Feature/lxc container runtime (#2514)

* wip: static-server errors

* wip: fix wifi

* wip: Fix the service_effects

* wip: Fix cors in the middleware

* wip(chore): Auth clean up the lint.

* wip(fix): Vhost

* wip: continue manager refactor

Co-authored-by: J H <Blu-J@users.noreply.github.com>

* wip: service manager refactor

* wip: Some fixes

* wip(fix): Fix the lib.rs

* wip

* wip(fix): Logs

* wip: bins

* wip(innspect): Add in the inspect

* wip: config

* wip(fix): Diagnostic

* wip(fix): Dependencies

* wip: context

* wip(fix) Sorta auth

* wip: warnings

* wip(fix): registry/admin

* wip(fix) marketplace

* wip(fix) Some more converted and fixed with the linter and config

* wip: Working on the static server

* wip(fix)static server

* wip: Remove some asynnc

* wip: Something about the request and regular rpc

* wip: gut install

Co-authored-by: J H <Blu-J@users.noreply.github.com>

* wip: Convert the static server into the new system

* wip delete file

* test

* wip(fix) vhost does not need the with safe defaults

* wip: Adding in the wifi

* wip: Fix the developer and the verify

* wip: new install flow

Co-authored-by: J H <Blu-J@users.noreply.github.com>

* fix middleware

* wip

* wip: Fix the auth

* wip

* continue service refactor

* feature: Service get_config

* feat: Action

* wip: Fighting the great fight against the borrow checker

* wip: Remove an error in a file that I just need to deel with later

* chore: Add in some more lifetime stuff to the services

* wip: Install fix on lifetime

* cleanup

* wip: Deal with the borrow later

* more cleanup

* resolve borrowchecker errors

* wip(feat): add in the handler for the socket, for now

* wip(feat): Update the service_effect_handler::action

* chore: Add in the changes to make sure the from_service goes to context

* chore: Change the

* refactor service map

* fix references to service map

* fill out restore

* wip: Before I work on the store stuff

* fix backup module

* handle some warnings

* feat: add in the ui components on the rust side

* feature: Update the procedures

* chore: Update the js side of the main and a few of the others

* chore: Update the rpc listener to match the persistant container

* wip: Working on updating some things to have a better name

* wip(feat): Try and get the rpc to return the correct shape?

* lxc wip

* wip(feat): Try and get the rpc to return the correct shape?

* build for container runtime wip

* remove container-init

* fix build

* fix error

* chore: Update to work I suppose

* lxc wip

* remove docker module and feature

* download alpine squashfs automatically

* overlays effect

Co-authored-by: Jade <Blu-J@users.noreply.github.com>

* chore: Add the overlay effect

* feat: Add the mounter in the main

* chore: Convert to use the mounts, still need to work with the sandbox

* install fixes

* fix ssl

* fixes from testing

* implement tmpfile for upload

* wip

* misc fixes

* cleanup

* cleanup

* better progress reporting

* progress for sideload

* return real guid

* add devmode script

* fix lxc rootfs path

* fix percentage bar

* fix progress bar styling

* fix build for unstable

* tweaks

* label progress

* tweaks

* update progress more often

* make symlink in rpc_client

* make socket dir

* fix parent path

* add start-cli to container

* add echo and gitInfo commands

* wip: Add the init + errors

* chore: Add in the exit effect for the system

* chore: Change the type to null for failure to parse

* move sigterm timeout to stopping status

* update order

* chore: Update the return type

* remove dbg

* change the map error

* chore: Update the thing to capture id

* chore add some life changes

* chore: Update the loging

* chore: Update the package to run module

* us From for RpcError

* chore: Update to use import instead

* chore: update

* chore: Use require for the backup

* fix a default

* update the type that is wrong

* chore: Update the type of the manifest

* chore: Update to make null

* only symlink if not exists

* get rid of double result

* better debug info for ErrorCollection

* chore: Update effects

* chore: fix

* mount assets and volumes

* add exec instead of spawn

* fix mounting in image

* fix overlay mounts

Co-authored-by: Jade <Blu-J@users.noreply.github.com>

* misc fixes

* feat: Fix two

* fix: systemForEmbassy main

* chore: Fix small part of main loop

* chore: Modify the bundle

* merge

* fixMain loop"

* move tsc to makefile

* chore: Update the return types of the health check

* fix client

* chore: Convert the todo to use tsmatches

* add in the fixes for the seen and create the hack to allow demo

* chore: Update to include the systemForStartOs

* chore UPdate to the latest types from the expected outout

* fixes

* fix typo

* Don't emit if failure on tsc

* wip

Co-authored-by: Jade <Blu-J@users.noreply.github.com>

* add s9pk api

* add inspection

* add inspect manifest

* newline after display serializable

* fix squashfs in image name

* edit manifest

Co-authored-by: Jade <Blu-J@users.noreply.github.com>

* wait for response on repl

* ignore sig for now

* ignore sig for now

* re-enable sig verification

* fix

* wip

* env and chroot

* add profiling logs

* set uid & gid in squashfs to 100000

* set uid of sqfs to 100000

* fix mksquashfs args

* add env to compat

* fix

* re-add docker feature flag

* fix docker output format being stupid

* here be dragons

* chore: Add in the cross compiling for something

* fix npm link

* extract logs from container on exit

* chore: Update for testing

* add log capture to drop trait

* chore: add in the modifications that I make

* chore: Update small things for no updates

* chore: Update the types of something

* chore: Make main not complain

* idmapped mounts

* idmapped volumes

* re-enable kiosk

* chore: Add in some logging for the new system

* bring in start-sdk

* remove avahi

* chore: Update the deps

* switch to musl

* chore: Update the version of prettier

* chore: Organize'

* chore: Update some of the headers back to the standard of fetch

* fix musl build

* fix idmapped mounts

* fix cross build

* use cross compiler for correct arch

* feat: Add in the faked ssl stuff for the effects

* @dr_bonez Did a solution here

* chore: Something that DrBonez

* chore: up

* wip: We have a working server!!!

* wip

* uninstall

* wip

* tes

---------

Co-authored-by: J H <dragondef@gmail.com>
Co-authored-by: J H <Blu-J@users.noreply.github.com>
Co-authored-by: J H <2364004+Blu-J@users.noreply.github.com>

core/startos/src/setup.rs

@@ -5,8 +5,8 @@ use std::time::Duration;
 use color_eyre::eyre::eyre;
 use josekit::jwk::Jwk;
 use openssl::x509::X509;
-use rpc_toolkit::command;
 use rpc_toolkit::yajrc::RpcError;
+use rpc_toolkit::{from_fn_async, HandlerExt, ParentHandler};
 use serde::{Deserialize, Serialize};
 use sqlx::Connection;
 use tokio::fs::File;
@@ -18,36 +18,55 @@ use tracing::instrument;
 use crate::account::AccountInfo;
 use crate::backup::restore::recover_full_embassy;
 use crate::backup::target::BackupTargetFS;
-use crate::context::rpc::RpcContextConfig;
 use crate::context::setup::SetupResult;
 use crate::context::SetupContext;
 use crate::disk::fsck::RepairStrategy;
 use crate::disk::main::DEFAULT_PASSWORD;
 use crate::disk::mount::filesystem::cifs::Cifs;
 use crate::disk::mount::filesystem::ReadWrite;
-use crate::disk::mount::guard::TmpMountGuard;
+use crate::disk::mount::guard::{GenericMountGuard, TmpMountGuard};
 use crate::disk::util::{pvscan, recovery_info, DiskInfo, EmbassyOsRecoveryInfo};
 use crate::disk::REPAIR_DISK_PATH;
 use crate::hostname::Hostname;
 use crate::init::{init, InitResult};
-use crate::middleware::encrypt::EncryptedWire;
 use crate::net::ssl::root_ca_start_time;
 use crate::prelude::*;
+use crate::util::crypto::EncryptedWire;
 use crate::util::io::{dir_copy, dir_size, Counter};
 use crate::{Error, ErrorKind, ResultExt};
 
-#[command(subcommands(status, disk, attach, execute, cifs, complete, get_pubkey, exit))]
-pub fn setup() -> Result<(), Error> {
-    Ok(())
+pub fn setup() -> ParentHandler {
+    ParentHandler::new()
+        .subcommand(
+            "status",
+            from_fn_async(status)
+                .with_metadata("authenticated", Value::Bool(false))
+                .no_cli(),
+        )
+        .subcommand("disk", disk())
+        .subcommand("attach", from_fn_async(attach).no_cli())
+        .subcommand("execute", from_fn_async(execute).no_cli())
+        .subcommand("cifs", cifs())
+        .subcommand("complete", from_fn_async(complete).no_cli())
+        .subcommand(
+            "get-pubkey",
+            from_fn_async(get_pubkey)
+                .with_metadata("authenticated", Value::Bool(false))
+                .no_cli(),
+        )
+        .subcommand("exit", from_fn_async(exit).no_cli())
 }
 
-#[command(subcommands(list_disks))]
-pub fn disk() -> Result<(), Error> {
-    Ok(())
+pub fn disk() -> ParentHandler {
+    ParentHandler::new().subcommand(
+        "list",
+        from_fn_async(list_disks)
+            .with_metadata("authenticated", Value::Bool(false))
+            .no_cli(),
+    )
 }
 
-#[command(rename = "list", rpc_only, metadata(authenticated = false))]
-pub async fn list_disks(#[context] ctx: SetupContext) -> Result<Vec<DiskInfo>, Error> {
+pub async fn list_disks(ctx: SetupContext) -> Result<Vec<DiskInfo>, Error> {
     crate::disk::util::list(&ctx.os_partitions).await
 }
 
@@ -55,8 +74,7 @@ async fn setup_init(
     ctx: &SetupContext,
     password: Option<String>,
 ) -> Result<(Hostname, OnionAddressV3, X509), Error> {
-    let InitResult { secret_store, db } =
-        init(&RpcContextConfig::load(ctx.config_path.clone()).await?).await?;
+    let InitResult { secret_store, db } = init(&ctx.config).await?;
     let mut secrets_handle = secret_store.acquire().await?;
     let mut secrets_tx = secrets_handle.begin().await?;
 
@@ -82,11 +100,17 @@ async fn setup_init(
     ))
 }
 
-#[command(rpc_only)]
+#[derive(Deserialize, Serialize)]
+#[serde(rename_all = "kebab-case")]
+pub struct AttachParams {
+    #[serde(rename = "embassy-password")]
+    password: Option<EncryptedWire>,
+    guid: Arc<String>,
+}
+
 pub async fn attach(
-    #[context] ctx: SetupContext,
-    #[arg] guid: Arc<String>,
-    #[arg(rename = "embassy-password")] password: Option<EncryptedWire>,
+    ctx: SetupContext,
+    AttachParams { password, guid }: AttachParams,
 ) -> Result<(), Error> {
     let mut status = ctx.setup_status.write().await;
     if status.is_some() {
@@ -169,8 +193,7 @@ pub struct SetupStatus {
     pub complete: bool,
 }
 
-#[command(rpc_only, metadata(authenticated = false))]
-pub async fn status(#[context] ctx: SetupContext) -> Result<Option<SetupStatus>, RpcError> {
+pub async fn status(ctx: SetupContext) -> Result<Option<SetupStatus>, RpcError> {
     ctx.setup_status.read().await.clone().transpose()
 }
 
@@ -178,25 +201,34 @@ pub async fn status(#[context] ctx: SetupContext) -> Result<Option<SetupStatus>,
 /// This way the frontend can send a secret, like the password for the setup/ recovory
 /// without knowing the password over clearnet. We use the public key shared across the network
 /// since it is fine to share the public, and encrypt against the public.
-#[command(rename = "get-pubkey", rpc_only, metadata(authenticated = false))]
-pub async fn get_pubkey(#[context] ctx: SetupContext) -> Result<Jwk, RpcError> {
+pub async fn get_pubkey(ctx: SetupContext) -> Result<Jwk, RpcError> {
     let secret = ctx.as_ref().clone();
     let pub_key = secret.to_public_key()?;
     Ok(pub_key)
 }
 
-#[command(subcommands(verify_cifs))]
-pub fn cifs() -> Result<(), Error> {
-    Ok(())
+pub fn cifs() -> ParentHandler {
+    ParentHandler::new().subcommand("verify", from_fn_async(verify_cifs).no_cli())
 }
 
-#[command(rename = "verify", rpc_only)]
+#[derive(Deserialize, Serialize)]
+#[serde(rename_all = "kebab-case")]
+pub struct VerifyCifsParams {
+    hostname: String,
+    path: PathBuf,
+    username: String,
+    password: Option<EncryptedWire>,
+}
+
+// #[command(rename = "verify", rpc_only)]
 pub async fn verify_cifs(
-    #[context] ctx: SetupContext,
-    #[arg] hostname: String,
-    #[arg] path: PathBuf,
-    #[arg] username: String,
-    #[arg] password: Option<EncryptedWire>,
+    ctx: SetupContext,
+    VerifyCifsParams {
+        hostname,
+        path,
+        username,
+        password,
+    }: VerifyCifsParams,
 ) -> Result<EmbassyOsRecoveryInfo, Error> {
     let password: Option<String> = password.map(|x| x.decrypt(&*ctx)).flatten();
     let guard = TmpMountGuard::mount(
@@ -209,12 +241,12 @@ pub async fn verify_cifs(
         ReadWrite,
     )
     .await?;
-    let embassy_os = recovery_info(&guard).await?;
+    let embassy_os = recovery_info(guard.path()).await?;
     guard.unmount().await?;
     embassy_os.ok_or_else(|| Error::new(eyre!("No Backup Found"), crate::ErrorKind::NotFound))
 }
 
-#[derive(Debug, Deserialize)]
+#[derive(Debug, Deserialize, Serialize)]
 #[serde(tag = "type")]
 #[serde(rename_all = "kebab-case")]
 pub enum RecoverySource {
@@ -222,13 +254,24 @@ pub enum RecoverySource {
     Backup { target: BackupTargetFS },
 }
 
-#[command(rpc_only)]
+#[derive(Deserialize, Serialize)]
+#[serde(rename_all = "kebab-case")]
+pub struct ExecuteParams {
+    embassy_logicalname: PathBuf,
+    embassy_password: EncryptedWire,
+    recovery_source: Option<RecoverySource>,
+    recovery_password: Option<EncryptedWire>,
+}
+
+// #[command(rpc_only)]
 pub async fn execute(
-    #[context] ctx: SetupContext,
-    #[arg(rename = "embassy-logicalname")] embassy_logicalname: PathBuf,
-    #[arg(rename = "embassy-password")] embassy_password: EncryptedWire,
-    #[arg(rename = "recovery-source")] recovery_source: Option<RecoverySource>,
-    #[arg(rename = "recovery-password")] recovery_password: Option<EncryptedWire>,
+    ctx: SetupContext,
+    ExecuteParams {
+        embassy_logicalname,
+        embassy_password,
+        recovery_source,
+        recovery_password,
+    }: ExecuteParams,
 ) -> Result<(), Error> {
     let embassy_password = match embassy_password.decrypt(&*ctx) {
         Some(a) => a,
@@ -312,8 +355,8 @@ pub async fn execute(
 }
 
 #[instrument(skip_all)]
-#[command(rpc_only)]
-pub async fn complete(#[context] ctx: SetupContext) -> Result<SetupResult, Error> {
+// #[command(rpc_only)]
+pub async fn complete(ctx: SetupContext) -> Result<SetupResult, Error> {
     let (guid, setup_result) = if let Some((guid, setup_result)) = &*ctx.setup_result.read().await {
         (guid.clone(), setup_result.clone())
     } else {
@@ -329,8 +372,8 @@ pub async fn complete(#[context] ctx: SetupContext) -> Result<SetupResult, Error
 }
 
 #[instrument(skip_all)]
-#[command(rpc_only)]
-pub async fn exit(#[context] ctx: SetupContext) -> Result<(), Error> {
+// #[command(rpc_only)]
+pub async fn exit(ctx: SetupContext) -> Result<(), Error> {
     ctx.shutdown.send(()).expect("failed to shutdown");
     Ok(())
 }
@@ -383,8 +426,7 @@ async fn fresh_setup(
     let sqlite_pool = ctx.secret_store().await?;
     account.save(&sqlite_pool).await?;
     sqlite_pool.close().await;
-    let InitResult { secret_store, .. } =
-        init(&RpcContextConfig::load(ctx.config_path.clone()).await?).await?;
+    let InitResult { secret_store, .. } = init(&ctx.config).await?;
     secret_store.close().await;
     Ok((
         account.hostname.clone(),