Feature/lxc container runtime (#2562)

* wip(fix): Dependencies

* wip: context

* wip(fix) Sorta auth

* wip: warnings

* wip(fix): registry/admin

* wip(fix) marketplace

* wip(fix) Some more converted and fixed with the linter and config

* wip: Working on the static server

* wip(fix)static server

* wip: Remove some asynnc

* wip: Something about the request and regular rpc

* wip: gut install

Co-authored-by: J H <Blu-J@users.noreply.github.com>

* wip: Convert the static server into the new system

* wip delete file

* test

* wip(fix) vhost does not need the with safe defaults

* wip: Adding in the wifi

* wip: Fix the developer and the verify

* wip: new install flow

Co-authored-by: J H <Blu-J@users.noreply.github.com>

* fix middleware

* wip

* wip: Fix the auth

* wip

* continue service refactor

* feature: Service get_config

* feat: Action

* wip: Fighting the great fight against the borrow checker

* wip: Remove an error in a file that I just need to deel with later

* chore: Add in some more lifetime stuff to the services

* wip: Install fix on lifetime

* cleanup

* wip: Deal with the borrow later

* more cleanup

* resolve borrowchecker errors

* wip(feat): add in the handler for the socket, for now

* wip(feat): Update the service_effect_handler::action

* chore: Add in the changes to make sure the from_service goes to context

* chore: Change the

* refactor service map

* fix references to service map

* fill out restore

* wip: Before I work on the store stuff

* fix backup module

* handle some warnings

* feat: add in the ui components on the rust side

* feature: Update the procedures

* chore: Update the js side of the main and a few of the others

* chore: Update the rpc listener to match the persistant container

* wip: Working on updating some things to have a better name

* wip(feat): Try and get the rpc to return the correct shape?

* lxc wip

* wip(feat): Try and get the rpc to return the correct shape?

* build for container runtime wip

* remove container-init

* fix build

* fix error

* chore: Update to work I suppose

* lxc wip

* remove docker module and feature

* download alpine squashfs automatically

* overlays effect

Co-authored-by: Jade <Blu-J@users.noreply.github.com>

* chore: Add the overlay effect

* feat: Add the mounter in the main

* chore: Convert to use the mounts, still need to work with the sandbox

* install fixes

* fix ssl

* fixes from testing

* implement tmpfile for upload

* wip

* misc fixes

* cleanup

* cleanup

* better progress reporting

* progress for sideload

* return real guid

* add devmode script

* fix lxc rootfs path

* fix percentage bar

* fix progress bar styling

* fix build for unstable

* tweaks

* label progress

* tweaks

* update progress more often

* make symlink in rpc_client

* make socket dir

* fix parent path

* add start-cli to container

* add echo and gitInfo commands

* wip: Add the init + errors

* chore: Add in the exit effect for the system

* chore: Change the type to null for failure to parse

* move sigterm timeout to stopping status

* update order

* chore: Update the return type

* remove dbg

* change the map error

* chore: Update the thing to capture id

* chore add some life changes

* chore: Update the loging

* chore: Update the package to run module

* us From for RpcError

* chore: Update to use import instead

* chore: update

* chore: Use require for the backup

* fix a default

* update the type that is wrong

* chore: Update the type of the manifest

* chore: Update to make null

* only symlink if not exists

* get rid of double result

* better debug info for ErrorCollection

* chore: Update effects

* chore: fix

* mount assets and volumes

* add exec instead of spawn

* fix mounting in image

* fix overlay mounts

Co-authored-by: Jade <Blu-J@users.noreply.github.com>

* misc fixes

* feat: Fix two

* fix: systemForEmbassy main

* chore: Fix small part of main loop

* chore: Modify the bundle

* merge

* fixMain loop"

* move tsc to makefile

* chore: Update the return types of the health check

* fix client

* chore: Convert the todo to use tsmatches

* add in the fixes for the seen and create the hack to allow demo

* chore: Update to include the systemForStartOs

* chore UPdate to the latest types from the expected outout

* fixes

* fix typo

* Don't emit if failure on tsc

* wip

Co-authored-by: Jade <Blu-J@users.noreply.github.com>

* add s9pk api

* add inspection

* add inspect manifest

* newline after display serializable

* fix squashfs in image name

* edit manifest

Co-authored-by: Jade <Blu-J@users.noreply.github.com>

* wait for response on repl

* ignore sig for now

* ignore sig for now

* re-enable sig verification

* fix

* wip

* env and chroot

* add profiling logs

* set uid & gid in squashfs to 100000

* set uid of sqfs to 100000

* fix mksquashfs args

* add env to compat

* fix

* re-add docker feature flag

* fix docker output format being stupid

* here be dragons

* chore: Add in the cross compiling for something

* fix npm link

* extract logs from container on exit

* chore: Update for testing

* add log capture to drop trait

* chore: add in the modifications that I make

* chore: Update small things for no updates

* chore: Update the types of something

* chore: Make main not complain

* idmapped mounts

* idmapped volumes

* re-enable kiosk

* chore: Add in some logging for the new system

* bring in start-sdk

* remove avahi

* chore: Update the deps

* switch to musl

* chore: Update the version of prettier

* chore: Organize'

* chore: Update some of the headers back to the standard of fetch

* fix musl build

* fix idmapped mounts

* fix cross build

* use cross compiler for correct arch

* feat: Add in the faked ssl stuff for the effects

* @dr_bonez Did a solution here

* chore: Something that DrBonez

* chore: up

* wip: We have a working server!!!

* wip

* uninstall

* wip

* tes

* misc fixes

* fix cli

* replace interface with host

* chore: Fix the types in some ts files

* chore: quick update for the system for embassy to update the types

* replace br-start9 with lxcbr0

* split patchdb into public/private

* chore: Add changes for config set

* Feat: Adding some debugging for the errors

* wip: Working on getting the set config to work

* chore: Update and fix the small issue with the deserialization

* lightning, masked, schemeOverride, invert host-iface relationship

* feat: Add in the changes for just the sdk

* feat: Add in the changes for the new effects I suppose for now

---------

Co-authored-by: J H <2364004+Blu-J@users.noreply.github.com>
Co-authored-by: J H <Blu-J@users.noreply.github.com>
Co-authored-by: J H <dragondef@gmail.com>
Co-authored-by: Matt Hill <mattnine@protonmail.com>

core/startos/src/version/mod.rs

@@ -70,8 +70,12 @@ where
         let semver = self.semver().into();
         let compat = self.compat().clone();
         db.mutate(|d| {
-            d.as_server_info_mut().as_version_mut().ser(&semver)?;
-            d.as_server_info_mut()
+            d.as_public_mut()
+                .as_server_info_mut()
+                .as_version_mut()
+                .ser(&semver)?;
+            d.as_public_mut()
+                .as_server_info_mut()
                 .as_eos_version_compat_mut()
                 .ser(&compat)?;
             Ok(())
@@ -166,7 +170,14 @@ where
 }
 
 pub async fn init(db: &PatchDb, secrets: &PgPool) -> Result<(), Error> {
-    let version = Version::from_util_version(db.peek().await.as_server_info().as_version().de()?);
+    let version = Version::from_util_version(
+        db.peek()
+            .await
+            .as_public()
+            .as_server_info()
+            .as_version()
+            .de()?,
+    );
 
     match version {
         Version::V0_3_4(v) => v.0.migrate_to(&Current::new(), db.clone(), secrets).await?,

core/startos/src/version/v0_3_4.rs

@@ -56,20 +56,23 @@ impl VersionT for Version {
         let mut account = AccountInfo::load(secrets).await?;
         let account = db
             .mutate(|d| {
-                d.as_server_info_mut().as_pubkey_mut().ser(
+                d.as_public_mut().as_server_info_mut().as_pubkey_mut().ser(
                     &ssh_key::PublicKey::from(Ed25519PublicKey::from(&account.key.ssh_key()))
                         .to_openssh()?,
                 )?;
-                d.as_server_info_mut().as_ca_fingerprint_mut().ser(
-                    &account
-                        .root_ca_cert
-                        .digest(MessageDigest::sha256())
-                        .unwrap()
-                        .iter()
-                        .map(|x| format!("{x:X}"))
-                        .join(":"),
-                )?;
-                let server_info = d.as_server_info();
+                d.as_public_mut()
+                    .as_server_info_mut()
+                    .as_ca_fingerprint_mut()
+                    .ser(
+                        &account
+                            .root_ca_cert
+                            .digest(MessageDigest::sha256())
+                            .unwrap()
+                            .iter()
+                            .map(|x| format!("{x:X}"))
+                            .join(":"),
+                    )?;
+                let server_info = d.as_public_mut().as_server_info();
                 account.hostname = server_info.as_hostname().de().map(Hostname)?;
                 account.server_id = server_info.as_id().de()?;
 
@@ -81,15 +84,16 @@ impl VersionT for Version {
 
         let parsed_url = Some(COMMUNITY_URL.parse().unwrap());
         db.mutate(|d| {
-            let mut ui = d.as_ui().de()?;
+            let mut ui = d.as_public().as_ui().de()?;
             use imbl_value::json;
             ui["marketplace"]["known-hosts"][COMMUNITY_URL] = json!({});
             ui["marketplace"]["known-hosts"][MAIN_REGISTRY] = json!({});
-            for package_id in d.as_package_data().keys()? {
+            for package_id in d.as_public().as_package_data().keys()? {
                 if !COMMUNITY_SERVICES.contains(&&*package_id.to_string()) {
                     continue;
                 }
-                d.as_package_data_mut()
+                d.as_public_mut()
+                    .as_package_data_mut()
                     .as_idx_mut(&package_id)
                     .or_not_found(&package_id)?
                     .as_installed_mut()
@@ -100,19 +104,20 @@ impl VersionT for Version {
             ui["theme"] = json!("Dark".to_string());
             ui["widgets"] = json!([]);
 
-            d.as_ui_mut().ser(&ui)
+            d.as_public_mut().as_ui_mut().ser(&ui)
         })
         .await
     }
     async fn down(&self, db: PatchDb, _secrets: &PgPool) -> Result<(), Error> {
         db.mutate(|d| {
-            let mut ui = d.as_ui().de()?;
+            let mut ui = d.as_public().as_ui().de()?;
             let parsed_url = Some(MAIN_REGISTRY.parse().unwrap());
-            for package_id in d.as_package_data().keys()? {
+            for package_id in d.as_public().as_package_data().keys()? {
                 if !COMMUNITY_SERVICES.contains(&&*package_id.to_string()) {
                     continue;
                 }
-                d.as_package_data_mut()
+                d.as_public_mut()
+                    .as_package_data_mut()
                     .as_idx_mut(&package_id)
                     .or_not_found(&package_id)?
                     .as_installed_mut()
@@ -128,7 +133,7 @@ impl VersionT for Version {
 
             ui["marketplace"]["known-hosts"][COMMUNITY_URL].take();
             ui["marketplace"]["known-hosts"][MAIN_REGISTRY].take();
-            d.as_ui_mut().ser(&ui)
+            d.as_public_mut().as_ui_mut().ser(&ui)
         })
         .await
     }

core/startos/src/version/v0_3_4_4.rs

@@ -26,7 +26,7 @@ impl VersionT for Version {
     }
     async fn up(&self, db: PatchDb, _secrets: &PgPool) -> Result<(), Error> {
         db.mutate(|v| {
-            let tor_address_lens = v.as_server_info_mut().as_tor_address_mut();
+            let tor_address_lens = v.as_public_mut().as_server_info_mut().as_tor_address_mut();
             let mut tor_addr = tor_address_lens.de()?;
             tor_addr
                 .set_scheme("https")

core/startos/src/version/v0_3_5.rs

@@ -30,7 +30,7 @@ impl VersionT for Version {
     async fn up(&self, db: PatchDb, _secrets: &PgPool) -> Result<(), Error> {
         let peek = db.peek().await;
         let mut url_replacements = BTreeMap::new();
-        for (_, pde) in peek.as_package_data().as_entries()? {
+        for (_, pde) in peek.as_public().as_package_data().as_entries()? {
             for (dependency, info) in pde
                 .as_installed()
                 .map(|i| i.as_dependency_info().as_entries())
@@ -63,7 +63,7 @@ impl VersionT for Version {
         }
         let prev_zram = db
             .mutate(|v| {
-                for (_, pde) in v.as_package_data_mut().as_entries_mut()? {
+                for (_, pde) in v.as_public_mut().as_package_data_mut().as_entries_mut()? {
                     for (dependency, info) in pde
                         .as_installed_mut()
                         .map(|i| i.as_dependency_info_mut().as_entries_mut())
@@ -95,7 +95,10 @@ impl VersionT for Version {
                         }
                     }
                 }
-                v.as_server_info_mut().as_zram_mut().replace(&true)
+                v.as_public_mut()
+                    .as_server_info_mut()
+                    .as_zram_mut()
+                    .replace(&true)
             })
             .await?;
         if !prev_zram {