0.4.0-alpha.8 (#2975)

* fix dependency input warning and extra comma

* clean up buttons during install in marketplace preview

* chore: grayscale and closing action-bar

* fix prerelease precedence

* fix duplicate url for addSsl on ssl proto

* no warning for soft uninstall

* fix: stop logs from repeating disconnected status and add 1 second delay between reconnection attempts

* fix stop on reactivation of critical task

* fix: fix disconnected toast

* fix: updates styles

* fix: updates styles

* misc fixes

* beta.33

* fix updates badge and initialization of marketplace preview controls

---------

Co-authored-by: waterplea <alexander@inkin.ru>
Co-authored-by: Aiden McClelland <me@drbonez.dev>

CONTRIBUTING.md

@@ -1,7 +1,6 @@
 # Contributing to StartOS
 
-This guide is for contributing to the StartOS. If you are interested in packaging a service for StartOS, visit the [service packaging guide](https://docs.start9.com/latest/developer-docs/). If you are interested in promoting, providing technical support, creating tutorials, or helping in other ways, please visit the [Start9 website](https://start9.com/contribute).
-
+This guide is for contributing to the StartOS. If you are interested in packaging a service for StartOS, visit the [service packaging guide](https://docs.start9.com/latest/packaging-guide/). If you are interested in promoting, providing technical support, creating tutorials, or helping in other ways, please visit the [Start9 website](https://start9.com/contribute).
 
 ## Collaboration
 
@@ -13,64 +12,77 @@ This guide is for contributing to the StartOS. If you are interested in packagin
 ```bash
 /
 ├── assets/
+├── container-runtime/
 ├── core/
 ├── build/
 ├── debian/
 ├── web/
 ├── image-recipe/
 ├── patch-db
-└── system-images/
+└── sdk/
 ```
+
 #### assets
+
 screenshots for the StartOS README
 
+#### container-runtime
+
+A NodeJS program that dynamically loads maintainer scripts and communicates with the OS to manage packages
+
 #### core
-An API, daemon (startd), CLI (start-cli), and SDK (start-sdk) that together provide the core functionality of StartOS.
+
+An API, daemon (startd), and CLI (start-cli) that together provide the core functionality of StartOS.
 
 #### build
+
 Auxiliary files and scripts to include in deployed StartOS images
 
 #### debian
+
 Maintainer scripts for the StartOS Debian package
 
 #### web
+
 Web UIs served under various conditions and used to interact with StartOS APIs.
 
 #### image-recipe
+
 Scripts for building StartOS images
 
 #### patch-db (submodule)
+
 A diff based data store used to synchronize data between the web interfaces and server.
 
-#### system-images
-Docker images that assist with creating backups.
+#### sdk
+
+A typescript sdk for building start-os packages
 
 ## Environment Setup
 
 #### Clone the StartOS repository
+
 ```sh
-git clone https://github.com/Start9Labs/start-os.git
+git clone https://github.com/Start9Labs/start-os.git --recurse-submodules
 cd start-os
 ```
 
-#### Load the PatchDB submodule
-```sh
-git submodule update --init --recursive
-```
-
 #### Continue to your project of interest for additional instructions:
+
 - [`core`](core/README.md)
 - [`web-interfaces`](web-interfaces/README.md)
 - [`build`](build/README.md)
 - [`patch-db`](https://github.com/Start9Labs/patch-db)
 
 ## Building
+
 This project uses [GNU Make](https://www.gnu.org/software/make/) to build its components. To build any specific component, simply run `make <TARGET>` replacing `<TARGET>` with the name of the target you'd like to build
 
 ### Requirements
+
 - [GNU Make](https://www.gnu.org/software/make/)
 - [Docker](https://docs.docker.com/get-docker/)
-- [NodeJS v18.15.0](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm)
+- [NodeJS v20.16.0](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm)
 - [sed](https://www.gnu.org/software/sed/)
 - [grep](https://www.gnu.org/software/grep/)
 - [awk](https://www.gnu.org/software/gawk/)
@@ -79,41 +91,43 @@ This project uses [GNU Make](https://www.gnu.org/software/make/) to build its co
 - [brotli](https://github.com/google/brotli)
 
 ### Environment variables
+
 - `PLATFORM`: which platform you would like to build for. Must be one of `x86_64`, `x86_64-nonfree`, `aarch64`, `aarch64-nonfree`, `raspberrypi`
-    - NOTE: `nonfree` images are for including `nonfree` firmware packages in the built ISO
+  - NOTE: `nonfree` images are for including `nonfree` firmware packages in the built ISO
 - `ENVIRONMENT`: a hyphen separated set of feature flags to enable
-    - `dev`: enables password ssh (INSECURE!) and does not compress frontends
-    - `unstable`: enables assertions that will cause errors on unexpected inconsistencies that are undesirable in production use either for performance or reliability reasons
-    - `docker`: use `docker` instead of `podman`
+  - `dev`: enables password ssh (INSECURE!) and does not compress frontends
+  - `unstable`: enables assertions that will cause errors on unexpected inconsistencies that are undesirable in production use either for performance or reliability reasons
+  - `docker`: use `docker` instead of `podman`
 - `GIT_BRANCH_AS_HASH`: set to `1` to use the current git branch name as the git hash so that the project does not need to be rebuilt on each commit
 
 ### Useful Make Targets
+
 - `iso`: Create a full `.iso` image
-    - Only possible from Debian
-    - Not available for `PLATFORM=raspberrypi`
-    - Additional Requirements:
-        - [debspawn](https://github.com/lkhq/debspawn)
+  - Only possible from Debian
+  - Not available for `PLATFORM=raspberrypi`
+  - Additional Requirements:
+    - [debspawn](https://github.com/lkhq/debspawn)
 - `img`: Create a full `.img` image
-    - Only possible from Debian
-    - Only available for `PLATFORM=raspberrypi`
-    - Additional Requirements:
-        - [debspawn](https://github.com/lkhq/debspawn)
+  - Only possible from Debian
+  - Only available for `PLATFORM=raspberrypi`
+  - Additional Requirements:
+    - [debspawn](https://github.com/lkhq/debspawn)
 - `format`: Run automatic code formatting for the project
-    - Additional Requirements:
-        - [rust](https://rustup.rs/)
+  - Additional Requirements:
+    - [rust](https://rustup.rs/)
 - `test`: Run automated tests for the project
-    - Additional Requirements:
-        - [rust](https://rustup.rs/)
+  - Additional Requirements:
+    - [rust](https://rustup.rs/)
 - `update`: Deploy the current working project to a device over ssh as if through an over-the-air update
-    - Requires an argument `REMOTE` which is the ssh address of the device, i.e. `start9@192.168.122.2`
+  - Requires an argument `REMOTE` which is the ssh address of the device, i.e. `start9@192.168.122.2`
 - `reflash`: Deploy the current working project to a device over ssh as if using a live `iso` image to reflash it
-    - Requires an argument `REMOTE` which is the ssh address of the device, i.e. `start9@192.168.122.2`
+  - Requires an argument `REMOTE` which is the ssh address of the device, i.e. `start9@192.168.122.2`
 - `update-overlay`: Deploy the current working project to a device over ssh to the in-memory overlay without restarting it
-    - WARNING: changes will be reverted after the device is rebooted
-    - WARNING: changes to `init` will not take effect as the device is already initialized
-    - Requires an argument `REMOTE` which is the ssh address of the device, i.e. `start9@192.168.122.2`
+  - WARNING: changes will be reverted after the device is rebooted
+  - WARNING: changes to `init` will not take effect as the device is already initialized
+  - Requires an argument `REMOTE` which is the ssh address of the device, i.e. `start9@192.168.122.2`
 - `wormhole`: Deploy the `startbox` to a device using [magic-wormhole](https://github.com/magic-wormhole/magic-wormhole)
-    - When the build it complete will emit a command to paste into the shell of the device to upgrade it
-    - Additional Requirements:
-        - [magic-wormhole](https://github.com/magic-wormhole/magic-wormhole)
-- `clean`: Delete all compiled artifacts
+  - When the build it complete will emit a command to paste into the shell of the device to upgrade it
+  - Additional Requirements:
+    - [magic-wormhole](https://github.com/magic-wormhole/magic-wormhole)
+- `clean`: Delete all compiled artifacts

DEVELOPMENT.md

@@ -25,15 +25,15 @@ docker buildx create --use
 curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh # proceed with default installation
 curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/master/install.sh | bash
 source ~/.bashrc
-nvm install 20
-nvm use 20
-nvm alias default 20 # this prevents your machine from reverting back to another version
+nvm install 22
+nvm use 22
+nvm alias default 22 # this prevents your machine from reverting back to another version
 ```
 
 ## Cloning the repository
 
 ```sh
-git clone --recursive https://github.com/Start9Labs/start-os.git --branch next/minor
+git clone --recursive https://github.com/Start9Labs/start-os.git --branch next/major
 cd start-os
 ```
 

Makefile

@@ -17,7 +17,7 @@ COMPAT_SRC := $(shell git ls-files system-images/compat/)
 UTILS_SRC := $(shell git ls-files system-images/utils/)
 BINFMT_SRC := $(shell git ls-files system-images/binfmt/)
 CORE_SRC := $(shell git ls-files core) $(shell git ls-files --recurse-submodules patch-db) $(GIT_HASH_FILE)
-WEB_SHARED_SRC := $(shell git ls-files web/projects/shared) $(shell ls -p web/ | grep -v / | sed 's/^/web\//g') web/node_modules/.package-lock.json web/config.json patch-db/client/dist/index.js sdk/baseDist/package.json web/patchdb-ui-seed.json sdk/dist/package.json
+WEB_SHARED_SRC := $(shell git ls-files web/projects/shared) $(shell git ls-files web/projects/marketplace) $(shell ls -p web/ | grep -v / | sed 's/^/web\//g') web/node_modules/.package-lock.json web/config.json patch-db/client/dist/index.js sdk/baseDist/package.json web/patchdb-ui-seed.json sdk/dist/package.json
 WEB_UI_SRC := $(shell git ls-files web/projects/ui)
 WEB_SETUP_WIZARD_SRC := $(shell git ls-files web/projects/setup-wizard)
 WEB_INSTALL_WIZARD_SRC := $(shell git ls-files web/projects/install-wizard)
@@ -49,7 +49,7 @@ endif
 
 .DELETE_ON_ERROR:
 
-.PHONY: all metadata install clean format cli uis ui reflash deb $(IMAGE_TYPE) squashfs sudo wormhole wormhole-deb test test-core test-sdk test-container-runtime registry
+.PHONY: all metadata install clean format cli uis ui reflash deb $(IMAGE_TYPE) squashfs wormhole wormhole-deb test test-core test-sdk test-container-runtime registry
 
 all: $(ALL_TARGETS)
 
@@ -58,9 +58,6 @@ touch:
 
 metadata: $(VERSION_FILE) $(PLATFORM_FILE) $(ENVIRONMENT_FILE) $(GIT_HASH_FILE)
 
-sudo:
-	sudo true
-
 clean:
 	rm -f system-images/**/*.tar
 	rm -rf system-images/compat/target
@@ -116,14 +113,14 @@ debian/control: build/lib/depends build/lib/conflicts
 	./debuild/control.sh
 
 results/$(BASENAME).deb: dpkg-build.sh $(DEBIAN_SRC) $(ALL_TARGETS)
-	PLATFORM=$(PLATFORM) ./dpkg-build.sh
+	PLATFORM=$(PLATFORM) REQUIRES=debian ./build/os-compat/run-compat.sh ./dpkg-build.sh
 
 $(IMAGE_TYPE): results/$(BASENAME).$(IMAGE_TYPE)
 
 squashfs: results/$(BASENAME).squashfs
 
 results/$(BASENAME).$(IMAGE_TYPE) results/$(BASENAME).squashfs: $(IMAGE_RECIPE_SRC) results/$(BASENAME).deb
-	./image-recipe/run-local-build.sh "results/$(BASENAME).deb"
+	REQUIRES=debian ./build/os-compat/run-compat.sh ./image-recipe/run-local-build.sh "results/$(BASENAME).deb"
 
 # For creating os images. DO NOT USE
 install: $(ALL_TARGETS) 
@@ -182,7 +179,7 @@ wormhole-squashfs: results/$(BASENAME).squashfs
 	$(eval SQFS_SIZE := $(shell du -s --bytes results/$(BASENAME).squashfs | awk '{print $$1}'))
 	@echo "Paste the following command into the shell of your StartOS server:"
 	@echo
-	@wormhole send results/$(BASENAME).squashfs 2>&1 | awk -Winteractive '/wormhole receive/ { printf "sudo sh -c '"'"'/usr/lib/startos/scripts/prune-images $(SQFS_SIZE) && cd /media/startos/images && wormhole receive --accept-file %s && mv $(BASENAME).squashfs $(SQFS_SUM).rootfs && ln -rsf ./$(SQFS_SUM).rootfs ../config/current.rootfs && sync && reboot'"'"'\n", $$3 }'
+	@wormhole send results/$(BASENAME).squashfs 2>&1 | awk -Winteractive '/wormhole receive/ { printf "sudo sh -c '"'"'/usr/lib/startos/scripts/prune-images $(SQFS_SIZE) && /usr/lib/startos/scripts/prune-boot && cd /media/startos/images && wormhole receive --accept-file %s && CHECKSUM=$(SQFS_SUM) /usr/lib/startos/scripts/use-img ./$(BASENAME).squashfs'"'"'\n", $$3 }'
 
 update: $(ALL_TARGETS)
 	@if [ -z "$(REMOTE)" ]; then >&2 echo "Must specify REMOTE" && false; fi
@@ -208,9 +205,9 @@ update-squashfs: results/$(BASENAME).squashfs
 	$(eval SQFS_SUM := $(shell b3sum results/$(BASENAME).squashfs))
 	$(eval SQFS_SIZE := $(shell du -s --bytes results/$(BASENAME).squashfs | awk '{print $$1}'))
 	$(call ssh,'/usr/lib/startos/scripts/prune-images $(SQFS_SIZE)')
-	$(call cp,results/$(BASENAME).squashfs,/media/startos/images/$(SQFS_SUM).rootfs)
-	$(call ssh,'sudo ln -rsf /media/startos/images/$(SQFS_SUM).rootfs /media/startos/config/current.rootfs')
-	$(call ssh,'sudo reboot')
+	$(call ssh,'/usr/lib/startos/scripts/prune-boot')
+	$(call cp,results/$(BASENAME).squashfs,/media/startos/images/next.rootfs)
+	$(call ssh,'sudo CHECKSUM=$(SQFS_SUM) /usr/lib/startos/scripts/use-img /media/startos/images/next.rootfs')
 
 emulate-reflash: $(ALL_TARGETS)
 	@if [ -z "$(REMOTE)" ]; then >&2 echo "Must specify REMOTE" && false; fi
@@ -222,10 +219,14 @@ emulate-reflash: $(ALL_TARGETS)
 upload-ota: results/$(BASENAME).squashfs
 	TARGET=$(TARGET) KEY=$(KEY) ./upload-ota.sh
 
-container-runtime/debian.$(ARCH).squashfs:
+container-runtime/debian.$(ARCH).squashfs: ./container-runtime/download-base-image.sh
 	ARCH=$(ARCH) ./container-runtime/download-base-image.sh
 
-container-runtime/node_modules/.package-lock.json: container-runtime/package.json container-runtime/package-lock.json sdk/dist/package.json
+container-runtime/package-lock.json: sdk/dist/package.json
+	npm --prefix container-runtime i
+	touch container-runtime/package-lock.json
+
+container-runtime/node_modules/.package-lock.json: container-runtime/package-lock.json
 	npm --prefix container-runtime ci
 	touch container-runtime/node_modules/.package-lock.json
 
@@ -254,8 +255,8 @@ container-runtime/dist/node_modules/.package-lock.json container-runtime/dist/pa
 	./container-runtime/install-dist-deps.sh
 	touch container-runtime/dist/node_modules/.package-lock.json
 
-container-runtime/rootfs.$(ARCH).squashfs: container-runtime/debian.$(ARCH).squashfs container-runtime/container-runtime.service container-runtime/update-image.sh container-runtime/deb-install.sh container-runtime/dist/index.js container-runtime/dist/node_modules/.package-lock.json core/target/$(ARCH)-unknown-linux-musl/release/containerbox | sudo
-	ARCH=$(ARCH) ./container-runtime/update-image.sh
+container-runtime/rootfs.$(ARCH).squashfs: container-runtime/debian.$(ARCH).squashfs container-runtime/container-runtime.service container-runtime/update-image.sh container-runtime/deb-install.sh container-runtime/dist/index.js container-runtime/dist/node_modules/.package-lock.json core/target/$(ARCH)-unknown-linux-musl/release/containerbox
+	ARCH=$(ARCH) REQUIRES=linux ./build/os-compat/run-compat.sh ./container-runtime/update-image.sh
 
 build/lib/depends build/lib/conflicts: build/dpkg-deps/*
 	build/dpkg-deps/generate.sh
@@ -280,7 +281,11 @@ core/target/$(ARCH)-unknown-linux-musl/release/containerbox: $(CORE_SRC) $(ENVIR
 	ARCH=$(ARCH) ./core/build-containerbox.sh
 	touch core/target/$(ARCH)-unknown-linux-musl/release/containerbox
 
-web/node_modules/.package-lock.json: web/package.json sdk/baseDist/package.json
+web/package-lock.json: web/package.json sdk/baseDist/package.json
+	npm --prefix web i
+	touch web/package-lock.json
+
+web/node_modules/.package-lock.json: web/package-lock.json
 	npm --prefix web ci
 	touch web/node_modules/.package-lock.json
 
@@ -298,7 +303,7 @@ web/dist/raw/setup-wizard/index.html: $(WEB_SETUP_WIZARD_SRC) $(WEB_SHARED_SRC)
 	touch web/dist/raw/setup-wizard/index.html
 
 web/dist/raw/install-wizard/index.html: $(WEB_INSTALL_WIZARD_SRC) $(WEB_SHARED_SRC) web/.angular/.updated
-	npm --prefix web run build:install-wiz
+	npm --prefix web run build:install
 	touch web/dist/raw/install-wizard/index.html
 
 $(COMPRESSED_WEB_UIS): $(WEB_UIS) $(ENVIRONMENT_FILE)

README.md

@@ -13,9 +13,6 @@
   <a href="https://twitter.com/start9labs">
     <img alt="X (formerly Twitter) Follow" src="https://img.shields.io/twitter/follow/start9labs">
   </a>
-  <a href="https://mastodon.start9labs.com">
-    <img src="https://img.shields.io/mastodon/follow/000000001?domain=https%3A%2F%2Fmastodon.start9labs.com&label=Follow&style=social">
-  </a>
   <a href="https://matrix.to/#/#community:matrix.start9labs.com">
     <img alt="Static Badge" src="https://img.shields.io/badge/community-matrix-yellow?logo=matrix">
   </a>

build/dpkg-deps/depends

@@ -36,6 +36,7 @@ man-db
 ncdu
 net-tools
 network-manager
+nfs-common
 nvme-cli
 nyx
 openssh-server
@@ -43,6 +44,7 @@ podman
 postgresql
 psmisc
 qemu-guest-agent
+rfkill
 rsync
 samba-common-bin
 smartmontools

build/lib/scripts/check-monitor

@@ -1,8 +0,0 @@
-#!/bin/sh
-
-
-if cat /sys/class/drm/*/status | grep -qw connected; then
-    exit 0
-else
-    exit 1
-fi

build/lib/scripts/enable-kiosk

@@ -91,15 +91,6 @@ cat > /home/kiosk/kiosk.sh << 'EOF'
 while ! curl "http://localhost" > /dev/null; do
     sleep 1
 done
-while ! /usr/lib/startos/scripts/check-monitor; do
-    sleep 15
-done
-(
-    while /usr/lib/startos/scripts/check-monitor; do
-        sleep 15
-    done
-    killall firefox-esr
-) &
 matchbox-window-manager -use_titlebar no &
 cp -r /home/kiosk/fx-profile /home/kiosk/fx-profile-tmp
 firefox-esr http://localhost --profile /home/kiosk/fx-profile-tmp

build/lib/scripts/forward-port

@@ -0,0 +1,26 @@
+#!/bin/bash
+
+if [ -z "$iiface" ] || [ -z "$oiface" ] || [ -z "$sip" ] || [ -z "$dip" ] || [ -z "$sport" ] || [ -z "$dport" ]; then
+    >&2 echo 'missing required env var'
+    exit 1
+fi
+
+kind="-A"
+
+if [ "$UNDO" = 1 ]; then
+    kind="-D"
+fi
+
+iptables -t nat "$kind" POSTROUTING -o $iiface -j MASQUERADE
+iptables -t nat "$kind" PREROUTING -i $iiface -p tcp --dport $sport -j DNAT --to-destination $dip:$dport
+iptables -t nat "$kind" PREROUTING -i $iiface -p udp --dport $sport -j DNAT --to-destination $dip:$dport
+iptables -t nat "$kind" PREROUTING -i $oiface -s $dip/24 -d $sip -p tcp --dport $sport -j DNAT --to-destination $dip:$dport
+iptables -t nat "$kind" PREROUTING -i $oiface -s $dip/24 -d $sip -p udp --dport $sport -j DNAT --to-destination $dip:$dport
+iptables -t nat "$kind" POSTROUTING -o $oiface -s $dip/24 -d $dip/32 -p tcp --dport $dport -j SNAT --to-source $sip:$sport
+iptables -t nat "$kind" POSTROUTING -o $oiface -s $dip/24 -d $dip/32 -p udp --dport $dport -j SNAT --to-source $sip:$sport
+
+
+iptables -t nat "$kind" PREROUTING -i $iiface -s $sip/32 -d $sip -p tcp --dport $sport -j DNAT --to-destination $dip:$dport
+iptables -t nat "$kind" PREROUTING -i $iiface -s $sip/32 -d $sip -p udp --dport $sport -j DNAT --to-destination $dip:$dport
+iptables -t nat "$kind" POSTROUTING -o $oiface -s $sip/32 -d $dip/32 -p tcp --dport $dport -j SNAT --to-source $sip:$sport
+iptables -t nat "$kind" POSTROUTING -o $oiface -s $sip/32 -d $dip/32 -p udp --dport $dport -j SNAT --to-source $sip:$sport

build/lib/scripts/prune-boot

@@ -0,0 +1,35 @@
+#!/bin/bash
+
+set -e
+
+if [ "$UID" -ne 0 ]; then
+    >&2 echo 'Must be run as root'
+    exit 1
+fi
+
+# Get the current kernel version
+current_kernel=$(uname -r)
+
+echo "Current kernel: $current_kernel"
+echo "Searching for old kernel files in /boot..."
+
+# Extract base kernel version (without possible suffixes)
+current_base=$(echo "$current_kernel" | sed 's/-.*//')
+
+cd /boot || { echo "/boot directory not found!"; exit 1; }
+
+for file in vmlinuz-* initrd.img-* System.map-* config-*; do
+    # Extract version from filename
+    version=$(echo "$file" | sed -E 's/^[^0-9]*([0-9][^ ]*).*/\1/')
+    # Skip if file matches current kernel version
+    if [[ "$file" == *"$current_kernel"* ]]; then
+        continue
+    fi
+    # Compare versions, delete if less than current
+    if dpkg --compare-versions "$version" lt "$current_kernel"; then
+        echo "Deleting $file (version $version is older than $current_kernel)"
+        sudo rm -f "$file"
+    fi
+done
+
+echo "Old kernel files deleted."

build/lib/scripts/use-img

@@ -0,0 +1,61 @@
+#!/bin/bash
+
+set -e
+
+if [ "$UID" -ne 0 ]; then
+    >&2 echo 'Must be run as root'
+    exit 1
+fi
+
+if [ -z "$1" ]; then
+    >&2 echo "usage: $0 <SQUASHFS>"
+    exit 1
+fi
+
+VERSION=$(unsquashfs -cat $1 /usr/lib/startos/VERSION.txt)
+GIT_HASH=$(unsquashfs -cat $1 /usr/lib/startos/GIT_HASH.txt)
+B3SUM=$(b3sum $1 | head -c 32)
+
+if [ -n "$CHECKSUM" ] && [ "$CHECKSUM" != "$B3SUM" ]; then
+    >&2 echo "CHECKSUM MISMATCH"
+    exit 2
+fi
+
+mv $1 /media/startos/images/${B3SUM}.rootfs
+ln -rsf /media/startos/images/${B3SUM}.rootfs /media/startos/config/current.rootfs
+
+unsquashfs -n -f -d / /media/startos/images/${B3SUM}.rootfs boot
+
+umount -R /media/startos/next 2> /dev/null || true
+umount -R /media/startos/lower 2> /dev/null || true
+umount -R /media/startos/upper 2> /dev/null || true
+
+rm -rf /media/startos/lower /media/startos/upper /media/startos/next
+mkdir /media/startos/upper
+mount -t tmpfs tmpfs /media/startos/upper
+mkdir -p /media/startos/lower /media/startos/upper/data /media/startos/upper/work /media/startos/next
+mount /media/startos/images/${B3SUM}.rootfs /media/startos/lower
+mount -t overlay \
+  -olowerdir=/media/startos/lower,upperdir=/media/startos/upper/data,workdir=/media/startos/upper/work \
+  overlay /media/startos/next
+mkdir -p /media/startos/next/media/startos/root
+mount --bind /media/startos/root /media/startos/next/media/startos/root
+mkdir -p /media/startos/next/dev
+mkdir -p /media/startos/next/sys
+mkdir -p /media/startos/next/proc
+mkdir -p /media/startos/next/boot
+mount --bind /dev /media/startos/next/dev
+mount --bind /sys /media/startos/next/sys
+mount --bind /proc /media/startos/next/proc
+mount --bind /boot /media/startos/next/boot
+
+chroot /media/startos/next update-grub2
+
+umount -R /media/startos/next
+umount -R /media/startos/upper
+umount -R /media/startos/lower
+rm -rf /media/startos/lower /media/startos/upper /media/startos/next
+
+sync
+
+reboot

build/os-compat/buildenv.Dockerfile

@@ -0,0 +1,56 @@
+FROM debian:bookworm
+
+RUN apt-get update && \
+    apt-get install -y \
+    ca-certificates \
+    curl \
+    gpg \
+    build-essential \
+    sed \
+    grep \
+    gawk \
+    jq \
+    gzip \
+    brotli \
+    qemu-user-static \
+    binfmt-support \
+    squashfs-tools \
+    git \
+    debspawn \
+    rsync \
+    b3sum \
+    fuse-overlayfs \
+    sudo \
+    systemd \
+    systemd-container \
+    systemd-sysv \
+    dbus \
+    dbus-user-session
+
+RUN systemctl mask \
+    systemd-firstboot.service \
+    systemd-udevd.service \
+    getty@tty1.service \
+    console-getty.service
+
+RUN git config --global --add safe.directory /root/start-os
+
+RUN mkdir -p /etc/debspawn && \
+    echo "AllowUnsafePermissions=true" > /etc/debspawn/global.toml
+
+ENV NVM_DIR=~/.nvm
+ENV NODE_VERSION=22
+RUN mkdir -p $NVM_DIR && \
+    curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/master/install.sh | bash && \
+    . $NVM_DIR/nvm.sh \
+    nvm install $NODE_VERSION && \
+    nvm use $NODE_VERSION && \
+    nvm alias default $NODE_VERSION && \
+    ln -s $(which node) /usr/bin/node && \
+    ln -s $(which npm) /usr/bin/npm
+
+RUN mkdir -p /root/start-os
+WORKDIR /root/start-os
+
+COPY docker-entrypoint.sh /docker-entrypoint.sh
+ENTRYPOINT [ "/docker-entrypoint.sh" ]

build/os-compat/docker-entrypoint.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+exec /lib/systemd/systemd --unit=multi-user.target --show-status=false --log-target=journal

build/os-compat/run-compat.sh

@@ -0,0 +1,27 @@
+#!/bin/bash
+
+if [ "$FORCE_COMPAT" = 1 ] || ( [ "$REQUIRES" = "linux" ] && [ "$(uname -s)" != "Linux" ] ) || ( [ "$REQUIRES" = "debian" ] && ! which dpkg > /dev/null ); then
+    project_pwd="$(cd "$(dirname "${BASH_SOURCE[0]}")"/../.. && pwd)/"
+    pwd="$(pwd)/"
+    if ! [[ "$pwd" = "$project_pwd"* ]]; then
+        >&2 echo "Must be run from start-os project dir"
+        exit 1
+    fi
+    rel_pwd="${pwd#"$project_pwd"}"
+
+    SYSTEMD_TTY="-P"
+    USE_TTY=
+    if tty -s; then
+        USE_TTY="-it"
+        SYSTEMD_TTY="-t"
+    fi
+
+    docker run -d --rm --name os-compat --privileged --security-opt apparmor=unconfined -v "${project_pwd}:/root/start-os" -v /lib/modules:/lib/modules:ro start9/build-env
+    while ! docker exec os-compat systemctl is-active --quiet multi-user.target 2> /dev/null; do sleep .5; done
+    docker exec -eARCH -eENVIRONMENT -ePLATFORM -eGIT_BRANCH_AS_HASH $USE_TTY -w "/root/start-os${rel_pwd}" os-compat $@
+    code=$?
+    docker stop os-compat
+    exit $code
+else 
+    exec $@
+fi

check-git-hash.sh

@@ -7,6 +7,7 @@ else
 fi
 
 if ! [ -f ./GIT_HASH.txt ] || [ "$(cat ./GIT_HASH.txt)" != "$GIT_HASH" ]; then
+    >&2 echo Git hash changed from "$([ -f ./GIT_HASH.txt ] && cat ./GIT_HASH.txt)" to "$GIT_HASH"
     echo -n "$GIT_HASH" > ./GIT_HASH.txt
 fi
 

container-runtime/container-runtime-failure.service

@@ -0,0 +1,6 @@
+[Unit]
+Description=StartOS Container Runtime Failure Handler
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/start-cli rebuild

container-runtime/container-runtime.service

@@ -1,11 +1,11 @@
 [Unit]
 Description=StartOS Container Runtime
+OnFailure=container-runtime-failure.service
 
 [Service]
 Type=simple
-ExecStart=/usr/bin/node --experimental-detect-module --unhandled-rejections=warn /usr/lib/startos/init/index.js
-Restart=always
-RestartSec=3
+ExecStart=/usr/bin/node --experimental-detect-module --trace-warnings --unhandled-rejections=warn /usr/lib/startos/init/index.js
+Restart=no
 
 [Install]
 WantedBy=multi-user.target

container-runtime/deb-install.sh

@@ -10,7 +10,7 @@ apt-get install -y curl rsync qemu-user-static
 
 curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash
 source ~/.bashrc
-nvm install 20
+nvm install 22
 ln -s $(which node) /usr/bin/node
 
 sed -i '/\(^\|#\)Storage=/c\Storage=persistent' /etc/systemd/journald.conf

container-runtime/download-base-image.sh

@@ -1,7 +1,5 @@
 #!/bin/bash
-
 cd "$(dirname "${BASH_SOURCE[0]}")"
-
 set -e
 
 DISTRO=debian
@@ -16,8 +14,9 @@ elif [ "$_ARCH" = "aarch64" ]; then
     _ARCH=arm64
 fi
 
-URL="https://images.linuxcontainers.org/$(curl -fsSL https://images.linuxcontainers.org/meta/1.0/index-system | grep "^$DISTRO;$VERSION;$_ARCH;$FLAVOR;" | head -n1 | sed 's/^.*;//g')/rootfs.squashfs"
+BASE_URL="https://images.linuxcontainers.org$(curl -fsSL https://images.linuxcontainers.org/meta/1.0/index-system | grep "^$DISTRO;$VERSION;$_ARCH;$FLAVOR;" | head -n1 | sed 's/^.*;//g')"
+OUTPUT_FILE="debian.${ARCH}.squashfs"
 
-echo "Downloading $URL to debian.${ARCH}.squashfs"
-
-curl -fsSL "$URL" > debian.${ARCH}.squashfs
+echo "Downloading ${BASE_URL}/rootfs.squashfs to $OUTPUT_FILE"
+curl -fsSL "${BASE_URL}/rootfs.squashfs" > "$OUTPUT_FILE"
+curl -fsSL "$BASE_URL/SHA256SUMS" | grep 'rootfs\.squashfs' | awk '{print $1"  '"$OUTPUT_FILE"'"}' | shasum -a 256 -c

container-runtime/package.json

@@ -26,8 +26,9 @@
     "isomorphic-fetch": "^3.0.0",
     "jsonpath": "^1.1.1",
     "lodash.merge": "^4.6.2",
+    "mime": "^4.0.7",
     "node-fetch": "^3.1.0",
-    "ts-matches": "^5.5.1",
+    "ts-matches": "^6.3.2",
     "tslib": "^2.5.3",
     "typescript": "^5.1.3",
     "yaml": "^2.3.1"

container-runtime/src/Adapters/EffectCreator.ts

@@ -1,4 +1,9 @@
-import { types as T, utils } from "@start9labs/start-sdk"
+import {
+  ExtendedVersion,
+  types as T,
+  utils,
+  VersionRange,
+} from "@start9labs/start-sdk"
 import * as net from "net"
 import { object, string, number, literals, some, unknown } from "ts-matches"
 import { Effects } from "../Models/Effects"
@@ -6,23 +11,19 @@ import { Effects } from "../Models/Effects"
 import { CallbackHolder } from "../Models/CallbackHolder"
 import { asError } from "@start9labs/start-sdk/base/lib/util"
 const matchRpcError = object({
-  error: object(
-    {
-      code: number,
-      message: string,
-      data: some(
-        string,
-        object(
-          {
-            details: string,
-            debug: string,
-          },
-          ["debug"],
-        ),
-      ),
-    },
-    ["data"],
-  ),
+  error: object({
+    code: number,
+    message: string,
+    data: some(
+      string,
+      object({
+        details: string,
+        debug: string.nullable().optional(),
+      }),
+    )
+      .nullable()
+      .optional(),
+  }),
 })
 const testRpcError = matchRpcError.test
 const testRpcResult = object({
@@ -104,7 +105,18 @@ const rpcRoundFor =
 export function makeEffects(context: EffectContext): Effects {
   const rpcRound = rpcRoundFor(context.procedureId)
   const self: Effects = {
+    child: (name) =>
+      makeEffects({ ...context, callbacks: context.callbacks?.child(name) }),
     constRetry: context.constRetry,
+    isInContext: !!context.callbacks,
+    onLeaveContext:
+      context.callbacks?.onLeaveContext?.bind(context.callbacks) ||
+      (() => {
+        console.warn(
+          "no context for this effects object",
+          new Error().stack?.replace(/^Error/, ""),
+        )
+      }),
     clearCallbacks(...[options]: Parameters<T.Effects["clearCallbacks"]>) {
       return rpcRound("clear-callbacks", {
         ...options,
@@ -126,22 +138,20 @@ export function makeEffects(context: EffectContext): Effects {
           ...options,
         }) as ReturnType<T.Effects["action"]["getInput"]>
       },
-      request(...[options]: Parameters<T.Effects["action"]["request"]>) {
-        return rpcRound("action.request", {
+      createTask(...[options]: Parameters<T.Effects["action"]["createTask"]>) {
+        return rpcRound("action.create-task", {
           ...options,
-        }) as ReturnType<T.Effects["action"]["request"]>
+        }) as ReturnType<T.Effects["action"]["createTask"]>
       },
       run(...[options]: Parameters<T.Effects["action"]["run"]>) {
         return rpcRound("action.run", {
           ...options,
         }) as ReturnType<T.Effects["action"]["run"]>
       },
-      clearRequests(
-        ...[options]: Parameters<T.Effects["action"]["clearRequests"]>
-      ) {
-        return rpcRound("action.clear-requests", {
+      clearTasks(...[options]: Parameters<T.Effects["action"]["clearTasks"]>) {
+        return rpcRound("action.clear-tasks", {
           ...options,
-        }) as ReturnType<T.Effects["action"]["clearRequests"]>
+        }) as ReturnType<T.Effects["action"]["clearTasks"]>
       },
     },
     bind(...[options]: Parameters<T.Effects["bind"]>) {
@@ -186,13 +196,6 @@ export function makeEffects(context: EffectContext): Effects {
         T.Effects["exportServiceInterface"]
       >
     }) as Effects["exportServiceInterface"],
-    exposeForDependents(
-      ...[options]: Parameters<T.Effects["exposeForDependents"]>
-    ) {
-      return rpcRound("expose-for-dependents", options) as ReturnType<
-        T.Effects["exposeForDependents"]
-      >
-    },
     getContainerIp(...[options]: Parameters<T.Effects["getContainerIp"]>) {
       return rpcRound("get-container-ip", options) as ReturnType<
         T.Effects["getContainerIp"]
@@ -254,6 +257,7 @@ export function makeEffects(context: EffectContext): Effects {
       return rpcRound("mount", options) as ReturnType<T.Effects["mount"]>
     },
     restart(...[]: Parameters<T.Effects["restart"]>) {
+      console.log("Restarting service...")
       return rpcRound("restart", {}) as ReturnType<T.Effects["restart"]>
     },
     setDependencies(
@@ -293,15 +297,6 @@ export function makeEffects(context: EffectContext): Effects {
     shutdown(...[]: Parameters<T.Effects["shutdown"]>) {
       return rpcRound("shutdown", {}) as ReturnType<T.Effects["shutdown"]>
     },
-    store: {
-      get: async (options: any) =>
-        rpcRound("store.get", {
-          ...options,
-          callback: context.callbacks?.addCallback(options.callback) || null,
-        }) as any,
-      set: async (options: any) =>
-        rpcRound("store.set", options) as ReturnType<T.Effects["store"]["set"]>,
-    } as T.Effects["store"],
     getDataVersion() {
       return rpcRound("get-data-version", {}) as ReturnType<
         T.Effects["getDataVersion"]
@@ -313,5 +308,15 @@ export function makeEffects(context: EffectContext): Effects {
       >
     },
   }
+  if (context.callbacks?.onLeaveContext)
+    self.onLeaveContext(() => {
+      self.isInContext = false
+      self.onLeaveContext = () => {
+        console.warn(
+          "this effects object is already out of context",
+          new Error().stack?.replace(/^Error/, ""),
+        )
+      }
+    })
   return self
 }

container-runtime/src/Adapters/RpcListener.ts

@@ -12,9 +12,15 @@ import {
   any,
   shape,
   anyOf,
+  literals,
 } from "ts-matches"
 
-import { types as T, utils } from "@start9labs/start-sdk"
+import {
+  ExtendedVersion,
+  types as T,
+  utils,
+  VersionRange,
+} from "@start9labs/start-sdk"
 import * as fs from "fs"
 
 import { CallbackHolder } from "../Models/CallbackHolder"
@@ -26,20 +32,16 @@ type MaybePromise<T> = T | Promise<T>
 export const matchRpcResult = anyOf(
   object({ result: any }),
   object({
-    error: object(
-      {
-        code: number,
-        message: string,
-        data: object(
-          {
-            details: string,
-            debug: any,
-          },
-          ["details", "debug"],
-        ),
-      },
-      ["data"],
-    ),
+    error: object({
+      code: number,
+      message: string,
+      data: object({
+        details: string.optional(),
+        debug: any.optional(),
+      })
+        .nullable()
+        .optional(),
+    }),
   }),
 )
 
@@ -54,38 +56,26 @@ const isResult = object({ result: any }).test
 
 const idType = some(string, number, literal(null))
 type IdType = null | string | number | undefined
-const runType = object(
-  {
-    id: idType,
-    method: literal("execute"),
-    params: object(
-      {
-        id: string,
-        procedure: string,
-        input: any,
-        timeout: number,
-      },
-      ["timeout"],
-    ),
-  },
-  ["id"],
-)
-const sandboxRunType = object(
-  {
-    id: idType,
-    method: literal("sandbox"),
-    params: object(
-      {
-        id: string,
-        procedure: string,
-        input: any,
-        timeout: number,
-      },
-      ["timeout"],
-    ),
-  },
-  ["id"],
-)
+const runType = object({
+  id: idType.optional(),
+  method: literal("execute"),
+  params: object({
+    id: string,
+    procedure: string,
+    input: any,
+    timeout: number.nullable().optional(),
+  }),
+})
+const sandboxRunType = object({
+  id: idType.optional(),
+  method: literal("sandbox"),
+  params: object({
+    id: string,
+    procedure: string,
+    input: any,
+    timeout: number.nullable().optional(),
+  }),
+})
 const callbackType = object({
   method: literal("callback"),
   params: object({
@@ -93,44 +83,37 @@ const callbackType = object({
     args: array,
   }),
 })
-const initType = object(
-  {
-    id: idType,
-    method: literal("init"),
-  },
-  ["id"],
-)
-const startType = object(
-  {
-    id: idType,
-    method: literal("start"),
-  },
-  ["id"],
-)
-const stopType = object(
-  {
-    id: idType,
-    method: literal("stop"),
-  },
-  ["id"],
-)
-const exitType = object(
-  {
-    id: idType,
-    method: literal("exit"),
-  },
-  ["id"],
-)
-const evalType = object(
-  {
-    id: idType,
-    method: literal("eval"),
-    params: object({
-      script: string,
-    }),
-  },
-  ["id"],
-)
+const initType = object({
+  id: idType.optional(),
+  method: literal("init"),
+  params: object({
+    id: string,
+    kind: literals("install", "update", "restore").nullable(),
+  }),
+})
+const startType = object({
+  id: idType.optional(),
+  method: literal("start"),
+})
+const stopType = object({
+  id: idType.optional(),
+  method: literal("stop"),
+})
+const exitType = object({
+  id: idType.optional(),
+  method: literal("exit"),
+  params: object({
+    id: string,
+    target: string.nullable(),
+  }),
+})
+const evalType = object({
+  id: idType.optional(),
+  method: literal("eval"),
+  params: object({
+    script: string,
+  }),
+})
 
 const jsonParse = (x: string) => JSON.parse(x)
 
@@ -171,6 +154,8 @@ export class RpcListener {
     if (!fs.existsSync(SOCKET_PARENT)) {
       fs.mkdirSync(SOCKET_PARENT, { recursive: true })
     }
+    if (fs.existsSync(SOCKET_PATH)) fs.rmSync(SOCKET_PATH, { force: true })
+
     this.unixSocketServer.listen(SOCKET_PATH)
 
     this.unixSocketServer.on("connection", (s) => {
@@ -238,21 +223,6 @@ export class RpcListener {
     return this._system
   }
 
-  private callbackHolders: Map<string, CallbackHolder> = new Map()
-  private removeCallbackHolderFor(procedure: string) {
-    const prev = this.callbackHolders.get(procedure)
-    if (prev) {
-      this.callbackHolders.delete(procedure)
-      this.callbacks?.removeChild(prev)
-    }
-  }
-  private callbackHolderFor(procedure: string): CallbackHolder {
-    this.removeCallbackHolderFor(procedure)
-    const callbackHolder = this.callbacks!.child()
-    this.callbackHolders.set(procedure, callbackHolder)
-    return callbackHolder
-  }
-
   callCallback(callback: number, args: any[]): void {
     if (this.callbacks) {
       this.callbacks
@@ -302,7 +272,8 @@ export class RpcListener {
         return null
       })
       .when(startType, async ({ id }) => {
-        const callbacks = this.callbackHolderFor("main")
+        const callbacks =
+          this.callbacks?.getChild("main") || this.callbacks?.child("main")
         const effects = makeEffects({
           procedureId: null,
           callbacks,
@@ -313,21 +284,35 @@ export class RpcListener {
         )
       })
       .when(stopType, async ({ id }) => {
-        this.removeCallbackHolderFor("main")
+        this.callbacks?.removeChild("main")
         return handleRpc(
           id,
           this.system.stop().then((result) => ({ result })),
         )
       })
-      .when(exitType, async ({ id }) => {
+      .when(exitType, async ({ id, params }) => {
         return handleRpc(
           id,
           (async () => {
-            if (this._system) await this._system.exit()
+            if (this._system) {
+              let target = null
+              if (params.target)
+                try {
+                  target = ExtendedVersion.parse(params.target)
+                } catch (_) {
+                  target = VersionRange.parse(params.target).normalize()
+                }
+              await this._system.exit(
+                makeEffects({
+                  procedureId: params.id,
+                }),
+                target,
+              )
+            }
           })().then((result) => ({ result })),
         )
       })
-      .when(initType, async ({ id }) => {
+      .when(initType, async ({ id, params }) => {
         return handleRpc(
           id,
           (async () => {
@@ -335,16 +320,19 @@ export class RpcListener {
               const system = await this.getDependencies.system()
               this.callbacks = new CallbackHolder(
                 makeEffects({
-                  procedureId: null,
+                  procedureId: params.id,
                 }),
               )
-              const callbacks = this.callbackHolderFor("containerInit")
-              await system.containerInit(
+              const callbacks = this.callbacks.child("init")
+              console.error("Initializing...")
+              await system.init(
                 makeEffects({
-                  procedureId: null,
+                  procedureId: params.id,
                   callbacks,
                 }),
+                params.kind,
               )
+              console.error("Initialization complete.")
               this._system = system
             }
           })().then((result) => ({ result })),
@@ -377,7 +365,7 @@ export class RpcListener {
         )
       })
       .when(
-        shape({ id: idType, method: string }, ["id"]),
+        shape({ id: idType.optional(), method: string }),
         ({ id, method }) => ({
           jsonrpc,
           id,
@@ -412,7 +400,7 @@ export class RpcListener {
     procedure: typeof jsonPath._TYPE,
     system: System,
     procedureId: string,
-    timeout: number | undefined,
+    timeout: number | null | undefined,
     input: any,
   ) {
     const ensureResultTypeShape = (
@@ -420,7 +408,7 @@ export class RpcListener {
     ): { result: any } => {
       return { result }
     }
-    const callbacks = this.callbackHolderFor(procedure)
+    const callbacks = this.callbacks?.child(procedure)
     const effects = makeEffects({
       procedureId,
       callbacks,
@@ -430,16 +418,6 @@ export class RpcListener {
       switch (procedure) {
         case "/backup/create":
           return system.createBackup(effects, timeout || null)
-        case "/backup/restore":
-          return system.restoreBackup(effects, timeout || null)
-        case "/packageInit":
-          return system.packageInit(effects, timeout || null)
-        case "/packageUninit":
-          return system.packageUninit(
-            effects,
-            string.optional().unsafeCast(input),
-            timeout || null,
-          )
         default:
           const procedures = unNestPath(procedure)
           switch (true) {
@@ -461,14 +439,10 @@ export class RpcListener {
     })().then(ensureResultTypeShape, (error) =>
       matches(error)
         .when(
-          object(
-            {
-              error: string,
-              code: number,
-            },
-            ["code"],
-            { code: 0 },
-          ),
+          object({
+            error: string,
+            code: number.defaultTo(0),
+          }),
           (error) => ({
             error: {
               code: error.code,

container-runtime/src/Adapters/Systems/SystemForEmbassy/DockerProcedureContainer.ts

@@ -7,13 +7,22 @@ import { Volume } from "./matchVolume"
 import {
   CommandOptions,
   ExecOptions,
-  ExecSpawnable,
+  SubContainerOwned,
 } from "@start9labs/start-sdk/package/lib/util/SubContainer"
+import { Mounts } from "@start9labs/start-sdk/package/lib/mainFn/Mounts"
+import { Manifest } from "@start9labs/start-sdk/base/lib/osBindings"
+import { BackupEffects } from "@start9labs/start-sdk/package/lib/backup/Backups"
+import { Drop } from "@start9labs/start-sdk/package/lib/util"
+import { SDKManifest } from "@start9labs/start-sdk/base/lib/types"
 export const exec = promisify(cp.exec)
 export const execFile = promisify(cp.execFile)
 
-export class DockerProcedureContainer {
-  private constructor(private readonly subcontainer: ExecSpawnable) {}
+export class DockerProcedureContainer extends Drop {
+  private constructor(
+    private readonly subcontainer: SubContainer<SDKManifest>,
+  ) {
+    super()
+  }
 
   static async of(
     effects: T.Effects,
@@ -21,7 +30,7 @@ export class DockerProcedureContainer {
     data: DockerProcedure,
     volumes: { [id: VolumeId]: Volume },
     name: string,
-    options: { subcontainer?: ExecSpawnable } = {},
+    options: { subcontainer?: SubContainer<SDKManifest> } = {},
   ) {
     const subcontainer =
       options?.subcontainer ??
@@ -41,9 +50,10 @@ export class DockerProcedureContainer {
     volumes: { [id: VolumeId]: Volume },
     name: string,
   ) {
-    const subcontainer = await SubContainer.of(
-      effects,
+    const subcontainer = await SubContainerOwned.of(
+      effects as BackupEffects,
       { imageId: data.image },
+      null,
       name,
     )
 
@@ -57,13 +67,19 @@ export class DockerProcedureContainer {
         const volumeMount = volumes[mount]
         if (volumeMount.type === "data") {
           await subcontainer.mount(
-            { type: "volume", id: mount, subpath: null, readonly: false },
-            mounts[mount],
+            Mounts.of().mountVolume({
+              volumeId: mount,
+              subpath: null,
+              mountpoint: mounts[mount],
+              readonly: false,
+            }),
           )
         } else if (volumeMount.type === "assets") {
           await subcontainer.mount(
-            { type: "assets", subpath: mount },
-            mounts[mount],
+            Mounts.of().mountAssets({
+              subpath: mount,
+              mountpoint: mounts[mount],
+            }),
           )
         } else if (volumeMount.type === "certificate") {
           const hostnames = [
@@ -95,21 +111,22 @@ export class DockerProcedureContainer {
             key,
           )
         } else if (volumeMount.type === "pointer") {
-          await effects
-            .mount({
-              location: path,
-              target: {
-                packageId: volumeMount["package-id"],
-                subpath: volumeMount.path,
-                readonly: volumeMount.readonly,
-                volumeId: volumeMount["volume-id"],
-              },
-            })
-            .catch(console.warn)
+          await effects.mount({
+            location: path,
+            target: {
+              packageId: volumeMount["package-id"],
+              subpath: volumeMount.path,
+              readonly: volumeMount.readonly,
+              volumeId: volumeMount["volume-id"],
+              filetype: "directory",
+            },
+          })
         } else if (volumeMount.type === "backup") {
           await subcontainer.mount(
-            { type: "backup", subpath: null },
-            mounts[mount],
+            Mounts.of().mountBackups({
+              subpath: null,
+              mountpoint: mounts[mount],
+            }),
           )
         }
       }
@@ -151,7 +168,11 @@ export class DockerProcedureContainer {
     }
   }
 
-  async spawn(commands: string[]): Promise<cp.ChildProcess> {
-    return await this.subcontainer.spawn(commands)
+  // async spawn(commands: string[]): Promise<cp.ChildProcess> {
+  //   return await this.subcontainer.spawn(commands)
+  // }
+
+  onDrop(): void {
+    this.subcontainer.destroy?.()
   }
 }

container-runtime/src/Adapters/Systems/SystemForEmbassy/MainLoop.ts

@@ -6,6 +6,8 @@ import { Daemon } from "@start9labs/start-sdk/package/lib/mainFn/Daemon"
 import { Effects } from "../../../Models/Effects"
 import { off } from "node:process"
 import { CommandController } from "@start9labs/start-sdk/package/lib/mainFn/CommandController"
+import { SDKManifest } from "@start9labs/start-sdk/base/lib/types"
+import { SubContainerRc } from "@start9labs/start-sdk/package/lib/util/SubContainer"
 
 const EMBASSY_HEALTH_INTERVAL = 15 * 1000
 const EMBASSY_PROPERTIES_LOOP = 30 * 1000
@@ -15,8 +17,13 @@ const EMBASSY_PROPERTIES_LOOP = 30 * 1000
  * Also, this has an ability to clean itself up too if need be.
  */
 export class MainLoop {
+  private subcontainerRc?: SubContainerRc<SDKManifest>
   get mainSubContainerHandle() {
-    return this.mainEvent?.daemon?.subContainerHandle
+    this.subcontainerRc =
+      this.subcontainerRc ??
+      this.mainEvent?.daemon?.subcontainerRc() ??
+      undefined
+    return this.subcontainerRc
   }
   private healthLoops?: {
     name: string
@@ -24,7 +31,7 @@ export class MainLoop {
   }[]
 
   private mainEvent?: {
-    daemon: Daemon
+    daemon: Daemon<SDKManifest>
   }
 
   private constructor(
@@ -55,28 +62,20 @@ export class MainLoop {
     if (jsMain) {
       throw new Error("Unreachable")
     }
-    const daemon = new Daemon(async () => {
-      const subcontainer = await DockerProcedureContainer.createSubContainer(
-        effects,
-        this.system.manifest.id,
-        this.system.manifest.main,
-        this.system.manifest.volumes,
-        `Main - ${currentCommand.join(" ")}`,
-      )
-      return CommandController.of()(
-        this.effects,
-        subcontainer,
-        currentCommand,
-        {
-          runAsInit: true,
-          env: {
-            TINI_SUBREAPER: "true",
-          },
-          sigtermTimeout: utils.inMs(
-            this.system.manifest.main["sigterm-timeout"],
-          ),
-        },
-      )
+    const subcontainer = await DockerProcedureContainer.createSubContainer(
+      effects,
+      this.system.manifest.id,
+      this.system.manifest.main,
+      this.system.manifest.volumes,
+      `Main - ${currentCommand.join(" ")}`,
+    )
+    const daemon = await Daemon.of()(this.effects, subcontainer, {
+      command: currentCommand,
+      runAsInit: true,
+      env: {
+        TINI_SUBREAPER: "true",
+      },
+      sigtermTimeout: utils.inMs(this.system.manifest.main["sigterm-timeout"]),
     })
 
     daemon.start()

container-runtime/src/Adapters/Systems/SystemForEmbassy/__fixtures__/rtlConfig.ts

@@ -0,0 +1,153 @@
+export default {
+  nodes: {
+    type: "list",
+    subtype: "union",
+    name: "Lightning Nodes",
+    description: "List of Lightning Network node instances to manage",
+    range: "[1,*)",
+    default: ["lnd"],
+    spec: {
+      type: "string",
+      "display-as": "{{name}}",
+      "unique-by": "name",
+      name: "Node Implementation",
+      tag: {
+        id: "type",
+        name: "Type",
+        description:
+          "- LND: Lightning Network Daemon from Lightning Labs\n- CLN: Core Lightning from Blockstream\n",
+        "variant-names": {
+          lnd: "Lightning Network Daemon (LND)",
+          "c-lightning": "Core Lightning (CLN)",
+        },
+      },
+      default: "lnd",
+      variants: {
+        lnd: {
+          name: {
+            type: "string",
+            name: "Node Name",
+            description: "Name of this node in the list",
+            default: "StartOS LND",
+            nullable: false,
+          },
+          "connection-settings": {
+            type: "union",
+            name: "Connection Settings",
+            description: "The Lightning Network Daemon node to connect to.",
+            tag: {
+              id: "type",
+              name: "Type",
+              description:
+                "- Internal: The Lightning Network Daemon service installed to your StartOS server.\n- External: A Lightning Network Daemon instance running on a remote device (advanced).\n",
+              "variant-names": {
+                internal: "Internal",
+                external: "External",
+              },
+            },
+            default: "internal",
+            variants: {
+              internal: {},
+              external: {
+                address: {
+                  type: "string",
+                  name: "Public Address",
+                  description:
+                    "The public address of your LND REST server\nNOTE: RTL does not support a .onion URL here\n",
+                  nullable: false,
+                },
+                "rest-port": {
+                  type: "number",
+                  name: "REST Port",
+                  description:
+                    "The port that your Lightning Network Daemon REST server is bound to",
+                  nullable: false,
+                  range: "[0,65535]",
+                  integral: true,
+                  default: 8080,
+                },
+                macaroon: {
+                  type: "string",
+                  name: "Macaroon",
+                  description:
+                    'Your admin.macaroon file, Base64URL encoded. This is the same as the value after "macaroon=" in your lndconnect URL.',
+                  nullable: false,
+                  masked: true,
+                  pattern: "[=A-Za-z0-9_-]+",
+                  "pattern-description":
+                    "Macaroon must be encoded in Base64URL format (only A-Z, a-z, 0-9, _, - and = allowed)",
+                },
+              },
+            },
+          },
+        },
+        "c-lightning": {
+          name: {
+            type: "string",
+            name: "Node Name",
+            description: "Name of this node in the list",
+            default: "StartOS CLN",
+            nullable: false,
+          },
+          "connection-settings": {
+            type: "union",
+            name: "Connection Settings",
+            description: "The Core Lightning (CLN) node to connect to.",
+            tag: {
+              id: "type",
+              name: "Type",
+              description:
+                "- Internal: The Core Lightning (CLN) service installed to your StartOS server.\n- External: A Core Lightning (CLN) instance running on a remote device (advanced).\n",
+              "variant-names": {
+                internal: "Internal",
+                external: "External",
+              },
+            },
+            default: "internal",
+            variants: {
+              internal: {},
+              external: {
+                address: {
+                  type: "string",
+                  name: "Public Address",
+                  description:
+                    "The public address of your CLNRest server\nNOTE: RTL does not support a .onion URL here\n",
+                  nullable: false,
+                },
+                "rest-port": {
+                  type: "number",
+                  name: "CLNRest Port",
+                  description: "The port that your CLNRest server is bound to",
+                  nullable: false,
+                  range: "[0,65535]",
+                  integral: true,
+                  default: 3010,
+                },
+                macaroon: {
+                  type: "string",
+                  name: "Rune",
+                  description:
+                    "Your CLNRest unrestricted Rune, Base64URL encoded.",
+                  nullable: false,
+                  masked: true,
+                },
+              },
+            },
+          },
+        },
+      },
+    },
+  },
+  password: {
+    type: "string",
+    name: "Password",
+    description: "The password for your Ride the Lightning dashboard",
+    nullable: false,
+    copyable: true,
+    masked: true,
+    default: {
+      charset: "a-z,A-Z,0-9",
+      len: 22,
+    },
+  },
+}

container-runtime/src/Adapters/Systems/SystemForEmbassy/__snapshots__/transformConfigSpec.test.ts.snap

@@ -1,5 +1,30 @@
 // Jest Snapshot v1, https://goo.gl/fbAQLP
 
+exports[`transformConfigSpec transformConfigSpec(RTL) 1`] = `
+{
+  "password": {
+    "default": {
+      "charset": "a-z,A-Z,0-9",
+      "len": 22,
+    },
+    "description": "The password for your Ride the Lightning dashboard",
+    "disabled": false,
+    "generate": null,
+    "immutable": false,
+    "inputmode": "text",
+    "masked": true,
+    "maxLength": null,
+    "minLength": null,
+    "name": "Password",
+    "patterns": [],
+    "placeholder": null,
+    "required": true,
+    "type": "text",
+    "warning": null,
+  },
+}
+`;
+
 exports[`transformConfigSpec transformConfigSpec(bitcoind) 1`] = `
 {
   "advanced": {

container-runtime/src/Adapters/Systems/SystemForEmbassy/index.ts

@@ -1,5 +1,8 @@
 import {
   ExtendedVersion,
+  FileHelper,
+  getDataVersion,
+  overlaps,
   types as T,
   utils,
   VersionRange,
@@ -55,8 +58,21 @@ function todo(): never {
 
 const MANIFEST_LOCATION = "/usr/lib/startos/package/embassyManifest.json"
 export const EMBASSY_JS_LOCATION = "/usr/lib/startos/package/embassy.js"
-const EMBASSY_POINTER_PATH_PREFIX = "/embassyConfig" as utils.StorePath
-const EMBASSY_DEPENDS_ON_PATH_PREFIX = "/embassyDependsOn" as utils.StorePath
+
+const configFile = FileHelper.json(
+  {
+    volumeId: "embassy",
+    subpath: "config.json",
+  },
+  matches.any,
+)
+const dependsOnFile = FileHelper.json(
+  {
+    volumeId: "embassy",
+    subpath: "dependsOn.json",
+  },
+  dictionary([string, array(string)]),
+)
 
 const matchResult = object({
   result: any,
@@ -94,47 +110,48 @@ const fromReturnType = <A>(a: U.ResultType<A>): A => {
   return assertNever(a)
 }
 
-const matchSetResult = object(
-  {
-    "depends-on": dictionary([string, array(string)]),
-    dependsOn: dictionary([string, array(string)]),
-    signal: literals(
-      "SIGTERM",
-      "SIGHUP",
-      "SIGINT",
-      "SIGQUIT",
-      "SIGILL",
-      "SIGTRAP",
-      "SIGABRT",
-      "SIGBUS",
-      "SIGFPE",
-      "SIGKILL",
-      "SIGUSR1",
-      "SIGSEGV",
-      "SIGUSR2",
-      "SIGPIPE",
-      "SIGALRM",
-      "SIGSTKFLT",
-      "SIGCHLD",
-      "SIGCONT",
-      "SIGSTOP",
-      "SIGTSTP",
-      "SIGTTIN",
-      "SIGTTOU",
-      "SIGURG",
-      "SIGXCPU",
-      "SIGXFSZ",
-      "SIGVTALRM",
-      "SIGPROF",
-      "SIGWINCH",
-      "SIGIO",
-      "SIGPWR",
-      "SIGSYS",
-      "SIGINFO",
-    ),
-  },
-  ["depends-on", "dependsOn"],
-)
+const matchSetResult = object({
+  "depends-on": dictionary([string, array(string)])
+    .nullable()
+    .optional(),
+  dependsOn: dictionary([string, array(string)])
+    .nullable()
+    .optional(),
+  signal: literals(
+    "SIGTERM",
+    "SIGHUP",
+    "SIGINT",
+    "SIGQUIT",
+    "SIGILL",
+    "SIGTRAP",
+    "SIGABRT",
+    "SIGBUS",
+    "SIGFPE",
+    "SIGKILL",
+    "SIGUSR1",
+    "SIGSEGV",
+    "SIGUSR2",
+    "SIGPIPE",
+    "SIGALRM",
+    "SIGSTKFLT",
+    "SIGCHLD",
+    "SIGCONT",
+    "SIGSTOP",
+    "SIGTSTP",
+    "SIGTTIN",
+    "SIGTTOU",
+    "SIGURG",
+    "SIGXCPU",
+    "SIGXFSZ",
+    "SIGVTALRM",
+    "SIGPROF",
+    "SIGWINCH",
+    "SIGIO",
+    "SIGPWR",
+    "SIGSYS",
+    "SIGINFO",
+  ),
+})
 
 type OldGetConfigRes = {
   config?: null | Record<string, unknown>
@@ -174,14 +191,14 @@ export type PackagePropertiesV2 = {
 }
 export type PackagePropertyString = {
   type: "string"
-  description?: string
+  description?: string | null
   value: string
   /** Let's the ui make this copyable button */
-  copyable?: boolean
+  copyable?: boolean | null
   /** Let the ui create a qr for this field */
-  qr?: boolean
+  qr?: boolean | null
   /** Hiding the value unless toggled off for field */
-  masked?: boolean
+  masked?: boolean | null
 }
 export type PackagePropertyObject = {
   value: PackagePropertiesV2
@@ -225,17 +242,14 @@ const matchPackagePropertyObject: Parser<unknown, PackagePropertyObject> =
   })
 
 const matchPackagePropertyString: Parser<unknown, PackagePropertyString> =
-  object(
-    {
-      type: literal("string"),
-      description: string,
-      value: string,
-      copyable: boolean,
-      qr: boolean,
-      masked: boolean,
-    },
-    ["copyable", "description", "qr", "masked"],
-  )
+  object({
+    type: literal("string"),
+    description: string.nullable().optional(),
+    value: string,
+    copyable: boolean.nullable().optional(),
+    qr: boolean.nullable().optional(),
+    masked: boolean.nullable().optional(),
+  })
 setMatchPackageProperties(
   dictionary([
     string,
@@ -298,9 +312,15 @@ export class SystemForEmbassy implements System {
     readonly moduleCode: Partial<U.ExpectedExports>,
   ) {}
 
-  async containerInit(effects: Effects): Promise<void> {
+  async init(
+    effects: Effects,
+    kind: "install" | "update" | "restore" | null,
+  ): Promise<void> {
+    if (kind === "restore") {
+      await this.restoreBackup(effects, null)
+    }
     for (let depId in this.manifest.dependencies) {
-      if (this.manifest.dependencies[depId].config) {
+      if (this.manifest.dependencies[depId]?.config) {
         await this.dependenciesAutoconfig(effects, depId, null)
       }
     }
@@ -308,6 +328,9 @@ export class SystemForEmbassy implements System {
     await this.exportActions(effects)
     await this.exportNetwork(effects)
     await this.containerSetDependencies(effects)
+    if (kind === "install" || kind === "update") {
+      await this.packageInit(effects, null)
+    }
   }
   async containerSetDependencies(effects: T.Effects) {
     const oldDeps: Record<string, string[]> = Object.fromEntries(
@@ -347,16 +370,23 @@ export class SystemForEmbassy implements System {
   }
 
   async packageInit(effects: Effects, timeoutMs: number | null): Promise<void> {
-    const previousVersion = await effects.getDataVersion()
+    const previousVersion = await getDataVersion(effects)
     if (previousVersion) {
-      if (
-        (await this.migration(effects, { from: previousVersion }, timeoutMs))
-          .configured
-      ) {
-        await effects.action.clearRequests({ only: ["needs-config"] })
+      const migrationRes = await this.migration(
+        effects,
+        { from: previousVersion },
+        timeoutMs,
+      )
+      if (migrationRes) {
+        if (migrationRes.configured)
+          await effects.action.clearTasks({ only: ["needs-config"] })
+        await configFile.write(
+          effects,
+          await this.getConfig(effects, timeoutMs),
+        )
       }
     } else if (this.manifest.config) {
-      await effects.action.request({
+      await effects.action.createTask({
         packageId: this.manifest.id,
         actionId: "config",
         severity: "critical",
@@ -364,9 +394,29 @@ export class SystemForEmbassy implements System {
         reason: "This service must be configured before it can be run",
       })
     }
+    const version = ExtendedVersion.parseEmver(this.manifest.version)
+    if (
+      this.manifest.id === "bitcoind" &&
+      this.manifest.title.toLowerCase().includes("knots")
+    )
+      version.flavor = "knots"
+
+    if (
+      this.manifest.id === "lnd" ||
+      this.manifest.id === "ride-the-lightning" ||
+      this.manifest.id === "datum"
+    ) {
+      version.upstream.prerelease = ["beta"]
+    } else if (
+      this.manifest.id === "lightning-terminal" ||
+      this.manifest.id === "robosats"
+    ) {
+      version.upstream.prerelease = ["alpha"]
+    }
     await effects.setDataVersion({
-      version: ExtendedVersion.parseEmver(this.manifest.version).toString(),
+      version: version.toString(),
     })
+    // @FullMetal: package hacks go here
   }
   async exportNetwork(effects: Effects) {
     for (const [id, interfaceValue] of Object.entries(
@@ -441,7 +491,7 @@ export class SystemForEmbassy implements System {
               masked: false,
               path: "",
               schemeOverride: null,
-              search: {},
+              query: {},
               username: null,
             }),
           ])
@@ -543,14 +593,17 @@ export class SystemForEmbassy implements System {
     }
     await effects.action.clear({ except: Object.keys(actions) })
   }
-  async packageUninit(
+  async uninit(
     effects: Effects,
-    nextVersion: Optional<string>,
-    timeoutMs: number | null,
+    target: ExtendedVersion | VersionRange | null,
+    timeoutMs?: number | null,
   ): Promise<void> {
     await this.currentRunning?.clean({ timeout: timeoutMs ?? undefined })
-    if (nextVersion) {
-      await this.migration(effects, { to: nextVersion }, timeoutMs)
+    if (
+      target &&
+      !overlaps(target, ExtendedVersion.parseEmver(this.manifest.version))
+    ) {
+      await this.migration(effects, { to: target }, timeoutMs ?? null)
     }
     await effects.setMainStatus({ status: "stopped" })
   }
@@ -577,11 +630,21 @@ export class SystemForEmbassy implements System {
       const moduleCode = await this.moduleCode
       await moduleCode.createBackup?.(polyfillEffects(effects, this.manifest))
     }
+    const dataVersion = await effects.getDataVersion()
+    if (dataVersion)
+      await fs.writeFile("/media/startos/backup/dataVersion.txt", dataVersion, {
+        encoding: "utf-8",
+      })
   }
   async restoreBackup(
     effects: Effects,
     timeoutMs: number | null,
   ): Promise<void> {
+    const store = await fs
+      .readFile("/media/startos/backup/store.json", {
+        encoding: "utf-8",
+      })
+      .catch((_) => null)
     const restoreBackup = this.manifest.backup.restore
     if (restoreBackup.type === "docker") {
       const commands = [restoreBackup.entrypoint, ...restoreBackup.args]
@@ -600,6 +663,13 @@ export class SystemForEmbassy implements System {
       const moduleCode = await this.moduleCode
       await moduleCode.restoreBackup?.(polyfillEffects(effects, this.manifest))
     }
+
+    const dataVersion = await fs
+      .readFile("/media/startos/backup/dataVersion.txt", {
+        encoding: "utf-8",
+      })
+      .catch((_) => null)
+    if (dataVersion) await effects.setDataVersion({ version: dataVersion })
   }
   async getConfig(effects: Effects, timeoutMs: number | null) {
     return this.getConfigUncleaned(effects, timeoutMs).then(convertToNewConfig)
@@ -649,10 +719,7 @@ export class SystemForEmbassy implements System {
       structuredClone(newConfigWithoutPointers as Record<string, unknown>),
     )
     await updateConfig(effects, this.manifest, spec, newConfig)
-    await effects.store.set({
-      path: EMBASSY_POINTER_PATH_PREFIX,
-      value: newConfig,
-    })
+    await configFile.write(effects, newConfig)
     const setConfigValue = this.manifest.config?.set
     if (!setConfigValue) return
     if (setConfigValue.type === "docker") {
@@ -706,15 +773,11 @@ export class SystemForEmbassy implements System {
     rawDepends: { [x: string]: readonly string[] },
     configuring: boolean,
   ) {
-    const storedDependsOn = (await effects.store.get({
-      packageId: this.manifest.id,
-      path: EMBASSY_DEPENDS_ON_PATH_PREFIX,
-    })) as Record<string, readonly string[]>
-
+    const storedDependsOn = await dependsOnFile.read().once()
     const requiredDeps = {
       ...Object.fromEntries(
-        Object.entries(this.manifest.dependencies || {})
-          ?.filter((x) => x[1].requirement.type === "required")
+        Object.entries(this.manifest.dependencies ?? {})
+          .filter(([k, v]) => v?.requirement.type === "required")
           .map((x) => [x[0], []]) || [],
       ),
     }
@@ -728,10 +791,7 @@ export class SystemForEmbassy implements System {
         ? storedDependsOn
         : requiredDeps
 
-    await effects.store.set({
-      path: EMBASSY_DEPENDS_ON_PATH_PREFIX,
-      value: dependsOn,
-    })
+    await dependsOnFile.write(effects, dependsOn)
 
     await effects.setDependencies({
       dependencies: Object.entries(dependsOn).flatMap(
@@ -755,31 +815,31 @@ export class SystemForEmbassy implements System {
 
   async migration(
     effects: Effects,
-    version: { from: string } | { to: string },
+    version:
+      | { from: VersionRange | ExtendedVersion }
+      | { to: VersionRange | ExtendedVersion },
     timeoutMs: number | null,
-  ): Promise<{ configured: boolean }> {
+  ): Promise<{ configured: boolean } | null> {
     let migration
     let args: [string, ...string[]]
     if ("from" in version) {
-      args = [version.from, "from"]
-      const fromExver = ExtendedVersion.parse(version.from)
+      args = [version.from.toString(), "from"]
       if (!this.manifest.migrations) return { configured: true }
       migration = Object.entries(this.manifest.migrations.from)
         .map(
           ([version, procedure]) =>
             [VersionRange.parseEmver(version), procedure] as const,
         )
-        .find(([versionEmver, _]) => versionEmver.satisfiedBy(fromExver))
+        .find(([versionEmver, _]) => overlaps(versionEmver, version.from))
     } else {
-      args = [version.to, "to"]
-      const toExver = ExtendedVersion.parse(version.to)
+      args = [version.to.toString(), "to"]
       if (!this.manifest.migrations) return { configured: true }
       migration = Object.entries(this.manifest.migrations.to)
         .map(
           ([version, procedure]) =>
             [VersionRange.parseEmver(version), procedure] as const,
         )
-        .find(([versionEmver, _]) => versionEmver.satisfiedBy(toExver))
+        .find(([versionEmver, _]) => overlaps(versionEmver, version.to))
     }
 
     if (migration) {
@@ -815,7 +875,7 @@ export class SystemForEmbassy implements System {
         })) as any
       }
     }
-    return { configured: true }
+    return null
   }
   async properties(
     effects: Effects,
@@ -969,43 +1029,51 @@ export class SystemForEmbassy implements System {
     timeoutMs: number | null,
   ): Promise<void> {
     // TODO: docker
-    const oldConfig = (await effects.store.get({
-      packageId: id,
-      path: EMBASSY_POINTER_PATH_PREFIX,
-      callback: () => {
-        this.dependenciesAutoconfig(effects, id, timeoutMs)
-      },
-    })) as U.Config
-    if (!oldConfig) return
-    const moduleCode = await this.moduleCode
-    const method = moduleCode?.dependencies?.[id]?.autoConfigure
-    if (!method) return
-    const newConfig = (await method(
-      polyfillEffects(effects, this.manifest),
-      JSON.parse(JSON.stringify(oldConfig)),
-    ).then((x) => {
-      if ("result" in x) return x.result
-      if ("error" in x) throw new Error("Error getting config: " + x.error)
-      throw new Error("Error getting config: " + x["error-code"][1])
-    })) as any
-    const diff = partialDiff(oldConfig, newConfig)
-    if (diff) {
-      await effects.action.request({
-        actionId: "config",
+    await effects.mount({
+      location: `/media/embassy/${id}`,
+      target: {
         packageId: id,
-        replayId: `${id}/config`,
-        severity: "important",
-        reason: `Configure this dependency for the needs of ${this.manifest.title}`,
-        input: {
-          kind: "partial",
-          value: diff.diff,
-        },
-        when: {
-          condition: "input-not-matches",
-          once: false,
-        },
+        volumeId: "embassy",
+        subpath: null,
+        readonly: true,
+        filetype: "directory",
+      },
+    })
+    configFile
+      .withPath(`/media/embassy/${id}/config.json`)
+      .read()
+      .onChange(effects, async (oldConfig: U.Config) => {
+        if (!oldConfig) return
+        const moduleCode = await this.moduleCode
+        const method = moduleCode?.dependencies?.[id]?.autoConfigure
+        if (!method) return
+        const newConfig = (await method(
+          polyfillEffects(effects, this.manifest),
+          JSON.parse(JSON.stringify(oldConfig)),
+        ).then((x) => {
+          if ("result" in x) return x.result
+          if ("error" in x) throw new Error("Error getting config: " + x.error)
+          throw new Error("Error getting config: " + x["error-code"][1])
+        })) as any
+        const diff = partialDiff(oldConfig, newConfig)
+        if (diff) {
+          await effects.action.createTask({
+            actionId: "config",
+            packageId: id,
+            replayId: `${id}/config`,
+            severity: "important",
+            reason: `Configure this dependency for the needs of ${this.manifest.title}`,
+            input: {
+              kind: "partial",
+              value: diff.diff,
+            },
+            when: {
+              condition: "input-not-matches",
+              once: false,
+            },
+          })
+        }
       })
-    }
   }
 }
 
@@ -1107,11 +1175,21 @@ async function updateConfig(
     ) {
       if (specValue.target === "config") {
         const jp = require("jsonpath")
-        const remoteConfig = await effects.store.get({
-          packageId: specValue["package-id"],
-          callback: () => effects.restart(),
-          path: EMBASSY_POINTER_PATH_PREFIX,
+        const depId = specValue["package-id"]
+        await effects.mount({
+          location: `/media/embassy/${depId}`,
+          target: {
+            packageId: depId,
+            volumeId: "embassy",
+            subpath: null,
+            readonly: true,
+            filetype: "directory",
+          },
         })
+        const remoteConfig = configFile
+          .withPath(`/media/embassy/${depId}/config.json`)
+          .read()
+          .once()
         console.debug(remoteConfig)
         const configValue = specValue.multi
           ? jp.query(remoteConfig, specValue.selector)

container-runtime/src/Adapters/Systems/SystemForEmbassy/matchManifest.ts

@@ -14,123 +14,113 @@ import {
 import { matchVolume } from "./matchVolume"
 import { matchDockerProcedure } from "../../../Models/DockerProcedure"
 
-const matchJsProcedure = object(
-  {
-    type: literal("script"),
-    args: array(unknown),
-  },
-  ["args"],
-  {
-    args: [],
-  },
-)
+const matchJsProcedure = object({
+  type: literal("script"),
+  args: array(unknown).nullable().optional().defaultTo([]),
+})
 
 const matchProcedure = some(matchDockerProcedure, matchJsProcedure)
 export type Procedure = typeof matchProcedure._TYPE
 
-const matchAction = object(
-  {
-    name: string,
-    description: string,
-    warning: string,
-    implementation: matchProcedure,
-    "allowed-statuses": array(literals("running", "stopped")),
-    "input-spec": unknown,
-  },
-  ["warning", "input-spec", "input-spec"],
-)
-export const matchManifest = object(
-  {
-    id: string,
-    title: string,
-    version: string,
-    main: matchDockerProcedure,
-    assets: object(
-      {
-        assets: string,
-        scripts: string,
-      },
-      ["assets", "scripts"],
+const matchAction = object({
+  name: string,
+  description: string,
+  warning: string.nullable().optional(),
+  implementation: matchProcedure,
+  "allowed-statuses": array(literals("running", "stopped")),
+  "input-spec": unknown.nullable().optional(),
+})
+export const matchManifest = object({
+  id: string,
+  title: string,
+  version: string,
+  main: matchDockerProcedure,
+  assets: object({
+    assets: string.nullable().optional(),
+    scripts: string.nullable().optional(),
+  })
+    .nullable()
+    .optional(),
+  "health-checks": dictionary([
+    string,
+    every(
+      matchProcedure,
+      object({
+        name: string,
+        ["success-message"]: string.nullable().optional(),
+      }),
     ),
-    "health-checks": dictionary([
-      string,
-      every(
-        matchProcedure,
-        object(
-          {
-            name: string,
-            ["success-message"]: string,
-          },
-          ["success-message"],
-        ),
-      ),
-    ]),
-    config: object({
-      get: matchProcedure,
-      set: matchProcedure,
+  ]),
+  config: object({
+    get: matchProcedure,
+    set: matchProcedure,
+  })
+    .nullable()
+    .optional(),
+  properties: matchProcedure.nullable().optional(),
+  volumes: dictionary([string, matchVolume]),
+  interfaces: dictionary([
+    string,
+    object({
+      name: string,
+      description: string,
+      "tor-config": object({
+        "port-mapping": dictionary([string, string]),
+      })
+        .nullable()
+        .optional(),
+      "lan-config": dictionary([
+        string,
+        object({
+          ssl: boolean,
+          internal: number,
+        }),
+      ])
+        .nullable()
+        .optional(),
+      ui: boolean,
+      protocols: array(string),
     }),
-    properties: matchProcedure,
-    volumes: dictionary([string, matchVolume]),
-    interfaces: dictionary([
-      string,
-      object(
-        {
-          name: string,
-          description: string,
-          "tor-config": object({
-            "port-mapping": dictionary([string, string]),
-          }),
-          "lan-config": dictionary([
-            string,
-            object({
-              ssl: boolean,
-              internal: number,
-            }),
-          ]),
-          ui: boolean,
-          protocols: array(string),
-        },
-        ["lan-config", "tor-config"],
+  ]),
+  backup: object({
+    create: matchProcedure,
+    restore: matchProcedure,
+  }),
+  migrations: object({
+    to: dictionary([string, matchProcedure]),
+    from: dictionary([string, matchProcedure]),
+  })
+    .nullable()
+    .optional(),
+  dependencies: dictionary([
+    string,
+    object({
+      version: string,
+      requirement: some(
+        object({
+          type: literal("opt-in"),
+          how: string,
+        }),
+        object({
+          type: literal("opt-out"),
+          how: string,
+        }),
+        object({
+          type: literal("required"),
+        }),
       ),
-    ]),
-    backup: object({
-      create: matchProcedure,
-      restore: matchProcedure,
-    }),
-    migrations: object({
-      to: dictionary([string, matchProcedure]),
-      from: dictionary([string, matchProcedure]),
-    }),
-    dependencies: dictionary([
-      string,
-      object(
-        {
-          version: string,
-          requirement: some(
-            object({
-              type: literal("opt-in"),
-              how: string,
-            }),
-            object({
-              type: literal("opt-out"),
-              how: string,
-            }),
-            object({
-              type: literal("required"),
-            }),
-          ),
-          description: string,
-          config: object({
-            check: matchProcedure,
-            "auto-configure": matchProcedure,
-          }),
-        },
-        ["description", "config"],
-      ),
-    ]),
+      description: string.nullable().optional(),
+      config: object({
+        check: matchProcedure,
+        "auto-configure": matchProcedure,
+      })
+        .nullable()
+        .optional(),
+    })
+      .nullable()
+      .optional(),
+  ]),
 
-    actions: dictionary([string, matchAction]),
-  },
-  ["config", "actions", "properties", "migrations", "dependencies"],
-)
+  actions: dictionary([string, matchAction]),
+})
 export type Manifest = typeof matchManifest._TYPE

container-runtime/src/Adapters/Systems/SystemForEmbassy/matchVolume.ts

@@ -1,12 +1,9 @@
 import { object, literal, string, boolean, some } from "ts-matches"
 
-const matchDataVolume = object(
-  {
-    type: literal("data"),
-    readonly: boolean,
-  },
-  ["readonly"],
-)
+const matchDataVolume = object({
+  type: literal("data"),
+  readonly: boolean.optional(),
+})
 const matchAssetVolume = object({
   type: literal("assets"),
 })

container-runtime/src/Adapters/Systems/SystemForEmbassy/polyfillEffects.ts

@@ -135,12 +135,9 @@ export const polyfillEffects = (
         [input.command, ...(input.args || [])].join(" "),
       )
       const daemon = promiseSubcontainer.then((subcontainer) =>
-        daemons.runCommand()(
-          effects,
-          subcontainer,
-          [input.command, ...(input.args || [])],
-          {},
-        ),
+        daemons.runCommand()(effects, subcontainer, {
+          command: [input.command, ...(input.args || [])],
+        }),
       )
       return {
         wait: () =>
@@ -169,12 +166,12 @@ export const polyfillEffects = (
           { imageId: manifest.main.image },
           commands,
           {
-            mounts: Mounts.of().addVolume(
-              input.volumeId,
-              null,
-              "/drive",
-              false,
-            ),
+            mounts: Mounts.of().mountVolume({
+              volumeId: input.volumeId,
+              subpath: null,
+              mountpoint: "/drive",
+              readonly: false,
+            }),
           },
           commands.join(" "),
         )
@@ -206,12 +203,12 @@ export const polyfillEffects = (
           { imageId: manifest.main.image },
           commands,
           {
-            mounts: Mounts.of().addVolume(
-              input.volumeId,
-              null,
-              "/drive",
-              false,
-            ),
+            mounts: Mounts.of().mountVolume({
+              volumeId: input.volumeId,
+              subpath: null,
+              mountpoint: "/drive",
+              readonly: false,
+            }),
           },
           commands.join(" "),
         )

container-runtime/src/Adapters/Systems/SystemForEmbassy/transformConfigSpec.test.ts

@@ -1,5 +1,10 @@
-import { matchOldConfigSpec, transformConfigSpec } from "./transformConfigSpec"
-import fixtureEmbasyPagesConfig from "./__fixtures__/embasyPagesConfig"
+import {
+  matchOldConfigSpec,
+  matchOldValueSpecList,
+  transformConfigSpec,
+} from "./transformConfigSpec"
+import fixtureEmbassyPagesConfig from "./__fixtures__/embassyPagesConfig"
+import fixtureRTLConfig from "./__fixtures__/rtlConfig"
 import searNXG from "./__fixtures__/searNXG"
 import bitcoind from "./__fixtures__/bitcoind"
 import nostr from "./__fixtures__/nostr"
@@ -8,14 +13,25 @@ import nostrConfig2 from "./__fixtures__/nostrConfig2"
 describe("transformConfigSpec", () => {
   test("matchOldConfigSpec(embassyPages.homepage.variants[web-page])", () => {
     matchOldConfigSpec.unsafeCast(
-      fixtureEmbasyPagesConfig.homepage.variants["web-page"],
+      fixtureEmbassyPagesConfig.homepage.variants["web-page"],
     )
   })
   test("matchOldConfigSpec(embassyPages)", () => {
-    matchOldConfigSpec.unsafeCast(fixtureEmbasyPagesConfig)
+    matchOldConfigSpec.unsafeCast(fixtureEmbassyPagesConfig)
   })
   test("transformConfigSpec(embassyPages)", () => {
-    const spec = matchOldConfigSpec.unsafeCast(fixtureEmbasyPagesConfig)
+    const spec = matchOldConfigSpec.unsafeCast(fixtureEmbassyPagesConfig)
+    expect(transformConfigSpec(spec)).toMatchSnapshot()
+  })
+
+  test("matchOldConfigSpec(RTL.nodes)", () => {
+    matchOldValueSpecList.unsafeCast(fixtureRTLConfig.nodes)
+  })
+  test("matchOldConfigSpec(RTL)", () => {
+    matchOldConfigSpec.unsafeCast(fixtureRTLConfig)
+  })
+  test("transformConfigSpec(RTL)", () => {
+    const spec = matchOldConfigSpec.unsafeCast(fixtureRTLConfig)
     expect(transformConfigSpec(spec)).toMatchSnapshot()
   })
 

container-runtime/src/Adapters/Systems/SystemForEmbassy/transformConfigSpec.ts

@@ -47,6 +47,7 @@ export function transformConfigSpec(oldSpec: OldConfigSpec): IST.InputSpec {
         immutable: false,
       }
     } else if (oldVal.type === "list") {
+      if (isUnionList(oldVal)) return inputSpec
       newVal = getListSpec(oldVal)
     } else if (oldVal.type === "number") {
       const range = Range.from(oldVal.range)
@@ -177,15 +178,17 @@ export function transformOldConfigToNew(
       }
     }
 
-    if (isList(val) && isObjectList(val)) {
+    if (isList(val)) {
       if (!config[key]) return obj
 
-      newVal = (config[key] as object[]).map((obj) =>
-        transformOldConfigToNew(
-          matchOldConfigSpec.unsafeCast(val.spec.spec),
-          obj,
-        ),
-      )
+      if (isObjectList(val)) {
+        newVal = (config[key] as object[]).map((obj) =>
+          transformOldConfigToNew(
+            matchOldConfigSpec.unsafeCast(val.spec.spec),
+            obj,
+          ),
+        )
+      } else if (isUnionList(val)) return obj
     }
 
     if (isPointer(val)) {
@@ -203,6 +206,7 @@ export function transformNewConfigToOld(
   spec: OldConfigSpec,
   config: Record<string, any>,
 ): Record<string, any> {
+  if (!config) return config
   return Object.entries(spec).reduce((obj, [key, val]) => {
     let newVal = config[key]
 
@@ -223,13 +227,15 @@ export function transformNewConfigToOld(
       }
     }
 
-    if (isList(val) && isObjectList(val)) {
-      newVal = (config[key] as object[]).map((obj) =>
-        transformNewConfigToOld(
-          matchOldConfigSpec.unsafeCast(val.spec.spec),
-          obj,
-        ),
-      )
+    if (isList(val)) {
+      if (isObjectList(val)) {
+        newVal = (config[key] as object[]).map((obj) =>
+          transformNewConfigToOld(
+            matchOldConfigSpec.unsafeCast(val.spec.spec),
+            obj,
+          ),
+        )
+      } else if (isUnionList(val)) return obj
     }
 
     return {
@@ -375,15 +381,17 @@ function isNumberList(
 ): val is OldValueSpecList & { subtype: "number" } {
   return val.subtype === "number"
 }
-
 function isObjectList(
   val: OldValueSpecList,
 ): val is OldValueSpecList & { subtype: "object" } {
-  if (["union"].includes(val.subtype)) {
-    throw new Error("Invalid list subtype. enum, string, and object permitted.")
-  }
   return val.subtype === "object"
 }
+function isUnionList(
+  val: OldValueSpecList,
+): val is OldValueSpecList & { subtype: "union" } {
+  return val.subtype === "union"
+}
+
 export type OldConfigSpec = Record<string, OldValueSpec>
 const [_matchOldConfigSpec, setMatchOldConfigSpec] = deferred<unknown>()
 export const matchOldConfigSpec = _matchOldConfigSpec as Parser<
@@ -396,100 +404,71 @@ export const matchOldDefaultString = anyOf(
 )
 type OldDefaultString = typeof matchOldDefaultString._TYPE
 
-export const matchOldValueSpecString = object(
-  {
-    type: literals("string"),
-    name: string,
-    masked: boolean,
-    copyable: boolean,
-    nullable: boolean,
-    placeholder: string,
-    pattern: string,
-    "pattern-description": string,
-    default: matchOldDefaultString,
-    textarea: boolean,
-    description: string,
-    warning: string,
-  },
-  [
-    "masked",
-    "copyable",
-    "nullable",
-    "placeholder",
-    "pattern",
-    "pattern-description",
-    "default",
-    "textarea",
-    "description",
-    "warning",
-  ],
-)
+export const matchOldValueSpecString = object({
+  type: literals("string"),
+  name: string,
+  masked: boolean.nullable().optional(),
+  copyable: boolean.nullable().optional(),
+  nullable: boolean.nullable().optional(),
+  placeholder: string.nullable().optional(),
+  pattern: string.nullable().optional(),
+  "pattern-description": string.nullable().optional(),
+  default: matchOldDefaultString.nullable().optional(),
+  textarea: boolean.nullable().optional(),
+  description: string.nullable().optional(),
+  warning: string.nullable().optional(),
+})
 
-export const matchOldValueSpecNumber = object(
-  {
-    type: literals("number"),
-    nullable: boolean,
-    name: string,
-    range: string,
-    integral: boolean,
-    default: number,
-    description: string,
-    warning: string,
-    units: string,
-    placeholder: anyOf(number, string),
-  },
-  ["default", "description", "warning", "units", "placeholder"],
-)
+export const matchOldValueSpecNumber = object({
+  type: literals("number"),
+  nullable: boolean,
+  name: string,
+  range: string,
+  integral: boolean,
+  default: number.nullable().optional(),
+  description: string.nullable().optional(),
+  warning: string.nullable().optional(),
+  units: string.nullable().optional(),
+  placeholder: anyOf(number, string).nullable().optional(),
+})
 type OldValueSpecNumber = typeof matchOldValueSpecNumber._TYPE
 
-export const matchOldValueSpecBoolean = object(
-  {
-    type: literals("boolean"),
-    default: boolean,
-    name: string,
-    description: string,
-    warning: string,
-  },
-  ["description", "warning"],
-)
+export const matchOldValueSpecBoolean = object({
+  type: literals("boolean"),
+  default: boolean,
+  name: string,
+  description: string.nullable().optional(),
+  warning: string.nullable().optional(),
+})
 type OldValueSpecBoolean = typeof matchOldValueSpecBoolean._TYPE
 
-const matchOldValueSpecObject = object(
-  {
-    type: literals("object"),
-    spec: _matchOldConfigSpec,
-    name: string,
-    description: string,
-    warning: string,
-  },
-  ["description", "warning"],
-)
+const matchOldValueSpecObject = object({
+  type: literals("object"),
+  spec: _matchOldConfigSpec,
+  name: string,
+  description: string.nullable().optional(),
+  warning: string.nullable().optional(),
+})
 type OldValueSpecObject = typeof matchOldValueSpecObject._TYPE
 
-const matchOldValueSpecEnum = object(
-  {
-    values: array(string),
-    "value-names": dictionary([string, string]),
-    type: literals("enum"),
-    default: string,
-    name: string,
-    description: string,
-    warning: string,
-  },
-  ["description", "warning"],
-)
+const matchOldValueSpecEnum = object({
+  values: array(string),
+  "value-names": dictionary([string, string]),
+  type: literals("enum"),
+  default: string,
+  name: string,
+  description: string.nullable().optional(),
+  warning: string.nullable().optional(),
+})
 type OldValueSpecEnum = typeof matchOldValueSpecEnum._TYPE
 
-const matchOldUnionTagSpec = object(
-  {
-    id: string, // The name of the field containing one of the union variants
-    "variant-names": dictionary([string, string]), // The name of each variant
-    name: string,
-    description: string,
-    warning: string,
-  },
-  ["description", "warning"],
-)
+const matchOldUnionTagSpec = object({
+  id: string, // The name of the field containing one of the union variants
+  "variant-names": dictionary([string, string]), // The name of each variant
+  name: string,
+  description: string.nullable().optional(),
+  warning: string.nullable().optional(),
+})
 const matchOldValueSpecUnion = object({
   type: literals("union"),
   tag: matchOldUnionTagSpec,
@@ -514,57 +493,51 @@ setOldUniqueBy(
   ),
 )
 
-const matchOldListValueSpecObject = object(
-  {
-    spec: _matchOldConfigSpec, // this is a mapped type of the config object at this level, replacing the object's values with specs on those values
-    "unique-by": matchOldUniqueBy, // indicates whether duplicates can be permitted in the list
-    "display-as": string, // this should be a handlebars template which can make use of the entire config which corresponds to 'spec'
-  },
-  ["display-as", "unique-by"],
-)
-const matchOldListValueSpecString = object(
-  {
-    masked: boolean,
-    copyable: boolean,
-    pattern: string,
-    "pattern-description": string,
-    placeholder: string,
-  },
-  ["pattern", "pattern-description", "placeholder", "copyable", "masked"],
-)
+const matchOldListValueSpecObject = object({
+  spec: _matchOldConfigSpec, // this is a mapped type of the config object at this level, replacing the object's values with specs on those values
+  "unique-by": matchOldUniqueBy.nullable().optional(), // indicates whether duplicates can be permitted in the list
+  "display-as": string.nullable().optional(), // this should be a handlebars template which can make use of the entire config which corresponds to 'spec'
+})
+const matchOldListValueSpecUnion = object({
+  "unique-by": matchOldUniqueBy.nullable().optional(),
+  "display-as": string.nullable().optional(),
+  tag: matchOldUnionTagSpec,
+  variants: dictionary([string, _matchOldConfigSpec]),
+})
+const matchOldListValueSpecString = object({
+  masked: boolean.nullable().optional(),
+  copyable: boolean.nullable().optional(),
+  pattern: string.nullable().optional(),
+  "pattern-description": string.nullable().optional(),
+  placeholder: string.nullable().optional(),
+})
 
 const matchOldListValueSpecEnum = object({
   values: array(string),
   "value-names": dictionary([string, string]),
 })
-const matchOldListValueSpecNumber = object(
-  {
-    range: string,
-    integral: boolean,
-    units: string,
-    placeholder: anyOf(number, string),
-  },
-  ["units", "placeholder"],
-)
+const matchOldListValueSpecNumber = object({
+  range: string,
+  integral: boolean,
+  units: string.nullable().optional(),
+  placeholder: anyOf(number, string).nullable().optional(),
+})
 
 // represents a spec for a list
-const matchOldValueSpecList = every(
-  object(
-    {
-      type: literals("list"),
-      range: string, // '[0,1]' (inclusive) OR '[0,*)' (right unbounded), normal math rules
-      default: anyOf(
-        array(string),
-        array(number),
-        array(matchOldDefaultString),
-        array(object),
-      ),
-      name: string,
-      description: string,
-      warning: string,
-    },
-    ["description", "warning"],
-  ),
+export const matchOldValueSpecList = every(
+  object({
+    type: literals("list"),
+    range: string, // '[0,1]' (inclusive) OR '[0,*)' (right unbounded), normal math rules
+    default: anyOf(
+      array(string),
+      array(number),
+      array(matchOldDefaultString),
+      array(object),
+    ),
+    name: string,
+    description: string.nullable().optional(),
+    warning: string.nullable().optional(),
+  }),
   anyOf(
     object({
       subtype: literals("string"),
@@ -582,6 +555,10 @@ const matchOldValueSpecList = every(
       subtype: literals("number"),
       spec: matchOldListValueSpecNumber,
     }),
+    object({
+      subtype: literals("union"),
+      spec: matchOldListValueSpecUnion,
+    }),
   ),
 )
 type OldValueSpecList = typeof matchOldValueSpecList._TYPE

container-runtime/src/Adapters/Systems/SystemForStartOs.ts

@@ -1,7 +1,6 @@
 import { System } from "../../Interfaces/System"
 import { Effects } from "../../Models/Effects"
-import { T, utils } from "@start9labs/start-sdk"
-import { Optional } from "ts-matches/lib/parsers/interfaces"
+import { ExtendedVersion, T, utils, VersionRange } from "@start9labs/start-sdk"
 
 export const STARTOS_JS_LOCATION = "/usr/lib/startos/package/index.js"
 
@@ -11,6 +10,7 @@ type RunningMain = {
 
 export class SystemForStartOs implements System {
   private runningMain: RunningMain | undefined
+  private starting: boolean = false
 
   static of() {
     return new SystemForStartOs(require(STARTOS_JS_LOCATION))
@@ -19,22 +19,23 @@ export class SystemForStartOs implements System {
   constructor(readonly abi: T.ABI) {
     this
   }
-  async containerInit(effects: Effects): Promise<void> {
-    return void (await this.abi.containerInit({ effects }))
-  }
-  async packageInit(
+
+  async init(
     effects: Effects,
+    kind: "install" | "update" | "restore" | null,
+  ): Promise<void> {
+    return void (await this.abi.init({ effects, kind }))
+  }
+
+  async exit(
+    effects: Effects,
+    target: ExtendedVersion | VersionRange | null,
     timeoutMs: number | null = null,
   ): Promise<void> {
-    return void (await this.abi.packageInit({ effects }))
-  }
-  async packageUninit(
-    effects: Effects,
-    nextVersion: Optional<string> = null,
-    timeoutMs: number | null = null,
-  ): Promise<void> {
-    return void (await this.abi.packageUninit({ effects, nextVersion }))
+    // TODO: stop?
+    return void (await this.abi.uninit({ effects, target }))
   }
+
   async createBackup(
     effects: T.Effects,
     timeoutMs: number | null,
@@ -43,14 +44,6 @@ export class SystemForStartOs implements System {
       effects,
     }))
   }
-  async restoreBackup(
-    effects: T.Effects,
-    timeoutMs: number | null,
-  ): Promise<void> {
-    return void (await this.abi.restoreBackup({
-      effects,
-    }))
-  }
   getActionInput(
     effects: Effects,
     id: string,
@@ -71,28 +64,31 @@ export class SystemForStartOs implements System {
     return action.run({ effects, input })
   }
 
-  async exit(): Promise<void> {}
-
   async start(effects: Effects): Promise<void> {
-    if (this.runningMain) return
-    effects.constRetry = utils.once(() => effects.restart())
-    let mainOnTerm: () => Promise<void> | undefined
-    const started = async (onTerm: () => Promise<void>) => {
-      await effects.setMainStatus({ status: "running" })
-      mainOnTerm = onTerm
-      return null
-    }
-    const daemons = await (
-      await this.abi.main({
-        effects,
-        started,
-      })
-    ).build()
-    this.runningMain = {
-      stop: async () => {
-        if (mainOnTerm) await mainOnTerm()
-        await daemons.term()
-      },
+    try {
+      if (this.runningMain || this.starting) return
+      this.starting = true
+      effects.constRetry = utils.once(() => effects.restart())
+      let mainOnTerm: () => Promise<void> | undefined
+      const started = async (onTerm: () => Promise<void>) => {
+        await effects.setMainStatus({ status: "running" })
+        mainOnTerm = onTerm
+        return null
+      }
+      const daemons = await (
+        await this.abi.main({
+          effects,
+          started,
+        })
+      ).build()
+      this.runningMain = {
+        stop: async () => {
+          if (mainOnTerm) await mainOnTerm()
+          await daemons.term()
+        },
+      }
+    } finally {
+      this.starting = false
     }
   }
 

container-runtime/src/Interfaces/System.ts

@@ -1,13 +1,13 @@
-import { types as T } from "@start9labs/start-sdk"
+import {
+  ExtendedVersion,
+  types as T,
+  VersionRange,
+} from "@start9labs/start-sdk"
 import { Effects } from "../Models/Effects"
 import { CallbackHolder } from "../Models/CallbackHolder"
-import { Optional } from "ts-matches/lib/parsers/interfaces"
 
 export type Procedure =
-  | "/packageInit"
-  | "/packageUninit"
   | "/backup/create"
-  | "/backup/restore"
   | `/actions/${string}/getInput`
   | `/actions/${string}/run`
 
@@ -15,20 +15,15 @@ export type ExecuteResult =
   | { ok: unknown }
   | { err: { code: number; message: string } }
 export type System = {
-  containerInit(effects: T.Effects): Promise<void>
+  init(
+    effects: T.Effects,
+    kind: "install" | "update" | "restore" | null,
+  ): Promise<void>
 
   start(effects: T.Effects): Promise<void>
   stop(): Promise<void>
 
-  packageInit(effects: Effects, timeoutMs: number | null): Promise<void>
-  packageUninit(
-    effects: Effects,
-    nextVersion: Optional<string>,
-    timeoutMs: number | null,
-  ): Promise<void>
-
   createBackup(effects: T.Effects, timeoutMs: number | null): Promise<void>
-  restoreBackup(effects: T.Effects, timeoutMs: number | null): Promise<void>
   runAction(
     effects: Effects,
     actionId: string,
@@ -41,7 +36,10 @@ export type System = {
     timeoutMs: number | null,
   ): Promise<T.ActionInput | null>
 
-  exit(): Promise<void>
+  exit(
+    effects: Effects,
+    target: ExtendedVersion | VersionRange | null,
+  ): Promise<void>
 }
 
 export type RunningMain = {

container-runtime/src/Models/CallbackHolder.ts

@@ -14,7 +14,8 @@ export class CallbackHolder {
   constructor(private effects?: T.Effects) {}
 
   private callbacks = new Map<number, Function>()
-  private children: WeakRef<CallbackHolder>[] = []
+  private onLeaveContextCallbacks: Function[] = []
+  private children: Map<string, CallbackHolder> = new Map()
   private newId() {
     return CallbackIdCell.inc++
   }
@@ -32,23 +33,30 @@ export class CallbackHolder {
       })
     return id
   }
-  child(): CallbackHolder {
-    const child = new CallbackHolder()
-    this.children.push(new WeakRef(child))
+  child(name: string): CallbackHolder {
+    this.removeChild(name)
+    const child = new CallbackHolder(this.effects)
+    this.children.set(name, child)
     return child
   }
-  removeChild(child: CallbackHolder) {
-    this.children = this.children.filter((c) => {
-      const ref = c.deref()
-      return ref && ref !== child
-    })
+
+  getChild(name: string): CallbackHolder | null {
+    return this.children.get(name) || null
+  }
+
+  removeChild(name: string) {
+    const child = this.children.get(name)
+    if (child) {
+      child.leaveContext()
+      this.children.delete(name)
+    }
   }
   private getCallback(index: number): Function | undefined {
     let callback = this.callbacks.get(index)
     if (callback) this.callbacks.delete(index)
     else {
-      for (let i = 0; i < this.children.length; i++) {
-        callback = this.children[i].deref()?.getCallback(index)
+      for (let [_, child] of this.children) {
+        callback = child.getCallback(index)
         if (callback) return callback
       }
     }
@@ -57,6 +65,25 @@ export class CallbackHolder {
   callCallback(index: number, args: any[]): Promise<unknown> {
     const callback = this.getCallback(index)
     if (!callback) return Promise.resolve()
-    return Promise.resolve().then(() => callback(...args))
+    return Promise.resolve()
+      .then(() => callback(...args))
+      .catch((e) => console.error("callback failed", e))
+  }
+  onLeaveContext(fn: Function) {
+    this.onLeaveContextCallbacks.push(fn)
+  }
+  leaveContext() {
+    for (let [_, child] of this.children) {
+      child.leaveContext()
+    }
+    this.children = new Map()
+    for (let fn of this.onLeaveContextCallbacks) {
+      try {
+        fn()
+      } catch (e) {
+        console.warn(e)
+      }
+    }
+    this.onLeaveContextCallbacks = []
   }
 }

container-runtime/src/Models/DockerProcedure.ts

@@ -17,31 +17,25 @@ const Path = string
 
 export type VolumeId = string
 export type Path = string
-export const matchDockerProcedure = object(
-  {
-    type: literal("docker"),
-    image: string,
-    system: boolean,
-    entrypoint: string,
-    args: array(string),
-    mounts: dictionary([VolumeId, Path]),
-    "io-format": literals(
-      "json",
-      "json-pretty",
-      "yaml",
-      "cbor",
-      "toml",
-      "toml-pretty",
-    ),
-    "sigterm-timeout": some(number, matchDuration),
-    inject: boolean,
-  },
-  ["io-format", "sigterm-timeout", "system", "args", "inject", "mounts"],
-  {
-    "sigterm-timeout": 30,
-    inject: false,
-    args: [],
-  },
-)
+export const matchDockerProcedure = object({
+  type: literal("docker"),
+  image: string,
+  system: boolean.optional(),
+  entrypoint: string,
+  args: array(string).defaultTo([]),
+  mounts: dictionary([VolumeId, Path]).optional(),
+  "io-format": literals(
+    "json",
+    "json-pretty",
+    "yaml",
+    "cbor",
+    "toml",
+    "toml-pretty",
+  )
+    .nullable()
+    .optional(),
+  "sigterm-timeout": some(number, matchDuration).onMismatch(30),
+  inject: boolean.defaultTo(false),
+})
 
 export type DockerProcedure = typeof matchDockerProcedure._TYPE

container-runtime/update-image.sh

@@ -4,8 +4,8 @@ cd "$(dirname "${BASH_SOURCE[0]}")"
 
 set -e
 
-if mountpoint tmp/combined; then sudo umount -R tmp/combined; fi
-if mountpoint tmp/lower; then sudo umount tmp/lower; fi
+if mountpoint -q tmp/combined; then sudo umount -R tmp/combined; fi
+if mountpoint -q tmp/lower; then sudo umount tmp/lower; fi
 sudo rm -rf tmp
 mkdir -p tmp/lower tmp/upper tmp/work tmp/combined
 if which squashfuse > /dev/null; then
@@ -13,7 +13,11 @@ if which squashfuse > /dev/null; then
 else
     sudo mount debian.${ARCH}.squashfs tmp/lower
 fi
-sudo mount -t overlay -olowerdir=tmp/lower,upperdir=tmp/upper,workdir=tmp/work overlay tmp/combined
+if which fuse-overlayfs > /dev/null; then
+    sudo fuse-overlayfs -olowerdir=tmp/lower,upperdir=tmp/upper,workdir=tmp/work overlay tmp/combined
+else
+    sudo mount -t overlay -olowerdir=tmp/lower,upperdir=tmp/upper,workdir=tmp/work overlay tmp/combined
+fi
 
 QEMU=
 if [ "$ARCH" != "$(uname -m)" ]; then
@@ -33,6 +37,8 @@ sudo rsync -a --copy-unsafe-links dist/ tmp/combined/usr/lib/startos/init/
 sudo chown -R 0:0 tmp/combined/usr/lib/startos/
 sudo cp container-runtime.service tmp/combined/lib/systemd/system/container-runtime.service
 sudo chown 0:0 tmp/combined/lib/systemd/system/container-runtime.service
+sudo cp container-runtime-failure.service tmp/combined/lib/systemd/system/container-runtime-failure.service
+sudo chown 0:0 tmp/combined/lib/systemd/system/container-runtime-failure.service
 sudo cp ../core/target/$ARCH-unknown-linux-musl/release/containerbox tmp/combined/usr/bin/start-cli
 sudo chown 0:0 tmp/combined/usr/bin/start-cli
 echo container-runtime | sha256sum | head -c 32 | cat - <(echo) | sudo tee tmp/combined/etc/machine-id

core/README.md

@@ -14,7 +14,7 @@
 ## Artifacts
 
 The StartOS backend is packed into a single binary `startbox` that is symlinked under
-several different names for different behaviour:
+several different names for different behavior:
 
 - `startd`: This is the main daemon of StartOS
 - `start-cli`: This is a CLI tool that will allow you to issue commands to

core/helpers/src/os_api.rs

@@ -0,0 +1,82 @@
+use std::sync::Arc;
+
+use color_eyre::Report;
+use models::InterfaceId;
+use models::PackageId;
+use serde_json::Value;
+use tokio::sync::mpsc;
+
+pub struct RuntimeDropped;
+
+pub struct Callback {
+    id: Arc<String>,
+    sender: mpsc::UnboundedSender<(Arc<String>, Vec<Value>)>,
+}
+impl Callback {
+    pub fn new(id: String, sender: mpsc::UnboundedSender<(Arc<String>, Vec<Value>)>) -> Self {
+        Self {
+            id: Arc::new(id),
+            sender,
+        }
+    }
+    pub fn is_listening(&self) -> bool {
+        self.sender.is_closed()
+    }
+    pub fn call(&self, args: Vec<Value>) -> Result<(), RuntimeDropped> {
+        self.sender
+            .send((self.id.clone(), args))
+            .map_err(|_| RuntimeDropped)
+    }
+}
+
+#[derive(serde::Deserialize, serde::Serialize, Debug, Clone)]
+#[serde(rename_all = "camelCase")]
+pub struct AddressSchemaOnion {
+    pub id: InterfaceId,
+    pub external_port: u16,
+}
+#[derive(serde::Deserialize, serde::Serialize, Debug, Clone)]
+#[serde(rename_all = "camelCase")]
+pub struct AddressSchemaLocal {
+    pub id: InterfaceId,
+    pub external_port: u16,
+}
+
+#[derive(serde::Deserialize, serde::Serialize, Debug, Clone)]
+#[serde(rename_all = "camelCase")]
+pub struct Address(pub String);
+#[derive(serde::Deserialize, serde::Serialize, Debug, Clone)]
+#[serde(rename_all = "camelCase")]
+pub struct Domain;
+#[derive(serde::Deserialize, serde::Serialize, Debug, Clone)]
+#[serde(rename_all = "camelCase")]
+pub struct Name;
+
+#[async_trait::async_trait]
+#[allow(unused_variables)]
+pub trait OsApi: Send + Sync + 'static {
+    async fn get_service_config(
+        &self,
+        id: PackageId,
+        path: &str,
+        callback: Option<Callback>,
+    ) -> Result<Vec<Value>, Report>;
+
+    async fn bind_local(
+        &self,
+        internal_port: u16,
+        address_schema: AddressSchemaLocal,
+    ) -> Result<Address, Report>;
+    async fn bind_onion(
+        &self,
+        internal_port: u16,
+        address_schema: AddressSchemaOnion,
+    ) -> Result<Address, Report>;
+
+    async fn unbind_local(&self, id: InterfaceId, external: u16) -> Result<(), Report>;
+    async fn unbind_onion(&self, id: InterfaceId, external: u16) -> Result<(), Report>;
+    fn set_started(&self) -> Result<(), Report>;
+    async fn restart(&self) -> Result<(), Report>;
+    async fn start(&self) -> Result<(), Report>;
+    async fn stop(&self) -> Result<(), Report>;
+}

core/models/Cargo.toml

@@ -6,12 +6,13 @@ edition = "2021"
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
 
 [dependencies]
-axum = "0.7.5"
-base64 = "0.21.4"
+axum = "0.8.4"
+base64 = "0.22.1"
 color-eyre = "0.6.2"
 ed25519-dalek = { version = "2.0.0", features = ["serde"] }
+gpt = "4.1.0"
 lazy_static = "1.4"
-mbrman = "0.5.2"
+mbrman = "0.6.0"
 exver = { version = "0.2.0", git = "https://github.com/Start9Labs/exver-rs.git", features = [
   "serde",
 ] }
@@ -21,21 +22,21 @@ openssl = { version = "0.10.57", features = ["vendored"] }
 patch-db = { version = "*", path = "../../patch-db/patch-db", features = [
   "trace",
 ] }
-rand = "0.8.5"
+rand = "0.9.1"
 regex = "1.10.2"
 reqwest = "0.12"
 rpc-toolkit = { git = "https://github.com/Start9Labs/rpc-toolkit.git", branch = "master" }
 rustls = "0.23"
 serde = { version = "1.0", features = ["derive", "rc"] }
 serde_json = "1.0"
-sqlx = { version = "0.7.2", features = [
+sqlx = { version = "0.8.6", features = [
   "chrono",
   "runtime-tokio-rustls",
   "postgres",
 ] }
 ssh-key = "0.6.2"
 ts-rs = { git = "https://github.com/dr-bonez/ts-rs.git", branch = "feature/top-level-as" } # "8"
-thiserror = "1.0"
+thiserror = "2.0"
 tokio = { version = "1", features = ["full"] }
 torut = { git = "https://github.com/Start9Labs/torut.git", branch = "update/dependencies" }
 tracing = "0.1.39"

core/models/src/errors.rs

@@ -10,6 +10,7 @@ use rpc_toolkit::yajrc::{
     RpcError, INVALID_PARAMS_ERROR, INVALID_REQUEST_ERROR, METHOD_NOT_FOUND_ERROR, PARSE_ERROR,
 };
 use serde::{Deserialize, Serialize};
+use tokio::task::JoinHandle;
 
 use crate::InvalidId;
 
@@ -91,6 +92,8 @@ pub enum ErrorKind {
     Cancelled = 73,
     Git = 74,
     DBus = 75,
+    InstallFailed = 76,
+    UpdateFailed = 77,
 }
 impl ErrorKind {
     pub fn as_str(&self) -> &'static str {
@@ -171,6 +174,8 @@ impl ErrorKind {
             Cancelled => "Cancelled",
             Git => "Git Error",
             DBus => "DBus Error",
+            InstallFailed => "Install Failed",
+            UpdateFailed => "Update Failed",
         }
     }
 }
@@ -185,6 +190,7 @@ pub struct Error {
     pub source: color_eyre::eyre::Error,
     pub kind: ErrorKind,
     pub revision: Option<Revision>,
+    pub task: Option<JoinHandle<()>>,
 }
 
 impl Display for Error {
@@ -198,6 +204,7 @@ impl Error {
             source: source.into(),
             kind,
             revision: None,
+            task: None,
         }
     }
     pub fn clone_output(&self) -> Self {
@@ -209,8 +216,20 @@ impl Error {
             .into(),
             kind: self.kind,
             revision: self.revision.clone(),
+            task: None,
         }
     }
+    pub fn with_task(mut self, task: JoinHandle<()>) -> Self {
+        self.task = Some(task);
+        self
+    }
+    pub async fn wait(mut self) -> Self {
+        if let Some(task) = &mut self.task {
+            task.await.log_err();
+        }
+        self.task.take();
+        self
+    }
 }
 impl axum::response::IntoResponse for Error {
     fn into_response(self) -> axum::response::Response {
@@ -304,6 +323,16 @@ impl From<mbrman::Error> for Error {
         Error::new(e, ErrorKind::DiskManagement)
     }
 }
+impl From<gpt::GptError> for Error {
+    fn from(e: gpt::GptError) -> Self {
+        Error::new(e, ErrorKind::DiskManagement)
+    }
+}
+impl From<gpt::mbr::MBRError> for Error {
+    fn from(e: gpt::mbr::MBRError) -> Self {
+        Error::new(e, ErrorKind::DiskManagement)
+    }
+}
 impl From<InvalidUri> for Error {
     fn from(e: InvalidUri) -> Self {
         Error::new(eyre!("{}", e), ErrorKind::ParseUrl)
@@ -526,6 +555,7 @@ where
             source: e.into(),
             kind,
             revision: None,
+            task: None,
         })
     }
 
@@ -539,6 +569,7 @@ where
                 kind,
                 source,
                 revision: None,
+                task: None,
             }
         })
     }
@@ -561,6 +592,7 @@ impl<T> ResultExt<T, Error> for Result<T, Error> {
             source: e.source,
             kind,
             revision: e.revision,
+            task: e.task,
         })
     }
 
@@ -574,6 +606,7 @@ impl<T> ResultExt<T, Error> for Result<T, Error> {
                 kind,
                 source,
                 revision: e.revision,
+                task: e.task,
             }
         })
     }

core/models/src/id/host.rs

@@ -63,8 +63,8 @@ impl AsRef<Path> for HostId {
 impl<'q> sqlx::Encode<'q, sqlx::Postgres> for HostId {
     fn encode_by_ref(
         &self,
-        buf: &mut <sqlx::Postgres as sqlx::database::HasArguments<'q>>::ArgumentBuffer,
-    ) -> sqlx::encode::IsNull {
+        buf: &mut <sqlx::Postgres as sqlx::Database>::ArgumentBuffer<'q>,
+    ) -> Result<sqlx::encode::IsNull, sqlx::error::BoxDynError> {
         <&str as sqlx::Encode<'q, sqlx::Postgres>>::encode_by_ref(&&**self, buf)
     }
 }

core/models/src/id/mod.rs

@@ -119,8 +119,8 @@ impl Serialize for Id {
 impl<'q> sqlx::Encode<'q, sqlx::Postgres> for Id {
     fn encode_by_ref(
         &self,
-        buf: &mut <sqlx::Postgres as sqlx::database::HasArguments<'q>>::ArgumentBuffer,
-    ) -> sqlx::encode::IsNull {
+        buf: &mut <sqlx::Postgres as sqlx::Database>::ArgumentBuffer<'q>,
+    ) -> Result<sqlx::encode::IsNull, sqlx::error::BoxDynError> {
         <&str as sqlx::Encode<'q, sqlx::Postgres>>::encode_by_ref(&&**self, buf)
     }
 }

core/models/src/id/package.rs

@@ -90,8 +90,8 @@ impl Serialize for PackageId {
 impl<'q> sqlx::Encode<'q, sqlx::Postgres> for PackageId {
     fn encode_by_ref(
         &self,
-        buf: &mut <sqlx::Postgres as sqlx::database::HasArguments<'q>>::ArgumentBuffer,
-    ) -> sqlx::encode::IsNull {
+        buf: &mut <sqlx::Postgres as sqlx::Database>::ArgumentBuffer<'q>,
+    ) -> Result<sqlx::encode::IsNull, sqlx::error::BoxDynError> {
         <&str as sqlx::Encode<'q, sqlx::Postgres>>::encode_by_ref(&&**self, buf)
     }
 }

core/models/src/id/service_interface.rs

@@ -47,8 +47,8 @@ impl AsRef<Path> for ServiceInterfaceId {
 impl<'q> sqlx::Encode<'q, sqlx::Postgres> for ServiceInterfaceId {
     fn encode_by_ref(
         &self,
-        buf: &mut <sqlx::Postgres as sqlx::database::HasArguments<'q>>::ArgumentBuffer,
-    ) -> sqlx::encode::IsNull {
+        buf: &mut <sqlx::Postgres as sqlx::Database>::ArgumentBuffer<'q>,
+    ) -> Result<sqlx::encode::IsNull, sqlx::error::BoxDynError> {
         <&str as sqlx::Encode<'q, sqlx::Postgres>>::encode_by_ref(&&**self, buf)
     }
 }

core/models/src/id/volume.rs

@@ -1,10 +1,11 @@
 use std::borrow::Borrow;
 use std::path::Path;
+use std::str::FromStr;
 
 use serde::{Deserialize, Deserializer, Serialize};
 use ts_rs::TS;
 
-use crate::Id;
+use crate::{Id, InvalidId};
 
 #[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, TS)]
 #[ts(type = "string")]
@@ -12,6 +13,15 @@ pub enum VolumeId {
     Backup,
     Custom(Id),
 }
+impl FromStr for VolumeId {
+    type Err = InvalidId;
+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        Ok(match s {
+            "BACKUP" => VolumeId::Backup,
+            s => VolumeId::Custom(Id::try_from(s.to_owned())?),
+        })
+    }
+}
 impl std::fmt::Display for VolumeId {
     fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         match self {

core/models/src/js_engine_types.rs

@@ -1,164 +0,0 @@
-use std::future::Future;
-use std::pin::Pin;
-use std::sync::Arc;
-use std::time::Duration;
-
-use color_eyre::eyre::bail;
-use container_init::{Input, Output, ProcessId, RpcId};
-use tokio::sync::mpsc::{UnboundedReceiver, UnboundedSender};
-use tokio::sync::Mutex;
-
-/// Used by the js-executor, it is the ability to just create a command in an already running exec
-pub type ExecCommand = Arc<
-    dyn Fn(
-            String,
-            Vec<String>,
-            UnboundedSender<container_init::Output>,
-            Option<Duration>,
-        ) -> Pin<Box<dyn Future<Output = Result<RpcId, String>> + 'static>>
-        + Send
-        + Sync
-        + 'static,
->;
-
-/// Used by the js-executor, it is the ability to just create a command in an already running exec
-pub type SendKillSignal = Arc<
-    dyn Fn(RpcId, u32) -> Pin<Box<dyn Future<Output = Result<(), String>> + 'static>>
-        + Send
-        + Sync
-        + 'static,
->;
-
-pub trait CommandInserter {
-    fn insert_command(
-        &self,
-        command: String,
-        args: Vec<String>,
-        sender: UnboundedSender<container_init::Output>,
-        timeout: Option<Duration>,
-    ) -> Pin<Box<dyn Future<Output = Option<RpcId>>>>;
-
-    fn send_signal(&self, id: RpcId, command: u32) -> Pin<Box<dyn Future<Output = ()>>>;
-}
-
-pub type ArcCommandInserter = Arc<Mutex<Option<Box<dyn CommandInserter>>>>;
-
-pub struct ExecutingCommand {
-    rpc_id: RpcId,
-    /// Will exist until killed
-    command_inserter: Arc<Mutex<Option<ArcCommandInserter>>>,
-    owned_futures: Arc<Mutex<Vec<Pin<Box<dyn Future<Output = ()>>>>>>,
-}
-
-impl ExecutingCommand {
-    pub async fn new(
-        command_inserter: ArcCommandInserter,
-        command: String,
-        args: Vec<String>,
-        timeout: Option<Duration>,
-    ) -> Result<ExecutingCommand, color_eyre::Report> {
-        let (sender, receiver) = tokio::sync::mpsc::unbounded_channel::<Output>();
-        let rpc_id = {
-            let locked_command_inserter = command_inserter.lock().await;
-            let locked_command_inserter = match &*locked_command_inserter {
-                Some(a) => a,
-                None => bail!("Expecting containers.main in the package manifest".to_string()),
-            };
-            match locked_command_inserter
-                .insert_command(command, args, sender, timeout)
-                .await
-            {
-                Some(a) => a,
-                None => bail!("Couldn't get command started ".to_string()),
-            }
-        };
-        let executing_commands = ExecutingCommand {
-            rpc_id,
-            command_inserter: Arc::new(Mutex::new(Some(command_inserter.clone()))),
-            owned_futures: Default::default(),
-        };
-        // let waiting = self.wait()
-        Ok(executing_commands)
-    }
-
-    async fn wait(
-        rpc_id: RpcId,
-        mut outputs: UnboundedReceiver<Output>,
-    ) -> Result<String, (Option<i32>, String)> {
-        let (process_id_send, process_id_recv) = tokio::sync::oneshot::channel::<ProcessId>();
-        let mut answer = String::new();
-        let mut command_error = String::new();
-        let mut status: Option<i32> = None;
-        let mut process_id_send = Some(process_id_send);
-        while let Some(output) = outputs.recv().await {
-            match output {
-                Output::ProcessId(process_id) => {
-                    if let Some(process_id_send) = process_id_send.take() {
-                        if let Err(err) = process_id_send.send(process_id) {
-                            tracing::error!(
-                                "Could not get a process id {process_id:?} sent for {rpc_id:?}"
-                            );
-                            tracing::debug!("{err:?}");
-                        }
-                    }
-                }
-                Output::Line(value) => {
-                    answer.push_str(&value);
-                    answer.push('\n');
-                }
-                Output::Error(error) => {
-                    command_error.push_str(&error);
-                    command_error.push('\n');
-                }
-                Output::Done(error_code) => {
-                    status = error_code;
-                    break;
-                }
-            }
-        }
-        if !command_error.is_empty() {
-            return Err((status, command_error));
-        }
-
-        Ok(answer)
-    }
-
-    async fn send_signal(&self, signal: u32) {
-        let locked = self.command_inserter.lock().await;
-        let inner = match &*locked {
-            Some(a) => a,
-            None => return,
-        };
-        let locked = inner.lock().await;
-        let command_inserter = match &*locked {
-            Some(a) => a,
-            None => return,
-        };
-        command_inserter.send_signal(self.rpc_id, signal);
-    }
-    /// Should only be called when output::done
-    async fn killed(&self) {
-        *self.owned_futures.lock().await = Default::default();
-        *self.command_inserter.lock().await = Default::default();
-    }
-    pub fn rpc_id(&self) -> RpcId {
-        self.rpc_id
-    }
-}
-
-impl Drop for ExecutingCommand {
-    fn drop(&mut self) {
-        let command_inserter = self.command_inserter.clone();
-        let rpc_id = self.rpc_id.clone();
-        tokio::spawn(async move {
-            let command_inserter_lock = command_inserter.lock().await;
-            let command_inserter = match &*command_inserter_lock {
-                Some(a) => a,
-                None => {
-                    return;
-                }
-            };
-            command_inserter.send_kill_command(rpc_id, 9).await;
-        });
-    }
-}

core/models/src/procedure_name.rs

@@ -4,25 +4,15 @@ use crate::ActionId;
 
 #[derive(Debug, Clone, Serialize, Deserialize)]
 pub enum ProcedureName {
-    GetConfig,
-    SetConfig,
     CreateBackup,
-    RestoreBackup,
     GetActionInput(ActionId),
     RunAction(ActionId),
-    PackageInit,
-    PackageUninit,
 }
 
 impl ProcedureName {
     pub fn js_function_name(&self) -> String {
         match self {
-            ProcedureName::PackageInit => "/packageInit".to_string(),
-            ProcedureName::PackageUninit => "/packageUninit".to_string(),
-            ProcedureName::SetConfig => "/config/set".to_string(),
-            ProcedureName::GetConfig => "/config/get".to_string(),
             ProcedureName::CreateBackup => "/backup/create".to_string(),
-            ProcedureName::RestoreBackup => "/backup/restore".to_string(),
             ProcedureName::RunAction(id) => format!("/actions/{}/run", id),
             ProcedureName::GetActionInput(id) => format!("/actions/{}/getInput", id),
         }

core/startos/Cargo.toml

@@ -14,7 +14,7 @@ keywords = [
 name = "start-os"
 readme = "README.md"
 repository = "https://github.com/Start9Labs/start-os"
-version = "0.3.6-alpha.18" # VERSION_BUMP
+version = "0.4.0-alpha.8" # VERSION_BUMP
 license = "MIT"
 
 [lib]
@@ -39,7 +39,7 @@ path = "src/main.rs"
 
 [features]
 cli = []
-container-runtime = ["procfs", "tty-spawn"]
+container-runtime = ["procfs", "pty-process"]
 daemon = ["mail-send"]
 registry = []
 default = ["cli", "daemon", "registry", "container-runtime"]
@@ -61,9 +61,9 @@ async-compression = { version = "0.4.4", features = [
 ] }
 async-stream = "0.3.5"
 async-trait = "0.1.74"
-axum = { version = "0.7.3", features = ["ws"] }
+axum = { version = "0.8.4", features = ["ws"] }
 barrage = "0.2.3"
-backhand = "0.18.0"
+backhand = "0.21.0"
 base32 = "0.5.0"
 base64 = "0.22.1"
 base64ct = "1.6.0"
@@ -74,7 +74,7 @@ chrono = { version = "0.4.31", features = ["serde"] }
 clap = { version = "4.4.12", features = ["string"] }
 color-eyre = "0.6.2"
 console = "0.15.7"
-console-subscriber = { version = "0.3.0", optional = true }
+console-subscriber = { version = "0.4.1", optional = true }
 const_format = "0.2.34"
 cookie = "0.18.0"
 cookie_store = "0.21.0"
@@ -96,7 +96,7 @@ exver = { version = "0.2.0", git = "https://github.com/Start9Labs/exver-rs.git",
 fd-lock-rs = "0.1.4"
 form_urlencoded = "1.2.1"
 futures = "0.3.28"
-gpt = "3.1.0"
+gpt = "4.1.0"
 helpers = { path = "../helpers" }
 hex = "0.4.3"
 hmac = "0.12.1"
@@ -116,8 +116,8 @@ id-pool = { version = "0.2.2", default-features = false, features = [
   "serde",
   "u16",
 ] }
-imbl = "2.0.3"
-imbl-value = "0.1.2"
+imbl = "4.0.1"
+imbl-value = "0.2.0"
 include_dir = { version = "0.7.3", features = ["metadata"] }
 indexmap = { version = "2.0.2", features = ["serde"] }
 indicatif = { version = "0.17.7", features = ["tokio"] }
@@ -125,10 +125,10 @@ integer-encoding = { version = "4.0.0", features = ["tokio_async"] }
 ipnet = { version = "2.8.0", features = ["serde"] }
 iprange = { version = "0.6.7", features = ["serde"] }
 isocountry = "0.3.2"
-itertools = "0.13.0"
+itertools = "0.14.0"
 jaq-core = "0.10.1"
 jaq-std = "0.10.0"
-josekit = "0.8.4"
+josekit = "0.10.3"
 jsonpath_lib = { git = "https://github.com/Start9Labs/jsonpath.git" }
 lazy_async_pool = "0.3.3"
 lazy_format = "2.0"
@@ -136,10 +136,10 @@ lazy_static = "1.4.0"
 libc = "0.2.149"
 log = "0.4.20"
 mio = "1"
-mbrman = "0.5.2"
+mbrman = "0.6.0"
 models = { version = "*", path = "../models" }
 new_mime_guess = "4"
-nix = { version = "0.29.0", features = [
+nix = { version = "0.30.1", features = [
   "fs",
   "mount",
   "net",
@@ -148,7 +148,7 @@ nix = { version = "0.29.0", features = [
   "signal",
   "user",
 ] }
-nom = "7.1.3"
+nom = "8.0.0"
 num = "0.4.1"
 num_enum = "0.7.0"
 num_cpus = "1.16.0"
@@ -163,9 +163,10 @@ pbkdf2 = "0.12.2"
 pin-project = "1.1.3"
 pkcs8 = { version = "0.10.2", features = ["std"] }
 prettytable-rs = "0.10.0"
-procfs = { version = "0.16.0", optional = true }
+procfs = { version = "0.17.0", optional = true }
 proptest = "1.3.1"
 proptest-derive = "0.5.0"
+pty-process = { version = "0.5.1", optional = true }
 qrcode = "0.14.1"
 rand = "0.9.0"
 regex = "1.10.2"
@@ -182,13 +183,14 @@ serde_json = "1.0"
 serde_toml = { package = "toml", version = "0.8.2" }
 serde_urlencoded = "0.7"
 serde_with = { version = "3.4.0", features = ["macros", "json"] }
-serde_yaml = { package = "serde_yml", version = "0.0.10" }
+serde_yaml = { package = "serde_yml", version = "0.0.12" }
+sha-crypt = "0.5.0"
 sha2 = "0.10.2"
 shell-words = "1"
 signal-hook = "0.3.17"
 simple-logging = "2.0.2"
 socket2 = "0.5.7"
-sqlx = { version = "0.7.2", features = [
+sqlx = { version = "0.8.6", features = [
   "chrono",
   "runtime-tokio-rustls",
   "postgres",
@@ -196,14 +198,15 @@ sqlx = { version = "0.7.2", features = [
 sscanf = "0.4.1"
 ssh-key = { version = "0.6.2", features = ["ed25519"] }
 tar = "0.4.40"
-thiserror = "1.0.49"
+termion = "4.0.5"
+thiserror = "2.0.12"
 textwrap = "0.16.1"
 tokio = { version = "1.38.1", features = ["full"] }
 tokio-rustls = "0.26.0"
 tokio-socks = "0.5.1"
 tokio-stream = { version = "0.1.14", features = ["io-util", "sync", "net"] }
 tokio-tar = { git = "https://github.com/dr-bonez/tokio-tar.git" }
-tokio-tungstenite = { version = "0.23.1", features = ["native-tls", "url"] }
+tokio-tungstenite = { version = "0.26.2", features = ["native-tls", "url"] }
 tokio-util = { version = "0.7.9", features = ["io"] }
 torut = { git = "https://github.com/Start9Labs/torut.git", branch = "update/dependencies", features = [
   "serialize",
@@ -216,8 +219,7 @@ tracing-journald = "0.3.0"
 tracing-subscriber = { version = "0.3.17", features = ["env-filter"] }
 trust-dns-server = "0.23.1"
 ts-rs = { git = "https://github.com/dr-bonez/ts-rs.git", branch = "feature/top-level-as" } # "8.1.0"
-tty-spawn = { version = "0.4.0", optional = true }
-typed-builder = "0.18.0"
+typed-builder = "0.21.0"
 unix-named-pipe = "0.2.0"
 url = { version = "2.4.1", features = ["serde"] }
 urlencoding = "2.1.3"

core/startos/src/account.rs

@@ -59,7 +59,13 @@ impl AccountInfo {
         let hostname = Hostname(db.as_public().as_server_info().as_hostname().de()?);
         let password = db.as_private().as_password().de()?;
         let key_store = db.as_private().as_key_store();
-        let tor_addrs = db.as_public().as_server_info().as_host().as_onions().de()?;
+        let tor_addrs = db
+            .as_public()
+            .as_server_info()
+            .as_network()
+            .as_host()
+            .as_onions()
+            .de()?;
         let tor_keys = tor_addrs
             .into_iter()
             .map(|tor_addr| key_store.as_onion().get_key(&tor_addr))
@@ -89,13 +95,17 @@ impl AccountInfo {
         server_info
             .as_pubkey_mut()
             .ser(&self.ssh_key.public_key().to_openssh()?)?;
-        server_info.as_host_mut().as_onions_mut().ser(
-            &self
-                .tor_keys
-                .iter()
-                .map(|tor_key| tor_key.public().get_onion_address())
-                .collect(),
-        )?;
+        server_info
+            .as_network_mut()
+            .as_host_mut()
+            .as_onions_mut()
+            .ser(
+                &self
+                    .tor_keys
+                    .iter()
+                    .map(|tor_key| tor_key.public().get_onion_address())
+                    .collect(),
+            )?;
         db.as_private_mut().as_password_mut().ser(&self.password)?;
         db.as_private_mut()
             .as_ssh_privkey_mut()

core/startos/src/action.rs

@@ -2,7 +2,7 @@ use std::fmt;
 
 use clap::{CommandFactory, FromArgMatches, Parser};
 pub use models::ActionId;
-use models::PackageId;
+use models::{PackageId, ReplayId};
 use qrcode::QrCode;
 use rpc_toolkit::{from_fn_async, Context, HandlerExt, ParentHandler};
 use serde::{Deserialize, Serialize};
@@ -10,6 +10,7 @@ use tracing::instrument;
 use ts_rs::TS;
 
 use crate::context::{CliContext, RpcContext};
+use crate::db::model::package::TaskSeverity;
 use crate::prelude::*;
 use crate::rpc_continuations::Guid;
 use crate::util::serde::{
@@ -38,6 +39,13 @@ pub fn action_api<C: Context>() -> ParentHandler<C> {
                 .with_about("Run service action")
                 .with_call_remote::<CliContext>(),
         )
+        .subcommand(
+            "clear-task",
+            from_fn_async(clear_task)
+                .no_display()
+                .with_about("Clear a service task")
+                .with_call_remote::<CliContext>(),
+        )
 }
 
 #[derive(Debug, Clone, Deserialize, Serialize, TS)]
@@ -83,6 +91,28 @@ pub enum ActionResult {
     #[serde(rename = "1")]
     V1(ActionResultV1),
 }
+impl ActionResult {
+    pub fn upcast(self) -> Self {
+        match self {
+            Self::V0(ActionResultV0 {
+                message,
+                value,
+                copyable,
+                qr,
+            }) => Self::V1(ActionResultV1 {
+                title: "Action Complete".into(),
+                message: Some(message),
+                result: value.map(|value| ActionResultValue::Single {
+                    value,
+                    copyable,
+                    qr,
+                    masked: false,
+                }),
+            }),
+            Self::V1(a) => Self::V1(a),
+        }
+    }
+}
 impl fmt::Display for ActionResult {
     fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
         match self {
@@ -222,14 +252,18 @@ impl fmt::Display for ActionResultV1 {
     }
 }
 
-pub fn display_action_result<T: Serialize>(params: WithIoFormat<T>, result: Option<ActionResult>) {
+pub fn display_action_result<T: Serialize>(
+    params: WithIoFormat<T>,
+    result: Option<ActionResult>,
+) -> Result<(), Error> {
     let Some(result) = result else {
-        return;
+        return Ok(());
     };
     if let Some(format) = params.format {
         return display_serializable(format, result);
     }
-    println!("{result}")
+    println!("{result}");
+    Ok(())
 }
 
 #[derive(Deserialize, Serialize, TS)]
@@ -308,4 +342,48 @@ pub async fn run_action(
         .or_not_found(lazy_format!("Manager for {}", package_id))?
         .run_action(Guid::new(), action_id, input.unwrap_or_default())
         .await
+        .map(|res| res.map(ActionResult::upcast))
+}
+
+#[derive(Deserialize, Serialize, Parser, TS)]
+#[serde(rename_all = "camelCase")]
+#[command(rename_all = "kebab-case")]
+pub struct ClearTaskParams {
+    pub package_id: PackageId,
+    pub replay_id: ReplayId,
+    #[arg(long)]
+    #[serde(default)]
+    pub force: bool,
+}
+
+#[instrument(skip_all)]
+pub async fn clear_task(
+    ctx: RpcContext,
+    ClearTaskParams {
+        package_id,
+        replay_id,
+        force,
+    }: ClearTaskParams,
+) -> Result<(), Error> {
+    ctx.db
+        .mutate(|db| {
+            if let Some(task) = db
+                .as_public_mut()
+                .as_package_data_mut()
+                .as_idx_mut(&package_id)
+                .or_not_found(&package_id)?
+                .as_tasks_mut()
+                .remove(&replay_id)?
+            {
+                if !force && task.as_task().as_severity().de()? == TaskSeverity::Critical {
+                    return Err(Error::new(
+                        eyre!("Cannot clear critical task"),
+                        ErrorKind::InvalidRequest,
+                    ));
+                }
+            }
+            Ok(())
+        })
+        .await
+        .result
 }

core/startos/src/auth.rs

@@ -9,6 +9,7 @@ use josekit::jwk::Jwk;
 use rpc_toolkit::yajrc::RpcError;
 use rpc_toolkit::{from_fn_async, Context, HandlerArgs, HandlerExt, ParentHandler};
 use serde::{Deserialize, Serialize};
+use tokio::io::AsyncWriteExt;
 use tracing::instrument;
 use ts_rs::TS;
 
@@ -19,6 +20,7 @@ use crate::middleware::auth::{
 };
 use crate::prelude::*;
 use crate::util::crypto::EncryptedWire;
+use crate::util::io::create_file_mod;
 use crate::util::serde::{display_serializable, HandlerExtSerde, WithIoFormat};
 use crate::{ensure_code, Error, ResultExt};
 
@@ -41,6 +43,33 @@ impl Map for Sessions {
     }
 }
 
+pub async fn write_shadow(password: &str) -> Result<(), Error> {
+    let hash: String = sha_crypt::sha512_simple(password, &sha_crypt::Sha512Params::default())
+        .map_err(|e| Error::new(eyre!("{e:?}"), ErrorKind::Serialization))?;
+    let shadow_contents = tokio::fs::read_to_string("/etc/shadow").await?;
+    let mut shadow_file =
+        create_file_mod("/media/startos/config/overlay/etc/shadow", 0o640).await?;
+    for line in shadow_contents.lines() {
+        match line.split_once(":") {
+            Some((user, rest)) if user == "start9" || user == "kiosk" => {
+                let (_, rest) = rest.split_once(":").ok_or_else(|| {
+                    Error::new(eyre!("malformed /etc/shadow"), ErrorKind::ParseSysInfo)
+                })?;
+                shadow_file
+                    .write_all(format!("{user}:{hash}:{rest}\n").as_bytes())
+                    .await?;
+            }
+            _ => {
+                shadow_file.write_all(line.as_bytes()).await?;
+                shadow_file.write_all(b"\n").await?;
+            }
+        }
+    }
+    shadow_file.sync_all().await?;
+    tokio::fs::copy("/media/startos/config/overlay/etc/shadow", "/etc/shadow").await?;
+    Ok(())
+}
+
 #[derive(Clone, Serialize, Deserialize, TS)]
 #[serde(untagged)]
 #[ts(export)]
@@ -197,9 +226,6 @@ pub struct LoginParams {
     user_agent: Option<String>,
     #[serde(default)]
     ephemeral: bool,
-    #[serde(default)]
-    #[ts(type = "any")]
-    metadata: Value,
 }
 
 #[instrument(skip_all)]
@@ -209,12 +235,11 @@ pub async fn login_impl(
         password,
         user_agent,
         ephemeral,
-        metadata,
     }: LoginParams,
 ) -> Result<LoginRes, Error> {
     let password = password.unwrap_or_default().decrypt(&ctx)?;
 
-    if ephemeral {
+    let tok = if ephemeral {
         check_password_against_db(&ctx.db.peek().await, &password)?;
         let hash_token = HashSessionToken::new();
         ctx.ephemeral_sessions.mutate(|s| {
@@ -224,7 +249,6 @@ pub async fn login_impl(
                     logged_in: Utc::now(),
                     last_active: Utc::now(),
                     user_agent,
-                    metadata,
                 },
             )
         });
@@ -240,7 +264,6 @@ pub async fn login_impl(
                         logged_in: Utc::now(),
                         last_active: Utc::now(),
                         user_agent,
-                        metadata,
                     },
                 )?;
 
@@ -248,7 +271,16 @@ pub async fn login_impl(
             })
             .await
             .result
+    }?;
+
+    if tokio::fs::metadata("/media/startos/config/overlay/etc/shadow")
+        .await
+        .is_err()
+    {
+        write_shadow(&password).await?;
     }
+
+    Ok(tok)
 }
 
 #[derive(Deserialize, Serialize, Parser, TS)]
@@ -277,10 +309,7 @@ pub struct Session {
     pub logged_in: DateTime<Utc>,
     #[ts(type = "string")]
     pub last_active: DateTime<Utc>,
-    #[ts(skip)]
     pub user_agent: Option<String>,
-    #[ts(type = "any")]
-    pub metadata: Value,
 }
 
 #[derive(Deserialize, Serialize, TS)]
@@ -299,9 +328,7 @@ pub fn session<C: Context>() -> ParentHandler<C> {
             from_fn_async(list)
                 .with_metadata("get_session", Value::Bool(true))
                 .with_display_serializable()
-                .with_custom_display_fn(|handle, result| {
-                    Ok(display_sessions(handle.params, result))
-                })
+                .with_custom_display_fn(|handle, result| display_sessions(handle.params, result))
                 .with_about("Display all server sessions")
                 .with_call_remote::<CliContext>(),
         )
@@ -314,7 +341,7 @@ pub fn session<C: Context>() -> ParentHandler<C> {
         )
 }
 
-fn display_sessions(params: WithIoFormat<ListParams>, arg: SessionList) {
+fn display_sessions(params: WithIoFormat<ListParams>, arg: SessionList) -> Result<(), Error> {
     use prettytable::*;
 
     if let Some(format) = params.format {
@@ -327,7 +354,6 @@ fn display_sessions(params: WithIoFormat<ListParams>, arg: SessionList) {
         "LOGGED IN",
         "LAST ACTIVE",
         "USER AGENT",
-        "METADATA",
     ]);
     for (id, session) in arg.sessions.0 {
         let mut row = row![
@@ -335,7 +361,6 @@ fn display_sessions(params: WithIoFormat<ListParams>, arg: SessionList) {
             &format!("{}", session.logged_in),
             &format!("{}", session.last_active),
             session.user_agent.as_deref().unwrap_or("N/A"),
-            &format!("{}", session.metadata),
         ];
         if Some(id) == arg.current {
             row.iter_mut()
@@ -344,7 +369,8 @@ fn display_sessions(params: WithIoFormat<ListParams>, arg: SessionList) {
         }
         table.add_row(row);
     }
-    table.print_tty(false).unwrap();
+    table.print_tty(false)?;
+    Ok(())
 }
 
 #[derive(Deserialize, Serialize, Parser, TS)]

core/startos/src/backup/backup_bulk.rs

@@ -223,18 +223,7 @@ fn assure_backing_up<'a>(
         .as_server_info_mut()
         .as_status_info_mut()
         .as_backup_progress_mut();
-    if backing_up
-        .clone()
-        .de()?
-        .iter()
-        .flat_map(|x| x.values())
-        .fold(false, |acc, x| {
-            if !x.complete {
-                return true;
-            }
-            acc
-        })
-    {
+    if backing_up.transpose_ref().is_some() {
         return Err(Error::new(
             eyre!("Server is already backing up!"),
             ErrorKind::InvalidRequest,
@@ -287,6 +276,22 @@ async fn perform_backup(
                         timestamp: Utc::now(),
                     },
                 );
+
+                ctx.db
+                    .mutate(|db| {
+                        if let Some(progress) = db
+                            .as_public_mut()
+                            .as_server_info_mut()
+                            .as_status_info_mut()
+                            .as_backup_progress_mut()
+                            .transpose_mut()
+                        {
+                            progress.insert(&id, &BackupProgress { complete: true })?;
+                        }
+                        Ok(())
+                    })
+                    .await
+                    .result?;
             }
             backup_report.insert(
                 id.clone(),

core/startos/src/backup/restore.rs

@@ -20,10 +20,13 @@ use crate::disk::mount::filesystem::ReadWrite;
 use crate::disk::mount::guard::{GenericMountGuard, TmpMountGuard};
 use crate::init::init;
 use crate::prelude::*;
+use crate::progress::ProgressUnits;
 use crate::s9pk::S9pk;
 use crate::service::service_map::DownloadInstallFuture;
 use crate::setup::SetupExecuteProgress;
+use crate::system::sync_kiosk;
 use crate::util::serde::IoFormat;
+use crate::PLATFORM;
 
 #[derive(Deserialize, Serialize, Parser, TS)]
 #[serde(rename_all = "camelCase")]
@@ -80,6 +83,7 @@ pub async fn recover_full_embassy(
     recovery_source: TmpMountGuard,
     server_id: &str,
     recovery_password: &str,
+    kiosk: Option<bool>,
     SetupExecuteProgress {
         init_phases,
         restore_phase,
@@ -105,8 +109,12 @@ pub async fn recover_full_embassy(
     )
     .with_kind(ErrorKind::PasswordHashGeneration)?;
 
+    let kiosk = Some(kiosk.unwrap_or(true)).filter(|_| &*PLATFORM != "raspberrypi");
+    sync_kiosk(kiosk).await?;
+
     let db = ctx.db().await?;
-    db.put(&ROOT, &Database::init(&os_backup.account)?).await?;
+    db.put(&ROOT, &Database::init(&os_backup.account, kiosk)?)
+        .await?;
     drop(db);
 
     let init_result = init(&ctx.webserver, &ctx.config, init_phases).await?;
@@ -129,6 +137,7 @@ pub async fn recover_full_embassy(
         .collect();
     let tasks = restore_packages(&rpc_ctx, backup_guard, ids).await?;
     restore_phase.set_total(tasks.len() as u64);
+    restore_phase.set_units(Some(ProgressUnits::Steps));
     let restore_phase = Arc::new(Mutex::new(restore_phase));
     stream::iter(tasks)
         .for_each_concurrent(5, |(id, res)| {
@@ -166,6 +175,7 @@ async fn restore_packages(
             .install(
                 ctx.clone(),
                 || S9pk::open(s9pk_path, Some(&id)),
+                None, // TODO: pull from metadata?
                 Some(backup_dir),
                 None,
             )

core/startos/src/backup/target/mod.rs

@@ -157,7 +157,7 @@ pub fn target<C: Context>() -> ParentHandler<C> {
             from_fn_async(info)
                 .with_display_serializable()
                 .with_custom_display_fn::<CliContext, _>(|params, info| {
-                    Ok(display_backup_info(params.params, info))
+                    display_backup_info(params.params, info)
                 })
                 .with_about("Display package backup information")
                 .with_call_remote::<CliContext>(),
@@ -227,7 +227,7 @@ pub struct PackageBackupInfo {
     pub timestamp: DateTime<Utc>,
 }
 
-fn display_backup_info(params: WithIoFormat<InfoParams>, info: BackupInfo) {
+fn display_backup_info(params: WithIoFormat<InfoParams>, info: BackupInfo) -> Result<(), Error> {
     use prettytable::*;
 
     if let Some(format) = params.format {
@@ -260,7 +260,8 @@ fn display_backup_info(params: WithIoFormat<InfoParams>, info: BackupInfo) {
         ];
         table.add_row(row);
     }
-    table.print_tty(false).unwrap();
+    table.print_tty(false)?;
+    Ok(())
 }
 
 #[derive(Deserialize, Serialize, Parser, TS)]
@@ -296,7 +297,7 @@ pub async fn info(
 }
 
 lazy_static::lazy_static! {
-    static ref USER_MOUNTS: Mutex<BTreeMap<BackupTargetId, BackupMountGuard<TmpMountGuard>>> =
+    static ref USER_MOUNTS: Mutex<BTreeMap<BackupTargetId, Result<BackupMountGuard<TmpMountGuard>, TmpMountGuard>>> =
         Mutex::new(BTreeMap::new());
 }
 
@@ -305,8 +306,11 @@ lazy_static::lazy_static! {
 #[command(rename_all = "kebab-case")]
 pub struct MountParams {
     target_id: BackupTargetId,
-    server_id: String,
+    #[arg(long)]
+    server_id: Option<String>,
     password: String,
+    #[arg(long)]
+    allow_partial: bool,
 }
 
 #[instrument(skip_all)]
@@ -316,24 +320,63 @@ pub async fn mount(
         target_id,
         server_id,
         password,
+        allow_partial,
     }: MountParams,
 ) -> Result<String, Error> {
+    let server_id = if let Some(server_id) = server_id {
+        server_id
+    } else {
+        ctx.db
+            .peek()
+            .await
+            .into_public()
+            .into_server_info()
+            .into_id()
+            .de()?
+    };
+
     let mut mounts = USER_MOUNTS.lock().await;
 
-    if let Some(existing) = mounts.get(&target_id) {
-        return Ok(existing.path().display().to_string());
-    }
+    let existing = mounts.get(&target_id);
 
-    let guard = BackupMountGuard::mount(
-        TmpMountGuard::mount(&target_id.clone().load(&ctx.db.peek().await)?, ReadWrite).await?,
-        &server_id,
-        &password,
-    )
-    .await?;
+    let base = match existing {
+        Some(Ok(a)) => return Ok(a.path().display().to_string()),
+        Some(Err(e)) => e.clone(),
+        None => {
+            TmpMountGuard::mount(&target_id.clone().load(&ctx.db.peek().await)?, ReadWrite).await?
+        }
+    };
+
+    let guard = match BackupMountGuard::mount(base.clone(), &server_id, &password).await {
+        Ok(a) => a,
+        Err(e) => {
+            if allow_partial {
+                mounts.insert(target_id, Err(base.clone()));
+                let enc_key = BackupMountGuard::<TmpMountGuard>::load_metadata(
+                    base.path(),
+                    &server_id,
+                    &password,
+                )
+                .await
+                .map(|(_, k)| k);
+                return Err(e)
+                    .with_ctx(|e| (
+                        e.kind,
+                        format!(
+                            "\nThe base filesystem did successfully mount at {:?}\nWrapped Key: {:?}",
+                            base.path(),
+                            enc_key
+                        )
+                    ));
+            } else {
+                return Err(e);
+            }
+        }
+    };
 
     let res = guard.path().display().to_string();
 
-    mounts.insert(target_id, guard);
+    mounts.insert(target_id, Ok(guard));
 
     Ok(res)
 }
@@ -350,11 +393,17 @@ pub async fn umount(_: RpcContext, UmountParams { target_id }: UmountParams) ->
     let mut mounts = USER_MOUNTS.lock().await; // TODO: move to context
     if let Some(target_id) = target_id {
         if let Some(existing) = mounts.remove(&target_id) {
-            existing.unmount().await?;
+            match existing {
+                Ok(e) => e.unmount().await?,
+                Err(e) => e.unmount().await?,
+            }
         }
     } else {
         for (_, existing) in std::mem::take(&mut *mounts) {
-            existing.unmount().await?;
+            match existing {
+                Ok(e) => e.unmount().await?,
+                Err(e) => e.unmount().await?,
+            }
         }
     }
 

core/startos/src/bins/startd.rs

@@ -1,6 +1,5 @@
 use std::cmp::max;
 use std::ffi::OsString;
-use std::net::IpAddr;
 use std::sync::Arc;
 use std::time::Duration;
 
@@ -14,7 +13,6 @@ use crate::context::config::ServerConfig;
 use crate::context::rpc::InitRpcContextPhases;
 use crate::context::{DiagnosticContext, InitContext, RpcContext};
 use crate::net::network_interface::SelfContainedNetworkInterfaceListener;
-use crate::net::utils::ipv6_is_local;
 use crate::net::web_server::{Acceptor, UpgradableListener, WebServer};
 use crate::shutdown::Shutdown;
 use crate::system::launch_metrics_task;

core/startos/src/config/hook.rs

@@ -0,0 +1,53 @@
+use helpers::Callback;
+use itertools::Itertools;
+use jsonpath_lib::Compiled;
+use models::PackageId;
+use serde_json::Value;
+
+use crate::context::RpcContext;
+
+pub struct ConfigHook {
+    pub path: Compiled,
+    pub prev: Vec<Value>,
+    pub callback: Callback,
+}
+
+impl RpcContext {
+    pub async fn add_config_hook(&self, id: PackageId, hook: ConfigHook) {
+        let mut hooks = self.config_hooks.lock().await;
+        let prev = hooks.remove(&id).unwrap_or_default();
+        hooks.insert(
+            id,
+            prev.into_iter()
+                .filter(|h| h.callback.is_listening())
+                .chain(std::iter::once(hook))
+                .collect(),
+        );
+    }
+
+    pub async fn call_config_hooks(&self, id: PackageId, config: &Value) {
+        let mut hooks = self.config_hooks.lock().await;
+        let mut prev = hooks.remove(&id).unwrap_or_default();
+        for hook in &mut prev {
+            let new = hook
+                .path
+                .select(config)
+                .unwrap_or_default()
+                .into_iter()
+                .cloned()
+                .collect_vec();
+            if new != hook.prev {
+                hook.callback
+                    .call(vec![Value::Array(new.clone())])
+                    .unwrap_or_default();
+                hook.prev = new;
+            }
+        }
+        hooks.insert(
+            id,
+            prev.into_iter()
+                .filter(|h| h.callback.is_listening())
+                .collect(),
+        );
+    }
+}

core/startos/src/context/cli.rs

@@ -37,6 +37,11 @@ pub struct CliContextSeed {
 }
 impl Drop for CliContextSeed {
     fn drop(&mut self) {
+        if let Some(rt) = self.runtime.take() {
+            if let Ok(rt) = Arc::try_unwrap(rt) {
+                rt.shutdown_background();
+            }
+        }
         let tmp = format!("{}.tmp", self.cookie_path.display());
         let parent_dir = self.cookie_path.parent().unwrap_or(Path::new("/"));
         if !parent_dir.exists() {

core/startos/src/context/rpc.rs

@@ -16,7 +16,7 @@ use models::{ActionId, PackageId};
 use reqwest::{Client, Proxy};
 use rpc_toolkit::yajrc::RpcError;
 use rpc_toolkit::{CallRemote, Context, Empty};
-use tokio::sync::{broadcast, watch, Mutex, RwLock};
+use tokio::sync::{broadcast, oneshot, watch, Mutex, RwLock};
 use tokio::time::Instant;
 use tracing::instrument;
 
@@ -24,6 +24,7 @@ use super::setup::CURRENT_SECRET;
 use crate::account::AccountInfo;
 use crate::auth::Sessions;
 use crate::context::config::ServerConfig;
+use crate::db::model::package::TaskSeverity;
 use crate::db::model::Database;
 use crate::disk::OsPartitionInfo;
 use crate::init::{check_time_is_synchronized, InitResult};
@@ -35,12 +36,12 @@ use crate::net::wifi::WpaCli;
 use crate::prelude::*;
 use crate::progress::{FullProgressTracker, PhaseProgressTrackerHandle};
 use crate::rpc_continuations::{Guid, OpenAuthedContinuations, RpcContinuations};
-use crate::service::action::update_requested_actions;
+use crate::service::action::update_tasks;
 use crate::service::effects::callbacks::ServiceCallbacks;
 use crate::service::ServiceMap;
 use crate::shutdown::Shutdown;
 use crate::util::lshw::LshwDevice;
-use crate::util::sync::SyncMutex;
+use crate::util::sync::{SyncMutex, Watch};
 
 pub struct RpcContextSeed {
     is_closed: AtomicBool,
@@ -56,14 +57,15 @@ pub struct RpcContextSeed {
     pub os_net_service: NetService,
     pub s9pk_arch: Option<&'static str>,
     pub services: ServiceMap,
-    pub metrics_cache: RwLock<Option<crate::system::Metrics>>,
+    pub cancellable_installs: SyncMutex<BTreeMap<PackageId, oneshot::Sender<()>>>,
+    pub metrics_cache: Watch<Option<crate::system::Metrics>>,
     pub shutdown: broadcast::Sender<Option<Shutdown>>,
     pub tor_socks: SocketAddr,
     pub lxc_manager: Arc<LxcManager>,
     pub open_authed_continuations: OpenAuthedContinuations<Option<InternedString>>,
     pub rpc_continuations: RpcContinuations,
     pub callbacks: Arc<ServiceCallbacks>,
-    pub wifi_manager: Option<Arc<RwLock<WpaCli>>>,
+    pub wifi_manager: Arc<RwLock<Option<WpaCli>>>,
     pub current_secret: Arc<Jwk>,
     pub client: Client,
     pub start_time: Instant,
@@ -101,14 +103,14 @@ impl InitRpcContextPhases {
 pub struct CleanupInitPhases {
     cleanup_sessions: PhaseProgressTrackerHandle,
     init_services: PhaseProgressTrackerHandle,
-    check_requested_actions: PhaseProgressTrackerHandle,
+    check_tasks: PhaseProgressTrackerHandle,
 }
 impl CleanupInitPhases {
     pub fn new(handle: &FullProgressTracker) -> Self {
         Self {
             cleanup_sessions: handle.add_phase("Cleaning up sessions".into(), Some(1)),
             init_services: handle.add_phase("Initializing services".into(), Some(10)),
-            check_requested_actions: handle.add_phase("Checking action requests".into(), Some(1)),
+            check_tasks: handle.add_phase("Checking action requests".into(), Some(1)),
         }
     }
 }
@@ -173,7 +175,7 @@ impl RpcContext {
         tracing::info!("Initialized Net Controller");
 
         let services = ServiceMap::default();
-        let metrics_cache = RwLock::<Option<crate::system::Metrics>>::new(None);
+        let metrics_cache = Watch::<Option<crate::system::Metrics>>::new(None);
         let tor_proxy_url = format!("socks5h://{tor_proxy}");
 
         let crons = SyncMutex::new(BTreeMap::new());
@@ -239,15 +241,14 @@ impl RpcContext {
                 Some(crate::ARCH)
             },
             services,
+            cancellable_installs: SyncMutex::new(BTreeMap::new()),
             metrics_cache,
             shutdown,
             tor_socks: tor_proxy,
             lxc_manager: Arc::new(LxcManager::new()),
             open_authed_continuations: OpenAuthedContinuations::new(),
             rpc_continuations: RpcContinuations::new(),
-            wifi_manager: wifi_interface
-                .clone()
-                .map(|i| Arc::new(RwLock::new(WpaCli::init(i)))),
+            wifi_manager: Arc::new(RwLock::new(wifi_interface.clone().map(|i| WpaCli::init(i)))),
             current_secret: Arc::new(
                 Jwk::generate_ec_key(josekit::jwk::alg::ec::EcCurve::P256).map_err(|e| {
                     tracing::debug!("{:?}", e);
@@ -307,7 +308,7 @@ impl RpcContext {
         CleanupInitPhases {
             mut cleanup_sessions,
             init_services,
-            mut check_requested_actions,
+            mut check_tasks,
         }: CleanupInitPhases,
     ) -> Result<(), Error> {
         cleanup_sessions.start();
@@ -369,35 +370,33 @@ impl RpcContext {
         tracing::info!("Initialized Services");
 
         // TODO
-        check_requested_actions.start();
+        check_tasks.start();
         let peek = self.db.peek().await;
         let mut action_input: OrdMap<PackageId, BTreeMap<ActionId, Value>> = OrdMap::new();
-        let requested_actions: BTreeSet<_> = peek
+        let tasks: BTreeSet<_> = peek
             .as_public()
             .as_package_data()
             .as_entries()?
             .into_iter()
             .map(|(_, pde)| {
-                Ok(pde
-                    .as_requested_actions()
-                    .as_entries()?
-                    .into_iter()
-                    .map(|(_, r)| {
-                        Ok::<_, Error>((
-                            r.as_request().as_package_id().de()?,
-                            r.as_request().as_action_id().de()?,
-                        ))
-                    }))
+                Ok(pde.as_tasks().as_entries()?.into_iter().map(|(_, r)| {
+                    Ok::<_, Error>((
+                        r.as_task().as_package_id().de()?,
+                        r.as_task().as_action_id().de()?,
+                    ))
+                }))
             })
             .flatten_ok()
             .map(|a| a.and_then(|a| a))
             .try_collect()?;
         let procedure_id = Guid::new();
-        for (package_id, action_id) in requested_actions {
+        for (package_id, action_id) in tasks {
             if let Some(service) = self.services.get(&package_id).await.as_ref() {
                 if let Some(input) = service
                     .get_action_input(procedure_id.clone(), action_id.clone())
-                    .await?
+                    .await
+                    .log_err()
+                    .flatten()
                     .and_then(|i| i.value)
                 {
                     action_input
@@ -407,28 +406,47 @@ impl RpcContext {
                 }
             }
         }
-        self.db
-            .mutate(|db| {
-                for (package_id, action_input) in &action_input {
-                    for (action_id, input) in action_input {
-                        for (_, pde) in db.as_public_mut().as_package_data_mut().as_entries_mut()? {
-                            pde.as_requested_actions_mut().mutate(|requested_actions| {
-                                Ok(update_requested_actions(
-                                    requested_actions,
-                                    package_id,
-                                    action_id,
-                                    input,
-                                    false,
-                                ))
-                            })?;
+        for id in
+            self.db
+                .mutate::<Vec<PackageId>>(|db| {
+                    for (package_id, action_input) in &action_input {
+                        for (action_id, input) in action_input {
+                            for (_, pde) in
+                                db.as_public_mut().as_package_data_mut().as_entries_mut()?
+                            {
+                                pde.as_tasks_mut().mutate(|tasks| {
+                                    Ok(update_tasks(tasks, package_id, action_id, input, false))
+                                })?;
+                            }
                         }
                     }
-                }
-                Ok(())
-            })
-            .await
-            .result?;
-        check_requested_actions.complete();
+                    db.as_public()
+                        .as_package_data()
+                        .as_entries()?
+                        .into_iter()
+                        .filter_map(|(id, pkg)| {
+                            (|| {
+                                if pkg.as_tasks().de()?.into_iter().any(|(_, t)| {
+                                    t.active && t.task.severity == TaskSeverity::Critical
+                                }) {
+                                    Ok(Some(id))
+                                } else {
+                                    Ok(None)
+                                }
+                            })()
+                            .transpose()
+                        })
+                        .collect()
+                })
+                .await
+                .result?
+        {
+            let svc = self.services.get(&id).await;
+            if let Some(svc) = &*svc {
+                svc.stop(procedure_id.clone(), false).await?;
+            }
+        }
+        check_tasks.complete();
 
         Ok(())
     }

core/startos/src/context/setup.rs

@@ -172,7 +172,8 @@ impl SetupContext {
                                         if let Some(progress) = progress {
                                             ws.send(ws::Message::Text(
                                                 serde_json::to_string(&progress)
-                                                    .with_kind(ErrorKind::Serialization)?,
+                                                    .with_kind(ErrorKind::Serialization)?
+                                                    .into(),
                                             ))
                                             .await
                                             .with_kind(ErrorKind::Network)?;

core/startos/src/control.rs

@@ -31,13 +31,12 @@ pub async fn start(ctx: RpcContext, ControlParams { id }: ControlParams) -> Resu
 }
 
 pub async fn stop(ctx: RpcContext, ControlParams { id }: ControlParams) -> Result<(), Error> {
-    // TODO: why did this return last_status before?
     ctx.services
         .get(&id)
         .await
         .as_ref()
         .ok_or_else(|| Error::new(eyre!("Manager not found"), crate::ErrorKind::InvalidRequest))?
-        .stop(Guid::new())
+        .stop(Guid::new(), true)
         .await?;
 
     Ok(())
@@ -49,7 +48,7 @@ pub async fn restart(ctx: RpcContext, ControlParams { id }: ControlParams) -> Re
         .await
         .as_ref()
         .ok_or_else(|| Error::new(eyre!("Manager not found"), crate::ErrorKind::InvalidRequest))?
-        .restart(Guid::new())
+        .restart(Guid::new(), false)
         .await?;
 
     Ok(())

core/startos/src/db/mod.rs

@@ -203,7 +203,9 @@ pub async fn subscribe(
                                 rev = sub.recv() => {
                                     if let Some(rev) = rev {
                                         ws.send(ws::Message::Text(
-                                            serde_json::to_string(&rev).with_kind(ErrorKind::Serialization)?,
+                                            serde_json::to_string(&rev)
+                                                .with_kind(ErrorKind::Serialization)?
+                                                .into(),
                                         ))
                                         .await
                                         .with_kind(ErrorKind::Network)?;

core/startos/src/db/model/mod.rs

@@ -27,9 +27,9 @@ pub struct Database {
     pub private: Private,
 }
 impl Database {
-    pub fn init(account: &AccountInfo) -> Result<Self, Error> {
+    pub fn init(account: &AccountInfo, kiosk: Option<bool>) -> Result<Self, Error> {
         Ok(Self {
-            public: Public::init(account)?,
+            public: Public::init(account, kiosk)?,
             private: Private {
                 key_store: KeyStore::new(account)?,
                 password: account.password.clone(),

core/startos/src/db/model/package.rs

@@ -3,10 +3,7 @@ use std::collections::{BTreeMap, BTreeSet};
 use chrono::{DateTime, Utc};
 use exver::VersionRange;
 use imbl_value::InternedString;
-use models::{
-    ActionId, DataUrl, HealthCheckId, HostId, PackageId, ReplayId, ServiceInterfaceId,
-    VersionString,
-};
+use models::{ActionId, DataUrl, HealthCheckId, HostId, PackageId, ReplayId, ServiceInterfaceId};
 use patch_db::json_ptr::JsonPointer;
 use patch_db::HasModel;
 use reqwest::Url;
@@ -365,7 +362,7 @@ impl Default for ActionVisibility {
 #[ts(export)]
 pub struct PackageDataEntry {
     pub state_info: PackageState,
-    pub data_version: Option<VersionString>,
+    pub data_version: Option<String>,
     pub status: MainStatus,
     #[ts(type = "string | null")]
     pub registry: Option<Url>,
@@ -376,8 +373,8 @@ pub struct PackageDataEntry {
     pub last_backup: Option<DateTime<Utc>>,
     pub current_dependencies: CurrentDependencies,
     pub actions: BTreeMap<ActionId, ActionMetadata>,
-    #[ts(as = "BTreeMap::<String, ActionRequestEntry>")]
-    pub requested_actions: BTreeMap<ReplayId, ActionRequestEntry>,
+    #[ts(as = "BTreeMap::<String, TaskEntry>")]
+    pub tasks: BTreeMap<ReplayId, TaskEntry>,
     pub service_interfaces: BTreeMap<ServiceInterfaceId, ServiceInterface>,
     pub hosts: Hosts,
     #[ts(type = "string[]")]
@@ -444,8 +441,8 @@ pub enum CurrentDependencyKind {
 #[serde(rename_all = "camelCase")]
 #[ts(export)]
 #[model = "Model<Self>"]
-pub struct ActionRequestEntry {
-    pub request: ActionRequest,
+pub struct TaskEntry {
+    pub task: Task,
     pub active: bool,
 }
 
@@ -453,58 +450,59 @@ pub struct ActionRequestEntry {
 #[serde(rename_all = "camelCase")]
 #[ts(export)]
 #[model = "Model<Self>"]
-pub struct ActionRequest {
+pub struct Task {
     pub package_id: PackageId,
     pub action_id: ActionId,
     #[serde(default)]
-    pub severity: ActionSeverity,
+    pub severity: TaskSeverity,
     #[ts(optional)]
     pub reason: Option<String>,
     #[ts(optional)]
-    pub when: Option<ActionRequestTrigger>,
+    pub when: Option<TaskTrigger>,
     #[ts(optional)]
-    pub input: Option<ActionRequestInput>,
+    pub input: Option<TaskInput>,
 }
 
-#[derive(Clone, Debug, Deserialize, Serialize, TS)]
+#[derive(Clone, Debug, Deserialize, Serialize, TS, PartialEq, Eq, PartialOrd, Ord)]
 #[serde(rename_all = "kebab-case")]
 #[ts(export)]
-pub enum ActionSeverity {
-    Critical,
+pub enum TaskSeverity {
+    Optional,
     Important,
+    Critical,
 }
-impl Default for ActionSeverity {
+impl Default for TaskSeverity {
     fn default() -> Self {
-        ActionSeverity::Important
+        TaskSeverity::Important
     }
 }
 
 #[derive(Clone, Debug, Deserialize, Serialize, TS)]
 #[serde(rename_all = "camelCase")]
 #[ts(export)]
-pub struct ActionRequestTrigger {
+pub struct TaskTrigger {
     #[serde(default)]
     pub once: bool,
-    pub condition: ActionRequestCondition,
+    pub condition: TaskCondition,
 }
 
 #[derive(Clone, Debug, Deserialize, Serialize, TS)]
 #[serde(rename_all = "kebab-case")]
 #[ts(export)]
-pub enum ActionRequestCondition {
+pub enum TaskCondition {
     InputNotMatches,
 }
 
 #[derive(Clone, Debug, Deserialize, Serialize, TS)]
 #[serde(rename_all = "kebab-case")]
 #[serde(tag = "kind")]
-pub enum ActionRequestInput {
+pub enum TaskInput {
     Partial {
         #[ts(type = "Record<string, unknown>")]
         value: Value,
     },
 }
-impl ActionRequestInput {
+impl TaskInput {
     pub fn matches(&self, input: Option<&Value>) -> bool {
         match self {
             Self::Partial { value } => match input {

core/startos/src/db/model/public.rs

@@ -40,7 +40,7 @@ pub struct Public {
     pub ui: Value,
 }
 impl Public {
-    pub fn init(account: &AccountInfo) -> Result<Self, Error> {
+    pub fn init(account: &AccountInfo, kiosk: Option<bool>) -> Result<Self, Error> {
         Ok(Self {
             server_info: ServerInfo {
                 arch: get_arch(),
@@ -48,44 +48,50 @@ impl Public {
                 id: account.server_id.clone(),
                 version: Current::default().semver(),
                 hostname: account.hostname.no_dot_host_name(),
-                host: Host {
-                    bindings: [(
-                        80,
-                        BindInfo {
-                            enabled: false,
-                            options: BindOptions {
-                                preferred_external_port: 80,
-                                add_ssl: Some(AddSslOptions {
-                                    preferred_external_port: 443,
-                                    alpn: Some(AlpnInfo::Specified(vec![
-                                        MaybeUtf8String("http/1.1".into()),
-                                        MaybeUtf8String("h2".into()),
-                                    ])),
-                                }),
-                                secure: None,
-                            },
-                            net: NetInfo {
-                                assigned_port: None,
-                                assigned_ssl_port: Some(443),
-                                public: false,
-                            },
-                        },
-                    )]
-                    .into_iter()
-                    .collect(),
-                    onions: account
-                        .tor_keys
-                        .iter()
-                        .map(|k| k.public().get_onion_address())
-                        .collect(),
-                    domains: BTreeMap::new(),
-                    hostname_info: BTreeMap::new(),
-                },
                 last_backup: None,
                 package_version_compat: Current::default().compat().clone(),
                 post_init_migration_todos: BTreeSet::new(),
-                network_interfaces: BTreeMap::new(),
-                acme: BTreeMap::new(),
+                network: NetworkInfo {
+                    host: Host {
+                        bindings: [(
+                            80,
+                            BindInfo {
+                                enabled: false,
+                                options: BindOptions {
+                                    preferred_external_port: 80,
+                                    add_ssl: Some(AddSslOptions {
+                                        preferred_external_port: 443,
+                                        alpn: Some(AlpnInfo::Specified(vec![
+                                            MaybeUtf8String("http/1.1".into()),
+                                            MaybeUtf8String("h2".into()),
+                                        ])),
+                                    }),
+                                    secure: None,
+                                },
+                                net: NetInfo {
+                                    assigned_port: None,
+                                    assigned_ssl_port: Some(443),
+                                    public: false,
+                                },
+                            },
+                        )]
+                        .into_iter()
+                        .collect(),
+                        onions: account
+                            .tor_keys
+                            .iter()
+                            .map(|k| k.public().get_onion_address())
+                            .collect(),
+                        domains: BTreeMap::new(),
+                        hostname_info: BTreeMap::new(),
+                    },
+                    wifi: WifiInfo {
+                        enabled: true,
+                        ..Default::default()
+                    },
+                    network_interfaces: BTreeMap::new(),
+                    acme: BTreeMap::new(),
+                },
                 status_info: ServerStatus {
                     backup_progress: None,
                     updated: false,
@@ -93,7 +99,6 @@ impl Public {
                     shutting_down: false,
                     restarting: false,
                 },
-                wifi: WifiInfo::default(),
                 unread_notification_count: 0,
                 password_hash: account.password.clone(),
                 pubkey: ssh_key::PublicKey::from(&account.ssh_key)
@@ -112,6 +117,7 @@ impl Public {
                 smtp: None,
                 ram: 0,
                 devices: Vec::new(),
+                kiosk,
             },
             package_data: AllPackageData::default(),
             ui: serde_json::from_str(include_str!(concat!(
@@ -145,7 +151,6 @@ pub struct ServerInfo {
     pub id: String,
     #[ts(type = "string")]
     pub hostname: InternedString,
-    pub host: Host,
     #[ts(type = "string")]
     pub version: Version,
     #[ts(type = "string")]
@@ -154,14 +159,9 @@ pub struct ServerInfo {
     pub post_init_migration_todos: BTreeSet<Version>,
     #[ts(type = "string | null")]
     pub last_backup: Option<DateTime<Utc>>,
-    #[ts(as = "BTreeMap::<String, NetworkInterfaceInfo>")]
-    #[serde(default)]
-    pub network_interfaces: BTreeMap<InternedString, NetworkInterfaceInfo>,
-    #[serde(default)]
-    pub acme: BTreeMap<AcmeProvider, AcmeSettings>,
+    pub network: NetworkInfo,
     #[serde(default)]
     pub status_info: ServerStatus,
-    pub wifi: WifiInfo,
     #[ts(type = "number")]
     pub unread_notification_count: u64,
     pub password_hash: String,
@@ -176,6 +176,21 @@ pub struct ServerInfo {
     #[ts(type = "number")]
     pub ram: u64,
     pub devices: Vec<LshwDevice>,
+    pub kiosk: Option<bool>,
+}
+
+#[derive(Debug, Default, Deserialize, Serialize, HasModel, TS)]
+#[serde(rename_all = "camelCase")]
+#[model = "Model<Self>"]
+#[ts(export)]
+pub struct NetworkInfo {
+    pub wifi: WifiInfo,
+    pub host: Host,
+    #[ts(as = "BTreeMap::<String, NetworkInterfaceInfo>")]
+    #[serde(default)]
+    pub network_interfaces: BTreeMap<InternedString, NetworkInterfaceInfo>,
+    #[serde(default)]
+    pub acme: BTreeMap<AcmeProvider, AcmeSettings>,
 }
 
 #[derive(Clone, Debug, Default, Deserialize, Serialize, HasModel, TS)]
@@ -183,12 +198,13 @@ pub struct ServerInfo {
 #[model = "Model<Self>"]
 #[ts(export)]
 pub struct NetworkInterfaceInfo {
-    pub public: Option<bool>,
+    pub inbound: Option<bool>,
+    pub outbound: Option<bool>,
     pub ip_info: Option<IpInfo>,
 }
 impl NetworkInterfaceInfo {
-    pub fn public(&self) -> bool {
-        self.public.unwrap_or_else(|| {
+    pub fn inbound(&self) -> bool {
+        self.inbound.unwrap_or_else(|| {
             !self.ip_info.as_ref().map_or(true, |ip_info| {
                 let ip4s = ip_info
                     .subnets
@@ -220,10 +236,13 @@ impl NetworkInterfaceInfo {
     }
 }
 
-#[derive(Clone, Debug, Default, PartialEq, Eq, Deserialize, Serialize, TS)]
+#[derive(Clone, Debug, Default, PartialEq, Eq, Deserialize, Serialize, TS, HasModel)]
 #[ts(export)]
 #[serde(rename_all = "camelCase")]
+#[model = "Model<Self>"]
 pub struct IpInfo {
+    #[ts(type = "string")]
+    pub name: InternedString,
     pub scope_id: u32,
     pub device_type: Option<NetworkInterfaceType>,
     #[ts(type = "string[]")]
@@ -276,6 +295,7 @@ pub struct ServerStatus {
 #[model = "Model<Self>"]
 #[ts(export)]
 pub struct WifiInfo {
+    pub enabled: bool,
     pub interface: Option<String>,
     pub ssids: BTreeSet<String>,
     pub selected: Option<String>,

core/startos/src/db/prelude.rs

@@ -1,4 +1,4 @@
-use std::collections::BTreeMap;
+use std::collections::{BTreeMap, BTreeSet};
 use std::marker::PhantomData;
 use std::str::FromStr;
 
@@ -267,7 +267,7 @@ where
     T::Key: FromStr + Ord + Clone,
     Error: From<<T::Key as FromStr>::Err>,
 {
-    pub fn keys(&self) -> Result<Vec<T::Key>, Error> {
+    pub fn keys(&self) -> Result<BTreeSet<T::Key>, Error> {
         use serde::de::Error;
         match &self.value {
             Value::Object(o) => o

core/startos/src/diagnostic.rs

@@ -7,10 +7,12 @@ use rpc_toolkit::{
 };
 
 use crate::context::{CliContext, DiagnosticContext, RpcContext};
+use crate::disk::repair;
 use crate::init::SYSTEM_REBUILD_PATH;
+use crate::prelude::*;
 use crate::shutdown::Shutdown;
 use crate::util::io::delete_file;
-use crate::{Error, DATA_DIR};
+use crate::DATA_DIR;
 
 pub fn diagnostic<C: Context>() -> ParentHandler<C> {
     ParentHandler::new()
@@ -74,7 +76,8 @@ pub fn restart(ctx: DiagnosticContext) -> Result<(), Error> {
                 .map(|guid| (guid, Path::new(DATA_DIR).to_owned())),
             restart: true,
         })
-        .expect("receiver dropped");
+        .map_err(|_| eyre!("receiver dropped"))
+        .log_err();
     Ok(())
 }
 pub async fn rebuild(ctx: DiagnosticContext) -> Result<(), Error> {
@@ -93,6 +96,15 @@ pub fn disk<C: Context>() -> ParentHandler<C> {
             .no_display()
             .with_about("Remove disk from filesystem"),
         )
+        .subcommand("repair", from_fn_async(|_: C| repair()).no_cli())
+        .subcommand(
+            "repair",
+            CallRemoteHandler::<CliContext, _, _>::new(
+                from_fn_async(|_: RpcContext| repair())
+                    .no_display()
+                    .with_about("Repair disk in the event of corruption"),
+            ),
+        )
 }
 
 pub async fn forget_disk<C: Context>(_: C) -> Result<(), Error> {

core/startos/src/disk/mod.rs

@@ -48,9 +48,7 @@ pub fn disk<C: Context>() -> ParentHandler<C> {
             "list",
             from_fn_async(list)
                 .with_display_serializable()
-                .with_custom_display_fn(|handle, result| {
-                    Ok(display_disk_info(handle.params, result))
-                })
+                .with_custom_display_fn(|handle, result| display_disk_info(handle.params, result))
                 .with_about("List disk info")
                 .with_call_remote::<CliContext>(),
         )
@@ -65,7 +63,7 @@ pub fn disk<C: Context>() -> ParentHandler<C> {
         )
 }
 
-fn display_disk_info(params: WithIoFormat<Empty>, args: Vec<DiskInfo>) {
+fn display_disk_info(params: WithIoFormat<Empty>, args: Vec<DiskInfo>) -> Result<(), Error> {
     use prettytable::*;
 
     if let Some(format) = params.format {
@@ -124,7 +122,8 @@ fn display_disk_info(params: WithIoFormat<Empty>, args: Vec<DiskInfo>) {
             table.add_row(row);
         }
     }
-    table.print_tty(false).unwrap();
+    table.print_tty(false)?;
+    Ok(())
 }
 
 // #[command(display(display_disk_info))]

core/startos/src/disk/mount/backup.rs

@@ -29,12 +29,11 @@ pub struct BackupMountGuard<G: GenericMountGuard> {
 }
 impl<G: GenericMountGuard> BackupMountGuard<G> {
     #[instrument(skip_all)]
-    pub async fn mount(
-        backup_disk_mount_guard: G,
+    pub async fn load_metadata(
+        backup_disk_path: &Path,
         server_id: &str,
         password: &str,
-    ) -> Result<Self, Error> {
-        let backup_disk_path = backup_disk_mount_guard.path();
+    ) -> Result<(StartOsRecoveryInfo, String), Error> {
         let backup_dir = backup_disk_path.join("StartOSBackups").join(server_id);
         let unencrypted_metadata_path = backup_dir.join("unencrypted-metadata.json");
         let crypt_path = backup_dir.join("crypt");
@@ -79,7 +78,6 @@ impl<G: GenericMountGuard> BackupMountGuard<G> {
                 &rand::random::<[u8; 32]>()[..],
             )
         };
-
         if unencrypted_metadata.password_hash.is_none() {
             unencrypted_metadata.password_hash = Some(
                 argon2::hash_encoded(
@@ -96,6 +94,20 @@ impl<G: GenericMountGuard> BackupMountGuard<G> {
                 &encrypt_slice(&enc_key, password),
             ));
         }
+        Ok((unencrypted_metadata, enc_key))
+    }
+    #[instrument(skip_all)]
+    pub async fn mount(
+        backup_disk_mount_guard: G,
+        server_id: &str,
+        password: &str,
+    ) -> Result<Self, Error> {
+        let backup_disk_path = backup_disk_mount_guard.path();
+        let (unencrypted_metadata, enc_key) =
+            Self::load_metadata(backup_disk_path, server_id, password).await?;
+        let backup_dir = backup_disk_path.join("StartOSBackups").join(server_id);
+        let unencrypted_metadata_path = backup_dir.join("unencrypted-metadata.json");
+        let crypt_path = backup_dir.join("crypt");
 
         if tokio::fs::metadata(&crypt_path).await.is_err() {
             tokio::fs::create_dir_all(&crypt_path).await.with_ctx(|_| {

core/startos/src/disk/mount/filesystem/bind.rs

@@ -3,28 +3,73 @@ use std::path::Path;
 
 use digest::generic_array::GenericArray;
 use digest::{Digest, OutputSizeUser};
+use serde::{Deserialize, Serialize};
 use sha2::Sha256;
+use ts_rs::TS;
 
 use super::FileSystem;
 use crate::prelude::*;
+use crate::util::io::create_file;
 
-pub struct Bind<SrcDir: AsRef<Path>> {
-    src_dir: SrcDir,
+#[derive(Debug, Clone, Serialize, Deserialize, TS)]
+#[ts(export)]
+#[serde(rename_all = "kebab-case")]
+pub enum FileType {
+    File,
+    Directory,
+    Infer,
 }
-impl<SrcDir: AsRef<Path>> Bind<SrcDir> {
-    pub fn new(src_dir: SrcDir) -> Self {
-        Self { src_dir }
+
+pub struct Bind<Src: AsRef<Path>> {
+    src: Src,
+    filetype: FileType,
+}
+impl<Src: AsRef<Path>> Bind<Src> {
+    pub fn new(src: Src) -> Self {
+        Self {
+            src,
+            filetype: FileType::Directory,
+        }
+    }
+    pub fn with_type(mut self, filetype: FileType) -> Self {
+        self.filetype = filetype;
+        self
     }
 }
-impl<SrcDir: AsRef<Path> + Send + Sync> FileSystem for Bind<SrcDir> {
+impl<Src: AsRef<Path> + Send + Sync> FileSystem for Bind<Src> {
     async fn source(&self) -> Result<Option<impl AsRef<Path>>, Error> {
-        Ok(Some(&self.src_dir))
+        Ok(Some(&self.src))
     }
     fn extra_args(&self) -> impl IntoIterator<Item = impl AsRef<std::ffi::OsStr>> {
         ["--bind"]
     }
-    async fn pre_mount(&self) -> Result<(), Error> {
-        tokio::fs::create_dir_all(self.src_dir.as_ref()).await?;
+    async fn pre_mount(&self, mountpoint: &Path) -> Result<(), Error> {
+        let from_meta = tokio::fs::metadata(&self.src).await.ok();
+        let to_meta = tokio::fs::metadata(&mountpoint).await.ok();
+        if matches!(self.filetype, FileType::File)
+            || (matches!(self.filetype, FileType::Infer)
+                && from_meta.as_ref().map_or(false, |m| m.is_file()))
+        {
+            if to_meta.as_ref().map_or(false, |m| m.is_dir()) {
+                tokio::fs::remove_dir(mountpoint).await?;
+            }
+            if from_meta.is_none() {
+                create_file(self.src.as_ref()).await?.sync_all().await?;
+            }
+            if to_meta.is_none() {
+                create_file(mountpoint).await?.sync_all().await?;
+            }
+        } else {
+            if to_meta.as_ref().map_or(false, |m| m.is_file()) {
+                tokio::fs::remove_file(mountpoint).await?;
+            }
+            if from_meta.is_none() {
+                tokio::fs::create_dir_all(self.src.as_ref()).await?;
+            }
+            if to_meta.is_none() {
+                tokio::fs::create_dir_all(mountpoint).await?;
+            }
+        }
         Ok(())
     }
     async fn source_hash(
@@ -33,12 +78,12 @@ impl<SrcDir: AsRef<Path> + Send + Sync> FileSystem for Bind<SrcDir> {
         let mut sha = Sha256::new();
         sha.update("Bind");
         sha.update(
-            tokio::fs::canonicalize(self.src_dir.as_ref())
+            tokio::fs::canonicalize(self.src.as_ref())
                 .await
                 .with_ctx(|_| {
                     (
                         crate::ErrorKind::Filesystem,
-                        self.src_dir.as_ref().display().to_string(),
+                        self.src.as_ref().display().to_string(),
                     )
                 })?
                 .as_os_str()

core/startos/src/disk/mount/filesystem/ecryptfs.rs

@@ -49,8 +49,7 @@ impl<EncryptedDir: AsRef<Path> + Send + Sync, Key: AsRef<str> + Send + Sync> Fil
         mountpoint: P,
         mount_type: super::MountType,
     ) -> Result<(), Error> {
-        self.pre_mount().await?;
-        tokio::fs::create_dir_all(mountpoint.as_ref()).await?;
+        self.pre_mount(mountpoint.as_ref()).await?;
         Command::new("mount")
             .args(
                 default_mount_command(self, mountpoint, mount_type)

core/startos/src/disk/mount/filesystem/idmapped.rs

@@ -1,5 +1,6 @@
 use std::ffi::OsStr;
 use std::fmt::Display;
+use std::os::unix::fs::MetadataExt;
 use std::path::Path;
 
 use digest::generic_array::GenericArray;
@@ -53,16 +54,38 @@ impl<Fs: FileSystem> FileSystem for IdMapped<Fs> {
     async fn source(&self) -> Result<Option<impl AsRef<Path>>, Error> {
         self.filesystem.source().await
     }
-    async fn pre_mount(&self) -> Result<(), Error> {
-        self.filesystem.pre_mount().await
+    async fn pre_mount(&self, mountpoint: &Path) -> Result<(), Error> {
+        self.filesystem.pre_mount(mountpoint).await?;
+        let info = tokio::fs::metadata(mountpoint).await?;
+        let uid_in_range = self.from_id <= info.uid() && self.from_id + self.range > info.uid();
+        let gid_in_range = self.from_id <= info.gid() && self.from_id + self.range > info.gid();
+        if uid_in_range || gid_in_range {
+            Command::new("chown")
+                .arg(format!(
+                    "{uid}:{gid}",
+                    uid = if uid_in_range {
+                        self.to_id + info.uid() - self.from_id
+                    } else {
+                        info.uid()
+                    },
+                    gid = if gid_in_range {
+                        self.to_id + info.gid() - self.from_id
+                    } else {
+                        info.gid()
+                    },
+                ))
+                .arg(&mountpoint)
+                .invoke(crate::ErrorKind::Filesystem)
+                .await?;
+        }
+        Ok(())
     }
     async fn mount<P: AsRef<Path> + Send>(
         &self,
         mountpoint: P,
         mount_type: MountType,
     ) -> Result<(), Error> {
-        self.pre_mount().await?;
-        tokio::fs::create_dir_all(mountpoint.as_ref()).await?;
+        self.pre_mount(mountpoint.as_ref()).await?;
         Command::new("mount.next")
             .args(
                 default_mount_command(self, mountpoint, mount_type)

core/startos/src/disk/mount/filesystem/mod.rs

@@ -69,8 +69,7 @@ pub(self) async fn default_mount_impl(
     mountpoint: impl AsRef<Path> + Send,
     mount_type: MountType,
 ) -> Result<(), Error> {
-    fs.pre_mount().await?;
-    tokio::fs::create_dir_all(mountpoint.as_ref()).await?;
+    fs.pre_mount(mountpoint.as_ref()).await?;
     Command::from(default_mount_command(fs, mountpoint, mount_type).await?)
         .capture(false)
         .invoke(ErrorKind::Filesystem)
@@ -92,8 +91,11 @@ pub trait FileSystem: Send + Sync {
     fn source(&self) -> impl Future<Output = Result<Option<impl AsRef<Path>>, Error>> + Send {
         async { Ok(None::<&Path>) }
     }
-    fn pre_mount(&self) -> impl Future<Output = Result<(), Error>> + Send {
-        async { Ok(()) }
+    fn pre_mount(&self, mountpoint: &Path) -> impl Future<Output = Result<(), Error>> + Send {
+        async move {
+            tokio::fs::create_dir_all(mountpoint).await?;
+            Ok(())
+        }
     }
     fn mount<P: AsRef<Path> + Send>(
         &self,

core/startos/src/disk/mount/filesystem/overlayfs.rs

@@ -41,9 +41,10 @@ impl<
             Box::new(lazy_format!("workdir={}", self.work.as_ref().display())),
         ]
     }
-    async fn pre_mount(&self) -> Result<(), Error> {
+    async fn pre_mount(&self, mountpoint: &Path) -> Result<(), Error> {
         tokio::fs::create_dir_all(self.upper.as_ref()).await?;
         tokio::fs::create_dir_all(self.work.as_ref()).await?;
+        tokio::fs::create_dir_all(mountpoint).await?;
         Ok(())
     }
     async fn source_hash(

core/startos/src/disk/util.rs

@@ -6,10 +6,9 @@ use color_eyre::eyre::{self, eyre};
 use futures::TryStreamExt;
 use nom::bytes::complete::{tag, take_till1};
 use nom::character::complete::multispace1;
-use nom::character::is_space;
 use nom::combinator::{opt, rest};
 use nom::sequence::{pair, preceded, terminated};
-use nom::IResult;
+use nom::{AsChar, IResult, Parser};
 use regex::Regex;
 use serde::{Deserialize, Serialize};
 use tokio::process::Command;
@@ -450,17 +449,17 @@ fn parse_pvscan_output(pvscan_output: &str) -> BTreeMap<PathBuf, Option<String>>
     fn parse_line(line: &str) -> IResult<&str, (&str, Option<&str>)> {
         let pv_parse = preceded(
             tag("  PV "),
-            terminated(take_till1(|c| is_space(c as u8)), multispace1),
+            terminated(take_till1(|c: char| c.is_space()), multispace1),
         );
         let vg_parse = preceded(
             opt(tag("is in exported ")),
             preceded(
                 tag("VG "),
-                terminated(take_till1(|c| is_space(c as u8)), multispace1),
+                terminated(take_till1(|c: char| c.is_space()), multispace1),
             ),
         );
         let mut parser = terminated(pair(pv_parse, opt(vg_parse)), rest);
-        parser(line)
+        parser.parse(line)
     }
     let lines = pvscan_output.lines();
     let n = lines.clone().count();

core/startos/src/hostname.rs

@@ -1,6 +1,6 @@
 use imbl_value::InternedString;
 use lazy_format::lazy_format;
-use rand::{thread_rng, Rng};
+use rand::{rng, Rng};
 use tokio::process::Command;
 use tracing::instrument;
 
@@ -34,7 +34,7 @@ impl Hostname {
 }
 
 pub fn generate_hostname() -> Hostname {
-    let mut rng = thread_rng();
+    let mut rng = rng();
     let adjective = &ADJECTIVES[rng.gen_range(0..ADJECTIVES.len())];
     let noun = &NOUNS[rng.gen_range(0..NOUNS.len())];
     Hostname(InternedString::from_display(&lazy_format!(

core/startos/src/init.rs

@@ -32,13 +32,13 @@ use crate::net::utils::find_wifi_iface;
 use crate::net::web_server::{UpgradableListener, WebServerAcceptorSetter};
 use crate::prelude::*;
 use crate::progress::{
-    FullProgress, FullProgressTracker, PhaseProgressTrackerHandle, PhasedProgressBar,
+    FullProgress, FullProgressTracker, PhaseProgressTrackerHandle, PhasedProgressBar, ProgressUnits,
 };
 use crate::rpc_continuations::{Guid, RpcContinuation};
 use crate::s9pk::v2::pack::{CONTAINER_DATADIR, CONTAINER_TOOL};
 use crate::ssh::SSH_DIR;
-use crate::system::get_mem_info;
-use crate::util::io::{create_file, IOHook};
+use crate::system::{get_mem_info, sync_kiosk};
+use crate::util::io::{create_file, open_file, IOHook};
 use crate::util::lshw::lshw;
 use crate::util::net::WebSocketExt;
 use crate::util::{cpupower, Invoke};
@@ -259,12 +259,14 @@ pub async fn run_script<P: AsRef<Path>>(path: P, mut progress: PhaseProgressTrac
     if let Err(e) = async {
         let script = tokio::fs::read_to_string(script).await?;
         progress.set_total(script.as_bytes().iter().filter(|b| **b == b'\n').count() as u64);
+        progress.set_units(Some(ProgressUnits::Bytes));
         let mut reader = IOHook::new(Cursor::new(script.as_bytes()));
         reader.post_read(|buf| progress += buf.iter().filter(|b| **b == b'\n').count() as u64);
         Command::new("/bin/bash")
             .input(Some(&mut reader))
             .invoke(ErrorKind::Unknown)
             .await?;
+        // TODO: inherit?
 
         Ok::<_, Error>(())
     }
@@ -397,6 +399,11 @@ pub async fn init(
         .invoke(crate::ErrorKind::Journald)
         .await?;
     mount_logs.complete();
+    tokio::io::copy(
+        &mut open_file("/run/startos/init.log").await?,
+        &mut tokio::io::stderr(),
+    )
+    .await?;
     tracing::info!("Mounted Logs");
 
     load_ca_cert.start();
@@ -415,7 +422,11 @@ pub async fn init(
     let wifi_interface = find_wifi_iface().await?;
     let wifi = db
         .mutate(|db| {
-            let wifi = db.as_public_mut().as_server_info_mut().as_wifi_mut();
+            let wifi = db
+                .as_public_mut()
+                .as_server_info_mut()
+                .as_network_mut()
+                .as_wifi_mut();
             wifi.as_interface_mut().ser(&wifi_interface)?;
             wifi.de()
         })
@@ -506,6 +517,7 @@ pub async fn init(
     enable_zram.complete();
 
     update_server_info.start();
+    sync_kiosk(server_info.as_kiosk().de()?).await?;
     let ram = get_mem_info().await?.total.0 as u64 * 1024 * 1024;
     let devices = lshw().await?;
     let status_info = ServerStatus {
@@ -615,7 +627,8 @@ pub async fn init_progress(ctx: InitContext) -> Result<InitProgressRes, Error> {
                             while let Some(progress) = stream.next().await {
                                 ws.send(ws::Message::Text(
                                     serde_json::to_string(&progress)
-                                        .with_kind(ErrorKind::Serialization)?,
+                                        .with_kind(ErrorKind::Serialization)?
+                                        .into(),
                                 ))
                                 .await
                                 .with_kind(ErrorKind::Network)?;

core/startos/src/install/mod.rs

@@ -23,7 +23,7 @@ use tracing::instrument;
 use ts_rs::TS;
 
 use crate::context::{CliContext, RpcContext};
-use crate::db::model::package::{ManifestPreference, PackageState, PackageStateMatchModelRef};
+use crate::db::model::package::{ManifestPreference, PackageStateMatchModelRef};
 use crate::prelude::*;
 use crate::progress::{FullProgress, FullProgressTracker, PhasedProgressBar};
 use crate::registry::context::{RegistryContext, RegistryUrlParams};
@@ -154,13 +154,16 @@ pub async fn install(
         })?
         .s9pk;
 
+    let progress_tracker = FullProgressTracker::new();
+    let download_progress = progress_tracker.add_phase("Downloading".into(), Some(100));
     let download = ctx
         .services
         .install(
             ctx.clone(),
-            || asset.deserialize_s9pk_buffered(ctx.client.clone()),
+            || asset.deserialize_s9pk_buffered(ctx.client.clone(), download_progress),
+            Some(registry),
             None::<Never>,
-            None,
+            Some(progress_tracker),
         )
         .await?;
     tokio::spawn(async move { download.await?.await });
@@ -188,10 +191,15 @@ pub async fn sideload(
     ctx: RpcContext,
     SideloadParams { session }: SideloadParams,
 ) -> Result<SideloadResponse, Error> {
-    let (upload, file) = upload(&ctx, session.clone()).await?;
     let (err_send, mut err_recv) = oneshot::channel::<Error>();
     let progress = Guid::new();
     let progress_tracker = FullProgressTracker::new();
+    let (upload, file) = upload(
+        &ctx,
+        session.clone(),
+        progress_tracker.add_phase("Uploading".into(), Some(100)),
+    )
+    .await?;
     let mut progress_listener = progress_tracker.stream(Some(Duration::from_millis(200)));
     ctx.rpc_continuations
         .add(
@@ -207,7 +215,8 @@ pub async fn sideload(
                                     if let Some(progress) = progress {
                                         ws.send(ws::Message::Text(
                                             serde_json::to_string(&progress)
-                                                .with_kind(ErrorKind::Serialization)?,
+                                                .with_kind(ErrorKind::Serialization)?
+                                                .into(),
                                         ))
                                         .await
                                         .with_kind(ErrorKind::Network)?;
@@ -250,6 +259,7 @@ pub async fn sideload(
                 .install(
                     ctx.clone(),
                     || crate::s9pk::load(file.clone(), || Ok(key.de()?.0), Some(&progress_tracker)),
+                    None,
                     None::<Never>,
                     Some(progress_tracker.clone()),
                 )
@@ -268,6 +278,24 @@ pub async fn sideload(
     Ok(SideloadResponse { upload, progress })
 }
 
+#[derive(Debug, Clone, Deserialize, Serialize, Parser, TS)]
+#[serde(rename_all = "camelCase")]
+#[command(rename_all = "kebab-case")]
+pub struct CancelInstallParams {
+    pub id: PackageId,
+}
+
+#[instrument(skip_all)]
+pub fn cancel_install(
+    ctx: RpcContext,
+    CancelInstallParams { id }: CancelInstallParams,
+) -> Result<(), Error> {
+    if let Some(cancel) = ctx.cancellable_installs.mutate(|c| c.remove(&id)) {
+        cancel.send(()).ok();
+    }
+    Ok(())
+}
+
 #[derive(Deserialize, Serialize, Parser)]
 pub struct QueryPackageParams {
     id: PackageId,
@@ -525,33 +553,18 @@ pub struct UninstallParams {
 pub async fn uninstall(
     ctx: RpcContext,
     UninstallParams { id, soft, force }: UninstallParams,
-) -> Result<PackageId, Error> {
-    ctx.db
-        .mutate(|db| {
-            let entry = db
-                .as_public_mut()
-                .as_package_data_mut()
-                .as_idx_mut(&id)
-                .or_not_found(&id)?;
-            entry.as_state_info_mut().map_mutate(|s| match s {
-                PackageState::Installed(s) => Ok(PackageState::Removing(s)),
-                _ => Err(Error::new(
-                    eyre!("Package {id} is not installed."),
-                    crate::ErrorKind::NotFound,
-                )),
-            })
-        })
-        .await
-        .result?;
-
-    let return_id = id.clone();
+) -> Result<(), Error> {
+    let fut = ctx
+        .services
+        .uninstall(ctx.clone(), id.clone(), soft, force)
+        .await?;
 
     tokio::spawn(async move {
-        if let Err(e) = ctx.services.uninstall(&ctx, &id, soft, force).await {
+        if let Err(e) = fut.await {
             tracing::error!("Error uninstalling service {id}: {e}");
             tracing::debug!("{e:?}");
         }
     });
 
-    Ok(return_id)
+    Ok(())
 }

core/startos/src/lib.rs

@@ -87,9 +87,9 @@ use crate::context::{
     CliContext, DiagnosticContext, InitContext, InstallContext, RpcContext, SetupContext,
 };
 use crate::disk::fsck::RequiresReboot;
-use crate::net::net;
 use crate::registry::context::{RegistryContext, RegistryUrlParams};
-use crate::util::serde::HandlerExtSerde;
+use crate::system::kiosk;
+use crate::util::serde::{display_serializable, HandlerExtSerde, WithIoFormat};
 
 #[derive(Deserialize, Serialize, Parser, TS)]
 #[serde(rename_all = "camelCase")]
@@ -118,7 +118,7 @@ impl std::fmt::Display for ApiState {
 }
 
 pub fn main_api<C: Context>() -> ParentHandler<C> {
-    let api = ParentHandler::new()
+    let mut api = ParentHandler::new()
         .subcommand(
             "git-info",
             from_fn(|_: C| version::git_info()).with_about("Display the githash of StartOS CLI"),
@@ -198,12 +198,9 @@ pub fn main_api<C: Context>() -> ParentHandler<C> {
             "util",
             util::rpc::util::<C>().with_about("Command for calculating the blake3 hash of a file"),
         );
-    #[cfg(feature = "dev")]
-    let api = api.subcommand(
-        "lxc",
-        lxc::dev::lxc::<C>()
-            .with_about("Commands related to lxc containers i.e. create, list, remove, connect"),
-    );
+    if &*PLATFORM != "raspberrypi" {
+        api = api.subcommand("kiosk", kiosk::<C>());
+    }
     api
 }
 
@@ -214,7 +211,7 @@ pub fn server<C: Context>() -> ParentHandler<C> {
             from_fn_async(system::time)
                 .with_display_serializable()
                 .with_custom_display_fn(|handle, result| {
-                    Ok(system::display_time(handle.params, result))
+                    system::display_time(handle.params, result)
                 })
                 .with_about("Display current time and server uptime")
                 .with_call_remote::<CliContext>()
@@ -242,10 +239,18 @@ pub fn server<C: Context>() -> ParentHandler<C> {
         )
         .subcommand(
             "metrics",
-            from_fn_async(system::metrics)
-                .with_display_serializable()
-                .with_about("Display information about the server i.e. temperature, RAM, CPU, and disk usage")
-                .with_call_remote::<CliContext>()
+            ParentHandler::<C, WithIoFormat<Empty>>::new()
+                .root_handler(
+                    from_fn_async(system::metrics)
+                        .with_display_serializable()
+                        .with_about("Display information about the server i.e. temperature, RAM, CPU, and disk usage")
+                        .with_call_remote::<CliContext>()
+                )
+                .subcommand(
+                    "follow", 
+                    from_fn_async(system::metrics_follow)
+                        .no_cli()
+                )
         )
         .subcommand(
             "shutdown",
@@ -341,6 +346,13 @@ pub fn package<C: Context>() -> ParentHandler<C> {
                 .no_display()
                 .with_about("Install a package from a marketplace or via sideloading"),
         )
+        .subcommand(
+            "cancel-install",
+            from_fn(install::cancel_install)
+                .no_display()
+                .with_about("Cancel an install of a package")
+                .with_call_remote::<CliContext>(),
+        )
         .subcommand(
             "uninstall",
             from_fn_async(install::uninstall)
@@ -395,6 +407,46 @@ pub fn package<C: Context>() -> ParentHandler<C> {
                 .with_about("Rebuild service container")
                 .with_call_remote::<CliContext>(),
         )
+        .subcommand(
+            "stats",
+            from_fn_async(lxc::stats)
+                .with_display_serializable()
+                .with_custom_display_fn(|args, res| {
+                    if let Some(format) = args.params.format {
+                        return display_serializable(format, res);
+                    }
+
+                    use prettytable::*;
+                    let mut table = table!([
+                        "Name",
+                        "Container ID",
+                        "Memory Usage",
+                        "Memory Limit",
+                        "Memory %"
+                    ]);
+                    for (id, stats) in res {
+                        if let Some(stats) = stats {
+                            table.add_row(row![
+                                &*id,
+                                &*stats.container_id,
+                                stats.memory_usage,
+                                stats.memory_limit,
+                                format!(
+                                    "{:.2}",
+                                    stats.memory_usage.0 as f64 / stats.memory_limit.0 as f64
+                                        * 100.0
+                                )
+                            ]);
+                        } else {
+                            table.add_row(row![&*id, "N/A", "0 MiB", "0 MiB", "0"]);
+                        }
+                    }
+                    table.print_tty(false)?;
+                    Ok(())
+                })
+                .with_about("List information related to the lxc containers i.e. CPU, Memory, Disk")
+                .with_call_remote::<CliContext>(),
+        )
         .subcommand("logs", logs::package_logs())
         .subcommand(
             "logs",

core/startos/src/logs.rs

@@ -75,7 +75,9 @@ async fn ws_handler(
     if let Some(first_entry) = first_entry {
         stream
             .send(ws::Message::Text(
-                serde_json::to_string(&first_entry).with_kind(ErrorKind::Serialization)?,
+                serde_json::to_string(&first_entry)
+                    .with_kind(ErrorKind::Serialization)?
+                    .into(),
             ))
             .await
             .with_kind(ErrorKind::Network)?;
@@ -88,7 +90,9 @@ async fn ws_handler(
                     let (_, log_entry) = entry.log_entry()?;
                     stream
                         .send(ws::Message::Text(
-                            serde_json::to_string(&log_entry).with_kind(ErrorKind::Serialization)?,
+                            serde_json::to_string(&log_entry)
+                                .with_kind(ErrorKind::Serialization)?
+                                .into(),
                         ))
                         .await
                         .with_kind(ErrorKind::Network)?;

core/startos/src/lxc/dev.rs

@@ -1,174 +0,0 @@
-use std::ops::Deref;
-
-use clap::Parser;
-use rpc_toolkit::{
-    from_fn_async, CallRemoteHandler, Context, Empty, HandlerArgs, HandlerExt, HandlerFor,
-    ParentHandler,
-};
-use serde::{Deserialize, Serialize};
-use ts_rs::TS;
-
-use crate::context::{CliContext, RpcContext};
-use crate::lxc::{ContainerId, LxcConfig};
-use crate::prelude::*;
-use crate::rpc_continuations::Guid;
-use crate::service::ServiceStats;
-
-pub fn lxc<C: Context>() -> ParentHandler<C> {
-    ParentHandler::new()
-        .subcommand(
-            "create",
-            from_fn_async(create)
-                .with_about("Create lxc container")
-                .with_call_remote::<CliContext>(),
-        )
-        .subcommand(
-            "list",
-            from_fn_async(list)
-                .with_custom_display_fn(|_, res| {
-                    use prettytable::*;
-                    let mut table = table!([bc => "GUID"]);
-                    for guid in res {
-                        table.add_row(row![&*guid]);
-                    }
-                    table.printstd();
-                    Ok(())
-                })
-                .with_about("List lxc containers")
-                .with_call_remote::<CliContext>(),
-        )
-        .subcommand(
-            "stats",
-            from_fn_async(stats)
-                .with_custom_display_fn(|_, res| {
-                    use prettytable::*;
-                    let mut table = table!([
-                        "Container ID",
-                        "Name",
-                        "Memory Usage",
-                        "Memory Limit",
-                        "Memory %"
-                    ]);
-                    for ServiceStats {
-                        container_id,
-                        package_id,
-                        memory_usage,
-                        memory_limit,
-                    } in res
-                    {
-                        table.add_row(row![
-                            &*container_id,
-                            &*package_id,
-                            memory_usage,
-                            memory_limit,
-                            format!(
-                                "{:.2}",
-                                memory_usage.0 as f64 / memory_limit.0 as f64 * 100.0
-                            )
-                        ]);
-                    }
-                    table.printstd();
-                    Ok(())
-                })
-                .with_about("List information related to the lxc containers i.e. CPU, Memory, Disk")
-                .with_call_remote::<CliContext>(),
-        )
-        .subcommand(
-            "remove",
-            from_fn_async(remove)
-                .no_display()
-                .with_about("Remove lxc container")
-                .with_call_remote::<CliContext>(),
-        )
-        .subcommand("connect", from_fn_async(connect_rpc).no_cli())
-        .subcommand(
-            "connect",
-            from_fn_async(connect_rpc_cli)
-                .no_display()
-                .with_about("Connect to a lxc container"),
-        )
-}
-
-pub async fn create(ctx: RpcContext) -> Result<ContainerId, Error> {
-    let container = ctx.lxc_manager.create(None, LxcConfig::default()).await?;
-    let guid = container.guid.deref().clone();
-    ctx.dev.lxc.lock().await.insert(guid.clone(), container);
-    Ok(guid)
-}
-
-pub async fn list(ctx: RpcContext) -> Result<Vec<ContainerId>, Error> {
-    Ok(ctx.dev.lxc.lock().await.keys().cloned().collect())
-}
-
-pub async fn stats(ctx: RpcContext) -> Result<Vec<ServiceStats>, Error> {
-    let ids = ctx.db.peek().await.as_public().as_package_data().keys()?;
-    let guids: Vec<_> = ctx.dev.lxc.lock().await.keys().cloned().collect();
-
-    let mut stats = Vec::with_capacity(guids.len());
-    for id in ids {
-        let service: tokio::sync::OwnedRwLockReadGuard<Option<crate::service::ServiceRef>> =
-            ctx.services.get(&id).await;
-
-        let service_ref = service.as_ref().or_not_found(&id)?;
-
-        stats.push(service_ref.stats().await?);
-    }
-    Ok(stats)
-}
-
-#[derive(Deserialize, Serialize, Parser, TS)]
-pub struct RemoveParams {
-    #[ts(type = "string")]
-    pub guid: ContainerId,
-}
-
-pub async fn remove(ctx: RpcContext, RemoveParams { guid }: RemoveParams) -> Result<(), Error> {
-    if let Some(container) = ctx.dev.lxc.lock().await.remove(&guid) {
-        container.exit().await?;
-    }
-    Ok(())
-}
-
-#[derive(Deserialize, Serialize, Parser, TS)]
-pub struct ConnectParams {
-    #[ts(type = "string")]
-    pub guid: ContainerId,
-}
-
-pub async fn connect_rpc(
-    ctx: RpcContext,
-    ConnectParams { guid }: ConnectParams,
-) -> Result<Guid, Error> {
-    super::connect(
-        &ctx,
-        ctx.dev.lxc.lock().await.get(&guid).ok_or_else(|| {
-            Error::new(eyre!("No container with guid: {guid}"), ErrorKind::NotFound)
-        })?,
-    )
-    .await
-}
-
-pub async fn connect_rpc_cli(
-    HandlerArgs {
-        context,
-        parent_method,
-        method,
-        params,
-        inherited_params,
-        raw_params,
-    }: HandlerArgs<CliContext, ConnectParams>,
-) -> Result<(), Error> {
-    let ctx = context.clone();
-    let guid = CallRemoteHandler::<CliContext, _, _>::new(from_fn_async(connect_rpc))
-        .handle_async(HandlerArgs {
-            context,
-            parent_method,
-            method,
-            params: rpc_toolkit::util::Flat(params, Empty {}),
-            inherited_params,
-            raw_params,
-        })
-        .await?;
-
-    super::connect_cli(&ctx, guid).await
-}

core/startos/src/lxc/mod.rs

@@ -1,4 +1,4 @@
-use std::collections::BTreeSet;
+use std::collections::{BTreeMap, BTreeSet};
 use std::net::Ipv4Addr;
 use std::path::Path;
 use std::sync::{Arc, Weak};
@@ -7,7 +7,7 @@ use std::time::Duration;
 use clap::builder::ValueParserFactory;
 use futures::{AsyncWriteExt, StreamExt};
 use imbl_value::{InOMap, InternedString};
-use models::{FromStrParser, InvalidId};
+use models::{FromStrParser, InvalidId, PackageId};
 use rpc_toolkit::yajrc::RpcError;
 use rpc_toolkit::{GenericRpcMethod, RpcRequest, RpcResponse};
 use rustyline_async::{ReadlineEvent, SharedWriter};
@@ -28,13 +28,11 @@ use crate::disk::mount::guard::{GenericMountGuard, MountGuard, TmpMountGuard};
 use crate::disk::mount::util::unmount;
 use crate::prelude::*;
 use crate::rpc_continuations::{Guid, RpcContinuation};
+use crate::service::ServiceStats;
 use crate::util::io::open_file;
 use crate::util::rpc_client::UnixRpcClient;
 use crate::util::{new_guid, Invoke};
 
-// #[cfg(feature = "dev")]
-pub mod dev;
-
 const LXC_CONTAINER_DIR: &str = "/var/lib/lxc";
 const RPC_DIR: &str = "media/startos/rpc"; // must not be absolute path
 pub const CONTAINER_RPC_SERVER_SOCKET: &str = "service.sock"; // must not be absolute path
@@ -91,8 +89,13 @@ impl LxcManager {
         log_mount: Option<&Path>,
         config: LxcConfig,
     ) -> Result<LxcContainer, Error> {
-        let container = LxcContainer::new(self, log_mount, config).await?;
         let mut guard = self.containers.lock().await;
+        let container = tokio::time::timeout(
+            Duration::from_secs(30),
+            LxcContainer::new(self, log_mount, config),
+        )
+        .await
+        .with_kind(ErrorKind::Timeout)??;
         *guard = std::mem::take(&mut *guard)
             .into_iter()
             .filter(|g| g.strong_count() > 0)
@@ -179,12 +182,6 @@ impl LxcContainer {
         .await?;
         // TODO: append config
         let rootfs_dir = container_dir.join("rootfs");
-        tokio::fs::create_dir_all(&rootfs_dir).await?;
-        Command::new("chown")
-            .arg("100000:100000")
-            .arg(&rootfs_dir)
-            .invoke(ErrorKind::Filesystem)
-            .await?;
         let rootfs = OverlayGuard::mount(
             TmpMountGuard::mount(
                 &IdMapped::new(
@@ -231,6 +228,17 @@ impl LxcContainer {
                 .arg(&log_mount_point)
                 .invoke(crate::ErrorKind::Filesystem)
                 .await?;
+            match Command::new("chattr")
+                .arg("-R")
+                .arg("+C")
+                .arg(&log_mount_point)
+                .invoke(ErrorKind::Filesystem)
+                .await
+            {
+                Ok(_) => Ok(()),
+                Err(e) if e.source.to_string().contains("Operation not supported") => Ok(()),
+                Err(e) => Err(e),
+            }?;
             Some(log_mount)
         } else {
             None
@@ -435,7 +443,8 @@ pub async fn connect(ctx: &RpcContext, container: &LxcContainer) -> Result<Guid,
                                     .await;
                                     ws.send(Message::Text(
                                         serde_json::to_string(&RpcResponse { id, result })
-                                            .with_kind(ErrorKind::Serialization)?,
+                                            .with_kind(ErrorKind::Serialization)?
+                                            .into(),
                                     ))
                                     .await
                                     .with_kind(ErrorKind::Network)?;
@@ -509,7 +518,7 @@ pub async fn connect_cli(ctx: &CliContext, guid: Guid) -> Result<(), Error> {
                 if let ReadlineEvent::Line(line) = line {
                     input.add_history_entry(line.clone());
                     if serde_json::from_str::<RpcRequest>(&line).is_ok() {
-                        ws.send(Message::Text(line))
+                        ws.send(Message::Text(line.into()))
                             .await
                             .with_kind(ErrorKind::Network)?;
                     } else {
@@ -537,7 +546,7 @@ pub async fn connect_cli(ctx: &CliContext, guid: Guid) -> Result<(), Error> {
                                         method: GenericRpcMethod::new(method.into()),
                                         params: Value::Object(params),
                                     }) {
-                                        Ok(a) => a,
+                                        Ok(a) => a.into(),
                                         Err(e) => {
                                             tracing::error!("Error Serializing Request: {e}");
                                             tracing::debug!("{e:?}");
@@ -569,3 +578,21 @@ pub async fn connect_cli(ctx: &CliContext, guid: Guid) -> Result<(), Error> {
 
     Ok(())
 }
+
+pub async fn stats(ctx: RpcContext) -> Result<BTreeMap<PackageId, Option<ServiceStats>>, Error> {
+    let ids = ctx.db.peek().await.as_public().as_package_data().keys()?;
+
+    let mut stats = BTreeMap::new();
+    for id in ids {
+        let service: tokio::sync::OwnedRwLockReadGuard<Option<crate::service::ServiceRef>> =
+            ctx.services.get(&id).await;
+
+        let Some(service_ref) = service.as_ref() else {
+            stats.insert(id, None);
+            continue;
+        };
+
+        stats.insert(id, Some(service_ref.stats().await?));
+    }
+    Ok(stats)
+}

core/startos/src/net/acme.rs

@@ -257,7 +257,7 @@ pub async fn init(
     ctx.db
         .mutate(|db| {
             db.as_public_mut()
-                .as_server_info_mut()
+                .as_server_info_mut().as_network_mut()
                 .as_acme_mut()
                 .insert(&provider, &AcmeSettings { contact })
         })
@@ -279,7 +279,7 @@ pub async fn remove(
     ctx.db
         .mutate(|db| {
             db.as_public_mut()
-                .as_server_info_mut()
+                .as_server_info_mut().as_network_mut()
                 .as_acme_mut()
                 .remove(&provider)
         })

core/startos/src/net/dns.rs

@@ -5,7 +5,6 @@ use std::sync::{Arc, Weak};
 use std::time::Duration;
 
 use color_eyre::eyre::eyre;
-use futures::TryFutureExt;
 use helpers::NonDetachingJoinHandle;
 use models::PackageId;
 use tokio::net::{TcpListener, UdpSocket};

core/startos/src/net/forward.rs

@@ -1,5 +1,5 @@
 use std::collections::{BTreeMap, BTreeSet};
-use std::net::SocketAddr;
+use std::net::{IpAddr, Ipv4Addr, SocketAddr};
 use std::sync::{Arc, Weak};
 
 use futures::channel::oneshot;
@@ -52,10 +52,13 @@ struct ForwardState {
     current: BTreeMap<u16, BTreeMap<InternedString, SocketAddr>>,
 }
 impl ForwardState {
-    async fn sync(&mut self, interfaces: &BTreeMap<InternedString, bool>) -> Result<(), Error> {
+    async fn sync(
+        &mut self,
+        interfaces: &BTreeMap<InternedString, (bool, Vec<Ipv4Addr>)>,
+    ) -> Result<(), Error> {
         let private_interfaces = interfaces
             .iter()
-            .filter(|(_, public)| !*public)
+            .filter(|(_, (public, _))| !*public)
             .map(|(i, _)| i)
             .collect::<BTreeSet<_>>();
         let all_interfaces = interfaces.keys().collect::<BTreeSet<_>>();
@@ -81,26 +84,30 @@ impl ForwardState {
                     let mut to_rm = actual
                         .difference(expected)
                         .copied()
-                        .cloned()
-                        .collect::<BTreeSet<_>>();
+                        .map(|i| (i.clone(), &interfaces[i].1))
+                        .collect::<BTreeMap<_, _>>();
                     let mut to_add = expected
                         .difference(&actual)
                         .copied()
-                        .cloned()
-                        .collect::<BTreeSet<_>>();
+                        .map(|i| (i.clone(), &interfaces[i].1))
+                        .collect::<BTreeMap<_, _>>();
                     for interface in actual.intersection(expected).copied() {
                         if cur[interface] != req.target {
-                            to_rm.insert(interface.clone());
-                            to_add.insert(interface.clone());
+                            to_rm.insert(interface.clone(), &interfaces[interface].1);
+                            to_add.insert(interface.clone(), &interfaces[interface].1);
                         }
                     }
-                    for interface in to_rm {
-                        unforward(external, &*interface, cur[&interface]).await?;
+                    for (interface, ips) in to_rm {
+                        for ip in ips {
+                            unforward(&*interface, (*ip, external).into(), cur[&interface]).await?;
+                        }
                         cur.remove(&interface);
                     }
-                    for interface in to_add {
-                        forward(external, &*interface, req.target).await?;
-                        cur.insert(interface, req.target);
+                    for (interface, ips) in to_add {
+                        cur.insert(interface.clone(), req.target);
+                        for ip in ips {
+                            forward(&*interface, (*ip, external).into(), cur[&interface]).await?;
+                        }
                     }
                 }
                 (Some(req), None) => {
@@ -112,16 +119,19 @@ impl ForwardState {
                     }
                     .into_iter()
                     .copied()
-                    .cloned()
                     {
-                        forward(external, &*interface, req.target).await?;
-                        cur.insert(interface, req.target);
+                        cur.insert(interface.clone(), req.target);
+                        for ip in &interfaces[interface].1 {
+                            forward(&**interface, (*ip, external).into(), req.target).await?;
+                        }
                     }
                 }
                 (None, Some(cur)) => {
                     let to_rm = cur.keys().cloned().collect::<BTreeSet<_>>();
                     for interface in to_rm {
-                        unforward(external, &*interface, cur[&interface]).await?;
+                        for ip in &interfaces[&interface].1 {
+                            unforward(&*interface, (*ip, external).into(), cur[&interface]).await?;
+                        }
                         cur.remove(&interface);
                     }
                     self.current.remove(&external);
@@ -155,7 +165,26 @@ impl LanPortForwardController {
             let mut interfaces = ip_info.peek_and_mark_seen(|ip_info| {
                 ip_info
                     .iter()
-                    .map(|(iface, info)| (iface.clone(), info.public()))
+                    .map(|(iface, info)| {
+                        (
+                            iface.clone(),
+                            (
+                                info.inbound(),
+                                info.ip_info.as_ref().map_or(Vec::new(), |i| {
+                                    i.subnets
+                                        .iter()
+                                        .filter_map(|s| {
+                                            if let IpAddr::V4(ip) = s.addr() {
+                                                Some(ip)
+                                            } else {
+                                                None
+                                            }
+                                        })
+                                        .collect()
+                                }),
+                            ),
+                        )
+                    })
                     .collect()
             });
             let mut reply: Option<oneshot::Sender<Result<(), Error>>> = None;
@@ -175,7 +204,21 @@ impl LanPortForwardController {
                         interfaces = ip_info.peek(|ip_info| {
                             ip_info
                                 .iter()
-                                .map(|(iface, info)| (iface.clone(), info.public()))
+                                .map(|(iface, info)| (iface.clone(), (
+                                    info.inbound(),
+                                    info.ip_info.as_ref().map_or(Vec::new(), |i| {
+                                        i.subnets
+                                            .iter()
+                                            .filter_map(|s| {
+                                                if let IpAddr::V4(ip) = s.addr() {
+                                                    Some(ip)
+                                                } else {
+                                                    None
+                                                }
+                                            })
+                                            .collect()
+                                    }),
+                                )))
                                 .collect()
                         });
                     }
@@ -222,86 +265,29 @@ impl LanPortForwardController {
     }
 }
 
-// iptables -I FORWARD -o br-start9 -p tcp -d 172.18.0.2 --dport 8333 -j ACCEPT
-// iptables -t nat -I PREROUTING -p tcp --dport 32768 -j DNAT --to 172.18.0.2:8333
-async fn forward(external: u16, interface: &str, target: SocketAddr) -> Result<(), Error> {
-    for proto in ["tcp", "udp"] {
-        Command::new("iptables")
-            .arg("-I")
-            .arg("FORWARD")
-            .arg("-i")
-            .arg(interface)
-            .arg("-o")
-            .arg(START9_BRIDGE_IFACE)
-            .arg("-p")
-            .arg(proto)
-            .arg("-d")
-            .arg(target.ip().to_string())
-            .arg("--dport")
-            .arg(target.port().to_string())
-            .arg("-j")
-            .arg("ACCEPT")
-            .invoke(crate::ErrorKind::Network)
-            .await?;
-        Command::new("iptables")
-            .arg("-t")
-            .arg("nat")
-            .arg("-I")
-            .arg("PREROUTING")
-            .arg("-i")
-            .arg(interface)
-            .arg("-p")
-            .arg(proto)
-            .arg("--dport")
-            .arg(external.to_string())
-            .arg("-j")
-            .arg("DNAT")
-            .arg("--to")
-            .arg(target.to_string())
-            .invoke(crate::ErrorKind::Network)
-            .await?;
-    }
+async fn forward(interface: &str, source: SocketAddr, target: SocketAddr) -> Result<(), Error> {
+    Command::new("/usr/lib/startos/scripts/forward-port")
+        .env("iiface", interface)
+        .env("oiface", START9_BRIDGE_IFACE)
+        .env("sip", source.ip().to_string())
+        .env("dip", target.ip().to_string())
+        .env("sport", source.port().to_string())
+        .env("dport", target.port().to_string())
+        .invoke(ErrorKind::Network)
+        .await?;
     Ok(())
 }
 
-// iptables -D FORWARD -o br-start9 -p tcp -d 172.18.0.2 --dport 8333 -j ACCEPT
-// iptables -t nat -D PREROUTING -p tcp --dport 32768 -j DNAT --to 172.18.0.2:8333
-async fn unforward(external: u16, interface: &str, target: SocketAddr) -> Result<(), Error> {
-    for proto in ["tcp", "udp"] {
-        Command::new("iptables")
-            .arg("-D")
-            .arg("FORWARD")
-            .arg("-i")
-            .arg(interface)
-            .arg("-o")
-            .arg(START9_BRIDGE_IFACE)
-            .arg("-p")
-            .arg(proto)
-            .arg("-d")
-            .arg(target.ip().to_string())
-            .arg("--dport")
-            .arg(target.port().to_string())
-            .arg("-j")
-            .arg("ACCEPT")
-            .invoke(crate::ErrorKind::Network)
-            .await?;
-        Command::new("iptables")
-            .arg("-t")
-            .arg("nat")
-            .arg("-D")
-            .arg("PREROUTING")
-            .arg("-i")
-            .arg(interface)
-            .arg("-p")
-            .arg(proto)
-            .arg("--dport")
-            .arg(external.to_string())
-            .arg("-j")
-            .arg("DNAT")
-            .arg("--to")
-            .arg(target.to_string())
-            .invoke(crate::ErrorKind::Network)
-            .await?;
-    }
+async fn unforward(interface: &str, source: SocketAddr, target: SocketAddr) -> Result<(), Error> {
+    Command::new("/usr/lib/startos/scripts/forward-port")
+        .env("UNDO", "1")
+        .env("iiface", interface)
+        .env("oiface", START9_BRIDGE_IFACE)
+        .env("sip", source.ip().to_string())
+        .env("dip", target.ip().to_string())
+        .env("sport", source.port().to_string())
+        .env("dport", target.port().to_string())
+        .invoke(ErrorKind::Network)
+        .await?;
     Ok(())
 }

core/startos/src/net/host/address.rs

@@ -186,7 +186,7 @@ pub async fn add_domain<Kind: HostApiKind>(
     ctx.db
         .mutate(|db| {
             if let Some(acme) = &acme {
-                if !db.as_public().as_server_info().as_acme().contains_key(&acme)? {
+                if !db.as_public().as_server_info().as_network().as_acme().contains_key(&acme)? {
                     return Err(Error::new(eyre!("unknown acme provider {}, please run acme.init for this provider first", acme.0), ErrorKind::InvalidRequest));
                 }
             }

core/startos/src/net/host/binding.rs

@@ -62,12 +62,14 @@ impl BindInfo {
     pub fn new(available_ports: &mut AvailablePorts, options: BindOptions) -> Result<Self, Error> {
         let mut assigned_port = None;
         let mut assigned_ssl_port = None;
-        if options.secure.is_some() {
-            assigned_port = Some(available_ports.alloc()?);
-        }
         if options.add_ssl.is_some() {
             assigned_ssl_port = Some(available_ports.alloc()?);
         }
+        if let Some(secure) = options.secure {
+            if !secure.ssl || !options.add_ssl.is_some() {
+                assigned_port = Some(available_ports.alloc()?);
+            }
+        }
         Ok(Self {
             enabled: true,
             options,
@@ -159,7 +161,7 @@ pub fn binding<C: Context, Kind: HostApiKind>(
                     use prettytable::*;
 
                     if let Some(format) = params.format {
-                        return Ok(display_serializable(format, res));
+                        return display_serializable(format, res);
                     }
 
                     let mut table = Table::new();
@@ -182,7 +184,7 @@ pub fn binding<C: Context, Kind: HostApiKind>(
                         ]);
                     }
 
-                    table.print_tty(false).unwrap();
+                    table.print_tty(false)?;
 
                     Ok(())
                 })

core/startos/src/net/host/mod.rs

@@ -85,7 +85,11 @@ pub fn host_for<'a>(
     host_id: &HostId,
 ) -> Result<&'a mut Model<Host>, Error> {
     let Some(package_id) = package_id else {
-        return Ok(db.as_public_mut().as_server_info_mut().as_host_mut());
+        return Ok(db
+            .as_public_mut()
+            .as_server_info_mut()
+            .as_network_mut()
+            .as_host_mut());
     };
     fn host_info<'a>(
         db: &'a mut DatabaseModel,
@@ -122,7 +126,7 @@ pub fn host_for<'a>(
 }
 
 pub fn all_hosts(db: &DatabaseModel) -> impl Iterator<Item = Result<&Model<Host>, Error>> {
-    [Ok(db.as_public().as_server_info().as_host())]
+    [Ok(db.as_public().as_server_info().as_network().as_host())]
         .into_iter()
         .chain(
             [db.as_public().as_package_data().as_entries()]
@@ -255,7 +259,7 @@ pub async fn list_hosts(
     ctx: RpcContext,
     _: Empty,
     package: PackageId,
-) -> Result<Vec<HostId>, Error> {
+) -> Result<BTreeSet<HostId>, Error> {
     ctx.db
         .peek()
         .await

core/startos/src/net/net_controller.rs

@@ -192,7 +192,11 @@ impl NetServiceData {
             let host = ctrl
                 .db
                 .mutate(|db| {
-                    let host = db.as_public_mut().as_server_info_mut().as_host_mut();
+                    let host = db
+                        .as_public_mut()
+                        .as_server_info_mut()
+                        .as_network_mut()
+                        .as_host_mut();
                     host.as_bindings_mut().mutate(|b| {
                         for (internal_port, info) in b {
                             if !except.contains(&BindId {
@@ -331,7 +335,7 @@ impl NetServiceData {
                 for (interface, public, ip_info) in
                     net_ifaces.iter().filter_map(|(interface, info)| {
                         if let Some(ip_info) = &info.ip_info {
-                            Some((interface, info.public(), ip_info))
+                            Some((interface, info.inbound(), ip_info))
                         } else {
                             None
                         }
@@ -488,7 +492,7 @@ impl NetServiceData {
                 let mut bind_hostname_info = hostname_info.remove(internal).unwrap_or_default();
                 bind_hostname_info.push(HostnameInfo::Onion {
                     hostname: OnionHostname {
-                        value: tor_addr.to_string(),
+                        value: InternedString::from_display(tor_addr),
                         port: ports.non_ssl,
                         ssl_port: ports.ssl,
                     },
@@ -626,6 +630,7 @@ impl NetServiceData {
                     .await
                     .as_public()
                     .as_server_info()
+                    .as_network()
                     .as_host()
                     .de()?,
             )

core/startos/src/net/network_interface.rs

@@ -47,7 +47,7 @@ pub fn network_interface_api<C: Context>() -> ParentHandler<C> {
                     use prettytable::*;
 
                     if let Some(format) = params.format {
-                        return Ok(display_serializable(format, res));
+                        return display_serializable(format, res);
                     }
 
                     let mut table = Table::new();
@@ -58,7 +58,7 @@ pub fn network_interface_api<C: Context>() -> ParentHandler<C> {
                             info.ip_info.as_ref()
                                 .and_then(|ip_info| ip_info.device_type)
                                 .map_or_else(|| "UNKNOWN".to_owned(), |ty| format!("{ty:?}")),
-                            info.public(),
+                            info.inbound(),
                             info.ip_info.as_ref().map_or_else(
                                 || "<DISCONNECTED>".to_owned(),
                                 |ip_info| ip_info.subnets
@@ -78,7 +78,7 @@ pub fn network_interface_api<C: Context>() -> ParentHandler<C> {
                         ]);
                     }
 
-                    table.print_tty(false).unwrap();
+                    table.print_tty(false)?;
 
                     Ok(())
                 })
@@ -86,18 +86,18 @@ pub fn network_interface_api<C: Context>() -> ParentHandler<C> {
                 .with_call_remote::<CliContext>(),
         )
         .subcommand(
-            "set-public",
-            from_fn_async(set_public)
+            "set-inbound",
+            from_fn_async(set_inbound)
                 .with_metadata("sync_db", Value::Bool(true))
                 .no_display()
-                .with_about("Indicate whether this interface is publicly addressable")
+                .with_about("Indicate whether this interface has inbound access from the WAN")
                 .with_call_remote::<CliContext>(),
         ).subcommand(
-            "unset-public",
-            from_fn_async(unset_public)
+            "unset-inbound",
+            from_fn_async(unset_inbound)
                 .with_metadata("sync_db", Value::Bool(true))
                 .no_display()
-                .with_about("Allow this interface to infer whether it is publicly addressable based on its IPv4 address")
+                .with_about("Allow this interface to infer whether it has inbound access from the WAN based on its IPv4 address")
                 .with_call_remote::<CliContext>(),
         ).subcommand("forget",
             from_fn_async(forget_iface)
@@ -116,36 +116,36 @@ async fn list_interfaces(
 
 #[derive(Debug, Clone, Deserialize, Serialize, Parser, TS)]
 #[ts(export)]
-struct NetworkInterfaceSetPublicParams {
+struct NetworkInterfaceSetInboundParams {
     #[ts(type = "string")]
     interface: InternedString,
-    public: Option<bool>,
+    inbound: Option<bool>,
 }
 
-async fn set_public(
+async fn set_inbound(
     ctx: RpcContext,
-    NetworkInterfaceSetPublicParams { interface, public }: NetworkInterfaceSetPublicParams,
+    NetworkInterfaceSetInboundParams { interface, inbound }: NetworkInterfaceSetInboundParams,
 ) -> Result<(), Error> {
     ctx.net_controller
         .net_iface
-        .set_public(&interface, Some(public.unwrap_or(true)))
+        .set_inbound(&interface, Some(inbound.unwrap_or(true)))
         .await
 }
 
 #[derive(Debug, Clone, Deserialize, Serialize, Parser, TS)]
 #[ts(export)]
-struct UnsetPublicParams {
+struct UnsetInboundParams {
     #[ts(type = "string")]
     interface: InternedString,
 }
 
-async fn unset_public(
+async fn unset_inbound(
     ctx: RpcContext,
-    UnsetPublicParams { interface }: UnsetPublicParams,
+    UnsetInboundParams { interface }: UnsetInboundParams,
 ) -> Result<(), Error> {
     ctx.net_controller
         .net_iface
-        .set_public(&interface, None)
+        .set_inbound(&interface, None)
         .await
 }
 
@@ -193,6 +193,9 @@ mod active_connection {
         default_service = "org.freedesktop.NetworkManager"
     )]
     pub trait ActiveConnection {
+        #[zbus(property)]
+        fn id(&self) -> Result<String, Error>;
+
         #[zbus(property)]
         fn state_flags(&self) -> Result<u32, Error>;
 
@@ -511,6 +514,8 @@ async fn watch_ip(
                                 _ => None,
                             };
 
+                            let name = InternedString::from(active_connection_proxy.id().await?);
+
                             let dhcp4_config = active_connection_proxy.dhcp4_config().await?;
                             let ip4_proxy =
                                 Ip4ConfigProxy::new(&connection, ip4_config.clone()).await?;
@@ -568,6 +573,7 @@ async fn watch_ip(
                                                 }
                                             };
                                             Some(IpInfo {
+                                                name: name.clone(),
                                                 scope_id,
                                                 device_type,
                                                 subnets,
@@ -579,11 +585,14 @@ async fn watch_ip(
                                         };
 
                                         write_to.send_if_modified(|m| {
-                                            let public = m.get(&iface).map_or(None, |i| i.public);
+                                            let (inbound, outbound) = m
+                                                .get(&iface)
+                                                .map_or((None, None), |i| (i.inbound, i.outbound));
                                             m.insert(
                                                 iface.clone(),
                                                 NetworkInterfaceInfo {
-                                                    public,
+                                                    inbound,
+                                                    outbound,
                                                     ip_info: ip_info.clone(),
                                                 },
                                             )
@@ -663,6 +672,7 @@ impl NetworkInterfaceController {
         db.mutate(|db| {
             db.as_public_mut()
                 .as_server_info_mut()
+                .as_network_mut()
                 .as_network_interfaces_mut()
                 .ser(info)
         })
@@ -732,6 +742,7 @@ impl NetworkInterfaceController {
                     .await
                     .as_public()
                     .as_server_info()
+                    .as_network()
                     .as_network_interfaces()
                     .de()
                 {
@@ -821,7 +832,7 @@ impl NetworkInterfaceController {
         Ok(listener)
     }
 
-    pub async fn set_public(
+    pub async fn set_inbound(
         &self,
         interface: &InternedString,
         public: Option<bool>,
@@ -829,7 +840,7 @@ impl NetworkInterfaceController {
         let mut sub = self
             .db
             .subscribe(
-                "/public/serverInfo/networkInterfaces"
+                "/public/serverInfo/network/networkInterfaces"
                     .parse::<JsonPointer<_, _>>()
                     .with_kind(ErrorKind::Database)?,
             )
@@ -844,7 +855,7 @@ impl NetworkInterfaceController {
                         return false;
                     }
                 }
-                .public,
+                .inbound,
                 public,
             );
             prev != public
@@ -862,7 +873,7 @@ impl NetworkInterfaceController {
         let mut sub = self
             .db
             .subscribe(
-                "/public/serverInfo/networkInterfaces"
+                "/public/serverInfo/network/networkInterfaces"
                     .parse::<JsonPointer<_, _>>()
                     .with_kind(ErrorKind::Database)?,
             )
@@ -956,8 +967,10 @@ impl ListenerMap {
     ) -> Result<(), Error> {
         let mut keep = BTreeSet::<SocketAddr>::new();
         for info in ip_info.values().chain([&NetworkInterfaceInfo {
-            public: Some(false),
+            inbound: Some(false),
+            outbound: Some(false),
             ip_info: Some(IpInfo {
+                name: "lo".into(),
                 scope_id: 1,
                 device_type: None,
                 subnets: [
@@ -970,7 +983,7 @@ impl ListenerMap {
                 ntp_servers: Default::default(),
             }),
         }]) {
-            if public || !info.public() {
+            if public || !info.inbound() {
                 if let Some(ip_info) = &info.ip_info {
                     for ipnet in &ip_info.subnets {
                         let addr = match ipnet.addr() {
@@ -989,7 +1002,7 @@ impl ListenerMap {
                         };
                         keep.insert(addr);
                         if let Some((_, is_public, wan_ip)) = self.listeners.get_mut(&addr) {
-                            *is_public = info.public();
+                            *is_public = info.inbound();
                             *wan_ip = info.ip_info.as_ref().and_then(|i| i.wan_ip);
                             continue;
                         }
@@ -1007,7 +1020,7 @@ impl ListenerMap {
                                         .into(),
                                 )
                                 .with_kind(ErrorKind::Network)?,
-                                info.public(),
+                                info.inbound(),
                                 info.ip_info.as_ref().and_then(|i| i.wan_ip),
                             ),
                         );

core/startos/src/net/service_interface.rs

@@ -1,6 +1,7 @@
 use std::net::{Ipv4Addr, Ipv6Addr};
 
 use imbl_value::InternedString;
+use lazy_format::lazy_format;
 use models::{HostId, ServiceInterfaceId};
 use serde::{Deserialize, Serialize};
 use ts_rs::TS;
@@ -21,15 +22,29 @@ pub enum HostnameInfo {
         hostname: OnionHostname,
     },
 }
+impl HostnameInfo {
+    pub fn to_san_hostname(&self) -> InternedString {
+        match self {
+            Self::Ip { hostname, .. } => hostname.to_san_hostname(),
+            Self::Onion { hostname } => hostname.to_san_hostname(),
+        }
+    }
+}
 
 #[derive(Clone, Debug, Deserialize, Serialize, TS)]
 #[ts(export)]
 #[serde(rename_all = "camelCase")]
 pub struct OnionHostname {
-    pub value: String,
+    #[ts(type = "string")]
+    pub value: InternedString,
     pub port: Option<u16>,
     pub ssl_port: Option<u16>,
 }
+impl OnionHostname {
+    pub fn to_san_hostname(&self) -> InternedString {
+        self.value.clone()
+    }
+}
 
 #[derive(Clone, Debug, Deserialize, Serialize, TS)]
 #[ts(export)]
@@ -64,6 +79,24 @@ pub enum IpHostname {
         ssl_port: Option<u16>,
     },
 }
+impl IpHostname {
+    pub fn to_san_hostname(&self) -> InternedString {
+        match self {
+            Self::Ipv4 { value, .. } => InternedString::from_display(value),
+            Self::Ipv6 { value, .. } => InternedString::from_display(value),
+            Self::Local { value, .. } => value.clone(),
+            Self::Domain {
+                domain, subdomain, ..
+            } => {
+                if let Some(subdomain) = subdomain {
+                    InternedString::from_display(&lazy_format!("{subdomain}.{domain}"))
+                } else {
+                    domain.clone()
+                }
+            }
+        }
+    }
+}
 
 #[derive(Clone, Debug, Deserialize, Serialize, TS)]
 #[ts(export)]

core/startos/src/net/static_server.rs

@@ -83,9 +83,9 @@ pub fn rpc_router<C: Context + Clone + AsRef<RpcContinuations>>(
     server: HttpServer<C>,
 ) -> Router {
     Router::new()
-        .route("/rpc/*path", any(server))
+        .route("/rpc/{*path}", any(server))
         .route(
-            "/ws/rpc/:guid",
+            "/ws/rpc/{guid}",
             get({
                 let ctx = ctx.clone();
                 move |x::Path(guid): x::Path<Guid>,
@@ -98,7 +98,7 @@ pub fn rpc_router<C: Context + Clone + AsRef<RpcContinuations>>(
             }),
         )
         .route(
-            "/rest/rpc/:guid",
+            "/rest/rpc/{guid}",
             any({
                 let ctx = ctx.clone();
                 move |x::Path(guid): x::Path<Guid>, request: x::Request| async move {
@@ -185,7 +185,7 @@ pub fn main_ui_router(ctx: RpcContext) -> Router {
             .middleware(Auth::new())
             .middleware(SyncDb::new())
     })
-    .route("/proxy/:url", {
+    .route("/proxy/{url}", {
         let ctx = ctx.clone();
         any(move |x::Path(url): x::Path<String>, request: Request| {
             let ctx = ctx.clone();
@@ -258,7 +258,7 @@ async fn proxy_request(ctx: RpcContext, request: Request, url: String) -> Result
 
 fn s9pk_router(ctx: RpcContext) -> Router {
     Router::new()
-        .route("/installed/:s9pk", {
+        .route("/installed/{s9pk}", {
             let ctx = ctx.clone();
             any(
                 |x::Path(s9pk): x::Path<String>, request: Request| async move {
@@ -282,7 +282,7 @@ fn s9pk_router(ctx: RpcContext) -> Router {
                 },
             )
         })
-        .route("/installed/:s9pk/*path", {
+        .route("/installed/{s9pk}/{*path}", {
             let ctx = ctx.clone();
             any(
                 |x::Path((s9pk, path)): x::Path<(String, PathBuf)>,
@@ -319,7 +319,7 @@ fn s9pk_router(ctx: RpcContext) -> Router {
             )
         })
         .route(
-            "/proxy/:url/*path",
+            "/proxy/{url}/{*path}",
             any(
                 |x::Path((url, path)): x::Path<(Url, PathBuf)>,
                  x::RawQuery(query): x::RawQuery,

core/startos/src/net/tor.rs

@@ -1,4 +1,4 @@
-use std::collections::BTreeMap;
+use std::collections::{BTreeMap, BTreeSet};
 use std::net::SocketAddr;
 use std::sync::atomic::AtomicBool;
 use std::sync::{Arc, Weak};
@@ -90,9 +90,7 @@ pub fn tor<C: Context>() -> ParentHandler<C> {
             "list-services",
             from_fn_async(list_services)
                 .with_display_serializable()
-                .with_custom_display_fn(|handle, result| {
-                    Ok(display_services(handle.params, result))
-                })
+                .with_custom_display_fn(|handle, result| display_services(handle.params, result))
                 .with_about("Display Tor V3 Onion Addresses")
                 .with_call_remote::<CliContext>(),
         )
@@ -181,7 +179,7 @@ pub async fn add_key(
     Ok(key.public().get_onion_address())
 }
 
-pub async fn list_keys(ctx: RpcContext) -> Result<Vec<OnionAddressV3>, Error> {
+pub async fn list_keys(ctx: RpcContext) -> Result<BTreeSet<OnionAddressV3>, Error> {
     ctx.db
         .peek()
         .await
@@ -210,7 +208,10 @@ pub async fn reset(
         .await
 }
 
-pub fn display_services(params: WithIoFormat<Empty>, services: Vec<OnionAddressV3>) {
+pub fn display_services(
+    params: WithIoFormat<Empty>,
+    services: Vec<OnionAddressV3>,
+) -> Result<(), Error> {
     use prettytable::*;
 
     if let Some(format) = params.format {
@@ -222,7 +223,8 @@ pub fn display_services(params: WithIoFormat<Empty>, services: Vec<OnionAddressV
         let row = row![&service.to_string()];
         table.add_row(row);
     }
-    table.print_tty(false).unwrap();
+    table.print_tty(false)?;
+    Ok(())
 }
 
 pub async fn list_services(ctx: RpcContext, _: Empty) -> Result<Vec<OnionAddressV3>, Error> {

core/startos/src/net/utils.rs

@@ -1,4 +1,3 @@
-use std::collections::BTreeMap;
 use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr, SocketAddrV6};
 use std::path::Path;
 
@@ -6,18 +5,13 @@ use async_stream::try_stream;
 use color_eyre::eyre::eyre;
 use futures::stream::BoxStream;
 use futures::{StreamExt, TryStreamExt};
-use helpers::NonDetachingJoinHandle;
 use imbl_value::InternedString;
 use ipnet::{IpNet, Ipv4Net, Ipv6Net};
 use nix::net::if_::if_nametoindex;
 use tokio::net::{TcpListener, TcpStream};
 use tokio::process::Command;
 
-use crate::db::model::public::NetworkInterfaceInfo;
-use crate::net::network_interface::NetworkInterfaceListener;
-use crate::net::web_server::Accept;
 use crate::prelude::*;
-use crate::util::sync::Watch;
 use crate::util::Invoke;
 
 pub fn ipv6_is_link_local(addr: Ipv6Addr) -> bool {

core/startos/src/net/vhost.rs

@@ -1,7 +1,6 @@
 use std::collections::{BTreeMap, BTreeSet};
 use std::net::{IpAddr, SocketAddr};
 use std::sync::{Arc, Weak};
-use std::time::Duration;
 
 use async_acme::acme::{Identifier, ACME_TLS_ALPN_NAME};
 use axum::body::Body;
@@ -451,6 +450,7 @@ impl VHostServer {
                         target.acme.as_ref().and_then(|a| {
                             peek.as_public()
                                 .as_server_info()
+                                .as_network()
                                 .as_acme()
                                 .as_idx(a)
                                 .map(|s| (domain, a, s))

core/startos/src/net/wifi.rs

@@ -18,27 +18,33 @@ use ts_rs::TS;
 use crate::context::{CliContext, RpcContext};
 use crate::db::model::public::WifiInfo;
 use crate::db::model::Database;
-use crate::net::utils::find_wifi_iface;
 use crate::prelude::*;
 use crate::util::serde::{display_serializable, HandlerExtSerde, WithIoFormat};
 use crate::util::Invoke;
 use crate::{Error, ErrorKind};
 
-type WifiManager = Arc<RwLock<WpaCli>>;
+type WifiManager = Arc<RwLock<Option<WpaCli>>>;
 
-pub fn wifi_manager(ctx: &RpcContext) -> Result<&WifiManager, Error> {
-    if let Some(wifi_manager) = ctx.wifi_manager.as_ref() {
-        Ok(wifi_manager)
-    } else {
-        Err(Error::new(
-            color_eyre::eyre::eyre!("No WiFi interface available"),
-            ErrorKind::Wifi,
-        ))
-    }
-}
+// pub fn wifi_manager(ctx: &RpcContext) -> Result<&WifiManager, Error> {
+//     if let Some(wifi_manager) = ctx.wifi_manager.as_ref() {
+//         Ok(wifi_manager)
+//     } else {
+//         Err(Error::new(
+//             color_eyre::eyre::eyre!("No WiFi interface available"),
+//             ErrorKind::Wifi,
+//         ))
+//     }
+// }
 
 pub fn wifi<C: Context>() -> ParentHandler<C> {
     ParentHandler::new()
+        .subcommand(
+            "set-enabled",
+            from_fn_async(set_enabled)
+                .no_display()
+                .with_about("Enable or disable wifi")
+                .with_call_remote::<CliContext>(),
+        )
         .subcommand(
             "add",
             from_fn_async(add)
@@ -64,9 +70,7 @@ pub fn wifi<C: Context>() -> ParentHandler<C> {
             "get",
             from_fn_async(get)
                 .with_display_serializable()
-                .with_custom_display_fn(|handle, result| {
-                    Ok(display_wifi_info(handle.params, result))
-                })
+                .with_custom_display_fn(|handle, result| display_wifi_info(handle.params, result))
                 .with_about("List wifi info")
                 .with_call_remote::<CliContext>(),
         )
@@ -80,12 +84,55 @@ pub fn wifi<C: Context>() -> ParentHandler<C> {
         )
 }
 
+#[derive(Deserialize, Serialize, Parser, TS)]
+#[serde(rename_all = "camelCase")]
+#[command(rename_all = "kebab-case")]
+pub struct SetWifiEnabledParams {
+    pub enabled: bool,
+}
+
+pub async fn set_enabled(
+    ctx: RpcContext,
+    SetWifiEnabledParams { enabled }: SetWifiEnabledParams,
+) -> Result<(), Error> {
+    if enabled {
+        Command::new("rfkill")
+            .arg("unblock")
+            .arg("all")
+            .invoke(ErrorKind::Wifi)
+            .await?;
+    } else {
+        Command::new("rfkill")
+            .arg("block")
+            .arg("all")
+            .invoke(ErrorKind::Wifi)
+            .await?;
+    }
+    let iface = if let Some(man) = ctx.wifi_manager.read().await.as_ref().filter(|_| enabled) {
+        Some(man.interface.clone())
+    } else {
+        None
+    };
+    ctx.db
+        .mutate(|d| {
+            d.as_public_mut()
+                .as_server_info_mut()
+                .as_network_mut()
+                .as_wifi_mut()
+                .as_interface_mut()
+                .ser(&iface)
+        })
+        .await
+        .result?;
+    Ok(())
+}
+
 pub fn available<C: Context>() -> ParentHandler<C> {
     ParentHandler::new().subcommand(
         "get",
         from_fn_async(get_available)
             .with_display_serializable()
-            .with_custom_display_fn(|handle, result| Ok(display_wifi_list(handle.params, result)))
+            .with_custom_display_fn(|handle, result| display_wifi_list(handle.params, result))
             .with_about("List available wifi networks")
             .with_call_remote::<CliContext>(),
     )
@@ -110,7 +157,7 @@ pub struct AddParams {
 }
 #[instrument(skip_all)]
 pub async fn add(ctx: RpcContext, AddParams { ssid, password }: AddParams) -> Result<(), Error> {
-    let wifi_manager = wifi_manager(&ctx)?;
+    let wifi_manager = ctx.wifi_manager.clone();
     if !ssid.is_ascii() {
         return Err(Error::new(
             color_eyre::eyre::eyre!("SSID may not have special characters"),
@@ -130,9 +177,14 @@ pub async fn add(ctx: RpcContext, AddParams { ssid, password }: AddParams) -> Re
         password: &Psk,
     ) -> Result<(), Error> {
         tracing::info!("Adding new WiFi network: '{}'", ssid.0);
-        let mut wpa_supplicant = wifi_manager.write().await;
+        let mut wpa_supplicant = wifi_manager.write_owned().await;
+        let wpa_supplicant = wpa_supplicant.as_mut().ok_or_else(|| {
+            Error::new(
+                color_eyre::eyre::eyre!("No WiFi interface available"),
+                ErrorKind::Wifi,
+            )
+        })?;
         wpa_supplicant.add_network(db, ssid, password).await?;
-        drop(wpa_supplicant);
         Ok(())
     }
     if let Err(err) = add_procedure(
@@ -154,6 +206,7 @@ pub async fn add(ctx: RpcContext, AddParams { ssid, password }: AddParams) -> Re
         .mutate(|db| {
             db.as_public_mut()
                 .as_server_info_mut()
+                .as_network_mut()
                 .as_wifi_mut()
                 .as_ssids_mut()
                 .mutate(|s| {
@@ -174,7 +227,7 @@ pub struct SsidParams {
 
 #[instrument(skip_all)]
 pub async fn connect(ctx: RpcContext, SsidParams { ssid }: SsidParams) -> Result<(), Error> {
-    let wifi_manager = wifi_manager(&ctx)?;
+    let wifi_manager = ctx.wifi_manager.clone();
     if !ssid.is_ascii() {
         return Err(Error::new(
             color_eyre::eyre::eyre!("SSID may not have special characters"),
@@ -186,10 +239,14 @@ pub async fn connect(ctx: RpcContext, SsidParams { ssid }: SsidParams) -> Result
         wifi_manager: WifiManager,
         ssid: &Ssid,
     ) -> Result<(), Error> {
-        let wpa_supplicant = wifi_manager.read().await;
+        let mut wpa_supplicant = wifi_manager.write_owned().await;
+        let wpa_supplicant = wpa_supplicant.as_mut().ok_or_else(|| {
+            Error::new(
+                color_eyre::eyre::eyre!("No WiFi interface available"),
+                ErrorKind::Wifi,
+            )
+        })?;
         let current = wpa_supplicant.get_current_network().await?;
-        drop(wpa_supplicant);
-        let mut wpa_supplicant = wifi_manager.write().await;
         let connected = wpa_supplicant.select_network(db.clone(), ssid).await?;
         if connected {
             tracing::info!("Successfully connected to WiFi: '{}'", ssid.0);
@@ -219,7 +276,11 @@ pub async fn connect(ctx: RpcContext, SsidParams { ssid }: SsidParams) -> Result
 
     ctx.db
         .mutate(|db| {
-            let wifi = db.as_public_mut().as_server_info_mut().as_wifi_mut();
+            let wifi = db
+                .as_public_mut()
+                .as_server_info_mut()
+                .as_network_mut()
+                .as_wifi_mut();
             wifi.as_ssids_mut().mutate(|s| {
                 s.insert(ssid.clone());
                 Ok(())
@@ -233,17 +294,22 @@ pub async fn connect(ctx: RpcContext, SsidParams { ssid }: SsidParams) -> Result
 
 #[instrument(skip_all)]
 pub async fn remove(ctx: RpcContext, SsidParams { ssid }: SsidParams) -> Result<(), Error> {
-    let wifi_manager = wifi_manager(&ctx)?;
+    let wifi_manager = ctx.wifi_manager.clone();
     if !ssid.is_ascii() {
         return Err(Error::new(
             color_eyre::eyre::eyre!("SSID may not have special characters"),
             ErrorKind::Wifi,
         ));
     }
-    let wpa_supplicant = wifi_manager.read().await;
+
+    let mut wpa_supplicant = wifi_manager.write_owned().await;
+    let wpa_supplicant = wpa_supplicant.as_mut().ok_or_else(|| {
+        Error::new(
+            color_eyre::eyre::eyre!("No WiFi interface available"),
+            ErrorKind::Wifi,
+        )
+    })?;
     let current = wpa_supplicant.get_current_network().await?;
-    drop(wpa_supplicant);
-    let mut wpa_supplicant = wifi_manager.write().await;
     let ssid = Ssid(ssid);
     let is_current_being_removed = matches!(current, Some(current) if current == ssid);
     let is_current_removed_and_no_hardwire =
@@ -256,7 +322,11 @@ pub async fn remove(ctx: RpcContext, SsidParams { ssid }: SsidParams) -> Result<
 
     ctx.db
         .mutate(|db| {
-            let wifi = db.as_public_mut().as_server_info_mut().as_wifi_mut();
+            let wifi = db
+                .as_public_mut()
+                .as_server_info_mut()
+                .as_network_mut()
+                .as_wifi_mut();
             wifi.as_ssids_mut().mutate(|s| {
                 s.remove(&ssid.0);
                 Ok(())
@@ -291,7 +361,7 @@ pub struct WifiListOut {
     security: Vec<String>,
 }
 pub type WifiList = HashMap<Ssid, WifiListInfoLow>;
-fn display_wifi_info(params: WithIoFormat<Empty>, info: WifiListInfo) {
+fn display_wifi_info(params: WithIoFormat<Empty>, info: WifiListInfo) -> Result<(), Error> {
     use prettytable::*;
 
     if let Some(format) = params.format {
@@ -352,10 +422,11 @@ fn display_wifi_info(params: WithIoFormat<Empty>, info: WifiListInfo) {
         ]);
     }
 
-    table_global.print_tty(false).unwrap();
+    table_global.print_tty(false)?;
+    Ok(())
 }
 
-fn display_wifi_list(params: WithIoFormat<Empty>, info: Vec<WifiListOut>) {
+fn display_wifi_list(params: WithIoFormat<Empty>, info: Vec<WifiListOut>) -> Result<(), Error> {
     use prettytable::*;
 
     if let Some(format) = params.format {
@@ -376,14 +447,21 @@ fn display_wifi_list(params: WithIoFormat<Empty>, info: Vec<WifiListOut>) {
         ]);
     }
 
-    table_global.print_tty(false).unwrap();
+    table_global.print_tty(false)?;
+    Ok(())
 }
 
 // #[command(display(display_wifi_info))]
 #[instrument(skip_all)]
 pub async fn get(ctx: RpcContext, _: Empty) -> Result<WifiListInfo, Error> {
-    let wifi_manager = wifi_manager(&ctx)?;
-    let wpa_supplicant = wifi_manager.read().await;
+    let wifi_manager = ctx.wifi_manager.clone();
+    let wpa_supplicant = wifi_manager.read_owned().await;
+    let wpa_supplicant = wpa_supplicant.as_ref().ok_or_else(|| {
+        Error::new(
+            color_eyre::eyre::eyre!("No WiFi interface available"),
+            ErrorKind::Wifi,
+        )
+    })?;
     let (list_networks, current_res, country_res, ethernet_res, signal_strengths) = tokio::join!(
         wpa_supplicant.list_networks_low(),
         wpa_supplicant.get_current_network(),
@@ -430,8 +508,14 @@ pub async fn get(ctx: RpcContext, _: Empty) -> Result<WifiListInfo, Error> {
 
 #[instrument(skip_all)]
 pub async fn get_available(ctx: RpcContext, _: Empty) -> Result<Vec<WifiListOut>, Error> {
-    let wifi_manager = wifi_manager(&ctx)?;
-    let wpa_supplicant = wifi_manager.read().await;
+    let wifi_manager = ctx.wifi_manager.clone();
+    let wpa_supplicant = wifi_manager.read_owned().await;
+    let wpa_supplicant = wpa_supplicant.as_ref().ok_or_else(|| {
+        Error::new(
+            color_eyre::eyre::eyre!("No WiFi interface available"),
+            ErrorKind::Wifi,
+        )
+    })?;
     let (wifi_list, network_list) = tokio::join!(
         wpa_supplicant.list_wifi_low(),
         wpa_supplicant.list_networks_low()
@@ -466,14 +550,20 @@ pub async fn set_country(
     ctx: RpcContext,
     SetCountryParams { country }: SetCountryParams,
 ) -> Result<(), Error> {
-    let wifi_manager = wifi_manager(&ctx)?;
+    let wifi_manager = ctx.wifi_manager.clone();
     if !interface_connected(&ctx.ethernet_interface).await? {
         return Err(Error::new(
             color_eyre::eyre::eyre!("Won't change country without hardwire connection"),
             crate::ErrorKind::Wifi,
         ));
     }
-    let mut wpa_supplicant = wifi_manager.write().await;
+    let mut wpa_supplicant = wifi_manager.write_owned().await;
+    let wpa_supplicant = wpa_supplicant.as_mut().ok_or_else(|| {
+        Error::new(
+            color_eyre::eyre::eyre!("No WiFi interface available"),
+            ErrorKind::Wifi,
+        )
+    })?;
     wpa_supplicant.set_country_low(country.alpha2()).await?;
     for (network_id, _wifi_info) in wpa_supplicant.list_networks_low().await? {
         wpa_supplicant.remove_network_low(network_id).await?;
@@ -737,6 +827,7 @@ impl WpaCli {
         db.mutate(|d| {
             d.as_public_mut()
                 .as_server_info_mut()
+                .as_network_mut()
                 .as_wifi_mut()
                 .as_last_region_mut()
                 .ser(&new_country)
@@ -913,13 +1004,21 @@ pub async fn synchronize_network_manager<P: AsRef<Path>>(
     crate::disk::mount::util::bind(&persistent, "/etc/NetworkManager/system-connections", false)
         .await?;
 
+    if !wifi.enabled {
+        Command::new("rfkill")
+            .arg("block")
+            .arg("all")
+            .invoke(ErrorKind::Wifi)
+            .await?;
+    }
+
     Command::new("systemctl")
         .arg("restart")
         .arg("NetworkManager")
         .invoke(ErrorKind::Wifi)
         .await?;
 
-    let Some(wifi_iface) = &wifi.interface else {
+    let Some(wifi_iface) = wifi.interface.as_ref().filter(|_| wifi.enabled) else {
         return Ok(());
     };
 

core/startos/src/notifications.rs

@@ -6,6 +6,7 @@ use chrono::{DateTime, Utc};
 use clap::builder::ValueParserFactory;
 use clap::Parser;
 use color_eyre::eyre::eyre;
+use helpers::const_true;
 use imbl_value::InternedString;
 use models::{FromStrParser, PackageId};
 use rpc_toolkit::{from_fn_async, Context, HandlerExt, ParentHandler};
@@ -15,7 +16,7 @@ use ts_rs::TS;
 
 use crate::backup::BackupReport;
 use crate::context::{CliContext, RpcContext};
-use crate::db::model::{Database, DatabaseModel};
+use crate::db::model::DatabaseModel;
 use crate::prelude::*;
 use crate::util::serde::HandlerExtSerde;
 
@@ -33,14 +34,35 @@ pub fn notification<C: Context>() -> ParentHandler<C> {
             "remove",
             from_fn_async(remove)
                 .no_display()
-                .with_about("Delete notification for a given id")
+                .with_about("Remove notification for given ids")
                 .with_call_remote::<CliContext>(),
         )
         .subcommand(
             "remove-before",
             from_fn_async(remove_before)
                 .no_display()
-                .with_about("Delete notifications preceding a given id")
+                .with_about("Remove notifications preceding a given id")
+                .with_call_remote::<CliContext>(),
+        )
+        .subcommand(
+            "mark-seen",
+            from_fn_async(mark_seen)
+                .no_display()
+                .with_about("Mark given notifications as seen")
+                .with_call_remote::<CliContext>(),
+        )
+        .subcommand(
+            "mark-seen-before",
+            from_fn_async(mark_seen_before)
+                .no_display()
+                .with_about("Mark notifications preceding a given id as seen")
+                .with_call_remote::<CliContext>(),
+        )
+        .subcommand(
+            "mark-unseen",
+            from_fn_async(mark_unseen)
+                .no_display()
+                .with_about("Mark given notifications as unseen")
                 .with_call_remote::<CliContext>(),
         )
         .subcommand(
@@ -55,7 +77,7 @@ pub fn notification<C: Context>() -> ParentHandler<C> {
 #[derive(Deserialize, Serialize, Parser, TS)]
 #[serde(rename_all = "camelCase")]
 #[command(rename_all = "kebab-case")]
-pub struct ListParams {
+pub struct ListNotificationParams {
     #[ts(type = "number | null")]
     before: Option<u32>,
     #[ts(type = "number | null")]
@@ -65,7 +87,7 @@ pub struct ListParams {
 #[instrument(skip_all)]
 pub async fn list(
     ctx: RpcContext,
-    ListParams { before, limit }: ListParams,
+    ListNotificationParams { before, limit }: ListNotificationParams,
 ) -> Result<Vec<NotificationWithId>, Error> {
     ctx.db
         .mutate(|db| {
@@ -122,39 +144,128 @@ pub async fn list(
 #[derive(Deserialize, Serialize, Parser, TS)]
 #[serde(rename_all = "camelCase")]
 #[command(rename_all = "kebab-case")]
-pub struct DeleteParams {
-    #[ts(type = "number")]
-    id: u32,
+pub struct ModifyNotificationParams {
+    #[ts(type = "number[]")]
+    ids: Vec<u32>,
 }
 
-pub async fn remove(ctx: RpcContext, DeleteParams { id }: DeleteParams) -> Result<(), Error> {
+pub async fn remove(
+    ctx: RpcContext,
+    ModifyNotificationParams { ids }: ModifyNotificationParams,
+) -> Result<(), Error> {
     ctx.db
         .mutate(|db| {
-            db.as_private_mut().as_notifications_mut().remove(&id)?;
+            let n = db.as_private_mut().as_notifications_mut();
+            for id in ids {
+                n.remove(&id)?;
+            }
             Ok(())
         })
         .await
         .result
 }
+
 #[derive(Deserialize, Serialize, Parser, TS)]
 #[serde(rename_all = "camelCase")]
 #[command(rename_all = "kebab-case")]
-pub struct DeleteBeforeParams {
+pub struct ModifyNotificationBeforeParams {
     #[ts(type = "number")]
     before: u32,
 }
 
 pub async fn remove_before(
     ctx: RpcContext,
-    DeleteBeforeParams { before }: DeleteBeforeParams,
+    ModifyNotificationBeforeParams { before }: ModifyNotificationBeforeParams,
 ) -> Result<(), Error> {
     ctx.db
         .mutate(|db| {
-            for id in db.as_private().as_notifications().keys()? {
-                if id < before {
-                    db.as_private_mut().as_notifications_mut().remove(&id)?;
+            let n = db.as_private_mut().as_notifications_mut();
+            for id in n.keys()?.range(..before) {
+                n.remove(&id)?;
+            }
+            Ok(())
+        })
+        .await
+        .result
+}
+
+pub async fn mark_seen(
+    ctx: RpcContext,
+    ModifyNotificationParams { ids }: ModifyNotificationParams,
+) -> Result<(), Error> {
+    ctx.db
+        .mutate(|db| {
+            let mut diff = 0;
+            let n = db.as_private_mut().as_notifications_mut();
+            for id in ids {
+                if !n
+                    .as_idx_mut(&id)
+                    .or_not_found(lazy_format!("Notification #{id}"))?
+                    .as_seen_mut()
+                    .replace(&true)?
+                {
+                    diff += 1;
                 }
             }
+            db.as_public_mut()
+                .as_server_info_mut()
+                .as_unread_notification_count_mut()
+                .mutate(|n| Ok(*n -= diff))?;
+            Ok(())
+        })
+        .await
+        .result
+}
+
+pub async fn mark_seen_before(
+    ctx: RpcContext,
+    ModifyNotificationBeforeParams { before }: ModifyNotificationBeforeParams,
+) -> Result<(), Error> {
+    ctx.db
+        .mutate(|db| {
+            let mut diff = 0;
+            let n = db.as_private_mut().as_notifications_mut();
+            for id in n.keys()?.range(..before) {
+                if !n
+                    .as_idx_mut(&id)
+                    .or_not_found(lazy_format!("Notification #{id}"))?
+                    .as_seen_mut()
+                    .replace(&true)?
+                {
+                    diff += 1;
+                }
+            }
+            db.as_public_mut()
+                .as_server_info_mut()
+                .as_unread_notification_count_mut()
+                .mutate(|n| Ok(*n -= diff))?;
+            Ok(())
+        })
+        .await
+        .result
+}
+
+pub async fn mark_unseen(
+    ctx: RpcContext,
+    ModifyNotificationParams { ids }: ModifyNotificationParams,
+) -> Result<(), Error> {
+    ctx.db
+        .mutate(|db| {
+            let mut diff = 0;
+            let n = db.as_private_mut().as_notifications_mut();
+            for id in ids {
+                if n.as_idx_mut(&id)
+                    .or_not_found(lazy_format!("Notification #{id}"))?
+                    .as_seen_mut()
+                    .replace(&false)?
+                {
+                    diff += 1;
+                }
+            }
+            db.as_public_mut()
+                .as_server_info_mut()
+                .as_unread_notification_count_mut()
+                .mutate(|n| Ok(*n += diff))?;
             Ok(())
         })
         .await
@@ -257,8 +368,9 @@ impl Map for Notifications {
     }
 }
 
-#[derive(Debug, Serialize, Deserialize)]
+#[derive(Debug, Serialize, Deserialize, HasModel)]
 #[serde(rename_all = "camelCase")]
+#[model = "Model<Self>"]
 pub struct Notification {
     pub package_id: Option<PackageId>,
     pub created_at: DateTime<Utc>,
@@ -267,6 +379,8 @@ pub struct Notification {
     pub title: String,
     pub message: String,
     pub data: Value,
+    #[serde(default = "const_true")]
+    pub seen: bool,
 }
 
 #[derive(Debug, Serialize, Deserialize)]
@@ -327,6 +441,7 @@ pub fn notify<T: NotificationType>(
             title,
             message,
             data,
+            seen: false,
         },
     )
 }

core/startos/src/os_install/gpt.rs

@@ -1,13 +1,12 @@
 use std::path::Path;
 
-use color_eyre::eyre::eyre;
 use gpt::disk::LogicalBlockSize;
 use gpt::GptConfig;
 
 use crate::disk::util::DiskInfo;
 use crate::disk::OsPartitionInfo;
 use crate::os_install::partition_for;
-use crate::Error;
+use crate::prelude::*;
 
 pub async fn partition(disk: &DiskInfo, overwrite: bool) -> Result<OsPartitionInfo, Error> {
     let efi = {
@@ -28,7 +27,6 @@ pub async fn partition(disk: &DiskInfo, overwrite: bool) -> Result<OsPartitionIn
                 (
                     GptConfig::new()
                         .writable(true)
-                        .initialized(false)
                         .logical_block_size(LogicalBlockSize::Lb512)
                         .create_from_device(device, None)?,
                     None,
@@ -36,7 +34,6 @@ pub async fn partition(disk: &DiskInfo, overwrite: bool) -> Result<OsPartitionIn
             } else {
                 let gpt = GptConfig::new()
                     .writable(true)
-                    .initialized(true)
                     .logical_block_size(LogicalBlockSize::Lb512)
                     .open_from_device(device)?;
                 let mut guid_part = None;
@@ -115,7 +112,12 @@ pub async fn partition(disk: &DiskInfo, overwrite: bool) -> Result<OsPartitionIn
                 )?;
             } else if let Some(guid_part) = guid_part {
                 let mut parts = gpt.partitions().clone();
-                parts.insert(gpt.find_next_partition_id(), guid_part);
+                parts.insert(
+                    gpt.find_next_partition_id().ok_or_else(|| {
+                        Error::new(eyre!("Partition table is full"), ErrorKind::DiskManagement)
+                    })?,
+                    guid_part,
+                );
                 gpt.update_partitions(parts)?;
             }