Feature/lxc container runtime (#2562)

* wip(fix): Dependencies

* wip: context

* wip(fix) Sorta auth

* wip: warnings

* wip(fix): registry/admin

* wip(fix) marketplace

* wip(fix) Some more converted and fixed with the linter and config

* wip: Working on the static server

* wip(fix)static server

* wip: Remove some asynnc

* wip: Something about the request and regular rpc

* wip: gut install

Co-authored-by: J H <Blu-J@users.noreply.github.com>

* wip: Convert the static server into the new system

* wip delete file

* test

* wip(fix) vhost does not need the with safe defaults

* wip: Adding in the wifi

* wip: Fix the developer and the verify

* wip: new install flow

Co-authored-by: J H <Blu-J@users.noreply.github.com>

* fix middleware

* wip

* wip: Fix the auth

* wip

* continue service refactor

* feature: Service get_config

* feat: Action

* wip: Fighting the great fight against the borrow checker

* wip: Remove an error in a file that I just need to deel with later

* chore: Add in some more lifetime stuff to the services

* wip: Install fix on lifetime

* cleanup

* wip: Deal with the borrow later

* more cleanup

* resolve borrowchecker errors

* wip(feat): add in the handler for the socket, for now

* wip(feat): Update the service_effect_handler::action

* chore: Add in the changes to make sure the from_service goes to context

* chore: Change the

* refactor service map

* fix references to service map

* fill out restore

* wip: Before I work on the store stuff

* fix backup module

* handle some warnings

* feat: add in the ui components on the rust side

* feature: Update the procedures

* chore: Update the js side of the main and a few of the others

* chore: Update the rpc listener to match the persistant container

* wip: Working on updating some things to have a better name

* wip(feat): Try and get the rpc to return the correct shape?

* lxc wip

* wip(feat): Try and get the rpc to return the correct shape?

* build for container runtime wip

* remove container-init

* fix build

* fix error

* chore: Update to work I suppose

* lxc wip

* remove docker module and feature

* download alpine squashfs automatically

* overlays effect

Co-authored-by: Jade <Blu-J@users.noreply.github.com>

* chore: Add the overlay effect

* feat: Add the mounter in the main

* chore: Convert to use the mounts, still need to work with the sandbox

* install fixes

* fix ssl

* fixes from testing

* implement tmpfile for upload

* wip

* misc fixes

* cleanup

* cleanup

* better progress reporting

* progress for sideload

* return real guid

* add devmode script

* fix lxc rootfs path

* fix percentage bar

* fix progress bar styling

* fix build for unstable

* tweaks

* label progress

* tweaks

* update progress more often

* make symlink in rpc_client

* make socket dir

* fix parent path

* add start-cli to container

* add echo and gitInfo commands

* wip: Add the init + errors

* chore: Add in the exit effect for the system

* chore: Change the type to null for failure to parse

* move sigterm timeout to stopping status

* update order

* chore: Update the return type

* remove dbg

* change the map error

* chore: Update the thing to capture id

* chore add some life changes

* chore: Update the loging

* chore: Update the package to run module

* us From for RpcError

* chore: Update to use import instead

* chore: update

* chore: Use require for the backup

* fix a default

* update the type that is wrong

* chore: Update the type of the manifest

* chore: Update to make null

* only symlink if not exists

* get rid of double result

* better debug info for ErrorCollection

* chore: Update effects

* chore: fix

* mount assets and volumes

* add exec instead of spawn

* fix mounting in image

* fix overlay mounts

Co-authored-by: Jade <Blu-J@users.noreply.github.com>

* misc fixes

* feat: Fix two

* fix: systemForEmbassy main

* chore: Fix small part of main loop

* chore: Modify the bundle

* merge

* fixMain loop"

* move tsc to makefile

* chore: Update the return types of the health check

* fix client

* chore: Convert the todo to use tsmatches

* add in the fixes for the seen and create the hack to allow demo

* chore: Update to include the systemForStartOs

* chore UPdate to the latest types from the expected outout

* fixes

* fix typo

* Don't emit if failure on tsc

* wip

Co-authored-by: Jade <Blu-J@users.noreply.github.com>

* add s9pk api

* add inspection

* add inspect manifest

* newline after display serializable

* fix squashfs in image name

* edit manifest

Co-authored-by: Jade <Blu-J@users.noreply.github.com>

* wait for response on repl

* ignore sig for now

* ignore sig for now

* re-enable sig verification

* fix

* wip

* env and chroot

* add profiling logs

* set uid & gid in squashfs to 100000

* set uid of sqfs to 100000

* fix mksquashfs args

* add env to compat

* fix

* re-add docker feature flag

* fix docker output format being stupid

* here be dragons

* chore: Add in the cross compiling for something

* fix npm link

* extract logs from container on exit

* chore: Update for testing

* add log capture to drop trait

* chore: add in the modifications that I make

* chore: Update small things for no updates

* chore: Update the types of something

* chore: Make main not complain

* idmapped mounts

* idmapped volumes

* re-enable kiosk

* chore: Add in some logging for the new system

* bring in start-sdk

* remove avahi

* chore: Update the deps

* switch to musl

* chore: Update the version of prettier

* chore: Organize'

* chore: Update some of the headers back to the standard of fetch

* fix musl build

* fix idmapped mounts

* fix cross build

* use cross compiler for correct arch

* feat: Add in the faked ssl stuff for the effects

* @dr_bonez Did a solution here

* chore: Something that DrBonez

* chore: up

* wip: We have a working server!!!

* wip

* uninstall

* wip

* tes

* misc fixes

* fix cli

* replace interface with host

* chore: Fix the types in some ts files

* chore: quick update for the system for embassy to update the types

* replace br-start9 with lxcbr0

* split patchdb into public/private

* chore: Add changes for config set

* Feat: Adding some debugging for the errors

* wip: Working on getting the set config to work

* chore: Update and fix the small issue with the deserialization

* lightning, masked, schemeOverride, invert host-iface relationship

* feat: Add in the changes for just the sdk

* feat: Add in the changes for the new effects I suppose for now

---------

Co-authored-by: J H <2364004+Blu-J@users.noreply.github.com>
Co-authored-by: J H <Blu-J@users.noreply.github.com>
Co-authored-by: J H <dragondef@gmail.com>
Co-authored-by: Matt Hill <mattnine@protonmail.com>

sdk/lib/util/getServiceInterface.ts

@@ -1,4 +1,11 @@
-import { AddressInfo, Effects, Hostname, ServiceInterface } from "../types"
+import {
+  AddressInfo,
+  Effects,
+  HostInfo,
+  Hostname,
+  HostnameInfo,
+  ServiceInterface,
+} from "../types"
 import * as regexes from "./regexes"
 import { ServiceInterfaceType } from "./utils"
 
@@ -22,7 +29,6 @@ export type Filled = {
   ipv4Hostnames: Hostname[]
   ipv6Hostnames: Hostname[]
   nonIpHostnames: Hostname[]
-  allHostnames: Hostname[]
 
   urls: UrlString[]
   onionUrls: UrlString[]
@@ -31,7 +37,6 @@ export type Filled = {
   ipv4Urls: UrlString[]
   ipv6Urls: UrlString[]
   nonIpUrls: UrlString[]
-  allUrls: UrlString[]
 }
 export type FilledAddressInfo = AddressInfo & Filled
 export type ServiceInterfaceFilled = {
@@ -44,6 +49,10 @@ export type ServiceInterfaceFilled = {
   hasPrimary: boolean
   /** Whether or not the interface disabled */
   disabled: boolean
+  /** Whether or not to mask the URIs for this interface. Useful if the URIs contain sensitive information, such as a password, macaroon, or API key */
+  masked: boolean
+  /** Information about the host for this binding */
+  hostInfo: HostInfo
   /** URI information */
   addressInfo: FilledAddressInfo
   /** Indicates if we are a ui/p2p/api for the kind of interface that this is representing */
@@ -62,75 +71,110 @@ const negate =
   (a: A) =>
     !fn(a)
 const unique = <A>(values: A[]) => Array.from(new Set(values))
+function stringifyHostname(info: HostnameInfo): Hostname {
+  let base: string
+  if ("kind" in info.hostname && info.hostname.kind === "domain") {
+    base = info.hostname.subdomain
+      ? `${info.hostname.subdomain}.${info.hostname.domain}`
+      : info.hostname.domain
+  } else {
+    base = info.hostname.value
+  }
+  if (info.hostname.port && info.hostname.sslPort) {
+    return `${base}:${info.hostname.port}` as Hostname
+  } else if (info.hostname.sslPort) {
+    return `${base}:${info.hostname.sslPort}` as Hostname
+  } else if (info.hostname.port) {
+    return `${base}:${info.hostname.port}` as Hostname
+  }
+  return base as Hostname
+}
 const addressHostToUrl = (
-  { options, username, suffix }: AddressInfo,
+  { bindOptions, username, suffix }: AddressInfo,
   host: Hostname,
 ): UrlString => {
   const scheme = host.endsWith(".onion")
-    ? options.scheme
-    : options.addSsl
-      ? options.addSsl.scheme
-      : options.scheme // TODO: encode whether hostname transport is "secure"?
+    ? bindOptions.scheme
+    : bindOptions.addSsl
+      ? bindOptions.addSsl.scheme
+      : bindOptions.scheme // TODO: encode whether hostname transport is "secure"?
   return `${scheme ? `${scheme}//` : ""}${
     username ? `${username}@` : ""
   }${host}${suffix}`
 }
 export const filledAddress = (
-  hostnames: Hostname[],
+  hostInfo: HostInfo,
   addressInfo: AddressInfo,
 ): FilledAddressInfo => {
   const toUrl = addressHostToUrl.bind(null, addressInfo)
+  const hostnameInfo =
+    hostInfo.kind == "multi"
+      ? hostInfo.hostnames
+      : hostInfo.hostname
+        ? [hostInfo.hostname]
+        : []
   return {
     ...addressInfo,
-    hostnames,
+    hostnames: hostnameInfo.flatMap((h) => stringifyHostname(h)),
     get onionHostnames() {
-      return hostnames.filter(regexes.torHostname.test)
+      return hostnameInfo
+        .filter((h) => h.kind === "onion")
+        .map((h) => stringifyHostname(h))
     },
     get localHostnames() {
-      return hostnames.filter(regexes.localHostname.test)
+      return hostnameInfo
+        .filter((h) => h.kind === "ip" && h.hostname.kind === "local")
+        .map((h) => stringifyHostname(h))
     },
     get ipHostnames() {
-      return hostnames.filter(either(regexes.ipv4.test, regexes.ipv6.test))
+      return hostnameInfo
+        .filter(
+          (h) =>
+            h.kind === "ip" &&
+            (h.hostname.kind === "ipv4" || h.hostname.kind === "ipv6"),
+        )
+        .map((h) => stringifyHostname(h))
     },
     get ipv4Hostnames() {
-      return hostnames.filter(regexes.ipv4.test)
+      return hostnameInfo
+        .filter((h) => h.kind === "ip" && h.hostname.kind === "ipv4")
+        .map((h) => stringifyHostname(h))
     },
     get ipv6Hostnames() {
-      return hostnames.filter(regexes.ipv6.test)
+      return hostnameInfo
+        .filter((h) => h.kind === "ip" && h.hostname.kind === "ipv6")
+        .map((h) => stringifyHostname(h))
     },
     get nonIpHostnames() {
-      return hostnames.filter(
-        negate(either(regexes.ipv4.test, regexes.ipv6.test)),
-      )
+      return hostnameInfo
+        .filter(
+          (h) =>
+            h.kind === "ip" &&
+            h.hostname.kind !== "ipv4" &&
+            h.hostname.kind !== "ipv6",
+        )
+        .map((h) => stringifyHostname(h))
     },
-    allHostnames: hostnames,
     get urls() {
-      return hostnames.map(toUrl)
+      return this.hostnames.map(toUrl)
     },
     get onionUrls() {
-      return hostnames.filter(regexes.torHostname.test).map(toUrl)
+      return this.onionHostnames.map(toUrl)
     },
     get localUrls() {
-      return hostnames.filter(regexes.localHostname.test).map(toUrl)
+      return this.localHostnames.map(toUrl)
     },
     get ipUrls() {
-      return hostnames
-        .filter(either(regexes.ipv4.test, regexes.ipv6.test))
-        .map(toUrl)
+      return this.ipHostnames.map(toUrl)
     },
     get ipv4Urls() {
-      return hostnames.filter(regexes.ipv4.test).map(toUrl)
+      return this.ipv4Hostnames.map(toUrl)
     },
     get ipv6Urls() {
-      return hostnames.filter(regexes.ipv6.test).map(toUrl)
+      return this.ipv6Hostnames.map(toUrl)
     },
     get nonIpUrls() {
-      return hostnames
-        .filter(negate(either(regexes.ipv4.test, regexes.ipv6.test)))
-        .map(toUrl)
-    },
-    get allUrls() {
-      return hostnames.map(toUrl)
+      return this.nonIpHostnames.map(toUrl)
     },
   }
 }
@@ -151,9 +195,9 @@ const makeInterfaceFilled = async ({
     packageId,
     callback,
   })
-  const hostIdRecord = await effects.getHostnames({
+  const hostInfo = await effects.getHostInfo({
     packageId,
-    hostId: serviceInterfaceValue.addressInfo.hostId,
+    serviceInterfaceId: serviceInterfaceValue.id,
     callback,
   })
   const primaryUrl = await effects.getPrimaryUrl({
@@ -165,7 +209,8 @@ const makeInterfaceFilled = async ({
   const interfaceFilled: ServiceInterfaceFilled = {
     ...serviceInterfaceValue,
     primaryUrl: primaryUrl,
-    addressInfo: filledAddress(hostIdRecord, serviceInterfaceValue.addressInfo),
+    hostInfo,
+    addressInfo: filledAddress(hostInfo, serviceInterfaceValue.addressInfo),
     get primaryHostname() {
       if (primaryUrl == null) return null
       return getHostname(primaryUrl)

sdk/lib/util/getServiceInterfaces.ts

@@ -20,19 +20,13 @@ const makeManyInterfaceFilled = async ({
   })
   const hostIdsRecord = Object.fromEntries(
     await Promise.all(
-      Array.from(
-        new Set(
-          serviceInterfaceValues
-            .flatMap((x) => x.addressInfo)
-            .map((x) => x.hostId),
-        ),
-      ).map(
-        async (hostId) =>
+      Array.from(new Set(serviceInterfaceValues.map((x) => x.id))).map(
+        async (id) =>
           [
-            hostId,
-            await effects.getHostnames({
+            id,
+            await effects.getHostInfo({
               packageId,
-              hostId,
+              serviceInterfaceId: id,
               callback,
             }),
           ] as const,
@@ -42,9 +36,9 @@ const makeManyInterfaceFilled = async ({
 
   const serviceInterfacesFilled: ServiceInterfaceFilled[] = await Promise.all(
     serviceInterfaceValues.map(async (serviceInterfaceValue) => {
-      const hostIdRecord = await effects.getHostnames({
+      const hostInfo = await effects.getHostInfo({
         packageId,
-        hostId: serviceInterfaceValue.addressInfo.hostId,
+        serviceInterfaceId: serviceInterfaceValue.id,
         callback,
       })
       const primaryUrl = await effects.getPrimaryUrl({
@@ -55,10 +49,8 @@ const makeManyInterfaceFilled = async ({
       return {
         ...serviceInterfaceValue,
         primaryUrl: primaryUrl,
-        addressInfo: filledAddress(
-          hostIdRecord,
-          serviceInterfaceValue.addressInfo,
-        ),
+        hostInfo,
+        addressInfo: filledAddress(hostInfo, serviceInterfaceValue.addressInfo),
         get primaryHostname() {
           if (primaryUrl == null) return null
           return getHostname(primaryUrl)

sdk/lib/util/utils.ts

@@ -25,7 +25,7 @@ import {
   NamedPath,
   Path,
 } from "../dependency/setupDependencyMounts"
-import { MultiHost, SingleHost, StaticHost } from "../interfaces/Host"
+import { MultiHost, Scheme, SingleHost, StaticHost } from "../interfaces/Host"
 import { ServiceInterfaceBuilder } from "../interfaces/ServiceInterfaceBuilder"
 import { GetServiceInterface, getServiceInterface } from "./getServiceInterface"
 import {
@@ -83,6 +83,8 @@ export type Utils<
     username: null | string
     path: string
     search: Record<string, string>
+    schemeOverride: { ssl: Scheme; noSsl: Scheme } | null
+    masked: boolean
   }) => ServiceInterfaceBuilder
   getSystemSmtp: () => GetSystemSmtp & WrapperOverWrite
   host: {
@@ -158,6 +160,8 @@ export const createUtils = <
       username: null | string
       path: string
       search: Record<string, string>
+      schemeOverride: { ssl: Scheme; noSsl: Scheme } | null
+      masked: boolean
     }) => new ServiceInterfaceBuilder({ ...options, effects }),
     childProcess,
     getSystemSmtp: () =>