Start9/start-os
2
0
Fork 0
mirror of https://github.com/Start9Labs/start-os.git synced 2026-04-04 14:29:45 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: support preferred external ports besides 443 (#3117)

Browse Source 
* docs: update preferred external port design in TODO

* docs: add user-controlled public/private and port forward mapping to design

* docs: overhaul interfaces page design with view/manage split and per-address controls

* docs: move address enable/disable to overflow menu, add SSL indicator, defer UI placement decisions

* chore: remove tor from startos core

Tor is being moved from a built-in OS feature to a service. This removes
the Arti-based Tor client, onion address management, hidden service
creation, and all related code from the core backend, frontend, and SDK.

- Delete core/src/net/tor/ module (~2060 lines)
- Remove OnionAddress, TorSecretKey, TorController from all consumers
- Remove HostnameInfo::Onion and HostAddress::Onion variants
- Remove onion CRUD RPC endpoints and tor subcommand
- Remove tor key handling from account and backup/restore
- Remove ~12 tor-related Cargo dependencies (arti-client, torut, etc.)
- Remove tor UI components, API methods, mock data, and routes
- Remove OnionHostname and tor patterns/regexes from SDK
- Add v0_4_0_alpha_20 database migration to strip onion data
- Bump version to 0.4.0-alpha.20

* chore: flatten HostnameInfo from enum to struct

HostnameInfo only had one variant (Ip) after removing Tor. Flatten it
into a plain struct with fields gateway, public, hostname. Remove all
kind === 'ip' type guards and narrowing across SDK, frontend, and
container runtime. Update DB migration to strip the kind field.

* chore: format RPCSpec.md markdown table

* docs: update TODO.md with DerivedAddressInfo design, remove completed tor task

* feat: implement preferred port allocation and per-address enable/disable

- Add AvailablePorts::try_alloc() with SSL tracking (BTreeMap<u16, bool>)
- Add DerivedAddressInfo on BindInfo with private_disabled/public_enabled/possible sets
- Add Bindings wrapper with Map impl for patchdb indexed access
- Flatten HostAddress from single-variant enum to struct
- Replace set-gateway-enabled RPC with set-address-enabled
- Remove hostname_info from Host; computed addresses now in BindInfo.addresses.possible
- Compute possible addresses inline in NetServiceData::update()
- Update DB migration, SDK types, frontend, and container-runtime

* feat: replace InterfaceFilter with ForwardRequirements, add WildcardListener, complete alpha.20 bump

- Replace DynInterfaceFilter with ForwardRequirements for per-IP forward
  precision with source-subnet iptables filtering for private forwards
- Add WildcardListener (binds [::]:port) to replace the per-gateway
  NetworkInterfaceListener/SelfContainedNetworkInterfaceListener/
  UpgradableListener infrastructure
- Update forward-port script with src_subnet and excluded_src env vars
- Remove unused filter types and listener infrastructure from gateway.rs
- Add availablePorts migration (IdPool -> BTreeMap<u16, bool>) to alpha.20
- Complete version bump to 0.4.0-alpha.20 in SDK and web

* outbound gateway support (#3120)

* Multiple (#3111)

* fix alerts i18n, fix status display, better, remove usb media, hide shutdown for install complete

* trigger chnage detection for localize pipe and round out implementing localize pipe for consistency even though not needed

* Fix PackageInfoShort to handle LocaleString on releaseNotes (#3112)

* Fix PackageInfoShort to handle LocaleString on releaseNotes

* fix: filter by target_version in get_matching_models and pass otherVersions from install

* chore: add exver documentation for ai agents

* frontend plus some be types

---------

Co-authored-by: Aiden McClelland <3732071+dr-bonez@users.noreply.github.com>

* feat: replace SourceFilter with IpNet, add policy routing, remove MASQUERADE

* build ts types and fix i18n

* fix license display in marketplace

* wip refactor

* chore: update ts bindings for preferred port design

* feat: refactor NetService to watch DB and reconcile network state

- NetService sync task now uses PatchDB DbWatch instead of being called
  directly after DB mutations
- Read gateways from DB instead of network interface context when
  updating host addresses
- gateway sync updates all host addresses in the DB
- Add Watch<u64> channel for callers to wait on sync completion
- Fix ts-rs codegen bug with #[ts(skip)] on flattened Plugin field
- Update SDK getServiceInterface.ts for new HostnameInfo shape
- Remove unnecessary HTTPS redirect in static_server.rs
- Fix tunnel/api.rs to filter for WAN IPv4 address

* re-arrange (#3123)

* new service interfacee page

* feat: add mdns hostname metadata variant and fix vhost routing

- Add HostnameMetadata::Mdns variant to distinguish mDNS from private domains
- Mark mDNS addresses as private (public: false) since mDNS is local-only
- Fall back to null SNI entry when hostname not found in vhost mapping
- Simplify public detection in ProxyTarget filter
- Pass hostname to update_addresses for mDNS domain name generation

* looking good

* feat: add port_forwards field to Host for tracking gateway forwarding rules

* update bindings for API types, add ARCHITECTURE (#3124)

* update binding for API types, add ARCHITECTURE

* translations

* fix: add CONNMARK restore-mark to mangle OUTPUT chain

The CONNMARK --restore-mark rule was only in PREROUTING, which handles
forwarded packets. Locally-bound listeners (e.g. vhost) generate replies
through the OUTPUT chain, where the fwmark was never restored. This
caused response packets to route via the default table instead of back
through the originating interface.

* chore: reserialize db on equal version, update bindings and docs

- Run de/ser roundtrip in pre_init even when db version matches, ensuring
  all #[serde(default)] fields are populated before any typed access
- Add patchdb.md documentation for TypedDbWatch patterns
- Update TS bindings for CheckPortParams, CheckPortRes, ifconfigUrl
- Update CLAUDE.md docs with patchdb and component-level references

* fix: include public gateways for IP-based addresses in vhost targets

The server hostname vhost construction only collected private IPs,
always setting public to empty. Public IP addresses (Ipv4/Ipv6 metadata
with public=true) were never added to the vhost target's public gateway
set, causing the vhost filter to reject public traffic for IP-based
addresses.

* fix: add TLS handshake timeout and fix accept loop deadlock

Two issues in TlsListener::poll_accept:

1. No timeout on TLS handshakes: LazyConfigAcceptor waits indefinitely
   for ClientHello. Attackers that complete TCP handshake but never send
   TLS data create zombie futures in `in_progress` that never complete.
   Fix: wrap the entire handshake in tokio::time::timeout(15s).

2. Missing waker on new-connection pending path: when a TCP connection
   is accepted and the TLS handshake is pending, poll_accept returned
   Pending without calling wake_by_ref(). Since the TcpListener returned
   Ready (not Pending), no waker was registered for it. With edge-
   triggered epoll and no other wakeup source, the task sleeps forever
   and remaining connections in the kernel accept queue are never
   drained. Fix: add cx.waker().wake_by_ref() so the task immediately
   re-polls and continues draining the accept queue.

* fix: switch BackgroundJobRunner from Vec to FuturesUnordered

BackgroundJobRunner stored active jobs in a Vec<BoxFuture> and polled
ALL of them on every wakeup — O(n) per poll. Since this runs in the
same tokio::select! as the WebServer accept loop, polling overhead from
active connections directly delayed acceptance of new connections.

FuturesUnordered only polls woken futures — O(woken) instead of O(n).

* chore: update bindings and use typed params for outbound gateway API

* feat: per-service and default outbound gateway routing

Add set-outbound-gateway RPC for packages and set-default-outbound RPC
for the server, with policy routing enforcement via ip rules. Fix
connmark restore to skip packets with existing fwmarks, add bridge
subnet routes to per-interface tables, and fix squashfs path in
update-image-local.sh.

* refactor: manifest wraps PackageMetadata, move dependency_metadata to PackageVersionInfo

Manifest now embeds PackageMetadata via #[serde(flatten)] instead of
duplicating ~14 fields. icon and dependency_metadata moved from
PackageMetadata to PackageVersionInfo since they are registry-enrichment
data loaded from the S9PK archive. merge_with now returns errors on
metadata/icon/dependency_metadata mismatches instead of silently ignoring
them.

* fix: replace .status() with .invoke() for iptables/ip commands

Using .status() leaks stderr directly to system logs, causing noisy
iptables error messages. Switch all networking CLI invocations to use
.invoke() which captures stderr properly. For check-then-act patterns
(iptables -C), use .invoke().await.is_err() instead of
.status().await.map_or(false, |s| s.success()).

* feat: add check-dns gateway endpoint and fix per-interface routing tables

Add a `check-dns` RPC endpoint that verifies whether a gateway's DNS
is properly configured for private domain resolution. Uses a three-tier
check: direct match (DNS == server IP), TXT challenge probe (DNS on
LAN), or failure (DNS off-subnet).

Fix per-interface routing tables to clone all non-default routes from
the main table instead of only the interface's own subnets. This
preserves LAN reachability when the priority-75 catch-all overrides
default routing. Filter out status-only flags (linkdown, dead) that
are invalid for `ip route add`.

* refactor: rename manifest metadata fields and improve error display

Rename wrapperRepo→packageRepo, marketingSite→marketingUrl,
docsUrl→docsUrls (array), remove supportSite. Add display_src/display_dbg
helpers to Error. Fix DepInfo description type to LocaleString. Update
web UI, SDK bindings, tests, and fixtures to match. Clean up cli_attach
error handling and remove dead commented code.

* chore: bump sdk version to 0.4.0-beta.49

* chore: add createTask decoupling TODO

* chore: add TODO to clear service error state on install/update

* round out dns check, dns server check, port forward check, and gateway port forwards

* chore: add TODOs for URL plugins, NAT hairpinning, and start-tunnel OTA updates

* version instead of os query param

* interface row clickable again, bu now with a chevron!

* feat: implement URL plugins with table/row actions and prefill support

- Add URL plugin effects (register, export_url, clear_urls) in core
- Add PluginHostnameInfo, HostnameMetadata::Plugin, and plugin registration types
- Implement plugin URL table in web UI with tableAction button and rowAction overflow menus
- Thread urlPluginMetadata (packageId, hostId, interfaceId, internalPort) as prefill to actions
- Add prefill support to PackageActionData so metadata passes through form dialogs
- Add i18n translations for plugin error messages
- Clean up plugin URLs on package uninstall

* feat: split row_actions into remove_action and overflow_actions for URL plugins

* touch up URL plugins table

* show table even when no addresses

* feat: NAT hairpinning, DNS static servers, clear service error on install

- Add POSTROUTING MASQUERADE rules for container and host hairpin NAT
- Allow bridge subnet containers to reach private forwards via LAN IPs
- Pass bridge_subnet env var from forward.rs to forward-port script
- Use DB-configured static DNS servers in resolver with DB watcher
- Fall back to resolv.conf servers when no static servers configured
- Clear service error state when install/update completes successfully
- Remove completed TODO items

* feat: builder-style InputSpec API, prefill plumbing, and port forward fix

- Add addKey() and add() builder methods to InputSpec with InputSpecTools
- Move OuterType to last generic param on Value, List, and all dynamic methods
- Plumb prefill through getActionInput end-to-end (core → container-runtime → SDK)
- Filter port_forwards to enabled addresses only
- Bump SDK to 0.4.0-beta.50

* fix: propagate host locale into LXC containers and write locale.conf

* chore: remove completed URL plugins TODO

* feat: OTA updates for start-tunnel via apt repository (untested)

- Add apt repo publish script (build/apt/publish-deb.sh) for S3-hosted repo
- Add apt source config and GPG key placeholder (apt/)
- Add tunnel.update.check and tunnel.update.apply RPC endpoints
- Wire up update API in tunnel frontend (api service + mock)
- Uses systemd-run --scope to survive service restart during update

* fix: publish script dpkg-name, s3cfg fallback, and --reinstall for apply

* chore: replace OTA updates TODO with UI TODO for MattDHill

* feat: add getOutboundGateway effect and simplify VersionGraph init/uninit

Add getOutboundGateway effect across core, container-runtime, and SDK
to let services query their effective outbound gateway with callback
support. Remove preInstall/uninstall hooks from VersionGraph as they
are no longer needed.

* frontend start-tunnel updates

* chore: remove completed TODO

* feat: tor hidden service key migration

* chore: migrate from ts-matches to zod across all TypeScript packages

* feat(core): allow setting server hostname

* send prefill for tasks and hide operations to hidden fields

* fix(core): preserve plugin URLs across binding updates

BindInfo::update was replacing addresses with a new DerivedAddressInfo
that cleared the available set, wiping plugin-exported URLs whenever
bind() was called. Also simplify update_addresses plugin preservation
to use retain in place rather than collecting into a separate set.

* minor cleanup from patch-db audit

* clean up prefill flow

* frontend support for setting and changing hostname

* feat(core): refactor hostname to ServerHostnameInfo with name/hostname pair

- Rename Hostname to ServerHostnameInfo, add name + hostname fields
- Add set_hostname_rpc for changing hostname at runtime
- Migrate alpha_20: generate serverInfo.name from hostname, delete ui.name
- Extract gateway.rs helpers to fix rustfmt nesting depth issue
- Add i18n key for hostname validation error
- Update SDK bindings

* add comments to everything potentially consumer facing (#3127)

* add comments to everything potentially consumer facing

* rework smtp

---------

Co-authored-by: Aiden McClelland <3732071+dr-bonez@users.noreply.github.com>

* implement server name

* setup changes

* clean up copy around addresses table

* feat: add zod-deep-partial, partialValidator on InputSpec, and z.deepPartial re-export

* fix: header color in zoom (#3128)

* fix: merge version ranges when adding existing package signer (#3125)

* fix: merge version ranges when adding existing package signer

   Previously, add_package_signer unconditionally inserted the new
   version range, overwriting any existing authorization for that signer.
   Now it OR-merges the new range with the existing one, so running
   signer add multiple times accumulates permissions rather than
   replacing them.

* add --merge flag to registry package signer add

  Default behavior remains overwrite. When --merge is passed, the new
  version range is OR-merged with the existing one, allowing admins to
  accumulate permissions incrementally.

* add missing attribute to TS type

* make merge optional

* upsert instead of insert

* VersionRange::None on upsert

* fix: header color in zoom

---------

Co-authored-by: Dominion5254 <musashidisciple@proton.me>

* update snake and add about this server to system general

* chore: bump sdk to beta.53, wrap z.deepPartial with passthrough

* reset instead of reset defaults

* action failure show dialog

* chore: bump sdk to beta.54, add device-info RPC, improve SDK abort handling and InputSpec filtering

- Bump SDK version to 0.4.0-beta.54
- Add `server.device-info` RPC endpoint and `s9pk select` CLI command
- Extract `HardwareRequirements::is_compatible()` method, reuse in registry filtering
- Add `AbortedError` class with `muteUnhandled` flag, replace generic abort errors
- Handle unhandled promise rejections in container-runtime with mute support
- Improve `InputSpec.filter()` with `keepByDefault` param and boolean filter values
- Accept readonly tuples in `CommandType` and `splitCommand`
- Remove `sync_host` calls from host API handlers (binding/address changes)
- Filter mDNS hostnames by secure gateway availability
- Derive mDNS enabled state from LAN IPs in web UI
- Add "Open UI" action to address table, disable mDNS toggle
- Hide debug details in service error component
- Update rpc-toolkit docs for no-params handlers

* fix: add --no-nvram to efi grub-install to preserve built-in boot order

* update snake

* diable actions when in error state

* chore: split out nvidia variant

* misc bugfixes

* create manage-release script (untested)

* fix: preserve z namespace types for sdk consumers

* sdk version bump

* new checkPort types

* multiple bugs and better port forward ux

* fix link

* chore: todos and formatting

* fix build

---------

Co-authored-by: Matt Hill <MattDHill@users.noreply.github.com>
Co-authored-by: Matt Hill <mattnine@protonmail.com>
Co-authored-by: Alex Inkin <alexander@inkin.ru>
Co-authored-by: Dominion5254 <musashidisciple@proton.me>
This commit is contained in:
Aiden McClelland
2026-03-04 04:37:31 -07:00
committed by GitHub
parent 26a68afdef
commit 3320391fcc
532 changed files with 21594 additions and 20559 deletions
Download Patch File Download Diff File Expand all files Collapse all files

72
core/ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,72 @@
# Core Architecture
The Rust backend daemon for StartOS.
## Binaries
The crate produces a single binary `startbox` that is symlinked under different names for different behavior:
- `startbox` / `startd` — Main daemon
- `start-cli` — CLI interface
- `start-container` — Runs inside LXC containers; communicates with host and manages subcontainers
- `registrybox` — Registry daemon
- `tunnelbox` — VPN/tunnel daemon
## Crate Structure
- `startos` — Core library that supports building `startbox`
- `helpers` — Utility functions used across both `startos` and `js-engine`
- `models` — Types shared across `startos`, `js-engine`, and `helpers`
## Key Modules
- `src/context/` — Context types (RpcContext, CliContext, InitContext, DiagnosticContext)
- `src/service/` — Service lifecycle management with actor pattern (`service_actor.rs`)
- `src/db/model/` — Patch-DB models (`public.rs` synced to frontend, `private.rs` backend-only)
- `src/net/` — Networking (DNS, ACME, WiFi, Tor via Arti, WireGuard)
- `src/s9pk/` — S9PK package format (merkle archive)
- `src/registry/` — Package registry management
## RPC Pattern
The API is JSON-RPC (not REST). All endpoints are RPC methods organized in a hierarchical command structure using [rpc-toolkit](https://github.com/Start9Labs/rpc-toolkit). Handlers are registered in a tree of `ParentHandler` nodes, with four handler types: `from_fn_async` (standard), `from_fn_async_local` (non-Send), `from_fn` (sync), and `from_fn_blocking` (blocking). Metadata like `.with_about()` drives middleware and documentation.
See [rpc-toolkit.md](rpc-toolkit.md) for full handler patterns and configuration.
## Patch-DB Patterns
Patch-DB provides diff-based state synchronization. Changes to `db/model/public.rs` automatically sync to the frontend.
**Key patterns:**
- `db.peek().await` — Get a read-only snapshot of the database state
- `db.mutate(|db| { ... }).await` — Apply mutations atomically, returns `MutateResult`
- `#[derive(HasModel)]` — Derive macro for types stored in the database, generates typed accessors
**Generated accessor types** (from `HasModel` derive):
- `as_field()` — Immutable reference: `&Model<T>`
- `as_field_mut()` — Mutable reference: `&mut Model<T>`
- `into_field()` — Owned value: `Model<T>`
**`Model<T>` APIs** (from `db/prelude.rs`):
- `.de()` — Deserialize to `T`
- `.ser(&value)` — Serialize from `T`
- `.mutate(|v| ...)` — Deserialize, mutate, reserialize
- For maps: `.keys()`, `.as_idx(&key)`, `.as_idx_mut(&key)`, `.insert()`, `.remove()`, `.contains_key()`
See [patchdb.md](patchdb.md) for `TypedDbWatch<T>` construction, API, and usage patterns.
## i18n
See [i18n-patterns.md](i18n-patterns.md) for internationalization key conventions and the `t!()` macro.
## Rust Utilities & Patterns
See [core-rust-patterns.md](core-rust-patterns.md) for common utilities (Invoke trait, Guard pattern, mount guards, Apply trait, etc.).
## Related Documentation
- [rpc-toolkit.md](rpc-toolkit.md) — JSON-RPC handler patterns
- [patchdb.md](patchdb.md) — Patch-DB watch patterns and TypedDbWatch
- [i18n-patterns.md](i18n-patterns.md) — Internationalization conventions
- [core-rust-patterns.md](core-rust-patterns.md) — Common Rust utilities
- [s9pk-structure.md](s9pk-structure.md) — S9PK package format

27
core/CLAUDE.md Normal file
View File

@@ -0,0 +1,27 @@
# Core — Rust Backend
The Rust backend daemon for StartOS.
## Architecture
See [ARCHITECTURE.md](ARCHITECTURE.md) for binaries, modules, Patch-DB patterns, and related documentation.
See [CONTRIBUTING.md](CONTRIBUTING.md) for how to add RPC endpoints, TS-exported types, and i18n keys.
## Quick Reference
```bash
cargo check -p start-os # Type check
make test-core # Run tests
make ts-bindings # Regenerate TS types after changing #[ts(export)] structs
cd sdk && make baseDist dist # Rebuild SDK after ts-bindings
```
## Operating Rules
- Always run `cargo check -p start-os` after modifying Rust code
- When adding RPC endpoints, follow the patterns in [rpc-toolkit.md](rpc-toolkit.md)
- When modifying `#[ts(export)]` types, regenerate bindings and rebuild the SDK (see [ARCHITECTURE.md](../ARCHITECTURE.md#build-pipeline))
- When adding i18n keys, add all 5 locales in `core/locales/i18n.yaml` (see [i18n-patterns.md](i18n-patterns.md))
- When using DB watches, follow the `TypedDbWatch<T>` patterns in [patchdb.md](patchdb.md)
- **Always use `.invoke(ErrorKind::...)` instead of `.status()` when running CLI commands** via `tokio::process::Command`. The `Invoke` trait (from `crate::util::Invoke`) captures stdout/stderr and checks exit codes properly. Using `.status()` leaks stderr directly to system logs, creating noise. For check-then-act patterns (e.g. `iptables -C`), use `.invoke(...).await.is_ok()` / `.is_err()` instead of `.status().await.map_or(false, |s| s.success())`.

49
core/CONTRIBUTING.md Normal file
View File

@@ -0,0 +1,49 @@
# Contributing to Core
For general environment setup, cloning, and build system, see the root [CONTRIBUTING.md](../CONTRIBUTING.md).
## Prerequisites
- [Rust](https://rustup.rs) (nightly for formatting)
- [rust-analyzer](https://rust-analyzer.github.io/) recommended
- [Docker](https://docs.docker.com/get-docker/) (for cross-compilation via `rust-zig-builder` container)
## Common Commands
```bash
cargo check -p start-os # Type check
cargo test --features=test # Run tests (or: make test-core)
make format # Format with nightly rustfmt
cd core && cargo test <test_name> --features=test # Run a specific test
```
## Adding a New RPC Endpoint
1. Define a params struct with `#[derive(Deserialize, Serialize)]`
2. Choose a handler type (`from_fn_async` for most cases)
3. Write the handler function: `async fn my_handler(ctx: RpcContext, params: MyParams) -> Result<MyResponse, Error>`
4. Register it in the appropriate `ParentHandler` tree
5. If params/response should be available in TypeScript, add `#[derive(TS)]` and `#[ts(export)]`
See [rpc-toolkit.md](rpc-toolkit.md) for full handler patterns and all four handler types.
## Adding TS-Exported Types
When a Rust type needs to be available in TypeScript (for the web frontend or SDK):
1. Add `ts_rs::TS` to the derive list and `#[ts(export)]` to the struct/enum
2. Use `#[serde(rename_all = "camelCase")]` for JS-friendly field names
3. For types that don't implement TS (like `DateTime<Utc>`, `exver::Version`), use `#[ts(type = "string")]` overrides
4. For `u64` fields that should be JS `number` (not `bigint`), use `#[ts(type = "number")]`
5. Run `make ts-bindings` to regenerate — files appear in `core/bindings/` then sync to `sdk/base/lib/osBindings/`
6. Rebuild the SDK: `cd sdk && make baseDist dist`
## Adding i18n Keys
1. Add the key to `core/locales/i18n.yaml` with all 5 language translations
2. Use the `t!("your.key.name")` macro in Rust code
3. Follow existing namespace conventions — match the module path where the key is used
4. Use kebab-case for multi-word segments
5. Translations are validated at compile time
See [i18n-patterns.md](i18n-patterns.md) for full conventions.

3387
core/Cargo.lock generated
View File

File diff suppressed because it is too large Load Diff

42
core/Cargo.toml
View File

@@ -15,7 +15,7 @@ license = "MIT"
name = "start-os"
readme = "README.md"
repository = "https://github.com/Start9Labs/start-os"
version = "0.4.0-alpha.19" # VERSION_BUMP
version = "0.4.0-alpha.20" # VERSION_BUMP
[lib]
name = "startos"
@@ -42,17 +42,6 @@ name = "tunnelbox"
path = "src/main/tunnelbox.rs"
[features]
arti = [
"arti-client",
"safelog",
"tor-cell",
"tor-hscrypto",
"tor-hsservice",
"tor-keymgr",
"tor-llcrypto",
"tor-proto",
"tor-rtcompat",
]
beta = []
console = ["console-subscriber", "tokio/tracing"]
default = []
@@ -62,16 +51,6 @@ unstable = ["backtrace-on-stack-overflow"]
[dependencies]
aes = { version = "0.7.5", features = ["ctr"] }
arti-client = { version = "0.33", features = [
"compression",
"ephemeral-keystore",
"experimental-api",
"onion-service-client",
"onion-service-service",
"rustls",
"static",
"tokio",
], default-features = false, git = "https://github.com/Start9Labs/arti.git", branch = "patch/disable-exit", optional = true }
async-acme = { version = "0.6.0", git = "https://github.com/dr-bonez/async-acme.git", features = [
"use_rustls",
"use_tokio",
@@ -100,7 +79,6 @@ console-subscriber = { version = "0.5.0", optional = true }
const_format = "0.2.34"
cookie = "0.18.0"
cookie_store = "0.22.0"
curve25519-dalek = "4.1.3"
der = { version = "0.7.9", features = ["derive", "pem"] }
digest = "0.10.7"
divrem = "1.0.0"
@@ -216,7 +194,6 @@ rpassword = "7.2.0"
rust-argon2 = "3.0.0"
rust-i18n = "3.1.5"
rpc-toolkit = { git = "https://github.com/Start9Labs/rpc-toolkit.git" }
safelog = { version = "0.4.8", git = "https://github.com/Start9Labs/arti.git", branch = "patch/disable-exit", optional = true }
semver = { version = "1.0.20", features = ["serde"] }
serde = { version = "1.0", features = ["derive", "rc"] }
serde_cbor = { package = "ciborium", version = "0.2.1" }
@@ -244,23 +221,6 @@ tokio-stream = { version = "0.1.14", features = ["io-util", "net", "sync"] }
tokio-tar = { git = "https://github.com/dr-bonez/tokio-tar.git" }
tokio-tungstenite = { version = "0.26.2", features = ["native-tls", "url"] }
tokio-util = { version = "0.7.9", features = ["io"] }
tor-cell = { version = "0.33", git = "https://github.com/Start9Labs/arti.git", branch = "patch/disable-exit", optional = true }
tor-hscrypto = { version = "0.33", features = [
"full",
], git = "https://github.com/Start9Labs/arti.git", branch = "patch/disable-exit", optional = true }
tor-hsservice = { version = "0.33", git = "https://github.com/Start9Labs/arti.git", branch = "patch/disable-exit", optional = true }
tor-keymgr = { version = "0.33", features = [
"ephemeral-keystore",
], git = "https://github.com/Start9Labs/arti.git", branch = "patch/disable-exit", optional = true }
tor-llcrypto = { version = "0.33", features = [
"full",
], git = "https://github.com/Start9Labs/arti.git", branch = "patch/disable-exit", optional = true }
tor-proto = { version = "0.33", git = "https://github.com/Start9Labs/arti.git", branch = "patch/disable-exit", optional = true }
tor-rtcompat = { version = "0.33", features = [
"rustls",
"tokio",
], git = "https://github.com/Start9Labs/arti.git", branch = "patch/disable-exit", optional = true }
torut = "0.2.1"
tower-service = "0.3.3"
tracing = "0.1.39"
tracing-error = "0.2.0"

8
core/README.md
View File

@@ -22,9 +22,7 @@ several different names for different behavior:
- `start-sdk`: This is a CLI tool that aids in building and packaging services
you wish to deploy to StartOS
## Questions
## Documentation
If you have questions about how various pieces of the backend system work. Open
an issue and tag the following people
- dr-bonez
- [ARCHITECTURE.md](ARCHITECTURE.md) — Backend architecture, modules, and patterns
- [CONTRIBUTING.md](CONTRIBUTING.md) — How to contribute to core

6
core/build/build-ts.sh
View File

@@ -7,11 +7,11 @@ source ./builder-alias.sh
set -ea
shopt -s expand_aliases
PROFILE=${PROFILE:-release}
PROFILE=${PROFILE:-debug}
if [ "${PROFILE}" = "release" ]; then
BUILD_FLAGS="--release"
else
if [ "$PROFILE" != "debug"]; then
if [ "$PROFILE" != "debug" ]; then
>&2 echo "Unknown profile $PROFILE: falling back to debug..."
PROFILE=debug
fi
@@ -38,7 +38,7 @@ if [[ "${ENVIRONMENT}" =~ (^|-)console($|-) ]]; then
fi
echo "FEATURES=\"$FEATURES\""
echo "RUSTFLAGS=\"$RUSTFLAGS\""
rust-zig-builder cargo test --manifest-path=./core/Cargo.toml $BUILD_FLAGS --features test,$FEATURES --locked 'export_bindings_'
rust-zig-builder cargo test --manifest-path=./core/Cargo.toml --lib $BUILD_FLAGS --features test,$FEATURES --locked 'export_bindings_'
if [ "$(ls -nd "core/bindings" | awk '{ print $3 }')" != "$UID" ]; then
rust-zig-builder sh -c "chown -R $UID:$UID core/target && chown -R $UID:$UID core/bindings && chown -R $UID:$UID /usr/local/cargo"
fi

249
core/core-rust-patterns.md Normal file
View File

@@ -0,0 +1,249 @@
# Utilities & Patterns
This document covers common utilities and patterns used throughout the StartOS codebase.
## Util Module (`core/src/util/`)
The `util` module contains reusable utilities. Key submodules:
| Module | Purpose |
|--------|---------|
| `actor/` | Actor pattern implementation for concurrent state management |
| `collections/` | Custom collection types |
| `crypto.rs` | Cryptographic utilities (encryption, hashing) |
| `future.rs` | Future/async utilities |
| `io.rs` | File I/O helpers (create_file, canonicalize, etc.) |
| `iter.rs` | Iterator extensions |
| `net.rs` | Network utilities |
| `rpc.rs` | RPC helpers |
| `rpc_client.rs` | RPC client utilities |
| `serde.rs` | Serialization helpers (Base64, display/fromstr, etc.) |
| `sync.rs` | Synchronization primitives (SyncMutex, etc.) |
## Command Invocation (`Invoke` trait)
The `Invoke` trait provides a clean way to run external commands with error handling:
```rust
use crate::util::Invoke;
// Simple invocation
tokio::process::Command::new("ls")
.arg("-la")
.invoke(ErrorKind::Filesystem)
.await?;
// With timeout
tokio::process::Command::new("slow-command")
.timeout(Some(Duration::from_secs(30)))
.invoke(ErrorKind::Timeout)
.await?;
// With input
let mut input = Cursor::new(b"input data");
tokio::process::Command::new("cat")
.input(Some(&mut input))
.invoke(ErrorKind::Filesystem)
.await?;
// Piped commands
tokio::process::Command::new("cat")
.arg("file.txt")
.pipe(&mut tokio::process::Command::new("grep").arg("pattern"))
.invoke(ErrorKind::Filesystem)
.await?;
```
## Guard Pattern
Guards ensure cleanup happens when they go out of scope.
### `GeneralGuard` / `GeneralBoxedGuard`
For arbitrary cleanup actions:
```rust
use crate::util::GeneralGuard;
let guard = GeneralGuard::new(|| {
println!("Cleanup runs on drop");
});
// Do work...
// Explicit drop with action
guard.drop();
// Or skip the action
// guard.drop_without_action();
```
### `FileLock`
File-based locking with automatic unlock:
```rust
use crate::util::FileLock;
let lock = FileLock::new("/path/to/lockfile", true).await?; // blocking=true
// Lock held until dropped or explicitly unlocked
lock.unlock().await?;
```
## Mount Guard Pattern (`core/src/disk/mount/guard.rs`)
RAII guards for filesystem mounts. Ensures filesystems are unmounted when guards are dropped.
### `MountGuard`
Basic mount guard:
```rust
use crate::disk::mount::guard::MountGuard;
use crate::disk::mount::filesystem::{MountType, ReadOnly};
let guard = MountGuard::mount(&filesystem, "/mnt/target", ReadOnly).await?;
// Use the mounted filesystem at guard.path()
do_something(guard.path()).await?;
// Explicit unmount (or auto-unmounts on drop)
guard.unmount(false).await?; // false = don't delete mountpoint
```
### `TmpMountGuard`
Reference-counted temporary mount (mounts to `/media/startos/tmp/`):
```rust
use crate::disk::mount::guard::TmpMountGuard;
use crate::disk::mount::filesystem::ReadOnly;
// Multiple clones share the same mount
let guard1 = TmpMountGuard::mount(&filesystem, ReadOnly).await?;
let guard2 = guard1.clone();
// Mount stays alive while any guard exists
// Auto-unmounts when last guard is dropped
```
### `GenericMountGuard` trait
All mount guards implement this trait:
```rust
pub trait GenericMountGuard: std::fmt::Debug + Send + Sync + 'static {
fn path(&self) -> &Path;
fn unmount(self) -> impl Future<Output = Result<(), Error>> + Send;
}
```
### `SubPath`
Wraps a mount guard to point to a subdirectory:
```rust
use crate::disk::mount::guard::SubPath;
let mount = TmpMountGuard::mount(&filesystem, ReadOnly).await?;
let subdir = SubPath::new(mount, "data/subdir");
// subdir.path() returns the full path including subdirectory
```
## FileSystem Implementations (`core/src/disk/mount/filesystem/`)
Various filesystem types that can be mounted:
| Type | Description |
|------|-------------|
| `bind.rs` | Bind mounts |
| `block_dev.rs` | Block device mounts |
| `cifs.rs` | CIFS/SMB network shares |
| `ecryptfs.rs` | Encrypted filesystem |
| `efivarfs.rs` | EFI variables |
| `httpdirfs.rs` | HTTP directory as filesystem |
| `idmapped.rs` | ID-mapped mounts |
| `label.rs` | Mount by label |
| `loop_dev.rs` | Loop device mounts |
| `overlayfs.rs` | Overlay filesystem |
## Other Useful Utilities
### `Apply` / `ApplyRef` traits
Fluent method chaining:
```rust
use crate::util::Apply;
let result = some_value
.apply(|v| transform(v))
.apply(|v| another_transform(v));
```
### `Container<T>`
Async-safe optional container:
```rust
use crate::util::Container;
let container = Container::new(None);
container.set(value).await;
let taken = container.take().await;
```
### `HashWriter<H, W>`
Write data while computing hash:
```rust
use crate::util::HashWriter;
use sha2::Sha256;
let writer = HashWriter::new(Sha256::new(), file);
// Write data...
let (hasher, file) = writer.finish();
let hash = hasher.finalize();
```
### `Never` type
Uninhabited type for impossible cases:
```rust
use crate::util::Never;
fn impossible() -> Never {
// This function can never return
}
let never: Never = impossible();
never.absurd::<String>() // Can convert to any type
```
### `MaybeOwned<'a, T>`
Either borrowed or owned data:
```rust
use crate::util::MaybeOwned;
fn accept_either(data: MaybeOwned<'_, String>) {
// Use &*data to access the value
}
accept_either(MaybeOwned::from(&existing_string));
accept_either(MaybeOwned::from(owned_string));
```
### `new_guid()`
Generate a random GUID:
```rust
use crate::util::new_guid;
let guid = new_guid(); // Returns InternedString
```

100
core/i18n-patterns.md Normal file
View File

@@ -0,0 +1,100 @@
# i18n Patterns in `core/`
## Library & Setup
**Crate:** [`rust-i18n`](https://crates.io/crates/rust-i18n) v3.1.5 (`core/Cargo.toml`)
**Initialization** (`core/src/lib.rs:3`):
```rust
rust_i18n::i18n!("locales", fallback = ["en_US"]);
```
This macro scans `core/locales/` at compile time and embeds all translations as constants.
**Prelude re-export** (`core/src/prelude.rs:4`):
```rust
pub use rust_i18n::t;
```
Most modules import `t!` via the prelude.
## Translation File
**Location:** `core/locales/i18n.yaml`
**Format:** YAML v2 (~755 keys)
**Supported languages:** `en_US`, `de_DE`, `es_ES`, `fr_FR`, `pl_PL`
**Entry structure:**
```yaml
namespace.sub.key-name:
en_US: "English text with %{param}"
de_DE: "German text with %{param}"
# ...
```
## Using `t!()`
```rust
// Simple key
t!("error.unknown")
// With parameter interpolation (%{name} in YAML)
t!("bins.deprecated.renamed", old = old_name, new = new_name)
```
## Key Naming Conventions
Keys use **dot-separated hierarchical namespaces** with **kebab-case** for multi-word segments:
```
<module>.<submodule>.<descriptive-name>
```
Examples:
- `error.incorrect-password` — error kind label
- `bins.start-init.updating-firmware` — startup phase message
- `backup.bulk.complete-title` — backup notification title
- `help.arg.acme-contact` — CLI help text for an argument
- `context.diagnostic.starting-diagnostic-ui` — diagnostic context status
### Top-Level Namespaces
| Namespace | Purpose |
|-----------|---------|
| `error.*` | `ErrorKind` display strings (see `src/error.rs`) |
| `bins.*` | CLI binary messages (deprecated, start-init, startd, etc.) |
| `init.*` | Initialization phase labels |
| `setup.*` | First-run setup messages |
| `context.*` | Context startup messages (diagnostic, setup, CLI) |
| `service.*` | Service lifecycle messages |
| `backup.*` | Backup/restore operation messages |
| `registry.*` | Package registry messages |
| `net.*` | Network-related messages |
| `middleware.*` | Request middleware messages (auth, etc.) |
| `disk.*` | Disk operation messages |
| `lxc.*` | Container management messages |
| `system.*` | System monitoring/metrics messages |
| `notifications.*` | User-facing notification messages |
| `update.*` | OS update messages |
| `util.*` | Utility messages (TUI, RPC) |
| `ssh.*` | SSH operation messages |
| `shutdown.*` | Shutdown-related messages |
| `logs.*` | Log-related messages |
| `auth.*` | Authentication messages |
| `help.*` | CLI help text (`help.arg.<arg-name>`) |
| `about.*` | CLI command descriptions |
## Locale Selection
`core/src/bins/mod.rs:15-36` — `set_locale_from_env()`:
1. Reads `LANG` environment variable
2. Strips `.UTF-8` suffix
3. Exact-matches against available locales, falls back to language-prefix match (e.g. `en_GB` matches `en_US`)
## Adding New Keys
1. Add the key to `core/locales/i18n.yaml` with all 5 language translations
2. Use the `t!("your.key.name")` macro in Rust code
3. Follow existing namespace conventions — match the module path where the key is used
4. Use kebab-case for multi-word segments
5. Translations are validated at compile time

118
core/locales/i18n.yaml
View File

@@ -197,6 +197,13 @@ setup.transferring-data:
fr_FR: "Transfert de données"
pl_PL: "Przesyłanie danych"
setup.password-required:
en_US: "Password is required for fresh setup"
de_DE: "Passwort ist für die Ersteinrichtung erforderlich"
es_ES: "Se requiere contraseña para la configuración inicial"
fr_FR: "Le mot de passe est requis pour la première configuration"
pl_PL: "Hasło jest wymagane do nowej konfiguracji"
# system.rs
system.governor-not-available:
en_US: "Governor %{governor} not available"
@@ -994,6 +1001,27 @@ disk.mount.binding:
fr_FR: "Liaison de %{src} à %{dst}"
pl_PL: "Wiązanie %{src} do %{dst}"
hostname.empty:
en_US: "Hostname cannot be empty"
de_DE: "Der Hostname darf nicht leer sein"
es_ES: "El nombre de host no puede estar vacío"
fr_FR: "Le nom d'hôte ne peut pas être vide"
pl_PL: "Nazwa hosta nie może być pusta"
hostname.invalid-character:
en_US: "Invalid character in hostname: %{char}"
de_DE: "Ungültiges Zeichen im Hostnamen: %{char}"
es_ES: "Carácter no válido en el nombre de host: %{char}"
fr_FR: "Caractère invalide dans le nom d'hôte : %{char}"
pl_PL: "Nieprawidłowy znak w nazwie hosta: %{char}"
hostname.must-provide-name-or-hostname:
en_US: "Must provide at least one of: name, hostname"
de_DE: "Es muss mindestens eines angegeben werden: name, hostname"
es_ES: "Se debe proporcionar al menos uno de: name, hostname"
fr_FR: "Vous devez fournir au moins l'un des éléments suivants : name, hostname"
pl_PL: "Należy podać co najmniej jedno z: name, hostname"
# init.rs
init.running-preinit:
en_US: "Running preinit.sh"
@@ -1243,6 +1271,21 @@ backup.target.cifs.target-not-found-id:
fr_FR: "ID de cible de sauvegarde %{id} non trouvé"
pl_PL: "Nie znaleziono ID celu kopii zapasowej %{id}"
# service/effects/net/plugin.rs
net.plugin.manifest-missing-plugin:
en_US: "manifest does not declare the \"%{plugin}\" plugin"
de_DE: "Manifest deklariert das Plugin \"%{plugin}\" nicht"
es_ES: "el manifiesto no declara el plugin \"%{plugin}\""
fr_FR: "le manifeste ne déclare pas le plugin \"%{plugin}\""
pl_PL: "manifest nie deklaruje wtyczki \"%{plugin}\""
net.plugin.binding-not-found:
en_US: "binding not found: %{binding}"
de_DE: "Bindung nicht gefunden: %{binding}"
es_ES: "enlace no encontrado: %{binding}"
fr_FR: "liaison introuvable : %{binding}"
pl_PL: "powiązanie nie znalezione: %{binding}"
# net/ssl.rs
net.ssl.unreachable:
en_US: "unreachable"
@@ -1790,6 +1833,28 @@ registry.package.remove-mirror.unauthorized:
fr_FR: "Non autorisé"
pl_PL: "Brak autoryzacji"
# registry/package/index.rs
registry.package.index.metadata-mismatch:
en_US: "package metadata mismatch: remove the existing version first, then re-add"
de_DE: "Paketmetadaten stimmen nicht überein: vorhandene Version zuerst entfernen, dann erneut hinzufügen"
es_ES: "discrepancia de metadatos del paquete: elimine la versión existente primero, luego vuelva a agregarla"
fr_FR: "discordance des métadonnées du paquet : supprimez d'abord la version existante, puis ajoutez-la à nouveau"
pl_PL: "niezgodność metadanych pakietu: najpierw usuń istniejącą wersję, a następnie dodaj ponownie"
registry.package.index.icon-mismatch:
en_US: "package icon mismatch: remove the existing version first, then re-add"
de_DE: "Paketsymbol stimmt nicht überein: vorhandene Version zuerst entfernen, dann erneut hinzufügen"
es_ES: "discrepancia del icono del paquete: elimine la versión existente primero, luego vuelva a agregarla"
fr_FR: "discordance de l'icône du paquet : supprimez d'abord la version existante, puis ajoutez-la à nouveau"
pl_PL: "niezgodność ikony pakietu: najpierw usuń istniejącą wersję, a następnie dodaj ponownie"
registry.package.index.dependency-metadata-mismatch:
en_US: "dependency metadata mismatch: remove the existing version first, then re-add"
de_DE: "Abhängigkeitsmetadaten stimmen nicht überein: vorhandene Version zuerst entfernen, dann erneut hinzufügen"
es_ES: "discrepancia de metadatos de dependencia: elimine la versión existente primero, luego vuelva a agregarla"
fr_FR: "discordance des métadonnées de dépendance : supprimez d'abord la version existante, puis ajoutez-la à nouveau"
pl_PL: "niezgodność metadanych zależności: najpierw usuń istniejącą wersję, a następnie dodaj ponownie"
# registry/package/get.rs
registry.package.get.version-not-found:
en_US: "Could not find a version of %{id} that satisfies %{version}"
@@ -3087,7 +3152,7 @@ help.arg.smtp-from:
fr_FR: "Adresse de l'expéditeur"
pl_PL: "Adres nadawcy e-mail"
help.arg.smtp-login:
help.arg.smtp-username:
en_US: "SMTP authentication username"
de_DE: "SMTP-Authentifizierungsbenutzername"
es_ES: "Nombre de usuario de autenticación SMTP"
@@ -3108,13 +3173,20 @@ help.arg.smtp-port:
fr_FR: "Port du serveur SMTP"
pl_PL: "Port serwera SMTP"
help.arg.smtp-server:
help.arg.smtp-host:
en_US: "SMTP server hostname"
de_DE: "SMTP-Server-Hostname"
es_ES: "Nombre de host del servidor SMTP"
fr_FR: "Nom d'hôte du serveur SMTP"
pl_PL: "Nazwa hosta serwera SMTP"
help.arg.smtp-security:
en_US: "Connection security mode (starttls or tls)"
de_DE: "Verbindungssicherheitsmodus (starttls oder tls)"
es_ES: "Modo de seguridad de conexión (starttls o tls)"
fr_FR: "Mode de sécurité de connexion (starttls ou tls)"
pl_PL: "Tryb zabezpieczeń połączenia (starttls lub tls)"
help.arg.smtp-to:
en_US: "Email recipient address"
de_DE: "E-Mail-Empfängeradresse"
@@ -3612,6 +3684,13 @@ help.arg.s9pk-file-path:
fr_FR: "Chemin vers le fichier de paquet s9pk"
pl_PL: "Ścieżka do pliku pakietu s9pk"
help.arg.s9pk-file-paths:
en_US: "Paths to s9pk package files"
de_DE: "Pfade zu s9pk-Paketdateien"
es_ES: "Rutas a los archivos de paquete s9pk"
fr_FR: "Chemins vers les fichiers de paquet s9pk"
pl_PL: "Ścieżki do plików pakietów s9pk"
help.arg.session-ids:
en_US: "Session identifiers"
de_DE: "Sitzungskennungen"
@@ -3935,6 +4014,13 @@ about.allow-gateway-infer-inbound-access-from-wan:
fr_FR: "Permettre à cette passerelle de déduire si elle a un accès entrant depuis le WAN en fonction de son adresse IPv4"
pl_PL: "Pozwól tej bramce wywnioskować, czy ma dostęp przychodzący z WAN na podstawie adresu IPv4"
about.apply-available-update:
en_US: "Apply available update"
de_DE: "Verfügbares Update anwenden"
es_ES: "Aplicar actualización disponible"
fr_FR: "Appliquer la mise à jour disponible"
pl_PL: "Zastosuj dostępną aktualizację"
about.calculate-blake3-hash-for-file:
en_US: "Calculate blake3 hash for a file"
de_DE: "Blake3-Hash für eine Datei berechnen"
@@ -3949,6 +4035,20 @@ about.cancel-install-package:
fr_FR: "Annuler l'installation d'un paquet"
pl_PL: "Anuluj instalację pakietu"
about.check-dns-configuration:
en_US: "Check DNS configuration for a gateway"
de_DE: "DNS-Konfiguration für ein Gateway prüfen"
es_ES: "Verificar la configuración DNS de un gateway"
fr_FR: "Vérifier la configuration DNS d'une passerelle"
pl_PL: "Sprawdź konfigurację DNS bramy"
about.check-for-updates:
en_US: "Check for available updates"
de_DE: "Nach verfügbaren Updates suchen"
es_ES: "Buscar actualizaciones disponibles"
fr_FR: "Vérifier les mises à jour disponibles"
pl_PL: "Sprawdź dostępne aktualizacje"
about.check-update-startos:
en_US: "Check a given registry for StartOS updates and update if available"
de_DE: "Ein bestimmtes Registry auf StartOS-Updates prüfen und bei Verfügbarkeit aktualisieren"
@@ -4887,6 +4987,13 @@ about.publish-s9pk:
fr_FR: "Publier s9pk dans le bucket S3 et indexer dans le registre"
pl_PL: "Opublikuj s9pk do bucketu S3 i zindeksuj w rejestrze"
about.select-s9pk-for-device:
en_US: "Select the best compatible s9pk for a target device"
de_DE: "Das beste kompatible s9pk für ein Zielgerät auswählen"
es_ES: "Seleccionar el s9pk más compatible para un dispositivo destino"
fr_FR: "Sélectionner le meilleur s9pk compatible pour un appareil cible"
pl_PL: "Wybierz najlepiej kompatybilny s9pk dla urządzenia docelowego"
about.rebuild-service-container:
en_US: "Rebuild service container"
de_DE: "Dienst-Container neu erstellen"
@@ -5139,6 +5246,13 @@ about.set-country:
fr_FR: "Définir le pays"
pl_PL: "Ustaw kraj"
about.set-hostname:
en_US: "Set the server hostname"
de_DE: "Den Server-Hostnamen festlegen"
es_ES: "Establecer el nombre de host del servidor"
fr_FR: "Définir le nom d'hôte du serveur"
pl_PL: "Ustaw nazwę hosta serwera"
about.set-gateway-enabled-for-binding:
en_US: "Set gateway enabled for binding"
de_DE: "Gateway für Bindung aktivieren"

105
core/patchdb.md Normal file
View File

@@ -0,0 +1,105 @@
# Patch-DB Patterns
## Model<T> and HasModel
Types stored in the database derive `HasModel`, which generates typed accessor methods on `Model<T>`:
```rust
#[derive(Debug, Deserialize, Serialize, HasModel)]
#[serde(rename_all = "camelCase")]
#[model = "Model<Self>"]
pub struct ServerInfo {
pub version: Version,
pub network: NetworkInfo,
// ...
}
```
**Generated accessors** (one per field):
- `as_version()` — `&Model<Version>`
- `as_version_mut()` — `&mut Model<Version>`
- `into_version()` — `Model<Version>`
**`Model<T>` APIs:**
- `.de()` — Deserialize to `T`
- `.ser(&value)` — Serialize from `T`
- `.mutate(|v| ...)` — Deserialize, mutate, reserialize
- For maps: `.keys()`, `.as_idx(&key)`, `.insert()`, `.remove()`, `.contains_key()`
## Database Access
```rust
// Read-only snapshot
let snap = db.peek().await;
let version = snap.as_public().as_server_info().as_version().de()?;
// Atomic mutation
db.mutate(|db| {
db.as_public_mut().as_server_info_mut().as_version_mut().ser(&new_version)?;
Ok(())
}).await;
```
## TypedDbWatch<T>
Watch a JSON pointer path for changes and deserialize as a typed value. Requires `T: HasModel`.
### Construction
```rust
use patch_db::json_ptr::JsonPointer;
let ptr: JsonPointer = "/public/serverInfo".parse().unwrap();
let mut watch = db.watch(ptr).await.typed::<ServerInfo>();
```
### API
- `watch.peek()?.de()?` — Get current value as `T`
- `watch.changed().await?` — Wait until the watched path changes
- `watch.peek()?.as_field().de()?` — Access nested fields via `HasModel` accessors
### Usage Patterns
**Wait for a condition, then proceed:**
```rust
// Wait for DB version to match current OS version
let current = Current::default().semver();
let mut watch = db
.watch("/public/serverInfo".parse().unwrap())
.await
.typed::<ServerInfo>();
loop {
let server_info = watch.peek()?.de()?;
if server_info.version == current {
break;
}
watch.changed().await?;
}
```
**React to changes in a loop:**
```rust
// From net_controller.rs — react to host changes
let mut watch = db
.watch("/public/serverInfo/network/host".parse().unwrap())
.await
.typed::<Host>();
loop {
if let Err(e) = watch.changed().await {
tracing::error!("DB watch disconnected: {e}");
break;
}
let host = watch.peek()?.de()?;
// ... process host ...
}
```
### Real Examples
- `net_controller.rs:469` — Watch `Hosts` for package network changes
- `net_controller.rs:493` — Watch `Host` for main UI network changes
- `service_actor.rs:37` — Watch `StatusInfo` for service state transitions
- `gateway.rs:1212` — Wait for DB migrations to complete before syncing

234
core/rpc-toolkit.md Normal file
View File

@@ -0,0 +1,234 @@
# rpc-toolkit
StartOS uses [rpc-toolkit](https://github.com/Start9Labs/rpc-toolkit) for its JSON-RPC API. This document covers the patterns used in this codebase.
## Overview
The API is JSON-RPC (not REST). All endpoints are RPC methods organized in a hierarchical command structure.
## Handler Functions
There are four types of handler functions, chosen based on the function's characteristics:
### `from_fn_async` - Async handlers
For standard async functions. Most handlers use this.
```rust
pub async fn my_handler(ctx: RpcContext, params: MyParams) -> Result<MyResponse, Error> {
// Can use .await
}
from_fn_async(my_handler)
```
If a handler takes no params, simply omit the params argument entirely (no need for `_: Empty`):
```rust
pub async fn no_params_handler(ctx: RpcContext) -> Result<MyResponse, Error> {
// ...
}
```
### `from_fn_async_local` - Non-thread-safe async handlers
For async functions that are not `Send` (cannot be safely moved between threads). Use when working with non-thread-safe types.
```rust
pub async fn cli_download(ctx: CliContext, params: Params) -> Result<(), Error> {
// Non-Send async operations
}
from_fn_async_local(cli_download)
```
### `from_fn_blocking` - Sync blocking handlers
For synchronous functions that perform blocking I/O or long computations.
```rust
pub fn query_dns(ctx: RpcContext, params: DnsParams) -> Result<DnsResponse, Error> {
// Blocking operations (file I/O, DNS lookup, etc.)
}
from_fn_blocking(query_dns)
```
### `from_fn` - Sync non-blocking handlers
For pure functions or quick synchronous operations with no I/O.
```rust
pub fn echo(ctx: RpcContext, params: EchoParams) -> Result<String, Error> {
Ok(params.message)
}
from_fn(echo)
```
## ParentHandler
Groups related RPC methods into a hierarchy:
```rust
use rpc_toolkit::{Context, HandlerExt, ParentHandler, from_fn_async};
pub fn my_api<C: Context>() -> ParentHandler<C> {
ParentHandler::new()
.subcommand("list", from_fn_async(list_handler).with_call_remote::<CliContext>())
.subcommand("create", from_fn_async(create_handler).with_call_remote::<CliContext>())
}
```
## Handler Extensions
Chain methods to configure handler behavior.
**Ordering rules:**
1. `with_about()` must come AFTER other CLI modifiers (`no_display()`, `with_custom_display_fn()`, etc.)
2. `with_call_remote()` must be the LAST adapter in the chain
| Method | Purpose |
|--------|---------|
| `.with_metadata("key", Value)` | Attach metadata for middleware |
| `.no_cli()` | RPC-only, not available via CLI |
| `.no_display()` | No CLI output |
| `.with_display_serializable()` | Default JSON/YAML output for CLI |
| `.with_custom_display_fn(\|_, res\| ...)` | Custom CLI output formatting |
| `.with_about("about.description")` | Add help text (i18n key) - **after CLI modifiers** |
| `.with_call_remote::<CliContext>()` | Enable CLI to call remotely - **must be last** |
### Correct ordering example:
```rust
from_fn_async(my_handler)
.with_metadata("sync_db", Value::Bool(true)) // metadata early
.no_display() // CLI modifier
.with_about("about.my-handler") // after CLI modifiers
.with_call_remote::<CliContext>() // always last
```
## Metadata by Middleware
Metadata tags are processed by different middleware. Group them logically:
### Auth Middleware (`middleware/auth/mod.rs`)
| Metadata | Default | Description |
|----------|---------|-------------|
| `authenticated` | `true` | Whether endpoint requires authentication. Set to `false` for public endpoints. |
### Session Auth Middleware (`middleware/auth/session.rs`)
| Metadata | Default | Description |
|----------|---------|-------------|
| `login` | `false` | Special handling for login endpoints (rate limiting, cookie setting) |
| `get_session` | `false` | Inject session ID into params as `__Auth_session` |
### Signature Auth Middleware (`middleware/auth/signature.rs`)
| Metadata | Default | Description |
|----------|---------|-------------|
| `get_signer` | `false` | Inject signer public key into params as `__Auth_signer` |
### Registry Auth (extends Signature Auth)
| Metadata | Default | Description |
|----------|---------|-------------|
| `admin` | `false` | Require admin privileges (signer must be in admin list) |
| `get_device_info` | `false` | Inject device info header for hardware filtering |
### Database Middleware (`middleware/db.rs`)
| Metadata | Default | Description |
|----------|---------|-------------|
| `sync_db` | `false` | Sync database after mutation, add `X-Patch-Sequence` header |
## Context Types
Different contexts for different execution environments:
- `RpcContext` - Web/RPC requests with full service access
- `CliContext` - CLI operations, calls remote RPC
- `InitContext` - During system initialization
- `DiagnosticContext` - Diagnostic/recovery mode
- `RegistryContext` - Registry daemon context
- `EffectContext` - Service effects context (container-to-host calls)
## Parameter Structs
Parameters use derive macros for JSON-RPC, CLI parsing, and TypeScript generation:
```rust
#[derive(Deserialize, Serialize, Parser, TS)]
#[serde(rename_all = "camelCase")] // JSON-RPC uses camelCase
#[command(rename_all = "kebab-case")] // CLI uses kebab-case
#[ts(export)] // Generate TypeScript types
pub struct MyParams {
pub package_id: PackageId,
}
```
### Middleware Injection
Auth middleware can inject values into params using special field names:
```rust
#[derive(Deserialize, Serialize, Parser, TS)]
pub struct MyParams {
#[ts(skip)]
#[serde(rename = "__Auth_session")] // Injected by session auth
session: InternedString,
#[ts(skip)]
#[serde(rename = "__Auth_signer")] // Injected by signature auth
signer: AnyVerifyingKey,
#[ts(skip)]
#[serde(rename = "__Auth_userAgent")] // Injected during login
user_agent: Option<String>,
}
```
## Common Patterns
### Adding a New RPC Endpoint
1. Define params struct with `Deserialize, Serialize, Parser, TS` (skip if no params needed)
2. Choose handler type based on sync/async and thread-safety
3. Write handler function taking `(Context, Params) -> Result<Response, Error>` (omit Params if none needed)
4. Add to parent handler with appropriate extensions (display modifiers before `with_about`)
5. TypeScript types auto-generated via `make ts-bindings`
### Public (Unauthenticated) Endpoint
```rust
from_fn_async(get_info)
.with_metadata("authenticated", Value::Bool(false))
.with_display_serializable()
.with_about("about.get-info")
.with_call_remote::<CliContext>() // last
```
### Mutating Endpoint with DB Sync
```rust
from_fn_async(update_config)
.with_metadata("sync_db", Value::Bool(true))
.no_display()
.with_about("about.update-config")
.with_call_remote::<CliContext>() // last
```
### Session-Aware Endpoint
```rust
from_fn_async(logout)
.with_metadata("get_session", Value::Bool(true))
.no_display()
.with_about("about.logout")
.with_call_remote::<CliContext>() // last
```
## File Locations
- Handler definitions: Throughout `core/src/` modules
- Main API tree: `core/src/lib.rs` (`main_api()`, `server()`, `package()`)
- Auth middleware: `core/src/middleware/auth/`
- DB middleware: `core/src/middleware/db.rs`
- Context types: `core/src/context/`

122
core/s9pk-structure.md Normal file
View File

@@ -0,0 +1,122 @@
# S9PK Package Format
S9PK is the package format for StartOS services. Version 2 uses a merkle archive structure for efficient downloading and cryptographic verification.
## File Format
S9PK files begin with a 3-byte header: `0x3b 0x3b 0x02` (magic bytes + version 2).
The archive is cryptographically signed using Ed25519 with prehashed content (SHA-512 over blake3 merkle root hash).
## Archive Structure
```
/
├── manifest.json # Package metadata (required)
├── icon.<ext> # Package icon - any image/* format (required)
├── LICENSE.md # License text (required)
├── dependencies/ # Dependency metadata (optional)
│ └── <package-id>/
│ ├── metadata.json # DependencyMetadata
│ └── icon.<ext> # Dependency icon
├── javascript.squashfs # Package JavaScript code (required)
├── assets.squashfs # Static assets (optional, legacy: assets/ directory)
└── images/ # Container images by architecture
└── <arch>/ # e.g., x86_64, aarch64, riscv64
├── <image-id>.squashfs # Container filesystem
├── <image-id>.json # Image metadata
└── <image-id>.env # Environment variables
```
## Components
### manifest.json
The package manifest contains all metadata:
| Field | Type | Description |
|-------|------|-------------|
| `id` | string | Package identifier (e.g., `bitcoind`) |
| `title` | string | Display name |
| `version` | string | Extended version string |
| `satisfies` | string[] | Version ranges this version satisfies |
| `releaseNotes` | string/object | Release notes (localized) |
| `canMigrateTo` | string | Version range for forward migration |
| `canMigrateFrom` | string | Version range for backward migration |
| `license` | string | License type |
| `wrapperRepo` | string | StartOS wrapper repository URL |
| `upstreamRepo` | string | Upstream project URL |
| `supportSite` | string | Support site URL |
| `marketingSite` | string | Marketing site URL |
| `donationUrl` | string? | Optional donation URL |
| `docsUrl` | string? | Optional documentation URL |
| `description` | object | Short and long descriptions (localized) |
| `images` | object | Image configurations by image ID |
| `volumes` | string[] | Volume IDs for persistent data |
| `alerts` | object | User alerts for lifecycle events |
| `dependencies` | object | Package dependencies |
| `hardwareRequirements` | object | Hardware requirements (arch, RAM, devices) |
| `hardwareAcceleration` | boolean | Whether package uses hardware acceleration |
| `gitHash` | string? | Git commit hash |
| `osVersion` | string | Minimum StartOS version |
| `sdkVersion` | string? | SDK version used to build |
### javascript.squashfs
Contains the package JavaScript that implements the `ABI` interface from `@start9labs/start-sdk-base`. This code runs in the container runtime and manages the package lifecycle.
The squashfs is mounted at `/usr/lib/startos/package/` and the runtime loads `index.js`.
### images/
Container images organized by architecture:
- **`<image-id>.squashfs`** - Container root filesystem
- **`<image-id>.json`** - Image metadata (entrypoint, user, workdir, etc.)
- **`<image-id>.env`** - Environment variables for the container
Images are built from Docker/Podman and converted to squashfs. The `ImageConfig` in manifest specifies:
- `arch` - Supported architectures
- `emulateMissingAs` - Fallback architecture for emulation
- `nvidiaContainer` - Whether to enable NVIDIA container support
### assets.squashfs
Static assets accessible to the package, mounted read-only at `/media/startos/assets/` in the container.
### dependencies/
Metadata for dependencies displayed in the UI:
- `metadata.json` - Just title for now
- `icon.<ext>` - Icon for the dependency
## Merkle Archive
The S9PK uses a merkle tree structure where each file and directory has a blake3 hash. This enables:
1. **Partial downloads** - Download and verify individual files
2. **Integrity verification** - Verify any subset of the archive
3. **Efficient updates** - Only download changed portions
4. **DOS protection** - Size limits enforced before downloading content
Files are sorted by priority for streaming (manifest first, then icon, license, dependencies, javascript, assets, images).
## Building S9PK
Use `start-cli s9pk pack` to build packages:
```bash
start-cli s9pk pack <manifest-path> -o <output.s9pk>
```
Images can be sourced from:
- Docker/Podman build (`--docker-build`)
- Existing Docker tag (`--docker-tag`)
- Pre-built squashfs files
## Related Code
- `core/src/s9pk/v2/mod.rs` - S9pk struct and serialization
- `core/src/s9pk/v2/manifest.rs` - Manifest types
- `core/src/s9pk/v2/pack.rs` - Packing logic
- `core/src/s9pk/merkle_archive/` - Merkle archive implementation

62
core/src/account.rs
View File

@@ -6,9 +6,8 @@ use openssl::pkey::{PKey, Private};
use openssl::x509::X509;
use crate::db::model::DatabaseModel;
use crate::hostname::{Hostname, generate_hostname, generate_id};
use crate::hostname::{ServerHostnameInfo, generate_hostname, generate_id};
use crate::net::ssl::{gen_nistp256, make_root_cert};
use crate::net::tor::TorSecretKey;
use crate::prelude::*;
use crate::util::serde::Pem;
@@ -24,21 +23,27 @@ fn hash_password(password: &str) -> Result<String, Error> {
#[derive(Clone)]
pub struct AccountInfo {
pub server_id: String,
pub hostname: Hostname,
pub hostname: ServerHostnameInfo,
pub password: String,
pub tor_keys: Vec<TorSecretKey>,
pub root_ca_key: PKey<Private>,
pub root_ca_cert: X509,
pub ssh_key: ssh_key::PrivateKey,
pub developer_key: ed25519_dalek::SigningKey,
}
impl AccountInfo {
pub fn new(password: &str, start_time: SystemTime) -> Result<Self, Error> {
pub fn new(
password: &str,
start_time: SystemTime,
hostname: Option<ServerHostnameInfo>,
) -> Result<Self, Error> {
let server_id = generate_id();
let hostname = generate_hostname();
let tor_key = vec![TorSecretKey::generate()];
let hostname = if let Some(h) = hostname {
h
} else {
ServerHostnameInfo::from_hostname(generate_hostname())
};
let root_ca_key = gen_nistp256()?;
let root_ca_cert = make_root_cert(&root_ca_key, &hostname, start_time)?;
let root_ca_cert = make_root_cert(&root_ca_key, &hostname.hostname, start_time)?;
let ssh_key = ssh_key::PrivateKey::from(ssh_key::private::Ed25519Keypair::random(
&mut ssh_key::rand_core::OsRng::default(),
));
@@ -48,7 +53,6 @@ impl AccountInfo {
server_id,
hostname,
password: hash_password(password)?,
tor_keys: tor_key,
root_ca_key,
root_ca_cert,
ssh_key,
@@ -58,20 +62,9 @@ impl AccountInfo {
pub fn load(db: &DatabaseModel) -> Result<Self, Error> {
let server_id = db.as_public().as_server_info().as_id().de()?;
let hostname = Hostname(db.as_public().as_server_info().as_hostname().de()?);
let hostname = ServerHostnameInfo::load(db.as_public().as_server_info())?;
let password = db.as_private().as_password().de()?;
let key_store = db.as_private().as_key_store();
let tor_addrs = db
.as_public()
.as_server_info()
.as_network()
.as_host()
.as_onions()
.de()?;
let tor_keys = tor_addrs
.into_iter()
.map(|tor_addr| key_store.as_onion().get_key(&tor_addr))
.collect::<Result<_, _>>()?;
let cert_store = key_store.as_local_certs();
let root_ca_key = cert_store.as_root_key().de()?.0;
let root_ca_cert = cert_store.as_root_cert().de()?.0;
@@ -82,7 +75,6 @@ impl AccountInfo {
server_id,
hostname,
password,
tor_keys,
root_ca_key,
root_ca_cert,
ssh_key,
@@ -93,21 +85,10 @@ impl AccountInfo {
pub fn save(&self, db: &mut DatabaseModel) -> Result<(), Error> {
let server_info = db.as_public_mut().as_server_info_mut();
server_info.as_id_mut().ser(&self.server_id)?;
server_info.as_hostname_mut().ser(&self.hostname.0)?;
self.hostname.save(server_info)?;
server_info
.as_pubkey_mut()
.ser(&self.ssh_key.public_key().to_openssh()?)?;
server_info
.as_network_mut()
.as_host_mut()
.as_onions_mut()
.ser(
&self
.tor_keys
.iter()
.map(|tor_key| tor_key.onion_address())
.collect(),
)?;
server_info.as_password_hash_mut().ser(&self.password)?;
db.as_private_mut().as_password_mut().ser(&self.password)?;
db.as_private_mut()
@@ -117,9 +98,6 @@ impl AccountInfo {
.as_developer_key_mut()
.ser(Pem::new_ref(&self.developer_key))?;
let key_store = db.as_private_mut().as_key_store_mut();
for tor_key in &self.tor_keys {
key_store.as_onion_mut().insert_key(tor_key)?;
}
let cert_store = key_store.as_local_certs_mut();
if cert_store.as_root_cert().de()?.0 != self.root_ca_cert {
cert_store
@@ -145,14 +123,8 @@ impl AccountInfo {
pub fn hostnames(&self) -> impl IntoIterator<Item = InternedString> + Send + '_ {
[
self.hostname.no_dot_host_name(),
self.hostname.local_domain_name(),
(*self.hostname.hostname).clone(),
self.hostname.hostname.local_domain_name(),
]
.into_iter()
.chain(
self.tor_keys
.iter()
.map(|k| InternedString::from_display(&k.onion_address())),
)
}
}

9
core/src/action.rs
View File

@@ -67,6 +67,10 @@ pub struct GetActionInputParams {
pub package_id: PackageId,
#[arg(help = "help.arg.action-id")]
pub action_id: ActionId,
#[ts(type = "Record<string, unknown> | null")]
#[serde(default)]
#[arg(skip)]
pub prefill: Option<Value>,
}
#[instrument(skip_all)]
@@ -75,6 +79,7 @@ pub async fn get_action_input(
GetActionInputParams {
package_id,
action_id,
prefill,
}: GetActionInputParams,
) -> Result<Option<ActionInput>, Error> {
ctx.services
@@ -82,7 +87,7 @@ pub async fn get_action_input(
.await
.as_ref()
.or_not_found(lazy_format!("Manager for {}", package_id))?
.get_action_input(Guid::new(), action_id)
.get_action_input(Guid::new(), action_id, prefill.unwrap_or(Value::Null))
.await
}
@@ -271,6 +276,7 @@ pub fn display_action_result<T: Serialize>(
}
#[derive(Deserialize, Serialize, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub struct RunActionParams {
pub package_id: PackageId,
@@ -362,6 +368,7 @@ pub async fn run_action(
}
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
pub struct ClearTaskParams {

2
core/src/auth.rs
View File

@@ -418,6 +418,7 @@ impl AsLogoutSessionId for KillSessionId {
}
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
pub struct KillParams {
@@ -435,6 +436,7 @@ pub async fn kill<C: SessionAuthContext>(
}
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
pub struct ResetPasswordParams {

7
core/src/backup/backup_bulk.rs
View File

@@ -30,6 +30,7 @@ use crate::util::serde::IoFormat;
use crate::version::VersionT;
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
pub struct BackupParams {
@@ -270,9 +271,9 @@ async fn perform_backup(
package_backups.insert(
id.clone(),
PackageBackupInfo {
os_version: manifest.as_os_version().de()?,
os_version: manifest.as_metadata().as_os_version().de()?,
version: manifest.as_version().de()?,
title: manifest.as_title().de()?,
title: manifest.as_metadata().as_title().de()?,
timestamp: Utc::now(),
},
);
@@ -337,7 +338,7 @@ async fn perform_backup(
let timestamp = Utc::now();
backup_guard.unencrypted_metadata.version = crate::version::Current::default().semver().into();
backup_guard.unencrypted_metadata.hostname = ctx.account.peek(|a| a.hostname.clone());
backup_guard.unencrypted_metadata.hostname = ctx.account.peek(|a| a.hostname.hostname.clone());
backup_guard.unencrypted_metadata.timestamp = timestamp.clone();
backup_guard.metadata.version = crate::version::Current::default().semver().into();
backup_guard.metadata.timestamp = Some(timestamp);

10
core/src/backup/mod.rs
View File

@@ -2,6 +2,7 @@ use std::collections::BTreeMap;
use rpc_toolkit::{Context, HandlerExt, ParentHandler, from_fn_async};
use serde::{Deserialize, Serialize};
use ts_rs::TS;
use crate::PackageId;
use crate::context::CliContext;
@@ -13,19 +14,22 @@ pub mod os;
pub mod restore;
pub mod target;
#[derive(Debug, Deserialize, Serialize)]
#[derive(Debug, Deserialize, Serialize, TS)]
#[ts(export)]
pub struct BackupReport {
server: ServerBackupReport,
packages: BTreeMap<PackageId, PackageBackupReport>,
}
#[derive(Debug, Deserialize, Serialize)]
#[derive(Debug, Deserialize, Serialize, TS)]
#[ts(export)]
pub struct ServerBackupReport {
attempted: bool,
error: Option<String>,
}
#[derive(Debug, Deserialize, Serialize)]
#[derive(Debug, Deserialize, Serialize, TS)]
#[ts(export)]
pub struct PackageBackupReport {
pub error: Option<String>,
}

41
core/src/backup/os.rs
View File

@@ -6,10 +6,8 @@ use serde::{Deserialize, Serialize};
use ssh_key::private::Ed25519Keypair;
use crate::account::AccountInfo;
use crate::hostname::{Hostname, generate_hostname, generate_id};
use crate::net::tor::TorSecretKey;
use crate::hostname::{ServerHostname, ServerHostnameInfo, generate_hostname, generate_id};
use crate::prelude::*;
use crate::util::crypto::ed25519_expand_key;
use crate::util::serde::{Base32, Base64, Pem};
pub struct OsBackup {
@@ -29,10 +27,12 @@ impl<'de> Deserialize<'de> for OsBackup {
.map_err(serde::de::Error::custom)?,
1 => patch_db::value::from_value::<OsBackupV1>(tagged.rest)
.map_err(serde::de::Error::custom)?
.project(),
.project()
.map_err(serde::de::Error::custom)?,
2 => patch_db::value::from_value::<OsBackupV2>(tagged.rest)
.map_err(serde::de::Error::custom)?
.project(),
.project()
.map_err(serde::de::Error::custom)?,
v => {
return Err(serde::de::Error::custom(&format!(
"Unknown backup version {v}"
@@ -77,7 +77,7 @@ impl OsBackupV0 {
Ok(OsBackup {
account: AccountInfo {
server_id: generate_id(),
hostname: generate_hostname(),
hostname: ServerHostnameInfo::from_hostname(generate_hostname()),
password: Default::default(),
root_ca_key: self.root_ca_key.0,
root_ca_cert: self.root_ca_cert.0,
@@ -85,10 +85,6 @@ impl OsBackupV0 {
&mut ssh_key::rand_core::OsRng::default(),
ssh_key::Algorithm::Ed25519,
)?,
tor_keys: TorSecretKey::from_bytes(self.tor_key.0)
.ok()
.into_iter()
.collect(),
developer_key: ed25519_dalek::SigningKey::generate(
&mut ssh_key::rand_core::OsRng::default(),
),
@@ -110,23 +106,19 @@ struct OsBackupV1 {
ui: Value, // JSON Value
}
impl OsBackupV1 {
fn project(self) -> OsBackup {
OsBackup {
fn project(self) -> Result<OsBackup, Error> {
Ok(OsBackup {
account: AccountInfo {
server_id: self.server_id,
hostname: Hostname(self.hostname),
hostname: ServerHostnameInfo::from_hostname(ServerHostname::new(self.hostname)?),
password: Default::default(),
root_ca_key: self.root_ca_key.0,
root_ca_cert: self.root_ca_cert.0,
ssh_key: ssh_key::PrivateKey::from(Ed25519Keypair::from_seed(&self.net_key.0)),
tor_keys: TorSecretKey::from_bytes(ed25519_expand_key(&self.net_key.0))
.ok()
.into_iter()
.collect(),
developer_key: ed25519_dalek::SigningKey::from_bytes(&self.net_key),
},
ui: self.ui,
}
})
}
}
@@ -140,34 +132,31 @@ struct OsBackupV2 {
root_ca_key: Pem<PKey<Private>>, // PEM Encoded OpenSSL Key
root_ca_cert: Pem<X509>, // PEM Encoded OpenSSL X509 Certificate
ssh_key: Pem<ssh_key::PrivateKey>, // PEM Encoded OpenSSH Key
tor_keys: Vec<TorSecretKey>, // Base64 Encoded Ed25519 Expanded Secret Key
compat_s9pk_key: Pem<ed25519_dalek::SigningKey>, // PEM Encoded ED25519 Key
ui: Value, // JSON Value
}
impl OsBackupV2 {
fn project(self) -> OsBackup {
OsBackup {
fn project(self) -> Result<OsBackup, Error> {
Ok(OsBackup {
account: AccountInfo {
server_id: self.server_id,
hostname: Hostname(self.hostname),
hostname: ServerHostnameInfo::from_hostname(ServerHostname::new(self.hostname)?),
password: Default::default(),
root_ca_key: self.root_ca_key.0,
root_ca_cert: self.root_ca_cert.0,
ssh_key: self.ssh_key.0,
tor_keys: self.tor_keys,
developer_key: self.compat_s9pk_key.0,
},
ui: self.ui,
}
})
}
fn unproject(backup: &OsBackup) -> Self {
Self {
server_id: backup.account.server_id.clone(),
hostname: backup.account.hostname.0.clone(),
hostname: (*backup.account.hostname.hostname).clone(),
root_ca_key: Pem(backup.account.root_ca_key.clone()),
root_ca_cert: Pem(backup.account.root_ca_cert.clone()),
ssh_key: Pem(backup.account.ssh_key.clone()),
tor_keys: backup.account.tor_keys.clone(),
compat_s9pk_key: Pem(backup.account.developer_key.clone()),
ui: backup.ui.clone(),
}

25
core/src/backup/restore.rs
View File

@@ -17,6 +17,7 @@ use crate::db::model::Database;
use crate::disk::mount::backup::BackupMountGuard;
use crate::disk::mount::filesystem::ReadWrite;
use crate::disk::mount::guard::{GenericMountGuard, TmpMountGuard};
use crate::hostname::ServerHostnameInfo;
use crate::init::init;
use crate::prelude::*;
use crate::progress::ProgressUnits;
@@ -30,6 +31,7 @@ use crate::{PLATFORM, PackageId};
#[derive(Deserialize, Serialize, Parser, TS)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
#[ts(export)]
pub struct RestorePackageParams {
#[arg(help = "help.arg.package-ids")]
pub ids: Vec<PackageId>,
@@ -84,11 +86,12 @@ pub async fn restore_packages_rpc(
pub async fn recover_full_server(
ctx: &SetupContext,
disk_guid: InternedString,
password: String,
password: Option<String>,
recovery_source: TmpMountGuard,
server_id: &str,
recovery_password: &str,
kiosk: Option<bool>,
hostname: Option<ServerHostnameInfo>,
SetupExecuteProgress {
init_phases,
restore_phase,
@@ -107,12 +110,18 @@ pub async fn recover_full_server(
.with_ctx(|_| (ErrorKind::Filesystem, os_backup_path.display().to_string()))?,
)?;
os_backup.account.password = argon2::hash_encoded(
password.as_bytes(),
&rand::random::<[u8; 16]>()[..],
&argon2::Config::rfc9106_low_mem(),
)
.with_kind(ErrorKind::PasswordHashGeneration)?;
if let Some(password) = password {
os_backup.account.password = argon2::hash_encoded(
password.as_bytes(),
&rand::random::<[u8; 16]>()[..],
&argon2::Config::rfc9106_low_mem(),
)
.with_kind(ErrorKind::PasswordHashGeneration)?;
}
if let Some(h) = hostname {
os_backup.account.hostname = h;
}
let kiosk = Some(kiosk.unwrap_or(true)).filter(|_| &*PLATFORM != "raspberrypi");
sync_kiosk(kiosk).await?;
@@ -182,7 +191,7 @@ pub async fn recover_full_server(
Ok((
SetupResult {
hostname: os_backup.account.hostname,
hostname: os_backup.account.hostname.hostname,
root_ca: Pem(os_backup.account.root_ca_cert),
needs_restart: ctx.install_rootfs.peek(|a| a.is_some()),
},

25
core/src/backup/target/cifs.rs
View File

@@ -36,7 +36,8 @@ impl Map for CifsTargets {
}
}
#[derive(Debug, Deserialize, Serialize)]
#[derive(Debug, Deserialize, Serialize, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub struct CifsBackupTarget {
hostname: String,
@@ -72,9 +73,10 @@ pub fn cifs<C: Context>() -> ParentHandler<C> {
}
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
pub struct AddParams {
pub struct CifsAddParams {
#[arg(help = "help.arg.cifs-hostname")]
pub hostname: String,
#[arg(help = "help.arg.cifs-path")]
@@ -87,12 +89,12 @@ pub struct AddParams {
pub async fn add(
ctx: RpcContext,
AddParams {
CifsAddParams {
hostname,
path,
username,
password,
}: AddParams,
}: CifsAddParams,
) -> Result<KeyVal<BackupTargetId, BackupTarget>, Error> {
let cifs = Cifs {
hostname,
@@ -131,9 +133,10 @@ pub async fn add(
}
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
pub struct UpdateParams {
pub struct CifsUpdateParams {
#[arg(help = "help.arg.backup-target-id")]
pub id: BackupTargetId,
#[arg(help = "help.arg.cifs-hostname")]
@@ -148,13 +151,13 @@ pub struct UpdateParams {
pub async fn update(
ctx: RpcContext,
UpdateParams {
CifsUpdateParams {
id,
hostname,
path,
username,
password,
}: UpdateParams,
}: CifsUpdateParams,
) -> Result<KeyVal<BackupTargetId, BackupTarget>, Error> {
let id = if let BackupTargetId::Cifs { id } = id {
id
@@ -207,14 +210,18 @@ pub async fn update(
}
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
pub struct RemoveParams {
pub struct CifsRemoveParams {
#[arg(help = "help.arg.backup-target-id")]
pub id: BackupTargetId,
}
pub async fn remove(ctx: RpcContext, RemoveParams { id }: RemoveParams) -> Result<(), Error> {
pub async fn remove(
ctx: RpcContext,
CifsRemoveParams { id }: CifsRemoveParams,
) -> Result<(), Error> {
let id = if let BackupTargetId::Cifs { id } = id {
id
} else {

18
core/src/backup/target/mod.rs
View File

@@ -34,7 +34,8 @@ use crate::util::{FromStrParser, VersionString};
pub mod cifs;
#[derive(Debug, Deserialize, Serialize)]
#[derive(Debug, Deserialize, Serialize, TS)]
#[ts(export)]
#[serde(tag = "type")]
#[serde(rename_all = "camelCase")]
pub enum BackupTarget {
@@ -49,7 +50,7 @@ pub enum BackupTarget {
}
#[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Clone, TS)]
#[ts(type = "string")]
#[ts(export, type = "string")]
pub enum BackupTargetId {
Disk { logicalname: PathBuf },
Cifs { id: u32 },
@@ -111,6 +112,7 @@ impl Serialize for BackupTargetId {
}
#[derive(Debug, Deserialize, Serialize, TS)]
#[ts(export)]
#[serde(tag = "type")]
#[serde(rename_all = "camelCase")]
pub enum BackupTargetFS {
@@ -210,20 +212,26 @@ pub async fn list(ctx: RpcContext) -> Result<BTreeMap<BackupTargetId, BackupTarg
.collect())
}
#[derive(Clone, Debug, Default, Deserialize, Serialize)]
#[derive(Clone, Debug, Default, Deserialize, Serialize, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub struct BackupInfo {
#[ts(type = "string")]
pub version: Version,
#[ts(type = "string | null")]
pub timestamp: Option<DateTime<Utc>>,
pub package_backups: BTreeMap<PackageId, PackageBackupInfo>,
}
#[derive(Clone, Debug, Deserialize, Serialize)]
#[derive(Clone, Debug, Deserialize, Serialize, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub struct PackageBackupInfo {
pub title: InternedString,
pub version: VersionString,
#[ts(type = "string")]
pub os_version: Version,
#[ts(type = "string")]
pub timestamp: DateTime<Utc>,
}
@@ -265,6 +273,7 @@ fn display_backup_info(params: WithIoFormat<InfoParams>, info: BackupInfo) -> Re
}
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
pub struct InfoParams {
@@ -387,6 +396,7 @@ pub async fn mount(
}
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
pub struct UmountParams {

6
core/src/bins/start_init.rs
View File

@@ -9,7 +9,7 @@ use crate::disk::fsck::RepairStrategy;
use crate::disk::main::DEFAULT_PASSWORD;
use crate::firmware::{check_for_firmware_update, update_firmware};
use crate::init::{InitPhases, STANDBY_MODE_PATH};
use crate::net::gateway::UpgradableListener;
use crate::net::gateway::WildcardListener;
use crate::net::web_server::WebServer;
use crate::prelude::*;
use crate::progress::FullProgressTracker;
@@ -19,7 +19,7 @@ use crate::{DATA_DIR, PLATFORM};
#[instrument(skip_all)]
async fn setup_or_init(
server: &mut WebServer<UpgradableListener>,
server: &mut WebServer<WildcardListener>,
config: &ServerConfig,
) -> Result<Result<(RpcContext, FullProgressTracker), Shutdown>, Error> {
if let Some(firmware) = check_for_firmware_update()
@@ -204,7 +204,7 @@ async fn setup_or_init(
#[instrument(skip_all)]
pub async fn main(
server: &mut WebServer<UpgradableListener>,
server: &mut WebServer<WildcardListener>,
config: &ServerConfig,
) -> Result<Result<(RpcContext, FullProgressTracker), Shutdown>, Error> {
if &*PLATFORM == "raspberrypi" && tokio::fs::metadata(STANDBY_MODE_PATH).await.is_ok() {

12
core/src/bins/startd.rs
View File

@@ -12,7 +12,7 @@ use tracing::instrument;
use crate::context::config::ServerConfig;
use crate::context::rpc::InitRpcContextPhases;
use crate::context::{DiagnosticContext, InitContext, RpcContext};
use crate::net::gateway::{BindTcp, SelfContainedNetworkInterfaceListener, UpgradableListener};
use crate::net::gateway::WildcardListener;
use crate::net::static_server::refresher;
use crate::net::web_server::{Acceptor, WebServer};
use crate::prelude::*;
@@ -23,7 +23,7 @@ use crate::util::logger::LOGGER;
#[instrument(skip_all)]
async fn inner_main(
server: &mut WebServer<UpgradableListener>,
server: &mut WebServer<WildcardListener>,
config: &ServerConfig,
) -> Result<Option<Shutdown>, Error> {
let rpc_ctx = if !tokio::fs::metadata("/run/startos/initialized")
@@ -70,7 +70,8 @@ async fn inner_main(
};
let (rpc_ctx, shutdown) = async {
crate::hostname::sync_hostname(&rpc_ctx.account.peek(|a| a.hostname.clone())).await?;
crate::hostname::sync_hostname(&rpc_ctx.account.peek(|a| a.hostname.hostname.clone()))
.await?;
let mut shutdown_recv = rpc_ctx.shutdown.subscribe();
@@ -147,10 +148,7 @@ pub fn main(args: impl IntoIterator<Item = OsString>) {
.build()
.expect(&t!("bins.startd.failed-to-initialize-runtime"));
let res = rt.block_on(async {
let mut server = WebServer::new(
Acceptor::bind_upgradable(SelfContainedNetworkInterfaceListener::bind(BindTcp, 80)),
refresher(),
);
let mut server = WebServer::new(Acceptor::new(WildcardListener::new(80)?), refresher());
match inner_main(&mut server, &config).await {
Ok(a) => {
server.shutdown().await;

9
core/src/bins/tunnel.rs
View File

@@ -7,13 +7,13 @@ use clap::Parser;
use futures::FutureExt;
use rpc_toolkit::CliApp;
use rust_i18n::t;
use tokio::net::TcpListener;
use tokio::signal::unix::signal;
use tracing::instrument;
use visit_rs::Visit;
use crate::context::CliContext;
use crate::context::config::ClientConfig;
use crate::net::gateway::{Bind, BindTcp};
use crate::net::tls::TlsListener;
use crate::net::web_server::{Accept, Acceptor, MetadataVisitor, WebServer};
use crate::prelude::*;
@@ -57,7 +57,12 @@ async fn inner_main(config: &TunnelConfig) -> Result<(), Error> {
if !a.contains_key(&key) {
match (|| {
Ok::<_, Error>(TlsListener::new(
BindTcp.bind(addr)?,
TcpListener::from_std(
mio::net::TcpListener::bind(addr)
.with_kind(ErrorKind::Network)?
.into(),
)
.with_kind(ErrorKind::Network)?,
TunnelCertHandler {
db: https_db.clone(),
crypto_provider: Arc::new(tokio_rustls::rustls::crypto::ring::default_provider()),

89
core/src/context/rpc.rs
View File

@@ -10,7 +10,6 @@ use std::time::Duration;
use chrono::{TimeDelta, Utc};
use imbl::OrdMap;
use imbl_value::InternedString;
use itertools::Itertools;
use josekit::jwk::Jwk;
use reqwest::{Client, Proxy};
use rpc_toolkit::yajrc::RpcError;
@@ -25,7 +24,6 @@ use crate::account::AccountInfo;
use crate::auth::Sessions;
use crate::context::config::ServerConfig;
use crate::db::model::Database;
use crate::db::model::package::TaskSeverity;
use crate::disk::OsPartitionInfo;
use crate::disk::mount::filesystem::bind::Bind;
use crate::disk::mount::filesystem::block_dev::BlockDev;
@@ -34,7 +32,7 @@ use crate::disk::mount::guard::MountGuard;
use crate::init::{InitResult, check_time_is_synchronized};
use crate::install::PKG_ARCHIVE_DIR;
use crate::lxc::LxcManager;
use crate::net::gateway::UpgradableListener;
use crate::net::gateway::WildcardListener;
use crate::net::net_controller::{NetController, NetService};
use crate::net::socks::DEFAULT_SOCKS_LISTEN;
use crate::net::utils::{find_eth_iface, find_wifi_iface};
@@ -44,7 +42,6 @@ use crate::prelude::*;
use crate::progress::{FullProgressTracker, PhaseProgressTrackerHandle};
use crate::rpc_continuations::{Guid, OpenAuthedContinuations, RpcContinuations};
use crate::service::ServiceMap;
use crate::service::action::update_tasks;
use crate::service::effects::callbacks::ServiceCallbacks;
use crate::service::effects::subcontainer::NVIDIA_OVERLAY_PATH;
use crate::shutdown::Shutdown;
@@ -53,7 +50,7 @@ use crate::util::future::NonDetachingJoinHandle;
use crate::util::io::{TmpDir, delete_file};
use crate::util::lshw::LshwDevice;
use crate::util::sync::{SyncMutex, SyncRwLock, Watch};
use crate::{ActionId, DATA_DIR, PLATFORM, PackageId};
use crate::{DATA_DIR, PLATFORM, PackageId};
pub struct RpcContextSeed {
is_closed: AtomicBool,
@@ -114,7 +111,6 @@ pub struct CleanupInitPhases {
cleanup_sessions: PhaseProgressTrackerHandle,
init_services: PhaseProgressTrackerHandle,
prune_s9pks: PhaseProgressTrackerHandle,
check_tasks: PhaseProgressTrackerHandle,
}
impl CleanupInitPhases {
pub fn new(handle: &FullProgressTracker) -> Self {
@@ -122,7 +118,6 @@ impl CleanupInitPhases {
cleanup_sessions: handle.add_phase("Cleaning up sessions".into(), Some(1)),
init_services: handle.add_phase("Initializing services".into(), Some(10)),
prune_s9pks: handle.add_phase("Pruning S9PKs".into(), Some(1)),
check_tasks: handle.add_phase("Checking action requests".into(), Some(1)),
}
}
}
@@ -132,7 +127,7 @@ pub struct RpcContext(Arc<RpcContextSeed>);
impl RpcContext {
#[instrument(skip_all)]
pub async fn init(
webserver: &WebServerAcceptorSetter<UpgradableListener>,
webserver: &WebServerAcceptorSetter<WildcardListener>,
config: &ServerConfig,
disk_guid: InternedString,
init_result: Option<InitResult>,
@@ -165,16 +160,15 @@ impl RpcContext {
{
(net_ctrl, os_net_service)
} else {
let net_ctrl =
Arc::new(NetController::init(db.clone(), &account.hostname, socks_proxy).await?);
webserver.try_upgrade(|a| net_ctrl.net_iface.watcher.upgrade_listener(a))?;
let net_ctrl = Arc::new(NetController::init(db.clone(), socks_proxy).await?);
webserver.send_modify(|wl| wl.set_ip_info(net_ctrl.net_iface.watcher.subscribe()));
let os_net_service = net_ctrl.os_bindings().await?;
(net_ctrl, os_net_service)
};
init_net_ctrl.complete();
tracing::info!("{}", t!("context.rpc.initialized-net-controller"));
if PLATFORM.ends_with("-nonfree") {
if PLATFORM.ends_with("-nvidia") {
if let Err(e) = Command::new("nvidia-smi")
.invoke(ErrorKind::ParseSysInfo)
.await
@@ -412,7 +406,6 @@ impl RpcContext {
mut cleanup_sessions,
mut init_services,
mut prune_s9pks,
mut check_tasks,
}: CleanupInitPhases,
) -> Result<(), Error> {
cleanup_sessions.start();
@@ -504,76 +497,6 @@ impl RpcContext {
}
prune_s9pks.complete();
check_tasks.start();
let mut action_input: OrdMap<PackageId, BTreeMap<ActionId, Value>> = OrdMap::new();
let tasks: BTreeSet<_> = peek
.as_public()
.as_package_data()
.as_entries()?
.into_iter()
.map(|(_, pde)| {
Ok(pde
.as_tasks()
.as_entries()?
.into_iter()
.map(|(_, r)| {
let t = r.as_task();
Ok::<_, Error>(if t.as_input().transpose_ref().is_some() {
Some((t.as_package_id().de()?, t.as_action_id().de()?))
} else {
None
})
})
.filter_map_ok(|a| a))
})
.flatten_ok()
.map(|a| a.and_then(|a| a))
.try_collect()?;
let procedure_id = Guid::new();
for (package_id, action_id) in tasks {
if let Some(service) = self.services.get(&package_id).await.as_ref() {
if let Some(input) = service
.get_action_input(procedure_id.clone(), action_id.clone())
.await
.log_err()
.flatten()
.and_then(|i| i.value)
{
action_input
.entry(package_id)
.or_default()
.insert(action_id, input);
}
}
}
self.db
.mutate(|db| {
for (package_id, action_input) in &action_input {
for (action_id, input) in action_input {
for (_, pde) in db.as_public_mut().as_package_data_mut().as_entries_mut()? {
pde.as_tasks_mut().mutate(|tasks| {
Ok(update_tasks(tasks, package_id, action_id, input, false))
})?;
}
}
}
for (_, pde) in db.as_public_mut().as_package_data_mut().as_entries_mut()? {
if pde
.as_tasks()
.de()?
.into_iter()
.any(|(_, t)| t.active && t.task.severity == TaskSeverity::Critical)
{
pde.as_status_info_mut().stop()?;
}
}
Ok(())
})
.await
.result?;
check_tasks.complete();
Ok(())
}
pub async fn call_remote<RemoteContext>(

10
core/src/context/setup.rs
View File

@@ -19,8 +19,8 @@ use crate::MAIN_DATA;
use crate::context::RpcContext;
use crate::context::config::ServerConfig;
use crate::disk::mount::guard::{MountGuard, TmpMountGuard};
use crate::hostname::Hostname;
use crate::net::gateway::UpgradableListener;
use crate::hostname::ServerHostname;
use crate::net::gateway::WildcardListener;
use crate::net::web_server::{WebServer, WebServerAcceptorSetter};
use crate::prelude::*;
use crate::progress::FullProgressTracker;
@@ -45,13 +45,13 @@ lazy_static::lazy_static! {
#[ts(export)]
pub struct SetupResult {
#[ts(type = "string")]
pub hostname: Hostname,
pub hostname: ServerHostname,
pub root_ca: Pem<X509>,
pub needs_restart: bool,
}
pub struct SetupContextSeed {
pub webserver: WebServerAcceptorSetter<UpgradableListener>,
pub webserver: WebServerAcceptorSetter<WildcardListener>,
pub config: SyncMutex<ServerConfig>,
pub disable_encryption: bool,
pub progress: FullProgressTracker,
@@ -70,7 +70,7 @@ pub struct SetupContext(Arc<SetupContextSeed>);
impl SetupContext {
#[instrument(skip_all)]
pub fn init(
webserver: &WebServer<UpgradableListener>,
webserver: &WebServer<WildcardListener>,
config: ServerConfig,
) -> Result<Self, Error> {
let (shutdown, _) = tokio::sync::broadcast::channel(1);

1
core/src/control.rs
View File

@@ -8,6 +8,7 @@ use crate::prelude::*;
use crate::{Error, PackageId};
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
pub struct ControlParams {

7
core/src/db/model/mod.rs
View File

@@ -45,7 +45,12 @@ impl Database {
.collect(),
ssh_privkey: Pem(account.ssh_key.clone()),
ssh_pubkeys: SshKeys::new(),
available_ports: AvailablePorts::new(),
available_ports: {
let mut ports = AvailablePorts::new();
ports.set_ssl(80, false);
ports.set_ssl(443, true);
ports
},
sessions: Sessions::new(),
notifications: Notifications::new(),
cifs: CifsTargets::new(),

21
core/src/db/model/package.rs
View File

@@ -18,7 +18,7 @@ use crate::s9pk::manifest::{LocaleString, Manifest};
use crate::status::StatusInfo;
use crate::util::DataUrl;
use crate::util::serde::{Pem, is_partial_of};
use crate::{ActionId, HealthCheckId, HostId, PackageId, ReplayId, ServiceInterfaceId};
use crate::{ActionId, GatewayId, HealthCheckId, HostId, PackageId, ReplayId, ServiceInterfaceId};
#[derive(Debug, Default, Deserialize, Serialize, TS)]
#[ts(export)]
@@ -381,6 +381,10 @@ pub struct PackageDataEntry {
pub hosts: Hosts,
#[ts(type = "string[]")]
pub store_exposed_dependents: Vec<JsonPointer>,
#[ts(type = "string | null")]
pub outbound_gateway: Option<GatewayId>,
#[serde(default)]
pub plugin: PackagePlugin,
}
impl AsRef<PackageDataEntry> for PackageDataEntry {
fn as_ref(&self) -> &PackageDataEntry {
@@ -388,6 +392,21 @@ impl AsRef<PackageDataEntry> for PackageDataEntry {
}
}
#[derive(Debug, Default, Deserialize, Serialize, HasModel, TS)]
#[serde(rename_all = "camelCase")]
#[model = "Model<Self>"]
#[ts(export)]
pub struct PackagePlugin {
pub url: Option<UrlPluginRegistration>,
}
#[derive(Debug, Clone, Deserialize, Serialize, TS)]
#[serde(rename_all = "camelCase")]
#[ts(export)]
pub struct UrlPluginRegistration {
pub table_action: ActionId,
}
#[derive(Debug, Clone, Default, Deserialize, Serialize, TS)]
#[ts(export)]
pub struct CurrentDependencies(pub BTreeMap<PackageId, CurrentDependencyInfo>);

136
core/src/db/model/public.rs
View File

@@ -13,6 +13,7 @@ use openssl::hash::MessageDigest;
use patch_db::{HasModel, Value};
use serde::{Deserialize, Serialize};
use ts_rs::TS;
use url::Url;
use crate::account::AccountInfo;
use crate::db::DbAccessByKey;
@@ -20,8 +21,9 @@ use crate::db::model::Database;
use crate::db::model::package::AllPackageData;
use crate::net::acme::AcmeProvider;
use crate::net::host::Host;
use crate::net::host::binding::{AddSslOptions, BindInfo, BindOptions, NetInfo};
use crate::net::utils::ipv6_is_local;
use crate::net::host::binding::{
AddSslOptions, BindInfo, BindOptions, Bindings, DerivedAddressInfo, NetInfo,
};
use crate::net::vhost::AlpnInfo;
use crate::prelude::*;
use crate::progress::FullProgress;
@@ -57,42 +59,43 @@ impl Public {
platform: get_platform(),
id: account.server_id.clone(),
version: Current::default().semver(),
hostname: account.hostname.no_dot_host_name(),
name: account.hostname.name.clone(),
hostname: (*account.hostname.hostname).clone(),
last_backup: None,
package_version_compat: Current::default().compat().clone(),
post_init_migration_todos: BTreeMap::new(),
network: NetworkInfo {
host: Host {
bindings: [(
80,
BindInfo {
enabled: false,
options: BindOptions {
preferred_external_port: 80,
add_ssl: Some(AddSslOptions {
preferred_external_port: 443,
add_x_forwarded_headers: false,
alpn: Some(AlpnInfo::Specified(vec![
MaybeUtf8String("h2".into()),
MaybeUtf8String("http/1.1".into()),
])),
}),
secure: None,
bindings: Bindings(
[(
80,
BindInfo {
enabled: false,
options: BindOptions {
preferred_external_port: 80,
add_ssl: Some(AddSslOptions {
preferred_external_port: 443,
add_x_forwarded_headers: false,
alpn: Some(AlpnInfo::Specified(vec![
MaybeUtf8String("h2".into()),
MaybeUtf8String("http/1.1".into()),
])),
}),
secure: None,
},
net: NetInfo {
assigned_port: None,
assigned_ssl_port: Some(443),
},
addresses: DerivedAddressInfo::default(),
},
net: NetInfo {
assigned_port: None,
assigned_ssl_port: Some(443),
private_disabled: OrdSet::new(),
public_enabled: OrdSet::new(),
},
},
)]
.into_iter()
.collect(),
onions: account.tor_keys.iter().map(|k| k.onion_address()).collect(),
)]
.into_iter()
.collect(),
),
public_domains: BTreeMap::new(),
private_domains: BTreeSet::new(),
hostname_info: BTreeMap::new(),
private_domains: BTreeMap::new(),
port_forwards: BTreeSet::new(),
},
wifi: WifiInfo {
enabled: true,
@@ -117,6 +120,7 @@ impl Public {
acme
},
dns: Default::default(),
default_outbound: None,
},
status_info: ServerStatus {
backup_progress: None,
@@ -141,6 +145,7 @@ impl Public {
zram: true,
governor: None,
smtp: None,
ifconfig_url: default_ifconfig_url(),
ram: 0,
devices: Vec::new(),
kiosk,
@@ -162,19 +167,21 @@ fn get_platform() -> InternedString {
(&*PLATFORM).into()
}
pub fn default_ifconfig_url() -> Url {
"https://ifconfig.co".parse().unwrap()
}
#[derive(Debug, Deserialize, Serialize, HasModel, TS)]
#[serde(rename_all = "camelCase")]
#[model = "Model<Self>"]
#[ts(export)]
pub struct ServerInfo {
#[serde(default = "get_arch")]
#[ts(type = "string")]
pub arch: InternedString,
#[serde(default = "get_platform")]
#[ts(type = "string")]
pub platform: InternedString,
pub id: String,
#[ts(type = "string")]
pub name: InternedString,
pub hostname: InternedString,
#[ts(type = "string")]
pub version: Version,
@@ -198,6 +205,9 @@ pub struct ServerInfo {
pub zram: bool,
pub governor: Option<Governor>,
pub smtp: Option<SmtpValue>,
#[serde(default = "default_ifconfig_url")]
#[ts(type = "string")]
pub ifconfig_url: Url,
#[ts(type = "number")]
pub ram: u64,
pub devices: Vec<LshwDevice>,
@@ -220,6 +230,9 @@ pub struct NetworkInfo {
pub acme: BTreeMap<AcmeProvider, AcmeSettings>,
#[serde(default)]
pub dns: DnsSettings,
#[serde(default)]
#[ts(type = "string | null")]
pub default_outbound: Option<GatewayId>,
}
#[derive(Debug, Default, Deserialize, Serialize, HasModel, TS)]
@@ -239,41 +252,12 @@ pub struct DnsSettings {
#[ts(export)]
pub struct NetworkInterfaceInfo {
pub name: Option<InternedString>,
pub public: Option<bool>,
pub secure: Option<bool>,
pub ip_info: Option<Arc<IpInfo>>,
#[serde(default, rename = "type")]
pub gateway_type: Option<GatewayType>,
}
impl NetworkInterfaceInfo {
pub fn public(&self) -> bool {
self.public.unwrap_or_else(|| {
!self.ip_info.as_ref().map_or(true, |ip_info| {
let ip4s = ip_info
.subnets
.iter()
.filter_map(|ipnet| {
if let IpAddr::V4(ip4) = ipnet.addr() {
Some(ip4)
} else {
None
}
})
.collect::<BTreeSet<_>>();
if !ip4s.is_empty() {
return ip4s
.iter()
.all(|ip4| ip4.is_loopback() || ip4.is_private() || ip4.is_link_local());
}
ip_info.subnets.iter().all(|ipnet| {
if let IpAddr::V6(ip6) = ipnet.addr() {
ipv6_is_local(ip6)
} else {
true
}
})
})
})
}
pub fn secure(&self) -> bool {
self.secure.unwrap_or(false)
}
@@ -310,6 +294,28 @@ pub enum NetworkInterfaceType {
Loopback,
}
#[derive(
Clone,
Copy,
Debug,
Default,
PartialEq,
Eq,
PartialOrd,
Ord,
Deserialize,
Serialize,
TS,
clap::ValueEnum,
)]
#[ts(export)]
#[serde(rename_all = "kebab-case")]
pub enum GatewayType {
#[default]
InboundOutbound,
OutboundOnly,
}
#[derive(Debug, Deserialize, Serialize, HasModel, TS)]
#[serde(rename_all = "camelCase")]
#[model = "Model<Self>"]

5
core/src/dependencies.rs
View File

@@ -45,7 +45,7 @@ impl TS for DepInfo {
"DepInfo".into()
}
fn inline() -> String {
"{ description: string | null, optional: boolean } & MetadataSrc".into()
"{ description: LocaleString | null, optional: boolean } & MetadataSrc".into()
}
fn inline_flattened() -> String {
Self::inline()
@@ -54,7 +54,8 @@ impl TS for DepInfo {
where
Self: 'static,
{
v.visit::<MetadataSrc>()
v.visit::<MetadataSrc>();
v.visit::<LocaleString>();
}
fn output_path() -> Option<&'static std::path::Path> {
Some(Path::new("DepInfo.ts"))

14
core/src/disk/util.rs
View File

@@ -19,7 +19,7 @@ use super::mount::filesystem::block_dev::BlockDev;
use super::mount::guard::TmpMountGuard;
use crate::disk::OsPartitionInfo;
use crate::disk::mount::guard::GenericMountGuard;
use crate::hostname::Hostname;
use crate::hostname::ServerHostname;
use crate::prelude::*;
use crate::util::Invoke;
use crate::util::serde::IoFormat;
@@ -43,22 +43,28 @@ pub struct DiskInfo {
pub guid: Option<InternedString>,
}
#[derive(Clone, Debug, Deserialize, Serialize)]
#[derive(Clone, Debug, Deserialize, Serialize, ts_rs::TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub struct PartitionInfo {
pub logicalname: PathBuf,
pub label: Option<String>,
#[ts(type = "number")]
pub capacity: u64,
#[ts(type = "number | null")]
pub used: Option<u64>,
pub start_os: BTreeMap<String, StartOsRecoveryInfo>,
pub guid: Option<InternedString>,
}
#[derive(Clone, Debug, Default, Deserialize, Serialize)]
#[derive(Clone, Debug, Default, Deserialize, Serialize, ts_rs::TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub struct StartOsRecoveryInfo {
pub hostname: Hostname,
pub hostname: ServerHostname,
#[ts(type = "string")]
pub version: exver::Version,
#[ts(type = "string")]
pub timestamp: DateTime<Utc>,
pub password_hash: Option<String>,
pub wrapped_key: Option<String>,

77
core/src/error.rs
View File

@@ -3,6 +3,7 @@ use std::fmt::{Debug, Display};
use axum::http::StatusCode;
use axum::http::uri::InvalidUri;
use color_eyre::eyre::eyre;
use imbl_value::InternedString;
use num_enum::TryFromPrimitive;
use patch_db::Value;
use rpc_toolkit::reqwest;
@@ -42,11 +43,11 @@ pub enum ErrorKind {
ParseUrl = 19,
DiskNotAvailable = 20,
BlockDevice = 21,
InvalidOnionAddress = 22,
// InvalidOnionAddress = 22,
Pack = 23,
ValidateS9pk = 24,
DiskCorrupted = 25, // Remove
Tor = 26,
// Tor = 26,
ConfigGen = 27,
ParseNumber = 28,
Database = 29,
@@ -126,11 +127,11 @@ impl ErrorKind {
ParseUrl => t!("error.parse-url"),
DiskNotAvailable => t!("error.disk-not-available"),
BlockDevice => t!("error.block-device"),
InvalidOnionAddress => t!("error.invalid-onion-address"),
// InvalidOnionAddress => t!("error.invalid-onion-address"),
Pack => t!("error.pack"),
ValidateS9pk => t!("error.validate-s9pk"),
DiskCorrupted => t!("error.disk-corrupted"), // Remove
Tor => t!("error.tor"),
// Tor => t!("error.tor"),
ConfigGen => t!("error.config-gen"),
ParseNumber => t!("error.parse-number"),
Database => t!("error.database"),
@@ -204,17 +205,12 @@ pub struct Error {
impl Display for Error {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{}: {:#}", &self.kind.as_str(), self.source)
write!(f, "{}: {}", &self.kind.as_str(), self.display_src())
}
}
impl Debug for Error {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(
f,
"{}: {:?}",
&self.kind.as_str(),
self.debug.as_ref().unwrap_or(&self.source)
)
write!(f, "{}: {}", &self.kind.as_str(), self.display_dbg())
}
}
impl Error {
@@ -235,8 +231,13 @@ impl Error {
}
pub fn clone_output(&self) -> Self {
Error {
source: eyre!("{}", self.source),
debug: self.debug.as_ref().map(|e| eyre!("{e}")),
source: eyre!("{:#}", self.source),
debug: Some(
self.debug
.as_ref()
.map(|e| eyre!("{e}"))
.unwrap_or_else(|| eyre!("{:?}", self.source)),
),
kind: self.kind,
info: self.info.clone(),
task: None,
@@ -257,6 +258,30 @@ impl Error {
self.task.take();
self
}
pub fn display_src(&self) -> impl Display {
struct D<'a>(&'a Error);
impl<'a> Display for D<'a> {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{:#}", self.0.source)
}
}
D(self)
}
pub fn display_dbg(&self) -> impl Display {
struct D<'a>(&'a Error);
impl<'a> Display for D<'a> {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
if let Some(debug) = &self.0.debug {
write!(f, "{}", debug)
} else {
write!(f, "{:?}", self.0.source)
}
}
}
D(self)
}
}
impl axum::response::IntoResponse for Error {
fn into_response(self) -> axum::response::Response {
@@ -370,17 +395,6 @@ impl From<reqwest::Error> for Error {
Error::new(e, kind)
}
}
#[cfg(feature = "arti")]
impl From<arti_client::Error> for Error {
fn from(e: arti_client::Error) -> Self {
Error::new(e, ErrorKind::Tor)
}
}
impl From<torut::control::ConnError> for Error {
fn from(e: torut::control::ConnError) -> Self {
Error::new(e, ErrorKind::Tor)
}
}
impl From<zbus::Error> for Error {
fn from(e: zbus::Error) -> Self {
Error::new(e, ErrorKind::DBus)
@@ -444,9 +458,11 @@ impl Debug for ErrorData {
impl std::error::Error for ErrorData {}
impl From<Error> for ErrorData {
fn from(value: Error) -> Self {
let details = value.display_src().to_string();
let debug = value.display_dbg().to_string();
Self {
details: value.to_string(),
debug: format!("{:?}", value),
details,
debug,
info: value.info,
}
}
@@ -634,13 +650,10 @@ impl<T> ResultExt<T, Error> for Result<T, Error> {
fn with_ctx<F: FnOnce(&Error) -> (ErrorKind, D), D: Display>(self, f: F) -> Result<T, Error> {
self.map_err(|e| {
let (kind, ctx) = f(&e);
let ctx = InternedString::from_display(&ctx);
let source = e.source;
let with_ctx = format!("{ctx}: {source}");
let source = source.wrap_err(with_ctx);
let debug = e.debug.map(|e| {
let with_ctx = format!("{ctx}: {e}");
e.wrap_err(with_ctx)
});
let source = source.wrap_err(ctx.clone());
let debug = e.debug.map(|e| e.wrap_err(ctx));
Error {
kind,
source,

250
core/src/hostname.rs
View File

@@ -1,25 +1,58 @@
use clap::Parser;
use imbl_value::InternedString;
use lazy_format::lazy_format;
use rand::{Rng, rng};
use serde::{Deserialize, Serialize};
use tokio::process::Command;
use tracing::instrument;
use ts_rs::TS;
use crate::context::RpcContext;
use crate::db::model::public::ServerInfo;
use crate::prelude::*;
use crate::util::Invoke;
use crate::{Error, ErrorKind};
#[derive(Clone, Debug, Default, serde::Deserialize, serde::Serialize)]
pub struct Hostname(pub InternedString);
lazy_static::lazy_static! {
static ref ADJECTIVES: Vec<String> = include_str!("./assets/adjectives.txt").lines().map(|x| x.to_string()).collect();
static ref NOUNS: Vec<String> = include_str!("./assets/nouns.txt").lines().map(|x| x.to_string()).collect();
}
impl AsRef<str> for Hostname {
fn as_ref(&self) -> &str {
#[derive(Clone, Debug, Default, serde::Deserialize, serde::Serialize, ts_rs::TS)]
#[ts(type = "string")]
pub struct ServerHostname(InternedString);
impl std::ops::Deref for ServerHostname {
type Target = InternedString;
fn deref(&self) -> &Self::Target {
&self.0
}
}
impl AsRef<str> for ServerHostname {
fn as_ref(&self) -> &str {
&***self
}
}
impl ServerHostname {
fn validate(&self) -> Result<(), Error> {
if self.0.is_empty() {
return Err(Error::new(
eyre!("{}", t!("hostname.empty")),
ErrorKind::InvalidRequest,
));
}
if let Some(c) = self
.0
.chars()
.find(|c| !(c.is_ascii_alphanumeric() || c == &'-') || c.is_ascii_uppercase())
{
return Err(Error::new(
eyre!("{}", t!("hostname.invalid-character", char = c)),
ErrorKind::InvalidRequest,
));
}
Ok(())
}
pub fn new(hostname: InternedString) -> Result<Self, Error> {
let res = Self(hostname);
res.validate()?;
Ok(res)
}
impl Hostname {
pub fn lan_address(&self) -> InternedString {
InternedString::from_display(&lazy_format!("https://{}.local", self.0))
}
@@ -28,17 +61,135 @@ impl Hostname {
InternedString::from_display(&lazy_format!("{}.local", self.0))
}
pub fn no_dot_host_name(&self) -> InternedString {
self.0.clone()
pub fn load(server_info: &Model<ServerInfo>) -> Result<Self, Error> {
Ok(Self(server_info.as_hostname().de()?))
}
pub fn save(&self, server_info: &mut Model<ServerInfo>) -> Result<(), Error> {
server_info.as_hostname_mut().ser(&**self)
}
}
pub fn generate_hostname() -> Hostname {
let mut rng = rng();
let adjective = &ADJECTIVES[rng.random_range(0..ADJECTIVES.len())];
let noun = &NOUNS[rng.random_range(0..NOUNS.len())];
Hostname(InternedString::from_display(&lazy_format!(
"{adjective}-{noun}"
#[derive(Clone, Debug, Default, serde::Deserialize, serde::Serialize, ts_rs::TS)]
#[ts(type = "string")]
pub struct ServerHostnameInfo {
pub name: InternedString,
pub hostname: ServerHostname,
}
lazy_static::lazy_static! {
static ref ADJECTIVES: Vec<String> = include_str!("./assets/adjectives.txt").lines().map(|x| x.to_string()).collect();
static ref NOUNS: Vec<String> = include_str!("./assets/nouns.txt").lines().map(|x| x.to_string()).collect();
}
impl AsRef<str> for ServerHostnameInfo {
fn as_ref(&self) -> &str {
&self.hostname
}
}
fn normalize(s: &str) -> InternedString {
let mut prev_was_dash = true;
let mut normalized = s
.chars()
.filter_map(|c| {
if c.is_alphanumeric() {
prev_was_dash = false;
Some(c.to_ascii_lowercase())
} else if (c == '-' || c.is_whitespace()) && !prev_was_dash {
prev_was_dash = true;
Some('-')
} else {
None
}
})
.collect::<String>();
while normalized.ends_with('-') {
normalized.pop();
}
if normalized.len() < 4 {
generate_hostname().0
} else {
normalized.into()
}
}
fn denormalize(s: &str) -> InternedString {
let mut cap = true;
s.chars()
.map(|c| {
if c == '-' {
cap = true;
' '
} else if cap {
cap = false;
c.to_ascii_uppercase()
} else {
c
}
})
.collect::<String>()
.into()
}
impl ServerHostnameInfo {
pub fn new(
name: Option<InternedString>,
hostname: Option<InternedString>,
) -> Result<Self, Error> {
Self::new_opt(name, hostname)
.map(|h| h.unwrap_or_else(|| ServerHostnameInfo::from_hostname(generate_hostname())))
}
pub fn new_opt(
name: Option<InternedString>,
hostname: Option<InternedString>,
) -> Result<Option<Self>, Error> {
let name = name.filter(|n| !n.is_empty());
let hostname = hostname.filter(|h| !h.is_empty());
Ok(match (name, hostname) {
(Some(name), Some(hostname)) => Some(ServerHostnameInfo {
name,
hostname: ServerHostname::new(hostname)?,
}),
(Some(name), None) => Some(ServerHostnameInfo::from_name(name)),
(None, Some(hostname)) => Some(ServerHostnameInfo::from_hostname(ServerHostname::new(
hostname,
)?)),
(None, None) => None,
})
}
pub fn from_hostname(hostname: ServerHostname) -> Self {
Self {
name: denormalize(&**hostname),
hostname,
}
}
pub fn from_name(name: InternedString) -> Self {
Self {
hostname: ServerHostname(normalize(&*name)),
name,
}
}
pub fn load(server_info: &Model<ServerInfo>) -> Result<Self, Error> {
Ok(Self {
name: server_info.as_name().de()?,
hostname: ServerHostname::load(server_info)?,
})
}
pub fn save(&self, server_info: &mut Model<ServerInfo>) -> Result<(), Error> {
server_info.as_name_mut().ser(&self.name)?;
self.hostname.save(server_info)
}
}
pub fn generate_hostname() -> ServerHostname {
let num = rand::random::<u16>();
ServerHostname(InternedString::from_display(&lazy_format!(
"startos-{num:04x}"
)))
}
@@ -48,17 +199,17 @@ pub fn generate_id() -> String {
}
#[instrument(skip_all)]
pub async fn get_current_hostname() -> Result<Hostname, Error> {
pub async fn get_current_hostname() -> Result<InternedString, Error> {
let out = Command::new("hostname")
.invoke(ErrorKind::ParseSysInfo)
.await?;
let out_string = String::from_utf8(out)?;
Ok(Hostname(out_string.trim().into()))
Ok(out_string.trim().into())
}
#[instrument(skip_all)]
pub async fn set_hostname(hostname: &Hostname) -> Result<(), Error> {
let hostname = &*hostname.0;
pub async fn set_hostname(hostname: &ServerHostname) -> Result<(), Error> {
let hostname = &***hostname;
Command::new("hostnamectl")
.arg("--static")
.arg("set-hostname")
@@ -77,7 +228,7 @@ pub async fn set_hostname(hostname: &Hostname) -> Result<(), Error> {
}
#[instrument(skip_all)]
pub async fn sync_hostname(hostname: &Hostname) -> Result<(), Error> {
pub async fn sync_hostname(hostname: &ServerHostname) -> Result<(), Error> {
set_hostname(hostname).await?;
Command::new("systemctl")
.arg("restart")
@@ -86,3 +237,54 @@ pub async fn sync_hostname(hostname: &Hostname) -> Result<(), Error> {
.await?;
Ok(())
}
#[derive(Deserialize, Serialize, Parser, TS)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
#[ts(export)]
pub struct SetServerHostnameParams {
name: Option<InternedString>,
hostname: Option<InternedString>,
}
pub async fn set_hostname_rpc(
ctx: RpcContext,
SetServerHostnameParams { name, hostname }: SetServerHostnameParams,
) -> Result<(), Error> {
let name = name.filter(|n| !n.is_empty());
let hostname = hostname
.filter(|h| !h.is_empty())
.map(ServerHostname::new)
.transpose()?;
if name.is_none() && hostname.is_none() {
return Err(Error::new(
eyre!("{}", t!("hostname.must-provide-name-or-hostname")),
ErrorKind::InvalidRequest,
));
};
let info = ctx
.db
.mutate(|db| {
let server_info = db.as_public_mut().as_server_info_mut();
if let Some(name) = name {
server_info.as_name_mut().ser(&name)?;
}
if let Some(hostname) = &hostname {
hostname.save(server_info)?;
}
ServerHostnameInfo::load(server_info)
})
.await
.result?;
ctx.account.mutate(|a| a.hostname = info.clone());
if let Some(h) = hostname {
sync_hostname(&h).await?;
}
Ok(())
}
#[test]
fn test_generate_hostname() {
assert_eq!(dbg!(generate_hostname().0).len(), 12);
}

20
core/src/init.rs
View File

@@ -18,9 +18,9 @@ use crate::context::{CliContext, InitContext, RpcContext};
use crate::db::model::Database;
use crate::db::model::public::ServerStatus;
use crate::developer::OS_DEVELOPER_KEY_PATH;
use crate::hostname::Hostname;
use crate::hostname::ServerHostname;
use crate::middleware::auth::local::LocalAuthContext;
use crate::net::gateway::UpgradableListener;
use crate::net::gateway::WildcardListener;
use crate::net::net_controller::{NetController, NetService};
use crate::net::socks::DEFAULT_SOCKS_LISTEN;
use crate::net::utils::find_wifi_iface;
@@ -144,7 +144,7 @@ pub async fn run_script<P: AsRef<Path>>(path: P, mut progress: PhaseProgressTrac
#[instrument(skip_all)]
pub async fn init(
webserver: &WebServerAcceptorSetter<UpgradableListener>,
webserver: &WebServerAcceptorSetter<WildcardListener>,
cfg: &ServerConfig,
InitPhases {
preinit,
@@ -191,15 +191,16 @@ pub async fn init(
.arg(OS_DEVELOPER_KEY_PATH)
.invoke(ErrorKind::Filesystem)
.await?;
let hostname = ServerHostname::load(peek.as_public().as_server_info())?;
crate::ssh::sync_keys(
&Hostname(peek.as_public().as_server_info().as_hostname().de()?),
&hostname,
&peek.as_private().as_ssh_privkey().de()?,
&peek.as_private().as_ssh_pubkeys().de()?,
SSH_DIR,
)
.await?;
crate::ssh::sync_keys(
&Hostname(peek.as_public().as_server_info().as_hostname().de()?),
&hostname,
&peek.as_private().as_ssh_privkey().de()?,
&Default::default(),
"/root/.ssh",
@@ -211,14 +212,9 @@ pub async fn init(
start_net.start();
let net_ctrl = Arc::new(
NetController::init(
db.clone(),
&account.hostname,
cfg.socks_listen.unwrap_or(DEFAULT_SOCKS_LISTEN),
)
.await?,
NetController::init(db.clone(), cfg.socks_listen.unwrap_or(DEFAULT_SOCKS_LISTEN)).await?,
);
webserver.try_upgrade(|a| net_ctrl.net_iface.watcher.upgrade_listener(a))?;
webserver.send_modify(|wl| wl.set_ip_info(net_ctrl.net_iface.watcher.subscribe()));
let os_net_service = net_ctrl.os_bindings().await?;
start_net.complete();

4
core/src/install/mod.rs
View File

@@ -177,6 +177,7 @@ pub async fn install(
}
#[derive(Deserialize, Serialize, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub struct SideloadParams {
#[ts(skip)]
@@ -185,6 +186,7 @@ pub struct SideloadParams {
}
#[derive(Deserialize, Serialize, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub struct SideloadResponse {
pub upload: Guid,
@@ -284,6 +286,7 @@ pub async fn sideload(
}
#[derive(Debug, Clone, Deserialize, Serialize, Parser, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
pub struct CancelInstallParams {
@@ -521,6 +524,7 @@ pub async fn cli_install(
}
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
pub struct UninstallParams {

37
core/src/lib.rs
View File

@@ -25,6 +25,9 @@ pub fn platform_to_arch(platform: &str) -> &str {
if let Some(arch) = platform.strip_suffix("-nonfree") {
return arch;
}
if let Some(arch) = platform.strip_suffix("-nvidia") {
return arch;
}
match platform {
"raspberrypi" | "rockchip64" => "aarch64",
_ => platform,
@@ -268,6 +271,18 @@ pub fn server<C: Context>() -> ParentHandler<C> {
.with_about("about.display-time-uptime")
.with_call_remote::<CliContext>(),
)
.subcommand(
"device-info",
ParentHandler::<C, WithIoFormat<Empty>>::new().root_handler(
from_fn_async(system::device_info)
.with_display_serializable()
.with_custom_display_fn(|handle, result| {
system::display_device_info(handle.params, result)
})
.with_about("about.get-device-info")
.with_call_remote::<CliContext>(),
),
)
.subcommand(
"experimental",
system::experimental::<C>().with_about("about.commands-experimental"),
@@ -377,6 +392,20 @@ pub fn server<C: Context>() -> ParentHandler<C> {
"host",
net::host::server_host_api::<C>().with_about("about.commands-host-system-ui"),
)
.subcommand(
"set-hostname",
from_fn_async(hostname::set_hostname_rpc)
.no_display()
.with_about("about.set-hostname")
.with_call_remote::<CliContext>(),
)
.subcommand(
"set-ifconfig-url",
from_fn_async(system::set_ifconfig_url)
.no_display()
.with_about("about.set-ifconfig-url")
.with_call_remote::<CliContext>(),
)
.subcommand(
"set-keyboard",
from_fn_async(system::set_keyboard)
@@ -548,4 +577,12 @@ pub fn package<C: Context>() -> ParentHandler<C> {
"host",
net::host::host_api::<C>().with_about("about.manage-network-hosts-package"),
)
.subcommand(
"set-outbound-gateway",
from_fn_async(net::gateway::set_outbound_gateway)
.with_metadata("sync_db", Value::Bool(true))
.no_display()
.with_about("about.set-outbound-gateway-package")
.with_call_remote::<CliContext>(),
)
}

19
core/src/logs.rs
View File

@@ -24,6 +24,7 @@ use tokio::process::{Child, Command};
use tokio_stream::wrappers::LinesStream;
use tokio_tungstenite::tungstenite::Message;
use tracing::instrument;
use ts_rs::TS;
use crate::PackageId;
use crate::context::{CliContext, RpcContext};
@@ -109,23 +110,28 @@ async fn ws_handler(
}
}
#[derive(serde::Serialize, serde::Deserialize, Debug, Clone)]
#[derive(serde::Serialize, serde::Deserialize, Debug, Clone, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub struct LogResponse {
#[ts(as = "Vec<LogEntry>")]
pub entries: Reversible<LogEntry>,
start_cursor: Option<String>,
end_cursor: Option<String>,
}
#[derive(serde::Serialize, serde::Deserialize, Debug, Clone)]
#[derive(serde::Serialize, serde::Deserialize, Debug, Clone, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub struct LogFollowResponse {
start_cursor: Option<String>,
guid: Guid,
}
#[derive(serde::Serialize, serde::Deserialize, Debug, Clone)]
#[derive(serde::Serialize, serde::Deserialize, Debug, Clone, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub struct LogEntry {
#[ts(type = "string")]
timestamp: DateTime<Utc>,
message: String,
boot_id: String,
@@ -321,14 +327,17 @@ impl From<BootIdentifier> for String {
}
}
#[derive(Deserialize, Serialize, Parser)]
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export, concrete(Extra = Empty), bound = "")]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
pub struct LogsParams<Extra: FromArgMatches + Args = Empty> {
#[command(flatten)]
#[serde(flatten)]
#[ts(skip)]
extra: Extra,
#[arg(short = 'l', long = "limit", help = "help.arg.log-limit")]
#[ts(optional)]
limit: Option<usize>,
#[arg(
short = 'c',
@@ -336,9 +345,11 @@ pub struct LogsParams<Extra: FromArgMatches + Args = Empty> {
conflicts_with = "follow",
help = "help.arg.log-cursor"
)]
#[ts(optional)]
cursor: Option<String>,
#[arg(short = 'b', long = "boot", help = "help.arg.log-boot")]
#[serde(default)]
#[ts(optional, type = "number | string")]
boot: Option<BootIdentifier>,
#[arg(
short = 'B',

3
core/src/lxc/config.template
View File

@@ -17,3 +17,6 @@ lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.rootfs.options = rshared
# Environment
lxc.environment = LANG={lang}

14
core/src/lxc/mod.rs
View File

@@ -174,10 +174,15 @@ impl LxcContainer {
config: LxcConfig,
) -> Result<Self, Error> {
let guid = new_guid();
let lang = std::env::var("LANG").unwrap_or_else(|_| "C.UTF-8".into());
let machine_id = hex::encode(rand::random::<[u8; 16]>());
let container_dir = Path::new(LXC_CONTAINER_DIR).join(&*guid);
tokio::fs::create_dir_all(&container_dir).await?;
let config_str = format!(include_str!("./config.template"), guid = &*guid);
let config_str = format!(
include_str!("./config.template"),
guid = &*guid,
lang = &lang,
);
tokio::fs::write(container_dir.join("config"), config_str).await?;
let rootfs_dir = container_dir.join("rootfs");
let rootfs = OverlayGuard::mount(
@@ -215,6 +220,13 @@ impl LxcContainer {
100000,
)
.await?;
write_file_owned_atomic(
rootfs_dir.join("etc/default/locale"),
format!("LANG={lang}\n"),
100000,
100000,
)
.await?;
Command::new("sed")
.arg("-i")
.arg(format!("s/LXC_NAME/{guid}/g"))

3
core/src/middleware/auth/session.rs
View File

@@ -20,9 +20,6 @@ use crate::context::RpcContext;
use crate::middleware::auth::DbContext;
use crate::prelude::*;
use crate::rpc_continuations::OpenAuthedContinuations;
use crate::util::Invoke;
use crate::util::io::{create_file_mod, read_file_to_string};
use crate::util::serde::{BASE64, const_true};
use crate::util::sync::SyncMutex;
pub trait SessionAuthContext: DbContext {

2
core/src/middleware/auth/signature.rs
View File

@@ -71,7 +71,7 @@ impl SignatureAuthContext for RpcContext {
.as_network()
.as_host()
.as_private_domains()
.de()
.keys()
.map(|k| k.into_iter())
.transpose(),
)

6
core/src/net/acme.rs
View File

@@ -461,7 +461,8 @@ impl ValueParserFactory for AcmeProvider {
}
}
#[derive(Deserialize, Serialize, Parser)]
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
pub struct InitAcmeParams {
#[arg(long, help = "help.arg.acme-provider")]
pub provider: AcmeProvider,
@@ -486,7 +487,8 @@ pub async fn init(
Ok(())
}
#[derive(Deserialize, Serialize, Parser)]
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
pub struct RemoveAcmeParams {
#[arg(long, help = "help.arg.acme-provider")]
pub provider: AcmeProvider,

211
core/src/net/dns.rs
View File

@@ -10,8 +10,9 @@ use color_eyre::eyre::eyre;
use futures::{FutureExt, StreamExt, TryStreamExt};
use hickory_server::authority::{AuthorityObject, Catalog, MessageResponseBuilder};
use hickory_server::proto::op::{Header, ResponseCode};
use hickory_server::proto::rr::{LowerName, Name, Record, RecordType};
use hickory_server::resolver::config::{ResolverConfig, ResolverOpts};
use hickory_server::proto::rr::{Name, Record, RecordType};
use hickory_server::proto::xfer::Protocol;
use hickory_server::resolver::config::{NameServerConfig, ResolverConfig, ResolverOpts};
use hickory_server::server::{Request, RequestHandler, ResponseHandler, ResponseInfo};
use hickory_server::store::forwarder::{ForwardAuthority, ForwardConfig};
use hickory_server::{ServerFuture, resolver as hickory_resolver};
@@ -25,6 +26,7 @@ use serde::{Deserialize, Serialize};
use tokio::net::{TcpListener, UdpSocket};
use tokio::sync::RwLock;
use tracing::instrument;
use ts_rs::TS;
use crate::context::{CliContext, RpcContext};
use crate::db::model::Database;
@@ -93,7 +95,8 @@ pub fn dns_api<C: Context>() -> ParentHandler<C> {
)
}
#[derive(Deserialize, Serialize, Parser)]
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
pub struct QueryDnsParams {
#[arg(help = "help.arg.fqdn")]
pub fqdn: InternedString,
@@ -133,7 +136,8 @@ pub fn query_dns<C: Context>(
.map_err(Error::from)
}
#[derive(Deserialize, Serialize, Parser)]
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
pub struct SetStaticDnsParams {
#[arg(help = "help.arg.dns-servers")]
pub servers: Option<Vec<String>>,
@@ -203,6 +207,7 @@ pub async fn dump_table(
struct ResolveMap {
private_domains: BTreeMap<InternedString, Weak<()>>,
services: BTreeMap<Option<PackageId>, BTreeMap<Ipv4Addr, Weak<()>>>,
challenges: BTreeMap<InternedString, (InternedString, Weak<()>)>,
}
pub struct DnsController {
@@ -237,22 +242,60 @@ impl Resolver {
let mut prev = crate::util::serde::hash_serializable::<sha2::Sha256, _>(&(
ResolverConfig::new(),
ResolverOpts::default(),
Option::<std::collections::VecDeque<SocketAddr>>::None,
))
.unwrap_or_default();
loop {
if let Err(e) = async {
let mut stream = file_string_stream("/run/systemd/resolve/resolv.conf")
.filter_map(|a| futures::future::ready(a.transpose()))
.boxed();
while let Some(conf) = stream.try_next().await? {
let (config, mut opts) =
hickory_resolver::system_conf::parse_resolv_conf(conf)
.with_kind(ErrorKind::ParseSysInfo)?;
opts.timeout = Duration::from_secs(30);
let res: Result<(), Error> = async {
let mut file_stream =
file_string_stream("/run/systemd/resolve/resolv.conf")
.filter_map(|a| futures::future::ready(a.transpose()))
.boxed();
let mut static_sub = db
.subscribe(
"/public/serverInfo/network/dns/staticServers"
.parse()
.unwrap(),
)
.await;
let mut last_config: Option<(ResolverConfig, ResolverOpts)> = None;
loop {
let got_file = tokio::select! {
res = file_stream.try_next() => {
let conf = res?
.ok_or_else(|| Error::new(
eyre!("resolv.conf stream ended"),
ErrorKind::Network,
))?;
let (config, mut opts) =
hickory_resolver::system_conf::parse_resolv_conf(conf)
.with_kind(ErrorKind::ParseSysInfo)?;
opts.timeout = Duration::from_secs(30);
last_config = Some((config, opts));
true
}
_ = static_sub.recv() => false,
};
let Some((ref config, ref opts)) = last_config else {
continue;
};
let static_servers: Option<std::collections::VecDeque<SocketAddr>> = db
.peek()
.await
.as_public()
.as_server_info()
.as_network()
.as_dns()
.as_static_servers()
.de()?;
let hash = crate::util::serde::hash_serializable::<sha2::Sha256, _>(
&(&config, &opts),
&(config, opts, &static_servers),
)?;
if hash != prev {
if hash == prev {
prev = hash;
continue;
}
if got_file {
db.mutate(|db| {
db.as_public_mut()
.as_server_info_mut()
@@ -271,44 +314,52 @@ impl Resolver {
})
.await
.result?;
let auth: Vec<Arc<dyn AuthorityObject>> = vec![Arc::new(
ForwardAuthority::builder_tokio(ForwardConfig {
name_servers: from_value(Value::Array(
config
.name_servers()
.into_iter()
.skip(4)
.map(to_value)
.collect::<Result<_, Error>>()?,
))?,
options: Some(opts),
}
let forward_servers = if let Some(servers) = &static_servers {
servers
.iter()
.flat_map(|addr| {
[
NameServerConfig::new(*addr, Protocol::Udp),
NameServerConfig::new(*addr, Protocol::Tcp),
]
})
.build()
.map_err(|e| Error::new(eyre!("{e}"), ErrorKind::Network))?,
)];
{
let mut guard = tokio::time::timeout(
Duration::from_secs(10),
catalog.write(),
)
.await
.map_err(|_| {
Error::new(
eyre!("{}", t!("net.dns.timeout-updating-catalog")),
ErrorKind::Timeout,
)
})?;
guard.upsert(Name::root().into(), auth);
drop(guard);
}
.map(|n| to_value(&n))
.collect::<Result<_, Error>>()?
} else {
config
.name_servers()
.into_iter()
.skip(4)
.map(to_value)
.collect::<Result<_, Error>>()?
};
let auth: Vec<Arc<dyn AuthorityObject>> = vec![Arc::new(
ForwardAuthority::builder_tokio(ForwardConfig {
name_servers: from_value(Value::Array(forward_servers))?,
options: Some(opts.clone()),
})
.build()
.map_err(|e| Error::new(eyre!("{e}"), ErrorKind::Network))?,
)];
{
let mut guard =
tokio::time::timeout(Duration::from_secs(10), catalog.write())
.await
.map_err(|_| {
Error::new(
eyre!("{}", t!("net.dns.timeout-updating-catalog")),
ErrorKind::Timeout,
)
})?;
guard.upsert(Name::root().into(), auth);
drop(guard);
}
prev = hash;
}
Ok::<_, Error>(())
}
.await
{
.await;
if let Err(e) = res {
tracing::error!("{e}");
tracing::debug!("{e:?}");
tokio::time::sleep(Duration::from_secs(1)).await;
@@ -399,7 +450,41 @@ impl RequestHandler for Resolver {
match async {
let req = request.request_info()?;
let query = req.query;
if let Some(ip) = self.resolve(query.name().borrow(), req.src.ip()) {
let name = query.name();
if STARTOS.zone_of(name) && query.query_type() == RecordType::TXT {
let name_str =
InternedString::intern(name.to_lowercase().to_utf8().trim_end_matches('.'));
if let Some(txt_value) = self.resolve.mutate(|r| {
r.challenges.retain(|_, (_, weak)| weak.strong_count() > 0);
r.challenges.remove(&name_str).map(|(val, _)| val)
}) {
let mut header = Header::response_from_request(request.header());
header.set_recursion_available(true);
return response_handle
.send_response(
MessageResponseBuilder::from_message_request(&*request).build(
header,
&[Record::from_rdata(
query.name().to_owned().into(),
0,
hickory_server::proto::rr::RData::TXT(
hickory_server::proto::rr::rdata::TXT::new(vec![
txt_value.to_string(),
]),
),
)],
[],
[],
[],
),
)
.await
.map(Some);
}
}
if let Some(ip) = self.resolve(name, req.src.ip()) {
match query.query_type() {
RecordType::A => {
let mut header = Header::response_from_request(request.header());
@@ -615,6 +700,34 @@ impl DnsController {
}
}
pub fn add_challenge(
&self,
domain: InternedString,
value: InternedString,
) -> Result<Arc<()>, Error> {
if let Some(resolve) = Weak::upgrade(&self.resolve) {
resolve.mutate(|writable| {
let entry = writable
.challenges
.entry(domain)
.or_insert_with(|| (value.clone(), Weak::new()));
let rc = if let Some(rc) = Weak::upgrade(&entry.1) {
rc
} else {
let new = Arc::new(());
*entry = (value, Arc::downgrade(&new));
new
};
Ok(rc)
})
} else {
Err(Error::new(
eyre!("{}", t!("net.dns.server-thread-exited")),
crate::ErrorKind::Network,
))
}
}
pub fn gc_private_domains<'a, BK: Ord + 'a>(
&self,
domains: impl IntoIterator<Item = &'a BK> + 'a,

246
core/src/net/forward.rs
View File

@@ -4,44 +4,90 @@ use std::sync::{Arc, Weak};
use std::time::Duration;
use futures::channel::oneshot;
use id_pool::IdPool;
use iddqd::{IdOrdItem, IdOrdMap};
use imbl::OrdMap;
use ipnet::{IpNet, Ipv4Net};
use rand::Rng;
use rpc_toolkit::{Context, HandlerArgs, HandlerExt, ParentHandler, from_fn_async};
use serde::{Deserialize, Serialize};
use tokio::process::Command;
use tokio::sync::mpsc;
use crate::GatewayId;
use crate::context::{CliContext, RpcContext};
use crate::db::model::public::NetworkInterfaceInfo;
use crate::net::gateway::{DynInterfaceFilter, InterfaceFilter};
use crate::prelude::*;
use crate::util::Invoke;
use crate::util::future::NonDetachingJoinHandle;
use crate::util::serde::{HandlerExtSerde, display_serializable};
use crate::util::sync::Watch;
use crate::{GatewayId, HOST_IP};
pub const START9_BRIDGE_IFACE: &str = "lxcbr0";
pub const FIRST_DYNAMIC_PRIVATE_PORT: u16 = 49152;
const EPHEMERAL_PORT_START: u16 = 49152;
// vhost.rs:89 — not allowed: <=1024, >=32768, 5355, 5432, 9050, 6010, 9051, 5353
const RESTRICTED_PORTS: &[u16] = &[5353, 5355, 5432, 6010, 9050, 9051];
fn is_restricted(port: u16) -> bool {
port <= 1024 || RESTRICTED_PORTS.contains(&port)
}
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
pub struct ForwardRequirements {
pub public_gateways: BTreeSet<GatewayId>,
pub private_ips: BTreeSet<IpAddr>,
pub secure: bool,
}
impl std::fmt::Display for ForwardRequirements {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(
f,
"ForwardRequirements {{ public: {:?}, private: {:?}, secure: {} }}",
self.public_gateways, self.private_ips, self.secure
)
}
}
#[derive(Debug, Deserialize, Serialize)]
pub struct AvailablePorts(IdPool);
pub struct AvailablePorts(BTreeMap<u16, bool>);
impl AvailablePorts {
pub fn new() -> Self {
Self(IdPool::new_ranged(FIRST_DYNAMIC_PRIVATE_PORT..u16::MAX))
Self(BTreeMap::new())
}
pub fn alloc(&mut self) -> Result<u16, Error> {
self.0.request_id().ok_or_else(|| {
Error::new(
eyre!("{}", t!("net.forward.no-dynamic-ports-available")),
ErrorKind::Network,
)
})
pub fn alloc(&mut self, ssl: bool) -> Result<u16, Error> {
let mut rng = rand::rng();
for _ in 0..1000 {
let port = rng.random_range(EPHEMERAL_PORT_START..u16::MAX);
if !self.0.contains_key(&port) {
self.0.insert(port, ssl);
return Ok(port);
}
}
Err(Error::new(
eyre!("{}", t!("net.forward.no-dynamic-ports-available")),
ErrorKind::Network,
))
}
/// Try to allocate a specific port. Returns Some(port) if available, None if taken/restricted.
pub fn try_alloc(&mut self, port: u16, ssl: bool) -> Option<u16> {
if is_restricted(port) || self.0.contains_key(&port) {
return None;
}
self.0.insert(port, ssl);
Some(port)
}
pub fn set_ssl(&mut self, port: u16, ssl: bool) {
self.0.insert(port, ssl);
}
/// Returns whether a given allocated port is SSL.
pub fn is_ssl(&self, port: u16) -> bool {
self.0.get(&port).copied().unwrap_or(false)
}
pub fn free(&mut self, ports: impl IntoIterator<Item = u16>) {
for port in ports {
self.0.return_id(port).unwrap_or_default();
self.0.remove(&port);
}
}
}
@@ -61,10 +107,10 @@ pub fn forward_api<C: Context>() -> ParentHandler<C> {
}
let mut table = Table::new();
table.add_row(row![bc => "FROM", "TO", "FILTER"]);
table.add_row(row![bc => "FROM", "TO", "REQS"]);
for (external, target) in res.0 {
table.add_row(row![external, target.target, target.filter]);
table.add_row(row![external, target.target, target.reqs]);
}
table.print_tty(false)?;
@@ -79,6 +125,7 @@ struct ForwardMapping {
source: SocketAddrV4,
target: SocketAddrV4,
target_prefix: u8,
src_filter: Option<IpNet>,
rc: Weak<()>,
}
@@ -93,9 +140,10 @@ impl PortForwardState {
source: SocketAddrV4,
target: SocketAddrV4,
target_prefix: u8,
src_filter: Option<IpNet>,
) -> Result<Arc<()>, Error> {
if let Some(existing) = self.mappings.get_mut(&source) {
if existing.target == target {
if existing.target == target && existing.src_filter == src_filter {
if let Some(existing_rc) = existing.rc.upgrade() {
return Ok(existing_rc);
} else {
@@ -104,21 +152,28 @@ impl PortForwardState {
return Ok(rc);
}
} else {
// Different target, need to remove old and add new
// Different target or src_filter, need to remove old and add new
if let Some(mapping) = self.mappings.remove(&source) {
unforward(mapping.source, mapping.target, mapping.target_prefix).await?;
unforward(
mapping.source,
mapping.target,
mapping.target_prefix,
mapping.src_filter.as_ref(),
)
.await?;
}
}
}
let rc = Arc::new(());
forward(source, target, target_prefix).await?;
forward(source, target, target_prefix, src_filter.as_ref()).await?;
self.mappings.insert(
source,
ForwardMapping {
source,
target,
target_prefix,
src_filter,
rc: Arc::downgrade(&rc),
},
);
@@ -136,7 +191,13 @@ impl PortForwardState {
for source in to_remove {
if let Some(mapping) = self.mappings.remove(&source) {
unforward(mapping.source, mapping.target, mapping.target_prefix).await?;
unforward(
mapping.source,
mapping.target,
mapping.target_prefix,
mapping.src_filter.as_ref(),
)
.await?;
}
}
Ok(())
@@ -157,9 +218,14 @@ impl Drop for PortForwardState {
let mappings = std::mem::take(&mut self.mappings);
tokio::spawn(async move {
for (_, mapping) in mappings {
unforward(mapping.source, mapping.target, mapping.target_prefix)
.await
.log_err();
unforward(
mapping.source,
mapping.target,
mapping.target_prefix,
mapping.src_filter.as_ref(),
)
.await
.log_err();
}
});
}
@@ -171,6 +237,7 @@ enum PortForwardCommand {
source: SocketAddrV4,
target: SocketAddrV4,
target_prefix: u8,
src_filter: Option<IpNet>,
respond: oneshot::Sender<Result<Arc<()>, Error>>,
},
Gc {
@@ -191,7 +258,13 @@ pub async fn add_iptables_rule(nat: bool, undo: bool, args: &[&str]) -> Result<(
if nat {
cmd.arg("-t").arg("nat");
}
if undo != !cmd.arg("-C").args(args).status().await?.success() {
let exists = cmd
.arg("-C")
.args(args)
.invoke(ErrorKind::Network)
.await
.is_ok();
if undo != !exists {
let mut cmd = Command::new("iptables");
if nat {
cmd.arg("-t").arg("nat");
@@ -257,9 +330,12 @@ impl PortForwardController {
source,
target,
target_prefix,
src_filter,
respond,
} => {
let result = state.add_forward(source, target, target_prefix).await;
let result = state
.add_forward(source, target, target_prefix, src_filter)
.await;
respond.send(result).ok();
}
PortForwardCommand::Gc { respond } => {
@@ -284,6 +360,7 @@ impl PortForwardController {
source: SocketAddrV4,
target: SocketAddrV4,
target_prefix: u8,
src_filter: Option<IpNet>,
) -> Result<Arc<()>, Error> {
let (send, recv) = oneshot::channel();
self.req
@@ -291,6 +368,7 @@ impl PortForwardController {
source,
target,
target_prefix,
src_filter,
respond: send,
})
.map_err(err_has_exited)?;
@@ -321,14 +399,14 @@ struct InterfaceForwardRequest {
external: u16,
target: SocketAddrV4,
target_prefix: u8,
filter: DynInterfaceFilter,
reqs: ForwardRequirements,
rc: Arc<()>,
}
#[derive(Clone)]
struct InterfaceForwardEntry {
external: u16,
filter: BTreeMap<DynInterfaceFilter, (SocketAddrV4, u8, Weak<()>)>,
targets: BTreeMap<ForwardRequirements, (SocketAddrV4, u8, Weak<()>)>,
// Maps source SocketAddr -> strong reference for the forward created in PortForwardController
forwards: BTreeMap<SocketAddrV4, Arc<()>>,
}
@@ -346,7 +424,7 @@ impl InterfaceForwardEntry {
fn new(external: u16) -> Self {
Self {
external,
filter: BTreeMap::new(),
targets: BTreeMap::new(),
forwards: BTreeMap::new(),
}
}
@@ -358,28 +436,37 @@ impl InterfaceForwardEntry {
) -> Result<(), Error> {
let mut keep = BTreeSet::<SocketAddrV4>::new();
for (iface, info) in ip_info.iter() {
if let Some((target, target_prefix)) = self
.filter
.iter()
.filter(|(_, (_, _, rc))| rc.strong_count() > 0)
.find(|(filter, _)| filter.filter(iface, info))
.map(|(_, (target, target_prefix, _))| (*target, *target_prefix))
{
if let Some(ip_info) = &info.ip_info {
for addr in ip_info.subnets.iter().filter_map(|net| {
if let IpAddr::V4(ip) = net.addr() {
Some(SocketAddrV4::new(ip, self.external))
} else {
None
for (gw_id, info) in ip_info.iter() {
if let Some(ip_info) = &info.ip_info {
for subnet in ip_info.subnets.iter() {
if let IpAddr::V4(ip) = subnet.addr() {
let addr = SocketAddrV4::new(ip, self.external);
if keep.contains(&addr) {
continue;
}
}) {
keep.insert(addr);
if !self.forwards.contains_key(&addr) {
let rc = port_forward
.add_forward(addr, target, target_prefix)
for (reqs, (target, target_prefix, rc)) in self.targets.iter() {
if rc.strong_count() == 0 {
continue;
}
if !reqs.secure && !info.secure() {
continue;
}
let src_filter = if reqs.public_gateways.contains(gw_id) {
None
} else if reqs.private_ips.contains(&IpAddr::V4(ip)) {
Some(subnet.trunc())
} else {
continue;
};
keep.insert(addr);
let fwd_rc = port_forward
.add_forward(addr, *target, *target_prefix, src_filter)
.await?;
self.forwards.insert(addr, rc);
self.forwards.insert(addr, fwd_rc);
break;
}
}
}
@@ -398,7 +485,7 @@ impl InterfaceForwardEntry {
external,
target,
target_prefix,
filter,
reqs,
mut rc,
}: InterfaceForwardRequest,
ip_info: &OrdMap<GatewayId, NetworkInterfaceInfo>,
@@ -412,8 +499,8 @@ impl InterfaceForwardEntry {
}
let entry = self
.filter
.entry(filter)
.targets
.entry(reqs)
.or_insert_with(|| (target, target_prefix, Arc::downgrade(&rc)));
if entry.0 != target {
entry.0 = target;
@@ -436,7 +523,7 @@ impl InterfaceForwardEntry {
ip_info: &OrdMap<GatewayId, NetworkInterfaceInfo>,
port_forward: &PortForwardController,
) -> Result<(), Error> {
self.filter.retain(|_, (_, _, rc)| rc.strong_count() > 0);
self.targets.retain(|_, (_, _, rc)| rc.strong_count() > 0);
self.update(ip_info, port_forward).await
}
@@ -495,7 +582,7 @@ pub struct ForwardTable(pub BTreeMap<u16, ForwardTarget>);
pub struct ForwardTarget {
pub target: SocketAddrV4,
pub target_prefix: u8,
pub filter: String,
pub reqs: String,
}
impl From<&InterfaceForwardState> for ForwardTable {
@@ -506,16 +593,16 @@ impl From<&InterfaceForwardState> for ForwardTable {
.iter()
.flat_map(|entry| {
entry
.filter
.targets
.iter()
.filter(|(_, (_, _, rc))| rc.strong_count() > 0)
.map(|(filter, (target, target_prefix, _))| {
.map(|(reqs, (target, target_prefix, _))| {
(
entry.external,
ForwardTarget {
target: *target,
target_prefix: *target_prefix,
filter: format!("{:#?}", filter),
reqs: format!("{reqs}"),
},
)
})
@@ -534,16 +621,6 @@ enum InterfaceForwardCommand {
DumpTable(oneshot::Sender<ForwardTable>),
}
#[test]
fn test() {
use crate::net::gateway::SecureFilter;
assert_ne!(
false.into_dyn(),
SecureFilter { secure: false }.into_dyn().into_dyn()
);
}
pub struct InterfacePortForwardController {
req: mpsc::UnboundedSender<InterfaceForwardCommand>,
_thread: NonDetachingJoinHandle<()>,
@@ -593,7 +670,7 @@ impl InterfacePortForwardController {
pub async fn add(
&self,
external: u16,
filter: DynInterfaceFilter,
reqs: ForwardRequirements,
target: SocketAddrV4,
target_prefix: u8,
) -> Result<Arc<()>, Error> {
@@ -605,7 +682,7 @@ impl InterfacePortForwardController {
external,
target,
target_prefix,
filter,
reqs,
rc,
},
send,
@@ -637,15 +714,25 @@ async fn forward(
source: SocketAddrV4,
target: SocketAddrV4,
target_prefix: u8,
src_filter: Option<&IpNet>,
) -> Result<(), Error> {
Command::new("/usr/lib/startos/scripts/forward-port")
.env("sip", source.ip().to_string())
let mut cmd = Command::new("/usr/lib/startos/scripts/forward-port");
cmd.env("sip", source.ip().to_string())
.env("dip", target.ip().to_string())
.env("dprefix", target_prefix.to_string())
.env("sport", source.port().to_string())
.env("dport", target.port().to_string())
.invoke(ErrorKind::Network)
.await?;
.env(
"bridge_subnet",
Ipv4Net::new(HOST_IP.into(), 24)
.with_kind(ErrorKind::ParseNetAddress)?
.trunc()
.to_string(),
);
if let Some(subnet) = src_filter {
cmd.env("src_subnet", subnet.to_string());
}
cmd.invoke(ErrorKind::Network).await?;
Ok(())
}
@@ -653,15 +740,18 @@ async fn unforward(
source: SocketAddrV4,
target: SocketAddrV4,
target_prefix: u8,
src_filter: Option<&IpNet>,
) -> Result<(), Error> {
Command::new("/usr/lib/startos/scripts/forward-port")
.env("UNDO", "1")
let mut cmd = Command::new("/usr/lib/startos/scripts/forward-port");
cmd.env("UNDO", "1")
.env("sip", source.ip().to_string())
.env("dip", target.ip().to_string())
.env("dprefix", target_prefix.to_string())
.env("sport", source.port().to_string())
.env("dport", target.port().to_string())
.invoke(ErrorKind::Network)
.await?;
.env("dport", target.port().to_string());
if let Some(subnet) = src_filter {
cmd.env("src_subnet", subnet.to_string());
}
cmd.invoke(ErrorKind::Network).await?;
Ok(())
}

1722
core/src/net/gateway.rs
View File

File diff suppressed because it is too large Load Diff

205
core/src/net/host/address.rs
View File

@@ -10,46 +10,29 @@ use ts_rs::TS;
use crate::GatewayId;
use crate::context::{CliContext, RpcContext};
use crate::db::model::DatabaseModel;
use crate::hostname::ServerHostname;
use crate::net::acme::AcmeProvider;
use crate::net::host::{HostApiKind, all_hosts};
use crate::net::tor::OnionAddress;
use crate::prelude::*;
use crate::util::serde::{HandlerExtSerde, display_serializable};
#[derive(Clone, Debug, Deserialize, Serialize)]
#[serde(rename_all = "kebab-case")]
#[serde(rename_all_fields = "camelCase")]
#[serde(tag = "kind")]
pub enum HostAddress {
Onion {
address: OnionAddress,
},
Domain {
address: InternedString,
public: Option<PublicDomainConfig>,
private: bool,
},
#[serde(rename_all = "camelCase")]
pub struct HostAddress {
pub address: InternedString,
pub public: Option<PublicDomainConfig>,
pub private: Option<BTreeSet<GatewayId>>,
}
#[derive(Debug, Clone, Deserialize, Serialize, TS)]
#[ts(export)]
pub struct PublicDomainConfig {
pub gateway: GatewayId,
pub acme: Option<AcmeProvider>,
}
fn handle_duplicates(db: &mut DatabaseModel) -> Result<(), Error> {
let mut onions = BTreeSet::<OnionAddress>::new();
let mut domains = BTreeSet::<InternedString>::new();
let check_onion = |onions: &mut BTreeSet<OnionAddress>, onion: OnionAddress| {
if onions.contains(&onion) {
return Err(Error::new(
eyre!("onion address {onion} is already in use"),
ErrorKind::InvalidRequest,
));
}
onions.insert(onion);
Ok(())
};
let check_domain = |domains: &mut BTreeSet<InternedString>, domain: InternedString| {
if domains.contains(&domain) {
return Err(Error::new(
@@ -68,35 +51,27 @@ fn handle_duplicates(db: &mut DatabaseModel) -> Result<(), Error> {
not_in_use.push(host);
continue;
}
for onion in host.as_onions().de()? {
check_onion(&mut onions, onion)?;
}
let public = host.as_public_domains().keys()?;
for domain in &public {
check_domain(&mut domains, domain.clone())?;
}
for domain in host.as_private_domains().de()? {
for domain in host.as_private_domains().keys()? {
if !public.contains(&domain) {
check_domain(&mut domains, domain)?;
}
}
}
for host in not_in_use {
host.as_onions_mut()
.mutate(|o| Ok(o.retain(|o| !onions.contains(o))))?;
host.as_public_domains_mut()
.mutate(|d| Ok(d.retain(|d, _| !domains.contains(d))))?;
host.as_private_domains_mut()
.mutate(|d| Ok(d.retain(|d| !domains.contains(d))))?;
.mutate(|d| Ok(d.retain(|d, _| !domains.contains(d))))?;
for onion in host.as_onions().de()? {
check_onion(&mut onions, onion)?;
}
let public = host.as_public_domains().keys()?;
for domain in &public {
check_domain(&mut domains, domain.clone())?;
}
for domain in host.as_private_domains().de()? {
for domain in host.as_private_domains().keys()? {
if !public.contains(&domain) {
check_domain(&mut domains, domain)?;
}
@@ -159,29 +134,6 @@ pub fn address_api<C: Context, Kind: HostApiKind>()
)
.with_inherited(Kind::inheritance),
)
.subcommand(
"onion",
ParentHandler::<C, Empty, Kind::Inheritance>::new()
.subcommand(
"add",
from_fn_async(add_onion::<Kind>)
.with_metadata("sync_db", Value::Bool(true))
.with_inherited(|_, a| a)
.no_display()
.with_about("about.add-address-to-host")
.with_call_remote::<CliContext>(),
)
.subcommand(
"remove",
from_fn_async(remove_onion::<Kind>)
.with_metadata("sync_db", Value::Bool(true))
.with_inherited(|_, a| a)
.no_display()
.with_about("about.remove-address-from-host")
.with_call_remote::<CliContext>(),
)
.with_inherited(Kind::inheritance),
)
.subcommand(
"list",
from_fn_async(list_addresses::<Kind>)
@@ -196,35 +148,7 @@ pub fn address_api<C: Context, Kind: HostApiKind>()
}
let mut table = Table::new();
table.add_row(row![bc => "ADDRESS", "PUBLIC", "ACME PROVIDER"]);
for address in &res {
match address {
HostAddress::Onion { address } => {
table.add_row(row![address, true, "N/A"]);
}
HostAddress::Domain {
address,
public: Some(PublicDomainConfig { gateway, acme }),
private,
} => {
table.add_row(row![
address,
&format!(
"{} ({gateway})",
if *private { "YES" } else { "ONLY" }
),
acme.as_ref().map(|a| a.0.as_str()).unwrap_or("NONE")
]);
}
HostAddress::Domain {
address,
public: None,
..
} => {
table.add_row(row![address, &format!("NO"), "N/A"]);
}
}
}
todo!("find a good way to represent this");
table.print_tty(false)?;
@@ -235,7 +159,8 @@ pub fn address_api<C: Context, Kind: HostApiKind>()
)
}
#[derive(Deserialize, Serialize, Parser)]
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
pub struct AddPublicDomainParams {
#[arg(help = "help.arg.fqdn")]
pub fqdn: InternedString,
@@ -271,11 +196,14 @@ pub async fn add_public_domain<Kind: HostApiKind>(
Kind::host_for(&inheritance, db)?
.as_public_domains_mut()
.insert(&fqdn, &PublicDomainConfig { acme, gateway })?;
handle_duplicates(db)
handle_duplicates(db)?;
let hostname = ServerHostname::load(db.as_public().as_server_info())?;
let gateways = db.as_public().as_server_info().as_network().as_gateways().de()?;
let ports = db.as_private().as_available_ports().de()?;
Kind::host_for(&inheritance, db)?.update_addresses(&hostname, &gateways, &ports)
})
.await
.result?;
Kind::sync_host(&ctx, inheritance).await?;
tokio::task::spawn_blocking(|| {
crate::net::dns::query_dns(ctx, crate::net::dns::QueryDnsParams { fqdn })
@@ -284,7 +212,8 @@ pub async fn add_public_domain<Kind: HostApiKind>(
.with_kind(ErrorKind::Unknown)?
}
#[derive(Deserialize, Serialize, Parser)]
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
pub struct RemoveDomainParams {
#[arg(help = "help.arg.fqdn")]
pub fqdn: InternedString,
@@ -299,36 +228,55 @@ pub async fn remove_public_domain<Kind: HostApiKind>(
.mutate(|db| {
Kind::host_for(&inheritance, db)?
.as_public_domains_mut()
.remove(&fqdn)
.remove(&fqdn)?;
let hostname = ServerHostname::load(db.as_public().as_server_info())?;
let gateways = db
.as_public()
.as_server_info()
.as_network()
.as_gateways()
.de()?;
let ports = db.as_private().as_available_ports().de()?;
Kind::host_for(&inheritance, db)?.update_addresses(&hostname, &gateways, &ports)
})
.await
.result?;
Kind::sync_host(&ctx, inheritance).await?;
Ok(())
}
#[derive(Deserialize, Serialize, Parser)]
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
pub struct AddPrivateDomainParams {
#[arg(help = "help.arg.fqdn")]
pub fqdn: InternedString,
pub gateway: GatewayId,
}
pub async fn add_private_domain<Kind: HostApiKind>(
ctx: RpcContext,
AddPrivateDomainParams { fqdn }: AddPrivateDomainParams,
AddPrivateDomainParams { fqdn, gateway }: AddPrivateDomainParams,
inheritance: Kind::Inheritance,
) -> Result<(), Error> {
ctx.db
.mutate(|db| {
Kind::host_for(&inheritance, db)?
.as_private_domains_mut()
.mutate(|d| Ok(d.insert(fqdn)))?;
handle_duplicates(db)
.upsert(&fqdn, || Ok(BTreeSet::new()))?
.mutate(|d| Ok(d.insert(gateway)))?;
handle_duplicates(db)?;
let hostname = ServerHostname::load(db.as_public().as_server_info())?;
let gateways = db
.as_public()
.as_server_info()
.as_network()
.as_gateways()
.de()?;
let ports = db.as_private().as_available_ports().de()?;
Kind::host_for(&inheritance, db)?.update_addresses(&hostname, &gateways, &ports)
})
.await
.result?;
Kind::sync_host(&ctx, inheritance).await?;
Ok(())
}
@@ -342,60 +290,19 @@ pub async fn remove_private_domain<Kind: HostApiKind>(
.mutate(|db| {
Kind::host_for(&inheritance, db)?
.as_private_domains_mut()
.mutate(|d| Ok(d.remove(&domain)))
.mutate(|d| Ok(d.remove(&domain)))?;
let hostname = ServerHostname::load(db.as_public().as_server_info())?;
let gateways = db
.as_public()
.as_server_info()
.as_network()
.as_gateways()
.de()?;
let ports = db.as_private().as_available_ports().de()?;
Kind::host_for(&inheritance, db)?.update_addresses(&hostname, &gateways, &ports)
})
.await
.result?;
Kind::sync_host(&ctx, inheritance).await?;
Ok(())
}
#[derive(Deserialize, Serialize, Parser)]
pub struct OnionParams {
#[arg(help = "help.arg.onion-address")]
pub onion: String,
}
pub async fn add_onion<Kind: HostApiKind>(
ctx: RpcContext,
OnionParams { onion }: OnionParams,
inheritance: Kind::Inheritance,
) -> Result<(), Error> {
let onion = onion.parse::<OnionAddress>()?;
ctx.db
.mutate(|db| {
db.as_private().as_key_store().as_onion().get_key(&onion)?;
Kind::host_for(&inheritance, db)?
.as_onions_mut()
.mutate(|a| Ok(a.insert(onion)))?;
handle_duplicates(db)
})
.await
.result?;
Kind::sync_host(&ctx, inheritance).await?;
Ok(())
}
pub async fn remove_onion<Kind: HostApiKind>(
ctx: RpcContext,
OnionParams { onion }: OnionParams,
inheritance: Kind::Inheritance,
) -> Result<(), Error> {
let onion = onion.parse::<OnionAddress>()?;
ctx.db
.mutate(|db| {
Kind::host_for(&inheritance, db)?
.as_onions_mut()
.mutate(|a| Ok(a.remove(&onion)))
})
.await
.result?;
Kind::sync_host(&ctx, inheritance).await?;
Ok(())
}

185
core/src/net/host/binding.rs
View File

@@ -1,23 +1,23 @@
use std::collections::{BTreeMap, BTreeSet};
use std::net::SocketAddr;
use std::str::FromStr;
use clap::Parser;
use clap::builder::ValueParserFactory;
use imbl::OrdSet;
use rpc_toolkit::{Context, Empty, HandlerArgs, HandlerExt, ParentHandler, from_fn_async};
use serde::{Deserialize, Serialize};
use ts_rs::TS;
use crate::HostId;
use crate::context::{CliContext, RpcContext};
use crate::db::model::public::NetworkInterfaceInfo;
use crate::db::prelude::Map;
use crate::net::forward::AvailablePorts;
use crate::net::gateway::InterfaceFilter;
use crate::net::host::HostApiKind;
use crate::net::service_interface::HostnameInfo;
use crate::net::vhost::AlpnInfo;
use crate::prelude::*;
use crate::util::FromStrParser;
use crate::util::serde::{HandlerExtSerde, display_serializable};
use crate::{GatewayId, HostId};
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize, TS)]
#[ts(export)]
@@ -45,25 +45,87 @@ impl FromStr for BindId {
}
}
#[derive(Debug, Deserialize, Serialize, TS)]
#[derive(Debug, Default, Clone, Deserialize, Serialize, TS, HasModel)]
#[serde(rename_all = "camelCase")]
#[ts(export)]
#[model = "Model<Self>"]
pub struct DerivedAddressInfo {
/// User override: enable these addresses (only for public IP & port)
pub enabled: BTreeSet<SocketAddr>,
/// User override: disable these addresses (only for domains and private IP & port)
pub disabled: BTreeSet<(InternedString, u16)>,
/// COMPUTED: NetServiceData::update — all possible addresses for this binding
pub available: BTreeSet<HostnameInfo>,
}
impl DerivedAddressInfo {
/// Returns addresses that are currently enabled after applying overrides.
/// Default: public IPs are disabled, everything else is enabled.
/// Explicit `enabled`/`disabled` overrides take precedence.
pub fn enabled(&self) -> BTreeSet<&HostnameInfo> {
self.available
.iter()
.filter(|h| {
if h.public && h.metadata.is_ip() {
// Public IPs: disabled by default, explicitly enabled via SocketAddr
h.to_socket_addr().map_or(
true, // should never happen, but would rather see them if it does
|sa| self.enabled.contains(&sa),
)
} else {
!self
.disabled
.contains(&(h.hostname.clone(), h.port.unwrap_or_default())) // disablable addresses will always have a port
}
})
.collect()
}
}
#[derive(Debug, Default, Deserialize, Serialize, HasModel, TS)]
#[model = "Model<Self>"]
#[ts(export)]
pub struct Bindings(pub BTreeMap<u16, BindInfo>);
impl Map for Bindings {
type Key = u16;
type Value = BindInfo;
fn key_str(key: &Self::Key) -> Result<impl AsRef<str>, Error> {
Self::key_string(key)
}
fn key_string(key: &Self::Key) -> Result<InternedString, Error> {
Ok(InternedString::from_display(key))
}
}
impl std::ops::Deref for Bindings {
type Target = BTreeMap<u16, BindInfo>;
fn deref(&self) -> &Self::Target {
&self.0
}
}
impl std::ops::DerefMut for Bindings {
fn deref_mut(&mut self) -> &mut Self::Target {
&mut self.0
}
}
#[derive(Debug, Deserialize, Serialize, HasModel, TS)]
#[serde(rename_all = "camelCase")]
#[model = "Model<Self>"]
#[ts(export)]
pub struct BindInfo {
pub enabled: bool,
pub options: BindOptions,
pub net: NetInfo,
pub addresses: DerivedAddressInfo,
}
#[derive(Clone, Debug, Deserialize, Serialize, TS, PartialEq, Eq, PartialOrd, Ord)]
#[serde(rename_all = "camelCase")]
#[ts(export)]
pub struct NetInfo {
#[ts(as = "BTreeSet::<GatewayId>")]
#[serde(default)]
pub private_disabled: OrdSet<GatewayId>,
#[ts(as = "BTreeSet::<GatewayId>")]
#[serde(default)]
pub public_enabled: OrdSet<GatewayId>,
pub assigned_port: Option<u16>,
pub assigned_ssl_port: Option<u16>,
}
@@ -71,25 +133,28 @@ impl BindInfo {
pub fn new(available_ports: &mut AvailablePorts, options: BindOptions) -> Result<Self, Error> {
let mut assigned_port = None;
let mut assigned_ssl_port = None;
if options.add_ssl.is_some() {
assigned_ssl_port = Some(available_ports.alloc()?);
if let Some(ssl) = &options.add_ssl {
assigned_ssl_port = available_ports
.try_alloc(ssl.preferred_external_port, true)
.or_else(|| Some(available_ports.alloc(true).ok()?));
}
if options
.secure
.map_or(true, |s| !(s.ssl && options.add_ssl.is_some()))
{
assigned_port = Some(available_ports.alloc()?);
assigned_port = available_ports
.try_alloc(options.preferred_external_port, false)
.or_else(|| Some(available_ports.alloc(false).ok()?));
}
Ok(Self {
enabled: true,
options,
net: NetInfo {
private_disabled: OrdSet::new(),
public_enabled: OrdSet::new(),
assigned_port,
assigned_ssl_port,
},
addresses: DerivedAddressInfo::default(),
})
}
pub fn update(
@@ -97,7 +162,11 @@ impl BindInfo {
available_ports: &mut AvailablePorts,
options: BindOptions,
) -> Result<Self, Error> {
let Self { net: mut lan, .. } = self;
let Self {
net: mut lan,
addresses,
..
} = self;
if options
.secure
.map_or(true, |s| !(s.ssl && options.add_ssl.is_some()))
@@ -105,19 +174,26 @@ impl BindInfo {
{
lan.assigned_port = if let Some(port) = lan.assigned_port.take() {
Some(port)
} else if let Some(port) =
available_ports.try_alloc(options.preferred_external_port, false)
{
Some(port)
} else {
Some(available_ports.alloc()?)
Some(available_ports.alloc(false)?)
};
} else {
if let Some(port) = lan.assigned_port.take() {
available_ports.free([port]);
}
}
if options.add_ssl.is_some() {
if let Some(ssl) = &options.add_ssl {
lan.assigned_ssl_port = if let Some(port) = lan.assigned_ssl_port.take() {
Some(port)
} else if let Some(port) = available_ports.try_alloc(ssl.preferred_external_port, true)
{
Some(port)
} else {
Some(available_ports.alloc()?)
Some(available_ports.alloc(true)?)
};
} else {
if let Some(port) = lan.assigned_ssl_port.take() {
@@ -128,22 +204,13 @@ impl BindInfo {
enabled: true,
options,
net: lan,
addresses,
})
}
pub fn disable(&mut self) {
self.enabled = false;
}
}
impl InterfaceFilter for NetInfo {
fn filter(&self, id: &GatewayId, info: &NetworkInterfaceInfo) -> bool {
info.ip_info.is_some()
&& if info.public() {
self.public_enabled.contains(id)
} else {
!self.private_disabled.contains(id)
}
}
}
#[derive(Debug, Clone, Copy, serde::Serialize, serde::Deserialize, TS)]
#[ts(export)]
@@ -188,7 +255,7 @@ pub fn binding<C: Context, Kind: HostApiKind>()
let mut table = Table::new();
table.add_row(row![bc => "INTERNAL PORT", "ENABLED", "EXTERNAL PORT", "EXTERNAL SSL PORT"]);
for (internal, info) in res {
for (internal, info) in res.iter() {
table.add_row(row![
internal,
info.enabled,
@@ -213,12 +280,12 @@ pub fn binding<C: Context, Kind: HostApiKind>()
.with_call_remote::<CliContext>(),
)
.subcommand(
"set-gateway-enabled",
from_fn_async(set_gateway_enabled::<Kind>)
"set-address-enabled",
from_fn_async(set_address_enabled::<Kind>)
.with_metadata("sync_db", Value::Bool(true))
.with_inherited(Kind::inheritance)
.no_display()
.with_about("about.set-gateway-enabled-for-binding")
.with_about("about.set-address-enabled-for-binding")
.with_call_remote::<CliContext>(),
)
}
@@ -227,7 +294,7 @@ pub async fn list_bindings<Kind: HostApiKind>(
ctx: RpcContext,
_: Empty,
inheritance: Kind::Inheritance,
) -> Result<BTreeMap<u16, BindInfo>, Error> {
) -> Result<Bindings, Error> {
Kind::host_for(&inheritance, &mut ctx.db.peek().await)?
.as_bindings()
.de()
@@ -236,50 +303,54 @@ pub async fn list_bindings<Kind: HostApiKind>(
#[derive(Deserialize, Serialize, Parser, TS)]
#[serde(rename_all = "camelCase")]
#[ts(export)]
pub struct BindingGatewaySetEnabledParams {
pub struct BindingSetAddressEnabledParams {
#[arg(help = "help.arg.internal-port")]
internal_port: u16,
#[arg(help = "help.arg.gateway-id")]
gateway: GatewayId,
#[arg(long, help = "help.arg.address")]
address: String,
#[arg(long, help = "help.arg.binding-enabled")]
enabled: Option<bool>,
}
pub async fn set_gateway_enabled<Kind: HostApiKind>(
pub async fn set_address_enabled<Kind: HostApiKind>(
ctx: RpcContext,
BindingGatewaySetEnabledParams {
BindingSetAddressEnabledParams {
internal_port,
gateway,
address,
enabled,
}: BindingGatewaySetEnabledParams,
}: BindingSetAddressEnabledParams,
inheritance: Kind::Inheritance,
) -> Result<(), Error> {
let enabled = enabled.unwrap_or(true);
let gateway_public = ctx
.net_controller
.net_iface
.watcher
.ip_info()
.get(&gateway)
.or_not_found(&gateway)?
.public();
let address: HostnameInfo =
serde_json::from_str(&address).with_kind(ErrorKind::Deserialization)?;
ctx.db
.mutate(|db| {
Kind::host_for(&inheritance, db)?
.as_bindings_mut()
.mutate(|b| {
let net = &mut b.get_mut(&internal_port).or_not_found(internal_port)?.net;
if gateway_public {
let bind = b.get_mut(&internal_port).or_not_found(internal_port)?;
if address.public && address.metadata.is_ip() {
// Public IPs: toggle via SocketAddr in `enabled` set
let sa = address.to_socket_addr().ok_or_else(|| {
Error::new(
eyre!("cannot convert address to socket addr"),
ErrorKind::InvalidRequest,
)
})?;
if enabled {
net.public_enabled.insert(gateway);
bind.addresses.enabled.insert(sa);
} else {
net.public_enabled.remove(&gateway);
bind.addresses.enabled.remove(&sa);
}
} else {
// Domains and private IPs: toggle via (host, port) in `disabled` set
let port = address.port.unwrap_or(if address.ssl { 443 } else { 80 });
let key = (address.hostname.clone(), port);
if enabled {
net.private_disabled.remove(&gateway);
bind.addresses.disabled.remove(&key);
} else {
net.private_disabled.insert(gateway);
bind.addresses.disabled.insert(key);
}
}
Ok(())
@@ -287,5 +358,5 @@ pub async fn set_gateway_enabled<Kind: HostApiKind>(
})
.await
.result?;
Kind::sync_host(&ctx, inheritance).await
Ok(())
}

353
core/src/net/host/mod.rs
View File

@@ -1,23 +1,26 @@
use std::collections::{BTreeMap, BTreeSet};
use std::future::Future;
use std::net::{IpAddr, SocketAddrV4};
use std::panic::RefUnwindSafe;
use clap::Parser;
use imbl::OrdMap;
use imbl_value::InternedString;
use itertools::Itertools;
use patch_db::DestructureMut;
use rpc_toolkit::{Context, Empty, HandlerExt, OrEmpty, ParentHandler, from_fn_async};
use serde::{Deserialize, Serialize};
use ts_rs::TS;
use crate::context::RpcContext;
use crate::db::model::DatabaseModel;
use crate::db::model::public::{NetworkInterfaceInfo, NetworkInterfaceType};
use crate::hostname::ServerHostname;
use crate::net::forward::AvailablePorts;
use crate::net::host::address::{HostAddress, PublicDomainConfig, address_api};
use crate::net::host::binding::{BindInfo, BindOptions, binding};
use crate::net::service_interface::HostnameInfo;
use crate::net::tor::OnionAddress;
use crate::net::host::binding::{BindInfo, BindOptions, Bindings, binding};
use crate::net::service_interface::{HostnameInfo, HostnameMetadata};
use crate::prelude::*;
use crate::{HostId, PackageId};
use crate::{GatewayId, HostId, PackageId};
pub mod address;
pub mod binding;
@@ -27,13 +30,23 @@ pub mod binding;
#[model = "Model<Self>"]
#[ts(export)]
pub struct Host {
pub bindings: BTreeMap<u16, BindInfo>,
#[ts(type = "string[]")]
pub onions: BTreeSet<OnionAddress>,
pub bindings: Bindings,
pub public_domains: BTreeMap<InternedString, PublicDomainConfig>,
pub private_domains: BTreeSet<InternedString>,
/// COMPUTED: NetService::update
pub hostname_info: BTreeMap<u16, Vec<HostnameInfo>>, // internal port -> Hostnames
pub private_domains: BTreeMap<InternedString, BTreeSet<GatewayId>>,
/// COMPUTED: port forwarding rules needed on gateways for public addresses to work.
#[serde(default)]
pub port_forwards: BTreeSet<PortForward>,
}
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Deserialize, Serialize, TS)]
#[serde(rename_all = "camelCase")]
#[ts(export)]
pub struct PortForward {
#[ts(type = "string")]
pub src: SocketAddrV4,
#[ts(type = "string")]
pub dst: SocketAddrV4,
pub gateway: GatewayId,
}
impl AsRef<Host> for Host {
@@ -46,31 +59,287 @@ impl Host {
Self::default()
}
pub fn addresses<'a>(&'a self) -> impl Iterator<Item = HostAddress> + 'a {
self.onions
self.public_domains
.iter()
.cloned()
.map(|address| HostAddress::Onion { address })
.chain(
self.public_domains
.iter()
.map(|(address, config)| HostAddress::Domain {
address: address.clone(),
public: Some(config.clone()),
private: self.private_domains.contains(address),
}),
)
.map(|(address, config)| HostAddress {
address: address.clone(),
public: Some(config.clone()),
private: self.private_domains.get(address).cloned(),
})
.chain(
self.private_domains
.iter()
.filter(|a| !self.public_domains.contains_key(*a))
.map(|address| HostAddress::Domain {
address: address.clone(),
.filter(|(domain, _)| !self.public_domains.contains_key(*domain))
.map(|(domain, gateways)| HostAddress {
address: domain.clone(),
public: None,
private: true,
private: Some(gateways.clone()),
}),
)
}
}
impl Model<Host> {
pub fn update_addresses(
&mut self,
mdns: &ServerHostname,
gateways: &OrdMap<GatewayId, NetworkInterfaceInfo>,
available_ports: &AvailablePorts,
) -> Result<(), Error> {
let this = self.destructure_mut();
// ips
for (_, bind) in this.bindings.as_entries_mut()? {
let net = bind.as_net().de()?;
let opt = bind.as_options().de()?;
// Preserve existing plugin-provided addresses across recomputation
let mut available = bind.as_addresses().as_available().de()?;
available.retain(|h| matches!(h.metadata, HostnameMetadata::Plugin { .. }));
for (gid, g) in gateways {
let Some(ip_info) = &g.ip_info else {
continue;
};
let gateway_secure = g.secure();
for subnet in &ip_info.subnets {
let host = InternedString::from_display(&subnet.addr());
let metadata = if subnet.addr().is_ipv4() {
HostnameMetadata::Ipv4 {
gateway: gid.clone(),
}
} else {
HostnameMetadata::Ipv6 {
gateway: gid.clone(),
scope_id: ip_info.scope_id,
}
};
if let Some(port) = net.assigned_port.filter(|_| {
opt.secure
.map_or(gateway_secure, |s| !(s.ssl && opt.add_ssl.is_some()))
}) {
available.insert(HostnameInfo {
ssl: opt.secure.map_or(false, |s| s.ssl),
public: false,
hostname: host.clone(),
port: Some(port),
metadata: metadata.clone(),
});
}
if let Some(port) = net.assigned_ssl_port {
available.insert(HostnameInfo {
ssl: true,
public: false,
hostname: host.clone(),
port: Some(port),
metadata,
});
}
}
if let Some(wan_ip) = &ip_info.wan_ip {
let host = InternedString::from_display(&wan_ip);
let metadata = HostnameMetadata::Ipv4 {
gateway: gid.clone(),
};
if let Some(port) = net.assigned_port.filter(|_| {
opt.secure.map_or(
false, // the public internet is never secure
|s| !(s.ssl && opt.add_ssl.is_some()),
)
}) {
available.insert(HostnameInfo {
ssl: opt.secure.map_or(false, |s| s.ssl),
public: true,
hostname: host.clone(),
port: Some(port),
metadata: metadata.clone(),
});
}
if let Some(port) = net.assigned_ssl_port {
available.insert(HostnameInfo {
ssl: true,
public: true,
hostname: host.clone(),
port: Some(port),
metadata,
});
}
}
}
// mdns
let mdns_host = mdns.local_domain_name();
let mdns_gateways: BTreeSet<GatewayId> = gateways
.iter()
.filter(|(_, g)| {
matches!(
g.ip_info.as_ref().and_then(|i| i.device_type),
Some(NetworkInterfaceType::Ethernet | NetworkInterfaceType::Wireless)
)
})
.map(|(id, _)| id.clone())
.collect();
if let Some(port) = net.assigned_port.filter(|_| {
opt.secure
.map_or(true, |s| !(s.ssl && opt.add_ssl.is_some()))
}) {
let mdns_gateways = if opt.secure.is_some() {
mdns_gateways.clone()
} else {
mdns_gateways
.iter()
.filter(|g| gateways.get(*g).map_or(false, |g| g.secure()))
.cloned()
.collect()
};
if !mdns_gateways.is_empty() {
available.insert(HostnameInfo {
ssl: opt.secure.map_or(false, |s| s.ssl),
public: false,
hostname: mdns_host.clone(),
port: Some(port),
metadata: HostnameMetadata::Mdns {
gateways: mdns_gateways,
},
});
}
}
if let Some(port) = net.assigned_ssl_port {
available.insert(HostnameInfo {
ssl: true,
public: false,
hostname: mdns_host,
port: Some(port),
metadata: HostnameMetadata::Mdns {
gateways: mdns_gateways,
},
});
}
// public domains
for (domain, info) in this.public_domains.de()? {
let metadata = HostnameMetadata::PublicDomain {
gateway: info.gateway.clone(),
};
if let Some(port) = net.assigned_port.filter(|_| {
opt.secure.map_or(
false, // the public internet is never secure
|s| !(s.ssl && opt.add_ssl.is_some()),
)
}) {
available.insert(HostnameInfo {
ssl: opt.secure.map_or(false, |s| s.ssl),
public: true,
hostname: domain.clone(),
port: Some(port),
metadata: metadata.clone(),
});
}
if let Some(mut port) = net.assigned_ssl_port {
if let Some(preferred) = opt
.add_ssl
.as_ref()
.map(|s| s.preferred_external_port)
.filter(|p| available_ports.is_ssl(*p))
{
port = preferred;
}
available.insert(HostnameInfo {
ssl: true,
public: true,
hostname: domain,
port: Some(port),
metadata,
});
}
}
// private domains
for (domain, domain_gateways) in this.private_domains.de()? {
if let Some(port) = net.assigned_port.filter(|_| {
opt.secure
.map_or(true, |s| !(s.ssl && opt.add_ssl.is_some()))
}) {
let gateways = if opt.secure.is_some() {
domain_gateways.clone()
} else {
domain_gateways
.iter()
.cloned()
.filter(|g| gateways.get(g).map_or(false, |g| g.secure()))
.collect()
};
available.insert(HostnameInfo {
ssl: opt.secure.map_or(false, |s| s.ssl),
public: true,
hostname: domain.clone(),
port: Some(port),
metadata: HostnameMetadata::PrivateDomain { gateways },
});
}
if let Some(mut port) = net.assigned_ssl_port {
if let Some(preferred) = opt
.add_ssl
.as_ref()
.map(|s| s.preferred_external_port)
.filter(|p| available_ports.is_ssl(*p))
{
port = preferred;
}
available.insert(HostnameInfo {
ssl: true,
public: true,
hostname: domain,
port: Some(port),
metadata: HostnameMetadata::PrivateDomain {
gateways: domain_gateways,
},
});
}
}
bind.as_addresses_mut().as_available_mut().ser(&available)?;
}
// compute port forwards from available public addresses
let bindings: Bindings = this.bindings.de()?;
let mut port_forwards = BTreeSet::new();
for bind in bindings.values() {
for addr in bind.addresses.enabled() {
if !addr.public {
continue;
}
let Some(port) = addr.port else {
continue;
};
let gw_id = match &addr.metadata {
HostnameMetadata::Ipv4 { gateway }
| HostnameMetadata::PublicDomain { gateway } => gateway,
_ => continue,
};
let Some(gw_info) = gateways.get(gw_id) else {
continue;
};
let Some(ip_info) = &gw_info.ip_info else {
continue;
};
let Some(wan_ip) = ip_info.wan_ip else {
continue;
};
for subnet in &ip_info.subnets {
let IpAddr::V4(addr) = subnet.addr() else {
continue;
};
port_forwards.insert(PortForward {
src: SocketAddrV4::new(wan_ip, port),
dst: SocketAddrV4::new(addr, port),
gateway: gw_id.clone(),
});
}
}
}
this.port_forwards.ser(&port_forwards)?;
Ok(())
}
}
#[derive(Debug, Default, Deserialize, Serialize, HasModel, TS)]
#[model = "Model<Self>"]
@@ -112,22 +381,7 @@ pub fn host_for<'a>(
.as_hosts_mut(),
)
}
let tor_key = if host_info(db, package_id)?.as_idx(host_id).is_none() {
Some(
db.as_private_mut()
.as_key_store_mut()
.as_onion_mut()
.new_key()?,
)
} else {
None
};
host_info(db, package_id)?.upsert(host_id, || {
let mut h = Host::new();
h.onions
.insert(tor_key.or_not_found("generated tor key")?.onion_address());
Ok(h)
})
host_info(db, package_id)?.upsert(host_id, || Ok(Host::new()))
}
pub fn all_hosts(db: &mut DatabaseModel) -> impl Iterator<Item = Result<&mut Model<Host>, Error>> {
@@ -185,10 +439,6 @@ pub trait HostApiKind: 'static {
inheritance: &Self::Inheritance,
db: &'a mut DatabaseModel,
) -> Result<&'a mut Model<Host>, Error>;
fn sync_host(
ctx: &RpcContext,
inheritance: Self::Inheritance,
) -> impl Future<Output = Result<(), Error>> + Send;
}
pub struct ForPackage;
impl HostApiKind for ForPackage {
@@ -207,12 +457,6 @@ impl HostApiKind for ForPackage {
) -> Result<&'a mut Model<Host>, Error> {
host_for(db, Some(package), host)
}
async fn sync_host(ctx: &RpcContext, (package, host): Self::Inheritance) -> Result<(), Error> {
let service = ctx.services.get(&package).await;
let service_ref = service.as_ref().or_not_found(&package)?;
service_ref.sync_host(host).await?;
Ok(())
}
}
pub struct ForServer;
impl HostApiKind for ForServer {
@@ -228,9 +472,6 @@ impl HostApiKind for ForServer {
) -> Result<&'a mut Model<Host>, Error> {
host_for(db, None, &HostId::default())
}
async fn sync_host(ctx: &RpcContext, _: Self::Inheritance) -> Result<(), Error> {
ctx.os_net_service.sync_host(HostId::default()).await
}
}
pub fn host_api<C: Context>() -> ParentHandler<C, RequiresPackageId> {

11
core/src/net/keys.rs
View File

@@ -3,28 +3,21 @@ use serde::{Deserialize, Serialize};
use crate::account::AccountInfo;
use crate::net::acme::AcmeCertStore;
use crate::net::ssl::CertStore;
use crate::net::tor::OnionStore;
use crate::prelude::*;
#[derive(Debug, Deserialize, Serialize, HasModel)]
#[model = "Model<Self>"]
#[serde(rename_all = "camelCase")]
pub struct KeyStore {
pub onion: OnionStore,
pub local_certs: CertStore,
#[serde(default)]
pub acme: AcmeCertStore,
}
impl KeyStore {
pub fn new(account: &AccountInfo) -> Result<Self, Error> {
let mut res = Self {
onion: OnionStore::new(),
Ok(Self {
local_certs: CertStore::new(account)?,
acme: AcmeCertStore::new(),
};
for tor_key in account.tor_keys.iter().cloned() {
res.onion.insert(tor_key);
}
Ok(res)
})
}
}

2
core/src/net/mod.rs
View File

@@ -14,7 +14,6 @@ pub mod socks;
pub mod ssl;
pub mod static_server;
pub mod tls;
pub mod tor;
pub mod tunnel;
pub mod utils;
pub mod vhost;
@@ -23,7 +22,6 @@ pub mod wifi;
pub fn net_api<C: Context>() -> ParentHandler<C> {
ParentHandler::new()
.subcommand("tor", tor::tor_api::<C>().with_about("about.tor-commands"))
.subcommand(
"acme",
acme::acme_api::<C>().with_about("about.setup-acme-certificate"),

1039
core/src/net/net_controller.rs
View File

File diff suppressed because it is too large Load Diff

189
core/src/net/service_interface.rs
View File

@@ -1,36 +1,142 @@
use std::net::{Ipv4Addr, Ipv6Addr};
use std::collections::BTreeSet;
use std::net::SocketAddr;
use imbl_value::InternedString;
use serde::{Deserialize, Serialize};
use ts_rs::TS;
use crate::{GatewayId, HostId, ServiceInterfaceId};
use crate::prelude::*;
use crate::{ActionId, GatewayId, HostId, PackageId, ServiceInterfaceId};
#[derive(Clone, Debug, Deserialize, Serialize, TS)]
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Deserialize, Serialize, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub struct HostnameInfo {
pub ssl: bool,
pub public: bool,
pub hostname: InternedString,
pub port: Option<u16>,
pub metadata: HostnameMetadata,
}
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Deserialize, Serialize, TS)]
#[ts(export)]
#[serde(rename_all = "kebab-case")]
#[serde(rename_all_fields = "camelCase")]
#[serde(tag = "kind")]
pub enum HostnameInfo {
Ip {
gateway: GatewayInfo,
public: bool,
hostname: IpHostname,
pub enum HostnameMetadata {
Ipv4 {
gateway: GatewayId,
},
Onion {
hostname: OnionHostname,
Ipv6 {
gateway: GatewayId,
scope_id: u32,
},
Mdns {
gateways: BTreeSet<GatewayId>,
},
PrivateDomain {
gateways: BTreeSet<GatewayId>,
},
PublicDomain {
gateway: GatewayId,
},
Plugin {
package_id: PackageId,
remove_action: Option<ActionId>,
overflow_actions: Vec<ActionId>,
#[ts(type = "unknown")]
#[serde(default)]
info: Value,
},
}
impl HostnameInfo {
pub fn to_socket_addr(&self) -> Option<SocketAddr> {
let ip = self.hostname.parse().ok()?;
Some(SocketAddr::new(ip, self.port?))
}
pub fn to_san_hostname(&self) -> InternedString {
self.hostname.clone()
}
}
impl HostnameMetadata {
pub fn is_ip(&self) -> bool {
matches!(self, Self::Ipv4 { .. } | Self::Ipv6 { .. })
}
pub fn gateways(&self) -> Box<dyn Iterator<Item = &GatewayId> + '_> {
match self {
Self::Ip { hostname, .. } => hostname.to_san_hostname(),
Self::Onion { hostname } => hostname.to_san_hostname(),
Self::Ipv4 { gateway }
| Self::Ipv6 { gateway, .. }
| Self::PublicDomain { gateway } => Box::new(std::iter::once(gateway)),
Self::PrivateDomain { gateways } | Self::Mdns { gateways } => Box::new(gateways.iter()),
Self::Plugin { .. } => Box::new(std::iter::empty()),
}
}
}
#[derive(Clone, Debug, Deserialize, Serialize, TS)]
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Deserialize, Serialize, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub struct PluginHostnameInfo {
pub package_id: Option<PackageId>,
pub host_id: HostId,
pub internal_port: u16,
pub ssl: bool,
pub public: bool,
#[ts(type = "string")]
pub hostname: InternedString,
pub port: Option<u16>,
#[ts(type = "unknown")]
#[serde(default)]
pub info: Value,
}
impl PluginHostnameInfo {
/// Convert to a `HostnameInfo` with `Plugin` metadata, using the given plugin package ID.
pub fn to_hostname_info(
&self,
plugin_package: &PackageId,
remove_action: Option<ActionId>,
overflow_actions: Vec<ActionId>,
) -> HostnameInfo {
HostnameInfo {
ssl: self.ssl,
public: self.public,
hostname: self.hostname.clone(),
port: self.port,
metadata: HostnameMetadata::Plugin {
package_id: plugin_package.clone(),
info: self.info.clone(),
remove_action,
overflow_actions,
},
}
}
/// Check if a `HostnameInfo` with Plugin metadata matches this `PluginHostnameInfo`
/// (comparing address fields only, not row_actions).
pub fn matches_hostname_info(&self, h: &HostnameInfo, plugin_package: &PackageId) -> bool {
match &h.metadata {
HostnameMetadata::Plugin {
package_id, info, ..
} => {
package_id == plugin_package
&& h.ssl == self.ssl
&& h.public == self.public
&& h.hostname == self.hostname
&& h.port == self.port
&& *info == self.info
}
_ => false,
}
}
}
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Deserialize, Serialize, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub struct GatewayInfo {
@@ -39,63 +145,6 @@ pub struct GatewayInfo {
pub public: bool,
}
#[derive(Clone, Debug, Deserialize, Serialize, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub struct OnionHostname {
#[ts(type = "string")]
pub value: InternedString,
pub port: Option<u16>,
pub ssl_port: Option<u16>,
}
impl OnionHostname {
pub fn to_san_hostname(&self) -> InternedString {
self.value.clone()
}
}
#[derive(Clone, Debug, Deserialize, Serialize, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
#[serde(rename_all_fields = "camelCase")]
#[serde(tag = "kind")]
pub enum IpHostname {
Ipv4 {
value: Ipv4Addr,
port: Option<u16>,
ssl_port: Option<u16>,
},
Ipv6 {
value: Ipv6Addr,
#[serde(default)]
scope_id: u32,
port: Option<u16>,
ssl_port: Option<u16>,
},
Local {
#[ts(type = "string")]
value: InternedString,
port: Option<u16>,
ssl_port: Option<u16>,
},
Domain {
#[ts(type = "string")]
value: InternedString,
port: Option<u16>,
ssl_port: Option<u16>,
},
}
impl IpHostname {
pub fn to_san_hostname(&self) -> InternedString {
match self {
Self::Ipv4 { value, .. } => InternedString::from_display(value),
Self::Ipv6 { value, .. } => InternedString::from_display(value),
Self::Local { value, .. } => value.clone(),
Self::Domain { value, .. } => value.clone(),
}
}
}
#[derive(Clone, Debug, Deserialize, Serialize, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]

38
core/src/net/socks.rs
View File

@@ -8,7 +8,6 @@ use socks5_impl::server::{AuthAdaptor, ClientConnection, Server};
use tokio::net::{TcpListener, TcpStream};
use crate::HOST_IP;
use crate::net::tor::TorController;
use crate::prelude::*;
use crate::util::actor::background::BackgroundJobQueue;
use crate::util::future::NonDetachingJoinHandle;
@@ -22,7 +21,7 @@ pub struct SocksController {
_thread: NonDetachingJoinHandle<()>,
}
impl SocksController {
pub fn new(listen: SocketAddr, tor: TorController) -> Result<Self, Error> {
pub fn new(listen: SocketAddr) -> Result<Self, Error> {
Ok(Self {
_thread: tokio::spawn(async move {
let auth: AuthAdaptor<()> = Arc::new(NoAuth);
@@ -45,7 +44,6 @@ impl SocksController {
loop {
match server.accept().await {
Ok((stream, _)) => {
let tor = tor.clone();
bg.add_job(async move {
if let Err(e) = async {
match stream
@@ -57,40 +55,6 @@ impl SocksController {
.await
.with_kind(ErrorKind::Network)?
{
ClientConnection::Connect(
reply,
Address::DomainAddress(domain, port),
) if domain.ends_with(".onion") => {
if let Ok(mut target) = tor
.connect_onion(&domain.parse()?, port)
.await
{
let mut sock = reply
.reply(
Reply::Succeeded,
Address::unspecified(),
)
.await
.with_kind(ErrorKind::Network)?;
tokio::io::copy_bidirectional(
&mut sock,
&mut target,
)
.await
.with_kind(ErrorKind::Network)?;
} else {
let mut sock = reply
.reply(
Reply::HostUnreachable,
Address::unspecified(),
)
.await
.with_kind(ErrorKind::Network)?;
sock.shutdown()
.await
.with_kind(ErrorKind::Network)?;
}
}
ClientConnection::Connect(reply, addr) => {
if let Ok(mut target) = match addr {
Address::DomainAddress(domain, port) => {

7
core/src/net/ssl.rs
View File

@@ -33,7 +33,7 @@ use crate::SOURCE_DATE;
use crate::account::AccountInfo;
use crate::db::model::Database;
use crate::db::{DbAccess, DbAccessMut};
use crate::hostname::Hostname;
use crate::hostname::ServerHostname;
use crate::init::check_time_is_synchronized;
use crate::net::gateway::GatewayInfo;
use crate::net::tls::TlsHandler;
@@ -283,7 +283,7 @@ pub fn gen_nistp256() -> Result<PKey<Private>, Error> {
#[instrument(skip_all)]
pub fn make_root_cert(
root_key: &PKey<Private>,
hostname: &Hostname,
hostname: &ServerHostname,
start_time: SystemTime,
) -> Result<X509, Error> {
let mut builder = X509Builder::new()?;
@@ -300,7 +300,8 @@ pub fn make_root_cert(
builder.set_serial_number(&*rand_serial()?)?;
let mut subject_name_builder = X509NameBuilder::new()?;
subject_name_builder.append_entry_by_text("CN", &format!("{} Local Root CA", &*hostname.0))?;
subject_name_builder
.append_entry_by_text("CN", &format!("{} Local Root CA", hostname.as_ref()))?;
subject_name_builder.append_entry_by_text("O", "Start9")?;
subject_name_builder.append_entry_by_text("OU", "StartOS")?;
let subject_name = subject_name_builder.build();

42
core/src/net/static_server.rs
View File

@@ -9,14 +9,14 @@ use async_compression::tokio::bufread::GzipEncoder;
use axum::Router;
use axum::body::Body;
use axum::extract::{self as x, Request};
use axum::response::{IntoResponse, Redirect, Response};
use axum::response::Response;
use axum::routing::{any, get};
use base64::display::Base64Display;
use digest::Digest;
use futures::future::ready;
use http::header::{
ACCEPT_ENCODING, ACCEPT_RANGES, CACHE_CONTROL, CONNECTION, CONTENT_ENCODING, CONTENT_LENGTH,
CONTENT_RANGE, CONTENT_TYPE, ETAG, HOST, RANGE,
CONTENT_RANGE, CONTENT_TYPE, ETAG, RANGE,
};
use http::request::Parts as RequestParts;
use http::{HeaderValue, Method, StatusCode};
@@ -31,13 +31,11 @@ use tokio_util::io::ReaderStream;
use url::Url;
use crate::context::{DiagnosticContext, InitContext, RpcContext, SetupContext};
use crate::hostname::Hostname;
use crate::hostname::ServerHostname;
use crate::middleware::auth::Auth;
use crate::middleware::auth::session::ValidSessionToken;
use crate::middleware::cors::Cors;
use crate::middleware::db::SyncDb;
use crate::net::gateway::GatewayInfo;
use crate::net::tls::TlsHandshakeInfo;
use crate::prelude::*;
use crate::rpc_continuations::{Guid, RpcContinuations};
use crate::s9pk::S9pk;
@@ -89,30 +87,6 @@ impl UiContext for RpcContext {
.middleware(SyncDb::new())
}
fn extend_router(self, router: Router) -> Router {
async fn https_redirect_if_public_http(
req: Request,
next: axum::middleware::Next,
) -> Response {
if req
.extensions()
.get::<GatewayInfo>()
.map_or(false, |p| p.info.public())
&& req.extensions().get::<TlsHandshakeInfo>().is_none()
{
Redirect::temporary(&format!(
"https://{}{}",
req.headers()
.get(HOST)
.and_then(|s| s.to_str().ok())
.unwrap_or("localhost"),
req.uri()
))
.into_response()
} else {
next.run(req).await
}
}
router
.route("/proxy/{url}", {
let ctx = self.clone();
@@ -131,12 +105,12 @@ impl UiContext for RpcContext {
get(move || {
let ctx = self.clone();
async move {
ctx.account
.peek(|account| cert_send(&account.root_ca_cert, &account.hostname))
ctx.account.peek(|account| {
cert_send(&account.root_ca_cert, &account.hostname.hostname)
})
}
}),
)
.layer(axum::middleware::from_fn(https_redirect_if_public_http))
}
}
@@ -446,7 +420,7 @@ pub fn bad_request() -> Response {
.unwrap()
}
fn cert_send(cert: &X509, hostname: &Hostname) -> Result<Response, Error> {
fn cert_send(cert: &X509, hostname: &ServerHostname) -> Result<Response, Error> {
let pem = cert.to_pem()?;
Response::builder()
.status(StatusCode::OK)
@@ -462,7 +436,7 @@ fn cert_send(cert: &X509, hostname: &Hostname) -> Result<Response, Error> {
.header(http::header::CONTENT_LENGTH, pem.len())
.header(
http::header::CONTENT_DISPOSITION,
format!("attachment; filename={}.crt", &hostname.0),
format!("attachment; filename={}.crt", hostname.as_ref()),
)
.body(Body::from(pem))
.with_kind(ErrorKind::Network)

15
core/src/net/tls.rs
View File

@@ -1,5 +1,6 @@
use std::sync::Arc;
use std::task::{Poll, ready};
use std::time::Duration;
use futures::future::BoxFuture;
use futures::stream::FuturesUnordered;
@@ -170,7 +171,7 @@ where
let (metadata, stream) = ready!(self.accept.poll_accept(cx)?);
let mut tls_handler = self.tls_handler.clone();
let mut fut = async move {
let res = async {
let res = match tokio::time::timeout(Duration::from_secs(15), async {
let mut acceptor =
LazyConfigAcceptor::new(Acceptor::default(), BackTrackingIO::new(stream));
let mut mid: tokio_rustls::StartHandshake<BackTrackingIO<AcceptStream>> =
@@ -233,14 +234,22 @@ where
}
Ok(None)
}
.await;
})
.await
{
Ok(res) => res,
Err(_) => {
tracing::trace!("TLS handshake timed out");
Ok(None)
}
};
(tls_handler, res)
}
.boxed();
match fut.poll_unpin(cx) {
Poll::Pending => {
in_progress.push(fut);
cx.waker().wake_by_ref();
Poll::Pending
}
Poll::Ready((handler, res)) => {

964
core/src/net/tor/arti.rs
View File

@@ -1,964 +0,0 @@
use std::borrow::Cow;
use std::collections::{BTreeMap, BTreeSet};
use std::net::SocketAddr;
use std::str::FromStr;
use std::sync::{Arc, Weak};
use std::time::{Duration, Instant};
use arti_client::config::onion_service::OnionServiceConfigBuilder;
use arti_client::{TorClient, TorClientConfig};
use base64::Engine;
use clap::Parser;
use color_eyre::eyre::eyre;
use futures::{FutureExt, StreamExt};
use imbl_value::InternedString;
use itertools::Itertools;
use rpc_toolkit::{Context, Empty, HandlerExt, ParentHandler, from_fn_async};
use serde::{Deserialize, Serialize};
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpStream;
use tokio::sync::Notify;
use tor_cell::relaycell::msg::Connected;
use tor_hscrypto::pk::{HsId, HsIdKeypair};
use tor_hsservice::status::State as ArtiOnionServiceState;
use tor_hsservice::{HsNickname, RunningOnionService};
use tor_keymgr::config::ArtiKeystoreKind;
use tor_proto::client::stream::IncomingStreamRequest;
use tor_rtcompat::tokio::TokioRustlsRuntime;
use ts_rs::TS;
use crate::context::{CliContext, RpcContext};
use crate::prelude::*;
use crate::util::actor::background::BackgroundJobQueue;
use crate::util::future::{NonDetachingJoinHandle, Until};
use crate::util::io::ReadWriter;
use crate::util::serde::{
BASE64, Base64, HandlerExtSerde, WithIoFormat, deserialize_from_str, display_serializable,
serialize_display,
};
use crate::util::sync::{SyncMutex, SyncRwLock, Watch};
const BOOTSTRAP_PROGRESS_TIMEOUT: Duration = Duration::from_secs(300);
const HS_BOOTSTRAP_TIMEOUT: Duration = Duration::from_secs(300);
const RETRY_COOLDOWN: Duration = Duration::from_secs(15);
const HEALTH_CHECK_FAILURE_ALLOWANCE: usize = 5;
const HEALTH_CHECK_COOLDOWN: Duration = Duration::from_secs(120);
#[derive(Debug, Clone, Copy)]
pub struct OnionAddress(pub HsId);
impl std::fmt::Display for OnionAddress {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
safelog::DisplayRedacted::fmt_unredacted(&self.0, f)
}
}
impl FromStr for OnionAddress {
type Err = Error;
fn from_str(s: &str) -> Result<Self, Self::Err> {
Ok(Self(
if s.ends_with(".onion") {
Cow::Borrowed(s)
} else {
Cow::Owned(format!("{s}.onion"))
}
.parse::<HsId>()
.with_kind(ErrorKind::Tor)?,
))
}
}
impl Serialize for OnionAddress {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
serialize_display(self, serializer)
}
}
impl<'de> Deserialize<'de> for OnionAddress {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: serde::Deserializer<'de>,
{
deserialize_from_str(deserializer)
}
}
impl PartialEq for OnionAddress {
fn eq(&self, other: &Self) -> bool {
self.0.as_ref() == other.0.as_ref()
}
}
impl Eq for OnionAddress {}
impl PartialOrd for OnionAddress {
fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {
self.0.as_ref().partial_cmp(other.0.as_ref())
}
}
impl Ord for OnionAddress {
fn cmp(&self, other: &Self) -> std::cmp::Ordering {
self.0.as_ref().cmp(other.0.as_ref())
}
}
pub struct TorSecretKey(pub HsIdKeypair);
impl TorSecretKey {
pub fn onion_address(&self) -> OnionAddress {
OnionAddress(HsId::from(self.0.as_ref().public().to_bytes()))
}
pub fn from_bytes(bytes: [u8; 64]) -> Result<Self, Error> {
Ok(Self(
tor_llcrypto::pk::ed25519::ExpandedKeypair::from_secret_key_bytes(bytes)
.ok_or_else(|| {
Error::new(
eyre!("{}", t!("net.tor.invalid-ed25519-key")),
ErrorKind::Tor,
)
})?
.into(),
))
}
pub fn generate() -> Self {
Self(
tor_llcrypto::pk::ed25519::ExpandedKeypair::from(
&tor_llcrypto::pk::ed25519::Keypair::generate(&mut rand::rng()),
)
.into(),
)
}
}
impl Clone for TorSecretKey {
fn clone(&self) -> Self {
Self(HsIdKeypair::from(
tor_llcrypto::pk::ed25519::ExpandedKeypair::from_secret_key_bytes(
self.0.as_ref().to_secret_key_bytes(),
)
.unwrap(),
))
}
}
impl std::fmt::Display for TorSecretKey {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(
f,
"{}",
BASE64.encode(self.0.as_ref().to_secret_key_bytes())
)
}
}
impl FromStr for TorSecretKey {
type Err = Error;
fn from_str(s: &str) -> Result<Self, Self::Err> {
Self::from_bytes(Base64::<[u8; 64]>::from_str(s)?.0)
}
}
impl Serialize for TorSecretKey {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
serialize_display(self, serializer)
}
}
impl<'de> Deserialize<'de> for TorSecretKey {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: serde::Deserializer<'de>,
{
deserialize_from_str(deserializer)
}
}
#[derive(Default, Deserialize, Serialize)]
pub struct OnionStore(BTreeMap<OnionAddress, TorSecretKey>);
impl Map for OnionStore {
type Key = OnionAddress;
type Value = TorSecretKey;
fn key_str(key: &Self::Key) -> Result<impl AsRef<str>, Error> {
Self::key_string(key)
}
fn key_string(key: &Self::Key) -> Result<imbl_value::InternedString, Error> {
Ok(InternedString::from_display(key))
}
}
impl OnionStore {
pub fn new() -> Self {
Self::default()
}
pub fn insert(&mut self, key: TorSecretKey) {
self.0.insert(key.onion_address(), key);
}
}
impl Model<OnionStore> {
pub fn new_key(&mut self) -> Result<TorSecretKey, Error> {
let key = TorSecretKey::generate();
self.insert(&key.onion_address(), &key)?;
Ok(key)
}
pub fn insert_key(&mut self, key: &TorSecretKey) -> Result<(), Error> {
self.insert(&key.onion_address(), &key)
}
pub fn get_key(&self, address: &OnionAddress) -> Result<TorSecretKey, Error> {
self.as_idx(address)
.or_not_found(lazy_format!("private key for {address}"))?
.de()
}
}
impl std::fmt::Debug for OnionStore {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
struct OnionStoreMap<'a>(&'a BTreeMap<OnionAddress, TorSecretKey>);
impl<'a> std::fmt::Debug for OnionStoreMap<'a> {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
#[derive(Debug)]
struct KeyFor(#[allow(unused)] OnionAddress);
let mut map = f.debug_map();
for (k, v) in self.0 {
map.key(k);
map.value(&KeyFor(v.onion_address()));
}
map.finish()
}
}
f.debug_tuple("OnionStore")
.field(&OnionStoreMap(&self.0))
.finish()
}
}
pub fn tor_api<C: Context>() -> ParentHandler<C> {
ParentHandler::new()
.subcommand(
"list-services",
from_fn_async(list_services)
.with_display_serializable()
.with_custom_display_fn(|handle, result| display_services(handle.params, result))
.with_about("about.display-tor-v3-onion-addresses")
.with_call_remote::<CliContext>(),
)
.subcommand(
"reset",
from_fn_async(reset)
.no_display()
.with_about("about.reset-tor-daemon")
.with_call_remote::<CliContext>(),
)
.subcommand(
"key",
key::<C>().with_about("about.manage-onion-service-key-store"),
)
}
pub fn key<C: Context>() -> ParentHandler<C> {
ParentHandler::new()
.subcommand(
"generate",
from_fn_async(generate_key)
.with_about("about.generate-onion-service-key-add-to-store")
.with_call_remote::<CliContext>(),
)
.subcommand(
"add",
from_fn_async(add_key)
.with_about("about.add-onion-service-key-to-store")
.with_call_remote::<CliContext>(),
)
.subcommand(
"list",
from_fn_async(list_keys)
.with_custom_display_fn(|_, res| {
for addr in res {
println!("{addr}");
}
Ok(())
})
.with_about("about.list-onion-services-with-keys-in-store")
.with_call_remote::<CliContext>(),
)
}
pub async fn generate_key(ctx: RpcContext) -> Result<OnionAddress, Error> {
ctx.db
.mutate(|db| {
Ok(db
.as_private_mut()
.as_key_store_mut()
.as_onion_mut()
.new_key()?
.onion_address())
})
.await
.result
}
#[derive(Deserialize, Serialize, Parser)]
pub struct AddKeyParams {
#[arg(help = "help.arg.onion-secret-key")]
pub key: Base64<[u8; 64]>,
}
pub async fn add_key(
ctx: RpcContext,
AddKeyParams { key }: AddKeyParams,
) -> Result<OnionAddress, Error> {
let key = TorSecretKey::from_bytes(key.0)?;
ctx.db
.mutate(|db| {
db.as_private_mut()
.as_key_store_mut()
.as_onion_mut()
.insert_key(&key)
})
.await
.result?;
Ok(key.onion_address())
}
pub async fn list_keys(ctx: RpcContext) -> Result<BTreeSet<OnionAddress>, Error> {
ctx.db
.peek()
.await
.into_private()
.into_key_store()
.into_onion()
.keys()
}
#[derive(Deserialize, Serialize, Parser, TS)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
pub struct ResetParams {
#[arg(
name = "wipe-state",
short = 'w',
long = "wipe-state",
help = "help.arg.wipe-tor-state"
)]
wipe_state: bool,
}
pub async fn reset(ctx: RpcContext, ResetParams { wipe_state }: ResetParams) -> Result<(), Error> {
ctx.net_controller.tor.reset(wipe_state).await
}
pub fn display_services(
params: WithIoFormat<Empty>,
services: BTreeMap<OnionAddress, OnionServiceInfo>,
) -> Result<(), Error> {
use prettytable::*;
if let Some(format) = params.format {
return display_serializable(format, services);
}
let mut table = Table::new();
table.add_row(row![bc => "ADDRESS", "STATE", "BINDINGS"]);
for (service, info) in services {
let row = row![
&service.to_string(),
&format!("{:?}", info.state),
&info
.bindings
.into_iter()
.map(|(port, addr)| lazy_format!("{port} -> {addr}"))
.join("; ")
];
table.add_row(row);
}
table.print_tty(false)?;
Ok(())
}
#[derive(Debug, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub enum OnionServiceState {
Shutdown,
Bootstrapping,
DegradedReachable,
DegradedUnreachable,
Running,
Recovering,
Broken,
}
impl From<ArtiOnionServiceState> for OnionServiceState {
fn from(value: ArtiOnionServiceState) -> Self {
match value {
ArtiOnionServiceState::Shutdown => Self::Shutdown,
ArtiOnionServiceState::Bootstrapping => Self::Bootstrapping,
ArtiOnionServiceState::DegradedReachable => Self::DegradedReachable,
ArtiOnionServiceState::DegradedUnreachable => Self::DegradedUnreachable,
ArtiOnionServiceState::Running => Self::Running,
ArtiOnionServiceState::Recovering => Self::Recovering,
ArtiOnionServiceState::Broken => Self::Broken,
_ => unreachable!(),
}
}
}
#[derive(Debug, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct OnionServiceInfo {
pub state: OnionServiceState,
pub bindings: BTreeMap<u16, SocketAddr>,
}
pub async fn list_services(
ctx: RpcContext,
_: Empty,
) -> Result<BTreeMap<OnionAddress, OnionServiceInfo>, Error> {
ctx.net_controller.tor.list_services().await
}
#[derive(Clone)]
pub struct TorController(Arc<TorControllerInner>);
struct TorControllerInner {
client: Watch<(usize, TorClient<TokioRustlsRuntime>)>,
_bootstrapper: NonDetachingJoinHandle<()>,
services: SyncMutex<BTreeMap<OnionAddress, OnionService>>,
reset: Arc<Notify>,
}
impl TorController {
pub fn new() -> Result<Self, Error> {
let mut config = TorClientConfig::builder();
config
.storage()
.keystore()
.primary()
.kind(ArtiKeystoreKind::Ephemeral.into());
let client = Watch::new((
0,
TorClient::with_runtime(TokioRustlsRuntime::current()?)
.config(config.build().with_kind(ErrorKind::Tor)?)
.local_resource_timeout(Duration::from_secs(0))
.create_unbootstrapped()?,
));
let reset = Arc::new(Notify::new());
let bootstrapper_reset = reset.clone();
let bootstrapper_client = client.clone();
let bootstrapper = tokio::spawn(async move {
loop {
let (epoch, client): (usize, _) = bootstrapper_client.read();
if let Err(e) = Until::new()
.with_async_fn(|| bootstrapper_reset.notified().map(Ok))
.run(async {
let mut events = client.bootstrap_events();
let bootstrap_fut =
client.bootstrap().map(|res| res.with_kind(ErrorKind::Tor));
let failure_fut = async {
let mut prev_frac = 0_f32;
let mut prev_inst = Instant::now();
while let Some(event) =
tokio::time::timeout(BOOTSTRAP_PROGRESS_TIMEOUT, events.next())
.await
.with_kind(ErrorKind::Tor)?
{
if event.ready_for_traffic() {
return Ok::<_, Error>(());
}
let frac = event.as_frac();
if frac == prev_frac {
if prev_inst.elapsed() > BOOTSTRAP_PROGRESS_TIMEOUT {
return Err(Error::new(
eyre!(
"{}",
t!(
"net.tor.bootstrap-no-progress",
duration = crate::util::serde::Duration::from(
BOOTSTRAP_PROGRESS_TIMEOUT
)
.to_string()
)
),
ErrorKind::Tor,
));
}
} else {
prev_frac = frac;
prev_inst = Instant::now();
}
}
futures::future::pending().await
};
if let Err::<(), Error>(e) = tokio::select! {
res = bootstrap_fut => res,
res = failure_fut => res,
} {
tracing::error!(
"{}",
t!("net.tor.bootstrap-error", error = e.to_string())
);
tracing::debug!("{e:?}");
} else {
bootstrapper_client.send_modify(|_| ());
for _ in 0..HEALTH_CHECK_FAILURE_ALLOWANCE {
if let Err::<(), Error>(e) = async {
loop {
let (bg, mut runner) = BackgroundJobQueue::new();
runner
.run_while(async {
const PING_BUF_LEN: usize = 8;
let key = TorSecretKey::generate();
let onion = key.onion_address();
let (hs, stream) = client
.launch_onion_service_with_hsid(
OnionServiceConfigBuilder::default()
.nickname(
onion
.to_string()
.trim_end_matches(".onion")
.parse::<HsNickname>()
.with_kind(ErrorKind::Tor)?,
)
.build()
.with_kind(ErrorKind::Tor)?,
key.clone().0,
)
.with_kind(ErrorKind::Tor)?;
bg.add_job(async move {
if let Err(e) = async {
let mut stream =
tor_hsservice::handle_rend_requests(
stream,
);
while let Some(req) = stream.next().await {
let mut stream = req
.accept(Connected::new_empty())
.await
.with_kind(ErrorKind::Tor)?;
let mut buf = [0; PING_BUF_LEN];
stream.read_exact(&mut buf).await?;
stream.write_all(&buf).await?;
stream.flush().await?;
stream.shutdown().await?;
}
Ok::<_, Error>(())
}
.await
{
tracing::error!(
"{}",
t!(
"net.tor.health-error",
error = e.to_string()
)
);
tracing::debug!("{e:?}");
}
});
tokio::time::timeout(HS_BOOTSTRAP_TIMEOUT, async {
let mut status = hs.status_events();
while let Some(status) = status.next().await {
if status.state().is_fully_reachable() {
return Ok(());
}
}
Err(Error::new(
eyre!(
"{}",
t!("net.tor.status-stream-ended")
),
ErrorKind::Tor,
))
})
.await
.with_kind(ErrorKind::Tor)??;
let mut stream = client
.connect((onion.to_string(), 8080))
.await?;
let mut ping_buf = [0; PING_BUF_LEN];
rand::fill(&mut ping_buf);
stream.write_all(&ping_buf).await?;
stream.flush().await?;
let mut ping_res = [0; PING_BUF_LEN];
stream.read_exact(&mut ping_res).await?;
ensure_code!(
ping_buf == ping_res,
ErrorKind::Tor,
"ping buffer mismatch"
);
stream.shutdown().await?;
Ok::<_, Error>(())
})
.await?;
tokio::time::sleep(HEALTH_CHECK_COOLDOWN).await;
}
}
.await
{
tracing::error!(
"{}",
t!("net.tor.client-health-error", error = e.to_string())
);
tracing::debug!("{e:?}");
}
}
tracing::error!(
"{}",
t!(
"net.tor.health-check-failed-recycling",
count = HEALTH_CHECK_FAILURE_ALLOWANCE
)
);
}
Ok(())
})
.await
{
tracing::error!(
"{}",
t!("net.tor.bootstrapper-error", error = e.to_string())
);
tracing::debug!("{e:?}");
}
if let Err::<(), Error>(e) = async {
tokio::time::sleep(RETRY_COOLDOWN).await;
bootstrapper_client.send((
epoch.wrapping_add(1),
TorClient::with_runtime(TokioRustlsRuntime::current()?)
.config(config.build().with_kind(ErrorKind::Tor)?)
.local_resource_timeout(Duration::from_secs(0))
.create_unbootstrapped_async()
.await?,
));
tracing::debug!("TorClient recycled");
Ok(())
}
.await
{
tracing::error!(
"{}",
t!("net.tor.client-creation-error", error = e.to_string())
);
tracing::debug!("{e:?}");
}
}
})
.into();
Ok(Self(Arc::new(TorControllerInner {
client,
_bootstrapper: bootstrapper,
services: SyncMutex::new(BTreeMap::new()),
reset,
})))
}
pub fn service(&self, key: TorSecretKey) -> Result<OnionService, Error> {
self.0.services.mutate(|s| {
use std::collections::btree_map::Entry;
let addr = key.onion_address();
match s.entry(addr) {
Entry::Occupied(e) => Ok(e.get().clone()),
Entry::Vacant(e) => Ok(e
.insert(OnionService::launch(self.0.client.clone(), key)?)
.clone()),
}
})
}
pub async fn gc(&self, addr: Option<OnionAddress>) -> Result<(), Error> {
if let Some(addr) = addr {
if let Some(s) = self.0.services.mutate(|s| {
let rm = if let Some(s) = s.get(&addr) {
!s.gc()
} else {
false
};
if rm { s.remove(&addr) } else { None }
}) {
s.shutdown().await
} else {
Ok(())
}
} else {
for s in self.0.services.mutate(|s| {
let mut rm = Vec::new();
s.retain(|_, s| {
if s.gc() {
true
} else {
rm.push(s.clone());
false
}
});
rm
}) {
s.shutdown().await?;
}
Ok(())
}
}
pub async fn reset(&self, wipe_state: bool) -> Result<(), Error> {
self.0.reset.notify_waiters();
Ok(())
}
pub async fn list_services(&self) -> Result<BTreeMap<OnionAddress, OnionServiceInfo>, Error> {
Ok(self
.0
.services
.peek(|s| s.iter().map(|(a, s)| (a.clone(), s.info())).collect()))
}
pub async fn connect_onion(
&self,
addr: &OnionAddress,
port: u16,
) -> Result<Box<dyn ReadWriter + Unpin + Send + Sync + 'static>, Error> {
if let Some(target) = self.0.services.peek(|s| {
s.get(addr).and_then(|s| {
s.0.bindings.peek(|b| {
b.get(&port).and_then(|b| {
b.iter()
.find(|(_, rc)| rc.strong_count() > 0)
.map(|(a, _)| *a)
})
})
})
}) {
let tcp_stream = TcpStream::connect(target)
.await
.with_kind(ErrorKind::Network)?;
if let Err(e) = socket2::SockRef::from(&tcp_stream).set_keepalive(true) {
tracing::error!(
"{}",
t!("net.tor.failed-to-set-tcp-keepalive", error = e.to_string())
);
tracing::debug!("{e:?}");
}
Ok(Box::new(tcp_stream))
} else {
let mut client = self.0.client.clone();
client
.wait_for(|(_, c)| c.bootstrap_status().ready_for_traffic())
.await;
let stream = client
.read()
.1
.connect((addr.to_string(), port))
.await
.with_kind(ErrorKind::Tor)?;
Ok(Box::new(stream))
}
}
}
#[derive(Clone)]
pub struct OnionService(Arc<OnionServiceData>);
struct OnionServiceData {
service: Arc<SyncMutex<Option<Arc<RunningOnionService>>>>,
bindings: Arc<SyncRwLock<BTreeMap<u16, BTreeMap<SocketAddr, Weak<()>>>>>,
_thread: NonDetachingJoinHandle<()>,
}
impl OnionService {
fn launch(
mut client: Watch<(usize, TorClient<TokioRustlsRuntime>)>,
key: TorSecretKey,
) -> Result<Self, Error> {
let service = Arc::new(SyncMutex::new(None));
let bindings = Arc::new(SyncRwLock::new(BTreeMap::<
u16,
BTreeMap<SocketAddr, Weak<()>>,
>::new()));
Ok(Self(Arc::new(OnionServiceData {
service: service.clone(),
bindings: bindings.clone(),
_thread: tokio::spawn(async move {
let (bg, mut runner) = BackgroundJobQueue::new();
runner
.run_while(async {
loop {
if let Err(e) = async {
client.wait_for(|(_,c)| c.bootstrap_status().ready_for_traffic()).await;
let epoch = client.peek(|(e, c)| {
ensure_code!(c.bootstrap_status().ready_for_traffic(), ErrorKind::Tor, "TorClient recycled");
Ok::<_, Error>(*e)
})?;
let addr = key.onion_address();
let (new_service, stream) = client.peek(|(_, c)| {
c.launch_onion_service_with_hsid(
OnionServiceConfigBuilder::default()
.nickname(
addr
.to_string()
.trim_end_matches(".onion")
.parse::<HsNickname>()
.with_kind(ErrorKind::Tor)?,
)
.build()
.with_kind(ErrorKind::Tor)?,
key.clone().0,
)
.with_kind(ErrorKind::Tor)
})?;
let mut status_stream = new_service.status_events();
let mut status = new_service.status();
if status.state().is_fully_reachable() {
tracing::debug!("{addr} is fully reachable");
} else {
tracing::debug!("{addr} is not fully reachable");
}
bg.add_job(async move {
while let Some(new_status) = status_stream.next().await {
if status.state().is_fully_reachable() && !new_status.state().is_fully_reachable() {
tracing::debug!("{addr} is no longer fully reachable");
} else if !status.state().is_fully_reachable() && new_status.state().is_fully_reachable() {
tracing::debug!("{addr} is now fully reachable");
}
status = new_status;
// TODO: health daemon?
}
});
service.replace(Some(new_service));
let mut stream = tor_hsservice::handle_rend_requests(stream);
while let Some(req) = tokio::select! {
req = stream.next() => req,
_ = client.wait_for(|(e, _)| *e != epoch) => None
} {
bg.add_job({
let bg = bg.clone();
let bindings = bindings.clone();
async move {
if let Err(e) = async {
let IncomingStreamRequest::Begin(begin) =
req.request()
else {
return req
.reject(tor_cell::relaycell::msg::End::new_with_reason(
tor_cell::relaycell::msg::EndReason::DONE,
))
.await
.with_kind(ErrorKind::Tor);
};
let Some(target) = bindings.peek(|b| {
b.get(&begin.port()).and_then(|a| {
a.iter()
.find(|(_, rc)| rc.strong_count() > 0)
.map(|(addr, _)| *addr)
})
}) else {
return req
.reject(tor_cell::relaycell::msg::End::new_with_reason(
tor_cell::relaycell::msg::EndReason::DONE,
))
.await
.with_kind(ErrorKind::Tor);
};
bg.add_job(async move {
if let Err(e) = async {
let mut outgoing =
TcpStream::connect(target)
.await
.with_kind(ErrorKind::Network)?;
if let Err(e) = socket2::SockRef::from(&outgoing).set_keepalive(true) {
tracing::error!("{}", t!("net.tor.failed-to-set-tcp-keepalive", error = e.to_string()));
tracing::debug!("{e:?}");
}
let mut incoming = req
.accept(Connected::new_empty())
.await
.with_kind(ErrorKind::Tor)?;
if let Err(e) =
tokio::io::copy_bidirectional(
&mut outgoing,
&mut incoming,
)
.await
{
tracing::trace!("Tor Stream Error: {e}");
tracing::trace!("{e:?}");
}
Ok::<_, Error>(())
}
.await
{
tracing::trace!("Tor Stream Error: {e}");
tracing::trace!("{e:?}");
}
});
Ok::<_, Error>(())
}
.await
{
tracing::trace!("Tor Request Error: {e}");
tracing::trace!("{e:?}");
}
}
});
}
Ok::<_, Error>(())
}
.await
{
tracing::error!("{}", t!("net.tor.client-error", error = e.to_string()));
tracing::debug!("{e:?}");
}
}
})
.await
})
.into(),
})))
}
pub async fn proxy_all<Rcs: FromIterator<Arc<()>>>(
&self,
bindings: impl IntoIterator<Item = (u16, SocketAddr)>,
) -> Result<Rcs, Error> {
Ok(self.0.bindings.mutate(|b| {
bindings
.into_iter()
.map(|(port, target)| {
let entry = b.entry(port).or_default().entry(target).or_default();
if let Some(rc) = entry.upgrade() {
rc
} else {
let rc = Arc::new(());
*entry = Arc::downgrade(&rc);
rc
}
})
.collect()
}))
}
pub fn gc(&self) -> bool {
self.0.bindings.mutate(|b| {
b.retain(|_, targets| {
targets.retain(|_, rc| rc.strong_count() > 0);
!targets.is_empty()
});
!b.is_empty()
})
}
pub async fn shutdown(self) -> Result<(), Error> {
self.0.service.replace(None);
self.0._thread.abort();
Ok(())
}
pub fn state(&self) -> OnionServiceState {
self.0
.service
.peek(|s| s.as_ref().map(|s| s.status().state().into()))
.unwrap_or(OnionServiceState::Bootstrapping)
}
pub fn info(&self) -> OnionServiceInfo {
OnionServiceInfo {
state: self.state(),
bindings: self.0.bindings.peek(|b| {
b.iter()
.filter_map(|(port, b)| {
b.iter()
.find(|(_, rc)| rc.strong_count() > 0)
.map(|(addr, _)| (*port, *addr))
})
.collect()
}),
}
}
}

1092
core/src/net/tor/ctor.rs
View File

File diff suppressed because it is too large Load Diff

10
core/src/net/tor/mod.rs
View File

@@ -1,10 +0,0 @@
#[cfg(feature = "arti")]
mod arti;
#[cfg(not(feature = "arti"))]
mod ctor;
#[cfg(feature = "arti")]
pub use arti::{OnionAddress, OnionStore, TorController, TorSecretKey, tor_api};
#[cfg(not(feature = "arti"))]
pub use ctor::{OnionAddress, OnionStore, TorController, TorSecretKey, tor_api};

58
core/src/net/tunnel.rs
View File

@@ -8,7 +8,7 @@ use ts_rs::TS;
use crate::GatewayId;
use crate::context::{CliContext, RpcContext};
use crate::db::model::public::{NetworkInterfaceInfo, NetworkInterfaceType};
use crate::db::model::public::{GatewayType, NetworkInterfaceInfo, NetworkInterfaceType};
use crate::net::host::all_hosts;
use crate::prelude::*;
use crate::util::Invoke;
@@ -32,14 +32,19 @@ pub fn tunnel_api<C: Context>() -> ParentHandler<C> {
}
#[derive(Debug, Clone, Deserialize, Serialize, Parser, TS)]
#[serde(rename_all = "camelCase")]
#[ts(export)]
pub struct AddTunnelParams {
#[arg(help = "help.arg.tunnel-name")]
name: InternedString,
#[arg(help = "help.arg.wireguard-config")]
config: String,
#[arg(help = "help.arg.is-public")]
public: bool,
#[arg(help = "help.arg.gateway-type")]
#[serde(default, rename = "type")]
gateway_type: Option<GatewayType>,
#[arg(help = "help.arg.set-as-default-outbound")]
#[serde(default)]
set_as_default_outbound: bool,
}
fn sanitize_config(config: &str) -> String {
@@ -64,7 +69,8 @@ pub async fn add_tunnel(
AddTunnelParams {
name,
config,
public,
gateway_type,
set_as_default_outbound,
}: AddTunnelParams,
) -> Result<GatewayId, Error> {
let ifaces = ctx.net_controller.net_iface.watcher.subscribe();
@@ -76,9 +82,9 @@ pub async fn add_tunnel(
iface.clone(),
NetworkInterfaceInfo {
name: Some(name),
public: Some(public),
secure: None,
ip_info: None,
gateway_type,
},
);
return true;
@@ -120,6 +126,19 @@ pub async fn add_tunnel(
sub.recv().await;
if set_as_default_outbound {
ctx.db
.mutate(|db| {
db.as_public_mut()
.as_server_info_mut()
.as_network_mut()
.as_default_outbound_mut()
.ser(&Some(iface.clone()))
})
.await
.result?;
}
Ok(iface)
}
@@ -156,10 +175,19 @@ pub async fn remove_tunnel(
ctx.db
.mutate(|db| {
let hostname = crate::hostname::ServerHostname::load(db.as_public().as_server_info())?;
let gateways = db
.as_public()
.as_server_info()
.as_network()
.as_gateways()
.de()?;
let ports = db.as_private().as_available_ports().de()?;
for host in all_hosts(db) {
let host = host?;
host.as_public_domains_mut()
.mutate(|p| Ok(p.retain(|_, v| v.gateway != id)))?;
host.update_addresses(&hostname, &gateways, &ports)?;
}
Ok(())
@@ -171,14 +199,24 @@ pub async fn remove_tunnel(
ctx.db
.mutate(|db| {
let hostname = crate::hostname::ServerHostname::load(db.as_public().as_server_info())?;
let gateways = db
.as_public()
.as_server_info()
.as_network()
.as_gateways()
.de()?;
let ports = db.as_private().as_available_ports().de()?;
for host in all_hosts(db) {
let host = host?;
host.as_bindings_mut().mutate(|b| {
Ok(b.values_mut().for_each(|v| {
v.net.private_disabled.remove(&id);
v.net.public_enabled.remove(&id);
}))
host.as_private_domains_mut().mutate(|d| {
for gateways in d.values_mut() {
gateways.remove(&id);
}
d.retain(|_, gateways| !gateways.is_empty());
Ok(())
})?;
host.update_addresses(&hostname, &gateways, &ports)?;
}
Ok(())

246
core/src/net/vhost.rs
View File

@@ -1,19 +1,19 @@
use std::any::Any;
use std::collections::{BTreeMap, BTreeSet};
use std::fmt;
use std::net::{IpAddr, SocketAddr};
use std::net::{IpAddr, SocketAddr, SocketAddrV6};
use std::sync::{Arc, Weak};
use std::task::{Poll, ready};
use std::time::Duration;
use async_acme::acme::ACME_TLS_ALPN_NAME;
use color_eyre::eyre::eyre;
use futures::FutureExt;
use futures::future::BoxFuture;
use imbl::OrdMap;
use imbl_value::{InOMap, InternedString};
use rpc_toolkit::{Context, HandlerArgs, HandlerExt, ParentHandler, from_fn};
use serde::{Deserialize, Serialize};
use tokio::net::TcpStream;
use tokio::net::{TcpListener, TcpStream};
use tokio_rustls::TlsConnector;
use tokio_rustls::rustls::crypto::CryptoProvider;
use tokio_rustls::rustls::pki_types::ServerName;
@@ -23,28 +23,28 @@ use tracing::instrument;
use ts_rs::TS;
use visit_rs::Visit;
use crate::ResultExt;
use crate::context::{CliContext, RpcContext};
use crate::db::model::Database;
use crate::db::model::public::AcmeSettings;
use crate::db::model::public::{AcmeSettings, NetworkInterfaceInfo};
use crate::db::{DbAccessByKey, DbAccessMut};
use crate::net::acme::{
AcmeCertStore, AcmeProvider, AcmeTlsAlpnCache, AcmeTlsHandler, GetAcmeProvider,
};
use crate::net::gateway::{
AnyFilter, BindTcp, DynInterfaceFilter, GatewayInfo, InterfaceFilter,
NetworkInterfaceController, NetworkInterfaceListener,
GatewayInfo, NetworkInterfaceController, NetworkInterfaceListenerAcceptMetadata,
};
use crate::net::ssl::{CertStore, RootCaTlsHandler};
use crate::net::tls::{
ChainedHandler, TlsHandlerWrapper, TlsListener, TlsMetadata, WrapTlsHandler,
};
use crate::net::utils::ipv6_is_link_local;
use crate::net::web_server::{Accept, AcceptStream, ExtractVisitor, TcpMetadata, extract};
use crate::prelude::*;
use crate::util::collections::EqSet;
use crate::util::future::{NonDetachingJoinHandle, WeakFuture};
use crate::util::serde::{HandlerExtSerde, MaybeUtf8String, display_serializable};
use crate::util::sync::{SyncMutex, Watch};
use crate::{GatewayId, ResultExt};
pub fn vhost_api<C: Context>() -> ParentHandler<C> {
ParentHandler::new().subcommand(
@@ -93,7 +93,7 @@ pub struct VHostController {
interfaces: Arc<NetworkInterfaceController>,
crypto_provider: Arc<CryptoProvider>,
acme_cache: AcmeTlsAlpnCache,
servers: SyncMutex<BTreeMap<u16, VHostServer<NetworkInterfaceListener>>>,
servers: SyncMutex<BTreeMap<u16, VHostServer<VHostBindListener>>>,
}
impl VHostController {
pub fn new(
@@ -114,14 +114,22 @@ impl VHostController {
&self,
hostname: Option<InternedString>,
external: u16,
target: DynVHostTarget<NetworkInterfaceListener>,
target: DynVHostTarget<VHostBindListener>,
) -> Result<Arc<()>, Error> {
self.servers.mutate(|writable| {
let server = if let Some(server) = writable.remove(&external) {
server
} else {
let bind_reqs = Watch::new(VHostBindRequirements::default());
let listener = VHostBindListener {
ip_info: self.interfaces.watcher.subscribe(),
port: external,
bind_reqs: bind_reqs.clone_unseen(),
listeners: BTreeMap::new(),
};
VHostServer::new(
self.interfaces.watcher.bind(BindTcp, external)?,
listener,
bind_reqs,
self.db.clone(),
self.crypto_provider.clone(),
self.acme_cache.clone(),
@@ -173,6 +181,142 @@ impl VHostController {
}
}
/// Union of all ProxyTargets' bind requirements for a VHostServer.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct VHostBindRequirements {
pub public_gateways: BTreeSet<GatewayId>,
pub private_ips: BTreeSet<IpAddr>,
}
fn compute_bind_reqs<A: Accept + 'static>(mapping: &Mapping<A>) -> VHostBindRequirements {
let mut reqs = VHostBindRequirements::default();
for (_, targets) in mapping {
for (target, rc) in targets {
if rc.strong_count() > 0 {
let (pub_gw, priv_ip) = target.0.bind_requirements();
reqs.public_gateways.extend(pub_gw);
reqs.private_ips.extend(priv_ip);
}
}
}
reqs
}
/// Listener that manages its own TCP listeners with IP-level precision.
/// Binds ALL IPs of public gateways and ONLY matching private IPs.
pub struct VHostBindListener {
ip_info: Watch<OrdMap<GatewayId, NetworkInterfaceInfo>>,
port: u16,
bind_reqs: Watch<VHostBindRequirements>,
listeners: BTreeMap<SocketAddr, (TcpListener, GatewayInfo)>,
}
fn update_vhost_listeners(
listeners: &mut BTreeMap<SocketAddr, (TcpListener, GatewayInfo)>,
port: u16,
ip_info: &OrdMap<GatewayId, NetworkInterfaceInfo>,
reqs: &VHostBindRequirements,
) -> Result<(), Error> {
let mut keep = BTreeSet::<SocketAddr>::new();
for (gw_id, info) in ip_info {
if let Some(ip_info) = &info.ip_info {
for ipnet in &ip_info.subnets {
let ip = ipnet.addr();
let should_bind =
reqs.public_gateways.contains(gw_id) || reqs.private_ips.contains(&ip);
if should_bind {
let addr = match ip {
IpAddr::V6(ip6) => SocketAddrV6::new(
ip6,
port,
0,
if ipv6_is_link_local(ip6) {
ip_info.scope_id
} else {
0
},
)
.into(),
ip => SocketAddr::new(ip, port),
};
keep.insert(addr);
if let Some((_, existing_info)) = listeners.get_mut(&addr) {
*existing_info = GatewayInfo {
id: gw_id.clone(),
info: info.clone(),
};
} else {
let tcp = TcpListener::from_std(
mio::net::TcpListener::bind(addr)
.with_kind(ErrorKind::Network)?
.into(),
)
.with_kind(ErrorKind::Network)?;
listeners.insert(
addr,
(
tcp,
GatewayInfo {
id: gw_id.clone(),
info: info.clone(),
},
),
);
}
}
}
}
}
listeners.retain(|key, _| keep.contains(key));
Ok(())
}
impl Accept for VHostBindListener {
type Metadata = NetworkInterfaceListenerAcceptMetadata;
fn poll_accept(
&mut self,
cx: &mut std::task::Context<'_>,
) -> Poll<Result<(Self::Metadata, AcceptStream), Error>> {
// Update listeners when ip_info or bind_reqs change
while self.ip_info.poll_changed(cx).is_ready() || self.bind_reqs.poll_changed(cx).is_ready()
{
let reqs = self.bind_reqs.read_and_mark_seen();
let listeners = &mut self.listeners;
let port = self.port;
self.ip_info.peek_and_mark_seen(|ip_info| {
update_vhost_listeners(listeners, port, ip_info, &reqs)
})?;
}
// Poll each listener for incoming connections
for (&addr, (listener, gw_info)) in &self.listeners {
match listener.poll_accept(cx) {
Poll::Ready(Ok((stream, peer_addr))) => {
if let Err(e) = socket2::SockRef::from(&stream).set_keepalive(true) {
tracing::error!("Failed to set tcp keepalive: {e}");
tracing::debug!("{e:?}");
}
return Poll::Ready(Ok((
NetworkInterfaceListenerAcceptMetadata {
inner: TcpMetadata {
local_addr: addr,
peer_addr,
},
info: gw_info.clone(),
},
Box::pin(stream),
)));
}
Poll::Ready(Err(e)) => {
tracing::trace!("VHostBindListener accept error on {addr}: {e}");
}
Poll::Pending => {}
}
}
Poll::Pending
}
}
pub trait VHostTarget<A: Accept>: std::fmt::Debug + Eq {
type PreprocessRes: Send + 'static;
#[allow(unused_variables)]
@@ -182,6 +326,10 @@ pub trait VHostTarget<A: Accept>: std::fmt::Debug + Eq {
fn acme(&self) -> Option<&AcmeProvider> {
None
}
/// Returns (public_gateways, private_ips) this target needs the listener to bind on.
fn bind_requirements(&self) -> (BTreeSet<GatewayId>, BTreeSet<IpAddr>) {
(BTreeSet::new(), BTreeSet::new())
}
fn preprocess<'a>(
&'a self,
prev: ServerConfig,
@@ -200,6 +348,7 @@ pub trait VHostTarget<A: Accept>: std::fmt::Debug + Eq {
pub trait DynVHostTargetT<A: Accept>: std::fmt::Debug + Any {
fn filter(&self, metadata: &<A as Accept>::Metadata) -> bool;
fn acme(&self) -> Option<&AcmeProvider>;
fn bind_requirements(&self) -> (BTreeSet<GatewayId>, BTreeSet<IpAddr>);
fn preprocess<'a>(
&'a self,
prev: ServerConfig,
@@ -224,6 +373,9 @@ impl<A: Accept, T: VHostTarget<A> + 'static> DynVHostTargetT<A> for T {
fn acme(&self) -> Option<&AcmeProvider> {
VHostTarget::acme(self)
}
fn bind_requirements(&self) -> (BTreeSet<GatewayId>, BTreeSet<IpAddr>) {
VHostTarget::bind_requirements(self)
}
fn preprocess<'a>(
&'a self,
prev: ServerConfig,
@@ -301,7 +453,8 @@ impl<A: Accept + 'static> Preprocessed<A> {
#[derive(Clone)]
pub struct ProxyTarget {
pub filter: DynInterfaceFilter,
pub public: BTreeSet<GatewayId>,
pub private: BTreeSet<IpAddr>,
pub acme: Option<AcmeProvider>,
pub addr: SocketAddr,
pub add_x_forwarded_headers: bool,
@@ -309,7 +462,8 @@ pub struct ProxyTarget {
}
impl PartialEq for ProxyTarget {
fn eq(&self, other: &Self) -> bool {
self.filter == other.filter
self.public == other.public
&& self.private == other.private
&& self.acme == other.acme
&& self.addr == other.addr
&& self.connect_ssl.as_ref().map(Arc::as_ptr)
@@ -320,7 +474,8 @@ impl Eq for ProxyTarget {}
impl fmt::Debug for ProxyTarget {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("ProxyTarget")
.field("filter", &self.filter)
.field("public", &self.public)
.field("private", &self.private)
.field("acme", &self.acme)
.field("addr", &self.addr)
.field("add_x_forwarded_headers", &self.add_x_forwarded_headers)
@@ -340,16 +495,35 @@ where
{
type PreprocessRes = AcceptStream;
fn filter(&self, metadata: &<A as Accept>::Metadata) -> bool {
let info = extract::<GatewayInfo, _>(metadata);
if info.is_none() {
tracing::warn!("No GatewayInfo on metadata");
let gw = extract::<GatewayInfo, _>(metadata);
let tcp = extract::<TcpMetadata, _>(metadata);
let (Some(gw), Some(tcp)) = (gw, tcp) else {
return false;
};
let Some(ip_info) = &gw.info.ip_info else {
return false;
};
let src = tcp.peer_addr.ip();
// Public: source is outside all known subnets (direct internet)
let is_public = !ip_info.subnets.iter().any(|s| s.contains(&src));
if is_public {
self.public.contains(&gw.id)
} else {
// Private: accept if connection arrived on an interface with a matching IP
ip_info
.subnets
.iter()
.any(|s| self.private.contains(&s.addr()))
}
info.as_ref()
.map_or(true, |i| self.filter.filter(&i.id, &i.info))
}
fn acme(&self) -> Option<&AcmeProvider> {
self.acme.as_ref()
}
fn bind_requirements(&self) -> (BTreeSet<GatewayId>, BTreeSet<IpAddr>) {
(self.public.clone(), self.private.clone())
}
async fn preprocess<'a>(
&'a self,
mut prev: ServerConfig,
@@ -518,6 +692,7 @@ where
let (target, rc) = self.0.peek(|m| {
m.get(&hello.server_name().map(InternedString::from))
.or_else(|| m.get(&None))
.into_iter()
.flatten()
.filter(|(_, rc)| rc.strong_count() > 0)
@@ -634,28 +809,15 @@ where
struct VHostServer<A: Accept + 'static> {
mapping: Watch<Mapping<A>>,
bind_reqs: Watch<VHostBindRequirements>,
_thread: NonDetachingJoinHandle<()>,
}
impl<'a> From<&'a BTreeMap<Option<InternedString>, BTreeMap<ProxyTarget, Weak<()>>>> for AnyFilter {
fn from(value: &'a BTreeMap<Option<InternedString>, BTreeMap<ProxyTarget, Weak<()>>>) -> Self {
Self(
value
.iter()
.flat_map(|(_, v)| {
v.iter()
.filter(|(_, r)| r.strong_count() > 0)
.map(|(t, _)| t.filter.clone())
})
.collect(),
)
}
}
impl<A: Accept> VHostServer<A> {
#[instrument(skip_all)]
fn new<M: HasModel>(
listener: A,
bind_reqs: Watch<VHostBindRequirements>,
db: TypedPatchDb<M>,
crypto_provider: Arc<CryptoProvider>,
acme_cache: AcmeTlsAlpnCache,
@@ -679,6 +841,7 @@ impl<A: Accept> VHostServer<A> {
let mapping = Watch::new(BTreeMap::new());
Self {
mapping: mapping.clone(),
bind_reqs,
_thread: tokio::spawn(async move {
let mut listener = VHostListener(TlsListener::new(
listener,
@@ -729,6 +892,9 @@ impl<A: Accept> VHostServer<A> {
targets.insert(target, Arc::downgrade(&rc));
writable.insert(hostname, targets);
res = Ok(rc);
if changed {
self.update_bind_reqs(writable);
}
changed
});
if self.mapping.watcher_count() > 1 {
@@ -752,9 +918,23 @@ impl<A: Accept> VHostServer<A> {
if !targets.is_empty() {
writable.insert(hostname, targets);
}
if pre != post {
self.update_bind_reqs(writable);
}
pre == post
});
}
fn update_bind_reqs(&self, mapping: &Mapping<A>) {
let new_reqs = compute_bind_reqs(mapping);
self.bind_reqs.send_if_modified(|reqs| {
if *reqs != new_reqs {
*reqs = new_reqs;
true
} else {
false
}
});
}
fn is_empty(&self) -> bool {
self.mapping.peek(|m| m.is_empty())
}

22
core/src/net/web_server.rs
View File

@@ -366,28 +366,6 @@ where
pub struct WebServerAcceptorSetter<A: Accept> {
acceptor: Watch<A>,
}
impl<A, B> WebServerAcceptorSetter<Option<Either<A, B>>>
where
A: Accept,
B: Accept<Metadata = A::Metadata>,
{
pub fn try_upgrade<F: FnOnce(A) -> Result<B, Error>>(&self, f: F) -> Result<(), Error> {
let mut res = Ok(());
self.acceptor.send_modify(|a| {
*a = match a.take() {
Some(Either::Left(a)) => match f(a) {
Ok(b) => Some(Either::Right(b)),
Err(e) => {
res = Err(e);
None
}
},
x => x,
}
});
res
}
}
impl<A: Accept> Deref for WebServerAcceptorSetter<A> {
type Target = Watch<A>;
fn deref(&self) -> &Self::Target {

30
core/src/net/wifi.rs
View File

@@ -85,6 +85,7 @@ pub fn wifi<C: Context>() -> ParentHandler<C> {
}
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
pub struct SetWifiEnabledParams {
@@ -150,16 +151,20 @@ pub fn country<C: Context>() -> ParentHandler<C> {
}
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
pub struct AddParams {
pub struct WifiAddParams {
#[arg(help = "help.arg.wifi-ssid")]
ssid: String,
#[arg(help = "help.arg.wifi-password")]
password: String,
}
#[instrument(skip_all)]
pub async fn add(ctx: RpcContext, AddParams { ssid, password }: AddParams) -> Result<(), Error> {
pub async fn add(
ctx: RpcContext,
WifiAddParams { ssid, password }: WifiAddParams,
) -> Result<(), Error> {
let wifi_manager = ctx.wifi_manager.clone();
if !ssid.is_ascii() {
return Err(Error::new(
@@ -229,15 +234,19 @@ pub async fn add(ctx: RpcContext, AddParams { ssid, password }: AddParams) -> Re
Ok(())
}
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
pub struct SsidParams {
pub struct WifiSsidParams {
#[arg(help = "help.arg.wifi-ssid")]
ssid: String,
}
#[instrument(skip_all)]
pub async fn connect(ctx: RpcContext, SsidParams { ssid }: SsidParams) -> Result<(), Error> {
pub async fn connect(
ctx: RpcContext,
WifiSsidParams { ssid }: WifiSsidParams,
) -> Result<(), Error> {
let wifi_manager = ctx.wifi_manager.clone();
if !ssid.is_ascii() {
return Err(Error::new(
@@ -311,7 +320,7 @@ pub async fn connect(ctx: RpcContext, SsidParams { ssid }: SsidParams) -> Result
}
#[instrument(skip_all)]
pub async fn remove(ctx: RpcContext, SsidParams { ssid }: SsidParams) -> Result<(), Error> {
pub async fn remove(ctx: RpcContext, WifiSsidParams { ssid }: WifiSsidParams) -> Result<(), Error> {
let wifi_manager = ctx.wifi_manager.clone();
if !ssid.is_ascii() {
return Err(Error::new(
@@ -359,11 +368,13 @@ pub async fn remove(ctx: RpcContext, SsidParams { ssid }: SsidParams) -> Result<
.result?;
Ok(())
}
#[derive(serde::Serialize, serde::Deserialize)]
#[derive(serde::Serialize, serde::Deserialize, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub struct WifiListInfo {
ssids: HashMap<Ssid, SignalStrength>,
connected: Option<Ssid>,
#[ts(type = "string | null")]
country: Option<CountryCode>,
ethernet: bool,
available_wifi: Vec<WifiListOut>,
@@ -374,7 +385,8 @@ pub struct WifiListInfoLow {
strength: SignalStrength,
security: Vec<String>,
}
#[derive(serde::Serialize, serde::Deserialize)]
#[derive(serde::Serialize, serde::Deserialize, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub struct WifiListOut {
ssid: Ssid,
@@ -560,6 +572,7 @@ pub async fn get_available(ctx: RpcContext, _: Empty) -> Result<Vec<WifiListOut>
}
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
pub struct SetCountryParams {
@@ -605,7 +618,7 @@ pub struct NetworkId(String);
/// Ssid are the names of the wifis, usually human readable.
#[derive(
Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, serde::Serialize, serde::Deserialize,
Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, serde::Serialize, serde::Deserialize, TS,
)]
pub struct Ssid(String);
@@ -622,6 +635,7 @@ pub struct Ssid(String);
Hash,
serde::Serialize,
serde::Deserialize,
TS,
)]
pub struct SignalStrength(u8);

13
core/src/notifications.rs
View File

@@ -75,6 +75,7 @@ pub fn notification<C: Context>() -> ParentHandler<C> {
}
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
pub struct ListNotificationParams {
@@ -140,6 +141,7 @@ pub async fn list(
}
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
pub struct ModifyNotificationParams {
@@ -175,6 +177,7 @@ pub async fn remove(
}
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
pub struct ModifyNotificationBeforeParams {
@@ -326,6 +329,7 @@ pub async fn create(
}
#[derive(Debug, Clone, PartialEq, Eq, Hash, serde::Serialize, serde::Deserialize, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub enum NotificationLevel {
Success,
@@ -396,26 +400,31 @@ impl Map for Notifications {
}
}
#[derive(Debug, Serialize, Deserialize, HasModel)]
#[derive(Debug, Serialize, Deserialize, HasModel, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
#[model = "Model<Self>"]
pub struct Notification {
pub package_id: Option<PackageId>,
#[ts(type = "string")]
pub created_at: DateTime<Utc>,
pub code: u32,
pub level: NotificationLevel,
pub title: String,
pub message: String,
#[ts(type = "any")]
pub data: Value,
#[serde(default = "const_true")]
pub seen: bool,
}
#[derive(Debug, Serialize, Deserialize)]
#[derive(Debug, Serialize, Deserialize, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub struct NotificationWithId {
id: u32,
#[serde(flatten)]
#[ts(flatten)]
notification: Notification,
}

1
core/src/os_install/mod.rs
View File

@@ -359,6 +359,7 @@ pub async fn install_os_to(
"riscv64" => install.arg("--target=riscv64-efi"),
_ => &mut install,
};
install.arg("--no-nvram");
}
install
.arg(disk_path)

17
core/src/registry/package/get.rs
View File

@@ -579,14 +579,12 @@ fn check_matching_info_short() {
use crate::s9pk::manifest::{Alerts, Description};
use crate::util::DataUrl;
let lang_map = |s: &str| {
LocaleString::LanguageMap([("en".into(), s.into())].into_iter().collect())
};
let lang_map =
|s: &str| LocaleString::LanguageMap([("en".into(), s.into())].into_iter().collect());
let info = PackageVersionInfo {
metadata: PackageMetadata {
title: "Test Package".into(),
icon: DataUrl::from_vec("image/png", vec![]),
description: Description {
short: lang_map("A short description"),
long: lang_map("A longer description of the test package"),
@@ -594,18 +592,19 @@ fn check_matching_info_short() {
release_notes: lang_map("Initial release"),
git_hash: None,
license: "MIT".into(),
wrapper_repo: "https://github.com/example/wrapper".parse().unwrap(),
package_repo: "https://github.com/example/wrapper".parse().unwrap(),
upstream_repo: "https://github.com/example/upstream".parse().unwrap(),
support_site: "https://example.com/support".parse().unwrap(),
marketing_site: "https://example.com".parse().unwrap(),
marketing_url: Some("https://example.com".parse().unwrap()),
donation_url: None,
docs_url: None,
docs_urls: Vec::new(),
alerts: Alerts::default(),
dependency_metadata: BTreeMap::new(),
os_version: exver::Version::new([0, 3, 6], []),
sdk_version: None,
hardware_acceleration: false,
plugins: BTreeSet::new(),
},
icon: DataUrl::from_vec("image/png", vec![]),
dependency_metadata: BTreeMap::new(),
source_version: None,
s9pks: Vec::new(),
};

150
core/src/registry/package/index.rs
View File

@@ -17,8 +17,11 @@ use crate::registry::device_info::DeviceInfo;
use crate::rpc_continuations::Guid;
use crate::s9pk::S9pk;
use crate::s9pk::git_hash::GitHash;
use crate::s9pk::manifest::{Alerts, Description, HardwareRequirements, LocaleString};
use crate::s9pk::manifest::{
Alerts, Description, HardwareRequirements, LocaleString, current_version,
};
use crate::s9pk::merkle_archive::source::FileSource;
use crate::service::effects::plugin::PluginId;
use crate::sign::commitment::merkle_archive::MerkleArchiveCommitment;
use crate::sign::{AnySignature, AnyVerifyingKey};
use crate::util::{DataUrl, VersionString};
@@ -69,75 +72,44 @@ impl DependencyMetadata {
}
}
#[derive(Debug, Deserialize, Serialize, HasModel, TS, PartialEq)]
fn placeholder_url() -> Url {
"https://example.com".parse().unwrap()
}
#[derive(Clone, Debug, Deserialize, Serialize, HasModel, TS, PartialEq)]
#[serde(rename_all = "camelCase")]
#[model = "Model<Self>"]
pub struct PackageMetadata {
#[ts(type = "string")]
pub title: InternedString,
pub icon: DataUrl<'static>,
pub description: Description,
pub release_notes: LocaleString,
pub git_hash: Option<GitHash>,
#[ts(type = "string")]
pub license: InternedString,
#[ts(type = "string")]
pub wrapper_repo: Url,
#[serde(default = "placeholder_url")] // TODO: remove
pub package_repo: Url,
#[ts(type = "string")]
pub upstream_repo: Url,
#[ts(type = "string")]
pub support_site: Url,
#[ts(type = "string")]
pub marketing_site: Url,
pub marketing_url: Option<Url>,
#[ts(type = "string | null")]
pub donation_url: Option<Url>,
#[ts(type = "string | null")]
pub docs_url: Option<Url>,
#[serde(default)]
#[ts(type = "string[]")]
pub docs_urls: Vec<Url>,
#[serde(default)]
pub alerts: Alerts,
pub dependency_metadata: BTreeMap<PackageId, DependencyMetadata>,
#[serde(default = "current_version")]
#[ts(type = "string")]
pub os_version: Version,
#[ts(type = "string | null")]
pub sdk_version: Option<Version>,
#[serde(default)]
pub hardware_acceleration: bool,
}
impl PackageMetadata {
pub async fn load<S: FileSource + Clone>(s9pk: &S9pk<S>) -> Result<Self, Error> {
let manifest = s9pk.as_manifest();
let mut dependency_metadata = BTreeMap::new();
for (id, info) in &manifest.dependencies.0 {
let metadata = s9pk.dependency_metadata(id).await?;
dependency_metadata.insert(
id.clone(),
DependencyMetadata {
title: metadata.map(|m| m.title),
icon: s9pk.dependency_icon_data_url(id).await?,
description: info.description.clone(),
optional: info.optional,
},
);
}
Ok(Self {
title: manifest.title.clone(),
icon: s9pk.icon_data_url().await?,
description: manifest.description.clone(),
release_notes: manifest.release_notes.clone(),
git_hash: manifest.git_hash.clone(),
license: manifest.license.clone(),
wrapper_repo: manifest.wrapper_repo.clone(),
upstream_repo: manifest.upstream_repo.clone(),
support_site: manifest.support_site.clone(),
marketing_site: manifest.marketing_site.clone(),
donation_url: manifest.donation_url.clone(),
docs_url: manifest.docs_url.clone(),
alerts: manifest.alerts.clone(),
dependency_metadata,
os_version: manifest.os_version.clone(),
sdk_version: manifest.sdk_version.clone(),
hardware_acceleration: manifest.hardware_acceleration.clone(),
})
}
#[serde(default)]
pub plugins: BTreeSet<PluginId>,
}
#[derive(Debug, Deserialize, Serialize, HasModel, TS)]
@@ -147,6 +119,8 @@ impl PackageMetadata {
pub struct PackageVersionInfo {
#[serde(flatten)]
pub metadata: PackageMetadata,
pub icon: DataUrl<'static>,
pub dependency_metadata: BTreeMap<PackageId, DependencyMetadata>,
#[ts(type = "string | null")]
pub source_version: Option<VersionRange>,
pub s9pks: Vec<(HardwareRequirements, RegistryAsset<MerkleArchiveCommitment>)>,
@@ -156,11 +130,28 @@ impl PackageVersionInfo {
s9pk: &S9pk<S>,
urls: Vec<Url>,
) -> Result<Self, Error> {
let manifest = s9pk.as_manifest();
let icon = s9pk.icon_data_url().await?;
let mut dependency_metadata = BTreeMap::new();
for (id, info) in &manifest.dependencies.0 {
let dep_meta = s9pk.dependency_metadata(id).await?;
dependency_metadata.insert(
id.clone(),
DependencyMetadata {
title: dep_meta.map(|m| m.title),
icon: s9pk.dependency_icon_data_url(id).await?,
description: info.description.clone(),
optional: info.optional,
},
);
}
Ok(Self {
metadata: PackageMetadata::load(s9pk).await?,
metadata: manifest.metadata.clone(),
icon,
dependency_metadata,
source_version: None, // TODO
s9pks: vec![(
s9pk.as_manifest().hardware_requirements.clone(),
manifest.hardware_requirements.clone(),
RegistryAsset {
published_at: Utc::now(),
urls,
@@ -176,6 +167,27 @@ impl PackageVersionInfo {
})
}
pub fn merge_with(&mut self, other: Self, replace_urls: bool) -> Result<(), Error> {
if self.metadata != other.metadata {
return Err(Error::new(
color_eyre::eyre::eyre!("{}", t!("registry.package.index.metadata-mismatch")),
ErrorKind::InvalidRequest,
));
}
if self.icon != other.icon {
return Err(Error::new(
color_eyre::eyre::eyre!("{}", t!("registry.package.index.icon-mismatch")),
ErrorKind::InvalidRequest,
));
}
if self.dependency_metadata != other.dependency_metadata {
return Err(Error::new(
color_eyre::eyre::eyre!(
"{}",
t!("registry.package.index.dependency-metadata-mismatch")
),
ErrorKind::InvalidRequest,
));
}
for (hw_req, asset) in other.s9pks {
if let Some((_, matching)) = self
.s9pks
@@ -221,10 +233,9 @@ impl PackageVersionInfo {
]);
table.add_row(row![br -> "GIT HASH", self.metadata.git_hash.as_deref().unwrap_or("N/A")]);
table.add_row(row![br -> "LICENSE", &self.metadata.license]);
table.add_row(row![br -> "PACKAGE REPO", &self.metadata.wrapper_repo.to_string()]);
table.add_row(row![br -> "PACKAGE REPO", &self.metadata.package_repo.to_string()]);
table.add_row(row![br -> "SERVICE REPO", &self.metadata.upstream_repo.to_string()]);
table.add_row(row![br -> "WEBSITE", &self.metadata.marketing_site.to_string()]);
table.add_row(row![br -> "SUPPORT", &self.metadata.support_site.to_string()]);
table.add_row(row![br -> "WEBSITE", self.metadata.marketing_url.as_ref().map_or("N/A".to_owned(), |u| u.to_string())]);
table
}
@@ -244,30 +255,7 @@ impl Model<PackageVersionInfo> {
}
if let Some(hw) = &device_info.hardware {
self.as_s9pks_mut().mutate(|s9pks| {
s9pks.retain(|(hw_req, _)| {
if let Some(arch) = &hw_req.arch {
if !arch.contains(&hw.arch) {
return false;
}
}
if let Some(ram) = hw_req.ram {
if hw.ram < ram {
return false;
}
}
if let Some(dev) = &hw.devices {
for device_filter in &hw_req.device {
if !dev
.iter()
.filter(|d| d.class() == &*device_filter.class)
.any(|d| device_filter.matches(d))
{
return false;
}
}
}
true
});
s9pks.retain(|(hw_req, _)| hw_req.is_compatible(hw));
if hw.devices.is_some() {
s9pks.sort_by_key(|(req, _)| req.specificity_desc());
} else {
@@ -287,19 +275,17 @@ impl Model<PackageVersionInfo> {
}
if let Some(locale) = device_info.os.language.as_deref() {
let metadata = self.as_metadata_mut();
metadata
self.as_metadata_mut()
.as_alerts_mut()
.mutate(|a| Ok(a.localize_for(locale)))?;
metadata
.as_dependency_metadata_mut()
self.as_dependency_metadata_mut()
.as_entries_mut()?
.into_iter()
.try_for_each(|(_, d)| d.mutate(|d| Ok(d.localize_for(locale))))?;
metadata
self.as_metadata_mut()
.as_description_mut()
.mutate(|d| Ok(d.localize_for(locale)))?;
metadata
self.as_metadata_mut()
.as_release_notes_mut()
.mutate(|r| Ok(r.localize_for(locale)))?;
}

110
core/src/s9pk/rpc.rs
View File

@@ -3,16 +3,17 @@ use std::path::PathBuf;
use std::sync::Arc;
use clap::Parser;
use rpc_toolkit::{Empty, HandlerExt, ParentHandler, from_fn_async};
use rpc_toolkit::{Empty, HandlerArgs, HandlerExt, ParentHandler, from_fn_async};
use serde::{Deserialize, Serialize};
use tokio::process::Command;
use ts_rs::TS;
use url::Url;
use crate::ImageId;
use crate::context::CliContext;
use crate::context::{CliContext, RpcContext};
use crate::prelude::*;
use crate::s9pk::manifest::Manifest;
use crate::registry::device_info::DeviceInfo;
use crate::s9pk::manifest::{HardwareRequirements, Manifest};
use crate::s9pk::merkle_archive::source::multi_cursor_file::MultiCursorFile;
use crate::s9pk::v2::SIG_CONTEXT;
use crate::s9pk::v2::pack::ImageConfig;
@@ -70,6 +71,15 @@ pub fn s9pk() -> ParentHandler<CliContext> {
.no_display()
.with_about("about.publish-s9pk"),
)
.subcommand(
"select",
from_fn_async(select)
.with_custom_display_fn(|_, path: PathBuf| {
println!("{}", path.display());
Ok(())
})
.with_about("about.select-s9pk-for-device"),
)
}
#[derive(Deserialize, Serialize, Parser)]
@@ -323,3 +333,97 @@ async fn publish(ctx: CliContext, S9pkPath { s9pk: s9pk_path }: S9pkPath) -> Res
.await?;
crate::registry::package::add::cli_add_package_impl(ctx, s9pk, vec![s3url], false).await
}
#[derive(Deserialize, Serialize, Parser)]
struct SelectParams {
#[arg(help = "help.arg.s9pk-file-paths")]
s9pks: Vec<PathBuf>,
}
async fn select(
HandlerArgs {
context,
params: SelectParams { s9pks },
..
}: HandlerArgs<CliContext, SelectParams>,
) -> Result<PathBuf, Error> {
// Resolve file list: use provided paths or scan cwd for *.s9pk
let paths = if s9pks.is_empty() {
let mut found = Vec::new();
let mut entries = tokio::fs::read_dir(".").await?;
while let Some(entry) = entries.next_entry().await? {
let path = entry.path();
if path.extension().and_then(|e| e.to_str()) == Some("s9pk") {
found.push(path);
}
}
if found.is_empty() {
return Err(Error::new(
eyre!("no .s9pk files found in current directory"),
ErrorKind::NotFound,
));
}
found
} else {
s9pks
};
// Fetch DeviceInfo from the target server
let device_info: DeviceInfo = from_value(
context
.call_remote::<RpcContext>("server.device-info", imbl_value::json!({}))
.await?,
)?;
// Filter and rank s9pk files by compatibility
let mut compatible: Vec<(PathBuf, HardwareRequirements)> = Vec::new();
for path in &paths {
let s9pk = match super::S9pk::open(path, None).await {
Ok(s9pk) => s9pk,
Err(e) => {
tracing::warn!("skipping {}: {e}", path.display());
continue;
}
};
let manifest = s9pk.as_manifest();
// OS version check: package's required OS version must be in server's compat range
if !manifest
.metadata
.os_version
.satisfies(&device_info.os.compat)
{
continue;
}
let hw_req = &manifest.hardware_requirements;
if let Some(hw) = &device_info.hardware {
if !hw_req.is_compatible(hw) {
continue;
}
}
compatible.push((path.clone(), hw_req.clone()));
}
if compatible.is_empty() {
return Err(Error::new(
eyre!(
"no compatible s9pk found for device (arch: {}, os: {})",
device_info
.hardware
.as_ref()
.map(|h| h.arch.to_string())
.unwrap_or_else(|| "unknown".into()),
device_info.os.version,
),
ErrorKind::NotFound,
));
}
// Sort by specificity (most specific first)
compatible.sort_by_key(|(_, req)| req.specificity_desc());
Ok(compatible.into_iter().next().unwrap().0)
}

39
core/src/s9pk/v2/compat.rs
View File

@@ -9,6 +9,7 @@ use tokio::process::Command;
use crate::dependencies::{DepInfo, Dependencies};
use crate::prelude::*;
use crate::registry::package::index::PackageMetadata;
use crate::s9pk::manifest::{DeviceFilter, LocaleString, Manifest};
use crate::s9pk::merkle_archive::directory_contents::DirectoryContents;
use crate::s9pk::merkle_archive::source::TmpSource;
@@ -195,20 +196,30 @@ impl TryFrom<ManifestV1> for Manifest {
}
Ok(Self {
id: value.id,
title: format!("{} (Legacy)", value.title).into(),
version: version.into(),
satisfies: BTreeSet::new(),
release_notes: LocaleString::Translated(value.release_notes),
can_migrate_from: VersionRange::any(),
can_migrate_to: VersionRange::none(),
license: value.license.into(),
wrapper_repo: value.wrapper_repo,
upstream_repo: value.upstream_repo,
support_site: value.support_site.unwrap_or_else(|| default_url.clone()),
marketing_site: value.marketing_site.unwrap_or_else(|| default_url.clone()),
donation_url: value.donation_url,
docs_url: None,
description: value.description,
metadata: PackageMetadata {
title: format!("{} (Legacy)", value.title).into(),
release_notes: LocaleString::Translated(value.release_notes),
license: value.license.into(),
package_repo: value.wrapper_repo,
upstream_repo: value.upstream_repo,
marketing_url: Some(value.marketing_site.unwrap_or_else(|| default_url.clone())),
donation_url: value.donation_url,
docs_urls: Vec::new(),
description: value.description,
alerts: value.alerts,
git_hash: value.git_hash,
os_version: value.eos_version,
sdk_version: None,
hardware_acceleration: match value.main {
PackageProcedure::Docker(d) => d.gpu_acceleration,
PackageProcedure::Script(_) => false,
},
plugins: BTreeSet::new(),
},
images: BTreeMap::new(),
volumes: value
.volumes
@@ -217,7 +228,6 @@ impl TryFrom<ManifestV1> for Manifest {
.map(|(id, _)| id.clone())
.chain([VolumeId::from_str("embassy").unwrap()])
.collect(),
alerts: value.alerts,
dependencies: Dependencies(
value
.dependencies
@@ -252,13 +262,6 @@ impl TryFrom<ManifestV1> for Manifest {
})
.collect(),
},
git_hash: value.git_hash,
os_version: value.eos_version,
sdk_version: None,
hardware_acceleration: match value.main {
PackageProcedure::Docker(d) => d.gpu_acceleration,
PackageProcedure::Script(_) => false,
},
})
}
}

63
core/src/s9pk/v2/manifest.rs
View File

@@ -7,12 +7,11 @@ use exver::{Version, VersionRange};
use imbl_value::{InOMap, InternedString};
use serde::{Deserialize, Serialize};
use ts_rs::TS;
use url::Url;
pub use crate::PackageId;
use crate::dependencies::Dependencies;
use crate::prelude::*;
use crate::s9pk::git_hash::GitHash;
use crate::registry::package::index::PackageMetadata;
use crate::s9pk::merkle_archive::directory_contents::DirectoryContents;
use crate::s9pk::merkle_archive::expected::{Expected, Filter};
use crate::s9pk::v2::pack::ImageConfig;
@@ -22,7 +21,7 @@ use crate::util::{FromStrParser, VersionString, mime};
use crate::version::{Current, VersionT};
use crate::{ImageId, VolumeId};
fn current_version() -> Version {
pub(crate) fn current_version() -> Version {
Current::default().semver()
}
@@ -32,46 +31,20 @@ fn current_version() -> Version {
#[ts(export)]
pub struct Manifest {
pub id: PackageId,
#[ts(type = "string")]
pub title: InternedString,
pub version: VersionString,
pub satisfies: BTreeSet<VersionString>,
pub release_notes: LocaleString,
#[ts(type = "string")]
pub can_migrate_to: VersionRange,
#[ts(type = "string")]
pub can_migrate_from: VersionRange,
#[ts(type = "string")]
pub license: InternedString, // type of license
#[ts(type = "string")]
pub wrapper_repo: Url,
#[ts(type = "string")]
pub upstream_repo: Url,
#[ts(type = "string")]
pub support_site: Url,
#[ts(type = "string")]
pub marketing_site: Url,
#[ts(type = "string | null")]
pub donation_url: Option<Url>,
#[ts(type = "string | null")]
pub docs_url: Option<Url>,
pub description: Description,
#[serde(flatten)]
pub metadata: PackageMetadata,
pub images: BTreeMap<ImageId, ImageConfig>,
pub volumes: BTreeSet<VolumeId>,
#[serde(default)]
pub alerts: Alerts,
#[serde(default)]
pub dependencies: Dependencies,
#[serde(default)]
pub hardware_requirements: HardwareRequirements,
#[serde(default)]
pub hardware_acceleration: bool,
pub git_hash: Option<GitHash>,
#[serde(default = "current_version")]
#[ts(type = "string")]
pub os_version: Version,
#[ts(type = "string | null")]
pub sdk_version: Option<Version>,
}
impl Manifest {
pub fn validate_for<'a, T: Clone>(
@@ -181,6 +154,32 @@ pub struct HardwareRequirements {
pub arch: Option<BTreeSet<InternedString>>,
}
impl HardwareRequirements {
/// Returns true if this s9pk's hardware requirements are satisfied by the given hardware.
pub fn is_compatible(&self, hw: &crate::registry::device_info::HardwareInfo) -> bool {
if let Some(arch) = &self.arch {
if !arch.contains(&hw.arch) {
return false;
}
}
if let Some(ram) = self.ram {
if hw.ram < ram {
return false;
}
}
if let Some(devices) = &hw.devices {
for device_filter in &self.device {
if !devices
.iter()
.filter(|d| d.class() == &*device_filter.class)
.any(|d| device_filter.matches(d))
{
return false;
}
}
}
true
}
/// returns a value that can be used as a sort key to get most specific requirements first
pub fn specificity_desc(&self) -> (u32, u32, u64) {
(
@@ -240,7 +239,7 @@ impl LocaleString {
pub fn localize(&mut self) {
self.localize_for(&*rust_i18n::locale());
}
pub fn localized(mut self) -> String {
pub fn localized(self) -> String {
self.localized_for(&*rust_i18n::locale())
}
}

4
core/src/s9pk/v2/pack.rs
View File

@@ -685,7 +685,7 @@ pub async fn pack(ctx: CliContext, params: PackParams) -> Result<(), Error> {
.await?;
let manifest = s9pk.as_manifest_mut();
manifest.git_hash = Some(GitHash::from_path(params.path()).await?);
manifest.metadata.git_hash = Some(GitHash::from_path(params.path()).await?);
if !params.arch.is_empty() {
let arches: BTreeSet<InternedString> = match manifest.hardware_requirements.arch.take() {
Some(a) => params
@@ -792,7 +792,7 @@ pub async fn pack(ctx: CliContext, params: PackParams) -> Result<(), Error> {
}
};
Some((
LocaleString::Translated(s9pk.as_manifest().title.to_string()),
LocaleString::Translated(s9pk.as_manifest().metadata.title.to_string()),
s9pk.icon_data_url().await?,
))
}

17
core/src/service/action.rs
View File

@@ -17,6 +17,7 @@ use crate::{ActionId, PackageId, ReplayId};
pub(super) struct GetActionInput {
id: ActionId,
prefill: Value,
}
impl Handler<GetActionInput> for ServiceActor {
type Response = Result<Option<ActionInput>, Error>;
@@ -26,7 +27,10 @@ impl Handler<GetActionInput> for ServiceActor {
async fn handle(
&mut self,
id: Guid,
GetActionInput { id: action_id }: GetActionInput,
GetActionInput {
id: action_id,
prefill,
}: GetActionInput,
_: &BackgroundJobQueue,
) -> Self::Response {
let container = &self.0.persistent_container;
@@ -34,7 +38,7 @@ impl Handler<GetActionInput> for ServiceActor {
.execute::<Option<ActionInput>>(
id,
ProcedureName::GetActionInput(action_id),
Value::Null,
json!({ "prefill": prefill }),
Some(Duration::from_secs(30)),
)
.await
@@ -47,6 +51,7 @@ impl Service {
&self,
id: Guid,
action_id: ActionId,
prefill: Value,
) -> Result<Option<ActionInput>, Error> {
if !self
.seed
@@ -67,7 +72,13 @@ impl Service {
return Ok(None);
}
self.actor
.send(id, GetActionInput { id: action_id })
.send(
id,
GetActionInput {
id: action_id,
prefill,
},
)
.await?
}
}

27
core/src/service/effects/action.rs
View File

@@ -122,6 +122,10 @@ pub struct GetActionInputParams {
package_id: Option<PackageId>,
#[arg(help = "help.arg.action-id")]
action_id: ActionId,
#[ts(type = "Record<string, unknown> | null")]
#[serde(default)]
#[arg(skip)]
prefill: Option<Value>,
}
async fn get_action_input(
context: EffectContext,
@@ -129,9 +133,11 @@ async fn get_action_input(
procedure_id,
package_id,
action_id,
prefill,
}: GetActionInputParams,
) -> Result<Option<ActionInput>, Error> {
let context = context.deref()?;
let prefill = prefill.unwrap_or(Value::Null);
if let Some(package_id) = package_id {
context
@@ -142,16 +148,18 @@ async fn get_action_input(
.await
.as_ref()
.or_not_found(&package_id)?
.get_action_input(procedure_id, action_id)
.get_action_input(procedure_id, action_id, prefill)
.await
} else {
context.get_action_input(procedure_id, action_id).await
context
.get_action_input(procedure_id, action_id, prefill)
.await
}
}
#[derive(Debug, Clone, Serialize, Deserialize, TS, Parser)]
#[serde(rename_all = "camelCase")]
#[ts(export)]
#[ts(export, rename = "EffectsRunActionParams")]
pub struct RunActionParams {
#[serde(default)]
#[ts(skip)]
@@ -243,11 +251,12 @@ async fn create_task(
.get(&task.package_id)
.await
.as_ref()
.filter(|s| s.is_initialized())
{
let Some(prev) = service
.get_action_input(procedure_id.clone(), task.action_id.clone())
.await?
else {
let prev = service
.get_action_input(procedure_id.clone(), task.action_id.clone(), Value::Null)
.await?;
let Some(prev) = prev else {
return Err(Error::new(
eyre!(
"{}",
@@ -270,7 +279,9 @@ async fn create_task(
true
}
} else {
true // update when service is installed
// Service not installed or not yet initialized — assume active.
// Will be retested when service init completes (Service::recheck_tasks).
true
}
}
},

123
core/src/service/effects/callbacks.rs
View File

@@ -5,12 +5,16 @@ use std::time::{Duration, SystemTime};
use clap::Parser;
use futures::future::join_all;
use imbl::{Vector, vector};
use imbl::{OrdMap, Vector, vector};
use imbl_value::InternedString;
use patch_db::TypedDbWatch;
use patch_db::json_ptr::JsonPointer;
use serde::{Deserialize, Serialize};
use tracing::warn;
use ts_rs::TS;
use crate::db::model::Database;
use crate::db::model::public::NetworkInterfaceInfo;
use crate::net::ssl::FullchainCertData;
use crate::prelude::*;
use crate::service::effects::context::EffectContext;
@@ -19,7 +23,7 @@ use crate::service::rpc::{CallbackHandle, CallbackId};
use crate::service::{Service, ServiceActorSeed};
use crate::util::collections::EqMap;
use crate::util::future::NonDetachingJoinHandle;
use crate::{HostId, PackageId, ServiceInterfaceId};
use crate::{GatewayId, HostId, PackageId, ServiceInterfaceId};
#[derive(Default)]
pub struct ServiceCallbacks(Mutex<ServiceCallbackMap>);
@@ -29,7 +33,8 @@ struct ServiceCallbackMap {
get_service_interface: BTreeMap<(PackageId, ServiceInterfaceId), Vec<CallbackHandler>>,
list_service_interfaces: BTreeMap<PackageId, Vec<CallbackHandler>>,
get_system_smtp: Vec<CallbackHandler>,
get_host_info: BTreeMap<(PackageId, HostId), Vec<CallbackHandler>>,
get_host_info:
BTreeMap<(PackageId, HostId), (NonDetachingJoinHandle<()>, Vec<CallbackHandler>)>,
get_ssl_certificate: EqMap<
(BTreeSet<InternedString>, FullchainCertData, Algorithm),
(NonDetachingJoinHandle<()>, Vec<CallbackHandler>),
@@ -37,6 +42,7 @@ struct ServiceCallbackMap {
get_status: BTreeMap<PackageId, Vec<CallbackHandler>>,
get_container_ip: BTreeMap<PackageId, Vec<CallbackHandler>>,
get_service_manifest: BTreeMap<PackageId, Vec<CallbackHandler>>,
get_outbound_gateway: BTreeMap<PackageId, (NonDetachingJoinHandle<()>, Vec<CallbackHandler>)>,
}
impl ServiceCallbacks {
@@ -57,7 +63,7 @@ impl ServiceCallbacks {
});
this.get_system_smtp
.retain(|h| h.handle.is_active() && h.seed.strong_count() > 0);
this.get_host_info.retain(|_, v| {
this.get_host_info.retain(|_, (_, v)| {
v.retain(|h| h.handle.is_active() && h.seed.strong_count() > 0);
!v.is_empty()
});
@@ -73,6 +79,10 @@ impl ServiceCallbacks {
v.retain(|h| h.handle.is_active() && h.seed.strong_count() > 0);
!v.is_empty()
});
this.get_outbound_gateway.retain(|_, (_, v)| {
v.retain(|h| h.handle.is_active() && h.seed.strong_count() > 0);
!v.is_empty()
});
})
}
@@ -141,29 +151,53 @@ impl ServiceCallbacks {
}
pub(super) fn add_get_host_info(
&self,
self: &Arc<Self>,
db: &TypedPatchDb<Database>,
package_id: PackageId,
host_id: HostId,
handler: CallbackHandler,
) {
self.mutate(|this| {
this.get_host_info
.entry((package_id, host_id))
.or_default()
.entry((package_id.clone(), host_id.clone()))
.or_insert_with(|| {
let ptr: JsonPointer =
format!("/public/packageData/{}/hosts/{}", package_id, host_id)
.parse()
.expect("valid json pointer");
let db = db.clone();
let callbacks = Arc::clone(self);
let key = (package_id, host_id);
(
tokio::spawn(async move {
let mut sub = db.subscribe(ptr).await;
while sub.recv().await.is_some() {
if let Some(cbs) = callbacks.mutate(|this| {
this.get_host_info
.remove(&key)
.map(|(_, handlers)| CallbackHandlers(handlers))
.filter(|cb| !cb.0.is_empty())
}) {
if let Err(e) = cbs.call(vector![]).await {
tracing::error!("Error in host info callback: {e}");
tracing::debug!("{e:?}");
}
}
// entry was removed when we consumed handlers,
// so stop watching — a new subscription will be
// created if the service re-registers
break;
}
})
.into(),
Vec::new(),
)
})
.1
.push(handler);
})
}
#[must_use]
pub fn get_host_info(&self, id: &(PackageId, HostId)) -> Option<CallbackHandlers> {
self.mutate(|this| {
Some(CallbackHandlers(
this.get_host_info.remove(id).unwrap_or_default(),
))
.filter(|cb| !cb.0.is_empty())
})
}
pub(super) fn add_get_ssl_certificate(
&self,
ctx: EffectContext,
@@ -256,6 +290,61 @@ impl ServiceCallbacks {
})
}
/// Register a callback for outbound gateway changes.
pub(super) fn add_get_outbound_gateway(
self: &Arc<Self>,
package_id: PackageId,
mut outbound_gateway: TypedDbWatch<Option<GatewayId>>,
mut default_outbound: Option<TypedDbWatch<Option<GatewayId>>>,
mut fallback: Option<TypedDbWatch<OrdMap<GatewayId, NetworkInterfaceInfo>>>,
handler: CallbackHandler,
) {
self.mutate(|this| {
this.get_outbound_gateway
.entry(package_id.clone())
.or_insert_with(|| {
let callbacks = Arc::clone(self);
let key = package_id;
(
tokio::spawn(async move {
tokio::select! {
_ = outbound_gateway.changed() => {}
_ = async {
if let Some(ref mut w) = default_outbound {
let _ = w.changed().await;
} else {
std::future::pending::<()>().await;
}
} => {}
_ = async {
if let Some(ref mut w) = fallback {
let _ = w.changed().await;
} else {
std::future::pending::<()>().await;
}
} => {}
}
if let Some(cbs) = callbacks.mutate(|this| {
this.get_outbound_gateway
.remove(&key)
.map(|(_, handlers)| CallbackHandlers(handlers))
.filter(|cb| !cb.0.is_empty())
}) {
if let Err(e) = cbs.call(vector![]).await {
tracing::error!("Error in outbound gateway callback: {e}");
tracing::debug!("{e:?}");
}
}
})
.into(),
Vec::new(),
)
})
.1
.push(handler);
})
}
pub(super) fn add_get_service_manifest(&self, package_id: PackageId, handler: CallbackHandler) {
self.mutate(|this| {
this.get_service_manifest

22
core/src/service/effects/mod.rs
View File

@@ -14,6 +14,7 @@ mod control;
mod dependency;
mod health;
mod net;
pub mod plugin;
mod prelude;
pub mod subcontainer;
mod system;
@@ -142,6 +143,10 @@ pub fn handler<C: Context>() -> ParentHandler<C> {
"get-container-ip",
from_fn_async(net::info::get_container_ip).no_cli(),
)
.subcommand(
"get-outbound-gateway",
from_fn_async(net::info::get_outbound_gateway).no_cli(),
)
.subcommand(
"get-os-ip",
from_fn(|_: C| Ok::<_, Error>(Ipv4Addr::from(HOST_IP))),
@@ -167,6 +172,23 @@ pub fn handler<C: Context>() -> ParentHandler<C> {
from_fn_async(net::ssl::get_ssl_certificate).no_cli(),
)
.subcommand("get-ssl-key", from_fn_async(net::ssl::get_ssl_key).no_cli())
// plugin
.subcommand(
"plugin",
ParentHandler::<C>::new().subcommand(
"url",
ParentHandler::<C>::new()
.subcommand("register", from_fn_async(net::plugin::register).no_cli())
.subcommand(
"export-url",
from_fn_async(net::plugin::export_url).no_cli(),
)
.subcommand(
"clear-urls",
from_fn_async(net::plugin::clear_urls).no_cli(),
),
),
)
.subcommand(
"set-data-version",
from_fn_async(version::set_data_version)

1
core/src/service/effects/net/host.rs
View File

@@ -29,6 +29,7 @@ pub async fn get_host_info(
if let Some(callback) = callback {
let callback = callback.register(&context.seed.persistent_container);
context.seed.ctx.callbacks.add_get_host_info(
&context.seed.ctx.db,
package_id.clone(),
host_id.clone(),
CallbackHandler::new(&context, callback),

122
core/src/service/effects/net/info.rs
View File

@@ -1,9 +1,16 @@
use std::net::Ipv4Addr;
use crate::PackageId;
use imbl::OrdMap;
use patch_db::TypedDbWatch;
use patch_db::json_ptr::JsonPointer;
use tokio::process::Command;
use crate::db::model::public::NetworkInterfaceInfo;
use crate::service::effects::callbacks::CallbackHandler;
use crate::service::effects::prelude::*;
use crate::service::rpc::CallbackId;
use crate::util::Invoke;
use crate::{GatewayId, PackageId};
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize, TS)]
#[serde(rename_all = "camelCase")]
@@ -51,3 +58,116 @@ pub async fn get_container_ip(
lxc.ip().await.map(Some)
}
}
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize, TS)]
#[serde(rename_all = "camelCase")]
#[ts(export)]
pub struct GetOutboundGatewayParams {
#[ts(optional)]
callback: Option<CallbackId>,
}
pub async fn get_outbound_gateway(
context: EffectContext,
GetOutboundGatewayParams { callback }: GetOutboundGatewayParams,
) -> Result<GatewayId, Error> {
let context = context.deref()?;
let ctx = &context.seed.ctx;
// Resolve the effective gateway; DB watches are created atomically
// with each read to avoid race conditions.
let (gw, pkg_watch, os_watch, gateways_watch) =
resolve_outbound_gateway(ctx, &context.seed.id).await?;
if let Some(callback) = callback {
let callback = callback.register(&context.seed.persistent_container);
context.seed.ctx.callbacks.add_get_outbound_gateway(
context.seed.id.clone(),
pkg_watch,
os_watch,
gateways_watch,
CallbackHandler::new(&context, callback),
);
}
Ok(gw)
}
async fn resolve_outbound_gateway(
ctx: &crate::context::RpcContext,
package_id: &PackageId,
) -> Result<
(
GatewayId,
TypedDbWatch<Option<GatewayId>>,
Option<TypedDbWatch<Option<GatewayId>>>,
Option<TypedDbWatch<OrdMap<GatewayId, NetworkInterfaceInfo>>>,
),
Error,
> {
// 1. Package-specific outbound gateway — subscribe before reading
let pkg_ptr: JsonPointer = format!("/public/packageData/{}/outboundGateway", package_id)
.parse()
.expect("valid json pointer");
let mut pkg_watch = ctx.db.watch(pkg_ptr).await;
let pkg_gw: Option<GatewayId> = imbl_value::from_value(pkg_watch.peek_and_mark_seen()?)?;
if let Some(gw) = pkg_gw {
return Ok((gw, pkg_watch.typed(), None, None));
}
// 2. OS-level default outbound — subscribe before reading
let os_ptr: JsonPointer = "/public/serverInfo/network/defaultOutbound"
.parse()
.expect("valid json pointer");
let mut os_watch = ctx.db.watch(os_ptr).await;
let default_outbound: Option<GatewayId> =
imbl_value::from_value(os_watch.peek_and_mark_seen()?)?;
if let Some(gw) = default_outbound {
return Ok((gw, pkg_watch.typed(), Some(os_watch.typed()), None));
}
// 3. Fall through to main routing table — watch gateways for changes
let gw_ptr: JsonPointer = "/public/serverInfo/network/gateways"
.parse()
.expect("valid json pointer");
let mut gateways_watch = ctx.db.watch(gw_ptr).await;
gateways_watch.peek_and_mark_seen()?;
let gw = default_route_interface().await?;
Ok((
gw,
pkg_watch.typed(),
Some(os_watch.typed()),
Some(gateways_watch.typed()),
))
}
/// Parses `ip route show table main` for the default route's `dev` field.
async fn default_route_interface() -> Result<GatewayId, Error> {
let output = Command::new("ip")
.arg("route")
.arg("show")
.arg("table")
.arg("main")
.invoke(ErrorKind::Network)
.await?;
let text = String::from_utf8_lossy(&output);
for line in text.lines() {
if line.starts_with("default ") {
let mut parts = line.split_whitespace();
while let Some(tok) = parts.next() {
if tok == "dev" {
if let Some(dev) = parts.next() {
return Ok(dev.parse().unwrap());
}
}
}
}
}
Err(Error::new(
eyre!("no default route found in main routing table"),
ErrorKind::Network,
))
}

1
core/src/service/effects/net/mod.rs
View File

@@ -2,4 +2,5 @@ pub mod bind;
pub mod host;
pub mod info;
pub mod interface;
pub mod plugin;
pub mod ssl;

176
core/src/service/effects/net/plugin.rs Normal file
View File

@@ -0,0 +1,176 @@
use std::collections::BTreeSet;
use std::sync::Arc;
use crate::ActionId;
use crate::net::host::{all_hosts, host_for};
use crate::net::service_interface::{HostnameMetadata, PluginHostnameInfo};
use crate::service::Service;
use crate::service::effects::plugin::PluginId;
use crate::service::effects::prelude::*;
fn require_url_plugin(context: &Arc<Service>) -> Result<(), Error> {
if !context
.seed
.persistent_container
.s9pk
.as_manifest()
.metadata
.plugins
.contains(&PluginId::UrlV0)
{
return Err(Error::new(
eyre!(
"{}",
t!("net.plugin.manifest-missing-plugin", plugin = "url-v0")
),
ErrorKind::InvalidRequest,
));
}
Ok(())
}
#[derive(Debug, Clone, Serialize, Deserialize, TS)]
#[serde(rename_all = "camelCase")]
#[ts(export)]
pub struct UrlPluginRegisterParams {
pub table_action: ActionId,
}
pub async fn register(
context: EffectContext,
UrlPluginRegisterParams { table_action }: UrlPluginRegisterParams,
) -> Result<(), Error> {
use crate::db::model::package::UrlPluginRegistration;
let context = context.deref()?;
require_url_plugin(&context)?;
let plugin_id = context.seed.id.clone();
context
.seed
.ctx
.db
.mutate(|db| {
db.as_public_mut()
.as_package_data_mut()
.as_idx_mut(&plugin_id)
.or_not_found(&plugin_id)?
.as_plugin_mut()
.as_url_mut()
.ser(&Some(UrlPluginRegistration { table_action }))?;
Ok(())
})
.await
.result?;
Ok(())
}
#[derive(Debug, Clone, Serialize, Deserialize, TS)]
#[serde(rename_all = "camelCase")]
#[ts(export)]
pub struct UrlPluginExportUrlParams {
pub hostname_info: PluginHostnameInfo,
pub remove_action: Option<ActionId>,
pub overflow_actions: Vec<ActionId>,
}
pub async fn export_url(
context: EffectContext,
UrlPluginExportUrlParams {
hostname_info,
remove_action,
overflow_actions,
}: UrlPluginExportUrlParams,
) -> Result<(), Error> {
let context = context.deref()?;
require_url_plugin(&context)?;
let plugin_id = context.seed.id.clone();
let entry = hostname_info.to_hostname_info(&plugin_id, remove_action, overflow_actions);
context
.seed
.ctx
.db
.mutate(|db| {
let host = host_for(
db,
hostname_info.package_id.as_ref(),
&hostname_info.host_id,
)?;
host.as_bindings_mut()
.as_idx_mut(&hostname_info.internal_port)
.or_not_found(t!(
"net.plugin.binding-not-found",
binding = format!(
"{}:{}:{}",
hostname_info.package_id.as_deref().unwrap_or("STARTOS"),
hostname_info.host_id,
hostname_info.internal_port
)
))?
.as_addresses_mut()
.as_available_mut()
.mutate(|available: &mut BTreeSet<_>| {
available.insert(entry);
Ok(())
})?;
Ok(())
})
.await
.result?;
Ok(())
}
#[derive(Debug, Clone, Serialize, Deserialize, TS)]
#[serde(rename_all = "camelCase")]
#[ts(export)]
pub struct UrlPluginClearUrlsParams {
pub except: BTreeSet<PluginHostnameInfo>,
}
pub async fn clear_urls(
context: EffectContext,
UrlPluginClearUrlsParams { except }: UrlPluginClearUrlsParams,
) -> Result<(), Error> {
let context = context.deref()?;
require_url_plugin(&context)?;
let plugin_id = context.seed.id.clone();
context
.seed
.ctx
.db
.mutate(|db| {
for host in all_hosts(db) {
let host = host?;
for (_, bind) in host.as_bindings_mut().as_entries_mut()? {
bind.as_addresses_mut().as_available_mut().mutate(
|available: &mut BTreeSet<_>| {
available.retain(|h| {
match &h.metadata {
HostnameMetadata::Plugin { package_id, .. }
if package_id == &plugin_id =>
{
// Keep if it matches any entry in the except list
except
.iter()
.any(|e| e.matches_hostname_info(h, &plugin_id))
}
_ => true,
}
});
Ok(())
},
)?;
}
}
Ok(())
})
.await
.result?;
Ok(())
}

32
core/src/service/effects/net/ssl.rs
View File

@@ -55,20 +55,18 @@ pub async fn get_ssl_certificate(
.map(|(_, m)| m.as_hosts().as_entries())
.flatten_ok()
.map_ok(|(_, m)| {
Ok(m.as_onions()
.de()?
.iter()
.map(InternedString::from_display)
.chain(m.as_public_domains().keys()?)
.chain(m.as_private_domains().de()?)
Ok(m.as_public_domains()
.keys()?
.into_iter()
.chain(m.as_private_domains().keys()?)
.chain(
m.as_hostname_info()
m.as_bindings()
.de()?
.values()
.flatten()
.flat_map(|b| b.addresses.available.iter().cloned())
.map(|h| h.to_san_hostname()),
)
.collect::<Vec<_>>())
.collect::<Vec<InternedString>>())
})
.map(|a| a.and_then(|a| a))
.flatten_ok()
@@ -181,20 +179,18 @@ pub async fn get_ssl_key(
.map(|m| m.as_hosts().as_entries())
.flatten_ok()
.map_ok(|(_, m)| {
Ok(m.as_onions()
.de()?
.iter()
.map(InternedString::from_display)
.chain(m.as_public_domains().keys()?)
.chain(m.as_private_domains().de()?)
Ok(m.as_public_domains()
.keys()?
.into_iter()
.chain(m.as_private_domains().keys()?)
.chain(
m.as_hostname_info()
m.as_bindings()
.de()?
.values()
.flatten()
.flat_map(|b| b.addresses.available.iter().cloned())
.map(|h| h.to_san_hostname()),
)
.collect::<Vec<_>>())
.collect::<Vec<InternedString>>())
})
.map(|a| a.and_then(|a| a))
.flatten_ok()

9
core/src/service/effects/plugin.rs Normal file
View File

@@ -0,0 +1,9 @@
use serde::{Deserialize, Serialize};
use ts_rs::TS;
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize, TS)]
#[serde(rename_all = "kebab-case")]
#[ts(export)]
pub enum PluginId {
UrlV0,
}

177
core/src/service/mod.rs
View File

@@ -16,7 +16,7 @@ use futures::{FutureExt, SinkExt, StreamExt, TryStreamExt};
use imbl_value::{InternedString, json};
use itertools::Itertools;
use nix::sys::signal::Signal;
use persistent_container::{PersistentContainer, Subcontainer};
use persistent_container::PersistentContainer;
use rpc_toolkit::HandlerArgs;
use rpc_toolkit::yajrc::RpcError;
use serde::{Deserialize, Serialize};
@@ -52,7 +52,7 @@ use crate::util::serde::Pem;
use crate::util::sync::SyncMutex;
use crate::util::tui::choose;
use crate::volume::data_dir;
use crate::{ActionId, CAP_1_KiB, DATA_DIR, HostId, ImageId, PackageId};
use crate::{ActionId, CAP_1_KiB, DATA_DIR, ImageId, PackageId};
pub mod action;
pub mod cli;
@@ -215,6 +215,84 @@ pub struct Service {
seed: Arc<ServiceActorSeed>,
}
impl Service {
pub fn is_initialized(&self) -> bool {
self.seed.persistent_container.state.borrow().rt_initialized
}
/// Re-evaluate all tasks that reference this service's actions.
/// Called after every service init to update task active state.
#[instrument(skip_all)]
async fn recheck_tasks(&self) -> Result<(), Error> {
let service_id = &self.seed.id;
let peek = self.seed.ctx.db.peek().await;
let mut action_input: BTreeMap<ActionId, Value> = BTreeMap::new();
let tasks: BTreeSet<_> = peek
.as_public()
.as_package_data()
.as_entries()?
.into_iter()
.map(|(_, pde)| {
Ok(pde
.as_tasks()
.as_entries()?
.into_iter()
.map(|(_, r)| {
let t = r.as_task();
Ok::<_, Error>(
if t.as_package_id().de()? == *service_id
&& t.as_input().transpose_ref().is_some()
{
Some(t.as_action_id().de()?)
} else {
None
},
)
})
.filter_map_ok(|a| a))
})
.flatten_ok()
.map(|a| a.and_then(|a| a))
.try_collect()?;
let procedure_id = Guid::new();
for action_id in tasks {
if let Some(input) = self
.get_action_input(procedure_id.clone(), action_id.clone(), Value::Null)
.await
.log_err()
.flatten()
.and_then(|i| i.value)
{
action_input.insert(action_id, input);
}
}
self.seed
.ctx
.db
.mutate(|db| {
for (action_id, input) in &action_input {
for (_, pde) in db.as_public_mut().as_package_data_mut().as_entries_mut()? {
pde.as_tasks_mut().mutate(|tasks| {
Ok(update_tasks(tasks, service_id, action_id, input, false))
})?;
}
}
for (_, pde) in db.as_public_mut().as_package_data_mut().as_entries_mut()? {
if pde
.as_tasks()
.de()?
.into_iter()
.any(|(_, t)| t.active && t.task.severity == TaskSeverity::Critical)
{
pde.as_status_info_mut().stop()?;
}
}
Ok(())
})
.await
.result?;
Ok(())
}
#[instrument(skip_all)]
async fn new(
ctx: RpcContext,
@@ -263,6 +341,7 @@ impl Service {
.persistent_container
.init(service.weak(), procedure_id, init_kind)
.await?;
service.recheck_tasks().await?;
if let Some(recovery_guard) = recovery_guard {
recovery_guard.unmount(true).await?;
}
@@ -489,70 +568,8 @@ impl Service {
)
.await?;
if let Some(mut progress) = progress {
progress.finalization_progress.complete();
progress.progress.complete();
tokio::task::yield_now().await;
}
let peek = ctx.db.peek().await;
let mut action_input: BTreeMap<ActionId, Value> = BTreeMap::new();
let tasks: BTreeSet<_> = peek
.as_public()
.as_package_data()
.as_entries()?
.into_iter()
.map(|(_, pde)| {
Ok(pde
.as_tasks()
.as_entries()?
.into_iter()
.map(|(_, r)| {
let t = r.as_task();
Ok::<_, Error>(
if t.as_package_id().de()? == manifest.id
&& t.as_input().transpose_ref().is_some()
{
Some(t.as_action_id().de()?)
} else {
None
},
)
})
.filter_map_ok(|a| a))
})
.flatten_ok()
.map(|a| a.and_then(|a| a))
.try_collect()?;
for action_id in tasks {
if peek
.as_public()
.as_package_data()
.as_idx(&manifest.id)
.or_not_found(&manifest.id)?
.as_actions()
.contains_key(&action_id)?
{
if let Some(input) = service
.get_action_input(procedure_id.clone(), action_id.clone())
.await
.log_err()
.flatten()
.and_then(|i| i.value)
{
action_input.insert(action_id, input);
}
}
}
ctx.db
.mutate(|db| {
for (action_id, input) in &action_input {
for (_, pde) in db.as_public_mut().as_package_data_mut().as_entries_mut()? {
pde.as_tasks_mut().mutate(|tasks| {
Ok(update_tasks(tasks, &manifest.id, action_id, input, false))
})?;
}
}
let entry = db
.as_public_mut()
.as_package_data_mut()
@@ -587,12 +604,19 @@ impl Service {
entry.as_developer_key_mut().ser(&Pem::new(developer_key))?;
entry.as_icon_mut().ser(&icon)?;
entry.as_registry_mut().ser(registry)?;
entry.as_status_info_mut().as_error_mut().ser(&None)?;
Ok(())
})
.await
.result?;
if let Some(mut progress) = progress {
progress.finalization_progress.complete();
progress.progress.complete();
tokio::task::yield_now().await;
}
// Trigger manifest callbacks after successful installation
let manifest = service.seed.persistent_container.s9pk.as_manifest();
if let Some(callbacks) = ctx.callbacks.get_service_manifest(&manifest.id) {
@@ -682,14 +706,6 @@ impl Service {
memory_usage: MiB::from_MiB(used),
})
}
pub async fn sync_host(&self, host_id: HostId) -> Result<(), Error> {
self.seed
.persistent_container
.net_service
.sync_host(host_id)
.await
}
}
struct ServiceActorSeed {
@@ -701,6 +717,7 @@ struct ServiceActorSeed {
}
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
pub struct RebuildParams {
#[arg(help = "help.arg.package-id")]
pub id: PackageId,
@@ -1194,6 +1211,9 @@ pub async fn cli_attach(
{
Ok(a) => a,
Err(e) => {
if e.kind != ErrorKind::InvalidRequest {
return Err(e);
}
let prompt = e.to_string();
let options: Vec<SubcontainerInfo> = from_value(e.info)?;
let choice = choose(&prompt, &options).await?;
@@ -1206,6 +1226,7 @@ pub async fn cli_attach(
)?;
let mut ws = context.ws_continuation(guid).await?;
print!("\r");
let (kill, thread_kill) = tokio::sync::oneshot::channel();
let (thread_send, recv) = tokio::sync::mpsc::channel(4 * CAP_1_KiB);
let stdin_thread: NonDetachingJoinHandle<()> = tokio::task::spawn_blocking(move || {
@@ -1234,18 +1255,6 @@ pub async fn cli_attach(
let mut stderr = Some(stderr);
loop {
futures::select_biased! {
// signal = tokio:: => {
// let exit = exit?;
// if current_out != "exit" {
// ws.send(Message::Text("exit".into()))
// .await
// .with_kind(ErrorKind::Network)?;
// current_out = "exit";
// }
// ws.send(Message::Binary(
// i32::to_be_bytes(exit.into_raw()).to_vec()
// )).await.with_kind(ErrorKind::Network)?;
// }
input = stdin.as_mut().map_or(
futures::future::Either::Left(futures::future::pending()),
|s| futures::future::Either::Right(s.recv())

2
core/src/service/persistent_container.rs
View File

@@ -97,7 +97,7 @@ impl PersistentContainer {
.join(&s9pk.as_manifest().id),
),
LxcConfig {
hardware_acceleration: s9pk.manifest.hardware_acceleration,
hardware_acceleration: s9pk.manifest.metadata.hardware_acceleration,
},
)
.await?;

2
core/src/service/service_map.rs
View File

@@ -259,6 +259,8 @@ impl ServiceMap {
service_interfaces: Default::default(),
hosts: Default::default(),
store_exposed_dependents: Default::default(),
outbound_gateway: None,
plugin: Default::default(),
},
)?;
};

21
core/src/service/uninstall.rs
View File

@@ -1,9 +1,12 @@
use std::collections::BTreeSet;
use std::path::Path;
use imbl::vector;
use crate::context::RpcContext;
use crate::db::model::package::{InstalledState, InstallingInfo, InstallingState, PackageState};
use crate::net::host::all_hosts;
use crate::net::service_interface::{HostnameInfo, HostnameMetadata};
use crate::prelude::*;
use crate::volume::PKG_VOLUME_DIR;
use crate::{DATA_DIR, PACKAGE_DATA, PackageId};
@@ -36,6 +39,24 @@ pub async fn cleanup(ctx: &RpcContext, id: &PackageId, soft: bool) -> Result<(),
Ok(())
})?;
d.as_private_mut().as_package_stores_mut().remove(&id)?;
// Remove plugin URLs exported by this package from all hosts
for host in all_hosts(d) {
let host = host?;
for (_, bind) in host.as_bindings_mut().as_entries_mut()? {
bind.as_addresses_mut().as_available_mut().mutate(
|available: &mut BTreeSet<HostnameInfo>| {
available.retain(|h| {
!matches!(
&h.metadata,
HostnameMetadata::Plugin { package_id, .. }
if package_id == id
)
});
Ok(())
},
)?;
}
}
Ok(Some(pde))
} else {
Ok(None)

80
core/src/setup.rs
View File

@@ -31,6 +31,7 @@ use crate::disk::mount::filesystem::ReadWrite;
use crate::disk::mount::filesystem::cifs::Cifs;
use crate::disk::mount::guard::{GenericMountGuard, TmpMountGuard};
use crate::disk::util::{DiskInfo, StartOsRecoveryInfo, pvscan, recovery_info};
use crate::hostname::ServerHostnameInfo;
use crate::init::{InitPhases, InitResult, init};
use crate::net::ssl::root_ca_start_time;
use crate::prelude::*;
@@ -115,6 +116,7 @@ async fn setup_init(
ctx: &SetupContext,
password: Option<String>,
kiosk: Option<bool>,
hostname: Option<ServerHostnameInfo>,
init_phases: InitPhases,
) -> Result<(AccountInfo, InitResult), Error> {
let init_result = init(&ctx.webserver, &ctx.config.peek(|c| c.clone()), init_phases).await?;
@@ -129,6 +131,9 @@ async fn setup_init(
if let Some(password) = &password {
account.set_password(password)?;
}
if let Some(hostname) = hostname {
account.hostname = hostname;
}
account.save(m)?;
let info = m.as_public_mut().as_server_info_mut();
info.as_password_hash_mut().ser(&account.password)?;
@@ -233,7 +238,8 @@ pub async fn attach(
}
disk_phase.complete();
let (account, net_ctrl) = setup_init(&setup_ctx, password, kiosk, init_phases).await?;
let (account, net_ctrl) =
setup_init(&setup_ctx, password, kiosk, None, init_phases).await?;
let rpc_ctx = RpcContext::init(
&setup_ctx.webserver,
@@ -246,7 +252,7 @@ pub async fn attach(
Ok((
SetupResult {
hostname: account.hostname,
hostname: account.hostname.hostname,
root_ca: Pem(account.root_ca_cert),
needs_restart: setup_ctx.install_rootfs.peek(|a| a.is_some()),
},
@@ -402,10 +408,12 @@ pub async fn setup_data_drive(
#[ts(export)]
pub struct SetupExecuteParams {
guid: InternedString,
password: EncryptedWire,
password: Option<EncryptedWire>,
recovery_source: Option<RecoverySource<EncryptedWire>>,
#[ts(optional)]
kiosk: Option<bool>,
name: Option<InternedString>,
hostname: Option<InternedString>,
}
// #[command(rpc_only)]
@@ -416,17 +424,20 @@ pub async fn execute(
password,
recovery_source,
kiosk,
name,
hostname,
}: SetupExecuteParams,
) -> Result<SetupProgress, Error> {
let password = match password.decrypt(&ctx) {
Some(a) => a,
None => {
return Err(Error::new(
color_eyre::eyre::eyre!("{}", t!("setup.couldnt-decode-startos-password")),
crate::ErrorKind::Unknown,
));
}
};
let password = password
.map(|p| {
p.decrypt(&ctx).ok_or_else(|| {
Error::new(
color_eyre::eyre::eyre!("{}", t!("setup.couldnt-decode-startos-password")),
crate::ErrorKind::Unknown,
)
})
})
.transpose()?;
let recovery = match recovery_source {
Some(RecoverySource::Backup {
target,
@@ -446,8 +457,10 @@ pub async fn execute(
None => None,
};
let hostname = ServerHostnameInfo::new_opt(name, hostname)?;
let setup_ctx = ctx.clone();
ctx.run_setup(move || execute_inner(setup_ctx, guid, password, recovery, kiosk))?;
ctx.run_setup(move || execute_inner(setup_ctx, guid, password, recovery, kiosk, hostname))?;
Ok(ctx.progress().await)
}
@@ -462,7 +475,7 @@ pub async fn complete(ctx: SetupContext) -> Result<SetupResult, Error> {
guid_file.sync_all().await?;
Command::new("systemd-firstboot")
.arg("--root=/media/startos/config/overlay/")
.arg(format!("--hostname={}", res.hostname.0))
.arg(format!("--hostname={}", res.hostname.as_ref()))
.invoke(ErrorKind::ParseSysInfo)
.await?;
Command::new("sync").invoke(ErrorKind::Filesystem).await?;
@@ -533,9 +546,10 @@ pub async fn shutdown(ctx: SetupContext) -> Result<(), Error> {
pub async fn execute_inner(
ctx: SetupContext,
guid: InternedString,
password: String,
password: Option<String>,
recovery_source: Option<RecoverySource<String>>,
kiosk: Option<bool>,
hostname: Option<ServerHostnameInfo>,
) -> Result<(SetupResult, RpcContext), Error> {
let progress = &ctx.progress;
let restore_phase = match recovery_source.as_ref() {
@@ -570,14 +584,30 @@ pub async fn execute_inner(
server_id,
recovery_password,
kiosk,
hostname,
progress,
)
.await
}
Some(RecoverySource::Migrate { guid: old_guid }) => {
migrate(&ctx, guid, &old_guid, password, kiosk, progress).await
migrate(&ctx, guid, &old_guid, password, kiosk, hostname, progress).await
}
None => {
fresh_setup(
&ctx,
guid,
&password.ok_or_else(|| {
Error::new(
eyre!("{}", t!("setup.password-required")),
ErrorKind::InvalidRequest,
)
})?,
kiosk,
hostname,
progress,
)
.await
}
None => fresh_setup(&ctx, guid, &password, kiosk, progress).await,
}
}
@@ -592,13 +622,14 @@ async fn fresh_setup(
guid: InternedString,
password: &str,
kiosk: Option<bool>,
hostname: Option<ServerHostnameInfo>,
SetupExecuteProgress {
init_phases,
rpc_ctx_phases,
..
}: SetupExecuteProgress,
) -> Result<(SetupResult, RpcContext), Error> {
let account = AccountInfo::new(password, root_ca_start_time().await)?;
let account = AccountInfo::new(password, root_ca_start_time().await, hostname)?;
let db = ctx.db().await?;
let kiosk = Some(kiosk.unwrap_or(true)).filter(|_| &*PLATFORM != "raspberrypi");
sync_kiosk(kiosk).await?;
@@ -635,7 +666,7 @@ async fn fresh_setup(
Ok((
SetupResult {
hostname: account.hostname,
hostname: account.hostname.hostname,
root_ca: Pem(account.root_ca_cert),
needs_restart: ctx.install_rootfs.peek(|a| a.is_some()),
},
@@ -647,11 +678,12 @@ async fn fresh_setup(
async fn recover(
ctx: &SetupContext,
guid: InternedString,
password: String,
password: Option<String>,
recovery_source: BackupTargetFS,
server_id: String,
recovery_password: String,
kiosk: Option<bool>,
hostname: Option<ServerHostnameInfo>,
progress: SetupExecuteProgress,
) -> Result<(SetupResult, RpcContext), Error> {
let recovery_source = TmpMountGuard::mount(&recovery_source, ReadWrite).await?;
@@ -663,6 +695,7 @@ async fn recover(
&server_id,
&recovery_password,
kiosk,
hostname,
progress,
)
.await
@@ -673,8 +706,9 @@ async fn migrate(
ctx: &SetupContext,
guid: InternedString,
old_guid: &str,
password: String,
password: Option<String>,
kiosk: Option<bool>,
hostname: Option<ServerHostnameInfo>,
SetupExecuteProgress {
init_phases,
restore_phase,
@@ -753,7 +787,7 @@ async fn migrate(
crate::disk::main::export(&old_guid, "/media/startos/migrate").await?;
restore_phase.complete();
let (account, net_ctrl) = setup_init(&ctx, Some(password), kiosk, init_phases).await?;
let (account, net_ctrl) = setup_init(&ctx, password, kiosk, hostname, init_phases).await?;
let rpc_ctx = RpcContext::init(
&ctx.webserver,
@@ -766,7 +800,7 @@ async fn migrate(
Ok((
SetupResult {
hostname: account.hostname,
hostname: account.hostname.hostname,
root_ca: Pem(account.root_ca_cert),
needs_restart: ctx.install_rootfs.peek(|a| a.is_some()),
},

24
core/src/ssh.rs
View File

@@ -12,7 +12,7 @@ use tracing::instrument;
use ts_rs::TS;
use crate::context::{CliContext, RpcContext};
use crate::hostname::Hostname;
use crate::hostname::ServerHostname;
use crate::prelude::*;
use crate::util::io::create_file;
use crate::util::serde::{HandlerExtSerde, Pem, WithIoFormat, display_serializable};
@@ -58,7 +58,8 @@ impl ValueParserFactory for SshPubKey {
}
}
#[derive(serde::Serialize, serde::Deserialize)]
#[derive(serde::Serialize, serde::Deserialize, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub struct SshKeyResponse {
pub alg: String,
@@ -115,15 +116,19 @@ pub fn ssh<C: Context>() -> ParentHandler<C> {
}
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
pub struct AddParams {
pub struct SshAddParams {
#[arg(help = "help.arg.ssh-public-key")]
key: SshPubKey,
}
#[instrument(skip_all)]
pub async fn add(ctx: RpcContext, AddParams { key }: AddParams) -> Result<SshKeyResponse, Error> {
pub async fn add(
ctx: RpcContext,
SshAddParams { key }: SshAddParams,
) -> Result<SshKeyResponse, Error> {
let mut key = WithTimeData::new(key);
let fingerprint = InternedString::intern(key.0.fingerprint_md5());
let (keys, res) = ctx
@@ -150,9 +155,10 @@ pub async fn add(ctx: RpcContext, AddParams { key }: AddParams) -> Result<SshKey
}
#[derive(Deserialize, Serialize, Parser, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
#[command(rename_all = "kebab-case")]
pub struct DeleteParams {
pub struct SshDeleteParams {
#[arg(help = "help.arg.ssh-fingerprint")]
#[ts(type = "string")]
fingerprint: InternedString,
@@ -161,7 +167,7 @@ pub struct DeleteParams {
#[instrument(skip_all)]
pub async fn remove(
ctx: RpcContext,
DeleteParams { fingerprint }: DeleteParams,
SshDeleteParams { fingerprint }: SshDeleteParams,
) -> Result<(), Error> {
let keys = ctx
.db
@@ -235,7 +241,7 @@ pub async fn list(ctx: RpcContext) -> Result<Vec<SshKeyResponse>, Error> {
#[instrument(skip_all)]
pub async fn sync_keys<P: AsRef<Path>>(
hostname: &Hostname,
hostname: &ServerHostname,
privkey: &Pem<ssh_key::PrivateKey>,
pubkeys: &SshKeys,
ssh_dir: P,
@@ -281,8 +287,8 @@ pub async fn sync_keys<P: AsRef<Path>>(
.to_openssh()
.with_kind(ErrorKind::OpenSsh)?
+ " start9@"
+ &*hostname.0)
.as_bytes(),
+ hostname.as_ref())
.as_bytes(),
)
.await?;
f.write_all(b"\n").await?;

177
core/src/system/mod.rs
View File

@@ -20,6 +20,7 @@ use crate::context::{CliContext, RpcContext};
use crate::disk::util::{get_available, get_used};
use crate::logs::{LogSource, LogsParams, SYSTEM_UNIT};
use crate::prelude::*;
use crate::registry::device_info::DeviceInfo;
use crate::rpc_continuations::{Guid, RpcContinuation, RpcContinuations};
use crate::shutdown::Shutdown;
use crate::util::Invoke;
@@ -191,7 +192,9 @@ pub async fn governor(
Ok(GovernorInfo { current, available })
}
#[derive(Serialize, Deserialize)]
#[derive(Serialize, Deserialize, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub struct TimeInfo {
now: String,
uptime: u64,
@@ -247,6 +250,64 @@ pub async fn time(ctx: RpcContext, _: Empty) -> Result<TimeInfo, Error> {
})
}
pub async fn device_info(ctx: RpcContext) -> Result<DeviceInfo, Error> {
DeviceInfo::load(&ctx).await
}
pub fn display_device_info(params: WithIoFormat<Empty>, info: DeviceInfo) -> Result<(), Error> {
use prettytable::*;
if let Some(format) = params.format {
return display_serializable(format, info);
}
let mut table = Table::new();
table.add_row(row![br -> "PLATFORM", &*info.os.platform]);
table.add_row(row![br -> "OS VERSION", info.os.version.to_string()]);
table.add_row(row![br -> "OS COMPAT", info.os.compat.to_string()]);
if let Some(lang) = &info.os.language {
table.add_row(row![br -> "LANGUAGE", &**lang]);
}
if let Some(hw) = &info.hardware {
table.add_row(row![br -> "ARCH", &*hw.arch]);
table.add_row(row![br -> "RAM", format_ram(hw.ram)]);
if let Some(devices) = &hw.devices {
for dev in devices {
let (class, desc) = match dev {
crate::util::lshw::LshwDevice::Processor(p) => (
"PROCESSOR",
p.product.as_deref().unwrap_or("unknown").to_string(),
),
crate::util::lshw::LshwDevice::Display(d) => (
"DISPLAY",
format!(
"{}{}",
d.product.as_deref().unwrap_or("unknown"),
d.driver
.as_deref()
.map(|drv| format!(" ({})", drv))
.unwrap_or_default()
),
),
};
table.add_row(row![br -> class, desc]);
}
}
}
table.print_tty(false)?;
Ok(())
}
fn format_ram(bytes: u64) -> String {
const GIB: u64 = 1024 * 1024 * 1024;
const MIB: u64 = 1024 * 1024;
if bytes >= GIB {
format!("{:.1} GiB", bytes as f64 / GIB as f64)
} else {
format!("{:.1} MiB", bytes as f64 / MIB as f64)
}
}
pub fn logs<C: Context + AsRef<RpcContinuations>>() -> ParentHandler<C, LogsParams> {
crate::logs::logs(|_: &C, _| async { Ok(LogSource::Unit(SYSTEM_UNIT)) })
}
@@ -331,6 +392,7 @@ pub struct MetricLeaf<T> {
}
#[derive(Clone, Copy, Debug, PartialEq, PartialOrd, TS)]
#[ts(type = "{ value: string, unit: string }")]
pub struct Celsius(f64);
impl fmt::Display for Celsius {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
@@ -359,6 +421,7 @@ impl<'de> Deserialize<'de> for Celsius {
}
}
#[derive(Clone, Debug, PartialEq, PartialOrd, TS)]
#[ts(type = "{ value: string, unit: string }")]
pub struct Percentage(f64);
impl Serialize for Percentage {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
@@ -385,6 +448,7 @@ impl<'de> Deserialize<'de> for Percentage {
}
#[derive(Clone, Debug, TS)]
#[ts(type = "{ value: string, unit: string }")]
pub struct MebiBytes(pub f64);
impl Serialize for MebiBytes {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
@@ -411,6 +475,7 @@ impl<'de> Deserialize<'de> for MebiBytes {
}
#[derive(Clone, Debug, PartialEq, PartialOrd, TS)]
#[ts(type = "{ value: string, unit: string }")]
pub struct GigaBytes(f64);
impl Serialize for GigaBytes {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
@@ -490,6 +555,7 @@ pub async fn metrics(ctx: RpcContext) -> Result<Metrics, Error> {
#[derive(Deserialize, Serialize, Clone, Debug, TS)]
#[serde(rename_all = "camelCase")]
#[ts(export)]
pub struct MetricsFollowResponse {
pub guid: Guid,
pub metrics: Metrics,
@@ -1042,20 +1108,36 @@ async fn get_disk_info() -> Result<MetricsDisk, Error> {
})
}
#[derive(
Debug, Clone, Copy, Default, serde::Serialize, serde::Deserialize, TS, clap::ValueEnum,
)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub enum SmtpSecurity {
#[default]
Starttls,
Tls,
}
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize, Parser, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub struct SmtpValue {
#[arg(long, help = "help.arg.smtp-server")]
pub server: String,
#[arg(long, help = "help.arg.smtp-host")]
#[serde(alias = "server")]
pub host: String,
#[arg(long, help = "help.arg.smtp-port")]
pub port: u16,
#[arg(long, help = "help.arg.smtp-from")]
pub from: String,
#[arg(long, help = "help.arg.smtp-login")]
pub login: String,
#[arg(long, help = "help.arg.smtp-username")]
#[serde(alias = "login")]
pub username: String,
#[arg(long, help = "help.arg.smtp-password")]
pub password: Option<String>,
#[arg(long, help = "help.arg.smtp-security")]
#[serde(default)]
pub security: SmtpSecurity,
}
pub async fn set_system_smtp(ctx: RpcContext, smtp: SmtpValue) -> Result<(), Error> {
let smtp = Some(smtp);
@@ -1088,51 +1170,89 @@ pub async fn clear_system_smtp(ctx: RpcContext) -> Result<(), Error> {
}
Ok(())
}
#[derive(Debug, Clone, Deserialize, Serialize, Parser)]
pub struct SetIfconfigUrlParams {
#[arg(help = "help.arg.ifconfig-url")]
pub url: url::Url,
}
pub async fn set_ifconfig_url(
ctx: RpcContext,
SetIfconfigUrlParams { url }: SetIfconfigUrlParams,
) -> Result<(), Error> {
ctx.db
.mutate(|db| {
db.as_public_mut()
.as_server_info_mut()
.as_ifconfig_url_mut()
.ser(&url)
})
.await
.result
}
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize, Parser, TS)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub struct TestSmtpParams {
#[arg(long, help = "help.arg.smtp-server")]
pub server: String,
#[arg(long, help = "help.arg.smtp-host")]
pub host: String,
#[arg(long, help = "help.arg.smtp-port")]
pub port: u16,
#[arg(long, help = "help.arg.smtp-from")]
pub from: String,
#[arg(long, help = "help.arg.smtp-to")]
pub to: String,
#[arg(long, help = "help.arg.smtp-login")]
pub login: String,
#[arg(long, help = "help.arg.smtp-username")]
pub username: String,
#[arg(long, help = "help.arg.smtp-password")]
pub password: String,
#[arg(long, help = "help.arg.smtp-security")]
#[serde(default)]
pub security: SmtpSecurity,
}
pub async fn test_smtp(
_: RpcContext,
TestSmtpParams {
server,
host,
port,
from,
to,
login,
username,
password,
security,
}: TestSmtpParams,
) -> Result<(), Error> {
use lettre::message::header::ContentType;
use lettre::transport::smtp::authentication::Credentials;
use lettre::transport::smtp::client::{Tls, TlsParameters};
use lettre::{AsyncSmtpTransport, AsyncTransport, Message, Tokio1Executor};
AsyncSmtpTransport::<Tokio1Executor>::relay(&server)?
.port(port)
.credentials(Credentials::new(login, password))
.build()
.send(
Message::builder()
.from(from.parse()?)
.to(to.parse()?)
.subject("StartOS Test Email")
.header(ContentType::TEXT_PLAIN)
.body("This is a test email sent from your StartOS Server".to_owned())?,
)
.await?;
let creds = Credentials::new(username, password);
let message = Message::builder()
.from(from.parse()?)
.to(to.parse()?)
.subject("StartOS Test Email")
.header(ContentType::TEXT_PLAIN)
.body("This is a test email sent from your StartOS Server".to_owned())?;
let transport = match security {
SmtpSecurity::Starttls => AsyncSmtpTransport::<Tokio1Executor>::relay(&host)?
.port(port)
.credentials(creds)
.build(),
SmtpSecurity::Tls => {
let tls = TlsParameters::new(host.clone())?;
AsyncSmtpTransport::<Tokio1Executor>::relay(&host)?
.port(port)
.tls(Tls::Wrapper(tls))
.credentials(creds)
.build()
}
};
transport.send(message).await?;
Ok(())
}
@@ -1211,6 +1331,7 @@ pub async fn set_keyboard(ctx: RpcContext, options: KeyboardOptions) -> Result<(
}
#[derive(Debug, Clone, Deserialize, Serialize, TS, Parser)]
#[ts(export)]
#[serde(rename_all = "camelCase")]
pub struct SetLanguageParams {
#[arg(help = "help.arg.language-code")]
@@ -1231,9 +1352,15 @@ pub async fn save_language(language: &str) -> Result<(), Error> {
"/media/startos/config/overlay/usr/lib/locale/locale-archive",
)
.await?;
let locale_content = format!("LANG={language}.UTF-8\n");
write_file_atomic(
"/media/startos/config/overlay/etc/default/locale",
format!("LANG={language}.UTF-8\n").as_bytes(),
locale_content.as_bytes(),
)
.await?;
write_file_atomic(
"/media/startos/config/overlay/etc/locale.conf",
locale_content.as_bytes(),
)
.await?;
Ok(())

30
core/src/tunnel/api.rs
View File

@@ -53,6 +53,24 @@ pub fn tunnel_api<C: Context>() -> ParentHandler<C> {
.with_call_remote::<CliContext>(),
),
)
.subcommand(
"update",
ParentHandler::<C>::new()
.subcommand(
"check",
from_fn_async(super::update::check_update)
.with_display_serializable()
.with_about("about.check-for-updates")
.with_call_remote::<CliContext>(),
)
.subcommand(
"apply",
from_fn_async(super::update::apply_update)
.with_display_serializable()
.with_about("about.apply-available-update")
.with_call_remote::<CliContext>(),
),
)
}
#[derive(Deserialize, Serialize, Parser)]
@@ -414,14 +432,11 @@ pub async fn show_config(
i.iter().find_map(|(_, info)| {
info.ip_info
.as_ref()
.filter(|_| info.public())
.iter()
.find_map(|info| info.subnets.iter().next())
.copied()
.and_then(|ip_info| ip_info.wan_ip)
.map(IpAddr::from)
})
})
.or_not_found("a public IP address")?
.addr()
};
Ok(client
.client_config(
@@ -459,7 +474,10 @@ pub async fn add_forward(
})
.map(|s| s.prefix_len())
.unwrap_or(32);
let rc = ctx.forward.add_forward(source, target, prefix).await?;
let rc = ctx
.forward
.add_forward(source, target, prefix, None)
.await?;
ctx.active_forwards.mutate(|m| {
m.insert(source, rc);
});

2
core/src/tunnel/context.rs
View File

@@ -199,7 +199,7 @@ impl TunnelContext {
})
.map(|s| s.prefix_len())
.unwrap_or(32);
active_forwards.insert(from, forward.add_forward(from, to, prefix).await?);
active_forwards.insert(from, forward.add_forward(from, to, prefix, None).await?);
}
Ok(Self(Arc::new(TunnelContextSeed {

1
core/src/tunnel/mod.rs
View File

@@ -9,6 +9,7 @@ pub mod api;
pub mod auth;
pub mod context;
pub mod db;
pub mod update;
pub mod web;
pub mod wg;

102
core/src/tunnel/update.rs Normal file
View File

@@ -0,0 +1,102 @@
use std::process::Stdio;
use rpc_toolkit::Empty;
use serde::{Deserialize, Serialize};
use tokio::process::Command;
use tracing::instrument;
use ts_rs::TS;
use crate::prelude::*;
use crate::tunnel::context::TunnelContext;
use crate::util::Invoke;
#[derive(Deserialize, Serialize, TS)]
#[serde(rename_all = "camelCase")]
pub struct TunnelUpdateResult {
/// "up-to-date", "update-available", or "updating"
pub status: String,
/// Currently installed version
pub installed: String,
/// Available candidate version
pub candidate: String,
}
#[instrument(skip_all)]
pub async fn check_update(_ctx: TunnelContext, _: Empty) -> Result<TunnelUpdateResult, Error> {
Command::new("apt-get")
.arg("update")
.invoke(ErrorKind::UpdateFailed)
.await?;
let policy_output = Command::new("apt-cache")
.arg("policy")
.arg("start-tunnel")
.invoke(ErrorKind::UpdateFailed)
.await?;
let policy_str = String::from_utf8_lossy(&policy_output).to_string();
let installed = parse_version_field(&policy_str, "Installed:");
let candidate = parse_version_field(&policy_str, "Candidate:");
let status = if installed == candidate {
"up-to-date"
} else {
"update-available"
};
Ok(TunnelUpdateResult {
status: status.to_string(),
installed: installed.unwrap_or_default(),
candidate: candidate.unwrap_or_default(),
})
}
#[instrument(skip_all)]
pub async fn apply_update(_ctx: TunnelContext, _: Empty) -> Result<TunnelUpdateResult, Error> {
let policy_output = Command::new("apt-cache")
.arg("policy")
.arg("start-tunnel")
.invoke(ErrorKind::UpdateFailed)
.await?;
let policy_str = String::from_utf8_lossy(&policy_output).to_string();
let installed = parse_version_field(&policy_str, "Installed:");
let candidate = parse_version_field(&policy_str, "Candidate:");
// Spawn in a separate cgroup via systemd-run so the process survives
// when the postinst script restarts start-tunneld.service.
// After the install completes, reboot the system.
// Uses --reinstall so the update applies even when versions match.
Command::new("systemd-run")
.arg("--scope")
.arg("--")
.arg("sh")
.arg("-c")
.arg("apt-get install --reinstall -y start-tunnel && reboot")
.env("DEBIAN_FRONTEND", "noninteractive")
.stdin(Stdio::null())
.stdout(Stdio::null())
.stderr(Stdio::null())
.spawn()
.with_kind(ErrorKind::UpdateFailed)?;
Ok(TunnelUpdateResult {
status: "updating".to_string(),
installed: installed.unwrap_or_default(),
candidate: candidate.unwrap_or_default(),
})
}
fn parse_version_field(policy: &str, field: &str) -> Option<String> {
policy
.lines()
.find(|l| l.trim().starts_with(field))
.and_then(|l| l.split_whitespace().nth(1))
.filter(|v| *v != "(none)")
.map(|s| s.to_string())
}
#[test]
fn export_bindings_tunnel_update() {
TunnelUpdateResult::export_all_to("bindings/tunnel").unwrap();
}

14
core/src/tunnel/web.rs
View File

@@ -18,7 +18,7 @@ use tokio_rustls::rustls::server::ClientHello;
use ts_rs::TS;
use crate::context::CliContext;
use crate::hostname::Hostname;
use crate::hostname::ServerHostname;
use crate::net::ssl::{SANInfo, root_ca_start_time};
use crate::net::tls::TlsHandler;
use crate::net::web_server::Accept;
@@ -292,7 +292,7 @@ pub async fn generate_certificate(
let root_key = crate::net::ssl::gen_nistp256()?;
let root_cert = crate::net::ssl::make_root_cert(
&root_key,
&Hostname("start-tunnel".into()),
&ServerHostname::new("start-tunnel".into())?,
root_ca_start_time().await,
)?;
let int_key = crate::net::ssl::gen_nistp256()?;
@@ -527,23 +527,23 @@ pub async fn init_web(ctx: CliContext) -> Result<(), Error> {
" 2. Paste the following command (**DO NOT** click Return): pbcopy < ~/Desktop/ca.crt\n",
" 3. Copy your Root CA (including -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----)\n",
" 4. Back in Terminal, click Return. ca.crt is saved to your Desktop\n",
" 5. Complete by trusting your Root CA: https://staging.docs.start9.com/device-guides/mac/ca.html\n",
" 5. Complete by trusting your Root CA: https://docs.start9.com/device-guides/mac/ca.html\n",
" - Linux\n",
" 1. Open gedit, nano, or any editor\n",
" 2. Copy/paste your Root CA (including -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----)\n",
" 3. Name the file ca.crt and save as plaintext\n",
" 5. Complete by trusting your Root CA: https://staging.docs.start9.com/device-guides/linux/ca.html\n",
" 4. Complete by trusting your Root CA: https://docs.start9.com/device-guides/linux/ca.html\n",
" - Windows\n",
" 1. Open the Notepad app\n",
" 2. Copy/paste your Root CA (including -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----)\n",
" 3. Name the file ca.crt and save as plaintext\n",
" 5. Complete by trusting your Root CA: https://staging.docs.start9.com/device-guides/windows/ca.html\n",
" 4. Complete by trusting your Root CA: https://docs.start9.com/device-guides/windows/ca.html\n",
" - Android/Graphene\n",
" 1. Send the ca.crt file (created above) to yourself\n",
" 2. Complete by trusting your Root CA: https://staging.docs.start9.com/device-guides/android/ca.html\n",
" 2. Complete by trusting your Root CA: https://docs.start9.com/device-guides/android/ca.html\n",
" - iOS\n",
" 1. Send the ca.crt file (created above) to yourself\n",
" 2. Complete by trusting your Root CA: https://staging.docs.start9.com/device-guides/ios/ca.html\n",
" 2. Complete by trusting your Root CA: https://docs.start9.com/device-guides/ios/ca.html\n",
));
return Ok(());

21
core/src/util/actor/background.rs
View File

@@ -1,5 +1,6 @@
use futures::future::BoxFuture;
use futures::{Future, FutureExt};
use futures::stream::FuturesUnordered;
use futures::{Future, FutureExt, StreamExt};
use tokio::sync::mpsc;
#[derive(Clone)]
@@ -11,7 +12,7 @@ impl BackgroundJobQueue {
Self(send),
BackgroundJobRunner {
recv,
jobs: Vec::new(),
jobs: FuturesUnordered::new(),
},
)
}
@@ -27,7 +28,7 @@ impl BackgroundJobQueue {
pub struct BackgroundJobRunner {
recv: mpsc::UnboundedReceiver<BoxFuture<'static, ()>>,
jobs: Vec<BoxFuture<'static, ()>>,
jobs: FuturesUnordered<BoxFuture<'static, ()>>,
}
impl BackgroundJobRunner {
pub fn is_empty(&self) -> bool {
@@ -43,19 +44,7 @@ impl Future for BackgroundJobRunner {
while let std::task::Poll::Ready(Some(job)) = self.recv.poll_recv(cx) {
self.jobs.push(job);
}
let complete = self
.jobs
.iter_mut()
.enumerate()
.filter_map(|(i, f)| match f.poll_unpin(cx) {
std::task::Poll::Pending => None,
std::task::Poll::Ready(_) => Some(i),
})
.collect::<Vec<_>>();
for idx in complete.into_iter().rev() {
#[allow(clippy::let_underscore_future)]
let _ = self.jobs.swap_remove(idx);
}
while let std::task::Poll::Ready(Some(())) = self.jobs.poll_next_unpin(cx) {}
if self.jobs.is_empty() && self.recv.is_closed() {
std::task::Poll::Ready(())
} else {

20
core/src/version/mod.rs
View File

@@ -59,8 +59,9 @@ mod v0_4_0_alpha_16;
mod v0_4_0_alpha_17;
mod v0_4_0_alpha_18;
mod v0_4_0_alpha_19;
mod v0_4_0_alpha_20;
pub type Current = v0_4_0_alpha_19::Version; // VERSION_BUMP
pub type Current = v0_4_0_alpha_20::Version; // VERSION_BUMP
impl Current {
#[instrument(skip(self, db))]
@@ -89,7 +90,13 @@ impl Current {
.await
.result?;
}
Ordering::Equal => (),
Ordering::Equal => {
db.apply_function(|db| {
Ok::<_, Error>((to_value(&from_value::<Database>(db.clone())?)?, ()))
})
.await
.result?;
}
}
Ok(())
}
@@ -181,7 +188,8 @@ enum Version {
V0_4_0_alpha_16(Wrapper<v0_4_0_alpha_16::Version>),
V0_4_0_alpha_17(Wrapper<v0_4_0_alpha_17::Version>),
V0_4_0_alpha_18(Wrapper<v0_4_0_alpha_18::Version>),
V0_4_0_alpha_19(Wrapper<v0_4_0_alpha_19::Version>), // VERSION_BUMP
V0_4_0_alpha_19(Wrapper<v0_4_0_alpha_19::Version>),
V0_4_0_alpha_20(Wrapper<v0_4_0_alpha_20::Version>), // VERSION_BUMP
Other(exver::Version),
}
@@ -243,7 +251,8 @@ impl Version {
Self::V0_4_0_alpha_16(v) => DynVersion(Box::new(v.0)),
Self::V0_4_0_alpha_17(v) => DynVersion(Box::new(v.0)),
Self::V0_4_0_alpha_18(v) => DynVersion(Box::new(v.0)),
Self::V0_4_0_alpha_19(v) => DynVersion(Box::new(v.0)), // VERSION_BUMP
Self::V0_4_0_alpha_19(v) => DynVersion(Box::new(v.0)),
Self::V0_4_0_alpha_20(v) => DynVersion(Box::new(v.0)), // VERSION_BUMP
Self::Other(v) => {
return Err(Error::new(
eyre!("unknown version {v}"),
@@ -297,7 +306,8 @@ impl Version {
Version::V0_4_0_alpha_16(Wrapper(x)) => x.semver(),
Version::V0_4_0_alpha_17(Wrapper(x)) => x.semver(),
Version::V0_4_0_alpha_18(Wrapper(x)) => x.semver(),
Version::V0_4_0_alpha_19(Wrapper(x)) => x.semver(), // VERSION_BUMP
Version::V0_4_0_alpha_19(Wrapper(x)) => x.semver(),
Version::V0_4_0_alpha_20(Wrapper(x)) => x.semver(), // VERSION_BUMP
Version::Other(x) => x.clone(),
}
}

14
core/src/version/update_details/v0_4_0.md
View File

@@ -10,13 +10,13 @@ A server is not a toy. It is a critical component of the computing paradigm, and
Start9 is paving new ground with StartOS, trying to create what most developers and IT professionals thought impossible; namely, an OS and user experience that affords a normal person the same independent control over their data and communications as an experienced Linux sysadmin.
The difficulty of our endeavor requires making mistakes; and our integrity and dedication to excellence require that we correct them. This means a willingness to discard bad ideas and broken parts, and if absolutely necessary, to tear it all down and start over. That is exactly what we did with StartOS v0.2.0 in 2020. It is what we did with StartOS v0.3.0 in 2022. And we are doing it now with StartOS v0.4.0 in 2025.
The difficulty of our endeavor requires making mistakes; and our integrity and dedication to excellence require that we correct them. This means a willingness to discard bad ideas and broken parts, and if absolutely necessary, to tear it all down and start over. That is exactly what we did with StartOS v0.2.0 in 2020. It is what we did with StartOS v0.3.0 in 2022. And we are doing it now with StartOS v0.4.0 in 2026.
v0.4.0 is a complete rewrite of StartOS, almost nothing survived. After nearly six years of building StartOS, we believe that we have finally arrived at the correct architecture and foundation that will allow us to deliver on the promise of sovereign computing.
## Changelog
### Improved User interface
### New User interface
We re-wrote the StartOS UI to be more performant, more intuitive, and better looking on both mobile and desktop. Enjoy.
@@ -28,6 +28,10 @@ StartOS v0.4.0 supports multiple languages and also makes it easy to add more la
Neither Docker nor Podman offer the reliability and flexibility needed for StartOS. Instead, v0.4.0 uses a nested container paradigm based on LXC for the outer container and Linux namespaces for sub containers. This architecture naturally supports multi container setups.
### Hardware Acceleration
Services can take advantage of (and require) the presence of certain hardware modules, such as Nvidia GPUs, for transcoding or inference purposes. For example, StartOS and Ollama can run natively on The Nvidia DGX Spark and take full advantage of the hardware/firmware stack to perform local inference against open source models.
### New S9PK archive format
The S9PK archive format has been overhauled to allow for signature verification of partial downloads, and allow direct mounting of container images without unpacking the s9pk.
@@ -80,13 +84,13 @@ The new start-fs fuse module unifies file system expectations for various platfo
StartOS now uses Extended Versioning (Exver), which consists of three parts: (1) a Semver-compliant upstream version, (2) a Semver-compliant wrapper version, and (3) an optional "flavor" prefix. Flavors can be thought of as alternative implementations of services, where a user would only want one or the other installed, and data can feasibly be migrating between the two. Another common characteristic of flavors is that they satisfy the same API requirement of dependents, though this is not strictly necessary. A valid Exver looks something like this: `#knots:29.0:1.0-beta.1`. This would translate to "the first beta release of StartOS wrapper version 1.0 of Bitcoin Knots version 29.0".
### ACME
### Let's Encrypt
StartOS now supports using ACME protocol to automatically obtain SSL/TLS certificates from widely trusted certificate authorities, such as Let's Encrypt, for your public domains. This means people visiting your public websites and APIs will not need to download and trust your server's Root CA.
StartOS now supports Let's Encrypt to automatically obtain SSL/TLS certificates for public domains. This means people visiting your public websites and APIs will not need to download and trust your server's Root CA.
### Gateways
Gateways connect your server to the Internet. They process outbound traffic, and under certain conditions, they also permit inbound traffic. For example, your router is a gateway. It is now possible add gateways to StartOS, such as StartTunnel, in order to more granularly control how your installed services are exposed to the Internet.
Gateways connect your server to the Internet, facilitating inbound and outbound traffic. Your router is a gateway. It is now possible to add Wireguard VPN gateways to your server to control how devices outside the LAN connect to your server and how your server connects out to the Internet.
### Static DNS Servers

168
core/src/version/v0_3_6_alpha_0.rs
View File

@@ -1,4 +1,4 @@
use std::collections::{BTreeMap, BTreeSet};
use std::collections::BTreeMap;
use std::ffi::OsStr;
use std::path::Path;
@@ -21,19 +21,16 @@ use crate::backup::target::cifs::CifsTargets;
use crate::context::RpcContext;
use crate::disk::mount::filesystem::cifs::Cifs;
use crate::disk::mount::util::unmount;
use crate::hostname::Hostname;
use crate::hostname::{ServerHostname, ServerHostnameInfo};
use crate::net::forward::AvailablePorts;
use crate::net::host::Host;
use crate::net::keys::KeyStore;
use crate::net::tor::{OnionAddress, TorSecretKey};
use crate::notifications::Notifications;
use crate::prelude::*;
use crate::s9pk::merkle_archive::source::multi_cursor_file::MultiCursorFile;
use crate::ssh::{SshKeys, SshPubKey};
use crate::util::Invoke;
use crate::util::crypto::ed25519_expand_key;
use crate::util::serde::Pem;
use crate::{DATA_DIR, HostId, Id, PACKAGE_DATA, PackageId, ReplayId};
use crate::{DATA_DIR, PACKAGE_DATA, PackageId, ReplayId};
lazy_static::lazy_static! {
static ref V0_3_6_alpha_0: exver::Version = exver::Version::new(
@@ -146,12 +143,7 @@ pub struct Version;
impl VersionT for Version {
type Previous = v0_3_5_2::Version;
type PreUpRes = (
AccountInfo,
SshKeys,
CifsTargets,
BTreeMap<PackageId, BTreeMap<HostId, TorSecretKey>>,
);
type PreUpRes = (AccountInfo, SshKeys, CifsTargets);
fn semver(self) -> exver::Version {
V0_3_6_alpha_0.clone()
}
@@ -166,21 +158,15 @@ impl VersionT for Version {
let cifs = previous_cifs(&pg).await?;
let tor_keys = previous_tor_keys(&pg).await?;
Command::new("systemctl")
.arg("stop")
.arg("postgresql@*.service")
.invoke(crate::ErrorKind::Database)
.await?;
Ok((account, ssh_keys, cifs, tor_keys))
Ok((account, ssh_keys, cifs))
}
fn up(
self,
db: &mut Value,
(account, ssh_keys, cifs, tor_keys): Self::PreUpRes,
) -> Result<Value, Error> {
fn up(self, db: &mut Value, (account, ssh_keys, cifs): Self::PreUpRes) -> Result<Value, Error> {
let prev_package_data = db["package-data"].clone();
let wifi = json!({
@@ -242,11 +228,7 @@ impl VersionT for Version {
"ui": db["ui"],
});
let mut keystore = KeyStore::new(&account)?;
for key in tor_keys.values().flat_map(|v| v.values()) {
assert!(key.is_valid());
keystore.onion.insert(key.clone());
}
let keystore = KeyStore::new(&account)?;
let private = {
let mut value = json!({});
@@ -350,20 +332,6 @@ impl VersionT for Version {
false
};
let onions = input[&*id]["installed"]["interface-addresses"]
.as_object()
.into_iter()
.flatten()
.filter_map(|(id, addrs)| {
addrs["tor-address"].as_str().map(|addr| {
Ok((
HostId::from(Id::try_from(id.clone())?),
addr.parse::<OnionAddress>()?,
))
})
})
.collect::<Result<BTreeMap<_, _>, Error>>()?;
if let Err(e) = async {
let package_s9pk = tokio::fs::File::open(path).await?;
let file = MultiCursorFile::open(&package_s9pk).await?;
@@ -381,11 +349,8 @@ impl VersionT for Version {
.await?
.await?;
let to_sync = ctx
.db
ctx.db
.mutate(|db| {
let mut to_sync = BTreeSet::new();
let package = db
.as_public_mut()
.as_package_data_mut()
@@ -396,29 +361,11 @@ impl VersionT for Version {
.as_tasks_mut()
.remove(&ReplayId::from("needs-config"))?;
}
for (id, onion) in onions {
package
.as_hosts_mut()
.upsert(&id, || Ok(Host::new()))?
.as_onions_mut()
.mutate(|o| {
o.clear();
o.insert(onion);
Ok(())
})?;
to_sync.insert(id);
}
Ok(to_sync)
Ok(())
})
.await
.result?;
if let Some(service) = &*ctx.services.get(&id).await {
for host_id in to_sync {
service.sync_host(host_id.clone()).await?;
}
}
Ok::<_, Error>(())
}
.await
@@ -481,42 +428,15 @@ async fn previous_account_info(pg: &sqlx::Pool<sqlx::Postgres>) -> Result<Accoun
password: account_query
.try_get("password")
.with_ctx(|_| (ErrorKind::Database, "password"))?,
tor_keys: vec![TorSecretKey::from_bytes(
if let Some(bytes) = account_query
.try_get::<Option<Vec<u8>>, _>("tor_key")
.with_ctx(|_| (ErrorKind::Database, "tor_key"))?
{
<[u8; 64]>::try_from(bytes).map_err(|e| {
Error::new(
eyre!("expected vec of len 64, got len {}", e.len()),
ErrorKind::ParseDbField,
)
})?
} else {
ed25519_expand_key(
&<[u8; 32]>::try_from(
account_query
.try_get::<Vec<u8>, _>("network_key")
.with_kind(ErrorKind::Database)?,
)
.map_err(|e| {
Error::new(
eyre!("expected vec of len 32, got len {}", e.len()),
ErrorKind::ParseDbField,
)
})?,
)
},
)?],
server_id: account_query
.try_get("server_id")
.with_ctx(|_| (ErrorKind::Database, "server_id"))?,
hostname: Hostname(
hostname: ServerHostnameInfo::from_hostname(ServerHostname::new(
account_query
.try_get::<String, _>("hostname")
.with_ctx(|_| (ErrorKind::Database, "hostname"))?
.into(),
),
)?),
root_ca_key: PKey::private_key_from_pem(
&account_query
.try_get::<String, _>("root_ca_key_pem")
@@ -578,69 +498,3 @@ async fn previous_ssh_keys(pg: &sqlx::Pool<sqlx::Postgres>) -> Result<SshKeys, E
};
Ok(ssh_keys)
}
#[tracing::instrument(skip_all)]
async fn previous_tor_keys(
pg: &sqlx::Pool<sqlx::Postgres>,
) -> Result<BTreeMap<PackageId, BTreeMap<HostId, TorSecretKey>>, Error> {
let mut res = BTreeMap::<PackageId, BTreeMap<HostId, TorSecretKey>>::new();
let net_key_query = sqlx::query(r#"SELECT * FROM network_keys"#)
.fetch_all(pg)
.await
.with_kind(ErrorKind::Database)?;
for row in net_key_query {
let package_id: PackageId = row
.try_get::<String, _>("package")
.with_ctx(|_| (ErrorKind::Database, "network_keys::package"))?
.parse()?;
let interface_id: HostId = row
.try_get::<String, _>("interface")
.with_ctx(|_| (ErrorKind::Database, "network_keys::interface"))?
.parse()?;
let key = TorSecretKey::from_bytes(ed25519_expand_key(
&<[u8; 32]>::try_from(
row.try_get::<Vec<u8>, _>("key")
.with_ctx(|_| (ErrorKind::Database, "network_keys::key"))?,
)
.map_err(|e| {
Error::new(
eyre!("expected vec of len 32, got len {}", e.len()),
ErrorKind::ParseDbField,
)
})?,
))?;
res.entry(package_id).or_default().insert(interface_id, key);
}
let tor_key_query = sqlx::query(r#"SELECT * FROM tor"#)
.fetch_all(pg)
.await
.with_kind(ErrorKind::Database)?;
for row in tor_key_query {
let package_id: PackageId = row
.try_get::<String, _>("package")
.with_ctx(|_| (ErrorKind::Database, "tor::package"))?
.parse()?;
let interface_id: HostId = row
.try_get::<String, _>("interface")
.with_ctx(|_| (ErrorKind::Database, "tor::interface"))?
.parse()?;
let key = TorSecretKey::from_bytes(
<[u8; 64]>::try_from(
row.try_get::<Vec<u8>, _>("key")
.with_ctx(|_| (ErrorKind::Database, "tor::key"))?,
)
.map_err(|e| {
Error::new(
eyre!("expected vec of len 64, got len {}", e.len()),
ErrorKind::ParseDbField,
)
})?,
)?;
res.entry(package_id).or_default().insert(interface_id, key);
}
Ok(res)
}

3
core/src/version/v0_3_6_alpha_10.rs
View File

@@ -8,7 +8,6 @@ use super::v0_3_5::V0_3_0_COMPAT;
use super::{VersionT, v0_3_6_alpha_9};
use crate::GatewayId;
use crate::net::host::address::PublicDomainConfig;
use crate::net::tor::OnionAddress;
use crate::prelude::*;
lazy_static::lazy_static! {
@@ -22,7 +21,7 @@ lazy_static::lazy_static! {
#[serde(rename_all = "camelCase")]
#[serde(tag = "kind")]
enum HostAddress {
Onion { address: OnionAddress },
Onion { address: String },
Domain { address: InternedString },
}

Some files were not shown because too many files have changed in this diff Show More