mirror of
https://github.com/Start9Labs/start-os.git
synced 2026-03-30 20:14:49 +00:00
mok ux, autofill device and pf forms, docss for st, docs for start-sdk
This commit is contained in:
Matt Hill
422
sdk/ARCHITECTURE.md
Normal file
422
sdk/ARCHITECTURE.md
Normal file
@@ -0,0 +1,422 @@
|
||||
# SDK Architecture
|
||||
|
||||
The Start SDK is split into two npm packages that form a layered architecture: **base** provides the foundational types, ABI contract, and effects interface; **package** builds on base to provide the developer-facing SDK facade.
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ package/ (@start9labs/start-sdk) │
|
||||
│ Developer-facing facade, daemon management, health checks, │
|
||||
│ backup system, file helpers, triggers, subcontainers │
|
||||
│ │
|
||||
│ ┌───────────────────────────────────────────────────────┐ │
|
||||
│ │ base/ (@start9labs/start-sdk-base) │ │
|
||||
│ │ ABI, Effects, OS bindings, actions/input builders, │ │
|
||||
│ │ ExVer parser, interfaces, dependencies, S9pk, utils │ │
|
||||
│ └───────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│ ▲
|
||||
│ Effects calls (RPC) │ Callbacks
|
||||
▼ │
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ StartOS Runtime (Rust supervisor) │
|
||||
│ Executes effects, manages containers, networking, storage │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
The SDK follows [Semantic Versioning](https://semver.org/) within the `0.4.0-beta.*` pre-release series. The SDK version tracks independently from the StartOS release versions.
|
||||
|
||||
## Base Package (`base/`)
|
||||
|
||||
The base package is a self-contained library of types, interfaces, and low-level builders. It has no dependency on the package layer and can be used independently when only type definitions or validation are needed.
|
||||
|
||||
### OS Bindings (`base/lib/osBindings/`)
|
||||
|
||||
~325 auto-generated TypeScript files defining every type exchanged between the SDK and the StartOS runtime. These cover the full surface area of the system: manifests, actions, health checks, service interfaces, bind parameters, dependency requirements, alerts, SSL, domains, SMTP, networking, images, and more.
|
||||
|
||||
All bindings are re-exported through `base/lib/osBindings/index.ts`.
|
||||
|
||||
Key types include:
|
||||
- `Manifest` — The full service package manifest as understood by the OS
|
||||
- `ActionMetadata` — Describes an action's name, description, visibility, and availability
|
||||
- `BindParams` — Port binding configuration (protocol, hostId, internal port)
|
||||
- `ServiceInterface` — A network endpoint exported to users
|
||||
- `DependencyRequirement` — Version range and health check requirements for a dependency
|
||||
- `SetHealth` — Health check result reporting
|
||||
- `HostnameInfo` / `Host` — Hostname and host metadata
|
||||
|
||||
### ABI and Core Types (`base/lib/types.ts`)
|
||||
|
||||
Defines the Application Binary Interface — the contract every service package must fulfill:
|
||||
|
||||
```typescript
|
||||
namespace ExpectedExports {
|
||||
main // Start the service daemon(s)
|
||||
init // Initialize on install/update/restore
|
||||
uninit // Clean up on uninstall/update/shutdown
|
||||
manifest // Service metadata
|
||||
actions // User-invocable operations
|
||||
createBackup // Export service data
|
||||
}
|
||||
```
|
||||
|
||||
Also defines foundational types used throughout the SDK:
|
||||
- `Daemon` / `DaemonReturned` — Running process handles with `wait()` and `term()`
|
||||
- `CommandType` — Shell string, argv array, or `UseEntrypoint`
|
||||
- `ServiceInterfaceType` — `'ui' | 'api' | 'p2p'`
|
||||
- `SmtpValue` — SMTP server configuration
|
||||
- `KnownError` — Structured user-facing errors
|
||||
- `DependsOn` — Package-to-health-check dependency mapping
|
||||
- `PathMaker`, `MaybePromise`, `DeepPartial`, `DeepReadonly` — Utility types
|
||||
|
||||
### Effects Interface (`base/lib/Effects.ts`)
|
||||
|
||||
The bridge between TypeScript service code and the StartOS runtime. Every runtime capability is accessed through an `Effects` object passed to lifecycle hooks.
|
||||
|
||||
Effects are organized by subsystem:
|
||||
|
||||
| Subsystem | Methods | Purpose |
|
||||
|-----------|---------|---------|
|
||||
| **Action** | `export`, `clear`, `getInput`, `run`, `createTask`, `clearTasks` | Register and invoke user actions |
|
||||
| **Control** | `restart`, `shutdown`, `getStatus`, `setMainStatus` | Service lifecycle control |
|
||||
| **Dependency** | `setDependencies`, `getDependencies`, `checkDependencies`, `mount`, `getInstalledPackages`, `getServiceManifest` | Inter-service dependency management |
|
||||
| **Health** | `setHealth` | Report health check results |
|
||||
| **Subcontainer** | `createFs`, `destroyFs` | Container filesystem management |
|
||||
| **Networking** | `bind`, `getServicePortForward`, `clearBindings`, `getHostInfo`, `getContainerIp`, `getOsIp`, `getOutboundGateway` | Port binding and network info |
|
||||
| **Interfaces** | `exportServiceInterface`, `getServiceInterface`, `listServiceInterfaces`, `clearServiceInterfaces` | Service endpoint management |
|
||||
| **Plugin** | `plugin.url.register`, `plugin.url.exportUrl`, `plugin.url.clearUrls` | Plugin system hooks |
|
||||
| **SSL** | `getSslCertificate`, `getSslKey` | TLS certificate management |
|
||||
| **System** | `getSystemSmtp`, `setDataVersion`, `getDataVersion` | System-wide configuration |
|
||||
|
||||
Effects also support reactive callbacks: many methods accept an optional `callback` parameter that the runtime invokes when the underlying value changes, enabling the reactive subscription patterns (`const()`, `watch()`, etc.).
|
||||
|
||||
### Action and Input System (`base/lib/actions/`)
|
||||
|
||||
#### Actions (`setupActions.ts`)
|
||||
|
||||
The `Action` class defines user-invocable operations. Two factory methods:
|
||||
- `Action.withInput(id, metadata, inputSpec, prefill, execute)` — Action with a validated form
|
||||
- `Action.withoutInput(id, metadata, execute)` — Action without user input
|
||||
|
||||
`Actions` is a typed map accumulated via `.addAction()` chaining.
|
||||
|
||||
#### Input Specification (`actions/input/`)
|
||||
|
||||
A builder-pattern system for declaring validated form inputs:
|
||||
|
||||
```
|
||||
inputSpec/
|
||||
├── builder/
|
||||
│ ├── inputSpec.ts — InputSpec.of() entry point
|
||||
│ ├── value.ts — Value class (individual form fields)
|
||||
│ ├── list.ts — List builder (arrays of values)
|
||||
│ └── variants.ts — Variants/Union builder (conditional fields)
|
||||
├── inputSpecTypes.ts — Type definitions for all field types
|
||||
└── inputSpecConstants.ts — Pre-built specs (SMTP, etc.)
|
||||
```
|
||||
|
||||
Supported field types via `Value`:
|
||||
- `text`, `textarea`, `number` — Text and numeric input
|
||||
- `toggle` — Boolean switch
|
||||
- `select`, `multiselect` — Single/multi-choice dropdown
|
||||
- `list` — Repeatable array of sub-values
|
||||
- `color`, `datetime` — Specialized pickers
|
||||
- `object` — Nested sub-form
|
||||
- `union` / `dynamicUnion` — Conditional fields based on a discriminator
|
||||
|
||||
### Dependencies (`base/lib/dependencies/`)
|
||||
|
||||
- `setupDependencies.ts` — Declare what the service depends on (package IDs, version ranges, health checks)
|
||||
- `dependencies.ts` — Runtime dependency checking via `checkDependencies()`
|
||||
|
||||
### Interfaces (`base/lib/interfaces/`)
|
||||
|
||||
Network interface declaration and port binding:
|
||||
|
||||
- `setupInterfaces.ts` — Top-level `setupServiceInterfaces()` function
|
||||
- `Host.ts` — `MultiHost` class for binding ports and exporting interfaces. A single MultiHost can bind a port and export multiple interfaces (e.g. a primary UI and admin UI on the same port with different paths)
|
||||
- `ServiceInterfaceBuilder.ts` — Builder for constructing `ServiceInterface` objects with name, type, description, scheme overrides, username, path, and query params
|
||||
- `setupExportedUrls.ts` — URL plugin support for exporting URLs to other services
|
||||
|
||||
### Initialization (`base/lib/inits/`)
|
||||
|
||||
- `setupInit.ts` — Compose init scripts that run on install, update, restore, or boot
|
||||
- `setupUninit.ts` — Compose uninit scripts that run on uninstall, update, or shutdown
|
||||
- `setupOnInit` / `setupOnUninit` — Register callbacks for specific init/uninit events
|
||||
|
||||
Init scripts receive a `kind` parameter (`'install' | 'update' | 'restore' | null`) so they can branch logic based on the initialization context.
|
||||
|
||||
### Extended Versioning (`base/lib/exver/`)
|
||||
|
||||
A PEG parser-based versioning system that extends semver:
|
||||
|
||||
- **`Version`** — Standard semantic version (`1.2.3-beta.1`)
|
||||
- **`ExtendedVersion` (ExVer)** — Adds an optional flavor prefix and a downstream version: `#flavor:upstream:downstream`
|
||||
- **`VersionRange`** — Boolean expressions over version comparisons (`>=1.0.0 && <2.0.0 || =3.0.0`)
|
||||
|
||||
The parser is generated from `exver.pegjs` via Peggy and emitted as `exver.ts`.
|
||||
|
||||
ExVer separates upstream project versions from StartOS wrapper versions, allowing the package maintainer's versioning to evolve independently from the upstream software.
|
||||
|
||||
### S9pk Format (`base/lib/s9pk/`)
|
||||
|
||||
Parser and verifier for `.s9pk` service package archives:
|
||||
|
||||
- `S9pk` class — Deserialize and inspect package contents
|
||||
- Merkle archive support for cryptographic verification of package integrity
|
||||
- Methods: `deserialize()`, `icon()`, `license()`, etc.
|
||||
|
||||
### Utilities (`base/lib/util/`)
|
||||
|
||||
~28 utility modules including:
|
||||
|
||||
**Reactive subscription wrappers** — Each wraps an Effects callback-based method into a consistent reactive API:
|
||||
- `Watchable` — Base class providing `const()`, `once()`, `watch()`, `onChange()`, `waitFor()`
|
||||
- `GetContainerIp`, `GetStatus`, `GetSystemSmtp`, `GetOutboundGateway`, `GetSslCertificate`, `GetHostInfo`, `GetServiceManifest` — Typed wrappers for specific Effects methods
|
||||
|
||||
**General utilities:**
|
||||
- `deepEqual` / `deepMerge` — Deep object comparison and merging
|
||||
- `patterns` — Hostname regex, port validators
|
||||
- `splitCommand` — Parse shell command strings into argv arrays
|
||||
- `Drop` — RAII-style cleanup utility
|
||||
- `graph` — Dependency graph utilities
|
||||
|
||||
## Package Layer (`package/`)
|
||||
|
||||
The package layer provides the developer-facing API. It re-exports everything from base and adds higher-level abstractions.
|
||||
|
||||
### StartSdk Facade (`package/lib/StartSdk.ts`)
|
||||
|
||||
The primary entry point for service developers. Constructed via a builder chain:
|
||||
|
||||
```typescript
|
||||
const sdk = StartSdk.of()
|
||||
.withManifest(manifest)
|
||||
.build(true)
|
||||
```
|
||||
|
||||
The `.build()` method returns an object containing the entire SDK surface area, organized by concern:
|
||||
|
||||
| Category | Members | Purpose |
|
||||
|----------|---------|---------|
|
||||
| **Manifest** | `manifest`, `volumes` | Access manifest data and volume paths |
|
||||
| **Actions** | `Action.withInput`, `Action.withoutInput`, `Actions`, `action.run`, `action.createTask`, `action.createOwnTask`, `action.clearTask` | Define and manage user actions |
|
||||
| **Daemons** | `Daemons.of`, `Daemon.of`, `setupMain` | Configure service processes |
|
||||
| **Health** | `healthCheck.checkPortListening`, `.checkWebUrl`, `.runHealthScript` | Built-in health checks |
|
||||
| **Interfaces** | `createInterface`, `MultiHost.of`, `setupInterfaces`, `serviceInterface.*` | Network endpoint management |
|
||||
| **Backups** | `setupBackups`, `Backups.ofVolumes`, `Backups.ofSyncs`, `Backups.withOptions` | Backup configuration |
|
||||
| **Dependencies** | `setupDependencies`, `checkDependencies` | Dependency declaration and verification |
|
||||
| **Init/Uninit** | `setupInit`, `setupUninit`, `setupOnInit`, `setupOnUninit` | Lifecycle hooks |
|
||||
| **Containers** | `SubContainer.of`, `SubContainer.withTemp`, `Mounts.of` | Container execution with mounts |
|
||||
| **Forms** | `InputSpec.of`, `Value`, `Variants`, `List` | Form input builders |
|
||||
| **Triggers** | `trigger.defaultTrigger`, `.cooldownTrigger`, `.changeOnFirstSuccess`, `.successFailure` | Health check polling strategies |
|
||||
| **Reactive** | `getContainerIp`, `getStatus`, `getSystemSmtp`, `getOutboundGateway`, `getSslCertificate`, `getServiceManifest` | Subscription-based data access |
|
||||
| **Plugins** | `plugin.url.register`, `plugin.url.exportUrl` | Plugin system (gated by manifest `plugins` field) |
|
||||
| **Effects** | `restart`, `shutdown`, `setHealth`, `mount`, `clearBindings`, ... | Direct effect wrappers |
|
||||
| **Utilities** | `nullIfEmpty`, `useEntrypoint`, `patterns`, `setDataVersion`, `getDataVersion` | Misc helpers |
|
||||
|
||||
### Daemon Management (`package/lib/mainFn/`)
|
||||
|
||||
The daemon subsystem manages long-running processes:
|
||||
|
||||
```
|
||||
mainFn/
|
||||
├── Daemons.ts — Multi-daemon topology builder
|
||||
├── Daemon.ts — Single daemon wrapper
|
||||
├── HealthDaemon.ts — Health check executor
|
||||
├── CommandController.ts — Command execution controller
|
||||
├── Mounts.ts — Volume/asset/dependency mount builder
|
||||
├── Oneshot.ts — One-time startup commands
|
||||
└── index.ts — setupMain() entry point
|
||||
```
|
||||
|
||||
**Daemons** is a builder that accumulates process definitions:
|
||||
```typescript
|
||||
sdk.Daemons.of(effects)
|
||||
.addDaemon('db', { /* command, ready probe, mounts */ })
|
||||
.addDaemon('app', { requires: ['db'], /* ... */ })
|
||||
.addHealthCheck('primary', { /* ... */ })
|
||||
```
|
||||
|
||||
Features:
|
||||
- Startup ordering via `requires` (dependency graph between daemons)
|
||||
- Ready probes (wait for a daemon to be ready before starting dependents)
|
||||
- Graceful shutdown with configurable signals and timeouts
|
||||
- One-shot commands that run before daemons start
|
||||
|
||||
**Mounts** declares what to attach to a container:
|
||||
```typescript
|
||||
sdk.Mounts.of()
|
||||
.mountVolume('main', '/data')
|
||||
.mountAssets('scripts', '/scripts')
|
||||
.mountDependency('bitcoind', 'main', '/bitcoin-data', { readonly: true })
|
||||
.mountBackup('/backup')
|
||||
```
|
||||
|
||||
### Health Checks (`package/lib/health/`)
|
||||
|
||||
```
|
||||
health/
|
||||
├── HealthCheck.ts — Periodic probe with startup grace period
|
||||
└── checkFns/
|
||||
├── checkPortListening.ts — TCP port connectivity check
|
||||
├── checkWebUrl.ts — HTTP(S) status code check
|
||||
└── runHealthScript.ts — Script exit code check
|
||||
```
|
||||
|
||||
Health checks are paired with **triggers** that control polling behavior:
|
||||
- `defaultTrigger` — Fixed interval (e.g. every 30s)
|
||||
- `cooldownTrigger` — Wait longer after failures
|
||||
- `changeOnFirstSuccess` — Rapid polling until first success, then slow down
|
||||
- `successFailure` — Different intervals for healthy vs unhealthy states
|
||||
|
||||
### Backup System (`package/lib/backup/`)
|
||||
|
||||
```
|
||||
backup/
|
||||
├── setupBackups.ts — Top-level setup function
|
||||
└── Backups.ts — Volume selection and rsync options
|
||||
```
|
||||
|
||||
Three builder patterns:
|
||||
- `Backups.ofVolumes('main', 'data')` — Back up entire volumes
|
||||
- `Backups.ofSyncs([{ dataPath, backupPath }])` — Custom sync pairs
|
||||
- `Backups.withOptions({ exclude: ['cache/'] })` — Rsync options
|
||||
|
||||
### File Helpers (`package/lib/util/fileHelper.ts`)
|
||||
|
||||
Type-safe configuration file management:
|
||||
|
||||
```typescript
|
||||
const configFile = FileHelper.yaml(effects, sdk.volumes.main.path('config.yml'), {
|
||||
port: 8080,
|
||||
debug: false,
|
||||
})
|
||||
|
||||
// Reactive reading
|
||||
const config = await configFile.read.const(effects)
|
||||
|
||||
// Partial merge
|
||||
await configFile.merge({ debug: true })
|
||||
|
||||
// Full write
|
||||
await configFile.write({ port: 9090, debug: true })
|
||||
```
|
||||
|
||||
Supported formats: JSON, YAML, TOML, INI, ENV, and custom parsers.
|
||||
|
||||
### Subcontainers (`package/lib/util/SubContainer.ts`)
|
||||
|
||||
Execute commands in isolated container environments:
|
||||
|
||||
```typescript
|
||||
// Long-lived subcontainer
|
||||
const container = await sdk.SubContainer.of(effects, { imageId: 'main' }, mounts, 'app')
|
||||
|
||||
// One-shot execution
|
||||
await sdk.SubContainer.withTemp(effects, { imageId: 'main' }, mounts, 'migrate', async (c) => {
|
||||
await c.exec(['run-migrations'])
|
||||
})
|
||||
```
|
||||
|
||||
### Manifest Building (`package/lib/manifest/`)
|
||||
|
||||
```typescript
|
||||
const manifest = setupManifest({
|
||||
id: 'my-service',
|
||||
title: 'My Service',
|
||||
license: 'MIT',
|
||||
description: { short: '...', long: '...' },
|
||||
images: { main: { source: { dockerTag: 'myimage:1.0' } } },
|
||||
volumes: { main: {} },
|
||||
dependencies: {},
|
||||
// ...
|
||||
})
|
||||
|
||||
export default buildManifest(manifest)
|
||||
```
|
||||
|
||||
`buildManifest()` finalizes the manifest with the current SDK version, OS version compatibility, and migration version ranges.
|
||||
|
||||
### Versioning (`package/lib/version/`)
|
||||
|
||||
Helpers for data version management during migrations:
|
||||
|
||||
```typescript
|
||||
await sdk.setDataVersion(effects, '1.2.0:0')
|
||||
const version = await sdk.getDataVersion(effects)
|
||||
```
|
||||
|
||||
Used in init scripts to track which migration version the service's data has been brought to.
|
||||
|
||||
### Internationalization (`package/lib/i18n/`)
|
||||
|
||||
```typescript
|
||||
const t = setupI18n({ en_US: enStrings, es_ES: esStrings })
|
||||
const greeting = t('hello', { name: 'World' }) // "Hello, World!" or "Hola, World!"
|
||||
```
|
||||
|
||||
Supports locale fallback and Intl-based formatting.
|
||||
|
||||
### Triggers (`package/lib/trigger/`)
|
||||
|
||||
Polling strategy functions that determine when health checks run:
|
||||
|
||||
```typescript
|
||||
sdk.trigger.defaultTrigger({ timeout: 30_000 })
|
||||
sdk.trigger.cooldownTrigger({ timeout: 30_000, cooldown: 60_000 })
|
||||
sdk.trigger.changeOnFirstSuccess({ first: 5_000, then: 30_000 })
|
||||
sdk.trigger.successFailure({ success: 60_000, failure: 10_000 })
|
||||
```
|
||||
|
||||
## Build Pipeline
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed build instructions, make targets, and development workflow.
|
||||
|
||||
At a high level: Peggy generates the ExVer parser, TypeScript compiles both packages in strict mode (base to `baseDist/`, package to `dist/`), hand-written `.js`/`.d.ts` pairs are copied into the output, and `node_modules` are bundled for self-contained distribution.
|
||||
|
||||
## Data Flow
|
||||
|
||||
A typical service package lifecycle:
|
||||
|
||||
```
|
||||
1. INSTALL / UPDATE / RESTORE
|
||||
├── init({ effects, kind })
|
||||
│ ├── Version migrations (if update)
|
||||
│ ├── setupDependencies()
|
||||
│ ├── setupInterfaces() → bind ports, export interfaces
|
||||
│ └── Actions registration → export actions to OS
|
||||
│
|
||||
2. MAIN
|
||||
│ setupMain() → Daemons.of(effects)
|
||||
│ ├── Oneshots run first
|
||||
│ ├── Daemons start in dependency order
|
||||
│ ├── Health checks begin polling
|
||||
│ └── Service runs until shutdown/restart
|
||||
│
|
||||
3. SHUTDOWN / UNINSTALL / UPDATE
|
||||
│ uninit({ effects, target })
|
||||
│ └── Version down-migrations (if needed)
|
||||
│
|
||||
4. BACKUP (user-triggered)
|
||||
createBackup({ effects })
|
||||
└── rsync volumes to backup location
|
||||
```
|
||||
|
||||
## Key Design Patterns
|
||||
|
||||
### Builder Pattern
|
||||
Most SDK APIs use immutable builder chains: `Daemons.of().addDaemon().addHealthCheck()`, `Mounts.of().mountVolume().mountAssets()`, `Actions.of().addAction()`. This provides type accumulation — each chained call narrows the type to reflect what has been configured.
|
||||
|
||||
### Effects as Capability System
|
||||
All runtime interactions go through the `Effects` object rather than direct system calls. This makes the runtime boundary explicit, enables the OS to mediate all side effects, and makes service code testable by providing mock effects.
|
||||
|
||||
### Reactive Subscriptions
|
||||
The `Watchable` base class provides a consistent API for values that can change over time:
|
||||
- `const(effects)` — Read once; if the value changes, triggers a retry of the enclosing context
|
||||
- `once()` — Read once without reactivity
|
||||
- `watch()` — Async generator yielding on each change
|
||||
- `onChange(callback)` — Invoke callback on each change
|
||||
- `waitFor(predicate)` — Block until a condition is met
|
||||
|
||||
### Type-safe Manifest Threading
|
||||
The manifest type flows through the entire SDK via generics. When you call `StartSdk.of().withManifest(manifest)`, the manifest's volume names, image IDs, dependency IDs, and plugin list become available as type constraints throughout all subsequent API calls. For example, `Mounts.of().mountVolume()` only accepts volume names declared in the manifest.
|
||||
Reference in New Issue
Block a user