start-os/sdk/ARCHITECTURE.md

# SDK Architecture

The Start SDK is split into two npm packages that form a layered architecture: **base** provides the foundational types, ABI contract, and effects interface; **package** builds on base to provide the developer-facing SDK facade.

```
┌─────────────────────────────────────────────────────────────┐
│  package/ (@start9labs/start-sdk)                           │
│  Developer-facing facade, daemon management, health checks, │
│  backup system, file helpers, triggers, subcontainers       │
│                                                             │
│  ┌───────────────────────────────────────────────────────┐  │
│  │  base/ (@start9labs/start-sdk-base)                   │  │
│  │  ABI, Effects, OS bindings, actions/input builders,   │  │
│  │  ExVer parser, interfaces, dependencies, S9pk, utils  │  │
│  └───────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
         │                                        ▲
         │ Effects calls (RPC)                    │ Callbacks
         ▼                                        │
┌─────────────────────────────────────────────────────────────┐
│  StartOS Runtime (Rust supervisor)                          │
│  Executes effects, manages containers, networking, storage  │
└─────────────────────────────────────────────────────────────┘
```

The SDK follows [Semantic Versioning](https://semver.org/) within the `0.4.0-beta.*` pre-release series. The SDK version tracks independently from the StartOS release versions.

## Base Package (`base/`)

The base package is a self-contained library of types, interfaces, and low-level builders. It has no dependency on the package layer and can be used independently when only type definitions or validation are needed.

### OS Bindings (`base/lib/osBindings/`)

~325 auto-generated TypeScript files defining every type exchanged between the SDK and the StartOS runtime. These cover the full surface area of the system: manifests, actions, health checks, service interfaces, bind parameters, dependency requirements, alerts, SSL, domains, SMTP, networking, images, and more.

All bindings are re-exported through `base/lib/osBindings/index.ts`.

Key types include:
- `Manifest` — The full service package manifest as understood by the OS
- `ActionMetadata` — Describes an action's name, description, visibility, and availability
- `BindParams` — Port binding configuration (protocol, hostId, internal port)
- `ServiceInterface` — A network endpoint exported to users
- `DependencyRequirement` — Version range and health check requirements for a dependency
- `SetHealth` — Health check result reporting
- `HostnameInfo` / `Host` — Hostname and host metadata

### ABI and Core Types (`base/lib/types.ts`)

Defines the Application Binary Interface — the contract every service package must fulfill:

```typescript
namespace ExpectedExports {
  main       // Start the service daemon(s)
  init       // Initialize on install/update/restore
  uninit     // Clean up on uninstall/update/shutdown
  manifest   // Service metadata
  actions    // User-invocable operations
  createBackup // Export service data
}
```

Also defines foundational types used throughout the SDK:
- `Daemon` / `DaemonReturned` — Running process handles with `wait()` and `term()`
- `CommandType` — Shell string, argv array, or `UseEntrypoint`
- `ServiceInterfaceType` — `'ui' | 'api' | 'p2p'`
- `SmtpValue` — SMTP server configuration
- `KnownError` — Structured user-facing errors
- `DependsOn` — Package-to-health-check dependency mapping
- `PathMaker`, `MaybePromise`, `DeepPartial`, `DeepReadonly` — Utility types

### Effects Interface (`base/lib/Effects.ts`)

The bridge between TypeScript service code and the StartOS runtime. Every runtime capability is accessed through an `Effects` object passed to lifecycle hooks.

Effects are organized by subsystem:

| Subsystem | Methods | Purpose |
|-----------|---------|---------|
| **Action** | `export`, `clear`, `getInput`, `run`, `createTask`, `clearTasks` | Register and invoke user actions |
| **Control** | `restart`, `shutdown`, `getStatus`, `setMainStatus` | Service lifecycle control |
| **Dependency** | `setDependencies`, `getDependencies`, `checkDependencies`, `mount`, `getInstalledPackages`, `getServiceManifest` | Inter-service dependency management |
| **Health** | `setHealth` | Report health check results |
| **Subcontainer** | `createFs`, `destroyFs` | Container filesystem management |
| **Networking** | `bind`, `getServicePortForward`, `clearBindings`, `getHostInfo`, `getContainerIp`, `getOsIp`, `getOutboundGateway` | Port binding and network info |
| **Interfaces** | `exportServiceInterface`, `getServiceInterface`, `listServiceInterfaces`, `clearServiceInterfaces` | Service endpoint management |
| **Plugin** | `plugin.url.register`, `plugin.url.exportUrl`, `plugin.url.clearUrls` | Plugin system hooks |
| **SSL** | `getSslCertificate`, `getSslKey` | TLS certificate management |
| **System** | `getSystemSmtp`, `setDataVersion`, `getDataVersion` | System-wide configuration |

Effects also support reactive callbacks: many methods accept an optional `callback` parameter that the runtime invokes when the underlying value changes, enabling the reactive subscription patterns (`const()`, `watch()`, etc.).

### Action and Input System (`base/lib/actions/`)

#### Actions (`setupActions.ts`)

The `Action` class defines user-invocable operations. Two factory methods:
- `Action.withInput(id, metadata, inputSpec, prefill, execute)` — Action with a validated form
- `Action.withoutInput(id, metadata, execute)` — Action without user input

`Actions` is a typed map accumulated via `.addAction()` chaining.

#### Input Specification (`actions/input/`)

A builder-pattern system for declaring validated form inputs:

```
inputSpec/
├── builder/
│   ├── inputSpec.ts   — InputSpec.of() entry point
│   ├── value.ts       — Value class (individual form fields)
│   ├── list.ts        — List builder (arrays of values)
│   └── variants.ts    — Variants/Union builder (conditional fields)
├── inputSpecTypes.ts  — Type definitions for all field types
└── inputSpecConstants.ts — Pre-built specs (SMTP, etc.)
```

Supported field types via `Value`:
- `text`, `textarea`, `number` — Text and numeric input
- `toggle` — Boolean switch
- `select`, `multiselect` — Single/multi-choice dropdown
- `list` — Repeatable array of sub-values
- `color`, `datetime` — Specialized pickers
- `object` — Nested sub-form
- `union` / `dynamicUnion` — Conditional fields based on a discriminator

### Dependencies (`base/lib/dependencies/`)

- `setupDependencies.ts` — Declare what the service depends on (package IDs, version ranges, health checks)
- `dependencies.ts` — Runtime dependency checking via `checkDependencies()`

### Interfaces (`base/lib/interfaces/`)

Network interface declaration and port binding:

- `setupInterfaces.ts` — Top-level `setupServiceInterfaces()` function
- `Host.ts` — `MultiHost` class for binding ports and exporting interfaces. A single MultiHost can bind a port and export multiple interfaces (e.g. a primary UI and admin UI on the same port with different paths)
- `ServiceInterfaceBuilder.ts` — Builder for constructing `ServiceInterface` objects with name, type, description, scheme overrides, username, path, and query params
- `setupExportedUrls.ts` — URL plugin support for exporting URLs to other services

### Initialization (`base/lib/inits/`)

- `setupInit.ts` — Compose init scripts that run on install, update, restore, or boot
- `setupUninit.ts` — Compose uninit scripts that run on uninstall, update, or shutdown
- `setupOnInit` / `setupOnUninit` — Register callbacks for specific init/uninit events

Init scripts receive a `kind` parameter (`'install' | 'update' | 'restore' | null`) so they can branch logic based on the initialization context.

### Extended Versioning (`base/lib/exver/`)

A PEG parser-based versioning system that extends semver:

- **`Version`** — Standard semantic version (`1.2.3-beta.1`)
- **`ExtendedVersion` (ExVer)** — Adds an optional flavor prefix and a downstream version: `#flavor:upstream:downstream`
- **`VersionRange`** — Boolean expressions over version comparisons (`>=1.0.0 && <2.0.0 || =3.0.0`)

The parser is generated from `exver.pegjs` via Peggy and emitted as `exver.ts`.

ExVer separates upstream project versions from StartOS wrapper versions, allowing the package maintainer's versioning to evolve independently from the upstream software.

### S9pk Format (`base/lib/s9pk/`)

Parser and verifier for `.s9pk` service package archives:

- `S9pk` class — Deserialize and inspect package contents
- Merkle archive support for cryptographic verification of package integrity
- Methods: `deserialize()`, `icon()`, `license()`, etc.

### Utilities (`base/lib/util/`)

~28 utility modules including:

**Reactive subscription wrappers** — Each wraps an Effects callback-based method into a consistent reactive API:
- `Watchable` — Base class providing `const()`, `once()`, `watch()`, `onChange()`, `waitFor()`
- `GetContainerIp`, `GetStatus`, `GetSystemSmtp`, `GetOutboundGateway`, `GetSslCertificate`, `GetHostInfo`, `GetServiceManifest` — Typed wrappers for specific Effects methods

**General utilities:**
- `deepEqual` / `deepMerge` — Deep object comparison and merging
- `patterns` — Hostname regex, port validators
- `splitCommand` — Parse shell command strings into argv arrays
- `Drop` — RAII-style cleanup utility
- `graph` — Dependency graph utilities

## Package Layer (`package/`)

The package layer provides the developer-facing API. It re-exports everything from base and adds higher-level abstractions.

### StartSdk Facade (`package/lib/StartSdk.ts`)

The primary entry point for service developers. Constructed via a builder chain:

```typescript
const sdk = StartSdk.of()
  .withManifest(manifest)
  .build(true)
```

The `.build()` method returns an object containing the entire SDK surface area, organized by concern:

| Category | Members | Purpose |
|----------|---------|---------|
| **Manifest** | `manifest`, `volumes` | Access manifest data and volume paths |
| **Actions** | `Action.withInput`, `Action.withoutInput`, `Actions`, `action.run`, `action.createTask`, `action.createOwnTask`, `action.clearTask` | Define and manage user actions |
| **Daemons** | `Daemons.of`, `Daemon.of`, `setupMain` | Configure service processes |
| **Health** | `healthCheck.checkPortListening`, `.checkWebUrl`, `.runHealthScript` | Built-in health checks |
| **Interfaces** | `createInterface`, `MultiHost.of`, `setupInterfaces`, `serviceInterface.*` | Network endpoint management |
| **Backups** | `setupBackups`, `Backups.ofVolumes`, `Backups.ofSyncs`, `Backups.withOptions` | Backup configuration |
| **Dependencies** | `setupDependencies`, `checkDependencies` | Dependency declaration and verification |
| **Init/Uninit** | `setupInit`, `setupUninit`, `setupOnInit`, `setupOnUninit` | Lifecycle hooks |
| **Containers** | `SubContainer.of`, `SubContainer.withTemp`, `Mounts.of` | Container execution with mounts |
| **Forms** | `InputSpec.of`, `Value`, `Variants`, `List` | Form input builders |
| **Triggers** | `trigger.defaultTrigger`, `.cooldownTrigger`, `.changeOnFirstSuccess`, `.successFailure` | Health check polling strategies |
| **Reactive** | `getContainerIp`, `getStatus`, `getSystemSmtp`, `getOutboundGateway`, `getSslCertificate`, `getServiceManifest` | Subscription-based data access |
| **Plugins** | `plugin.url.register`, `plugin.url.exportUrl` | Plugin system (gated by manifest `plugins` field) |
| **Effects** | `restart`, `shutdown`, `setHealth`, `mount`, `clearBindings`, ... | Direct effect wrappers |
| **Utilities** | `nullIfEmpty`, `useEntrypoint`, `patterns`, `setDataVersion`, `getDataVersion` | Misc helpers |

### Daemon Management (`package/lib/mainFn/`)

The daemon subsystem manages long-running processes:

```
mainFn/
├── Daemons.ts          — Multi-daemon topology builder
├── Daemon.ts           — Single daemon wrapper
├── HealthDaemon.ts     — Health check executor
├── CommandController.ts — Command execution controller
├── Mounts.ts           — Volume/asset/dependency mount builder
├── Oneshot.ts          — One-time startup commands
└── index.ts            — setupMain() entry point
```

**Daemons** is a builder that accumulates process definitions:
```typescript
sdk.Daemons.of(effects)
  .addDaemon('db', { /* command, ready probe, mounts */ })
  .addDaemon('app', { requires: ['db'], /* ... */ })
  .addHealthCheck('primary', { /* ... */ })
```

Features:
- Startup ordering via `requires` (dependency graph between daemons)
- Ready probes (wait for a daemon to be ready before starting dependents)
- Graceful shutdown with configurable signals and timeouts
- One-shot commands that run before daemons start

**Mounts** declares what to attach to a container:
```typescript
sdk.Mounts.of()
  .mountVolume('main', '/data')
  .mountAssets('scripts', '/scripts')
  .mountDependency('bitcoind', 'main', '/bitcoin-data', { readonly: true })
  .mountBackup('/backup')
```

### Health Checks (`package/lib/health/`)

```
health/
├── HealthCheck.ts       — Periodic probe with startup grace period
└── checkFns/
    ├── checkPortListening.ts — TCP port connectivity check
    ├── checkWebUrl.ts        — HTTP(S) status code check
    └── runHealthScript.ts    — Script exit code check
```

Health checks are paired with **triggers** that control polling behavior:
- `defaultTrigger` — Fixed interval (e.g. every 30s)
- `cooldownTrigger` — Wait longer after failures
- `changeOnFirstSuccess` — Rapid polling until first success, then slow down
- `successFailure` — Different intervals for healthy vs unhealthy states

### Backup System (`package/lib/backup/`)

```
backup/
├── setupBackups.ts — Top-level setup function
└── Backups.ts      — Volume selection and rsync options
```

Three builder patterns:
- `Backups.ofVolumes('main', 'data')` — Back up entire volumes
- `Backups.ofSyncs([{ dataPath, backupPath }])` — Custom sync pairs
- `Backups.withOptions({ exclude: ['cache/'] })` — Rsync options

### File Helpers (`package/lib/util/fileHelper.ts`)

Type-safe configuration file management:

```typescript
const configFile = FileHelper.yaml(effects, sdk.volumes.main.path('config.yml'), {
  port: 8080,
  debug: false,
})

// Reactive reading
const config = await configFile.read.const(effects)

// Partial merge
await configFile.merge({ debug: true })

// Full write
await configFile.write({ port: 9090, debug: true })
```

Supported formats: JSON, YAML, TOML, INI, ENV, and custom parsers.

### Subcontainers (`package/lib/util/SubContainer.ts`)

Execute commands in isolated container environments:

```typescript
// Long-lived subcontainer
const container = await sdk.SubContainer.of(effects, { imageId: 'main' }, mounts, 'app')

// One-shot execution
await sdk.SubContainer.withTemp(effects, { imageId: 'main' }, mounts, 'migrate', async (c) => {
  await c.exec(['run-migrations'])
})
```

### Manifest Building (`package/lib/manifest/`)

```typescript
const manifest = setupManifest({
  id: 'my-service',
  title: 'My Service',
  license: 'MIT',
  description: { short: '...', long: '...' },
  images: { main: { source: { dockerTag: 'myimage:1.0' } } },
  volumes: { main: {} },
  dependencies: {},
  // ...
})

export default buildManifest(manifest)
```

`buildManifest()` finalizes the manifest with the current SDK version, OS version compatibility, and migration version ranges.

### Versioning (`package/lib/version/`)

Helpers for data version management during migrations:

```typescript
await sdk.setDataVersion(effects, '1.2.0:0')
const version = await sdk.getDataVersion(effects)
```

Used in init scripts to track which migration version the service's data has been brought to.

### Internationalization (`package/lib/i18n/`)

```typescript
const t = setupI18n({ en_US: enStrings, es_ES: esStrings })
const greeting = t('hello', { name: 'World' }) // "Hello, World!" or "Hola, World!"
```

Supports locale fallback and Intl-based formatting.

### Triggers (`package/lib/trigger/`)

Polling strategy functions that determine when health checks run:

```typescript
sdk.trigger.defaultTrigger({ timeout: 30_000 })
sdk.trigger.cooldownTrigger({ timeout: 30_000, cooldown: 60_000 })
sdk.trigger.changeOnFirstSuccess({ first: 5_000, then: 30_000 })
sdk.trigger.successFailure({ success: 60_000, failure: 10_000 })
```

## Build Pipeline

See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed build instructions, make targets, and development workflow.

At a high level: Peggy generates the ExVer parser, TypeScript compiles both packages in strict mode (base to `baseDist/`, package to `dist/`), hand-written `.js`/`.d.ts` pairs are copied into the output, and `node_modules` are bundled for self-contained distribution.

## Data Flow

A typical service package lifecycle:

```
1. INSTALL / UPDATE / RESTORE
   ├── init({ effects, kind })
   │   ├── Version migrations (if update)
   │   ├── setupDependencies()
   │   ├── setupInterfaces() → bind ports, export interfaces
   │   └── Actions registration → export actions to OS
   │
2. MAIN
   │   setupMain() → Daemons.of(effects)
   │   ├── Oneshots run first
   │   ├── Daemons start in dependency order
   │   ├── Health checks begin polling
   │   └── Service runs until shutdown/restart
   │
3. SHUTDOWN / UNINSTALL / UPDATE
   │   uninit({ effects, target })
   │   └── Version down-migrations (if needed)
   │
4. BACKUP (user-triggered)
       createBackup({ effects })
       └── rsync volumes to backup location
```

## Key Design Patterns

### Builder Pattern
Most SDK APIs use immutable builder chains: `Daemons.of().addDaemon().addHealthCheck()`, `Mounts.of().mountVolume().mountAssets()`, `Actions.of().addAction()`. This provides type accumulation — each chained call narrows the type to reflect what has been configured.

### Effects as Capability System
All runtime interactions go through the `Effects` object rather than direct system calls. This makes the runtime boundary explicit, enables the OS to mediate all side effects, and makes service code testable by providing mock effects.

### Reactive Subscriptions
The `Watchable` base class provides a consistent API for values that can change over time:
- `const(effects)` — Read once; if the value changes, triggers a retry of the enclosing context
- `once()` — Read once without reactivity
- `watch()` — Async generator yielding on each change
- `onChange(callback)` — Invoke callback on each change
- `waitFor(predicate)` — Block until a condition is met

### Type-safe Manifest Threading
The manifest type flows through the entire SDK via generics. When you call `StartSdk.of().withManifest(manifest)`, the manifest's volume names, image IDs, dependency IDs, and plugin list become available as type constraints throughout all subsequent API calls. For example, `Mounts.of().mountVolume()` only accepts volume names declared in the manifest.