start-os/sdk/ARCHITECTURE.md

SDK Architecture

The Start SDK is split into two npm packages that form a layered architecture: base provides the foundational types, ABI contract, and effects interface; package builds on base to provide the developer-facing SDK facade.

┌─────────────────────────────────────────────────────────────┐
│  package/ (@start9labs/start-sdk)                           │
│  Developer-facing facade, daemon management, health checks, │
│  backup system, file helpers, triggers, subcontainers       │
│                                                             │
│  ┌───────────────────────────────────────────────────────┐  │
│  │  base/ (@start9labs/start-sdk-base)                   │  │
│  │  ABI, Effects, OS bindings, actions/input builders,   │  │
│  │  ExVer parser, interfaces, dependencies, S9pk, utils  │  │
│  └───────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
         │                                        ▲
         │ Effects calls (RPC)                    │ Callbacks
         ▼                                        │
┌─────────────────────────────────────────────────────────────┐
│  StartOS Runtime (Rust supervisor)                          │
│  Executes effects, manages containers, networking, storage  │
└─────────────────────────────────────────────────────────────┘

The SDK follows Semantic Versioning within the 0.4.0-beta.* pre-release series. The SDK version tracks independently from the StartOS release versions.

Base Package (base/)

The base package is a self-contained library of types, interfaces, and low-level builders. It has no dependency on the package layer and can be used independently when only type definitions or validation are needed.

OS Bindings (base/lib/osBindings/)

~325 auto-generated TypeScript files defining every type exchanged between the SDK and the StartOS runtime. These cover the full surface area of the system: manifests, actions, health checks, service interfaces, bind parameters, dependency requirements, alerts, SSL, domains, SMTP, networking, images, and more.

All bindings are re-exported through base/lib/osBindings/index.ts.

Key types include:

• Manifest — The full service package manifest as understood by the OS
• ActionMetadata — Describes an action's name, description, visibility, and availability
• BindParams — Port binding configuration (protocol, hostId, internal port)
• ServiceInterface — A network endpoint exported to users
• DependencyRequirement — Version range and health check requirements for a dependency
• SetHealth — Health check result reporting
• HostnameInfo / Host — Hostname and host metadata

ABI and Core Types (base/lib/types.ts)

Defines the Application Binary Interface — the contract every service package must fulfill:

namespace ExpectedExports {
  main       // Start the service daemon(s)
  init       // Initialize on install/update/restore
  uninit     // Clean up on uninstall/update/shutdown
  manifest   // Service metadata
  actions    // User-invocable operations
  createBackup // Export service data
}

Also defines foundational types used throughout the SDK:

• Daemon / DaemonReturned — Running process handles with wait() and term()
• CommandType — Shell string, argv array, or UseEntrypoint
• ServiceInterfaceType — 'ui' | 'api' | 'p2p'
• SmtpValue — SMTP server configuration
• KnownError — Structured user-facing errors
• DependsOn — Package-to-health-check dependency mapping
• PathMaker, MaybePromise, DeepPartial, DeepReadonly — Utility types

Effects Interface (base/lib/Effects.ts)

The bridge between TypeScript service code and the StartOS runtime. Every runtime capability is accessed through an Effects object passed to lifecycle hooks.

Effects are organized by subsystem:

Subsystem	Methods	Purpose
Action	export, clear, getInput, run, createTask, clearTasks	Register and invoke user actions
Control	restart, shutdown, getStatus, setMainStatus	Service lifecycle control
Dependency	setDependencies, getDependencies, checkDependencies, mount, getInstalledPackages, getServiceManifest	Inter-service dependency management
Health	setHealth	Report health check results
Subcontainer	createFs, destroyFs	Container filesystem management
Networking	bind, getServicePortForward, clearBindings, getHostInfo, getContainerIp, getOsIp, getOutboundGateway	Port binding and network info
Interfaces	exportServiceInterface, getServiceInterface, listServiceInterfaces, clearServiceInterfaces	Service endpoint management
Plugin	plugin.url.register, plugin.url.exportUrl, plugin.url.clearUrls	Plugin system hooks
SSL	getSslCertificate, getSslKey	TLS certificate management
System	getSystemSmtp, setDataVersion, getDataVersion	System-wide configuration

Effects also support reactive callbacks: many methods accept an optional callback parameter that the runtime invokes when the underlying value changes, enabling the reactive subscription patterns (const(), watch(), etc.).

Action and Input System (base/lib/actions/)

Actions (setupActions.ts)

The Action class defines user-invocable operations. Two factory methods:

• Action.withInput(id, metadata, inputSpec, prefill, execute) — Action with a validated form
• Action.withoutInput(id, metadata, execute) — Action without user input

Actions is a typed map accumulated via .addAction() chaining.

Input Specification (actions/input/)

A builder-pattern system for declaring validated form inputs:

inputSpec/
├── builder/
│   ├── inputSpec.ts   — InputSpec.of() entry point
│   ├── value.ts       — Value class (individual form fields)
│   ├── list.ts        — List builder (arrays of values)
│   └── variants.ts    — Variants/Union builder (conditional fields)
├── inputSpecTypes.ts  — Type definitions for all field types
└── inputSpecConstants.ts — Pre-built specs (SMTP, etc.)

Supported field types via Value:

• text, textarea, number — Text and numeric input
• toggle — Boolean switch
• select, multiselect — Single/multi-choice dropdown
• list — Repeatable array of sub-values
• color, datetime — Specialized pickers
• object — Nested sub-form
• union / dynamicUnion — Conditional fields based on a discriminator

Dependencies (base/lib/dependencies/)

• setupDependencies.ts — Declare what the service depends on (package IDs, version ranges, health checks)
• dependencies.ts — Runtime dependency checking via checkDependencies()

Interfaces (base/lib/interfaces/)

Network interface declaration and port binding:

• setupInterfaces.ts — Top-level setupServiceInterfaces() function
• Host.ts — MultiHost class for binding ports and exporting interfaces. A single MultiHost can bind a port and export multiple interfaces (e.g. a primary UI and admin UI on the same port with different paths)
• ServiceInterfaceBuilder.ts — Builder for constructing ServiceInterface objects with name, type, description, scheme overrides, username, path, and query params
• setupExportedUrls.ts — URL plugin support for exporting URLs to other services

Initialization (base/lib/inits/)

• setupInit.ts — Compose init scripts that run on install, update, restore, or boot
• setupUninit.ts — Compose uninit scripts that run on uninstall, update, or shutdown
• setupOnInit / setupOnUninit — Register callbacks for specific init/uninit events

Init scripts receive a kind parameter ('install' | 'update' | 'restore' | null) so they can branch logic based on the initialization context.

Extended Versioning (base/lib/exver/)

A PEG parser-based versioning system that extends semver:

• Version — Standard semantic version (1.2.3-beta.1)
• ExtendedVersion (ExVer) — Adds an optional flavor prefix and a downstream version: #flavor:upstream:downstream
• VersionRange — Boolean expressions over version comparisons (>=1.0.0 && <2.0.0 || =3.0.0)

The parser is generated from exver.pegjs via Peggy and emitted as exver.ts.

ExVer separates upstream project versions from StartOS wrapper versions, allowing the package maintainer's versioning to evolve independently from the upstream software.

S9pk Format (base/lib/s9pk/)

Parser and verifier for .s9pk service package archives:

• S9pk class — Deserialize and inspect package contents
• Merkle archive support for cryptographic verification of package integrity
• Methods: deserialize(), icon(), license(), etc.

Utilities (base/lib/util/)

~28 utility modules including:

Reactive subscription wrappers — Each wraps an Effects callback-based method into a consistent reactive API:

• Watchable — Base class providing const(), once(), watch(), onChange(), waitFor()
• GetContainerIp, GetStatus, GetSystemSmtp, GetOutboundGateway, GetSslCertificate, GetHostInfo, GetServiceManifest — Typed wrappers for specific Effects methods

General utilities:

• deepEqual / deepMerge — Deep object comparison and merging
• patterns — Hostname regex, port validators
• splitCommand — Parse shell command strings into argv arrays
• Drop — RAII-style cleanup utility
• graph — Dependency graph utilities

Package Layer (package/)

The package layer provides the developer-facing API. It re-exports everything from base and adds higher-level abstractions.

StartSdk Facade (package/lib/StartSdk.ts)

The primary entry point for service developers. Constructed via a builder chain:

const sdk = StartSdk.of()
  .withManifest(manifest)
  .build(true)

The .build() method returns an object containing the entire SDK surface area, organized by concern:

Category	Members	Purpose
Manifest	manifest, volumes	Access manifest data and volume paths
Actions	Action.withInput, Action.withoutInput, Actions, action.run, action.createTask, action.createOwnTask, action.clearTask	Define and manage user actions
Daemons	Daemons.of, Daemon.of, setupMain	Configure service processes
Health	healthCheck.checkPortListening, .checkWebUrl, .runHealthScript	Built-in health checks
Interfaces	createInterface, MultiHost.of, setupInterfaces, serviceInterface.*	Network endpoint management
Backups	setupBackups, Backups.ofVolumes, Backups.ofSyncs, Backups.withOptions	Backup configuration
Dependencies	setupDependencies, checkDependencies	Dependency declaration and verification
Init/Uninit	setupInit, setupUninit, setupOnInit, setupOnUninit	Lifecycle hooks
Containers	SubContainer.of, SubContainer.withTemp, Mounts.of	Container execution with mounts
Forms	InputSpec.of, Value, Variants, List	Form input builders
Triggers	trigger.defaultTrigger, .cooldownTrigger, .changeOnFirstSuccess, .successFailure	Health check polling strategies
Reactive	getContainerIp, getStatus, getSystemSmtp, getOutboundGateway, getSslCertificate, getServiceManifest	Subscription-based data access
Plugins	plugin.url.register, plugin.url.exportUrl	Plugin system (gated by manifest plugins field)
Effects	restart, shutdown, setHealth, mount, clearBindings, ...	Direct effect wrappers
Utilities	nullIfEmpty, useEntrypoint, patterns, setDataVersion, getDataVersion	Misc helpers

Daemon Management (package/lib/mainFn/)

The daemon subsystem manages long-running processes:

mainFn/
├── Daemons.ts          — Multi-daemon topology builder
├── Daemon.ts           — Single daemon wrapper
├── HealthDaemon.ts     — Health check executor
├── CommandController.ts — Command execution controller
├── Mounts.ts           — Volume/asset/dependency mount builder
├── Oneshot.ts          — One-time startup commands
└── index.ts            — setupMain() entry point

Daemons is a builder that accumulates process definitions:

sdk.Daemons.of(effects)
  .addDaemon('db', { /* command, ready probe, mounts */ })
  .addDaemon('app', { requires: ['db'], /* ... */ })
  .addHealthCheck('primary', { /* ... */ })

Features:

• Startup ordering via requires (dependency graph between daemons)
• Ready probes (wait for a daemon to be ready before starting dependents)
• Graceful shutdown with configurable signals and timeouts
• One-shot commands that run before daemons start

Mounts declares what to attach to a container:

sdk.Mounts.of()
  .mountVolume('main', '/data')
  .mountAssets('scripts', '/scripts')
  .mountDependency('bitcoind', 'main', '/bitcoin-data', { readonly: true })
  .mountBackup('/backup')

Health Checks (package/lib/health/)

health/
├── HealthCheck.ts       — Periodic probe with startup grace period
└── checkFns/
    ├── checkPortListening.ts — TCP port connectivity check
    ├── checkWebUrl.ts        — HTTP(S) status code check
    └── runHealthScript.ts    — Script exit code check

Health checks are paired with triggers that control polling behavior:

• defaultTrigger — Fixed interval (e.g. every 30s)
• cooldownTrigger — Wait longer after failures
• changeOnFirstSuccess — Rapid polling until first success, then slow down
• successFailure — Different intervals for healthy vs unhealthy states

Backup System (package/lib/backup/)

backup/
├── setupBackups.ts — Top-level setup function
└── Backups.ts      — Volume selection and rsync options

Three builder patterns:

• Backups.ofVolumes('main', 'data') — Back up entire volumes
• Backups.ofSyncs([{ dataPath, backupPath }]) — Custom sync pairs
• Backups.withOptions({ exclude: ['cache/'] }) — Rsync options

File Helpers (package/lib/util/fileHelper.ts)

Type-safe configuration file management:

const configFile = FileHelper.yaml(effects, sdk.volumes.main.path('config.yml'), {
  port: 8080,
  debug: false,
})

// Reactive reading
const config = await configFile.read.const(effects)

// Partial merge
await configFile.merge({ debug: true })

// Full write
await configFile.write({ port: 9090, debug: true })

Supported formats: JSON, YAML, TOML, INI, ENV, and custom parsers.

Subcontainers (package/lib/util/SubContainer.ts)

Execute commands in isolated container environments:

// Long-lived subcontainer
const container = await sdk.SubContainer.of(effects, { imageId: 'main' }, mounts, 'app')

// One-shot execution
await sdk.SubContainer.withTemp(effects, { imageId: 'main' }, mounts, 'migrate', async (c) => {
  await c.exec(['run-migrations'])
})

Manifest Building (package/lib/manifest/)

const manifest = setupManifest({
  id: 'my-service',
  title: 'My Service',
  license: 'MIT',
  description: { short: '...', long: '...' },
  images: { main: { source: { dockerTag: 'myimage:1.0' } } },
  volumes: { main: {} },
  dependencies: {},
  // ...
})

export default buildManifest(manifest)

buildManifest() finalizes the manifest with the current SDK version, OS version compatibility, and migration version ranges.

Versioning (package/lib/version/)

Helpers for data version management during migrations:

await sdk.setDataVersion(effects, '1.2.0:0')
const version = await sdk.getDataVersion(effects)

Used in init scripts to track which migration version the service's data has been brought to.

Internationalization (package/lib/i18n/)

const t = setupI18n({ en_US: enStrings, es_ES: esStrings })
const greeting = t('hello', { name: 'World' }) // "Hello, World!" or "Hola, World!"

Supports locale fallback and Intl-based formatting.

Triggers (package/lib/trigger/)

Polling strategy functions that determine when health checks run:

sdk.trigger.defaultTrigger({ timeout: 30_000 })
sdk.trigger.cooldownTrigger({ timeout: 30_000, cooldown: 60_000 })
sdk.trigger.changeOnFirstSuccess({ first: 5_000, then: 30_000 })
sdk.trigger.successFailure({ success: 60_000, failure: 10_000 })

Build Pipeline

See CONTRIBUTING.md for detailed build instructions, make targets, and development workflow.

At a high level: Peggy generates the ExVer parser, TypeScript compiles both packages in strict mode (base to baseDist/, package to dist/), hand-written .js/.d.ts pairs are copied into the output, and node_modules are bundled for self-contained distribution.

Data Flow

A typical service package lifecycle:

1. INSTALL / UPDATE / RESTORE
   ├── init({ effects, kind })
   │   ├── Version migrations (if update)
   │   ├── setupDependencies()
   │   ├── setupInterfaces() → bind ports, export interfaces
   │   └── Actions registration → export actions to OS
   │
2. MAIN
   │   setupMain() → Daemons.of(effects)
   │   ├── Oneshots run first
   │   ├── Daemons start in dependency order
   │   ├── Health checks begin polling
   │   └── Service runs until shutdown/restart
   │
3. SHUTDOWN / UNINSTALL / UPDATE
   │   uninit({ effects, target })
   │   └── Version down-migrations (if needed)
   │
4. BACKUP (user-triggered)
       createBackup({ effects })
       └── rsync volumes to backup location

Key Design Patterns

Builder Pattern

Most SDK APIs use immutable builder chains: Daemons.of().addDaemon().addHealthCheck(), Mounts.of().mountVolume().mountAssets(), Actions.of().addAction(). This provides type accumulation — each chained call narrows the type to reflect what has been configured.

Effects as Capability System

All runtime interactions go through the Effects object rather than direct system calls. This makes the runtime boundary explicit, enables the OS to mediate all side effects, and makes service code testable by providing mock effects.

Reactive Subscriptions

The Watchable base class provides a consistent API for values that can change over time:

• const(effects) — Read once; if the value changes, triggers a retry of the enclosing context
• once() — Read once without reactivity
• watch() — Async generator yielding on each change
• onChange(callback) — Invoke callback on each change
• waitFor(predicate) — Block until a condition is met

Type-safe Manifest Threading

The manifest type flows through the entire SDK via generics. When you call StartSdk.of().withManifest(manifest), the manifest's volume names, image IDs, dependency IDs, and plugin list become available as type constraints throughout all subsequent API calls. For example, Mounts.of().mountVolume() only accepts volume names declared in the manifest.