Start9/documentation
2
0
Fork 0
mirror of https://github.com/Start9Labs/documentation.git synced 2026-03-26 10:21:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

re-arrange the user docs (#76)

Browse Source 
* re-arrange the user docs

* Minor description edit

* update styles

* fix service links

Co-authored-by: kn0wmad <kn0wmad@protonmail.com>
Co-authored-by: Lucy Cifferello <12953208+elvece@users.noreply.github.com>
This commit is contained in:
Matt Hill
2022-02-18 08:49:53 -07:00
committed by GitHub
parent c38d5c1a0c
commit aed86e7e6c
151 changed files with 500 additions and 781 deletions
Download Patch File Download Diff File Expand all files Collapse all files

41
site/source/developer-docs/specification/backups.rst Normal file
View File

@@ -0,0 +1,41 @@
.. _service_backups:
=======
Backups
=======
Backups are defined by the service developer in the backups section of the Manifest. By default, this action is a command executed within a Docker container.
EmbassyOS provides a system utility for creating backups. This system utility is the `duplicity <https://en.wikipedia.org/wiki/Duplicity_(software)>`_ library that is run inside the Docker image entitled _compat_.
The path to be backed up can be specified in the Manifest. In the create backup example below, it is the last argument:
.. code:: yaml
...
# create backup
entrypoint: compat
# Arguments to pass into the entrypoint executable. In this example, the full command run will be: `compat duplicity <package-id> /mnt/backup /root/data`
args:
- duplicity
- <package-id>
- /mnt/backup
# For duplicity, the backup mount point needs to be something other than `/root`, so we default to `/root/data`
- /root/data
...
For restoration purposes, it might be beneficial to ignore certain files or folders. For instance, ignore the shared/public folder that is mounted for dependencies that expose this feature as it causes data inconsistencies on restore.
In this case, create a ``.backupignore`` file in the mounted volume directory. This file contains a list of relative paths to the ignored files.
Example
-------
The `BTCPay Server wrapper <https://github.com/Start9Labs/btcpayserver-wrapper/blob/master/configurator/src/templates/.backupignore.templates>`_ demonstrates a good use of a backupignore template.
Ultimately, ``/datadir/.backupignore`` gets populated with:
.. code::
/root/volumes/btcpayserver/start9/public
/root/volumes/btcpayserver/start9/shared

21
site/source/developer-docs/specification/compat/index.rst Normal file
View File

@@ -0,0 +1,21 @@
.. _compat:
============
Compat Image
============
With the release of EmbassyOS v0.3.0, system utility Docker images are preloaded for service packager convenience.
The ``Compat`` image is a backwards compatible Docker image that hosts EmbassyOS features used in the v0.2.x series. It was created as a convenience for service packagers to help migrate their service configurations from 0.2.x to 0.3.0.
It exposes functionality to make use of:
- ``ConfigRules`` language
- Duplicity backups
.. This is for the side navigation display
.. toctree::
:hidden:
rules

111
site/source/developer-docs/specification/compat/rules.rst Normal file
View File

@@ -0,0 +1,111 @@
.. _config_rules:
============
Config Rules
============
This file defines the configuration rules, or the rule-set that defines dependencies between config variables. In practice, config rules are for auto-configuring self dependencies. Self dependencies are internal dependencies of a service, such as if the setting of one config variable informs the option of another setting. These "dependencies" are configured as rules.
A rule is a boolean expression that we demand to be true. It is not true if the expression fails the rule parser. Set of predicates. xt
They follow the `Backus–Naur <https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_form>`_ meta-syntax for writing rules.
Rules are composed of two main concepts:
* Variables - accessor into a configuration
* Terms - either a variable or type literal (ie. a boolean term is a boolean variable, a boolean expression, or a comparison operation between numbers or strings)
Variables can be booleans, numbers, or strings, and have a different syntax depending on the type. These type annotations check your config rules against your config spec and throw an error if invalid.
- ``?`` - Casts to boolean value. If the value is not a boolean, this notes whether or not the value is null.
- ``#`` - Treat the value as a number. If it is not a number, the value will be parsed as NaN. String numbers are not currently supported.
- ``'`` - Cast the value into a string. Applies to any value except for an object or a list.
- ``!`` - Equals not.
.. note::
Config rules are processed in order.
If application does not satisfy a rule, a set of suggestions should be provided. These suggestions are in the form of the operation to preform:
- ``Set`` - set the value
- ``Push`` - add to the value (such as to a list)
- ``Delete`` - delete the value
.. code:: typescript
enum SuggestionVariant = Set | Delete | Push
interface Set {
var: String, // fully qualified path without typecast
// one of the following three variants are required
to: Option<String> // a string expression, use when tying another config value
to-value: Option<String>
to-entropy: Option<{
charset: String (eg. 'a-z,A-Z,0-9')
len: Number
}>
}
interface Delete {
src: String, // path to key - removes if in a list
}
interface Push {
to: String,
value: String, // string literal of value to be set
}
SET Examples
------------
.. code:: yaml
- SET:
# the key in config you want to set
var: 'users.[first(item => ''item.name = "c-lightning")].password'
# the value in config that you will set
to-entropy:
charset: "a-z,A-Z,0-9"
len: 22
- SET:
var: 'users.[first(item => ''item.name = "c-lightning")].fetch-blocks'
to-value: true
PUSH Examples
-------------
.. code:: yaml
- PUSH:
to: "users"
value:
name: c-lightning
allowed-calls: []
- PUSH:
to: 'users.[first(item => ''item.name = "c-lightning")].allowed-calls'
value: "getnetworkinfo"
Full example from `c-lightning manifest <https://github.com/Start9Labs/c-lightning-wrapper/blob/master/manifest.yaml>`_:
.. code:: yaml
config:
- rule: '''users.*.name = "c-lightning"'
description: 'Must have an RPC user named "c-lightning"'
suggestions:
- PUSH:
to: "users"
value:
name: c-lightning
allowed-calls: []
- SET:
var: 'users.[first(item => ''item.name = "c-lightning")].password'
to-entropy:
charset: "a-z,A-Z,0-9"
len: 22

490
site/source/developer-docs/specification/config-spec.rst Normal file
View File

@@ -0,0 +1,490 @@
.. _config_spec:
===========
Config Spec
===========
Introduction
============
Most self-hosted applications require the user to tell the application how to behave using a config file in a specific format, environment variables, command-line arguments, or some combination of these inputs. One of the coolest features of EmbassyOS is that the services' configuration will be available to the user as a slick GUI that always produces a valid configuration no matter how little experience or skill the user has.
With EmbassyOS, this means a services' configuration requires a file to define the particular format to ensure it integrates smoothly with the user interface. This format enables clean handling of improper values and dependency management.
This file defines the *structure* of the service's native config and should be curated according to the ``ConfigSpec`` type, which is a detailed mapping of the configuration options with acceptable values, defined patterns, and defaults.
For example, if the user chooses config option A, then config option B must be between 5 and 10. This enables a simple GUI configuration experience, complete with validations and protections, for users. They do not have to worry about the consequences of a wrong value or manually editing a config file.
Purpose
=======
The ``ConfigSpec`` exists primarily an input specification and secondarily for input validation.
.. figure:: /_static/images/services/service5.png
:width: 80%
:alt: Synapse Config
The file containing the ``ConfigSpec`` defines the structure of configuration options your service depends on to run. It additionally can include configuration options that users might want to enable for more advanced or customized usage. Ultimately, these values influence the UI elements for a user to interact with. Specifically, they evaluate to the options available when managing a service, such as:
- Prior to service installation when the user needs to be made aware of any necessary dependency configurations
- When the user installs a service and the service is in the "Needs Config" state
- Whenever a user edits a service config
- When config pointers get updated
The neat part about this file is that each ``ValueSpec`` type gets translated into a specific front end component. For instance, boolean values display as a toggle button, such as in the Synapse example of ``Enable Registration``.
.. figure:: /_static/images/services/synapseconfig.png
:width: 80%
:alt: Example boolean toggle
Another advantage is the ability to define default values. These values automatically get populated if the user selects the ``Default`` option when setting up a service in ``Needs Config`` state. This is incredibly convenient for users who want to get up and running quickly.
Implementation
==============
The following section contains implementation specifications for the structure of the file containing the ``ConfigSpec``. This config specification if of the type:
.. code-block::
:caption: ConfigSpec
key: ValueSpec
ValueSpec Type: Boolean | Enum | List | Number | Object | String | Union | Pointer
- All keys are ``kebab-case`` strings, which correspond to the package id
- All values are one the following specs (ie. ``ValueSpec`` type):
- :ref:`boolean <boolean>`
- :ref:`enum <enum>`
- :ref:`list <list>`
- :ref:`number <number>`
- :ref:`object <object>`
- :ref:`string <string>`
- :ref:`union <union>`
- :ref:`pointer <pointer>`
- In the examples for each value spec type below, ``Option`` means the key is optional. Otherwise, the key is required.
- Descriptions are optional, but recommended
- Name corresponds to the name of the config field
- Find a complete example :ref:`here <example_config_spec>`
- Change warning text displays when the value is altered
Specs
=====
These are the possible value types with examples for each key of the config specification.
.. _boolean:
Boolean
-------
Config value specification denoted as a boolean value. A default value is required.
.. code-block::
:caption: ValueSpec Type
type: boolean
name: String
description: Option<String>
warning: Option<String>
default: Boolean
Example
^^^^^^^^
.. code:: yaml
fetch-blocks:
type: boolean
name: Fetch Blocks
description: Fetch blocks from the network if pruned from disk
default: true
.. _enum:
Enum
----
Config value specification denoted as an enum value. Enums values must be a unique set. If no default is provided, ``null`` will be the assumed value.
.. code-block::
:caption: ValueSpec Type
type: enum
name: String
description: Option<String>
warning: Option<String>
default: Option<Enum>
values: Set<String>
Example
^^^^^^^
.. code:: yaml
theme-mode:
type: enum
name: Theme Mode
values:
- NIGHT
- DAY
valueNames:
NIGHT: Night
DAY: Day
default: NIGHT
.. _list:
List
----
The list type describes an array of values. The values must consist of the same subtype, which can be any of the ValueSpec types available in the EmbassyOS config specification.
Lists of any type do not contain the default for each item in list. The list *itself* can have a default. If no default is provided, ``null`` will be the assumed value.
Range is loosely based off mathematical range syntax, with infinity replaced with ``*``:
``[ || ]`` = inclusive
``( || )`` = noninclusive
``*`` = infinity on either end
eg:
.. code::
[0,*) - all numbers to infinity including 0
.. code-block::
:caption: ValueSpec Type
type: list
name: String
description: Option<String>
subtype: Enum<enum | number | object | string | union>
range: NumRange<unsigned integer>
spec: ValueSpec
default: ValueSpec
Example:
^^^^^^^^
.. code:: yaml
allowed-calls:
type: list
name: Allowed Calls
description: The list of all RPC methods this user is allowed to make
subtype: enum
range: "[0, *)"
spec:
type: enum
values:
- item
- item
.. _number:
Number
------
A number value within an optionally defined range. Nullable field is required. If ``nullable`` is true, the default is assumed to be ``null`` if it is not provided.
.. code-block::
:caption: ValueSpec Type
type: number
name: String
description: Option<String>
change-warning: Option<String>
default: Boolean
nullable: Boolean
range: Option<NumRange<64 bit floating point>>
integral: Boolean
units: Option<String>
Example
^^^^^^^
.. code:: yaml
type: number
name: Peer Message Timeout
description: How long to wait for a response from a peer before failing
nullable: false
integral: true
units: Seconds
range: "[0, *)"
default: 30
.. _object:
Object Type
-----------
A nested representation of a ``ConfigSpec``. The object type takes the same structure under the ``spec`` key as a ``ConfigSpec``: a key indicates the field name, and the value denotes the ``ValueSpec`` type for that field.
.. code-block::
:caption: ValueSpec Type
type: object
name: String
description: Option<String>
warning: Option<String>
display-as: Option<String>
# indicates whether duplicates can be permitted
unique-by: UniqueBy
spec: ConfigSpec
type UniqueBy = null | string | { any: UniqueBy[] } | { all: UniqueBy[] }
Example
^^^^^^^
.. code:: yaml
type: object
name: Advanced
description: Advanced settings for Bitcoin Proxy
nullable: false
spec:
tor-only:
type: boolean
name: Only Tor Peers
description: Use Tor for all peer connections
default: false
peer-timeout:
type: number
name: Peer Message Timeout
description: How long to wait for a response from a peer before failing
nullable: false
integral: true
units: Seconds
range: "[0, *)"
default: 30
max-peer-age:
type: number
name: Maximum Peer Age
description: How long to wait before refreshing the peer list
nullable: false
integral: true
units: Seconds
range: "[0, *)"
default: 300
max-peer-concurrency:
type: number
name: Maximum Peer Concurrency
description: How many peers to reach out to concurrently for block data
nullable: true
integral: true
range: "[1, *)"
default: 1
.. _string:
String
------
There are various options for string values. They can optionally be marked as copyable or masked, such as for passwords, which will reflect the UI element display. A pattern, expressed in regex, can be denoted. If it exists, this field requires both the pattern type (ie. Regex) and pattern description (ie. an explanation of the pattern requirements).
If the default type is ``Entropy``, the charset can optionally specify an inclusive ranged character set (ie. "a-f,0-9").
.. code-block::
:caption: ValueSpec Type
type: string
name: String
description: Option<String>
warning: Option<String>
masked: Option<boolean>
copyable: Option<boolean>
# Placeholder text in UI input box
placeholder: Option<String>
nullable: Boolean
default: String | Entropy
pattern: Option<Regex>
pattern-description: Option<String>
.. code-block::
:caption: Entropy Type
charset: Option<String>
len: integer
Examples
^^^^^^^^
.. code::
color:
type: string
name: Color
description: Color value for the Lightning Network
nullable: false
pattern: "[0-9a-fA-F]{6}"
pattern-description: |
Must be a valid 6 digit hexadecimal RGB value. The first two digits are red, middle two are green and final two are
blue
default:
charset: "a-f,0-9"
len: 6
password:
type: string
name: Password
description: The password for the RPC User
nullable: false
copyable: true
masked: true
default:
charset: "a-z,A-Z,0-9"
len: 22
.. _pointer:
Pointer
--------
The pointer type *points* to a config value on another service installed on EmbassyOS (ie. package subtype) or to the EmbassyOS system (ie. system subtype). When pointing to another service, the ``index`` field indicates the path to the desired config variable.
.. code-block::
:caption: ValueSpec Type
type: pointer
name: String
description: Option<String>
warning: Option<String>
subtype: Enum< package | system>
package-id: String (*always* kebab case)
target: AppPointerSpecVariants | SystemPointerSpecVariants
index: Option<String> (dependent on target being AppPointerSpecVariants)
AppPointerSpecVariants = TorAddress | TorKey | Config
SystemPointerSpecVariants = HostIp
Example
^^^^^^^
.. code::
user:
type: pointer
name: RPC Username
description: The username for the RPC user for Bitcoin Core
subtype: package
package-id: bitcoind
target: config
index: "rpc.username"
.. _union:
Union
-----
This type describes a necessary dependency. Multiple variants can be expressed to enable the user the option to connect to another service (internal dependency) or outside source (external dependency).
For example, the Bitcoin Proxy service is united with an instance of Bitcoin. Three variants are defined: internal, external, and a quick connect. In this case, internal refers to the Bitcoin Core instance running on EmbassyOS, and defines the RPC credentials necessary for connecting; external refers to a Bitcoin Core node running on a different device, and defines the RPC credentials necessary for connecting; quick connect refers to yet another method of connecting to a Bitcoin Core node, optimized for convenience.
Default is required and corresponds to one of the variants.
``Tag`` is the key that will be rendered on the UI element.
.. code-block::
:caption: ValueSpec Type
type: union
name: String
description: Option<String>
change-warning: Option<String>
default: Boolean
tag: Tag
variants: Map<String, ConfigSpec>
display-as: Option<String>
unique-by: any | all | exactly | notUnique
.. code-block::
:caption: Tag Type
id: String
name: String
description: Option<String>
variant-names: Map<String, String>
.. _example_config_spec:
Example
^^^^^^^
.. code:: yaml
bitcoind:
type: union
name: Bitcoin Core
description: The Bitcoin Core node to connect to
tag:
id: type
name: Type
description: |
- Internal: The Bitcoin Core service installed to your Embassy
- External: A Bitcoin Core node running on a different device
- Quick Connect: A Quick Connect URL for an unpruned Bitcoin Core node
variant-names:
internal: Internal
external: External
quick-connect: Quick Connect
default: internal
variants:
internal:
user:
type: pointer
name: RPC Username
description: The username for the RPC user for Bitcoin Core
subtype: app
app-id: bitcoind
target: config
index: "rpc.username"
password:
type: pointer
name: RPC Password
description: The password for the RPC user for Bitcoin Core
subtype: app
app-id: bitcoind
target: config
index: "rpc.password"
external:
addressext:
type: string
name: Public Address
description: The public address of your Bitcoin Core RPC server
nullable: false
userext:
type: string
name: RPC Username
description: The username for the RPC user on your Bitcoin Core RPC server
nullable: false
passwordext:
type: string
name: RPC Password
description: The password for the RPC user on your Bitcoin Core RPC server
nullable: false
quick-connect:
quick-connect-url:
type: string
name: Quick Connect URL
description: The Quick Connect URL for your Bitcoin Core RPC server
nullable: false
pattern: 'btcstandup://[^:]*:[^@]*@[a-zA-Z0-9.-]+:[0-9]+(/(\?(label=.+)?)?)?'
patternDescription: Must be a valid Quick Connect URL. For help, check out https://github.com/BlockchainCommons/Gordian/blob/master/Docs/Quick-Connect-API.md

142
site/source/developer-docs/specification/dependencies.rst Normal file
View File

@@ -0,0 +1,142 @@
.. _dependencies-spec:
============
Dependencies
============
Background
----------
When it comes to running your own server, managing dependencies is perhaps the most difficult part. The term "dependency hell" emerged from this sentiment. Even the best developers have lost significant amounts of time trying to find, correct, or clarify documentation for dependencies or dependency configuration. Individuals who manage their own servers have specific technical skills and are willing to devote the time and effort necessary to maintain them. Companies have entire departments dedicated to this task.
Some other personal server products aimed at non-technical individuals tackle this problem by simply hard coding dependencies. Since EmbassyOS is a platform for running all open-source, self-hosted software, it is not possible to hard code every possible service, service dependency, and service dependency configuration forever. Instead, Start9 built a new, unprecedented operating system that touts a generalized, intuitive approach to dependency management and service configuration. EmbassyOS enables users to easily and selectively install, uninstall, and update any service they want without getting stuck in dependency hell.
This may sound cool or neat, but it is more than that: *it's novel*. This has never been done before.
The key to making the system work is a new, domain-specific-language (DSL) and set of standards that are used by developers to define the rules and requirements of their services. Run in the context of EmbassyOS, these rules and requirements appear as simple UI elements, such as inputs, toggles, and drop downs, and they are enforced by validations and clear user instructions. Using this system, what previously required serious time and expertise, can now be done by anyone in a matter of seconds.
This DSL is utilized in the :ref:`config rules <config_rules>` and :ref:`dependencies <service-dependencies>` key in the service manifest.
Context
-------
Many services depend on other libraries and services on EmbassyOS (such as Bitcoin), sometimes even a particular version of those services. These need to be specified by the developers so that EmbassyOS can handle installing these dependencies under the hood.
Requirement
-----------
Dependencies are defined in the Manifest.
The key of each field in the dependencies object is the lowercase, kebab-case package id of the service that is depended on.
Only the type of requirement must be defined. Requirement types include:
- "Required" - which means it's always required.
- "Opt-in" - which means the dependency may be required if you change the service's config ie. you can opt into the selection
- "Opt-out" - which means the dependency will be required according to the default service config ie. you can opt out of the selection
Advanced Configuration
----------------------
If a dependency requires a more advanced configuration, rule checks and auto configurations can be implemented. These functions utilize a preloaded Docker image within EmbassyOS entitled _compat_. This system utility provides the mechanisms for checking dependency configuration compatibility by providing a file with the defined rules as input.
In this case, each dependency contains a set of rules that need to be fulfilled as true if the dependency is to be properly installed. The "config rules" here are for auto-configuring dependencies - the action defined by the rule will be executed if the service is auto configured with defaults during initial setup. This simplifies and streamlines the user experience. The interface should provide suggestions for the behavior if the denoted rule cannot be met with previous configurations.
Example
-------
Here is an example from BTCPay Server.
First, an except from the dependencies section of the Manifest. We see here that ``bitcoind`` is a dependency of BTCPay Server. It is required to be installed at at least version 0.21.1.2.
There also exists a rule that states the ``advanced.peers.listen`` config option in bitcoind must be equal to ``true``. If this value is not set as such, ``SET`` the value to true.
These actions all take place during the initial service configuration, before the service is started for the first time.
.. code:: yaml
...
dependencies:
# Key must be the package id of another service that exists in the marketplace
bitcoind:
# The version range that is acceptable for this dependency
version: "^0.21.1.2"
# Describes if the dependency is critical to the service functioning. If the dependency is critical, the service will stop if this dependency is stopped.
critical: false
requirement:
# "Opt-out" means the dependency will be required according to the default config. "Opt-in" means the dependency may be required if you change the config. And "required" just means it's always required.
type: "opt-in"
# An explanation of how to opt-in or opt-out. This value is optional for type=required
how: Can alternatively configure an external bitcoind node.
# Description of the necessity of the dependency relationship
description: Used to fetch validated blocks.
# This defines the actions to check dependency rules and auto configure them if possible
config:
# This levies requirements on the configuration of the dependency and suggests ways to remedy any incompatibilities.
check:
type: docker
image: compat
system: true
entrypoint: compat
args:
- dependency
- check
- btcpayserver
- "bitcoind"
- /datadir
- "/mnt/assets/bitcoind_config_rules.yaml"
mounts:
main: /datadir
compat: /mnt/assets
io-format: yaml
# This implements default values on the configuration of the dependency
auto-configure:
type: docker
image: compat
system: true
entrypoint: compat
args:
- dependency
- "auto-configure"
- btcpayserver
- "bitcoind"
- /datadir
- "/mnt/assets/bitcoind_config_rules.yaml"
mounts:
main: /datadir
compat: /mnt/assets
io-format: yaml
...
Secondly, an except from it's dependency's configuration rules file:
.. code:: yaml
- rule: "advanced.peers.listen?"
description: Peer port must be listening on the network.
suggestions:
- SET:
var: advanced.peers.listen
to-value: true
.. note::
Dependency config rules are processed in order.
Optional Dependencies
---------------------
Configuring a service to have multiple optional dependencies is also possible. This can be done by defining a file that specifies the condition under which the dependency should become activated.
For example, in BTCPay Server's config, a user can select either LND or C-lightning as an internal lightning node implementation. To do this, we define the following file:
.. code:: yaml
lnd:
condition: '''lightning.type = "lnd"'
health_checks: []
c-lightning:
condition: '''lightning.type = "c-lightning"'
health_checks: []
This file gets passed in as an argument during BTCPay Server's config set function in its Manifest, shown `here <https://github.com/Start9Labs/btcpayserver-wrapper/blob/master/manifest.yaml#L86>`__.

107
site/source/developer-docs/specification/docker.rst Normal file
View File

@@ -0,0 +1,107 @@
.. _service_docker:
==========
Dockerfile
==========
This purpose of the Dockerfile is to define how to build the image for the service. It declares the environment and building stages.
The Dockerfile is responsible for mounting the service application to whatever volume the manifest specifies, typically ``/root/.<service-id>``.
Since the build requires specific arm runtime environments, these base images can be imported into the Dockerfile as a starting point so the base system does not need to be completely redefined. This enables importing a specific build environment version that encapsulates any necessary libraries for running the service build, eg. golang, rust.
For instance:
``FROM alpine:3.12``
``FROM arm32v7/alpine``
``FROM arm32v7/golang:alpine``
We recommended using ``alpine`` since it produces the smallest image. We try to keep the image under 100MB when possible.
Image
-----
Each service boils down to a Docker image. We're not going to dive into Docker specifics in this guide, since there exists tons of `resources <https://docs.docker.com/>`__ for developing with this containerization tool.
The image is immutable, meaning that when the service is updated, the image is replaced with a completely new image containing the updated features.
When installed, the service image is mounted to the EmbassyOS image. This path is defined in the :ref:`manifest <service_manifest>` configuration file:
.. code:: yaml
...
main:
type: docker
image: main
entrypoint: "docker_entrypoint.sh"
args: []
mounts:
# Specified where to mount the volume in the container
main: /root
io-format: yaml
...
In EmbassyOS, the mount specified in the Manifest gets bound to ``/embassy-data/package-data/volumes/main/<service-id>``.
Volumes
-------
Volumes are the mechanism for persisting data generated by and used by Docker containers. In EmbassyOS, services can declare as one or many volumes in the Manifest:
.. code:: yaml
volumes:
# This key correlates to the image name specified above. The default is "main".
main:
type: data
Volume Types
^^^^^^^^^^^^
1. ``data`` - A volume to persist application data. All service application data is stored in this directory and is persisted across updates.
2. ``assets`` - A volume to store miscellaneous assets that do not need to be in the main image.
If an asset volume is specified, there must exist a folder entitled assets, and a subfolder entitled the name of the asset volume:
.. code-block::
:emphazize-lines: 9,10,11,16,17
properties:
type: docker
image: compat
system: true
entrypoint: compat
args:
- properties
- /root
mounts:
main: /root
compat: /mnt/assets
volumes:
# This is the image where files from the project asset directory will go
main:
type: data
compat:
type: assets
# required project folder structure
├── assets
│ └── compat
.. warning:: Any files in the mounted volume path will be overwritten when a backup restore occurs.
Entrypoint
----------
The ``docker_entrypoint.sh`` defines what to do when the service application starts.
It consists of a bash script that completes any environment setup (eg. creating folder substructure), sets any environment variables, and executes the run command. The only required feature of this file is to execute the run command for the service.
Example
-------
The `LND wrapper <https://github.com/Start9Labs/lnd-wrapper/blob/master/Dockerfile>`_ features a well defined Dockerfile, for example.

113
site/source/developer-docs/specification/index.rst Normal file
View File

@@ -0,0 +1,113 @@
.. _service-packaging-spec:
=============
Specification
=============
The following guides provide an in depth overview of the full capabilities available for packaging a service.
.. raw:: html
<div class="topics-grid grid-container full">
<div class="grid-x grid-margin-x">
.. topic-box::
:title: Docker
:link: docker
:class: large-4
:anchor: View
Learn how to setup the main Docker image for your service.
.. topic-box::
:title: Manifest
:link: manifest
:class: large-4
:anchor: Begin
Understand the function of a Manifest file and its type.
.. topic-box::
:title: Config Spec
:link: config-spec
:class: large-4
:anchor: Begin
Learn the purpose and utility of a config specification.
.. topic-box::
:title: Properties
:link: properties
:class: large-4
:anchor: View
Understand the purpose and requirements of service properties.
.. topic-box::
:title: Dependencies
:link: dependencies
:class: large-4
:anchor: View
Learn how to configure dependency options.
.. topic-box::
:title: Backups
:link: advanced
:class: large-4
:anchor: View
Learn how to configure backup options.
.. topic-box::
:title: Instructions
:link: instructions
:class: large-4
:anchor: View
Understand how an instructions file is relevant to a service.
.. topic-box::
:title: Package
:link: package
:class: large-4
:anchor: View
Learn how to package service components into a single file format.
.. topic-box::
:title: Wrapper
:link: wrapper
:class: large-4
:anchor: View
Understand the purpose of a wrapper repository.
.. topic-box::
:title: Submission
:link: submission
:class: large-4
:anchor: View
Learn about how to submit a package for review to a marketplace.
.. raw:: html
</div></div>
.. toctree::
:hidden:
docker
manifest
config-spec
properties
dependencies
backups
instructions
package
wrapper
makefile
submission
compat/index

61
site/source/developer-docs/specification/instructions.rst Normal file
View File

@@ -0,0 +1,61 @@
.. _service_instructions:
============
Instructions
============
Within each wrapper repository, a file should exist that includes instructions for a service. This file should include any pertinent documentation, instructions, external integrations, or other details about the service that users or developers might find relevant.
This markdown formatted file will be rendered in the "Instructions" section within the service page on EmbassyOS:
.. figure:: /_static/images/services/service-instruct.png
:width: 80%
:alt: Embassy Pages Instructions
Below is the markdown file for Embassy Pages' instructions, as shown above:
.. code-block::
# Embassy Pages
## About
Embassy Pages is a simple web server that uses directories inside File Browser to serve Tor websites. Your website could be a blog, a resume, a portfolio, a business landing page, a product brochure, or just a set of static folders and/or files that you want to share with the world.
When you first install Embassy Pages, there will be a default Homepage hosted at the root, <tor-address>.onion. You can change the behavior of this page, and you can also create Subdomain websites. For example, one site could be hello.<tor-address>.onion and another could be goodbye.<tor-address>.onion. What is served from the Homepage and each Subdomain is totally up to you.
Self-hosting Tor websites using Embassy Pages is easy, permissionless, and censorship-resistant; there are no trusted third parties involved.
Anyone can do it. No one can stop it.
## Instructions
1. Inside `Config`, you can easily change the behavior of your Homepage to serve:
a. A list of hyperlinks to all your Subdomains
b. A personal web page
c. An automatic redirect to a Subdomain
d. A static web page that tells anyone visiting it to fuck off, politely.
2. Inside `Config`, you can create one or more Subdomains, giving each a unique name.
3. To serve a personal website, simply upload the website directory to File Browser. Then, inside the settings for a particular page (either your Homepage or a Subdomain), enter the path to that directory. For example, a path of websites/blog would tell Embassy Pages that it can find the blog website inside the websites directory in File Browser.
4. A list of all your Subdomains can be found inside the Properties section of your Embassy Pages service.
## Restore from Backup
1. In `Config`, select "Restore from Backup".
2. After the restoration completes, navigate to the Marketplace.
3. Downgrade Embassy Pages.
4. Upgrade Embassy Pages to the latest version.
5. Start Embassy Pages.
6. Embassy Pages is now successfully restored!
Example
-------
The `bitcoind wrapper <https://github.com/Start9Labs/bitcoind-wrapper/tree/master/docs>`_ demonstrates a good use of instructions and external integrations.
.. role:: raw-html(raw)
:format: html
:raw-html:`<br />`

28
site/source/developer-docs/specification/makefile.rst Normal file
View File

@@ -0,0 +1,28 @@
.. _service_makefile:
========
Makefile
========
.. note:: *This file is optional*
A Makefile serves as a convenience for outlining the build. This helps streamline additional developer contributions to the project. Please reference the GNU `documentation <https://www.gnu.org/software/make/manual/html_node/Introduction.html>`_ for implementation details.
An alternative to using ``make`` is to use the `nix <https://nixos.wiki/wiki/Nix>`_ specification.
This purpose of this file is to:
- Read the docker container and build the project
- Build all prerequisites for running the docker file
- Build all dependencies
- Package assets and docker image using ``embassy-sdk pack``
Example
-------
The `LND wrapper <https://github.com/Start9Labs/lnd-wrapper/blob/master/Makefile>`_ features a well defined Makefile, for example.
.. role:: raw-html(raw)
:format: html
:raw-html:`<br />`

299
site/source/developer-docs/specification/manifest.rst Normal file
View File

@@ -0,0 +1,299 @@
.. _service_manifest:
========
Manifest
========
Overview
--------
This file describes the service and it's requirements. It is used to:
- create a listing in the marketplace
- denote any installation considerations
- define dependency requirements
- define actions such as health checks, backups, and configuration
- define alerts and other messaging / descriptions for the user interface
Each time a service is updated, the Manifest should be updated to include the new version, release notes, and any pertinent updates to the install, uninstall, or restoration flows. All this information is displayed in the marketplace listing, and the optionally denoted alerts will be displayed when appropriate to provide the user with more information.
There is nothing you need to do as a developer to set up Tor for running a service. This is *completely* handled by EmbassyOS - a Tor address will be automatically generated when the service is installed. Just define an interface with a tor config in the Manifest file. You do, however, need to ensure the service is in fact capable of running over Tor.
The Manifest is also responsible for outlining service :ref:`dependencies <dependencies-spec>`. By defining rules using the :ref:`EmbassyOS DSL specification <config_rules>`, users can easily and selectively install, uninstall, and update any service without getting stuck in dependency hell. EmbassyOS presents this information in a polished install/uninstall/update wizard, so there's no need for editing configuration files or jumping into the command line. For you as a developer, this simply means populating this key in the manifest!
Formatting
----------
- Serialization language:``yaml | toml | json``
- Case style: ``kebab-case``
Type definitions
----------------
Below are the types and sub-type definitions, with necessary elaborations. Any item that contains ``Option<>`` is an optional field.
.. code:: yaml
# The package identifier used by the OS
id: String
# A human readable service title
title: String
# Service version - accepts up to four digits, where the last confirms to revisions necessary for EmbassyOS - see documentation: https://github.com/Start9Labs/emver-rs. This value will change with each release of the service.
version: Number
# Release notes for the update - can be a string, paragraph or URL
release-notes: String
# The type of license for the project. Include the LICENSE in the root of the project directory. A license is required for a Start9 package.
license: String
# The Start9 wrapper repository URL for the package. This repo contains the manifest file (this), any scripts necessary for configuration, backups, actions, or health checks (more below). This key must exist. But could be embedded into the source repository.
wrapper-repo: String
# The original project repository URL. There is no upstream repo in this example
upstream-repo: String
# URL to the support site / channel for the project. This key can be omitted if none exists, or it can link to the original project repository issues.
support-site: String
# URL to the marketing site for the project. This key can be omitted if none exists, or it can link to the original project repository.
marketing-site: String
# The series of commands to build the project into an s9pk for arm64/v8. In this case we are using a Makefile with the simple build command "make".
build: List<String>
# Minimum required version of EmbassyOS
min-os-version: Number
# Human readable descriptors for the service. These are used throughout the EmbassyOS user interface, primarily in the marketplace.
description:
# This is the first description visible to the user in the marketplace.
short: String
# This description will display with additional details in the service's individual marketplace page
long: String
# These assets are static files necessary for packaging the service for Start9 (into an s9pk). Each value is a path to the specified asset. If an asset is missing from this list, or otherwise denoted, it will be defaulted to the values denoted below.
assets:
# Default = LICENSE.md
license: String
# Default = icon.png
icon: String
# Default = INSTRUCTIONS.md
instructions: String
# Default = image.tar
docker-images: String
# The main action for initializing the service. Currently, the only type of action available is docker.
main:
# Docker is currently the only action implementation
type: String
# Identifier for the main image volume, which will be used when other actions need to mount to this volume.
image: String
# The executable binary for starting the initialization action. For docker actions, this is typically a "docker_entrypoint.sh" file. See the Dockerfile and the docker_entrypoint.sh in this project for additional details.
entrypoint: String
# Any arguments that should be passed into the entrypoint executable
args: List<String>
# Specifies where to mount the data volume(s), if there are any. Mounts for pointer dependency volumes are also denoted here. These are necessary if data needs to be read from / written to these volumes.
mounts:
# Specifies where on the service's file system its persistence directory should be mounted prior to service startup
main: String
# Health checks
health-checks:
main:
name: String
description: String
type: String
image: String
entrypoint: String
args: List<String>
# When `inject` is true, the health check will use the main image to run the health check. This is faster as there is no need to spin up an additional docker container
# When `inject` is false, the health check will use whatever image is specified. This is useful when using a system image with additional utilities to run a health check. If inject=false, then system must equal true
inject: Boolean
# Optional if false - indicates if an image that is preloaded onto the system will be used
system: Boolean
# Required - valid values are yaml, toml, json
io-format: Enum<json|yaml|toml>
# Specifies how to get and set configuration file values for the service. NOTE: This stanza can be left empty (null) if the service has no configuration options.
config:
# The config action to run to get the specified config file (default is config.yaml)
get:
# The type of implementation to run this action (currently, only Docker is available)
type: String
# The Docker image to run the action command in. This could be the service's main image, or an image that is preloaded onto the system, like compat (which holds compatible helper functions for default functionality)
image: String
# Indicates if an image that is preloaded onto the system will be used
system: Boolean
# The initial run command to execute the config get action
entrypoint: String
# Any arguments that need to be passed into the run command
args: List<String>
# The locations at which to mount the specified Docker images
mounts:
compat: String
main: String
# Required - valid values are yaml, toml, json
io-format: Enum<yaml|json|toml>
# The config action to run to set the specified config file (default is config.yaml). Details for the keys below are the same as above.
set:
type: String
image: String
system: Boolean
entrypoint: String
args: List<String>
mounts:
compat: String
main: String
io-format: Enum<yaml|json|toml>
# This is a key value map specifying dependent services that this service needs in order to function. The keys are the package id's on which you depend. NOTE: if developing a standalone service, you may leave this stanza as an empty object (the key dependencies is required)
dependencies:
# Key must be the package id of another service that exists in the marketplace
filebrowser:
# The version range that is acceptable for this dependency
version: Emver
# Describes if the dependency is critical to the service functioning. If the dependency is critical, the service will stop if this dependency is stopped.
critical: Boolean
# Specifies the requirement type of the dependency
requirement:
# "Opt-out" means the dependency will be required according to the default config. "Opt-in" means the dependency may be required if you change the config. And "required" just means it's always required.
type: Enum<opt-in|opt-out|required>
# An explanation of how to opt-in or opt-out. This value is optional for type=required
how: String
# Description of the dependency relationship
description: String
# This is a list of rules that levies requirements on the configuration of the dependency and suggests ways to remedy any incompatibilities. Documentation of this feature is outside the scope of this example.
config: ~
# This denotes any data, asset, or pointer volumes that should be connected when the "docker run" command is invoked
volumes:
# This is the image where files from the project asset directory will go
main:
type: Enum<data|asset>
# This is an example of an asset volume
compat:
type: Enum<data|asset>
# This specifies how to configure the port mapping for exposing the service over TOR and LAN (if applicable). Many interfaces can be specified depending on the needs of the service. If it can be launched over a Local Area Network connection, specify a `lan-config`. Otherwise, at minimum, a `tor-config` must be specified.
interfaces:
# This key is the internal name that the OS will use to configure the interface
main:
# A human readable name for display in the UI
name: String
# A descriptive description of what the interface does
description: String
tor-config:
# Port mappings are from the external port to the internal container port
port-mapping:
80: String
# Port mappings are from the external port to the internal container port
lan-config:
80:
ssl: Boolean
internal: Number
# Denotes if the service has a user interface to display
ui: Boolean
# Denotes the protocol specifications used by this interface
protocols: List<String>
# Alerts: omitting these will result in using the default alerts in EmbassyOS, except for start, which has no default.
alerts:
install-alert: String
uninstall-alert: String
restore-alert: String
start-alert: String
# Specifies how backups should be run for this service. The default EmbassyOS provided option is to use the duplicity backup library on a system image (compat)
backup:
create:
type: String
image: String
system: Boolean
entrypoint: String
# Arguments to pass into the entrypoint.
args: List<String>
mounts:
# BACKUP is the default volume that is used for backups. This is whatever backup drive is mounted to the device, or a network filesystem.
# The value here donates where the mount point will be. The backup drive is mounted to this location.
BACKUP: String
main: String
restore:
type: String
image: String
system: Boolean
entrypoint: String
args: List<String>
mounts:
BACKUP: String
main: String
# Commands that can be issued from the UI. NOTE: if no actions are required, this section can be left as an empty object
actions:
hello-world-action:
name: String
description: String
warning: Option<String>
# Indicates what state the service can be in while executing the action
allowed-statuses: List<String>
# Defines how the action is run
implementation:
type: String
image: String
entrypoint: String
args: [List<String>
# Same as note on health-checks
inject: Boolean
# Required - valid values are yaml, toml, json
io-format: Enum<yaml|json|toml>
.. code:: typescript
interface Dependencies [{
serviceId: DepInfo
}]
interface DepInfo {
version: VersionRange // ie. ^0.11.1.1
requirement: Enum<"opt-in"|"opt-out"|"required">,
description?: String,
config: Optional<{
check: ActionImplementation,
auto-configure: ActionImplementation,
}>,
}
interface ActionImplementation {
image: String,
system: Bool,
entrypoint: String,
args: List<String>,
mounts: Map<VolumeId, Path>,
io_format: Option<Enum<json|yaml|toml>>,
inject: Bool,
}
interface ConfigRule {
rule: String, // ie. 'users.*.name = "lnd"
description: String,
suggestions: [SuggestionVariant]
}
interface SuggestionVariant {
SET: {
var: String,
to: SetVariant,
},
DELETE: {
src: String,
},
PUSH: {
to: String,
value: Value,
},
}
interface SetVariant {
to: Option<String>,
to-value: Option<Value>, // ie. true/false
to-entropy: Option<{
charset: String // ie. 'a-z,A-Z,0-9'
len: number
}>
}
.. role:: raw-html(raw)
:format: html
:raw-html:`<br />`
Examples
========
- `Hello World <https://github.com/Start9Labs/hello-world-wrapper/blob/master/manifest.yaml>`__
- `Filebrowser <https://github.com/Start9Labs/filebrowser-wrapper/blob/master/manifest.yaml>`__
- `Embassy-pages <https://github.com/Start9Labs/embassy-pages-wrapper/blob/master/manifest.toml>`__
- `Photoview <https://github.com/Start9Labs/embassyos-photoview-wrapper/blob/master/manifest.yaml>`__
- `BTCPay Server <https://github.com/Start9Labs/btcpayserver-wrapper/blob/master/manifest.yaml>`__
- `Synapse <https://github.com/Start9Labs/synapse-wrapper/blob/master/manifest.yaml>`__

33
site/source/developer-docs/specification/package.rst Normal file
View File

@@ -0,0 +1,33 @@
.. _package:
==============
Package Bundle
==============
All assets get bundled into a custom filetype with an extension of ``.s9pk``, short for Start9 Package. This is the file that will be downloaded from a marketplace. When the user clicks a service's "Install" button, EmbassyOS unpacks and installs the service.
The minimum necessary files for this bundle are:
- image.tar (the result of a Dockerfile build)
- instructions.md
- LICENSE
- icon.png
Each new version release of a service should include the updated version of these files re-bundled into a new ``.s9pk``. .
Building
--------
To package all components into an ``.s9pk``, run the following command from the root of your project directory:
.. code:: bash
embassy-sdk pack
Let's also make sure to verify the validity of the package:
.. code:: bash
embassy-sdk verify s9pk /path/to/hello-world.s9pk
If anything goes wrong, an error message will indicate the missing component or other failure.

46
site/source/developer-docs/specification/properties.rst Normal file
View File

@@ -0,0 +1,46 @@
.. _service_properties:
==========
Properties
==========
The output of this step is a file titled ``stats.yaml``. This file contains a mapping of the values that will be displayed in the ``Properties`` section in a service's menu.
Due to the fact that config variables are only available when the service is running, this file can only be populated at runtime.
.. figure:: /_static/images/services/service-properties.png
:width: 80%
:alt: Service Properties
Service Properties
.. code:: typescript
:caption: Properties Type
interface PropertiesString {
type: 'string'
name: string
value: string
description: string | null
copyable: boolean
qr: boolean
masked: boolean
}
interface PropertiesObject {
type: 'object'
name: string
value: PropertiesObject | PropertiesString
}
Example
-------
A good example of the configurator producing the ``stats.yaml`` file can be found `here <https://github.com/Start9Labs/lnd-wrapper/blob/master/configurator/src/main.rs>`_.
.. role:: raw-html(raw)
:format: html
:raw-html:`<br />`

7
site/source/developer-docs/specification/submission.rst Normal file
View File

@@ -0,0 +1,7 @@
.. _submission:
==================
Submission Process
==================
When you have built and tested your project for EmbassyOS, please send Start9 a submission with the project repository to ``dev@start9labs.com``. After being reviewed for security and compatibility, the service will be deployed to the marketplace and available for all EmbassyOS users to download.

50
site/source/developer-docs/specification/wrapper.rst Normal file
View File

@@ -0,0 +1,50 @@
.. _service_wrapper:
=======
Wrapper
=======
Each service is bound with a wrapper repository, which contains everything you need to build a service.
The purpose of this repo is:
- To define the necessary Manifest and configuration options (ie. config spec and rules)
- To build the project into the ``.s9pk`` format digestible to EmbassyOS
- Link to the source project as a git submodule
- Define the Dockerfile for running the project on EmbassyOS
- Provide documentation for the project, especially user runbook instructions
File Structure
--------------
The project structure should be used as a model:
.. code-block:: bash
├── Dockerfile
├── LICENSE
├── Makefile
├── README.md
├── assets
│ └── compat
│ ├── config_rules.yaml
│ └── config_spec.yaml
├── docker_entrypoint.sh
├── <submodule-project>
├── <package-id>>.s9pk
├── icon.png
├── image.tar
├── instructions.md
└── manifest.yaml
Submodule
---------
`Git submodules <https://www.git-scm.com/book/en/v2/Git-Tools-Submodules>`__ allow use of another project while in the working project directory. Setting up this feature enables linking of the source service repository so that its context is available. The submodule is added into the wrapper so that the wrapper can build the submodule and also track the exact code that's being built.
Run:
.. code:: bash
git submodule add <link_to_source_project>