Start9/documentation
2
0
Fork 0
mirror of https://github.com/Start9Labs/documentation.git synced 2026-03-26 10:21:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

More scaffolding

Browse Source
This commit is contained in:
kn0wmad
2021-12-05 23:29:52 -07:00
committed by Lucy Cifferello
parent 4dc72ad620
commit 03d33ca4ec
130 changed files with 4892 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

0
source/dev-docs/dev-faq/faq-contributing.rst Normal file
View File

0
source/dev-docs/dev-faq/faq-service-packaging.rst Normal file
View File

0
source/dev-docs/example.rst Normal file
View File

12
source/dev-docs/index.rst Normal file
View File

@@ -0,0 +1,12 @@
***********************
Developer Documentation
***********************
An overview of EmbassyOS general capabilities.
.. toctree::
:maxdepth: 2
service-packaging/index
example
dev-faq/index

29
source/dev-docs/service-packaging/backups.rst Normal file
View File

@@ -0,0 +1,29 @@
.. _service_backups:
***************
Service Backups
***************
Everything within the root of the mounted volume directory will be stored in an EmbassyOS backup. This includes the config (config.yaml) and properties (stats.yaml) files, as well as any other persisted data within the volume directory.
For restoration purposes, it might be beneficial to ignore certain files or folders. For instance, ignore the shared/public folder that is mounted for dependencies that expose this feature as it causes data inconsistencies on restore.
In this case, create a `.backupignore` file. This file contains a list of relative paths to the ignored files.
Example
=======
The `btcpayserver wrapper <https://github.com/Start9Labs/btcpayserver-wrapper/blob/master/configurator/src/templates/.backupignore.templates>`_ demonstrates a good use of a backupignore template.
Ultimately, ``/datadir/.backupignore`` gets populated with:
.. code::
/root/volumes/btcpayserver/start9/public
/root/volumes/btcpayserver/start9/shared
.. role:: raw-html(raw)
:format: html
:raw-html:`<br />`

617
source/dev-docs/service-packaging/config.rst Normal file
View File

@@ -0,0 +1,617 @@
.. _service_config:
****************************
Service Config Specification
****************************
Overview
========
Most self-hosted applications require the user to tell the app how to behave using a config file in a specific format, environment variables, command-line arguments, or some combination of these inputs. One of the coolest features of EmbassyOS is that, when packaged correctly, the app's configuration will be available to the user as a slick GUI that always produces a valid configuration no matter how little experience or skill the user has.
With EmbassyOS, this means a service wrappers' configuration requires a particular format and rule structure to ensure it integrates smoothly with the user interface. This format enables clean handling of improper values and dependency management.
The outcome of this step is two files:
:ref:`config_spec.yaml <config_spec>`
:ref:`config_rules.yaml <config_rules>`
These files contain a detailed mapping of configuration options with acceptable values, defaults, and relational rule-sets.
For example, if the user chooses config option A, then config option B must be between 5 and 10. This enables a simple GUI configuration experience, complete with validations and protections, for users. They do not have to worry about the consequences of a wrong value in a ``.conf`` file.
.. _config_spec:
Config Spec
===========
Overview
--------
.. figure:: /_static/images/service/bitcoin_config.png
:width: 80%
:alt: Bitcoin Config
This file defines the structure of configuration options your service depends on to run. It additionally can include configuration options that users might want to enable for more advanced or customized usage. Ultimately, these values influence the UI elements for a user to interact with. Specifically, they evaluate to the options available when managing a service, such as:
- Prior to service installation when the user needs to be made aware of any necessary dependency configurations
- When the user installs a service and the service is in the "Needs Config" state
- Whenever a user edits a service config
- When config pointers get updated
The neat part about this file is that each ValueSpec type gets translated into a specific front end component. For instance, boolean values display as a toggle button.
.. figure:: /_static/images/service/boolean_toggle.png
:width: 80%
:alt: Example boolean toggle
Another advantage is the ability to define default values. These values automatically get populated if the user selects the ``Default`` option when setting up a service in ``Needs Config`` state. This is super convenient for users who want to get up and running quickly.
Types
-----
ConfigSpec Type:
.. code::
key: ValueSpec
ValueSpec Type: Boolean | Enum | List | Number | Object | String | Union | Pointer (see below for details)
Implementation Guide
--------------------
The following section contains implementation specifications for the ``config_spec.yaml`` file.
- All keys are ``kebab-case`` strings, which correspond to the service (app) id
- All values are one the following specs (ie. ``ValueSpec`` type):
:ref:`boolean <boolean>`
:ref:`enum <enum>`
:ref:`list <list>`
:ref:`number <number>`
:ref:`object <object>`
:ref:`string <string>`
:ref:`union <union>`
:ref:`pointer <pointer>`
- In the examples for each value spec type below, ``Option`` means the key is optional. Otherwise, the key is required.
- Descriptions are optional, but recommended
- Name corresponds to the name of the config field
- Find a complete example :ref:`here <example_config_spec>`
- Change warning text displays when the value is altered
.. _boolean:
Boolean
^^^^^^^
Config value specification denoted as a boolean value. A default value is required.
``ValueSpec`` Type:
.. code::
type: boolean
name: String
description: Option<String>
changeWarning: Option<String>
default: Boolean
Example:
.. code:: yaml
fetch-blocks:
type: boolean
name: Fetch Blocks
description: Fetch blocks from the network if pruned from disk
default: true
.. _enum:
Enum
^^^^
Config value specification denoted as an enum value. Enums values must be a unique set. If no default is provided, ``null`` will be the assumed value.
ValueSpec Type:
.. code::
type: enum
name: String
description: Option<String>
changeWarning: Option<String>
default: Option<Enum>
values: Set<String>
.. code:: yaml
theme-mode:
type: enum
name: Theme Mode
values:
- NIGHT
- DAY
valueNames:
NIGHT: Night
DAY: Day
default: NIGHT
.. _list:
List
^^^^
The list type describes an array of values. The values must consist of the same subtype, which can be any of the ValueSpec types available in the EmbassyOS config specification.
Lists of any type do not contain the default for each item in list. The list *itself* can have a default. If no default is provided, ``null`` will be the assumed value.
Range is loosely based off mathematical range syntax, with infinity replaced with ``*``:
``[ || ]`` = inclusive
``( || )`` = noninclusive
``*`` = infinity on either end
eg:
.. code::
[0,*) - all numbers to infinity including 0
ValueSpec Type:
.. code::
type: list
name: String
description: Option<String>
subtype: enum || number || object || string || union
range: NumRange<unsigned integer>
spec: ValueSpec
default: ValueSpec
Example:
.. code:: yaml
allowed-calls:
type: list
name: Allowed Calls
description: The list of all RPC methods this user is allowed to make
subtype: enum
range: "[0, *)"
spec:
type: enum
values:
- item
- item
.. _number:
Number
^^^^^^
A number value within an optionally defined range. Nullable field is required. If ``nullable`` is true, the default is assumed to be ``null`` if it is not provided.
ValueSpec Type:
.. code::
type: number
name: String
description: Option<String>
changeWarning: Option<String>
default: Boolean
nullable: Boolean
range: Option<NumRange<64 bit floating point>>
integral: Boolean
units: Option<String>
Example:
.. code:: yaml
type: number
name: Peer Message Timeout
description: How long to wait for a response from a peer before failing
nullable: false
integral: true
units: Seconds
range: "[0, *)"
default: 30
.. _object:
Object Type
^^^^^^^^^^^
A nested representation of a ConfigSpec. The object type takes the same structure under the ``spec`` key as a ConfigSpec: a key indicates the field name, and the value denotes the ValueSpec type for that field.
There is no default option for the object type. Rather, the option ``null-by-default`` should be used to indicate the default as ``null``. If null by default is true, nullable must be true. If null by default is false, nullable could be either.
``unique-by`` indicates whether duplicates can be permitted in the list.
ValueSpec Type:
.. code::
type: object
name: String
description: Option<String>
changeWarning: Option<String>
nullable: Boolean
null-by-default: Boolean
display-as: Option<String>
unique-by: UniqueBy
spec: ConfigSpec
type UniqueBy = null | string | { any: UniqueBy[] } | { all: UniqueBy[] }
Example:
.. code:: yaml
type: object
name: Advanced
description: Advanced settings for Bitcoin Proxy
nullable: false
spec:
tor-only:
type: boolean
name: Only Tor Peers
description: Use Tor for all peer connections
default: false
peer-timeout:
type: number
name: Peer Message Timeout
description: How long to wait for a response from a peer before failing
nullable: false
integral: true
units: Seconds
range: "[0, *)"
default: 30
max-peer-age:
type: number
name: Maximum Peer Age
description: How long to wait before refreshing the peer list
nullable: false
integral: true
units: Seconds
range: "[0, *)"
default: 300
max-peer-concurrency:
type: number
name: Maximum Peer Concurrency
description: How many peers to reach out to concurrently for block data
nullable: true
integral: true
range: "[1, *)"
default: 1
.. _string:
String
^^^^^^
There are various options for string values. They can optionally be marked as copyable or masked, such as for passwords, which will reflect the UI element display. A pattern, expressed in regex, can be denoted. If it exists, this field requires both the pattern type (ie. Regex) and pattern description (ie. an explanation of the pattern requirements).
If the default type is ``Entropy``, the charset can optionally specify an inclusive ranged character set (ie. "a-f,0-9").
ValueSpec Type:
.. code::
type: string
name: String
description: Option<String>
changeWarning: Option<String>
copyable: Option<boolean>
masked: Option<boolean>
nullable: Boolean
default: String | Entropy
pattern: Option<Regex>
pattern-description: Option<String>
Entropy Type:
.. code::
charset: Option<String>
len: integer
Examples:
.. code::
color:
type: string
name: Color
description: Color value for the Lightning Network
nullable: false
pattern: "[0-9a-fA-F]{6}"
patternDescription: |
Must be a valid 6 digit hexadecimal RGB value. The first two digits are red, middle two are green and final two are
blue
default:
charset: "a-f,0-9"
len: 6
password:
type: string
name: Password
description: The password for the RPC User
nullable: false
copyable: true
masked: true
default:
charset: "a-z,A-Z,0-9"
len: 22
.. _pointer:
Pointer
^^^^^^^
The pointer type *points* to a config value on another service installed on EmbassyOS (ie. app subtype) or to the EmbassyOS system (ie. system subtype). When pointing to another service, the ``index`` field indicates the path to the desired config variable.
ValueSpec Type:
.. code::
type: pointer
name: String
description: Option<String>
changeWarning: Option<String>
subtype: app | system
app-id: String (*always* kebab case)
target: AppPointerSpecVariants | SystemPointerSpecVariants
index: Option<String> (dependent on target being AppPointerSpecVariants)
AppPointerSpecVariants = LanAddress | TorAddress | TorKey | Config
SystemPointerSpecVariants = HostIp
Example:
.. code::
user:
type: pointer
name: RPC Username
description: The username for the RPC user for Bitcoin Core
subtype: app
app-id: bitcoind
target: config
index: "rpc.username"
.. _union:
Union
^^^^^
This type describes a necessary dependency. Multiple variants can be expressed to enable the user the option to connect to another service (internal dependency) or outside source (external dependency).
For example, the Bitcoin Proxy service is united with an instance of Bitcoin. Three variants are defined: internal, external, and a quick connect. In this case, internal refers to the Bitcoin Core instance running on EmbassyOS, and defines the RPC credentials necessary for connecting; external refers to a Bitcoin Core node running on a different device, and defines the RPC credentials necessary for connecting; quick connect refers to yet another method of connecting to a Bitcoin Core node, optimized for convenience.
Default is required and corresponds to one of the variants.
``Tag`` is the key that will be rendered on the UI element.
ValueSpec Type;
.. code::
type: union
name: String
description: Option<String>
changeWarning: Option<String>
default: Boolean
tag: Tag
variants: Map<String, ConfigSpec>
display-as: Option<String>
unique-by: any | all | exactly | notUnique
Tag Type:
.. code::
id: String
name: String
description: Option<String>
variant-names: Map<String, String>
.. _example_config_spec:
Example:
.. code:: yaml
bitcoind:
type: union
name: Bitcoin Core
description: The Bitcoin Core node to connect to
tag:
id: type
name: Type
description: |
- Internal: The Bitcoin Core service installed to your Embassy
- External: A Bitcoin Core node running on a different device
- Quick Connect: A Quick Connect URL for an unpruned Bitcoin Core node
variant-names:
internal: Internal
external: External
quick-connect: Quick Connect
default: internal
variants:
internal:
address:
type: pointer
name: Local Address
description: The LAN IP address of your Bitcoin Core service
subtype: app
app-id: bitcoind
target: lan-address
user:
type: pointer
name: RPC Username
description: The username for the RPC user for Bitcoin Core
subtype: app
app-id: bitcoind
target: config
index: "rpc.username"
password:
type: pointer
name: RPC Password
description: The password for the RPC user for Bitcoin Core
subtype: app
app-id: bitcoind
target: config
index: "rpc.password"
external:
addressext:
type: string
name: Public Address
description: The public address of your Bitcoin Core RPC server
nullable: false
userext:
type: string
name: RPC Username
description: The username for the RPC user on your Bitcoin Core RPC server
nullable: false
passwordext:
type: string
name: RPC Password
description: The password for the RPC user on your Bitcoin Core RPC server
nullable: false
quick-connect:
quick-connect-url:
type: string
name: Quick Connect URL
description: The Quick Connect URL for your Bitcoin Core RPC server
nullable: false
pattern: 'btcstandup://[^:]*:[^@]*@[a-zA-Z0-9.-]+:[0-9]+(/(\?(label=.+)?)?)?'
patternDescription: Must be a valid Quick Connect URL. For help, check out https://github.com/BlockchainCommons/Gordian/blob/master/Docs/Quick-Connect-API.md
.. _config_rules:
Config Rules
============
This file defines the configuration rules, or the rule-set that defines dependencies between config variables. In practice, config rules are for auto-configuring self dependencies. Self dependencies are internal dependencies of a service, such as if the setting of one config variable informs the option of another setting. These "dependencies" are configured as rules.
A rule is a boolean expression that we demand to be true. It is not true if the expression fails the rule parser.
They follow the `Backus–Naur <https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_form>`_ meta-syntax for writing rules.
Rules are composed of two main concepts:
* Variables - accessor into a configuration
* Terms - either a variable or type literal (ie. a boolean term is a boolean variable, a boolean expression, or a comparison operation between numbers or strings)
Variables can be booleans, numbers, or strings, and have a different syntax depending on the type. These type annotations check your config rules against your config spec and throw an error if invalid.
- ``?`` - Casts to boolean value. If the value is not a boolean, this notes whether or not the value is null.
- ``#`` - Treat the value as a number. If it is not a number, the value will be parsed as NaN. String numbers are not currently supported.
- ``'`` - Cast the value into a string. Applies to any value except for an object or a list.
- ``!`` - Equals not.
.. note::
Config rules are processed in order.
If application does not satisfy a rule, a set of suggestions should be provided. These suggestions are in the form of the operation to preform:
- ``Set`` - set the value
- ``Push`` - add to the value (such as to a list)
- ``Delete`` - delete the value
.. code:: typescript
enum SuggestionVariant = Set | Delete | Push
interface Set {
var: String, // fully qualified path without typecast
// one of the following three variants are required
to: Option<String> // a string expression, use when tying another config value
to-value: Option<String>
to-entropy: Option<{
charset: String (eg. 'a-z,A-Z,0-9')
len: Number
}>
}
interface Delete {
src: String, // path to key - removes if in a list
}
interface Push {
to: String,
value: String, // string literal of value to be set
}
Set Examples:
.. code:: yaml
- SET:
# the key in config you want to set
var: 'users.[first(item => ''item.name = "c-lightning")].password'
# the value in config that you will set
to-entropy:
charset: "a-z,A-Z,0-9"
len: 22
- SET:
var: 'users.[first(item => ''item.name = "c-lightning")].fetch-blocks'
to-value: true
Push Examples:
.. code:: yaml
- PUSH:
to: "users"
value:
name: c-lightning
allowed-calls: []
- PUSH:
to: 'users.[first(item => ''item.name = "c-lightning")].allowed-calls'
value: "getnetworkinfo"
Full example from `c-lightning manifest <https://github.com/Start9Labs/c-lightning-wrapper/blob/master/manifest.yaml>`_:
.. code:: yaml
config:
- rule: '''users.*.name = "c-lightning"'
description: 'Must have an RPC user named "c-lightning"'
suggestions:
- PUSH:
to: "users"
value:
name: c-lightning
allowed-calls: []
- SET:
var: 'users.[first(item => ''item.name = "c-lightning")].password'
to-entropy:
charset: "a-z,A-Z,0-9"
len: 22
.. role:: raw-html(raw)
:format: html
:raw-html:`<br />`

40
source/dev-docs/service-packaging/docker.rst Normal file
View File

@@ -0,0 +1,40 @@
.. _service_docker:
******************
Service Dockerfile
******************
Dockerfile
==========
This purpose of the Dockerfile is to define how to build the image for the service. It declares the environment and building stages.
The Dockerfile is responsible for mounting the service application to whatever volume the manifest specifies, typically ``/root/.<service-id>``.
Since the build requires specific arm runtime environments, these base images can be imported into the Dockerfile as a starting point so the base system does not need to be completely redefined. This enables importing a specific build environment version that encapsulates any necessary libraries for running the service build, eg. golang, rust.
For instance:
``FROM alpine:3.12``
``FROM arm32v7/alpine``
``FROM arm32v7/golang:alpine``
We recommended using ``alpine`` since it produces the smallest image. We try to keep the image under 100MB when possible.
Entry-point
===========
The ``docker_entrypoint.sh`` defines what to do when the service application starts.
It consists of a bash script that completes any environment setup (eg. creating folder substructure), sets any environment variables, and executes the run command. The only required feature of this file is to execute the run commands for EmbassyOS.
Example
=======
The `LND wrapper <https://github.com/Start9Labs/lnd-wrapper/blob/master/Dockerfile>`_ features a well defined Dockerfile, for example.
.. role:: raw-html(raw)
:format: html
:raw-html:`<br />`

44
source/dev-docs/service-packaging/index copy.rst Normal file
View File

@@ -0,0 +1,44 @@
.. _service_pacakge:
***********************
Service Packaging Guide
***********************
Welcome! If you are here, you are interested in becoming part of the mission to change the future of personal computing. This guide will take you through the process of packaging a service for EmbassyOS, a novel, self-hosted, sovereign computing platform.
A service in this context is any open source project that has been bundled into the appropriate format to run on EmbassyOS. By configuring and packaging a project according to this guide, it can be installed on EmbassyOS with no command line or technical expertise required. This opens up an entire platform for self-hosted software to run independent of third parties in a completely private and sovereign way for all individuals.
This guide is technical, but breaks down the steps for deployment. If you have any feedback or questions concerning this guide, please don't hesitate to `reach out <https://matrix.to/#/#community-dev:matrix.start9labs.com>`_ or submit a pull request with alterations.
To start, you will need to acquire EmbassyOS for testing the packaged service. This can be done by:
- building from `source <https://github.com/Start9Labs/embassy-os/blob/master/CONTRIBUTING.md#setting-up-your-development-environment>`_
- following the :ref:`DIY <diy>` guide
- :ref:`purchasing <purchasing>` the ready to run personal server
While you are waiting to receive or assemble a device, you can begin the process of packaging a project. The sections below outline these steps in detail.
Happy building!
.. toctree::
:maxdepth: 1
Overview <overview>
Wrapper <wrapper>
Manifest <manifest>
Docker <docker>
Makefile <makefile>
Config <config>
Properties <properties>
Instructions <instructions>
Backups <backups>
Submission process
------------------
When you have built and tested your project for EmbassyOS, please send Start9 a submission with the project repository to ``dev@start9labs.com``. After being reviewed for security and compatibility, the service will be deployed to the marketplace and available for all EmbassyOS users to download.
.. role:: raw-html(raw)
:format: html
:raw-html:`<br />`

18
source/dev-docs/service-packaging/index.rst Normal file
View File

@@ -0,0 +1,18 @@
*****************
Service Packaging
*****************
An overview of EmbassyOS general capabilities.
.. toctree::
:maxdepth: 2
overview
wrapper
manifest
docker
makefile
config
properties
instructions
backups

23
source/dev-docs/service-packaging/instructions.rst Normal file
View File

@@ -0,0 +1,23 @@
.. _service_instructions:
************************************
Service Instructions & Documentation
************************************
Within each wrapper repository should exist a ``docs`` directory. This folder should include any pertinent documentation, instructions, external integrations, or other details about the service that users or developers might find relevant.
If an ``instructions.md`` file is included, this markdown formatted file will be rendered in the "Instructions" tab within the service detail menu on EmbassyOS:
.. figure:: /_static/images/service/bitcoin_instructions.png
:width: 80%
:alt: Bitcoin Instructions
Example
=======
The `bitcoind wrapper <https://github.com/Start9Labs/bitcoind-wrapper/tree/master/docs>`_ demonstrates a good use of instructions and external integrations.
.. role:: raw-html(raw)
:format: html
:raw-html:`<br />`

30
source/dev-docs/service-packaging/makefile.rst Normal file
View File

@@ -0,0 +1,30 @@
.. _service_makefile:
****************
Service Makefile
****************
.. note::
*This file is optional*
A Makefile serves as a convenience for outlining the build. This helps streamline additional developer contributions to the project. Please reference the GNU `documentation <https://www.gnu.org/software/make/manual/html_node/Introduction.html>`_ for implementation details.
An alternative to using ``make`` is to use the `nix <https://nixos.wiki/wiki/Nix>`_ specification.
This purpose of this file is to:
- Read the docker container and build the project
- Build all prerequisites for running the docker file
- Build all dependencies
- Package ``config_rules.yaml``, ``config_spec.yaml``, ``manifest.yaml``, and ``image.tar`` into an ``.s9pk`` extension by invoking ``appmgr``.
Example
=======
The `LND wrapper <https://github.com/Start9Labs/lnd-wrapper/blob/master/Makefile>`_ features a well defined Makefile, for example.
.. role:: raw-html(raw)
:format: html
:raw-html:`<br />`

352
source/dev-docs/service-packaging/manifest.rst Normal file
View File

@@ -0,0 +1,352 @@
.. _service_manifest:
****************
Service Manifest
****************
Overview
========
This file describes the service and it's requirements. It is used to:
- create a listing in the marketplace
- denote any installation considerations
- define dependency requirements
Each time a service is updated, the manifest should be updated to include the new version, release notes, and any pertinent updates to the install, uninstall, or restoration flows. All this information is displayed in the marketplace listing, and the optionally denoted alerts will be displayed when appropriate to provide the user with more information.
For instance, `LND displays alerts <https://github.com/Start9Labs/lnd-wrapper/blob/master/manifest.yaml#L28>`_ around restoration since data will be overwritten.
There is nothing you need to do as a developer to set up Tor for running a service. This is *completely* handled by EmbassyOS - a Tor address will be automatically generated when the service is installed. Just define the port and which version of Tor the service needs in this manifest file! You do, however, need to ensure the service is in fact capable of running over Tor.
The manifest is also responsible for outlining service :ref:`dependencies <dependencies>`. By defining rules using the :ref:`EmbassyOS DSL specification <config_rules>`, users can easily and selectively install, uninstall, and update any service without getting stuck in dependency hell. EmbassyOS presents this information in a polished install/uninstall/update wizard, so there's no need for editing configuration files or jumping into the command line. For you as a developer, this simply means populating this key in the manifest!
Formatting
==========
- Serialization language:``.yaml``
- Case style: ``kebab-case``
Type definitions
================
Below are the types and sub-type definitions, with necessary elaborations. Any item that contains ``Option<>`` is an optional field.
.. code:: yaml
# manifest version compatibility ie. v0 (this is currently the only option)
compat: v0
# service id, used for uniqueness and BE identification ie. bitcoind
id: String
# version number of the release conforming to the semantic versioning scheme
version: String
# display name of service
title: String
# an object containing a short and long description of the service. TODO character lengths
description:
short: String
long: String
# a link to the release tag notes in GitHub, or a short description TODO character length
release-notes: String
# a notification message that should caution the user with any service particularities, eg. beta tech
install-alert: Option<String>
# a notification message warning users of potential problems with uninstalling, such as dependency failures or data loss
uninstall-alert: Option<String>
# a notification message containing warning or details about backup restoration
restore-alert: Option<String>
# denoting the existence of instructions.md
has-instructions: Boolean
# required semver version range of EmbassyOS to run service eg. ">=1.0.0"
os-version-required: VersionReq
# recommended semver version range of EmbassyOS to run service eg."^1.0.0"
os-version-recommended: VersionReq
# a list of objects of ports to run the service on localhost and tor
ports:
- internal: String
tor: String
# currently only tar is supported
image:
type: String
# shared memory container size
shm-size-mb: Option<usize>
# path to mount the image on the volume, ie: /root/bitcoind
mount: String
# read only data exposed to dependencies (path is relevant to mount)
public: Option<String>
# shared filesystem segment with each of your dependencies (path is relevant to mount)
shared: Option<String>
# deprecated - will default to an empty vector
assets: []
# version of tor support, eg. v2, v3
hidden-service-version: String
# A map of dependent services, see below for more details
dependencies: Dependencies
.. _dependencies:
Dependencies
============
Many services depend on other libraries and services on EmbassyOS (such as Bitcoin), sometimes even a particular version of those services, which need to be specified by the developers so that EmbassyOS can handle installing these dependencies under the hood.
The key of each field in the dependencies object is the lowercase, kebab-case app ID of the service that is depended on. Each dependency contains a set of rules that need to be fulfilled as true if the dependency is to be properly installed. The "config rules" here are for auto-configuring dependencies - the action defined by the rule will be executed if the service is auto configured with defaults during initial setup. This simplifies and streamlines the user experience. The interface should provide suggestions for the behavior if the denoted rule cannot be met with previous configurations.
Let's take this snippet for example:
.. code:: yaml
...
dependencies:
btc-rpc-proxy:
version: "^0.1.0"
optional: Can configure an external bitcoin node.
description: Required for fetching validated blocks.
config:
- rule: '''users.*.name = "c-lightning"'
description: 'Must have an RPC user named "c-lightning"'
suggestions:
- PUSH:
to: 'users'
value:
name: c-lightning
...
.. role:: raw-html(raw)
:format: html
:raw-html:`<br />`
The service ``btc-rpc-proxy`` is a dependency of the service ``c-lightning``. ``c-lightning`` requires it to be installed at a version >=0.1.0 <0.2.0. There exists a rule that states the config option ``user.name`` must be equal to "c-lightning". If this value does not exist for ``user.name`` when accessed, ``PUSH`` the value "c-lighting" to the field. This all takes place during the initial service configuration, before the service is started for the first time.
.. note::
Dependency config rules are processed in order.
Type definitions
----------------
Types for ``manifest.yaml`` fields:
.. code:: typescript
interface Dependencies [{
serviceId: DepInfo
}]
interface DepInfo {
version: String // ie. 0.11.1.1
optional?: String,
description?: String,
config: [ConfigRule],
],
}
interface ConfigRule {
rule: String, // ie. 'users.*.name = "lnd"
description: String,
suggestions: [SuggestionVariant]
}
interface SuggestionVariant {
SET: {
var: String,
to: SetVariant,
},
DELETE: {
src: String,
},
PUSH: {
to: String,
value: Value,
},
}
interface SetVariant {
to: Option<String>,
to-value: Option<Value>, // ie. true/false
to-entropy: Option<{
charset: String // ie. 'a-z,A-Z,0-9'
len: number
}>
}
.. role:: raw-html(raw)
:format: html
:raw-html:`<br />`
Examples
--------
Actual ``manifest.yaml`` files for existing services:
LND
^^^
.. code:: yaml
compat: v0
id: lnd
version: 0.11.1.1
title: Lightning Network Daemon
description:
short: "A complete implementation of a Lightning Network node by Lightning Labs"
long: "LND fully conforms to the Lightning Network specification (BOLTs). BOLT stands for: Basis of Lightning Technology. In the current state lnd is capable of: creating channels, closing channels, managing all channel states (including the exceptional ones!), maintaining a fully authenticated+validated channel graph, performing path finding within the network, passively forwarding incoming payments, sending outgoing onion-encrypted payments through the network, updating advertised fee schedules, and automatic channel management (autopilot)."
release-notes: https://github.com/lightningnetwork/lnd/releases/tag/v0.11.1-beta
ports:
- internal: 8080
tor: 8080
- internal: 9735
tor: 9735
- internal: 9911
tor: 9911
- internal: 10009
tor: 10009
image:
type: tar
mount: /root/.lnd
public: public
has-instructions: true
os-version-required: ">=0.2.8"
os-version-recommended: ">=0.2.8"
install-alert: |
READ CAREFULLY! LND and the Lightning Network are considered beta software. Please use with caution and do not risk more money than you are willing to lose. We encourage frequent backups. If for any reason, you need to restore LND from a backup, your on-chain wallet will be restored, but all your channels will be closed and their funds returned to your on-chain wallet, minus fees. It may also take some time for this process to occur.
uninstall-alert: "READ CAREFULLY! Uninstalling LND will result in permanent loss of data, including its private keys for its on-chain wallet and all channel states. Please make a backup if you have any funds in your on-chain wallet or in any channels. Recovering from backup will restore your on-chain wallet, but due to the architecture of the Lightning Network, your channels cannot be recovered. All your channels will be closed and their funds returned to your on-chain wallet, minus fees. \n"
restore-alert: |
Restoring LND will overwrite its current data, including its on-chain wallet and channels. Any channels opened since the last backup will be forgotten and may linger indefinitely, and channels contained in the backup will be closed and their funds returned to your on-chain wallet, minus fees.
assets: []
hidden-service-version: v3
dependencies:
btc-rpc-proxy:
version: "^0.2.4"
optional: Can alternatively configure an external bitcoin node.
description: Used to fetch validated blocks.
config:
- rule: '''users.*.name = "lnd"'
description: 'Must have an RPC user named "lnd"'
suggestions:
- PUSH:
to: "users"
value:
name: lnd
allowed-calls: []
- SET:
var: 'users.[first(item => ''item.name = "lnd")].password'
to-entropy:
charset: "a-z,A-Z,0-9"
len: 22
- rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getinfo"'
description: 'RPC user "lnd" must have "getinfo" enabled'
suggestions:
- PUSH:
to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
value: "getinfo"
- rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getbestblockhash"'
description: 'RPC user "lnd" must have "getbestblockhash" enabled'
suggestions:
- PUSH:
to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
value: "getbestblockhash"
- rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "gettxout"'
description: 'RPC user "lnd" must have "gettxout" enabled'
suggestions:
- PUSH:
to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
value: "gettxout"
- rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getblockchaininfo"'
description: 'RPC user "lnd" must have "getblockchaininfo" enabled'
suggestions:
- PUSH:
to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
value: "getblockchaininfo"
- rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "sendrawtransaction"'
description: 'RPC user "lnd" must have "sendrawtransaction" enabled'
suggestions:
- PUSH:
to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
value: "sendrawtransaction"
- rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getblockhash"'
description: 'RPC user "lnd" must have "getblockhash" enabled'
suggestions:
- PUSH:
to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
value: "getblockhash"
- rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getblock"'
description: 'RPC user "lnd" must have "getblock" enabled'
suggestions:
- PUSH:
to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
value: "getblock"
- rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getblockheader"'
description: 'RPC user "lnd" must have "getblockheader" enabled'
suggestions:
- PUSH:
to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
value: "getblockheader"
- rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "estimatesmartfee"'
description: 'RPC user "lnd" must have "estimatesmartfee" enabled'
suggestions:
- PUSH:
to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
value: "estimatesmartfee"
- rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getnetworkinfo"'
description: 'RPC user "lnd" must have "getnetworkinfo" enabled'
suggestions:
- PUSH:
to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
value: "getnetworkinfo"
- rule: 'users.[first(item => ''item.name = "lnd")].fetch-blocks?'
description: 'RPC user "lnd" must have "Fetch Blocks" enabled'
suggestions:
- SET:
var: 'users.[first(item => ''item.name = "lnd")].fetch-blocks'
to-value: true
bitcoind:
version: "^0.21.0"
optional: Can alternatively configure an external bitcoin node.
description: Used to subscribe to new block events.
config:
- rule: "zmq-enabled?"
description: "Must have an ZeroMQ enabled"
suggestions:
- SET:
var: "zmq-enabled"
to-value: true
Cups
^^^^
.. code:: yaml
compat: v0
id: cups
version: "0.3.6"
title: "Cups Messenger"
description:
short: "Real private messaging"
long: "Cups is a private, self-hosted, peer-to-peer, Tor-based, instant messenger. Unlike other end-to-end encrypted messengers, with Cups on the Embassy there are no trusted third parties."
release-notes: |
Features
- Adds instructions defined by EmbassyOS 0.2.4 instructions feature
ports:
- internal: 59001
tor: 59001
- internal: 80
tor: 80
image:
type: tar
mount: /root
has-instructions: true
os-version-required: ">=0.2.4"
os-version-recommended: ">=0.2.4"
assets:
- src: httpd.conf
dst: "."
overwrite: true
- src: www
dst: "."
overwrite: true
hidden-service-version: v3
.. role:: raw-html(raw)
:format: html
:raw-html:`<br />`

103
source/dev-docs/service-packaging/overview.rst Normal file
View File

@@ -0,0 +1,103 @@
.. _service_package_overview:
************************
Service Package Overview
************************
There are several main components that comprise a service package for EmbassyOS. This overview will introduce them and following sections will dive into technical specifics.
First, let's get your system setup with the necessary software dependencies.
Environment Setup
=================
At minimum, ``docker``, ``docker-buildx``, and ``appmgr`` are required. The rest of the recommendations serve to optimize the build process.
Recommended Dependencies
------------------------
`docker <https://docs.docker.com/get-docker>`_
`docker-buildx <https://docs.docker.com/buildx/working-with-buildx/>`_
`appmgr <https://github.com/Start9Labs/embassy-os/tree/master/appmgr>`_
`cargo <https://doc.rust-lang.org/cargo/>`_
`yq (version 4) <https://mikefarah.gitbook.io/yq>`_
`make <https://www.gnu.org/software/make/>`_
`musl <https://github.com/Start9Labs/rust-musl-cross>`_
Package Components
==================
Image
-----
Each service boils down to a Docker image. We're not going to dive into Docker specifics in this guide, since there exists tons of `resources <https://docs.docker.com/>`_ for developing with this containerization tool.
Because the service ultimately runs on a Raspberry Pi, the created Docker image should be a snapshot of an arm based linux environment. The service image is then mounted to the EmbassyOS image during installation. This path is defined in the :ref:`manifest <service_manifest>` configuration file.
The image is immutable, meaning that when the service is updated, the image is replaced with a completely new image containing the updated features.
Volume
------
Each service application gets a volume allocated to it by EmbassyOS. All service application data is stored in this directory and is persisted across updates.
The volume directory (for seeding data into the volume) is typically: ``/root/volumes/<service-id>``.
.. warning::
Any files that are in the image at the volume path will be overwritten when a backup restore occurs.
Dependencies
------------
When it comes to running your own server, managing dependencies is perhaps the most difficult part. The term "dependency hell" emerged from this sentiment. Even the best developers have lost significant amounts of time trying to find, correct, or clarify documentation for dependencies or dependency configuration. Individuals who manage their own servers have specific technical skills and are willing to devote the time and effort necessary to maintain them. Companies have entire departments dedicated to this feat.
Some other personal server products aimed at non-technical individuals tackle this problem by simply hard coding dependencies. Since EmbassyOS is a platform for running all open-source, self-hosted software, it is not possible to hard code every possible service, service dependency, and service dependency configuration forever. Instead, Start9 built a new, unprecedented operating system that touts a generalized, intuitive approach to dependency management and service configuration. EmbassyOS enables users to easily and selectively install, uninstall, and update any service they want without getting stuck in dependency hell.
This may sound cool or neat, but it is more than that: *it’s novel*. This has never been done before.
The key to making the system work is a new, domain-specific-language (DSL) and set of standards that are used by developers to define the rules and requirements of their services. Run in the context of EmbassyOS, these rules and requirements appear as simple UI elements, such as inputs, toggles, and drop downs, and they are enforced by validations and clear user instructions. Using this system, what previously required involved time and expertise, can now be done by anyone in a matter of seconds.
This DSL is utilized in the :ref:`config rules <config_rules>` and :ref:`dependencies <dependencies>` key in the service manifest.
Manifest
--------
This file describes the service and its requirements. It is used to:
- create a listing in the marketplace
- denote any installation considerations
- define dependency requirements
Each time a service is updated, the manifest should be updated to include the new version, release notes, and any pertinent updates to the install, uninstall, or restoration flows. All this information is displayed in the marketplace listing, and the optionally denoted alerts will be displayed when appropriate to provide the user with more information about the particularities of the service.
For instance, `LND displays alerts <https://github.com/Start9Labs/lnd-wrapper/blob/master/manifest.yaml#L28>`_ around restoration since data will be overwritten.
There is nothing you need to do as a developer to enable the service to run over Tor. This is *completely* handled by EmbassyOS - a Tor address will be automatically generated and an Nginx server configured (if a client application) when the service is installed. Just define which version of Tor your service needs in this manifest file!
Config
------
Most self-hosted applications require the user to tell the app how to behave using a config file in a specific format, environment variables, command-line arguments, or some combination of these inputs. One of the coolest features of EmbassyOS is that, when packaged correctly, the app's :ref:`configuration <service_config>` will be available to the user as a slick GUI that always produces a valid configuration no matter how little experience or skill the user has.
With EmbassyOS, this means a service wrappers' configuration requires a particular format and rule structure to ensure it integrates smoothly with the user interface. This format enables clean handling of improper values and configuration dependencies.
.s9pk Bundle
------------
The configuration and manifest files get bundled into the ``.s9pk`` package, which is `built using appmgr <https://github.com/Start9Labs/embassy-os/tree/master/appmgr>`_. Each new version release should include the updated version of these files re-bundled into a new binary. This is the file that will be downloaded from the marketplace. Upon user initiation, EmbassyOS then un-packages and installs the service.
Hello World Example
-------------------
For reference, the `Hello world <https://github.com/Start9Labs/hello-world-wrapper>`_ repository should be used as an example. A project template can be cloned using the "Use this template" button in GitHub.
.. role:: raw-html(raw)
:format: html
:raw-html:`<br />`

25
source/dev-docs/service-packaging/properties.rst Normal file
View File

@@ -0,0 +1,25 @@
.. _service_properties:
******************
Service Properties
******************
The output of this step is a file titled ``stats.yaml``. This file contains a mapping of the values that will be displayed in the ``Properties`` section in a service's menu.
.. figure:: /_static/images/service/service_properties.png
:width: 80%
:alt: Service Properties
Service Properties Tab
Due to the fact that config variables are only available when the service is running, this file can only be populated at runtime. In services Start9 has wrapped or created, we typically handle this in ``configurator/src/main.rs``. This file is essentially a script that acquires the necessary values at runtime from config (eg. ``/root/.lnd/start9/config.yaml``), and dynamically populating them to the ``stats.yaml`` file, marking each value with the appropriate keys for UI display (ie. masked or copyable). Running the configurator is typically included in the ``docker_entrypoint.sh`` file.
Example
=======
A good example of the configurator producing the ``stats.yaml`` file can be found `here <https://github.com/Start9Labs/lnd-wrapper/blob/master/configurator/src/main.rs>`_.
.. role:: raw-html(raw)
:format: html
:raw-html:`<br />`

42
source/dev-docs/service-packaging/wrapper.rst Normal file
View File

@@ -0,0 +1,42 @@
.. _service_wrapper:
***************
Service Wrapper
***************
Each service is bound with a wrapper repository, which contains everything you need to build a service.
The purpose of this repo is:
- Denote any dependencies required to run and build the project
- To define the necessary, ``config_rules.yaml``, ``config_spec.yaml`` and ``manifest.yaml`` options
- To build the project into the ``.s9pk`` format digestible to EmbassyOS
- Link to the source project as a git submodule
- Define the docker file for running the project on EmbassyOS
- Provide documentation for the project, especially user runbook instructions
- symlink of ``instructions.md`` from ``docs`` directory to wrapper repo root, if included
File Structure
==============
The project structure should be used as a model:
.. code-block:: bash
├── Dockerfile
├── Makefile (optional)
├── README.md
├── config_rules.yaml
├── config_spec.yaml
├── <submodule_project_dir>
├── docker_entrypoint.sh
├── docs
│ └── instructions.md
└── manifest.yaml
Submodule
==========
`Git submodules <https://www.git-scm.com/book/en/v2/Git-Tools-Submodules>`_ allow use of another project while in the working project directory. Setting up this feature enables linking of the source service repository so that its context is available.
Run ``git submodule add <link_to_source_project>``