Merge pull request #4 from Start9Labs/feature/service-sdk

Feature/service sdk

source/_static/default.css

@@ -51,7 +51,7 @@ h1, h2 {
 
 }
 
-.admonition.warning .admonition-title, .admonition.note .admonition-title  {
+.admonition.warning .admonition-title, .admonition.note .admonition-title,  .admonition.important .admonition-title   {
     background-color: #666;
 }
 

source/contributing/adding-services.rst

@@ -1,13 +0,0 @@
-***************
-Adding Services
-***************
-
-We are finalizing the OS infrastructure requirements to ensure a smooth process for developer contribution. This includes wrapping everything up into a software development kit for developing services on the EmbassyOS platform.
-
-If you are interested in developing a service, keep the following in mind:
-
-Marketplace services need to provide a "service manifest" and a "config spec", which the SDK will facilitate. The service manifest is for the store listing. The config spec is a detailed mapping of configuration options with acceptable values, defaults, and relational rule-sets. For example, if the user chooses config option A, then config option B must be between 5 and 10. That way, users can enjoy a simple GUI configuration experience, complete with validations and protections, and not have to worry about screwing up a .conf file in some unknown way. 
-
-The config spec will also denote any dependencies. Many services depend on other libraries and services on the Embassy (such as Bitcoin), sometimes even a particular version of those services, which need to be specified by the developers so that EmbassyOS can handle installing these dependencies under the hood.
-
-Check back soon for more details!

source/contributing/index.rst

@@ -0,0 +1,33 @@
+.. _service_sdk:
+
+***********************
+Service Packaging Guide
+***********************
+
+This technical guide covers the necessary steps to package up a service for deployment to the EmbassyOS platform. To start, you will need to acquire :ref:`EmbassyOS <purchasing>` for testing the packaged service.
+
+While you are waiting to assemble a device, you can begin the process of building your project in the format necessary to run on the Embassy. The sections below outline these steps in detail.
+
+For reference, the `Hello world <https://github.com/Start9Labs/hello-world-wrapper>`_ repository should be used as an example. A project template can be cloned using the "Use this template" button in GitHub.
+
+.. toctree::
+  :maxdepth: 2
+
+  Overview <services/overview>
+  Wrapper <services/wrapper>
+  Docker <services/docker>
+  Makefile <services/makefile>
+  Manifest <services/manifest>
+  Config <services/config>
+  Properties <services/properties>
+  Docs <services/docs>
+
+Submission process
+------------------
+
+When you have built and tested your project for EmbassyOS, please send us a submission with the project repository to ``dev@start9labs.com``. Reach out with questions in the Matrix community `dev channel <https://matrix.to/#/#community-dev:matrix.start9labs.com>`_.
+
+.. role:: raw-html(raw)
+    :format: html
+
+:raw-html:`<br />`

source/contributing/services/config.rst

@@ -0,0 +1,578 @@
+.. _service_config:
+
+****************************
+Service Config Specification
+****************************
+
+Configuration options are a crucial component of any application. EmbassyOS requires a particular format to ensure your service integrates smoothly with the user interface. This enables clean handling of improper values and dependency management.
+
+The outcome of this step is two files:
+
+:ref:`config_spec.yaml <config_spec>`
+
+:ref:`config_rules.yaml <config_rules>`
+
+These files contain a detailed mapping of configuration options with acceptable values, defaults and relational rule-sets. 
+
+For example, if the user chooses config option A, then config option B must be between 5 and 10. This enables a simple GUI configuration experience, complete with validations and protections, for users. They do not have to worry about the consequences of a wrong value in a ``.conf`` file. 
+
+.. _config_spec:
+
+Config Spec
+===========
+
+Overview
+--------
+
+.. figure:: /_static/images/service/bitcoin_config.png
+  :width: 80%
+  :alt: Bitcoin Config
+
+This file defines the structure of configuration options your service depends on to run. It additionally can include configuration options that users might want to enable for more advanced or customized usage. Ultimately, these values influence the UI elements for a user to interact with. Specifically, they evaluate to the options available when managing a service, such as:
+
+- Prior to service installation when the user needs to be made aware of any necessary dependency configurations
+- When the user installs a service and the service is in the "Needs Config" state
+- Whenever a user edits a service config
+- When config pointers get updated
+
+The neat part about this file is that each ValueSpec type gets translated into a specific front end component. For instance, boolean values display as a toggle button.
+
+.. figure:: /_static/images/service/boolean_toggle.png
+  :width: 80%
+  :alt: Example boolean toggle
+
+Another advantage is the ability to define default values. These values automatically get populated if the user selects the ``Default`` option when setting up a service in ``Needs Config`` state. This is super convenient for users who want to get up and running quickly.
+
+Types
+-----
+
+ConfigSpec Type:
+
+.. code::
+
+    key: ValueSpec
+
+    ValueSpec Type: Boolean | Enum | List | Number | Object | String | Union | Pointer (see below for details)
+
+Implementation Guide
+--------------------
+
+The following section contains implementation specifications for the ``config_spec.yaml`` file.
+
+- All keys are ``kebab-case`` strings, which correspond to the service (app) id
+- All values are one the following specs (ie. ``ValueSpec`` type):
+
+    :ref:`boolean <boolean>`
+
+    :ref:`enum <enum>`
+
+    :ref:`list <list>`
+
+    :ref:`number <number>`
+
+    :ref:`object <object>`
+
+    :ref:`string <string>`
+
+    :ref:`union <union>`
+
+    :ref:`pointer <pointer>`
+
+- In the examples for each value spec type below, ``Option`` means the key is optional. Otherwise, the key is required.
+- Descriptions are optional but recommended
+- Name corresponds to the name of the config field
+- Find a complete example :ref:`here <example_config_spec>`
+- Change warning text displays when the value is altered
+
+.. _boolean:
+
+Boolean
+^^^^^^^
+
+Config value specification denoted as a boolean value. A default value is required.
+
+``ValueSpec`` Type:
+
+.. code::
+
+    type: boolean
+    name: String
+    description: Option<String>
+    changeWarning: Option<String>
+    default: Boolean
+
+Example:
+
+.. code:: yaml
+
+    fetch-blocks:
+        type: boolean
+        name: Fetch Blocks
+        description: Fetch blocks from the network if pruned from disk
+        default: true
+
+.. _enum:
+
+Enum
+^^^^
+
+Config value specification denoted as an enum value. Enums values must be a unique set. If no default is provided, ``null`` will be the assumed value.
+
+ValueSpec Type:
+
+.. code::
+
+    type: enum
+    name: String
+    description: Option<String>
+    changeWarning: Option<String>
+    default: Option<Enum>
+    values: Set<String>
+
+.. code:: yaml
+
+    theme-mode:
+        type: enum
+        name: Theme Mode
+        values:
+                - NIGHT
+                - DAY
+        valueNames:
+                NIGHT: Night
+                DAY: Day
+        default: NIGHT
+
+.. _list:
+
+List
+^^^^
+
+The list type describes an array of values. The values must consist of the same subtype, which can be any of the ValueSpec types available in the EmbassyOS config specification. 
+Lists of any type do not contain the default for each item in list. The list *itself* can be have a default. If no default is provided, ``null`` will be the assumed value.
+
+Range is loosely based off mathematical range syntax, with infinity replaced with ``*``:
+
+    ``[ || ]`` = inclusive
+
+    ``( || )`` = noninclusive
+
+    ``*`` = infinity on either end
+
+eg:
+
+.. code::
+
+    [0,*) - all numbers to infinity including 0
+
+ValueSpec Type:
+
+.. code:: 
+
+    type: list
+    name: String
+    description: Option<String>
+    subtype: enum || number || object || string || union
+    range: NumRange<unsigned integer>
+    spec: ValueSpec
+    default: ValueSpec
+
+Example:
+
+.. code:: yaml
+
+    allowed-calls:
+        type: list
+        name: Allowed Calls
+        description: The list of all RPC methods this user is allowed to make
+        subtype: enum
+        range: "[0, *)"
+        spec:
+            type: enum
+            values:
+                - item
+                - item
+
+
+.. _number:
+
+Number
+^^^^^^
+
+A number value within an optionally defined range. Nullable field is required. If ``nullable`` is true, the default is assumed to be ``null`` if it is not provided.
+
+ValueSpec Type:
+
+.. code::
+
+    type: number
+    name: String
+    description: Option<String>
+    changeWarning: Option<String>
+    default: Boolean
+    nullable: Boolean
+    range: Option<NumRange<64 bit floating point>>
+    integral: Boolean
+    units: Option<String>
+
+Example:
+
+.. code:: yaml
+
+    type: number
+    name: Peer Message Timeout
+    description: How long to wait for a response from a peer before failing
+    nullable: false
+    integral: true
+    units: Seconds
+    range: "[0, *)"
+    default: 30
+
+.. _object:
+
+Object Type
+^^^^^^^^^^^
+
+A nested representation of a ConfigSpec. The object type takes the same structure under the ``spec`` key as a ConfigSpec: a key indicates the field name, and the value denotes the ValueSpec type for that field. 
+
+There is no default option for the object type. Rather, the option ``null-by-default`` should be used to indicate the default as ``null``. If null by default is true, nullable must be true. If null by default is false, nullable could be either.
+
+``unique-by`` indicates whether duplicates can be permitted in the list.
+
+ValueSpec Type:
+
+.. code::
+
+    type: object
+    name: String
+    description: Option<String>
+    changeWarning: Option<String>
+    nullable: Boolean
+    null-by-default: Boolean
+    display-as: Option<String>
+    unique-by: UniqueBy
+    spec: ConfigSpec
+
+    type UniqueBy = null | string | { any: UniqueBy[] } | { all: UniqueBy[] }
+
+Example:
+
+.. code:: yaml
+
+    type: object
+    name: Advanced
+    description: Advanced settings for Bitcoin Proxy
+    nullable: false
+    spec:
+        tor-only:
+            type: boolean
+            name: Only Tor Peers
+            description: Use Tor for all peer connections
+            default: false
+        peer-timeout:
+            type: number
+            name: Peer Message Timeout
+            description: How long to wait for a response from a peer before failing
+            nullable: false
+            integral: true
+            units: Seconds
+            range: "[0, *)"
+            default: 30
+        max-peer-age:
+            type: number
+            name: Maximum Peer Age
+            description: How long to wait before refreshing the peer list
+            nullable: false
+            integral: true
+            units: Seconds
+            range: "[0, *)"
+            default: 300
+        max-peer-concurrency:
+            type: number
+            name: Maximum Peer Concurrency
+            description: How many peers to reach out to concurrently for block data
+            nullable: true
+            integral: true
+            range: "[1, *)"
+            default: 1
+
+.. _string:
+
+String
+^^^^^^
+
+There are various options for string values. They can optionally be marked as copyable or masked, such as for passwords, which will reflect the UI element display. A pattern, expressed in regex, can be denoted. If it exists, this field requires both the pattern type (ie. Regex) and pattern description (ie. an explanation of the pattern requirements).
+
+If the default type is ``Entropy``, the charset can optionally specify an inclusive ranged character set (ie. "a-f,0-9").
+
+ValueSpec Type:
+
+.. code::
+
+    type: string
+    name: String
+    description: Option<String>
+    changeWarning: Option<String>
+    copyable: Option<boolean>
+    masked: Option<boolean>
+    nullable: Boolean
+    default: String | Entropy
+    pattern: Option<Regex>
+    pattern-description: Option<String>
+
+Entropy Type:
+
+.. code::
+
+    charset: Option<String>
+    len: integer
+
+Examples:
+
+.. code::
+
+    color:
+        type: string
+        name: Color
+        description: Color value for the Lightning Network
+        nullable: false
+        pattern: "[0-9a-fA-F]{6}"
+        patternDescription: |
+                Must be a valid 6 digit hexadecimal RGB value. The first two digits are red, middle two are green and final two are
+                blue
+        default:
+                charset: "a-f,0-9"
+                len: 6
+
+    password:
+        type: string
+        name: Password
+        description: The password for the RPC User
+        nullable: false
+        copyable: true
+        masked: true
+        default:
+            charset: "a-z,A-Z,0-9"
+            len: 22
+
+.. _pointer:
+
+Pointer
+^^^^^^^
+
+The pointer type *points* to a config value on another service installed on EmbassyOS (ie. app subtype) or to the EmbassyOS system (ie. system subtype). When pointing to another service, the ``index`` field indicates the path to the desired config variable.
+
+ValueSpec Type:
+
+.. code::
+
+    type: pointer
+    name: String
+    description: Option<String>
+    changeWarning: Option<String>
+    subtype: app | system
+    app-id: String (*always* kebab case)
+    target: AppPointerSpecVariants | SystemPointerSpecVariants
+    index: Option<String> (dependent on target being AppPointerSpecVariants)
+
+    AppPointerSpecVariants = LanAddress | TorAddress | TorKey | Config 
+    SystemPointerSpecVariants = HostIp
+
+Example:
+
+.. code::
+
+    user:
+        type: pointer
+        name: RPC Username
+        description: The username for the RPC user for Bitcoin Core
+        subtype: app
+        app-id: bitcoind
+        target: config
+        index: "rpc.username"
+
+.. _union:
+
+Union
+^^^^^
+This type describes a necessary dependency. Multiple variants can be expressed to enable the user the option to connect to another service (internal dependency) or outside source (external dependency).
+
+For example, the Bitcoin Proxy service is united with an instance of Bitcoin. Three variants are defined: internal, external, and a quick connect. In this case, internal refers to the Bitcoin Core instance running on EmbassyOS, and defines the RPC credentials necessary for connecting; external refers to a Bitcoin Core node running on a different device, and defines the RPC credentials necessary for connecting; quick connect refers to yet another method of connecting to a Bitcoin Core node, optimized for convenience.
+
+Default is required and corresponds to one of the variants. 
+
+``Tag`` is the key that will be rendered on the UI element.
+
+ValueSpec Type;
+
+.. code::
+
+    type: union
+    name: String
+    description: Option<String>
+    changeWarning: Option<String>
+    default: Boolean
+    tag: Tag
+    variants: Map<String, ConfigSpec>
+    display-as: Option<String>
+    unique-by: any | all | exactly | notUnique
+
+Tag Type:
+
+.. code::
+
+    id: String
+    name: String
+    description: Option<String>
+    variant-names: Map<String, String>
+
+.. _example_config_spec:
+
+Example:
+
+.. code:: yaml
+
+    bitcoind:
+        type: union
+        name: Bitcoin Core
+        description: The Bitcoin Core node to connect to
+        tag:
+            id: type
+            name: Type
+            description: |
+            - Internal: The Bitcoin Core service installed to your Embassy
+            - External: A Bitcoin Core node running on a different device
+            - Quick Connect: A Quick Connect URL for an unpruned Bitcoin Core node
+            variant-names:
+            internal: Internal
+            external: External
+            quick-connect: Quick Connect
+        default: internal
+        variants:
+            internal:
+                address:
+                    type: pointer
+                    name: Local Address
+                    description: The LAN IP address of your Bitcoin Core service
+                    subtype: app
+                    app-id: bitcoind
+                    target: lan-address
+                user:
+                    type: pointer
+                    name: RPC Username
+                    description: The username for the RPC user for Bitcoin Core
+                    subtype: app
+                    app-id: bitcoind
+                    target: config
+                    index: "rpc.username"
+                password:
+                    type: pointer
+                    name: RPC Password
+                    description: The password for the RPC user for Bitcoin Core
+                    subtype: app
+                    app-id: bitcoind
+                    target: config
+                    index: "rpc.password"
+            external:
+                addressext:
+                    type: string
+                    name: Public Address
+                    description: The public address of your Bitcoin Core RPC server
+                    nullable: false
+                userext:
+                    type: string
+                    name: RPC Username
+                    description: The username for the RPC user on your Bitcoin Core RPC server
+                    nullable: false
+                passwordext:
+                    type: string
+                    name: RPC Password
+                    description: The password for the RPC user on your Bitcoin Core RPC server
+                    nullable: false
+            quick-connect:
+                quick-connect-url:
+                    type: string
+                    name: Quick Connect URL
+                    description: The Quick Connect URL for your Bitcoin Core RPC server
+                    nullable: false
+                    pattern: 'btcstandup://[^:]*:[^@]*@[a-zA-Z0-9.-]+:[0-9]+(/(\?(label=.+)?)?)?'
+                    patternDescription: Must be a valid Quick Connect URL. For help, check out https://github.com/BlockchainCommons/Gordian/blob/master/Docs/Quick-Connect-API.md
+
+
+.. _config_rules:
+
+Config Rules
+============
+
+This file defines the configuration rules, or the rule-set that defines dependencies between config variables. 
+
+A rule is a boolean expression that we demand to be true. It is not true if the expression fails the rule parser.
+
+They follow the `Backus–Naur <https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_form>`_ meta-syntax for writing rules.
+
+Rules are composed of two main concepts:
+
+* Variables - accessor into a configuration
+* Terms - either a variable or type literal (ie. a boolean term is a boolean variable, a boolean expression, or a comparison operation between numbers or strings)
+
+Variables can be booleans, numbers, or strings, and have a different syntax depending on the type:
+
+- ``?`` - Casts to boolean value. If the value is not a boolean, this notes whether or not the value is null.
+- ``#`` - Treat the value as a number. If it is not a number, the value will be parsed as NaN. String numbers are not currently supported. 
+- ``'`` - Cast the value into a string. Applies to any value except for an object or a list.
+- ``!`` - Equals not.
+
+If application does not satisfy a rule, a set of suggestions should be provided. These suggestions are in the form of the operation to preform:
+
+    - ``Set`` - set the value 
+
+    - ``Push`` - add to the value (such as to a list)
+
+    - ``Delete`` - delete the value
+
+.. code:: typescript 
+
+    enum SuggestionVariant = Set | Delete | Push
+
+    interface Set {
+        var: String,
+        to: SetVariant,
+    }
+    interface Delete {
+        src: String,
+    }
+    interface Push {
+        to: String,
+        value: Value,
+    }
+
+    enum SetVariant = To | ToValue | ToEntropy
+
+    type To = String
+    type ToValue = Value
+    interface ToEntropy {
+        charset: String (eg. 'a-z,A-Z,0-9')
+        len: Number
+    }
+
+SetVariant Examples:
+
+.. code:: yaml
+
+    to: 'users'
+
+    to-entropy: {
+        charset: 'a-z,A-Z,0-9'
+        len: 22
+    }
+
+    to: true
+
+.. code:: yaml
+
+    - rule: 'rpc.enable? OR !(''advanced.pruning.mode = "manual")'
+      description: "RPC must be enabled for manual pruning."
+
+.. role:: raw-html(raw)
+    :format: html
+
+:raw-html:`<br />`

source/contributing/services/docker.rst

@@ -0,0 +1,34 @@
+.. _service_docker:
+
+******************
+Service Dockerfile
+******************
+
+Dockerfile
+==========
+
+This file defines how to build the image for the service. It declares the environment and building stages.
+The Dockerfile is responsible for mounting the service application to whatever volume the manifest specifies, typically ``/root``.
+
+Since the build requires specific arm runtime environments, these base images can be imported into the Dockerfile as a starting point so the base system does not need to be completely redefined. This enables importing a specific build environment version that encapsulates any necessary libraries for running the service build, eg. golang, rust.
+
+For instance: 
+
+``FROM alpine:3.12``
+``FROM arm32v7/alpine``
+``arm32v7/golang:alpine``
+
+
+Entry-point
+===========
+
+File specified as: ``docker_entrypoint.sh``
+
+This file defines what to do when the service application starts.
+
+It consists of a bash script that completes any environment setup (eg. creating folder substructure), sets any environment variables, and executes the run command. The only required feature of this file is to execute the run commands on EmbassyOS.
+
+.. role:: raw-html(raw)
+    :format: html
+
+:raw-html:`<br />`

source/contributing/services/docs.rst

@@ -0,0 +1,18 @@
+.. _service_docs:
+
+*********************
+Service Documentation
+*********************
+
+Within each wrapper repository should exist a ``docs`` directory. This folder should include any pertinent documentation, instructions, external integrations, or other details about the service that users or developers might find relevant.
+
+If an ``instructions.md`` file is included, this markdown formatted file will be rendered in the "Instructions" tab within the service detail menu on EmbassyOS:
+
+.. figure:: /_static/images/service/bitcoin_instructions.png
+  :width: 80%
+  :alt: Bitcoin Instructions
+
+.. role:: raw-html(raw)
+    :format: html
+
+:raw-html:`<br />`

source/contributing/services/makefile.rst

@@ -0,0 +1,23 @@
+.. _service_makefile:
+
+****************
+Service Makefile
+****************
+
+*This file is optional*
+
+A Makefile serves as a convenience for building and for additional developer contributions to your project. Please reference the GNU `documentation <https://www.gnu.org/software/make/manual/html_node/Introduction.html>`_ for implementation details.
+
+An alternative to using ``make`` is to use the `nix <https://nixos.wiki/wiki/Nix>`_ specification.
+
+This purpose of this file is to:
+
+- Read the docker container and build the project
+- Build all prerequisite for running the docker file
+- Build all dependencies 
+- Package ``config.yaml``, ``manifest.yaml``, and ``image.tar`` into an ``.s9pk`` extension by invoke ``appmgr``. 
+
+.. role:: raw-html(raw)
+    :format: html
+
+:raw-html:`<br />`

source/contributing/services/manifest.rst

@@ -0,0 +1,346 @@
+.. _service_manifest:
+
+****************
+Service Manifest
+****************
+
+Overview
+========
+
+This file describes the service and it's requirements. It is used by the marketplace to create a service listing. 
+
+Important of note is the :ref:`dependencies <dependencies>` key, which contains rules for optional or required other services that are depended on for the service to successfully run.
+
+Formatting
+==========
+
+- Serialization language:``.yaml``
+- Case style: ``kebab-case``
+
+Type definitions
+================
+Below are the types and sub-type definitions, with necessary elaborations. Any item that contains ``Option<>`` is an optional field.
+
+Example
+=======
+
+.. code:: yaml
+
+    # manifest version compatibility ie. v0 (this is currently the only option)
+    compat: v0
+    # service id, used for uniqueness and BE identification ie. bitcoind
+    id: String
+    # version number of the release conforming to the semantic versioning scheme
+    version: String
+    # display name of service
+    title: String
+    # an object containing a short and long description of the service. TODO character lengths
+    description:
+        short: String
+        long: String
+    # a link to the release tag notes in GitHub, or a short description TODO character length
+    release_notes: String
+    # a notification message that should caution the user with any service particularities, eg. beta tech
+    install_alert: Option<String>
+    # a notification message warning users of potential problems with uninstalling, such as dependency failures or data loss
+    uninstall_alert: Option<String>
+    # a notification message containing warning or details about backup restoration
+    restore_alert: Option<String>
+    # denoting the existence of instructions.md
+    has_instructions: Boolean
+    # required semver version range of EmbassyOS to run service eg. ">=1.0.0"
+    os_version_required: VersionReq
+    # recommended semver version range of EmbassyOS to run service eg."^1.0.0"
+    os_version_recommended: VersionReq
+    # a list of objects of ports to run the service on localhost and tor
+    ports:
+        - internal: String
+          tor: String
+    # currently only tar is supported
+    image:
+        type: String
+    # shared memory container size
+    shm_size_mb: Option<usize>
+    # path to mount the image on the volume, ie: /root/bitcoind
+    mount: String
+    # public directory path
+    public: Option<String>
+    # shared directory path
+    shared: Option<String>
+    # a list of objecting containing the source and destination directories of persistent assets, either that should be copied over during build, or to persist when service started, and if the volume directory should be overwritten when the release is copied over
+    # src: path to file within the assets directory that is in the build directory
+    # dst: path within volume to place it
+    assets:
+      - src: String TODO confirm type
+        dst: String TODO confirm type
+        overwrite: Boolean
+    # version of tor support, eg. v1, v2, v3
+    hidden_service_version: String
+    # A map of dependent services, see below for more details
+    dependencies: Dependencies
+
+.. _dependencies:
+
+Dependencies
+------------
+
+Many services depend on other libraries and services on EmbassyOS (such as Bitcoin), sometimes even a particular version of those services, which need to be specified by the developers so that EmbassyOS can handle installing these dependencies under the hood.
+
+The key of each field in the dependencies object is the lowercase, kebab-case app ID of the service that is depended on. Each dependency contains a set of rules that need to be fulfilled as true if the dependency is to be properly installed. The interface should provide suggestions for the behavior if the denoted rule cannot be met with previous configurations.
+
+Let's take this snippet for example:
+
+.. code:: yaml
+
+    ...
+    dependencies:
+        btc-rpc-proxy:
+            version: "^0.1.0"
+            optional: Can configure an external bitcoin node.
+            description: Required for fetching validated blocks.
+            config:
+            - rule: '''users.*.name = "c-lightning"'
+                description: 'Must have an RPC user named "c-lightning"'
+                suggestions:
+                - PUSH:
+                    to: 'users'
+                    value:
+                        name: c-lightning
+    ...
+
+.. role:: raw-html(raw)
+    :format: html
+
+:raw-html:`<br />`
+
+The service ``btc-rpc-proxy`` is a dependency of the service ``c-lightning``. ``c-lightning`` requires it to be installed at a version >=0.1.0 <0.2.0. There exists a rule that states the config option ``user.name`` must be equal to "c-lightning". If this value does not exist for ``user.name`` when accessed, ``PUSH`` the value "c-lighting" to the field. 
+
+Types for ``manifest.yaml`` fields:
+
+.. code:: typescript
+
+    interface Dependencies [{
+        serviceId: DepInfo
+    }]
+
+    interface DepInfo {
+        version: String // ie. 0.11.1.1
+        optional?: String,
+        description?: String,
+        config: [ConfigRule],
+        ],
+    }
+
+    interface ConfigRule {
+        rule: String, // ie. '''users.*.name = "lnd"'
+        description: String,
+        suggestions: [SuggestionVariant]
+    }
+
+    interface SuggestionVariant {
+        SET: {
+            var: String,
+            to: SetVariant,
+        },
+        DELETE: {
+            src: String,
+        },
+        PUSH: {
+            to: String,
+            value: Value,
+        },
+    }
+
+    interface SetVariant {
+        to: Option<String>,
+        to-value: Option<Value>, // ie. true/false
+        to-entropy: Option<{
+            charset: String // ie. 'a-z,A-Z,0-9'
+            len: number
+        }>
+    }
+
+.. role:: raw-html(raw)
+    :format: html
+
+:raw-html:`<br />`
+
+----
+
+Examples
+--------
+
+Actual ``manifest.yaml`` files for existing services:
+
+LND
+^^^
+
+.. code:: yaml
+
+    compat: v0
+    id: lnd
+    version: 0.11.1.1
+    title: Lightning Network Daemon
+    description:
+    short: "A complete implementation of a Lightning Network node by Lightning Labs"
+    long: "LND fully conforms to the Lightning Network specification (BOLTs). BOLT stands for: Basis of Lightning Technology. In the current state lnd is capable of: creating channels, closing channels, managing all channel states (including the exceptional ones!), maintaining a fully authenticated+validated channel graph, performing path finding within the network, passively forwarding incoming payments, sending outgoing onion-encrypted payments through the network, updating advertised fee schedules, and automatic channel management (autopilot)."
+    release-notes: https://github.com/lightningnetwork/lnd/releases/tag/v0.11.1-beta
+    ports:
+    - internal: 8080
+        tor: 8080
+    - internal: 9735
+        tor: 9735
+    - internal: 9911
+        tor: 9911
+    - internal: 10009
+        tor: 10009
+    image:
+    type: tar
+    mount: /root/.lnd
+    public: public
+    has-instructions: true
+    os-version-required: ">=0.2.8"
+    os-version-recommended: ">=0.2.8"
+    install-alert: |
+    READ CAREFULLY! LND and the Lightning Network are considered beta software. Please use with caution and do not risk more money than you are willing to lose. We encourage frequent backups. If for any reason, you need to restore LND from a backup, your on-chain wallet will be restored, but all your channels will be closed and their funds returned to your on-chain wallet, minus fees. It may also take some time for this process to occur.
+    uninstall-alert: "READ CAREFULLY! Uninstalling LND will result in permanent loss of data, including its private keys for its on-chain wallet and all channel states. Please make a backup if you have any funds in your on-chain wallet or in any channels. Recovering from backup will restore your on-chain wallet, but due to the architecture of the Lightning Network, your channels cannot be recovered. All your channels will be closed and their funds returned to your on-chain wallet, minus fees. \n"
+    restore-alert: |
+    Restoring LND will overwrite its current data, including its on-chain wallet and channels. Any channels opened since the last backup will be forgotten and may linger indefinitely, and channels contained in the backup will be closed and their funds returned to your on-chain wallet, minus fees.
+    assets: []
+    hidden-service-version: v3
+    dependencies:
+    btc-rpc-proxy:
+        version: "^0.2.4"
+        optional: Can alternatively configure an external bitcoin node.
+        description: Used to fetch validated blocks.
+        config:
+        - rule: '''users.*.name = "lnd"'
+            description: 'Must have an RPC user named "lnd"'
+            suggestions:
+            - PUSH:
+                to: "users"
+                value:
+                    name: lnd
+                    allowed-calls: []
+            - SET:
+                var: 'users.[first(item => ''item.name = "lnd")].password'
+                to-entropy:
+                    charset: "a-z,A-Z,0-9"
+                    len: 22
+        - rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getinfo"'
+            description: 'RPC user "lnd" must have "getinfo" enabled'
+            suggestions:
+            - PUSH:
+                to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
+                value: "getinfo"
+        - rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getbestblockhash"'
+            description: 'RPC user "lnd" must have "getbestblockhash" enabled'
+            suggestions:
+            - PUSH:
+                to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
+                value: "getbestblockhash"
+        - rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "gettxout"'
+            description: 'RPC user "lnd" must have "gettxout" enabled'
+            suggestions:
+            - PUSH:
+                to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
+                value: "gettxout"
+        - rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getblockchaininfo"'
+            description: 'RPC user "lnd" must have "getblockchaininfo" enabled'
+            suggestions:
+            - PUSH:
+                to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
+                value: "getblockchaininfo"
+        - rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "sendrawtransaction"'
+            description: 'RPC user "lnd" must have "sendrawtransaction" enabled'
+            suggestions:
+            - PUSH:
+                to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
+                value: "sendrawtransaction"
+        - rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getblockhash"'
+            description: 'RPC user "lnd" must have "getblockhash" enabled'
+            suggestions:
+            - PUSH:
+                to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
+                value: "getblockhash"
+        - rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getblock"'
+            description: 'RPC user "lnd" must have "getblock" enabled'
+            suggestions:
+            - PUSH:
+                to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
+                value: "getblock"
+        - rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getblockheader"'
+            description: 'RPC user "lnd" must have "getblockheader" enabled'
+            suggestions:
+            - PUSH:
+                to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
+                value: "getblockheader"
+        - rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "estimatesmartfee"'
+            description: 'RPC user "lnd" must have "estimatesmartfee" enabled'
+            suggestions:
+            - PUSH:
+                to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
+                value: "estimatesmartfee"
+        - rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getnetworkinfo"'
+            description: 'RPC user "lnd" must have "getnetworkinfo" enabled'
+            suggestions:
+            - PUSH:
+                to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
+                value: "getnetworkinfo"
+        - rule: 'users.[first(item => ''item.name = "lnd")].fetch-blocks?'
+            description: 'RPC user "lnd" must have "Fetch Blocks" enabled'
+            suggestions:
+            - SET:
+                var: 'users.[first(item => ''item.name = "lnd")].fetch-blocks'
+                to-value: true
+    bitcoind:
+        version: "^0.21.0"
+        optional: Can alternatively configure an external bitcoin node.
+        description: Used to subscribe to new block events.
+        config:
+        - rule: "zmq-enabled?"
+            description: "Must have an ZeroMQ enabled"
+            suggestions:
+            - SET:
+                var: "zmq-enabled"
+                to-value: true
+
+Cups
+^^^^
+
+.. code:: yaml
+
+    compat: v0
+    id: cups
+    version: "0.3.6"
+    title: "Cups Messenger"
+    description:
+    short: "Real private messaging"
+    long: "Cups is a private, self-hosted, peer-to-peer, Tor-based, instant messenger. Unlike other end-to-end encrypted messengers, with Cups on the Embassy there are no trusted third parties."
+    release-notes: |
+    Features
+        - Adds instructions defined by EmbassyOS 0.2.4 instructions feature
+    ports:
+        - internal: 59001
+            tor: 59001
+        - internal: 80
+            tor: 80
+    image:
+    type: tar
+    mount: /root
+    has-instructions: true
+    os-version-required: ">=0.2.4"
+    os-version-recommended: ">=0.2.4"
+    assets:
+        - src: httpd.conf
+            dst: "."
+            overwrite: true
+        - src: www
+            dst: "."
+            overwrite: true
+    hidden-service-version: v3
+
+.. role:: raw-html(raw)
+    :format: html
+
+:raw-html:`<br />`

source/contributing/services/overview.rst

@@ -0,0 +1,48 @@
+.. _service_package_overview:
+
+************************
+Service Package Overview
+************************
+
+Let's start at a high level. What actually is a service, in a technical sense, on the Embassy?
+
+Each service is an image. An image is a snapshot of a linux environment in which an application runs. All service application data is stored in a volume directory. This volume is persisted across updates. The image is immutable: when the service is updated, the image is replaced with the complete new image containing the updated features. 
+
+The volume directory (for seeding data into the volume) is typically: ``/root/volumes/<serviceId>``.
+
+.. important::
+    Any files that are in the image at the volume path will be overwritten when a backup restore occurs.
+
+A service is mounted to the EmbassyOS image. Each service application gets a volume allocated to it by EmbassyOS. Within the Docker container that encompasses the service image, a path is specified as to where it should mount on EmbassyOS. This path is specified in the :ref:`manifest <service_manifest>`.
+
+The config spec, config rules, and manifest get bundled into the ``.s9pk`` package, `built using appmgr <https://github.com/Start9Labs/appmgr>`_. Each new version release should include the updated version of these files re-bundled into a new binary. 
+
+.. note::
+    Due to tech debt, code in appmgr may reference ``app-id`` instead of ``service-id``.
+
+Environment Setup
+=================
+
+At minimum, docker, docker-builx, and appmgr are required to package a service for EmbassyOS.
+
+Recommended Dependencies
+------------------------
+
+`docker <https://docs.docker.com/get-docker>`_
+
+`docker-buildx <https://docs.docker.com/buildx/working-with-buildx/>`_
+
+`appmgr <https://github.com/Start9Labs/embassy-os/tree/master/appmgr>`_
+
+`cargo <https://doc.rust-lang.org/cargo/>`_
+
+`yq (version 4) <https://mikefarah.gitbook.io/yq>`_
+
+`make <https://www.gnu.org/software/make/>`_
+
+`musl <https://github.com/Start9Labs/rust-musl-cross>`_
+
+.. role:: raw-html(raw)
+    :format: html
+
+:raw-html:`<br />`

source/contributing/services/properties.rst

@@ -0,0 +1,22 @@
+.. _service_properties:
+
+******************
+Service Properties
+******************
+
+The output of this step is a file titled ``stats.yaml``. This file contains a mapping of the values that will be displayed in the ``Properties`` section in a service's menu.
+
+.. figure:: /_static/images/service/service_properties.png
+  :width: 40%
+  :alt: Service Properties
+
+  Service Properties Tab
+
+Due to the fact that config variables are only available when the service is running, this file can only be populated at runtime. In services Start9 has wrapped or created, we typically handle this in ``configurator/src/main.rs``. This file is essentially a script that acquires the necessary values at runtime from config (eg. ``/root/.lnd/start9/config.yaml``), and dynamically populating them to the ``stats.yaml`` file, marking each value with the appropriate keys for UI display (ie. masked or copyable). Running the configurator is typically included in the ``docker_entrypoint.sh`` file.
+
+A good example of the a configurator producing the ``stats.yaml`` file can be found `here <https://github.com/Start9Labs/btc-rpc-proxy-wrapper/blob/master/configurator/src/main.rs>`_.
+
+.. role:: raw-html(raw)
+    :format: html
+
+:raw-html:`<br />`

source/contributing/services/wrapper.rst

@@ -0,0 +1,60 @@
+.. _service_wrapper:
+
+***************
+Service Wrapper
+***************
+
+Each service is bound with a wrapper repository, which contains everything you need to build a service.
+
+The purpose of this repo is:
+
+- Denote any dependencies required to run and build the project
+- To define the necessary, ``config_rules.yaml``, ``config_spec.yaml`` and ``manifest.yaml`` options
+- To build the project into the ``.s9pk`` format digestible to EmbassyOS
+- Link to the source project as a git submodule
+- Define the docker file for running the project on EmbassyOS
+- Provide documentation for the project, especially user runbook instructions
+- symlink of ``instructions.md`` from ``docs`` directory to wrapper repo root, if included
+
+File Structure
+==============
+
+The project structure should be used as a model:
+
+.. code-block:: bash
+
+    ├── Dockerfile
+    ├── Makefile (optional)
+    ├── README.md
+    ├── assets
+    │   ├── httpd.conf (optional)
+    │   └── httpd.conf.template (optional)
+    ├── config_rules.yaml
+    ├── config_spec.yaml
+    ├── <submodule_project_dir>
+    ├── docker_entrypoint.sh
+    ├── docs
+    │   └── instructions.md
+    ├── instructions.md -> docs/instructions.md (symlink)
+    └── manifest.yaml
+
+Submodule
+==========
+
+Git submodules allow use of another project while in the working project directory. Setting up this feature enables linking of the source service repository so that it's context is available.
+
+Run ``git submodule add <link_to_source_project>``
+
+For reference: `git-submodules <https://www.git-scm.com/book/en/v2/Git-Tools-Submodules>`_
+
+Assets
+======
+
+Whenever a service is stopped, any file that is not contained within in the ``/assets`` directory will be cleared from memory. Any unsaved changes will be reverted. This folder acts as a persistance storage container.
+
+In this folder belongs any assets that are unique configurations to your service. For instance, bitcoind's ``.conf`` file is saved here. 
+
+.. role:: raw-html(raw)
+    :format: html
+
+:raw-html:`<br />`

source/index.rst

@@ -35,7 +35,7 @@ Here you will find guidance and information about the Start9 Embassy personal se
   :maxdepth: 2
   :caption: Contributing
 
-  contributing/adding-services
+  contributing/index
 
 .. toctree::
   :maxdepth: 2

source/misc-guides/tor-firefox/android.rst

@@ -11,7 +11,7 @@ Once Tor is setup on your system, you will need to edit your phone's private DNS
 
 Navigate to `Network & Internet > Advanced > Private DNS` and toggle Private DNS Mode "off".
 
-.. figure:: /_static/images/private_dns.png
+.. figure:: /_static/images/tor/private_dns_off.png
   :width: 50%
   :alt: Private DNS toggle