documentation/site/source/support/dev-docs/packaging-example.rst

.. _packaging-example:

=========================
Service Packaging Example
=========================

.. contents::
   :depth: 4
   :local:


Welcome!  The following guide will provide the prerequisites, introduce a brief overview of the packaging process, use an example demonstrating how to package a service, and finally, describe the submission process.  This essentially describes how you can take an existing app (or one you have written yourself), and wrap it up into an ``s9pk`` such that it can be added to an EmbassyOS (EOS) Marketplace!  The ``s9pk`` is the final product, which is the portable version of a package that is understood by EOS, and can be distributed to any EOS users either directly, or via a Marketplace.

Pre-requisites
--------------

EmbassyOS (EOS)
===============

It is **HIGHLY RECOMMENDED** to have a copy of EmbassyOS for testing your packaged service.

There are 3 options for this:
    #. Build from `source <https://github.com/Start9Labs/embassy-os/build>`_
    #. Follow the :ref:`DIY guide <diy>` to build on a Raspberry Pi
    #. :ref:`Purchase <purchasing>` a device or copy of the OS

Development Environment
=======================

Once you have EOS installed, you'll want to get your development system set up with the necessary software.

At minimum you will need the following:
    #. `Docker <https://docs.docker.com/get-docker>`_
    #. `Docker-buildx <https://docs.docker.com/buildx/working-with-buildx/>`_
    #. `Embassy-SDK <https://github.com/Start9Labs/embassy-os/tree/master/backend>`_

The following are recommended:
    #. `Cargo <https://doc.rust-lang.org/cargo/>`_
    #. `yq <https://mikefarah.gitbook.io/yq/>`_ (version 4)
    #. `make <https://www.gnu.org/software/make/>`_
    #. `rust-musl-cross <https://github.com/Start9Labs/rust-musl-cross>`_ (For cross compiling Rust to Alpine, not needed otherwise)

Overview
--------

Components
==========

The package is made up of the following parts:

    #. Container Image - Each service is running in a container (typically Docker).  Best results will come from an arm based linux; [Alpine](https://www.alpinelinux.org/) is highly recommended.
    #. Volume(s) - Each service gets one or more volumes, allocated by EOS.  This is the filesystem where the service data will be stored and mounted by the container. The volume directory within EOS (for seeding data into the volume) is located at `/embassy-data/package-data/volumes/<service-id>`
    #. Dependencies - Rules and requirements of your service, which appear as UI elements, such as inputs, toggles, and drop-downs.  These are enforced by validations and clear user instructions.  EmbassyOS has a unique and powerful system for managing dependencies which allows anyone to have the power of systems administrators without an advanced skillset.
    #. Manifest - Describes the service and its requirements.  This file describes the marketplace listing, installation considerations, configuration and dependency requirements, health checks, backups and additional info.
    #. Config - EOS makes a service's configuration available to the user in the GUI and must be valid regardless of user skill.
    #. Instructions & Docs - Directions for setup, use, external integrations, etc.
    #. License - The Open Source license of the service wapper.
    #. Icon - Image used to identify the service in the UI.

The ``.s9pk`` is the image, manifest, license, icon, and instructions files bundled into a package. Optional additional assets for use with system images can also be bundled. This is the file a user downloads from the Marketplace, at which point EOS uses to unpack assets and install the service.

Check :ref:`here <service_package_overview>` for a detailed overview of package components.

Service Wrapper Repo and Submodules
===================================

See :ref:`here <service_wrapper>` for how to structure your service wrapper's git repository.

Git submodules allow the use of another project while in the working project directory.  In this case, you can use an existing app's git repo in order to source its code in your service wrapper.  You can add the submodule into the wrapper so that you can build the submodule and also track the exact code you're building.

Simply run:

  .. code:: bash

    git submodule add <link_to_source_project>

Example - Hello World
---------------------

Okay, let's actually package a service!  For this example, we're going to use an example service `Hello World <https://github.com/Start9Labs/hello-world>`_.  This repository can also be used as a template to quickly get started with your service.  The guide will provide good overview of service packaging, but obviously your app will be different, so don't hesitate to reach out to our community `dev chat <https://matrix.to/#/#community-dev:matrix.start9labs.com>`_ with questions.

The commands below assume a Linux development environment with all the recommended dependencies listed above installed.  To get started quickly, we'll use Start9's wrapper template.

Clone the Template Repo and Edit the Manifest
=============================================

#. Clone and rename the repo (or alternatively, use the template generation button found on the github `repo <https://github.com/Start9Labs/hello-world-wrapper>`_)

  .. code-block:: bash

    git clone https://github.com/Start9Labs/hello-world-wrapper
    cd hello-world-wrapper

#. Edit the ``README.md`` to explain what the service is, what dependencies are required, build/install/contribute instructions, and any other information you'd like.

#. Edit the ``manifest`` file.  This must be in ``.json``, ``.toml``, or ``.yaml`` format and in ``kebab-case`` style.  You can see descriptions of each key (and some notes) in our 'Hello World' example ``manifest.yaml`` below:

Manifest example
================

.. code-block:: yaml

  id: hello-world
  title: "Hello World"
  version: 0.2.0 # Service version
  release-notes: "Upgrade to EmbassyOS 2.16.0 and then to v0.3.0"
  license: mit
  wrapper-repo: "https://github.com/Start9Labs/hello-world-wrapper"
  upstream-repo: "https://github.com/Start9Labs/hello-world-wrapper" # There is no upstream repo in this example
  support-site: "https://docs.start9.com/"
  marketing-site: "https://start9.com/"
  build: ["make"] # Series of commands to build into an s9pk, in this case we are using make and all the build commands are in the makefile
  min-os-version: "0.3.0" # Minimum required version of EmbassyOS
  description:
    short: Example service
    long: |
      Hello World is a simple example of a service wrapper that launches a web interface to say hello and nothing more.
  assets:
    license: LICENSE # default = LICENSE.md
    icon: icon.png # default = icon.png
    instructions: docs/instructions.md # default = INSTRUCTIONS.md
    docker-images: image.tar # default = image.tar
  main:
    type: docker
    image: main
    entrypoint: "/usr/local/bin/docker_entrypoint.sh"
    args: []
    mounts: {} # Specifies where to put volumes, if there are any.  Empty in this example
  health-checks: {} # Health check config would go here
  config: ~ # Configuration options, none for hello-world, but see below example for format:
  #    get:
  #     type: docker
  #     image: compat
  #     entrypoint: compat
  #     args:
  #       - "config"
  #       - "get"
  #       - "/root/.hello-world/start9/config.yaml"
  #       - "/mnt/assets/config_spec.yaml"
  #     mounts:
  #       compat: "/mnt/assets"
  #       main: "/root"
  #     io-format: yaml
  #   set:
  #     type: docker
  #     image: compat
  #     entrypoint: compat
  #     args:
  #       - "config"
  #       - "set"
  #       - "/root/.hello-world/start9/config.yaml"
  #     mounts:
  #       main: "/root"
  #     io-format: yaml
  dependencies: {} # Service pre-requisites, none for hello-world, but see below example (which would make BTC Proxy a dependency) for format:
    # btc-rpc-proxy:
    #   version: ">=0.3.2.1 <0.4.0"
    #   recommended: true
    #   critical: false
    #   optional: Can alternatively configure an external bitcoin node.
    #   description: Used to fetch validated blocks.
    #   config:
    #     check:
    #       type: docker
    #       image: compat
    #       system: true
    #       # the compat image will contain a tool to check the config rules against the specified dependency
    #       entrypoint: compat
    #       args:
    #         - "dependency"
    #         - "check"
    #         - "/mnt/assets/btc-rpc-proxy.rules.yaml"
    #       mounts:
    #         compat: "/mnt/assets"
    #     auto-configure:
    #       type: docker
    #       image: compat
    #       # if true, the denoted image is prebuilt and comes stock with EOS
    #       # uncommon: if you want something not prebuilt with EOS, you can bundle multiple docker images into the `image.tar` during the `make` build process
    #       system: true
    #       entrypoint: compat
    #       args:
    #         - "dependency"
    #         - "auto-configure"
    #         - "/mnt/assets/btc-rpc-proxy.rules.yaml"
    #       mounts:
    #         compat: "/mnt/assets"
  volumes: # this is the image where data will go from 0.2.x
    main:
      type: data # this is the image where files from the project asset directory will go
    compat:
      type: assets
  interfaces:
    main:
      name: Network Interface
      description: Specifies the interface to listen on for HTTP connections.
      tor-config:
        port-mapping:
          80: "80"
      lan-config:
        80:
          ssl: false
          mapping: 80
      ui: true
      protocols:
        - tcp
        - http
  alerts: {}
  backup:
    create:
      type: docker
      image: compat # default backup process of the compat docker image is duplicity - EOS will have access to the image defined here
      system: true
      entrypoint: compat # command to run the backup executable, in this case, duplicity
      args: # arguments to pass into the entrypoint, in this case duplicity - in this example, the full command run will be: `duplicity hello-world file:///mnt/backup /root`
        - duplicity
        - hello-world
        - /mnt/backup
        - /root
      mounts:
        # BACKUP is the default volume that is used for backups.  This is whatever backup drive is mounted to the defice, or a network filesystem.
        # The value here donates where the data mount point will be.  Backup drive is mounted to this location, which contains previous backups.
        BACKUP: "/mnt/backup"
        main: "/root"
    restore:
      type: docker
      image: compat
      system: true
      entrypoint: compat
      args:
        - duplicity
        - hello-world
        - /root
        - /mnt/backup
      mounts:
        # See above comments under `backup: -> mounts:`
        BACKUP: "/mnt/backup"
        main: "/root"
  actions: {} # Commands that can be issued from the UI.  None for hello-world, but see below example (resetting a root user) for format:
    # reset-root-user:
    #   name: Reset Root User
    #   description: Resets your root user (the first user) to username "admin" and a random password; restores any lost admin privileges.
    #   warning: This will invalidate existing sessions and password managers if you have them set up.
    #   allowed-statuses:
    #     - stopped
    #   implementation:
    #     type: docker
    #     image: main
    #     system: true
    #     entrypoint: docker_entrypoint.sh
    #     args: ["reset-root-user"]
    #     mounts:
    #       main: "/root"

Note the ``dependencies`` and ``volumes`` sections, which may access another service, e.g. File Browser, such that files stored on a user's Embassy can be accessed in your service.

For details on all the different possible dependency, type, and subtype definitions available for the ``manifest`` file, please see :ref:`here <service_manifest>`.

Edit the Dockerfile and Entrypoint
==================================

Next, it's time to edit the ``Dockerfile``.  This defines how to build the image for the package by declaring the environment, building stages, and copying any binaries or assets needed to run the service or its health checks to the image filesystem.

#. We start by importing a base image, in this case Alpine, as recommended.

  .. code:: docker

    FROM arm64v8/alpine:3.12

#. Next we issue some commands, which in this example simply updates repositories, installs required software, and finally creates a directory for nginx.

  .. code:: docker

    RUN apk update
    RUN apk add tini

    RUN mkdir /run/nginx

#. Next we will add the cross-compiled binary of ``hello-world`` to ``/usr/local/bin/`` and add the ``docker_entrypoint.sh`` file from the repository.  Then we set permissions for ``docker_entrypoint.sh``.

  .. code:: docker

    ADD ./hello-world/target/aarch64-unknown-linux-musl/release/hello-world /usr/local/bin/hello-world
    ADD ./docker_entrypoint.sh /usr/local/bin/docker_entrypoint.sh
    RUN chmod a+x /usr/local/bin/docker_entrypoint.sh

#. Next, we set a working directory, and set the location of the entrypoint. Exposing ports is not necessary for EOS, but its often useful to leave this line for clarity.

  .. code:: docker

    WORKDIR /root

    EXPOSE 80

    ENTRYPOINT ["/usr/local/bin/docker_entrypoint.sh"]

#. Great, let's take a look at our final Hello World ``Dockerfile``:

  .. code:: docker

    FROM arm64v8/alpine:3.12

    RUN apk update
    RUN apk add tini

    ADD ./hello-world/target/aarch64-unknown-linux-musl/release/hello-world /usr/local/bin/hello-world
    ADD ./docker_entrypoint.sh /usr/local/bin/docker_entrypoint.sh
    RUN chmod a+x /usr/local/bin/docker_entrypoint.sh

    WORKDIR /root

    EXPOSE 80

    ENTRYPOINT ["/usr/local/bin/docker_entrypoint.sh"]

Docker Entrypoint
=================

#. Okay, let's move on to our ``docker_entrypoint.sh`` file.  This is a script that defines what to do when the service starts, and often acts as an init system.  It will need to complete any environment setup (such as folder substructure), set any environment variables, and execute the run command.  It's also PID 1 in the docker container, so it does all of the signal handling and container exits when it is stopped/exits.  If you have built a "configurator," this will also need to be called to execute here.  More on configurators below.  Let's take a look at our (extremely basic) Hello World example:

  .. code:: bash

    #!/bin/sh

    export HOST_IP=$(ip -4 route list match 0/0 | awk '{print $3}')

    exec tini hello-world

#. We've defined the file, exported the IP address of the Embassy (host), and run the program.

For a more detailed ``docker_entrypoint.sh``, please check out the `filebrowser-wrapper <https://github.com/Start9Labs/filebrowser-wrapper/blob/master/docker_entrypoint.sh>`_.  Additional details on the ``Dockerfile`` and ``docker_entrypoint`` can be found `here <https://docs.start9.com/contributing/services/docker.html>`_.

Configurators
.............

- Broadly, a "configurator" is any code that translates the configuration coming from the OS to a format the service can understand. Technically all services with a config have one of these (so, most services on Embassy). The configurator also writes the stats.yaml file, which is used in properties.

- Narrowly, a configurator is a piece of code separate from the docker_entrypoint.sh script, that does the same as the task above. If you can configure the service in bash, inside the docker_entrypoint.sh script, then you don't need a separate piece of code called "configurator", as we have in LND, CL, and synapse, for example. You might want to create a separate configurator if configuring your service would be complicated or impossible in bash. In the case of CL and LND (and bitcoind, where the configurator is actually called a "manager"), the configurator actually has a long-running element that runs concurrently with the service itself.

There's really no reason a service has to be engineered in this manner. It's really up to the package maintainer how they want to accomplish translation of the config and implementing the properties action, which doesn't even require a stats.yaml file as of 0.3.

You can check out an example configurator in the `BitcoinD Wrapper <https://github.com/Start9Labs/bitcoind-wrapper/tree/master/manager>`_, which in this case is called a manager.

Makefile (Optional)
===================

Here, we will create a ``Makefile``, which is optional, but recommended as it outlines the build and streamlines additional developer contributions.  Alternatively, you could use ``nix``, ``bash``, ``python``, ``perl``, ``ruby``, etc instead of ``make`` for build orchestration.

Our example ``Makefile`` is again fairly simple for Hello World.  Let's take a look:

.. code-block:: Makefile

  ASSETS := $(shell yq e '.assets.[].src' manifest.yaml)
  ASSET_PATHS := $(addprefix assets/,$(ASSETS))
  VERSION := $(shell toml get hello-world/Cargo.toml package.version)
  HELLO_WORLD_SRC := $(shell find ./hello-world/src) hello-world/Cargo.toml hello-world/Cargo.lock
  S9PK_PATH=$(shell find . -name hello-world.s9pk -print)

  .DELETE_ON_ERROR:

  all: verify

  verify: hello-world.s9pk $(S9PK_PATH)
	  embassy-sdk verify $(S9PK_PATH)

  # embassy-sdk pack errors come from here, check your manifest, config, instructions, and icon
  hello-world.s9pk: manifest.yaml assets/compat/config_spec.yaml config_rules.yaml image.tar docs/instructions.md $(ASSET_PATHS)
	  embassy-sdk pack

  image.tar: Dockerfile docker_entrypoint.sh hello-world/target/aarch64-unknown-linux-musl/release/hello-world
	  DOCKER_CLI_EXPERIMENTAL=enabled docker buildx build --tag start9/hello-world --platform=linux/arm64 -o type=docker,dest=image.tar .

  hello-world/target/aarch64-unknown-linux-musl/release/hello-world: $(HELLO_WORLD_SRC)
    docker run --rm -it -v ~/.cargo/registry:/root/.cargo/registry -v "$(shell pwd)"/hello-world:/home/rust/src start9/rust-musl-cross:aarch64-musl cargo +beta build --release
    docker run --rm -it -v ~/.cargo/registry:/root/.cargo/registry -v "$(shell pwd)"/hello-world:/home/rust/src start9/rust-musl-cross:aarch64-musl musl-strip target/aarch64-unknown-linux-musl/release/hello-world

  manifest.yaml: hello-world/Cargo.toml
    yq e -i '.version = $(VERSION)' manifest.yaml

#. The first 5 lines set environment variables.

#. The next line simply removes the progress of a ``make`` process if it fails.

  .. code-block:: Makefile

    .DELETE_ON_ERROR:

#. The ``all`` step is run when the ``make`` command is issued.  This attempts the ``verify`` step, which requires that the ``hello-world.s9pk`` must first be built, which first requires the ``image.tar``, and so on.  In this case, each step requires the next and each step indicates the necessary existence of a file. If an indicated file has been altered, such as the `docker_entrypoint.sh`, then any step that contains this file will be rebuilt.

#. So the ``.s9pk`` is created with the ``embassy-sdk pack`` command, supplied with the ``manifest``, ``config_spec``, previously created ``image.tar``, and ``instructions.md``.  Your project may likely also contain a ``config_rules`` file.  Some of these files we have not yet edited, but that will come shortly.

#. The ``image.tar`` is built below this, the cross-compiled ``hello-world`` source code, and ``manifest`` at the bottom.

For more details on creating a ``Makefile`` for your project, please check :ref:`here <service_makefile>`.

Service Config Specification and Rules
======================================

Most self-hosted packages require a configuration.  With EmbassyOS, these config options are provided to the user in a friendly GUI, and invalid configs are not permitted.  This allows the user to manage their software without a lot of technical skill, and minimal risk of borking their software.

In the config section of the `manifest`, you can
Two files are created in this process:

``config_spec.yaml`` for specifying all the config options your package depends on to run

``config_rules.yaml`` for defining the ruleset that defines dependencies between config variables

These are stored in ``assets/compat/`` for 0.2.x compatibility, and in ``/assets/`` for anything built for v0.3.0 and up (almost certainly what you're doing).  These files contain a detailed mapping of configuration options with acceptable values, defaults, and relational rule-sets.  Hello World has no configuration, as you can see `here <https://github.com/Start9Labs/hello-world-wrapper/blob/0.3.0/assets/compat/config_spec.yaml>`_.  Instead, let's take a look at our ``config_spec`` for Embassy Pages, which actually has some config options:

.. code-block:: yaml

  homepage:
    name: Homepage
    description: The page that will be displayed when your Embassy Pages .onion address is visited. Since this page is technically publicly accessible, you can choose to which type of page to display.
    type: union
    default: welcome
    tag:
      id: type
      name: Type
      variant-names:
        welcome: Welcome
        index: Subdomain Index
        filebrowser: Web Page
        redirect: Redirect
        fuck-off: Fuck Off
    variants:
      welcome: {}
      index: {}
      filebrowser:
        directory:
          type: string
          name: Directory Path
          description: The path to the directory in File Browser that contains the static files of your website. For example, a value of "websites/resume_site" would tell Embassy Pages to look for that directory in File Browser.
          pattern: "^(\\.|[a-zA-Z0-9_ -][a-zA-Z0-9_ .-]*|([a-zA-Z0-9_ .-][a-zA-Z0-9_ -]+\\.*)+)(/[a-zA-Z0-9_ -][a-zA-Z0-9_ .-]*|/([a-zA-Z0-9_ .-][a-zA-Z0-9_ -]+\\.*)+)*/?$"
          pattern-description: Must be a valid relative file path
          nullable: false
      redirect:
        target:
          type: string
          name: Target Subdomain
          description: The name of the subdomain to redirect users to. This must be a valid subdomain site within your Embassy Pages.
          pattern: '^[a-z-]+$'
          pattern-description: May contain only lowercase characters and hyphens.
          nullable: false
      fuck-off: {}
  subdomains:
    type: list
    name: Subdomains
    description: The websites you want to serve.
    default: []
    range: '[0, *)'
    subtype: object
    spec:
      unique-by: name
      display-as: "{{name}}"
      spec:
        name:
          type: string
          nullable: false
          name: Subdomain name
          description: The subdomain of your Embassy Pages .onion address to host the website on. For example, a value of "me" would produce a website hosted at http://me.myaddress.onion.
          pattern: "^[a-z-]+$"
          pattern-description: "May contain only lowercase characters and hyphens"
        settings:
          type: union
          name: Settings
          description: The desired behavior you want to occur when the subdomain is visited. You can either redirect to another subdomain, or load a web page from File Browser.
          default: filebrowser
          tag:
            id: type
            name: Type
            variant-names:
              filebrowser: Web Page
              redirect: Redirect
          variants:
            filebrowser:
              directory:
                type: string
                name: Directory Path
                description: The path to the directory in File Browser that contains the static files of your website. For example, a value of "websites/resume_site" would tell Embassy Pages to look for that directory in File Browser.
                pattern: "^(\\.|[a-zA-Z0-9_ -][a-zA-Z0-9_ .-]*|([a-zA-Z0-9_ .-][a-zA-Z0-9_ -]+\\.*)+)(/[a-zA-Z0-9_ -][a-zA-Z0-9_ .-]*|/([a-zA-Z0-9_ .-][a-zA-Z0-9_ -]+\\.*)+)*/?$"
                pattern-description: Must be a valid relative file path
                nullable: false
            redirect:
              target:
                type: string
                name: Target Subdomain
                description: The subdomain of your Embassy Pages .onion address to redirect to. This should be the name of another subdomain on Embassy Pages. Leave empty to redirect to the homepage.
                pattern: '^[a-z-]+$'
                pattern-description: May contain only lowercase characters and hyphens.
                nullable: false

We essentially have 2 config options (homepage and subdomains), with all of their specifications nested below them.  Looking at the homepage, it contains a ``union`` type, which is a necessary dependency, which can be of 5 variants (welcome, index, filebrowser, redirect, or fuck-off).  The below images show how this is displayed in the UI.

  .. figure:: /_static/images/dev/pages0.png
    :width: 60%
    :alt: Pages Config

  .. figure:: /_static/images/dev/pages1.png
    :width: 60%
    :alt: Pages Union

For all the possible types, please check our :ref:`Service Config Spec <service_config>`.

In our example, there is *no need* for a ``config_rules`` file.  This is because there is not a rule-set required to define dependencies between config variables.  An example of when this would be required would be the following code, from the [LND wrapper](https://github.com/Start9Labs/lnd-wrapper/blob/master/config_rules.yaml):

.. code-block:: yaml

  ---
  - rule: '!(max-chan-size?) OR !(min-chan-size?) OR (#max-chan-size > #min-chan-size)'
    description: "Maximum Channel Size must exceed Minimum Channel Size"

Here we see that a Maximum Channel Size **MUST** be one of 3 possible options in order to be a valid config.

Properties
==========

Next we need to create the Properties section for our package, to display any relevant info.  The result of this step is a ``stats.yaml`` file, which is only populated at runtime.  These commands will be issued in the ``docker_entrypoint`` file (or ``configurator``, if you are using one).

.. ***STATS.YAML IS APPARENTLY BEING DEPRECATED, THIS SECTION NEEDS COMMENT***  Possibly this is not actually the case?

Instructions
============

Instructions are the basic directions or any particular details that you would like to convey to the user to help get them on their way.  Each wrapper repo should contain a ``docs`` directory which can include anything you'd like, but specifically if you include an ``instructions.md`` file, formatted in Markdown language, it will be displayed simply for the user as shown below.

  .. figure:: /_static/images/dev/instructions.png
    :width: 60%
    :alt: Instructions

You can find the ``instructions.md`` file for Embassy Pages `here <https://github.com/Start9Labs/embassy-pages-wrapper/tree/master/docs>`_ if you are interested.

Backups
=======

Everything in the root folder of the mounted system directory will be stored in an EOS backup.  If you want to ignore any particular files for backup, you can create a ``.backupignore`` file and add the relative paths of any directories you would like ignored.

Submission Process
------------------

When you have built and tested your project for EmbassyOS, please send Start9 a submission with the project repository to dev@start9labs.com. After being reviewed for security and compatibility, the service will be deployed to the marketplace and available for all EmbassyOS users to download.

If you are deploying to an alternative marketplace, please shout it out in our community channels!