mirror of
https://github.com/Start9Labs/documentation.git
synced 2026-03-26 10:21:53 +00:00
revap dev docs
This commit is contained in:
Lucy Cifferello
@@ -1,13 +0,0 @@
|
||||
.. _dev-faq:
|
||||
|
||||
=============
|
||||
Developer FAQ
|
||||
=============
|
||||
|
||||
Frequently Asked Questions by and for developers. If you have a suggestion, please use the Github link at the top of the page.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 3
|
||||
|
||||
faq-contributing
|
||||
faq-service-packaging
|
||||
@@ -1,52 +0,0 @@
|
||||
.. _embassy-cli:
|
||||
|
||||
===========
|
||||
Embassy-CLI
|
||||
===========
|
||||
|
||||
.. warning:: This is for advanced users only!! Anything you do while SSH'd into your Embassy is NOT SUPPORTED, unless under the guidance of a Start9 technician.
|
||||
|
||||
When interacting with Embassy directly, you will invariably want to use ``embassy-cli``. Embassy-CLI can control EmbassyOS in many of the same ways that the UI can, but with finer controls and deeper insights.
|
||||
|
||||
In order to use Embassy-CLI, you will first need to authorize yourself with the following command...
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
embassy-cli auth login
|
||||
|
||||
and enter your Embassy Master Password at the prompt.
|
||||
|
||||
|
||||
You can enter ``embassy-cli help`` or ``embassy-cli <option> help`` to get an overview of capabilities:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
Embassy CLI
|
||||
|
||||
USAGE:
|
||||
embassy-cli [OPTIONS] <SUBCOMMAND>
|
||||
|
||||
FLAGS:
|
||||
--help Prints help information
|
||||
-V, --version Prints version information
|
||||
|
||||
OPTIONS:
|
||||
-c, --config <config>
|
||||
-h, --host <host>
|
||||
-p, --proxy <proxy>
|
||||
|
||||
SUBCOMMANDS:
|
||||
auth Login/logout and manage sessions
|
||||
backup Manage backups
|
||||
db Interact with the Database
|
||||
disk Information on the external drive
|
||||
echo Echoes
|
||||
git-info Print the current git hash
|
||||
help Prints this message or the help of the given subcommand(s)
|
||||
inspect Details on services and their components
|
||||
net Network information
|
||||
notification Control UI notifications
|
||||
package Interact with packages
|
||||
server EmbassyOS operations and information
|
||||
ssh Manage SSH keys
|
||||
wifi Manage WiFi networks
|
||||
@@ -1,42 +0,0 @@
|
||||
.. _embassy-sdk:
|
||||
|
||||
===========
|
||||
Embassy-SDK
|
||||
===========
|
||||
|
||||
.. warning:: This is for developers only!! Anything you do while SSH'd into your Embassy is NOT SUPPORTED, unless under the guidance of a Start9 technician.
|
||||
|
||||
You can install to your system by running the following from you ``embassy-os`` directory:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
cargo install --bin=embassy-sdk --path=. --no-default-features --verbose
|
||||
|
||||
|
||||
You can enter ``embassy-sdk help`` or ``embassy-sdk <option> help`` to get an overview of capabilities:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
embassy-sdk --help
|
||||
Embassy SDK
|
||||
|
||||
USAGE:
|
||||
embassy-sdk [OPTIONS] <SUBCOMMAND>
|
||||
|
||||
FLAGS:
|
||||
-h, --help Prints help information
|
||||
-V, --version Prints version information
|
||||
|
||||
OPTIONS:
|
||||
-c, --config <config>
|
||||
|
||||
SUBCOMMANDS:
|
||||
git-info Print current git hash
|
||||
help Prints this message or the help of the given subcommand(s)
|
||||
init Initialize
|
||||
inspect Inspect a .s9pk
|
||||
pack Package a service into the .s9pk format
|
||||
verify Verify a .s9pk is valid
|
||||
|
||||
|
||||
Please visit https://github.com/Start9Labs/embassy-os/tree/master/backend for more details
|
||||
@@ -1,22 +0,0 @@
|
||||
.. _embassyd:
|
||||
|
||||
========
|
||||
EmbassyD
|
||||
========
|
||||
|
||||
.. warning:: This is for advanced users only!! Anything you do while SSH'd into your Embassy is NOT SUPPORTED, unless under the guidance of a Start9 technician.
|
||||
|
||||
EmbassyD is the daemon that runs everything that could be considered EmbassyOS.
|
||||
|
||||
When SSH'd into your Embassy, you may see the status of EmbassyD with the following:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
systemctl status embassyd
|
||||
|
||||
If you need to restart the daemon for some reason, you can do so with:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
systemctl restart embassyd
|
||||
|
||||
@@ -1,15 +0,0 @@
|
||||
.. _dev-tools:
|
||||
|
||||
===============
|
||||
Developer Tools
|
||||
===============
|
||||
|
||||
Brief overviews of Developer Tools that will come in handy when :ref:`packaging a service<service-packaging>`, or interacting directly with the OS to learn or contribute to the project.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
embassyd
|
||||
embassy-cli
|
||||
embassy-sdk
|
||||
service-container
|
||||
@@ -1,19 +0,0 @@
|
||||
.. _exec-service-container:
|
||||
|
||||
==================
|
||||
Service Containers
|
||||
==================
|
||||
|
||||
For simplicity, EmbassyOS is designed to be completely operated via the :ref:`WebUI<web-ui>`, however, your device belongs to you and if you want to "lift the hood" and access the internals of the software, you may do so. This guide will show you how to enter a Service's :ref:`container<containers>` in order to directly interface with it and issue commands.
|
||||
|
||||
.. warning:: The following guide is for those that have advanced command line skills, or those who are being guided by a Start9 support technician. Nothing you do inside a container is supported unless under the direction of Start9. Here be dragons!!
|
||||
|
||||
This guide assumes you already have :ref:`SSH Access<ssh-setup>` setup on your machine.
|
||||
|
||||
Open a terminal, ssh into your device, and enter the following command, replacing ``<package-id>`` with the name of the service you are trying to access. You can list the services with ``embassy-cli package list`` (you will need to be logged in under ``embassy-cli`` for this). You will need to either be ``root`` or use ``sudo``.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
docker exec -it <package-id>.embassy sh
|
||||
|
||||
This will give you a shell within the container. Reminder that nothing you do in here is supported, please be careful!
|
||||
@@ -1,24 +0,0 @@
|
||||
.. _dev-docs:
|
||||
|
||||
==============
|
||||
Developer Docs
|
||||
==============
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 3
|
||||
:hidden:
|
||||
|
||||
dev-tools/index
|
||||
packaging-example
|
||||
service-packaging/index
|
||||
dev-faq/index
|
||||
|
||||
.. panel-box::
|
||||
:title: Overview
|
||||
:id: "dev-docs"
|
||||
:class: my-panel
|
||||
|
||||
* :doc:`Developer Tools <dev-tools/index>` - An overview of tools for those interested in contributing to the OS or packaging a service.
|
||||
* :doc:`Package Example <packaging-example>` - A walkthrough of how to package a service using an example.
|
||||
* :doc:`Service Packaging <service-packaging/index>` - The full specification for packaging a service.
|
||||
* :doc:`Developer FAQ <dev-faq/index>` - Frequently asked questions by developers in regard to EmbassyOS and packaging services.
|
||||
@@ -1,543 +0,0 @@
|
||||
.. _packaging-example:
|
||||
|
||||
=========================
|
||||
Service Packaging Example
|
||||
=========================
|
||||
|
||||
.. contents::
|
||||
:depth: 4
|
||||
:local:
|
||||
|
||||
|
||||
Welcome! The following guide will provide the prerequisites, introduce a brief overview of the packaging process, use an example demonstrating how to package a service, and finally, describe the submission process. This essentially describes how you can take an existing app (or one you have written yourself), and wrap it up into an ``s9pk`` such that it can be added to an EmbassyOS (EOS) Marketplace! The ``s9pk`` is the final product, which is the portable version of a package that is understood by EOS, and can be distributed to any EOS users either directly, or via a Marketplace.
|
||||
|
||||
Pre-requisites
|
||||
--------------
|
||||
|
||||
EmbassyOS (EOS)
|
||||
===============
|
||||
|
||||
It is **HIGHLY RECOMMENDED** to have a copy of EmbassyOS for testing your packaged service.
|
||||
|
||||
There are 3 options for this:
|
||||
#. Build from `source <https://github.com/Start9Labs/embassy-os/build>`_
|
||||
#. Follow the :ref:`DIY guide <diy>` to build on a Raspberry Pi
|
||||
#. :ref:`Purchase <purchasing>` a device or copy of the OS
|
||||
|
||||
Development Environment
|
||||
=======================
|
||||
|
||||
Once you have EOS installed, you'll want to get your development system set up with the necessary software.
|
||||
|
||||
At minimum you will need the following:
|
||||
#. `Docker <https://docs.docker.com/get-docker>`_
|
||||
#. `Docker-buildx <https://docs.docker.com/buildx/working-with-buildx/>`_
|
||||
#. `Embassy-SDK <https://github.com/Start9Labs/embassy-os/tree/master/backend>`_
|
||||
|
||||
The following are recommended:
|
||||
#. `Cargo <https://doc.rust-lang.org/cargo/>`_
|
||||
#. `yq <https://mikefarah.gitbook.io/yq/>`_ (version 4)
|
||||
#. `make <https://www.gnu.org/software/make/>`_
|
||||
#. `rust-musl-cross <https://github.com/Start9Labs/rust-musl-cross>`_ (For cross compiling Rust to Alpine, not needed otherwise)
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
Components
|
||||
==========
|
||||
|
||||
The package is made up of the following parts:
|
||||
|
||||
#. Container Image - Each service is running in a container (typically Docker). Best results will come from an arm based linux; [Alpine](https://www.alpinelinux.org/) is highly recommended.
|
||||
#. Volume(s) - Each service gets one or more volumes, allocated by EOS. This is the filesystem where the service data will be stored and mounted by the container. The volume directory within EOS (for seeding data into the volume) is located at `/embassy-data/package-data/volumes/<service-id>`
|
||||
#. Dependencies - Rules and requirements of your service, which appear as UI elements, such as inputs, toggles, and drop-downs. These are enforced by validations and clear user instructions. EmbassyOS has a unique and powerful system for managing dependencies which allows anyone to have the power of systems administrators without an advanced skillset.
|
||||
#. Manifest - Describes the service and its requirements. This file describes the marketplace listing, installation considerations, configuration and dependency requirements, health checks, backups and additional info.
|
||||
#. Config - EOS makes a service's configuration available to the user in the GUI and must be valid regardless of user skill.
|
||||
#. Instructions & Docs - Directions for setup, use, external integrations, etc.
|
||||
#. License - The Open Source license of the service wapper.
|
||||
#. Icon - Image used to identify the service in the UI.
|
||||
|
||||
The ``.s9pk`` is the image, manifest, license, icon, and instructions files bundled into a package. Optional additional assets for use with system images can also be bundled. This is the file a user downloads from the Marketplace, at which point EOS uses to unpack assets and install the service.
|
||||
|
||||
Check :ref:`here <service_package_overview>` for a detailed overview of package components.
|
||||
|
||||
Service Wrapper Repo and Submodules
|
||||
===================================
|
||||
|
||||
See :ref:`here <service_wrapper>` for how to structure your service wrapper's git repository.
|
||||
|
||||
Git submodules allow the use of another project while in the working project directory. In this case, you can use an existing app's git repo in order to source its code in your service wrapper. You can add the submodule into the wrapper so that you can build the submodule and also track the exact code you're building.
|
||||
|
||||
Simply run:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
git submodule add <link_to_source_project>
|
||||
|
||||
Example - Hello World
|
||||
---------------------
|
||||
|
||||
Okay, let's actually package a service! For this example, we're going to use an example service `Hello World <https://github.com/Start9Labs/hello-world>`_. This repository can also be used as a template to quickly get started with your service. The guide will provide good overview of service packaging, but obviously your app will be different, so don't hesitate to reach out to our community `dev chat <https://matrix.to/#/#community-dev:matrix.start9labs.com>`_ with questions.
|
||||
|
||||
The commands below assume a Linux development environment with all the recommended dependencies listed above installed. To get started quickly, we'll use Start9's wrapper template.
|
||||
|
||||
Clone the Template Repo and Edit the Manifest
|
||||
=============================================
|
||||
|
||||
#. Clone and rename the repo (or alternatively, use the template generation button found on the github `repo <https://github.com/Start9Labs/hello-world-wrapper>`_)
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
git clone https://github.com/Start9Labs/hello-world-wrapper
|
||||
cd hello-world-wrapper
|
||||
|
||||
#. Edit the ``README.md`` to explain what the service is, what dependencies are required, build/install/contribute instructions, and any other information you'd like.
|
||||
|
||||
#. Edit the ``manifest`` file. This must be in ``.json``, ``.toml``, or ``.yaml`` format and in ``kebab-case`` style. You can see descriptions of each key (and some notes) in our 'Hello World' example ``manifest.yaml`` below:
|
||||
|
||||
Manifest example
|
||||
================
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
id: hello-world
|
||||
title: "Hello World"
|
||||
version: 0.2.0 # Service version
|
||||
release-notes: "Upgrade to EmbassyOS 2.16.0 and then to v0.3.0"
|
||||
license: mit
|
||||
wrapper-repo: "https://github.com/Start9Labs/hello-world-wrapper"
|
||||
upstream-repo: "https://github.com/Start9Labs/hello-world-wrapper" # There is no upstream repo in this example
|
||||
support-site: "https://docs.start9.com/"
|
||||
marketing-site: "https://start9.com/"
|
||||
build: ["make"] # Series of commands to build into an s9pk, in this case we are using make and all the build commands are in the makefile
|
||||
min-os-version: "0.3.0" # Minimum required version of EmbassyOS
|
||||
description:
|
||||
short: Example service
|
||||
long: |
|
||||
Hello World is a simple example of a service wrapper that launches a web interface to say hello and nothing more.
|
||||
assets:
|
||||
license: LICENSE # default = LICENSE.md
|
||||
icon: icon.png # default = icon.png
|
||||
instructions: docs/instructions.md # default = INSTRUCTIONS.md
|
||||
docker-images: image.tar # default = image.tar
|
||||
main:
|
||||
type: docker
|
||||
image: main
|
||||
entrypoint: "/usr/local/bin/docker_entrypoint.sh"
|
||||
args: []
|
||||
mounts: {} # Specifies where to put volumes, if there are any. Empty in this example
|
||||
health-checks: {} # Health check config would go here
|
||||
config: ~ # Configuration options, none for hello-world, but see below example for format:
|
||||
# get:
|
||||
# type: docker
|
||||
# image: compat
|
||||
# entrypoint: compat
|
||||
# args:
|
||||
# - "config"
|
||||
# - "get"
|
||||
# - "/root/.hello-world/start9/config.yaml"
|
||||
# - "/mnt/assets/config_spec.yaml"
|
||||
# mounts:
|
||||
# compat: "/mnt/assets"
|
||||
# main: "/root"
|
||||
# io-format: yaml
|
||||
# set:
|
||||
# type: docker
|
||||
# image: compat
|
||||
# entrypoint: compat
|
||||
# args:
|
||||
# - "config"
|
||||
# - "set"
|
||||
# - "/root/.hello-world/start9/config.yaml"
|
||||
# mounts:
|
||||
# main: "/root"
|
||||
# io-format: yaml
|
||||
dependencies: {} # Service pre-requisites, none for hello-world, but see below example (which would make BTC Proxy a dependency) for format:
|
||||
# btc-rpc-proxy:
|
||||
# version: ">=0.3.2.1 <0.4.0"
|
||||
# recommended: true
|
||||
# critical: false
|
||||
# optional: Can alternatively configure an external bitcoin node.
|
||||
# description: Used to fetch validated blocks.
|
||||
# config:
|
||||
# check:
|
||||
# type: docker
|
||||
# image: compat
|
||||
# system: true
|
||||
# # the compat image will contain a tool to check the config rules against the specified dependency
|
||||
# entrypoint: compat
|
||||
# args:
|
||||
# - "dependency"
|
||||
# - "check"
|
||||
# - "/mnt/assets/btc-rpc-proxy.rules.yaml"
|
||||
# mounts:
|
||||
# compat: "/mnt/assets"
|
||||
# auto-configure:
|
||||
# type: docker
|
||||
# image: compat
|
||||
# # if true, the denoted image is prebuilt and comes stock with EOS
|
||||
# # uncommon: if you want something not prebuilt with EOS, you can bundle multiple docker images into the `image.tar` during the `make` build process
|
||||
# system: true
|
||||
# entrypoint: compat
|
||||
# args:
|
||||
# - "dependency"
|
||||
# - "auto-configure"
|
||||
# - "/mnt/assets/btc-rpc-proxy.rules.yaml"
|
||||
# mounts:
|
||||
# compat: "/mnt/assets"
|
||||
volumes: # this is the image where data will go from 0.2.x
|
||||
main:
|
||||
type: data # this is the image where files from the project asset directory will go
|
||||
compat:
|
||||
type: assets
|
||||
interfaces:
|
||||
main:
|
||||
name: Network Interface
|
||||
description: Specifies the interface to listen on for HTTP connections.
|
||||
tor-config:
|
||||
port-mapping:
|
||||
80: "80"
|
||||
lan-config:
|
||||
80:
|
||||
ssl: false
|
||||
mapping: 80
|
||||
ui: true
|
||||
protocols:
|
||||
- tcp
|
||||
- http
|
||||
alerts: {}
|
||||
backup:
|
||||
create:
|
||||
type: docker
|
||||
image: compat # default backup process of the compat docker image is duplicity - EOS will have access to the image defined here
|
||||
system: true
|
||||
entrypoint: compat # command to run the backup executable, in this case, duplicity
|
||||
args: # arguments to pass into the entrypoint, in this case duplicity - in this example, the full command run will be: `duplicity hello-world file:///mnt/backup /root`
|
||||
- duplicity
|
||||
- hello-world
|
||||
- /mnt/backup
|
||||
- /root
|
||||
mounts:
|
||||
# BACKUP is the default volume that is used for backups. This is whatever backup drive is mounted to the defice, or a network filesystem.
|
||||
# The value here donates where the data mount point will be. Backup drive is mounted to this location, which contains previous backups.
|
||||
BACKUP: "/mnt/backup"
|
||||
main: "/root"
|
||||
restore:
|
||||
type: docker
|
||||
image: compat
|
||||
system: true
|
||||
entrypoint: compat
|
||||
args:
|
||||
- duplicity
|
||||
- hello-world
|
||||
- /root
|
||||
- /mnt/backup
|
||||
mounts:
|
||||
# See above comments under `backup: -> mounts:`
|
||||
BACKUP: "/mnt/backup"
|
||||
main: "/root"
|
||||
actions: {} # Commands that can be issued from the UI. None for hello-world, but see below example (resetting a root user) for format:
|
||||
# reset-root-user:
|
||||
# name: Reset Root User
|
||||
# description: Resets your root user (the first user) to username "admin" and a random password; restores any lost admin privileges.
|
||||
# warning: This will invalidate existing sessions and password managers if you have them set up.
|
||||
# allowed-statuses:
|
||||
# - stopped
|
||||
# implementation:
|
||||
# type: docker
|
||||
# image: main
|
||||
# system: true
|
||||
# entrypoint: docker_entrypoint.sh
|
||||
# args: ["reset-root-user"]
|
||||
# mounts:
|
||||
# main: "/root"
|
||||
|
||||
Note the ``dependencies`` and ``volumes`` sections, which may access another service, e.g. File Browser, such that files stored on a user's Embassy can be accessed in your service.
|
||||
|
||||
For details on all the different possible dependency, type, and subtype definitions available for the ``manifest`` file, please see :ref:`here <service_manifest>`.
|
||||
|
||||
Edit the Dockerfile and Entrypoint
|
||||
==================================
|
||||
|
||||
Next, it's time to edit the ``Dockerfile``. This defines how to build the image for the package by declaring the environment, building stages, and copying any binaries or assets needed to run the service or its health checks to the image filesystem.
|
||||
|
||||
#. We start by importing a base image, in this case Alpine, as recommended.
|
||||
|
||||
.. code:: docker
|
||||
|
||||
FROM arm64v8/alpine:3.12
|
||||
|
||||
#. Next we issue some commands, which in this example simply updates repositories, installs required software, and finally creates a directory for nginx.
|
||||
|
||||
.. code:: docker
|
||||
|
||||
RUN apk update
|
||||
RUN apk add tini
|
||||
|
||||
RUN mkdir /run/nginx
|
||||
|
||||
#. Next we will add the cross-compiled binary of ``hello-world`` to ``/usr/local/bin/`` and add the ``docker_entrypoint.sh`` file from the repository. Then we set permissions for ``docker_entrypoint.sh``.
|
||||
|
||||
.. code:: docker
|
||||
|
||||
ADD ./hello-world/target/aarch64-unknown-linux-musl/release/hello-world /usr/local/bin/hello-world
|
||||
ADD ./docker_entrypoint.sh /usr/local/bin/docker_entrypoint.sh
|
||||
RUN chmod a+x /usr/local/bin/docker_entrypoint.sh
|
||||
|
||||
#. Next, we set a working directory, and set the location of the entrypoint. Exposing ports is not necessary for EOS, but its often useful to leave this line for clarity.
|
||||
|
||||
.. code:: docker
|
||||
|
||||
WORKDIR /root
|
||||
|
||||
EXPOSE 80
|
||||
|
||||
ENTRYPOINT ["/usr/local/bin/docker_entrypoint.sh"]
|
||||
|
||||
#. Great, let's take a look at our final Hello World ``Dockerfile``:
|
||||
|
||||
.. code:: docker
|
||||
|
||||
FROM arm64v8/alpine:3.12
|
||||
|
||||
RUN apk update
|
||||
RUN apk add tini
|
||||
|
||||
ADD ./hello-world/target/aarch64-unknown-linux-musl/release/hello-world /usr/local/bin/hello-world
|
||||
ADD ./docker_entrypoint.sh /usr/local/bin/docker_entrypoint.sh
|
||||
RUN chmod a+x /usr/local/bin/docker_entrypoint.sh
|
||||
|
||||
WORKDIR /root
|
||||
|
||||
EXPOSE 80
|
||||
|
||||
ENTRYPOINT ["/usr/local/bin/docker_entrypoint.sh"]
|
||||
|
||||
Docker Entrypoint
|
||||
=================
|
||||
|
||||
#. Okay, let's move on to our ``docker_entrypoint.sh`` file. This is a script that defines what to do when the service starts, and often acts as an init system. It will need to complete any environment setup (such as folder substructure), set any environment variables, and execute the run command. It's also PID 1 in the docker container, so it does all of the signal handling and container exits when it is stopped/exits. If you have built a "configurator," this will also need to be called to execute here. More on configurators below. Let's take a look at our (extremely basic) Hello World example:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
#!/bin/sh
|
||||
|
||||
export HOST_IP=$(ip -4 route list match 0/0 | awk '{print $3}')
|
||||
|
||||
exec tini hello-world
|
||||
|
||||
#. We've defined the file, exported the IP address of the Embassy (host), and run the program.
|
||||
|
||||
For a more detailed ``docker_entrypoint.sh``, please check out the `filebrowser-wrapper <https://github.com/Start9Labs/filebrowser-wrapper/blob/master/docker_entrypoint.sh>`_. Additional details on the ``Dockerfile`` and ``docker_entrypoint`` can be found `here <https://docs.start9.com/contributing/services/docker.html>`_.
|
||||
|
||||
Configurators
|
||||
.............
|
||||
|
||||
- Broadly, a "configurator" is any code that translates the configuration coming from the OS to a format the service can understand. Technically all services with a config have one of these (so, most services on Embassy). The configurator also writes the stats.yaml file, which is used in properties.
|
||||
|
||||
- Narrowly, a configurator is a piece of code separate from the docker_entrypoint.sh script, that does the same as the task above. If you can configure the service in bash, inside the docker_entrypoint.sh script, then you don't need a separate piece of code called "configurator", as we have in LND, CL, and synapse, for example. You might want to create a separate configurator if configuring your service would be complicated or impossible in bash. In the case of CL and LND (and bitcoind, where the configurator is actually called a "manager"), the configurator actually has a long-running element that runs concurrently with the service itself.
|
||||
|
||||
There's really no reason a service has to be engineered in this manner. It's really up to the package maintainer how they want to accomplish translation of the config and implementing the properties action, which doesn't even require a stats.yaml file as of 0.3.
|
||||
|
||||
You can check out an example configurator in the `BitcoinD Wrapper <https://github.com/Start9Labs/bitcoind-wrapper/tree/master/manager>`_, which in this case is called a manager.
|
||||
|
||||
Makefile (Optional)
|
||||
===================
|
||||
|
||||
Here, we will create a ``Makefile``, which is optional, but recommended as it outlines the build and streamlines additional developer contributions. Alternatively, you could use ``nix``, ``bash``, ``python``, ``perl``, ``ruby``, etc instead of ``make`` for build orchestration.
|
||||
|
||||
Our example ``Makefile`` is again fairly simple for Hello World. Let's take a look:
|
||||
|
||||
.. code-block:: Makefile
|
||||
|
||||
ASSETS := $(shell yq e '.assets.[].src' manifest.yaml)
|
||||
ASSET_PATHS := $(addprefix assets/,$(ASSETS))
|
||||
VERSION := $(shell toml get hello-world/Cargo.toml package.version)
|
||||
HELLO_WORLD_SRC := $(shell find ./hello-world/src) hello-world/Cargo.toml hello-world/Cargo.lock
|
||||
S9PK_PATH=$(shell find . -name hello-world.s9pk -print)
|
||||
|
||||
.DELETE_ON_ERROR:
|
||||
|
||||
all: verify
|
||||
|
||||
verify: hello-world.s9pk $(S9PK_PATH)
|
||||
embassy-sdk verify $(S9PK_PATH)
|
||||
|
||||
# embassy-sdk pack errors come from here, check your manifest, config, instructions, and icon
|
||||
hello-world.s9pk: manifest.yaml assets/compat/config_spec.yaml config_rules.yaml image.tar docs/instructions.md $(ASSET_PATHS)
|
||||
embassy-sdk pack
|
||||
|
||||
image.tar: Dockerfile docker_entrypoint.sh hello-world/target/aarch64-unknown-linux-musl/release/hello-world
|
||||
DOCKER_CLI_EXPERIMENTAL=enabled docker buildx build --tag start9/hello-world --platform=linux/arm64 -o type=docker,dest=image.tar .
|
||||
|
||||
hello-world/target/aarch64-unknown-linux-musl/release/hello-world: $(HELLO_WORLD_SRC)
|
||||
docker run --rm -it -v ~/.cargo/registry:/root/.cargo/registry -v "$(shell pwd)"/hello-world:/home/rust/src start9/rust-musl-cross:aarch64-musl cargo +beta build --release
|
||||
docker run --rm -it -v ~/.cargo/registry:/root/.cargo/registry -v "$(shell pwd)"/hello-world:/home/rust/src start9/rust-musl-cross:aarch64-musl musl-strip target/aarch64-unknown-linux-musl/release/hello-world
|
||||
|
||||
manifest.yaml: hello-world/Cargo.toml
|
||||
yq e -i '.version = $(VERSION)' manifest.yaml
|
||||
|
||||
#. The first 5 lines set environment variables.
|
||||
|
||||
#. The next line simply removes the progress of a ``make`` process if it fails.
|
||||
|
||||
.. code-block:: Makefile
|
||||
|
||||
.DELETE_ON_ERROR:
|
||||
|
||||
#. The ``all`` step is run when the ``make`` command is issued. This attempts the ``verify`` step, which requires that the ``hello-world.s9pk`` must first be built, which first requires the ``image.tar``, and so on. In this case, each step requires the next and each step indicates the necessary existence of a file. If an indicated file has been altered, such as the `docker_entrypoint.sh`, then any step that contains this file will be rebuilt.
|
||||
|
||||
#. So the ``.s9pk`` is created with the ``embassy-sdk pack`` command, supplied with the ``manifest``, ``config_spec``, previously created ``image.tar``, and ``instructions.md``. Your project may likely also contain a ``config_rules`` file. Some of these files we have not yet edited, but that will come shortly.
|
||||
|
||||
#. The ``image.tar`` is built below this, the cross-compiled ``hello-world`` source code, and ``manifest`` at the bottom.
|
||||
|
||||
For more details on creating a ``Makefile`` for your project, please check :ref:`here <service_makefile>`.
|
||||
|
||||
Service Config Specification and Rules
|
||||
======================================
|
||||
|
||||
Most self-hosted packages require a configuration. With EmbassyOS, these config options are provided to the user in a friendly GUI, and invalid configs are not permitted. This allows the user to manage their software without a lot of technical skill, and minimal risk of borking their software.
|
||||
|
||||
In the config section of the `manifest`, you can
|
||||
Two files are created in this process:
|
||||
|
||||
``config_spec.yaml`` for specifying all the config options your package depends on to run
|
||||
|
||||
``config_rules.yaml`` for defining the ruleset that defines dependencies between config variables
|
||||
|
||||
These are stored in ``assets/compat/`` for 0.2.x compatibility, and in ``/assets/`` for anything built for v0.3.0 and up (almost certainly what you're doing). These files contain a detailed mapping of configuration options with acceptable values, defaults, and relational rule-sets. Hello World has no configuration, as you can see `here <https://github.com/Start9Labs/hello-world-wrapper/blob/0.3.0/assets/compat/config_spec.yaml>`_. Instead, let's take a look at our ``config_spec`` for Embassy Pages, which actually has some config options:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
homepage:
|
||||
name: Homepage
|
||||
description: The page that will be displayed when your Embassy Pages .onion address is visited. Since this page is technically publicly accessible, you can choose to which type of page to display.
|
||||
type: union
|
||||
default: welcome
|
||||
tag:
|
||||
id: type
|
||||
name: Type
|
||||
variant-names:
|
||||
welcome: Welcome
|
||||
index: Subdomain Index
|
||||
filebrowser: Web Page
|
||||
redirect: Redirect
|
||||
fuck-off: Fuck Off
|
||||
variants:
|
||||
welcome: {}
|
||||
index: {}
|
||||
filebrowser:
|
||||
directory:
|
||||
type: string
|
||||
name: Directory Path
|
||||
description: The path to the directory in File Browser that contains the static files of your website. For example, a value of "websites/resume_site" would tell Embassy Pages to look for that directory in File Browser.
|
||||
pattern: "^(\\.|[a-zA-Z0-9_ -][a-zA-Z0-9_ .-]*|([a-zA-Z0-9_ .-][a-zA-Z0-9_ -]+\\.*)+)(/[a-zA-Z0-9_ -][a-zA-Z0-9_ .-]*|/([a-zA-Z0-9_ .-][a-zA-Z0-9_ -]+\\.*)+)*/?$"
|
||||
pattern-description: Must be a valid relative file path
|
||||
nullable: false
|
||||
redirect:
|
||||
target:
|
||||
type: string
|
||||
name: Target Subdomain
|
||||
description: The name of the subdomain to redirect users to. This must be a valid subdomain site within your Embassy Pages.
|
||||
pattern: '^[a-z-]+$'
|
||||
pattern-description: May contain only lowercase characters and hyphens.
|
||||
nullable: false
|
||||
fuck-off: {}
|
||||
subdomains:
|
||||
type: list
|
||||
name: Subdomains
|
||||
description: The websites you want to serve.
|
||||
default: []
|
||||
range: '[0, *)'
|
||||
subtype: object
|
||||
spec:
|
||||
unique-by: name
|
||||
display-as: "{{name}}"
|
||||
spec:
|
||||
name:
|
||||
type: string
|
||||
nullable: false
|
||||
name: Subdomain name
|
||||
description: The subdomain of your Embassy Pages .onion address to host the website on. For example, a value of "me" would produce a website hosted at http://me.myaddress.onion.
|
||||
pattern: "^[a-z-]+$"
|
||||
pattern-description: "May contain only lowercase characters and hyphens"
|
||||
settings:
|
||||
type: union
|
||||
name: Settings
|
||||
description: The desired behavior you want to occur when the subdomain is visited. You can either redirect to another subdomain, or load a web page from File Browser.
|
||||
default: filebrowser
|
||||
tag:
|
||||
id: type
|
||||
name: Type
|
||||
variant-names:
|
||||
filebrowser: Web Page
|
||||
redirect: Redirect
|
||||
variants:
|
||||
filebrowser:
|
||||
directory:
|
||||
type: string
|
||||
name: Directory Path
|
||||
description: The path to the directory in File Browser that contains the static files of your website. For example, a value of "websites/resume_site" would tell Embassy Pages to look for that directory in File Browser.
|
||||
pattern: "^(\\.|[a-zA-Z0-9_ -][a-zA-Z0-9_ .-]*|([a-zA-Z0-9_ .-][a-zA-Z0-9_ -]+\\.*)+)(/[a-zA-Z0-9_ -][a-zA-Z0-9_ .-]*|/([a-zA-Z0-9_ .-][a-zA-Z0-9_ -]+\\.*)+)*/?$"
|
||||
pattern-description: Must be a valid relative file path
|
||||
nullable: false
|
||||
redirect:
|
||||
target:
|
||||
type: string
|
||||
name: Target Subdomain
|
||||
description: The subdomain of your Embassy Pages .onion address to redirect to. This should be the name of another subdomain on Embassy Pages. Leave empty to redirect to the homepage.
|
||||
pattern: '^[a-z-]+$'
|
||||
pattern-description: May contain only lowercase characters and hyphens.
|
||||
nullable: false
|
||||
|
||||
We essentially have 2 config options (homepage and subdomains), with all of their specifications nested below them. Looking at the homepage, it contains a ``union`` type, which is a necessary dependency, which can be of 5 variants (welcome, index, filebrowser, redirect, or fuck-off). The below images show how this is displayed in the UI.
|
||||
|
||||
.. figure:: /_static/images/dev/pages0.png
|
||||
:width: 60%
|
||||
:alt: Pages Config
|
||||
|
||||
.. figure:: /_static/images/dev/pages1.png
|
||||
:width: 60%
|
||||
:alt: Pages Union
|
||||
|
||||
For all the possible types, please check our :ref:`Service Config Spec <service_config>`.
|
||||
|
||||
In our example, there is *no need* for a ``config_rules`` file. This is because there is not a rule-set required to define dependencies between config variables. An example of when this would be required would be the following code, from the [LND wrapper](https://github.com/Start9Labs/lnd-wrapper/blob/master/config_rules.yaml):
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
---
|
||||
- rule: '!(max-chan-size?) OR !(min-chan-size?) OR (#max-chan-size > #min-chan-size)'
|
||||
description: "Maximum Channel Size must exceed Minimum Channel Size"
|
||||
|
||||
Here we see that a Maximum Channel Size **MUST** be one of 3 possible options in order to be a valid config.
|
||||
|
||||
Properties
|
||||
==========
|
||||
|
||||
Next we need to create the Properties section for our package, to display any relevant info. The result of this step is a ``stats.yaml`` file, which is only populated at runtime. These commands will be issued in the ``docker_entrypoint`` file (or ``configurator``, if you are using one).
|
||||
|
||||
.. ***STATS.YAML IS APPARENTLY BEING DEPRECATED, THIS SECTION NEEDS COMMENT*** Possibly this is not actually the case?
|
||||
|
||||
Instructions
|
||||
============
|
||||
|
||||
Instructions are the basic directions or any particular details that you would like to convey to the user to help get them on their way. Each wrapper repo should contain a ``docs`` directory which can include anything you'd like, but specifically if you include an ``instructions.md`` file, formatted in Markdown language, it will be displayed simply for the user as shown below.
|
||||
|
||||
.. figure:: /_static/images/dev/instructions.png
|
||||
:width: 60%
|
||||
:alt: Instructions
|
||||
|
||||
You can find the ``instructions.md`` file for Embassy Pages `here <https://github.com/Start9Labs/embassy-pages-wrapper/tree/master/docs>`_ if you are interested.
|
||||
|
||||
Backups
|
||||
=======
|
||||
|
||||
Everything in the root folder of the mounted system directory will be stored in an EOS backup. If you want to ignore any particular files for backup, you can create a ``.backupignore`` file and add the relative paths of any directories you would like ignored.
|
||||
|
||||
Submission Process
|
||||
------------------
|
||||
|
||||
When you have built and tested your project for EmbassyOS, please send Start9 a submission with the project repository to dev@start9labs.com. After being reviewed for security and compatibility, the service will be deployed to the marketplace and available for all EmbassyOS users to download.
|
||||
|
||||
If you are deploying to an alternative marketplace, please shout it out in our community channels!
|
||||
@@ -1,29 +0,0 @@
|
||||
.. _service_backups:
|
||||
|
||||
=======
|
||||
Backups
|
||||
=======
|
||||
|
||||
Everything within the root of the mounted volume directory will be stored in an EmbassyOS backup. This includes the config (config.yaml) and properties (stats.yaml) files, as well as any other persisted data within the volume directory.
|
||||
|
||||
For restoration purposes, it might be beneficial to ignore certain files or folders. For instance, ignore the shared/public folder that is mounted for dependencies that expose this feature as it causes data inconsistencies on restore.
|
||||
|
||||
In this case, create a `.backupignore` file. This file contains a list of relative paths to the ignored files.
|
||||
|
||||
|
||||
Example
|
||||
-------
|
||||
|
||||
The `btcpayserver wrapper <https://github.com/Start9Labs/btcpayserver-wrapper/blob/master/configurator/src/templates/.backupignore.templates>`_ demonstrates a good use of a backupignore template.
|
||||
|
||||
Ultimately, ``/datadir/.backupignore`` gets populated with:
|
||||
|
||||
.. code::
|
||||
|
||||
/root/volumes/btcpayserver/start9/public
|
||||
/root/volumes/btcpayserver/start9/shared
|
||||
|
||||
.. role:: raw-html(raw)
|
||||
:format: html
|
||||
|
||||
:raw-html:`<br />`
|
||||
@@ -1,618 +0,0 @@
|
||||
.. _service-config:
|
||||
|
||||
=============
|
||||
Configuration
|
||||
=============
|
||||
|
||||
Introduction
|
||||
------------
|
||||
|
||||
Most self-hosted applications require the user to tell the app how to behave using a config file in a specific format, environment variables, command-line arguments, or some combination of these inputs. One of the coolest features of EmbassyOS is that, when packaged correctly, the app's configuration will be available to the user as a slick GUI that always produces a valid configuration no matter how little experience or skill the user has.
|
||||
|
||||
With EmbassyOS, this means a service wrappers' configuration requires a particular format and rule structure to ensure it integrates smoothly with the user interface. This format enables clean handling of improper values and dependency management.
|
||||
|
||||
The outcome of this step is two files:
|
||||
|
||||
:ref:`config_spec.yaml <config_spec>`
|
||||
|
||||
:ref:`config_rules.yaml <config_rules>`
|
||||
|
||||
These files contain a detailed mapping of configuration options with acceptable values, defaults, and relational rule-sets.
|
||||
|
||||
For example, if the user chooses config option A, then config option B must be between 5 and 10. This enables a simple GUI configuration experience, complete with validations and protections, for users. They do not have to worry about the consequences of a wrong value in a ``.conf`` file.
|
||||
|
||||
.. _config_spec:
|
||||
|
||||
Config Spec
|
||||
-----------
|
||||
|
||||
Overview
|
||||
========
|
||||
|
||||
.. figure:: /_static/images/services/service5.png
|
||||
:width: 80%
|
||||
:alt: Synapse Config
|
||||
|
||||
This file defines the structure of configuration options your service depends on to run. It additionally can include configuration options that users might want to enable for more advanced or customized usage. Ultimately, these values influence the UI elements for a user to interact with. Specifically, they evaluate to the options available when managing a service, such as:
|
||||
|
||||
- Prior to service installation when the user needs to be made aware of any necessary dependency configurations
|
||||
- When the user installs a service and the service is in the "Needs Config" state
|
||||
- Whenever a user edits a service config
|
||||
- When config pointers get updated
|
||||
|
||||
The neat part about this file is that each ValueSpec type gets translated into a specific front end component. For instance, boolean values display as a toggle button, such as in the Synapse exmple of ``Enable Registration``.
|
||||
|
||||
.. figure:: /_static/images/services/synapseconfig.png
|
||||
:width: 80%
|
||||
:alt: Example boolean toggle
|
||||
|
||||
Another advantage is the ability to define default values. These values automatically get populated if the user selects the ``Default`` option when setting up a service in ``Needs Config`` state. This is super convenient for users who want to get up and running quickly.
|
||||
|
||||
Types
|
||||
=====
|
||||
|
||||
ConfigSpec Type:
|
||||
|
||||
.. code::
|
||||
|
||||
key: ValueSpec
|
||||
|
||||
ValueSpec Type: Boolean | Enum | List | Number | Object | String | Union | Pointer (see below for details)
|
||||
|
||||
Implementation Guide
|
||||
====================
|
||||
|
||||
The following section contains implementation specifications for the ``config_spec.yaml`` file.
|
||||
|
||||
- All keys are ``kebab-case`` strings, which correspond to the service (app) id
|
||||
- All values are one the following specs (ie. ``ValueSpec`` type):
|
||||
|
||||
:ref:`boolean <boolean>`
|
||||
|
||||
:ref:`enum <enum>`
|
||||
|
||||
:ref:`list <list>`
|
||||
|
||||
:ref:`number <number>`
|
||||
|
||||
:ref:`object <object>`
|
||||
|
||||
:ref:`string <string>`
|
||||
|
||||
:ref:`union <union>`
|
||||
|
||||
:ref:`pointer <pointer>`
|
||||
|
||||
- In the examples for each value spec type below, ``Option`` means the key is optional. Otherwise, the key is required.
|
||||
- Descriptions are optional, but recommended
|
||||
- Name corresponds to the name of the config field
|
||||
- Find a complete example :ref:`here <example_config_spec>`
|
||||
- Change warning text displays when the value is altered
|
||||
|
||||
.. _boolean:
|
||||
|
||||
Boolean
|
||||
.......
|
||||
|
||||
Config value specification denoted as a boolean value. A default value is required.
|
||||
|
||||
``ValueSpec`` Type:
|
||||
|
||||
.. code::
|
||||
|
||||
type: boolean
|
||||
name: String
|
||||
description: Option<String>
|
||||
changeWarning: Option<String>
|
||||
default: Boolean
|
||||
|
||||
Example:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
fetch-blocks:
|
||||
type: boolean
|
||||
name: Fetch Blocks
|
||||
description: Fetch blocks from the network if pruned from disk
|
||||
default: true
|
||||
|
||||
.. _enum:
|
||||
|
||||
Enum
|
||||
....
|
||||
|
||||
Config value specification denoted as an enum value. Enums values must be a unique set. If no default is provided, ``null`` will be the assumed value.
|
||||
|
||||
ValueSpec Type:
|
||||
|
||||
.. code::
|
||||
|
||||
type: enum
|
||||
name: String
|
||||
description: Option<String>
|
||||
changeWarning: Option<String>
|
||||
default: Option<Enum>
|
||||
values: Set<String>
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
theme-mode:
|
||||
type: enum
|
||||
name: Theme Mode
|
||||
values:
|
||||
- NIGHT
|
||||
- DAY
|
||||
valueNames:
|
||||
NIGHT: Night
|
||||
DAY: Day
|
||||
default: NIGHT
|
||||
|
||||
.. _list:
|
||||
|
||||
List
|
||||
....
|
||||
|
||||
The list type describes an array of values. The values must consist of the same subtype, which can be any of the ValueSpec types available in the EmbassyOS config specification.
|
||||
Lists of any type do not contain the default for each item in list. The list *itself* can have a default. If no default is provided, ``null`` will be the assumed value.
|
||||
|
||||
Range is loosely based off mathematical range syntax, with infinity replaced with ``*``:
|
||||
|
||||
``[ || ]`` = inclusive
|
||||
|
||||
``( || )`` = noninclusive
|
||||
|
||||
``*`` = infinity on either end
|
||||
|
||||
eg:
|
||||
|
||||
.. code::
|
||||
|
||||
[0,*) - all numbers to infinity including 0
|
||||
|
||||
ValueSpec Type:
|
||||
|
||||
.. code::
|
||||
|
||||
type: list
|
||||
name: String
|
||||
description: Option<String>
|
||||
subtype: enum || number || object || string || union
|
||||
range: NumRange<unsigned integer>
|
||||
spec: ValueSpec
|
||||
default: ValueSpec
|
||||
|
||||
Example:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
allowed-calls:
|
||||
type: list
|
||||
name: Allowed Calls
|
||||
description: The list of all RPC methods this user is allowed to make
|
||||
subtype: enum
|
||||
range: "[0, *)"
|
||||
spec:
|
||||
type: enum
|
||||
values:
|
||||
- item
|
||||
- item
|
||||
|
||||
|
||||
.. _number:
|
||||
|
||||
Number
|
||||
......
|
||||
|
||||
A number value within an optionally defined range. Nullable field is required. If ``nullable`` is true, the default is assumed to be ``null`` if it is not provided.
|
||||
|
||||
ValueSpec Type:
|
||||
|
||||
.. code::
|
||||
|
||||
type: number
|
||||
name: String
|
||||
description: Option<String>
|
||||
changeWarning: Option<String>
|
||||
default: Boolean
|
||||
nullable: Boolean
|
||||
range: Option<NumRange<64 bit floating point>>
|
||||
integral: Boolean
|
||||
units: Option<String>
|
||||
|
||||
Example:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
type: number
|
||||
name: Peer Message Timeout
|
||||
description: How long to wait for a response from a peer before failing
|
||||
nullable: false
|
||||
integral: true
|
||||
units: Seconds
|
||||
range: "[0, *)"
|
||||
default: 30
|
||||
|
||||
.. _object:
|
||||
|
||||
Object Type
|
||||
...........
|
||||
|
||||
A nested representation of a ConfigSpec. The object type takes the same structure under the ``spec`` key as a ConfigSpec: a key indicates the field name, and the value denotes the ValueSpec type for that field.
|
||||
|
||||
There is no default option for the object type. Rather, the option ``null-by-default`` should be used to indicate the default as ``null``. If null by default is true, nullable must be true. If null by default is false, nullable could be either.
|
||||
|
||||
``unique-by`` indicates whether duplicates can be permitted in the list.
|
||||
|
||||
ValueSpec Type:
|
||||
|
||||
.. code::
|
||||
|
||||
type: object
|
||||
name: String
|
||||
description: Option<String>
|
||||
changeWarning: Option<String>
|
||||
nullable: Boolean
|
||||
null-by-default: Boolean
|
||||
display-as: Option<String>
|
||||
unique-by: UniqueBy
|
||||
spec: ConfigSpec
|
||||
|
||||
type UniqueBy = null | string | { any: UniqueBy[] } | { all: UniqueBy[] }
|
||||
|
||||
Example:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
type: object
|
||||
name: Advanced
|
||||
description: Advanced settings for Bitcoin Proxy
|
||||
nullable: false
|
||||
spec:
|
||||
tor-only:
|
||||
type: boolean
|
||||
name: Only Tor Peers
|
||||
description: Use Tor for all peer connections
|
||||
default: false
|
||||
peer-timeout:
|
||||
type: number
|
||||
name: Peer Message Timeout
|
||||
description: How long to wait for a response from a peer before failing
|
||||
nullable: false
|
||||
integral: true
|
||||
units: Seconds
|
||||
range: "[0, *)"
|
||||
default: 30
|
||||
max-peer-age:
|
||||
type: number
|
||||
name: Maximum Peer Age
|
||||
description: How long to wait before refreshing the peer list
|
||||
nullable: false
|
||||
integral: true
|
||||
units: Seconds
|
||||
range: "[0, *)"
|
||||
default: 300
|
||||
max-peer-concurrency:
|
||||
type: number
|
||||
name: Maximum Peer Concurrency
|
||||
description: How many peers to reach out to concurrently for block data
|
||||
nullable: true
|
||||
integral: true
|
||||
range: "[1, *)"
|
||||
default: 1
|
||||
|
||||
.. _string:
|
||||
|
||||
String
|
||||
......
|
||||
|
||||
There are various options for string values. They can optionally be marked as copyable or masked, such as for passwords, which will reflect the UI element display. A pattern, expressed in regex, can be denoted. If it exists, this field requires both the pattern type (ie. Regex) and pattern description (ie. an explanation of the pattern requirements).
|
||||
|
||||
If the default type is ``Entropy``, the charset can optionally specify an inclusive ranged character set (ie. "a-f,0-9").
|
||||
|
||||
ValueSpec Type:
|
||||
|
||||
.. code::
|
||||
|
||||
type: string
|
||||
name: String
|
||||
description: Option<String>
|
||||
changeWarning: Option<String>
|
||||
copyable: Option<boolean>
|
||||
masked: Option<boolean>
|
||||
nullable: Boolean
|
||||
default: String | Entropy
|
||||
pattern: Option<Regex>
|
||||
pattern-description: Option<String>
|
||||
|
||||
Entropy Type:
|
||||
|
||||
.. code::
|
||||
|
||||
charset: Option<String>
|
||||
len: integer
|
||||
|
||||
Examples:
|
||||
|
||||
.. code::
|
||||
|
||||
color:
|
||||
type: string
|
||||
name: Color
|
||||
description: Color value for the Lightning Network
|
||||
nullable: false
|
||||
pattern: "[0-9a-fA-F]{6}"
|
||||
patternDescription: |
|
||||
Must be a valid 6 digit hexadecimal RGB value. The first two digits are red, middle two are green and final two are
|
||||
blue
|
||||
default:
|
||||
charset: "a-f,0-9"
|
||||
len: 6
|
||||
|
||||
password:
|
||||
type: string
|
||||
name: Password
|
||||
description: The password for the RPC User
|
||||
nullable: false
|
||||
copyable: true
|
||||
masked: true
|
||||
default:
|
||||
charset: "a-z,A-Z,0-9"
|
||||
len: 22
|
||||
|
||||
.. _pointer:
|
||||
|
||||
Pointer
|
||||
.......
|
||||
|
||||
The pointer type *points* to a config value on another service installed on EmbassyOS (ie. app subtype) or to the EmbassyOS system (ie. system subtype). When pointing to another service, the ``index`` field indicates the path to the desired config variable.
|
||||
|
||||
ValueSpec Type:
|
||||
|
||||
.. code::
|
||||
|
||||
type: pointer
|
||||
name: String
|
||||
description: Option<String>
|
||||
changeWarning: Option<String>
|
||||
subtype: app | system
|
||||
app-id: String (*always* kebab case)
|
||||
target: AppPointerSpecVariants | SystemPointerSpecVariants
|
||||
index: Option<String> (dependent on target being AppPointerSpecVariants)
|
||||
|
||||
AppPointerSpecVariants = LanAddress | TorAddress | TorKey | Config
|
||||
SystemPointerSpecVariants = HostIp
|
||||
|
||||
Example:
|
||||
|
||||
.. code::
|
||||
|
||||
user:
|
||||
type: pointer
|
||||
name: RPC Username
|
||||
description: The username for the RPC user for Bitcoin Core
|
||||
subtype: app
|
||||
app-id: bitcoind
|
||||
target: config
|
||||
index: "rpc.username"
|
||||
|
||||
.. _union:
|
||||
|
||||
Union
|
||||
.....
|
||||
|
||||
This type describes a necessary dependency. Multiple variants can be expressed to enable the user the option to connect to another service (internal dependency) or outside source (external dependency).
|
||||
|
||||
For example, the Bitcoin Proxy service is united with an instance of Bitcoin. Three variants are defined: internal, external, and a quick connect. In this case, internal refers to the Bitcoin Core instance running on EmbassyOS, and defines the RPC credentials necessary for connecting; external refers to a Bitcoin Core node running on a different device, and defines the RPC credentials necessary for connecting; quick connect refers to yet another method of connecting to a Bitcoin Core node, optimized for convenience.
|
||||
|
||||
Default is required and corresponds to one of the variants.
|
||||
|
||||
``Tag`` is the key that will be rendered on the UI element.
|
||||
|
||||
ValueSpec Type;
|
||||
|
||||
.. code::
|
||||
|
||||
type: union
|
||||
name: String
|
||||
description: Option<String>
|
||||
changeWarning: Option<String>
|
||||
default: Boolean
|
||||
tag: Tag
|
||||
variants: Map<String, ConfigSpec>
|
||||
display-as: Option<String>
|
||||
unique-by: any | all | exactly | notUnique
|
||||
|
||||
Tag Type:
|
||||
|
||||
.. code::
|
||||
|
||||
id: String
|
||||
name: String
|
||||
description: Option<String>
|
||||
variant-names: Map<String, String>
|
||||
|
||||
.. _example_config_spec:
|
||||
|
||||
Example:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
bitcoind:
|
||||
type: union
|
||||
name: Bitcoin Core
|
||||
description: The Bitcoin Core node to connect to
|
||||
tag:
|
||||
id: type
|
||||
name: Type
|
||||
description: |
|
||||
- Internal: The Bitcoin Core service installed to your Embassy
|
||||
- External: A Bitcoin Core node running on a different device
|
||||
- Quick Connect: A Quick Connect URL for an unpruned Bitcoin Core node
|
||||
variant-names:
|
||||
internal: Internal
|
||||
external: External
|
||||
quick-connect: Quick Connect
|
||||
default: internal
|
||||
variants:
|
||||
internal:
|
||||
address:
|
||||
type: pointer
|
||||
name: Local Address
|
||||
description: The LAN IP address of your Bitcoin Core service
|
||||
subtype: app
|
||||
app-id: bitcoind
|
||||
target: lan-address
|
||||
user:
|
||||
type: pointer
|
||||
name: RPC Username
|
||||
description: The username for the RPC user for Bitcoin Core
|
||||
subtype: app
|
||||
app-id: bitcoind
|
||||
target: config
|
||||
index: "rpc.username"
|
||||
password:
|
||||
type: pointer
|
||||
name: RPC Password
|
||||
description: The password for the RPC user for Bitcoin Core
|
||||
subtype: app
|
||||
app-id: bitcoind
|
||||
target: config
|
||||
index: "rpc.password"
|
||||
external:
|
||||
addressext:
|
||||
type: string
|
||||
name: Public Address
|
||||
description: The public address of your Bitcoin Core RPC server
|
||||
nullable: false
|
||||
userext:
|
||||
type: string
|
||||
name: RPC Username
|
||||
description: The username for the RPC user on your Bitcoin Core RPC server
|
||||
nullable: false
|
||||
passwordext:
|
||||
type: string
|
||||
name: RPC Password
|
||||
description: The password for the RPC user on your Bitcoin Core RPC server
|
||||
nullable: false
|
||||
quick-connect:
|
||||
quick-connect-url:
|
||||
type: string
|
||||
name: Quick Connect URL
|
||||
description: The Quick Connect URL for your Bitcoin Core RPC server
|
||||
nullable: false
|
||||
pattern: 'btcstandup://[^:]*:[^@]*@[a-zA-Z0-9.-]+:[0-9]+(/(\?(label=.+)?)?)?'
|
||||
patternDescription: Must be a valid Quick Connect URL. For help, check out https://github.com/BlockchainCommons/Gordian/blob/master/Docs/Quick-Connect-API.md
|
||||
|
||||
|
||||
.. _config_rules:
|
||||
|
||||
Config Rules
|
||||
------------
|
||||
|
||||
This file defines the configuration rules, or the rule-set that defines dependencies between config variables. In practice, config rules are for auto-configuring self dependencies. Self dependencies are internal dependencies of a service, such as if the setting of one config variable informs the option of another setting. These "dependencies" are configured as rules.
|
||||
|
||||
A rule is a boolean expression that we demand to be true. It is not true if the expression fails the rule parser.
|
||||
|
||||
They follow the `Backus–Naur <https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_form>`_ meta-syntax for writing rules.
|
||||
|
||||
Rules are composed of two main concepts:
|
||||
|
||||
* Variables - accessor into a configuration
|
||||
* Terms - either a variable or type literal (ie. a boolean term is a boolean variable, a boolean expression, or a comparison operation between numbers or strings)
|
||||
|
||||
Variables can be booleans, numbers, or strings, and have a different syntax depending on the type. These type annotations check your config rules against your config spec and throw an error if invalid.
|
||||
|
||||
- ``?`` - Casts to boolean value. If the value is not a boolean, this notes whether or not the value is null.
|
||||
- ``#`` - Treat the value as a number. If it is not a number, the value will be parsed as NaN. String numbers are not currently supported.
|
||||
- ``'`` - Cast the value into a string. Applies to any value except for an object or a list.
|
||||
- ``!`` - Equals not.
|
||||
|
||||
.. note::
|
||||
Config rules are processed in order.
|
||||
|
||||
If application does not satisfy a rule, a set of suggestions should be provided. These suggestions are in the form of the operation to preform:
|
||||
|
||||
- ``Set`` - set the value
|
||||
|
||||
- ``Push`` - add to the value (such as to a list)
|
||||
|
||||
- ``Delete`` - delete the value
|
||||
|
||||
.. code:: typescript
|
||||
|
||||
enum SuggestionVariant = Set | Delete | Push
|
||||
|
||||
interface Set {
|
||||
var: String, // fully qualified path without typecast
|
||||
// one of the following three variants are required
|
||||
to: Option<String> // a string expression, use when tying another config value
|
||||
to-value: Option<String>
|
||||
to-entropy: Option<{
|
||||
charset: String (eg. 'a-z,A-Z,0-9')
|
||||
len: Number
|
||||
}>
|
||||
}
|
||||
|
||||
interface Delete {
|
||||
src: String, // path to key - removes if in a list
|
||||
}
|
||||
|
||||
interface Push {
|
||||
to: String,
|
||||
value: String, // string literal of value to be set
|
||||
}
|
||||
|
||||
Set Examples:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
- SET:
|
||||
# the key in config you want to set
|
||||
var: 'users.[first(item => ''item.name = "c-lightning")].password'
|
||||
# the value in config that you will set
|
||||
to-entropy:
|
||||
charset: "a-z,A-Z,0-9"
|
||||
len: 22
|
||||
|
||||
- SET:
|
||||
var: 'users.[first(item => ''item.name = "c-lightning")].fetch-blocks'
|
||||
to-value: true
|
||||
|
||||
|
||||
Push Examples:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
- PUSH:
|
||||
to: "users"
|
||||
value:
|
||||
name: c-lightning
|
||||
allowed-calls: []
|
||||
|
||||
- PUSH:
|
||||
to: 'users.[first(item => ''item.name = "c-lightning")].allowed-calls'
|
||||
value: "getnetworkinfo"
|
||||
|
||||
Full example from `c-lightning manifest <https://github.com/Start9Labs/c-lightning-wrapper/blob/master/manifest.yaml>`_:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
config:
|
||||
- rule: '''users.*.name = "c-lightning"'
|
||||
description: 'Must have an RPC user named "c-lightning"'
|
||||
suggestions:
|
||||
- PUSH:
|
||||
to: "users"
|
||||
value:
|
||||
name: c-lightning
|
||||
allowed-calls: []
|
||||
- SET:
|
||||
var: 'users.[first(item => ''item.name = "c-lightning")].password'
|
||||
to-entropy:
|
||||
charset: "a-z,A-Z,0-9"
|
||||
len: 22
|
||||
|
||||
.. role:: raw-html(raw)
|
||||
:format: html
|
||||
|
||||
:raw-html:`<br />`
|
||||
@@ -1,40 +0,0 @@
|
||||
.. _service_docker:
|
||||
|
||||
==========
|
||||
Dockerfile
|
||||
==========
|
||||
|
||||
Dockerfile
|
||||
----------
|
||||
|
||||
This purpose of the Dockerfile is to define how to build the image for the service. It declares the environment and building stages.
|
||||
The Dockerfile is responsible for mounting the service application to whatever volume the manifest specifies, typically ``/root/.<service-id>``.
|
||||
|
||||
Since the build requires specific arm runtime environments, these base images can be imported into the Dockerfile as a starting point so the base system does not need to be completely redefined. This enables importing a specific build environment version that encapsulates any necessary libraries for running the service build, eg. golang, rust.
|
||||
|
||||
For instance:
|
||||
|
||||
``FROM alpine:3.12``
|
||||
|
||||
``FROM arm32v7/alpine``
|
||||
|
||||
``FROM arm32v7/golang:alpine``
|
||||
|
||||
We recommended using ``alpine`` since it produces the smallest image. We try to keep the image under 100MB when possible.
|
||||
|
||||
Entry-point
|
||||
-----------
|
||||
|
||||
The ``docker_entrypoint.sh`` defines what to do when the service application starts.
|
||||
|
||||
It consists of a bash script that completes any environment setup (eg. creating folder substructure), sets any environment variables, and executes the run command. The only required feature of this file is to execute the run commands for EmbassyOS.
|
||||
|
||||
Example
|
||||
-------
|
||||
|
||||
The `LND wrapper <https://github.com/Start9Labs/lnd-wrapper/blob/master/Dockerfile>`_ features a well defined Dockerfile, for example.
|
||||
|
||||
.. role:: raw-html(raw)
|
||||
:format: html
|
||||
|
||||
:raw-html:`<br />`
|
||||
@@ -1,35 +0,0 @@
|
||||
.. _service-packaging-spec:
|
||||
|
||||
======================
|
||||
Service Packaging Spec
|
||||
======================
|
||||
|
||||
Welcome! If you are here, you are interested in becoming part of the mission to change the future of personal computing. This guide will take you through the process of packaging a service for EmbassyOS, a novel, self-hosted, sovereign computing platform.
|
||||
|
||||
A service in this context is any open source project that has been bundled into the appropriate format to run on EmbassyOS. By configuring and packaging a project according to this guide, it can be installed on EmbassyOS with no command line or technical expertise required. This opens up an entire platform for self-hosted software to run independent of third parties in a completely private and sovereign way for all individuals.
|
||||
|
||||
This guide is technical, but breaks down the steps for deployment. If you have any feedback or questions concerning this guide, please don't hesitate to `reach out <https://matrix.to/#/#community-dev:matrix.start9labs.com>`_ or submit a pull request with alterations.
|
||||
|
||||
To start, you will need to acquire EmbassyOS for testing the packaged service. This can be done by:
|
||||
|
||||
- building from `source <https://github.com/Start9Labs/embassy-os/blob/master/CONTRIBUTING.md#setting-up-your-development-environment>`_
|
||||
- following the :ref:`DIY <diy>` guide
|
||||
- :ref:`purchasing <purchasing>` the ready to run personal server
|
||||
|
||||
While you are waiting to receive or assemble a device, you can begin the process of packaging a project. The next sections outline these steps in detail.
|
||||
|
||||
Happy building!
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
|
||||
overview
|
||||
wrapper
|
||||
manifest
|
||||
docker
|
||||
makefile
|
||||
config
|
||||
properties
|
||||
instructions
|
||||
backups
|
||||
submission
|
||||
@@ -1,61 +0,0 @@
|
||||
.. _service_instructions:
|
||||
|
||||
===================
|
||||
Instructions & Docs
|
||||
===================
|
||||
|
||||
Within each wrapper repository should exist a ``docs`` directory. This folder should include any pertinent documentation, instructions, external integrations, or other details about the service that users or developers might find relevant.
|
||||
|
||||
If an ``instructions.md`` file is included, this markdown formatted file will be rendered in the "Instructions" section within the service page on EmbassyOS:
|
||||
|
||||
.. figure:: /_static/images/services/service-instruct.png
|
||||
:width: 80%
|
||||
:alt: Embassy Pages Instructions
|
||||
|
||||
Below is the markdown file for Embassy Pages' instructions, as shown above:
|
||||
|
||||
.. code-block::
|
||||
|
||||
# Embassy Pages
|
||||
|
||||
## About
|
||||
|
||||
Embassy Pages is a simple web server that uses directories inside File Browser to serve Tor websites. Your website could be a blog, a resume, a portfolio, a business landing page, a product brochure, or just a set of static folders and/or files that you want to share with the world.
|
||||
|
||||
When you first install Embassy Pages, there will be a default Homepage hosted at the root, <tor-address>.onion. You can change the behavior of this page, and you can also create Subdomain websites. For example, one site could be hello.<tor-address>.onion and another could be goodbye.<tor-address>.onion. What is served from the Homepage and each Subdomain is totally up to you.
|
||||
|
||||
Self-hosting Tor websites using Embassy Pages is easy, permissionless, and censorship-resistant; there are no trusted third parties involved.
|
||||
Anyone can do it. No one can stop it.
|
||||
|
||||
## Instructions
|
||||
|
||||
1. Inside `Config`, you can easily change the behavior of your Homepage to serve:
|
||||
a. A list of hyperlinks to all your Subdomains
|
||||
b. A personal web page
|
||||
c. An automatic redirect to a Subdomain
|
||||
d. A static web page that tells anyone visiting it to fuck off, politely.
|
||||
|
||||
2. Inside `Config`, you can create one or more Subdomains, giving each a unique name.
|
||||
|
||||
3. To serve a personal website, simply upload the website directory to File Browser. Then, inside the settings for a particular page (either your Homepage or a Subdomain), enter the path to that directory. For example, a path of websites/blog would tell Embassy Pages that it can find the blog website inside the websites directory in File Browser.
|
||||
|
||||
4. A list of all your Subdomains can be found inside the Properties section of your Embassy Pages service.
|
||||
|
||||
## Restore from Backup
|
||||
|
||||
1. In `Config`, select "Restore from Backup".
|
||||
2. After the restoration completes, navigate to the Marketplace.
|
||||
3. Downgrade Embassy Pages.
|
||||
4. Upgrade Embassy Pages to the latest version.
|
||||
5. Start Embassy Pages.
|
||||
6. Embassy Pages is now successfully restored!
|
||||
|
||||
Example
|
||||
-------
|
||||
|
||||
The `bitcoind wrapper <https://github.com/Start9Labs/bitcoind-wrapper/tree/master/docs>`_ demonstrates a good use of instructions and external integrations.
|
||||
|
||||
.. role:: raw-html(raw)
|
||||
:format: html
|
||||
|
||||
:raw-html:`<br />`
|
||||
@@ -1,28 +0,0 @@
|
||||
.. _service_makefile:
|
||||
|
||||
========
|
||||
Makefile
|
||||
========
|
||||
|
||||
.. note:: *This file is optional*
|
||||
|
||||
A Makefile serves as a convenience for outlining the build. This helps streamline additional developer contributions to the project. Please reference the GNU `documentation <https://www.gnu.org/software/make/manual/html_node/Introduction.html>`_ for implementation details.
|
||||
|
||||
An alternative to using ``make`` is to use the `nix <https://nixos.wiki/wiki/Nix>`_ specification.
|
||||
|
||||
This purpose of this file is to:
|
||||
|
||||
- Read the docker container and build the project
|
||||
- Build all prerequisites for running the docker file
|
||||
- Build all dependencies
|
||||
- Package ``config_rules.yaml``, ``config_spec.yaml``, ``manifest.yaml``, and ``image.tar`` into an ``.s9pk`` extension by invoking ``appmgr``.
|
||||
|
||||
Example
|
||||
-------
|
||||
|
||||
The `LND wrapper <https://github.com/Start9Labs/lnd-wrapper/blob/master/Makefile>`_ features a well defined Makefile, for example.
|
||||
|
||||
.. role:: raw-html(raw)
|
||||
:format: html
|
||||
|
||||
:raw-html:`<br />`
|
||||
@@ -1,353 +0,0 @@
|
||||
.. _service_manifest:
|
||||
|
||||
========
|
||||
Manifest
|
||||
========
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
This file describes the service and it's requirements. It is used to:
|
||||
|
||||
- create a listing in the marketplace
|
||||
- denote any installation considerations
|
||||
- define dependency requirements
|
||||
|
||||
Each time a service is updated, the manifest should be updated to include the new version, release notes, and any pertinent updates to the install, uninstall, or restoration flows. All this information is displayed in the marketplace listing, and the optionally denoted alerts will be displayed when appropriate to provide the user with more information.
|
||||
|
||||
For instance, `LND displays alerts <https://github.com/Start9Labs/lnd-wrapper/blob/87daf4e5ed7231e22aaa28be533e794f67f98289/manifest.yaml#L30>`_ around restoration since data will be overwritten.
|
||||
|
||||
There is nothing you need to do as a developer to set up Tor for running a service. This is *completely* handled by EmbassyOS - a Tor address will be automatically generated when the service is installed. Just define the port and which version of Tor the service needs in this manifest file! You do, however, need to ensure the service is in fact capable of running over Tor.
|
||||
|
||||
The manifest is also responsible for outlining service :ref:`dependencies <dependencies>`. By defining rules using the :ref:`EmbassyOS DSL specification <config_rules>`, users can easily and selectively install, uninstall, and update any service without getting stuck in dependency hell. EmbassyOS presents this information in a polished install/uninstall/update wizard, so there's no need for editing configuration files or jumping into the command line. For you as a developer, this simply means populating this key in the manifest!
|
||||
|
||||
Formatting
|
||||
----------
|
||||
|
||||
- Serialization language:``.yaml``
|
||||
- Case style: ``kebab-case``
|
||||
|
||||
Type definitions
|
||||
----------------
|
||||
|
||||
Below are the types and sub-type definitions, with necessary elaborations. Any item that contains ``Option<>`` is an optional field.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
# manifest version compatibility ie. v0 (this is currently the only option)
|
||||
compat: v0
|
||||
# service id, used for uniqueness and BE identification ie. bitcoind
|
||||
id: String
|
||||
# version number of the release conforming to the semantic versioning scheme
|
||||
version: String
|
||||
# display name of service
|
||||
title: String
|
||||
# an object containing a short and long description of the service. TODO character lengths
|
||||
description:
|
||||
short: String
|
||||
long: String
|
||||
# a link to the release tag notes in GitHub, or a short description TODO character length
|
||||
release-notes: String
|
||||
# a notification message that should caution the user with any service particularities, eg. beta tech
|
||||
install-alert: Option<String>
|
||||
# a notification message warning users of potential problems with uninstalling, such as dependency failures or data loss
|
||||
uninstall-alert: Option<String>
|
||||
# a notification message containing warning or details about backup restoration
|
||||
restore-alert: Option<String>
|
||||
# denoting the existence of instructions.md
|
||||
has-instructions: Boolean
|
||||
# required semver version range of EmbassyOS to run service eg. ">=1.0.0"
|
||||
os-version-required: VersionReq
|
||||
# recommended semver version range of EmbassyOS to run service eg."^1.0.0"
|
||||
os-version-recommended: VersionReq
|
||||
# a list of objects of ports to run the service on localhost and tor
|
||||
ports:
|
||||
- internal: String
|
||||
tor: String
|
||||
# currently only tar is supported
|
||||
image:
|
||||
type: String
|
||||
# shared memory container size
|
||||
shm-size-mb: Option<usize>
|
||||
# path to mount the image on the volume, ie: /root/bitcoind
|
||||
mount: String
|
||||
# read only data exposed to dependencies (path is relevant to mount)
|
||||
public: Option<String>
|
||||
# shared filesystem segment with each of your dependencies (path is relevant to mount)
|
||||
shared: Option<String>
|
||||
# deprecated - will default to an empty vector
|
||||
assets: []
|
||||
# version of tor support, eg. v2, v3
|
||||
hidden-service-version: String
|
||||
# A map of dependent services, see below for more details
|
||||
dependencies: Dependencies
|
||||
|
||||
.. _dependencies-spec:
|
||||
|
||||
Dependencies
|
||||
------------
|
||||
|
||||
Many services depend on other libraries and services on EmbassyOS (such as Bitcoin), sometimes even a particular version of those services, which need to be specified by the developers so that EmbassyOS can handle installing these dependencies under the hood.
|
||||
|
||||
The key of each field in the dependencies object is the lowercase, kebab-case app ID of the service that is depended on. Each dependency contains a set of rules that need to be fulfilled as true if the dependency is to be properly installed. The "config rules" here are for auto-configuring dependencies - the action defined by the rule will be executed if the service is auto configured with defaults during initial setup. This simplifies and streamlines the user experience. The interface should provide suggestions for the behavior if the denoted rule cannot be met with previous configurations.
|
||||
|
||||
Let's take this snippet for example:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
...
|
||||
dependencies:
|
||||
btc-rpc-proxy:
|
||||
version: "^0.1.0"
|
||||
optional: Can configure an external bitcoin node.
|
||||
description: Required for fetching validated blocks.
|
||||
config:
|
||||
- rule: '''users.*.name = "c-lightning"'
|
||||
description: 'Must have an RPC user named "c-lightning"'
|
||||
suggestions:
|
||||
- PUSH:
|
||||
to: 'users'
|
||||
value:
|
||||
name: c-lightning
|
||||
...
|
||||
|
||||
.. role:: raw-html(raw)
|
||||
:format: html
|
||||
|
||||
:raw-html:`<br />`
|
||||
|
||||
The service ``btc-rpc-proxy`` is a dependency of the service ``c-lightning``. ``c-lightning`` requires it to be installed at a version >=0.1.0 <0.2.0. There exists a rule that states the config option ``user.name`` must be equal to "c-lightning". If this value does not exist for ``user.name`` when accessed, ``PUSH`` the value "c-lighting" to the field. This all takes place during the initial service configuration, before the service is started for the first time.
|
||||
|
||||
.. note::
|
||||
Dependency config rules are processed in order.
|
||||
|
||||
Type definitions
|
||||
================
|
||||
|
||||
Types for ``manifest.yaml`` fields:
|
||||
|
||||
.. code:: typescript
|
||||
|
||||
interface Dependencies [{
|
||||
serviceId: DepInfo
|
||||
}]
|
||||
|
||||
interface DepInfo {
|
||||
version: String // ie. 0.11.1.1
|
||||
optional?: String,
|
||||
description?: String,
|
||||
config: [ConfigRule],
|
||||
],
|
||||
}
|
||||
|
||||
interface ConfigRule {
|
||||
rule: String, // ie. 'users.*.name = "lnd"
|
||||
description: String,
|
||||
suggestions: [SuggestionVariant]
|
||||
}
|
||||
|
||||
interface SuggestionVariant {
|
||||
SET: {
|
||||
var: String,
|
||||
to: SetVariant,
|
||||
},
|
||||
DELETE: {
|
||||
src: String,
|
||||
},
|
||||
PUSH: {
|
||||
to: String,
|
||||
value: Value,
|
||||
},
|
||||
}
|
||||
|
||||
interface SetVariant {
|
||||
to: Option<String>,
|
||||
to-value: Option<Value>, // ie. true/false
|
||||
to-entropy: Option<{
|
||||
charset: String // ie. 'a-z,A-Z,0-9'
|
||||
len: number
|
||||
}>
|
||||
}
|
||||
|
||||
.. role:: raw-html(raw)
|
||||
:format: html
|
||||
|
||||
:raw-html:`<br />`
|
||||
|
||||
Examples
|
||||
========
|
||||
|
||||
Actual ``manifest.yaml`` files for existing services:
|
||||
|
||||
LND
|
||||
...
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
compat: v0
|
||||
id: lnd
|
||||
version: 0.11.1.1
|
||||
title: Lightning Network Daemon
|
||||
description:
|
||||
short: "A complete implementation of a Lightning Network node by Lightning Labs"
|
||||
long: "LND fully conforms to the Lightning Network specification (BOLTs). BOLT stands for: Basis of Lightning Technology. In the current state lnd is capable of: creating channels, closing channels, managing all channel states (including the exceptional ones!), maintaining a fully authenticated+validated channel graph, performing path finding within the network, passively forwarding incoming payments, sending outgoing onion-encrypted payments through the network, updating advertised fee schedules, and automatic channel management (autopilot)."
|
||||
release-notes: https://github.com/lightningnetwork/lnd/releases/tag/v0.11.1-beta
|
||||
ports:
|
||||
- internal: 8080
|
||||
tor: 8080
|
||||
- internal: 9735
|
||||
tor: 9735
|
||||
- internal: 9911
|
||||
tor: 9911
|
||||
- internal: 10009
|
||||
tor: 10009
|
||||
image:
|
||||
type: tar
|
||||
mount: /root/.lnd
|
||||
public: public
|
||||
has-instructions: true
|
||||
os-version-required: ">=0.2.8"
|
||||
os-version-recommended: ">=0.2.8"
|
||||
install-alert: |
|
||||
READ CAREFULLY! LND and the Lightning Network are considered beta software. Please use with caution and do not risk more money than you are willing to lose. We encourage frequent backups. If for any reason, you need to restore LND from a backup, your on-chain wallet will be restored, but all your channels will be closed and their funds returned to your on-chain wallet, minus fees. It may also take some time for this process to occur.
|
||||
uninstall-alert: "READ CAREFULLY! Uninstalling LND will result in permanent loss of data, including its private keys for its on-chain wallet and all channel states. Please make a backup if you have any funds in your on-chain wallet or in any channels. Recovering from backup will restore your on-chain wallet, but due to the architecture of the Lightning Network, your channels cannot be recovered. All your channels will be closed and their funds returned to your on-chain wallet, minus fees. \n"
|
||||
restore-alert: |
|
||||
Restoring LND will overwrite its current data, including its on-chain wallet and channels. Any channels opened since the last backup will be forgotten and may linger indefinitely, and channels contained in the backup will be closed and their funds returned to your on-chain wallet, minus fees.
|
||||
assets: []
|
||||
hidden-service-version: v3
|
||||
dependencies:
|
||||
btc-rpc-proxy:
|
||||
version: "^0.2.4"
|
||||
optional: Can alternatively configure an external bitcoin node.
|
||||
description: Used to fetch validated blocks.
|
||||
config:
|
||||
- rule: '''users.*.name = "lnd"'
|
||||
description: 'Must have an RPC user named "lnd"'
|
||||
suggestions:
|
||||
- PUSH:
|
||||
to: "users"
|
||||
value:
|
||||
name: lnd
|
||||
allowed-calls: []
|
||||
- SET:
|
||||
var: 'users.[first(item => ''item.name = "lnd")].password'
|
||||
to-entropy:
|
||||
charset: "a-z,A-Z,0-9"
|
||||
len: 22
|
||||
- rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getinfo"'
|
||||
description: 'RPC user "lnd" must have "getinfo" enabled'
|
||||
suggestions:
|
||||
- PUSH:
|
||||
to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
|
||||
value: "getinfo"
|
||||
- rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getbestblockhash"'
|
||||
description: 'RPC user "lnd" must have "getbestblockhash" enabled'
|
||||
suggestions:
|
||||
- PUSH:
|
||||
to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
|
||||
value: "getbestblockhash"
|
||||
- rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "gettxout"'
|
||||
description: 'RPC user "lnd" must have "gettxout" enabled'
|
||||
suggestions:
|
||||
- PUSH:
|
||||
to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
|
||||
value: "gettxout"
|
||||
- rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getblockchaininfo"'
|
||||
description: 'RPC user "lnd" must have "getblockchaininfo" enabled'
|
||||
suggestions:
|
||||
- PUSH:
|
||||
to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
|
||||
value: "getblockchaininfo"
|
||||
- rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "sendrawtransaction"'
|
||||
description: 'RPC user "lnd" must have "sendrawtransaction" enabled'
|
||||
suggestions:
|
||||
- PUSH:
|
||||
to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
|
||||
value: "sendrawtransaction"
|
||||
- rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getblockhash"'
|
||||
description: 'RPC user "lnd" must have "getblockhash" enabled'
|
||||
suggestions:
|
||||
- PUSH:
|
||||
to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
|
||||
value: "getblockhash"
|
||||
- rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getblock"'
|
||||
description: 'RPC user "lnd" must have "getblock" enabled'
|
||||
suggestions:
|
||||
- PUSH:
|
||||
to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
|
||||
value: "getblock"
|
||||
- rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getblockheader"'
|
||||
description: 'RPC user "lnd" must have "getblockheader" enabled'
|
||||
suggestions:
|
||||
- PUSH:
|
||||
to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
|
||||
value: "getblockheader"
|
||||
- rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "estimatesmartfee"'
|
||||
description: 'RPC user "lnd" must have "estimatesmartfee" enabled'
|
||||
suggestions:
|
||||
- PUSH:
|
||||
to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
|
||||
value: "estimatesmartfee"
|
||||
- rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getnetworkinfo"'
|
||||
description: 'RPC user "lnd" must have "getnetworkinfo" enabled'
|
||||
suggestions:
|
||||
- PUSH:
|
||||
to: 'users.[first(item => ''item.name = "lnd")].allowed-calls'
|
||||
value: "getnetworkinfo"
|
||||
- rule: 'users.[first(item => ''item.name = "lnd")].fetch-blocks?'
|
||||
description: 'RPC user "lnd" must have "Fetch Blocks" enabled'
|
||||
suggestions:
|
||||
- SET:
|
||||
var: 'users.[first(item => ''item.name = "lnd")].fetch-blocks'
|
||||
to-value: true
|
||||
bitcoind:
|
||||
version: "^0.21.0"
|
||||
optional: Can alternatively configure an external bitcoin node.
|
||||
description: Used to subscribe to new block events.
|
||||
config:
|
||||
- rule: "zmq-enabled?"
|
||||
description: "Must have an ZeroMQ enabled"
|
||||
suggestions:
|
||||
- SET:
|
||||
var: "zmq-enabled"
|
||||
to-value: true
|
||||
|
||||
Cups
|
||||
....
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
compat: v0
|
||||
id: cups
|
||||
version: "0.3.6"
|
||||
title: "Cups Messenger"
|
||||
description:
|
||||
short: "Real private messaging"
|
||||
long: "Cups is a private, self-hosted, peer-to-peer, Tor-based, instant messenger. Unlike other end-to-end encrypted messengers, with Cups on the Embassy there are no trusted third parties."
|
||||
release-notes: |
|
||||
Features
|
||||
- Adds instructions defined by EmbassyOS 0.2.4 instructions feature
|
||||
ports:
|
||||
- internal: 59001
|
||||
tor: 59001
|
||||
- internal: 80
|
||||
tor: 80
|
||||
image:
|
||||
type: tar
|
||||
mount: /root
|
||||
has-instructions: true
|
||||
os-version-required: ">=0.2.4"
|
||||
os-version-recommended: ">=0.2.4"
|
||||
assets:
|
||||
- src: httpd.conf
|
||||
dst: "."
|
||||
overwrite: true
|
||||
- src: www
|
||||
dst: "."
|
||||
overwrite: true
|
||||
hidden-service-version: v3
|
||||
|
||||
.. role:: raw-html(raw)
|
||||
:format: html
|
||||
|
||||
:raw-html:`<br />`
|
||||
@@ -1,107 +0,0 @@
|
||||
.. _service_package_overview:
|
||||
|
||||
==================
|
||||
Packaging Overview
|
||||
==================
|
||||
|
||||
.. contents::
|
||||
:depth: 2
|
||||
:local:
|
||||
|
||||
There are several main components that comprise a service package for EmbassyOS. This overview will introduce them and following sections will dive into technical specifics.
|
||||
|
||||
First, let's get your system setup with the necessary software dependencies.
|
||||
|
||||
Environment Setup
|
||||
-----------------
|
||||
|
||||
At minimum, ``docker``, ``docker-buildx``, and ``embassy-sdk`` are required. The rest of the recommendations serve to optimize the build process.
|
||||
|
||||
Recommended Dependencies
|
||||
========================
|
||||
|
||||
`docker <https://docs.docker.com/get-docker>`_
|
||||
|
||||
`docker-buildx <https://docs.docker.com/buildx/working-with-buildx/>`_
|
||||
|
||||
`embassy-sdk <https://github.com/Start9Labs/embassy-os/tree/master/backend>`_
|
||||
|
||||
`cargo <https://doc.rust-lang.org/cargo/>`_
|
||||
|
||||
`yq (version 4) <https://mikefarah.gitbook.io/yq>`_
|
||||
|
||||
`make <https://www.gnu.org/software/make/>`_
|
||||
|
||||
`musl <https://github.com/Start9Labs/rust-musl-cross>`_
|
||||
|
||||
Package Components
|
||||
------------------
|
||||
|
||||
Image
|
||||
=====
|
||||
|
||||
Each service boils down to a Docker image. We're not going to dive into Docker specifics in this guide, since there exists tons of `resources <https://docs.docker.com/>`_ for developing with this containerization tool.
|
||||
|
||||
Because the service ultimately runs on a Raspberry Pi, the created Docker image should be a snapshot of an arm based linux environment. The service image is then mounted to the EmbassyOS image during installation. This path is defined in the :ref:`manifest <service-manifest>` configuration file.
|
||||
|
||||
The image is immutable, meaning that when the service is updated, the image is replaced with a completely new image containing the updated features.
|
||||
|
||||
Volumes
|
||||
=======
|
||||
|
||||
Each service application gets one or more volumes allocated to it by EmbassyOS. All service application data is stored in this directory and is persisted across updates.
|
||||
|
||||
The default volume directory (for seeding data into the volume) is typically: ``/root/volumes/<service-id>``.
|
||||
|
||||
.. warning:: Any files that are in the image at the volume path will be overwritten when a backup restore occurs.
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
When it comes to running your own server, managing dependencies is perhaps the most difficult part. The term "dependency hell" emerged from this sentiment. Even the best developers have lost significant amounts of time trying to find, correct, or clarify documentation for dependencies or dependency configuration. Individuals who manage their own servers have specific technical skills and are willing to devote the time and effort necessary to maintain them. Companies have entire departments dedicated to this task.
|
||||
|
||||
Some other personal server products aimed at non-technical individuals tackle this problem by simply hard coding dependencies. Since EmbassyOS is a platform for running all open-source, self-hosted software, it is not possible to hard code every possible service, service dependency, and service dependency configuration forever. Instead, Start9 built a new, unprecedented operating system that touts a generalized, intuitive approach to dependency management and service configuration. EmbassyOS enables users to easily and selectively install, uninstall, and update any service they want without getting stuck in dependency hell.
|
||||
|
||||
This may sound cool or neat, but it is more than that: *it's novel*. This has never been done before.
|
||||
|
||||
The key to making the system work is a new, domain-specific-language (DSL) and set of standards that are used by developers to define the rules and requirements of their services. Run in the context of EmbassyOS, these rules and requirements appear as simple UI elements, such as inputs, toggles, and drop downs, and they are enforced by validations and clear user instructions. Using this system, what previously required serious time and expertise, can now be done by anyone in a matter of seconds.
|
||||
|
||||
This DSL is utilized in the :ref:`config rules <config_rules>` and :ref:`dependencies <service-dependencies>` key in the service manifest.
|
||||
|
||||
Manifest
|
||||
========
|
||||
|
||||
This file describes the service and its requirements. It is used to:
|
||||
|
||||
- create a listing in the marketplace
|
||||
- denote any installation considerations
|
||||
- define dependency requirements
|
||||
|
||||
Each time a service is updated, the manifest should be updated to include the new version, release notes, and any pertinent updates to the install, uninstall, or restoration flows. All this information is displayed in the marketplace listing, and the optionally denoted alerts will be displayed when appropriate to provide the user with more information about the particularities of the service.
|
||||
|
||||
For instance, `LND displays alerts <https://github.com/Start9Labs/lnd-wrapper/blob/master/manifest.yaml#L28>`_ around restoration since data will be overwritten.
|
||||
|
||||
There is nothing you need to do as a developer to enable the service to run over Tor/LAN. This is *completely* handled by EmbassyOS - a Tor and LAN address will be automatically generated and an Nginx server configured (if a client application) when the service is installed.
|
||||
|
||||
Config
|
||||
======
|
||||
|
||||
Most self-hosted applications require the user to tell the app how to behave using a config file in a specific format, environment variables, command-line arguments, or some combination of these inputs. One of the coolest features of EmbassyOS is that, when packaged correctly, the app's :ref:`configuration <service-config>` will be available to the user as a slick GUI that always produces a valid configuration no matter how little experience or skill the user has.
|
||||
|
||||
With EmbassyOS, this means a service wrappers' configuration requires a particular format and rule structure to ensure it integrates smoothly with the user interface. This format enables clean handling of improper values and configuration dependencies.
|
||||
|
||||
.s9pk Bundle
|
||||
============
|
||||
|
||||
The configuration and manifest files get bundled into the ``.s9pk`` package, which is `built using embassy-sdk <https://github.com/Start9Labs/embassy-os/tree/master/backend>`_. Each new version release should include the updated version of these files re-bundled into a new binary. This is the file that will be downloaded from the marketplace. When the user clicks the service's "Install" button, EmbassyOS unpacks and installs the service.
|
||||
|
||||
Hello World Example
|
||||
===================
|
||||
|
||||
For reference, the `Hello world <https://github.com/Start9Labs/hello-world-wrapper>`_ repository should be used as an example. A project template can be cloned using the "Use this template" button in GitHub. For a detailed walkthrough of this example, please see the :ref:`Packaging Example<packaging-example>`.
|
||||
|
||||
|
||||
.. role:: raw-html(raw)
|
||||
:format: html
|
||||
|
||||
:raw-html:`<br />`
|
||||
@@ -1,25 +0,0 @@
|
||||
.. _service_properties:
|
||||
|
||||
==========
|
||||
Properties
|
||||
==========
|
||||
|
||||
The output of this step is a file titled ``stats.yaml``. This file contains a mapping of the values that will be displayed in the ``Properties`` section in a service's menu.
|
||||
|
||||
.. figure:: /_static/images/services/service-properties.png
|
||||
:width: 80%
|
||||
:alt: Service Properties
|
||||
|
||||
Service Properties
|
||||
|
||||
Due to the fact that config variables are only available when the service is running, this file can only be populated at runtime. In services Start9 has wrapped or created, we typically handle this in ``configurator/src/main.rs``. This file is essentially a script that acquires the necessary values at runtime from config (eg. ``/root/.lnd/start9/config.yaml``), and dynamically populating them to the ``stats.yaml`` file, marking each value with the appropriate keys for UI display (ie. masked or copyable). Running the configurator is typically included in the ``docker_entrypoint.sh`` file.
|
||||
|
||||
Example
|
||||
-------
|
||||
|
||||
A good example of the configurator producing the ``stats.yaml`` file can be found `here <https://github.com/Start9Labs/lnd-wrapper/blob/master/configurator/src/main.rs>`_.
|
||||
|
||||
.. role:: raw-html(raw)
|
||||
:format: html
|
||||
|
||||
:raw-html:`<br />`
|
||||
@@ -1,7 +0,0 @@
|
||||
.. _submission:
|
||||
|
||||
==================
|
||||
Submission Process
|
||||
==================
|
||||
|
||||
When you have built and tested your project for EmbassyOS, please send Start9 a submission with the project repository to ``dev@start9labs.com``. After being reviewed for security and compatibility, the service will be deployed to the marketplace and available for all EmbassyOS users to download.
|
||||
@@ -1,42 +0,0 @@
|
||||
.. _service_wrapper:
|
||||
|
||||
=======
|
||||
Wrapper
|
||||
=======
|
||||
|
||||
Each service is bound with a wrapper repository, which contains everything you need to build a service.
|
||||
|
||||
The purpose of this repo is:
|
||||
|
||||
- Denote any dependencies required to run and build the project
|
||||
- To define the necessary, ``config_rules.yaml``, ``config_spec.yaml`` and ``manifest.yaml`` options
|
||||
- To build the project into the ``.s9pk`` format digestible to EmbassyOS
|
||||
- Link to the source project as a git submodule
|
||||
- Define the docker file for running the project on EmbassyOS
|
||||
- Provide documentation for the project, especially user runbook instructions
|
||||
- symlink of ``instructions.md`` from ``docs`` directory to wrapper repo root, if included
|
||||
|
||||
File Structure
|
||||
--------------
|
||||
|
||||
The project structure should be used as a model:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
├── Dockerfile
|
||||
├── Makefile (optional)
|
||||
├── README.md
|
||||
├── config_rules.yaml
|
||||
├── config_spec.yaml
|
||||
├── <submodule_project_dir>
|
||||
├── docker_entrypoint.sh
|
||||
├── docs
|
||||
│ └── instructions.md
|
||||
└── manifest.yaml
|
||||
|
||||
Submodule
|
||||
---------
|
||||
|
||||
`Git submodules <https://www.git-scm.com/book/en/v2/Git-Tools-Submodules>`_ allow use of another project while in the working project directory. Setting up this feature enables linking of the source service repository so that its context is available. The submodule is added into the wrapper so that the wrapper can build the submodule and also track the exact code that's being built.
|
||||
|
||||
Run ``git submodule add <link_to_source_project>``
|
||||
@@ -11,16 +11,3 @@ Please follow the guide `here <https://github.com/Start9Labs/embassy-os/blob/mas
|
||||
Do I need to be a professional developer or have coding experience to contribute?
|
||||
---------------------------------------------------------------------------------
|
||||
Absolutely not! Contributions can be as simple as finding a typo in our docs, making a suggestion on GitHub, creating educational or promotional content, and the list goes on! Everyone that wants to contribute can do so in some unique way. Please don't hesitate to :ref:`Contact <contact>` us for ideas if you're not sure where to begin.
|
||||
|
||||
|
||||
|
||||
.. Expound the following into FAQs:
|
||||
.. --------------------------------
|
||||
|
||||
.. - Release notes need to be in quotations? As it is a string? (they aren’t by default) – this is a yaml thing and SOMETIMES they are required and SOMETIMES NOT
|
||||
.. - Alerts in LND example are not wrapped in quotes, same with CUPS release notes – yaml thing
|
||||
.. - Ports in manifest – can leave as default (80), as it is a docker container?? - Sure
|
||||
|
||||
.. - Type definitions – is this just for dependencies? Can a blurb be added here for extra clarity?
|
||||
.. - Config spec – Admin pass?
|
||||
|
||||
@@ -39,10 +39,6 @@ How can my service access the data directory of another service?
|
||||
|
||||
Check out `LND <https://github.com/Start9Labs/lnd-wrapper/blob/master/manifest.yaml>`_ and `RTL <https://github.com/Start9Labs/ride-the-lightning-wrapper/blob/master/manifest.yaml>`_ as an example.
|
||||
|
||||
The guide says that a Makefile is optional, why is this?
|
||||
--------------------------------------------------------
|
||||
The ``Makefile`` simplifies the development process by putting all your build steps into one place so that you can simply use the ``make`` to build with. A ``Makefile`` is not required however, and you may instead choose to use something like ``nix``, ``bash``, ``perl``, ``python``, or ``ruby`` for your build orchestration.
|
||||
|
||||
Why am I getting the error "No rule to make target yq, needed by manifest.yaml?"
|
||||
--------------------------------------------------------------------------------
|
||||
This is the message you get when you're missing the executable on your ``$PATH``.
|
||||
This is the message you get when you're missing the yq executable on your ``$PATH``. You can find details to `install yq here <https://github.com/mikefarah/yq#install>`_.`
|
||||
Reference in New Issue
Block a user