Start9/documentation
2
0
Fork 0
mirror of https://github.com/Start9Labs/documentation.git synced 2026-03-26 10:21:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

re-arrange the user docs (#76)

Browse Source 
* re-arrange the user docs

* Minor description edit

* update styles

* fix service links

Co-authored-by: kn0wmad <kn0wmad@protonmail.com>
Co-authored-by: Lucy Cifferello <12953208+elvece@users.noreply.github.com>
This commit is contained in:
Matt Hill
2022-02-18 08:49:53 -07:00
committed by GitHub
parent c38d5c1a0c
commit aed86e7e6c
151 changed files with 500 additions and 781 deletions
Download Patch File Download Diff File Expand all files Collapse all files

52
site/source/developer-docs/advanced/dev-tools/embassy-cli.rst Normal file
View File

@@ -0,0 +1,52 @@
.. _embassy-cli:
==============================
Embassy Command Line Interface
==============================
.. warning:: This is for advanced users only!! Anything you do while SSH'd into your Embassy is NOT SUPPORTED, unless under the guidance of a Start9 technician.
When interacting with Embassy directly, you will invariably want to use ``embassy-cli``. Embassy-CLI can control EmbassyOS in many of the same ways that the UI can, but with finer controls and deeper insights.
In order to use Embassy-CLI, you will first need to authorize yourself with the following command...
.. code-block:: bash
embassy-cli auth login
and enter your Embassy Master Password at the prompt.
You can enter ``embassy-cli help`` or ``embassy-cli <option> help`` to get an overview of capabilities:
.. code-block:: bash
Embassy CLI
USAGE:
embassy-cli [OPTIONS] <SUBCOMMAND>
FLAGS:
--help Prints help information
-V, --version Prints version information
OPTIONS:
-c, --config <config>
-h, --host <host>
-p, --proxy <proxy>
SUBCOMMANDS:
auth Login/logout and manage sessions
backup Manage backups
db Interact with the Database
disk Information on the external drive
echo Echoes
git-info Print the current git hash
help Prints this message or the help of the given subcommand(s)
inspect Details on services and their components
net Network information
notification Control UI notifications
package Interact with packages
server EmbassyOS operations and information
ssh Manage SSH keys
wifi Manage WiFi networks

43
site/source/developer-docs/advanced/dev-tools/embassy-sdk.rst Normal file
View File

@@ -0,0 +1,43 @@
.. _embassy-sdk:
================================
Embassy Software Development Kit
================================
.. warning:: This is for developers only!! Anything you do while SSH'd into your Embassy is NOT SUPPORTED, unless under the guidance of a Start9 technician.
You can install to your system by running the following from your ``embassy-os`` directory:
.. code-block:: bash
cd backend
./install-sdk.sh
You can enter ``embassy-sdk help`` or ``embassy-sdk <option> help`` to get an overview of capabilities:
.. code-block:: bash
embassy-sdk --help
Embassy SDK
USAGE:
embassy-sdk [OPTIONS] <SUBCOMMAND>
FLAGS:
-h, --help Prints help information
-V, --version Prints version information
OPTIONS:
-c, --config <config>
SUBCOMMANDS:
git-info Print current git hash
help Prints this message or the help of the given subcommand(s)
init Initialize
inspect Inspect a .s9pk
pack Package a service into the .s9pk format
verify Verify a .s9pk is valid
Please visit the `EmbassyOS backend <https://github.com/Start9Labs/embassy-os/tree/master/backend>`_ for more details

21
site/source/developer-docs/advanced/dev-tools/embassyd.rst Normal file
View File

@@ -0,0 +1,21 @@
.. _embassyd:
==============
Embassy Daemon
==============
.. warning:: This is for advanced users only!! Anything you do while SSH'd into your Embassy is NOT SUPPORTED, unless under the guidance of a Start9 technician.
``embassyd`` is the daemon that runs everything that could be considered EmbassyOS.
When SSH'd into your Embassy, you may see the status of EmbassyD with the following:
.. code-block:: bash
systemctl status embassyd
If you need to restart the daemon for some reason, you can do so with:
.. code-block:: bash
systemctl restart embassyd

15
site/source/developer-docs/advanced/dev-tools/index.rst Normal file
View File

@@ -0,0 +1,15 @@
.. _dev-tools:
===============
Developer Tools
===============
Brief overviews of Developer Tools that will come in handy when :ref:`packaging a service<developer-docs>`, or interacting directly with the OS to learn or contribute to the project.
.. toctree::
:maxdepth: 2
embassyd
embassy-cli
embassy-sdk
service-container

19
site/source/developer-docs/advanced/dev-tools/service-container.rst Normal file
View File

@@ -0,0 +1,19 @@
.. _exec-service-container:
==================
Service Containers
==================
For simplicity, EmbassyOS is designed to be completely operated via the :ref:`WebUI <web-ui>`, however, your device belongs to you and if you want to "lift the hood" and access the internals of the software, you may do so. This guide will show you how to enter a Service's Docker container in order to directly interface with it and issue commands.
.. warning:: The following guide is for those that have advanced command line skills, or those who are being guided by a Start9 support technician. Nothing you do inside a container is supported unless under the direction of Start9. Here be dragons!!
This guide assumes you already have :ref:`SSH Access<ssh>` setup on your machine.
Open a terminal, ssh into your device, and enter the following command, replacing ``<package-id>`` with the name of the service you are trying to access. You can list the services with ``embassy-cli package list`` (you will need to be logged in under ``embassy-cli`` for this). You will need to either be ``root`` or use ``sudo``.
.. code-block:: bash
docker exec -it <package-id>.embassy sh
This will give you a shell within the container. Reminder that nothing you do in here is supported, please be careful!

42
site/source/developer-docs/advanced/index.rst Normal file
View File

@@ -0,0 +1,42 @@
.. _advanced-packaging:
===========================
Advanced Packaging Overview
===========================
This section included advanced commands for working with packages in EmbassyOS.
.. raw:: html
<div class="topics-grid grid-container full">
<div class="grid-x grid-margin-x">
.. topic-box::
:title: Developer Tools
:link: dev-tools
:icon: scylla-icon scylla-icon--developers-blog
:class: large-4
:anchor: View
EmbassyOS developer CLI tools
.. topic-box::
:title: Service Containers
:link: dev-tools/service-container
:icon: scylla-icon scylla-icon--overview
:class: large-4
:anchor: View
How to interact with containers on the Embassy
.. raw:: html
</div></div>
.. This is for the side navigation display
.. toctree::
:maxdepth: 1
:hidden:
dev-tools/index

30
site/source/developer-docs/build-package-example/01_setup.rst Normal file
View File

@@ -0,0 +1,30 @@
.. _packaging-setup:
==========================
Step 1 - Context and Setup
==========================
What we'll build
----------------
We'll create a web application that produces a hello world page. This web application will be the service we package for the Embassy. During this guide we will:
1. Create a simple hello world web service
2. Create a Dockerfile
3. Create a Package Manifest
4. Create a hosted repository which will contain all the service components
5. Create the packaged service file
You can find the `complete code <https://github.com/Start9Labs/hello-world-wrapper>`_ referenced in this guide on GitHub.
Download required tools
-----------------------
If necessary, download any of the system requirements:
- Docker
- EmbassySDK
- A code editor
- Terminal
For more details and installation steps for these tools, visit the :ref:`environment setup <environment-setup>` page.

72
site/source/developer-docs/build-package-example/02_create-service.rst Normal file
View File

@@ -0,0 +1,72 @@
.. _packaging-create-service:
=======================
Step 2 - Create Service
=======================
For this example, we are going to create a simple Rust project that serves a static web page.
.. code:: bash
mkdir hello-world
cargo init
touch src/index.html
In ``index.html`` add:
.. code:: html
<html>
<head>
<title>Hello World!</title>
</head>
<body>
<h1>Hello World!</h1>
</body>
</html>
In ``main.rs`` add:
.. code:: rust
use hyper::service::{make_service_fn, service_fn};
use hyper::{body::Bytes, Body, Request, Response, Server};
use std::convert::Infallible;
use std::net::SocketAddr;
async fn handle(_req: Request<Body>) -> Result<Response<Body>, Infallible> {
Ok(Response::new(Body::from(Bytes::from_static(
include_bytes!("index.html"),
))))
}
#[tokio::main]
async fn main() {
// Construct our SocketAddr to listen on...
let addr = SocketAddr::from(([0, 0, 0, 0], 80));
// And a MakeService to handle each connection...
let make_service = make_service_fn(|_conn| async { Ok::<_, Infallible>(service_fn(handle)) });
// Then bind and serve...
let server = Server::bind(&addr).serve(make_service);
// And run forever...
if let Err(e) = server.await {
eprintln!("server error: {}", e);
}
}
**That's it!** We now have the code for our service.
Let's build and run it!
.. code:: bash
# build the project
cargo build
# start the executable
target/debug/hello-world
Visit `<http://localhost:80>`_ to see your running web page!

39
site/source/developer-docs/build-package-example/03_hardware-compile.rst Normal file
View File

@@ -0,0 +1,39 @@
.. _packaging-compile-raspi:
===========================
Step 3 - Build for Hardware
===========================
EmbassyOS is run on the arm-v8 architecture, specifically the aarch64 state, for the RaspberryPi.
Depending on the programming language or libraries used in a project, you might need to set up an environment to *cross compile* the executable for this runtime environment.
We were able to easily build and run our project locally. However, Rust is one of those programming languages that needs to be cross compiled. This step can be skipped if it does not apply to your project.
Since Start9 uses Rust throughout its service ecosystem, the team has built a helper to cross compile Rust projects for the Embassy.
This Rust cross compiler for aarch64 can be downloaded and build from `GitHub <https://github.com/Start9Labs/rust-musl-cross>`_:
.. code:: bash
git clone https://github.com/Start9Labs/rust-musl-cross.git
cd rust-musl-cross
chmod a+x ./build.sh
./build.sh
This actually builds a Docker container for us to use locally so we can emulate the environment we need to compile our project for aarch64!
Next, we enable cross-arch emulated builds in Docker
.. code:: bash
docker run --privileged --rm linuxkit/binfmt:v0.8
Finally, we run the following command to build the project, replacing `/absolute/path/to` with the output of `pwd` as necessary:
.. code:: bash
docker run --rm -it -v ~/.cargo/registry:/root/.cargo/registry -v "/absolute/path/to"/hello-world:/home/rust/src start9/rust-musl-cross:aarch64-musl cargo build --release
Awesome! We can now see the compiled executable here: `./hello-world/target/aarch64-unknown-linux-musl/release/hello-world`

77
site/source/developer-docs/build-package-example/04_docker-build.rst Normal file
View File

@@ -0,0 +1,77 @@
.. _packaging-docker-build:
==========================
Step 4 - Build with Docker
==========================
Now that we have our code properly built/compiled, we can create a Dockerfile. This file defines how to build the Docker image for the service by declaring the environment, building stages, and copying any binaries or assets needed to run the service to the Docker image filesystem.
In other words, the Dockerfile serves as a recipe for creating a Docker image, from which Docker containers are spun up. This is ultimately what runs an instance of your service on the Embassy.
1. Create the necessary Docker files:
.. code:: bash
touch Dockerfile
touch docker_entrypoint.sh
2. We'll start editing the ``Dockerfile`` by importing a base image, in this case Alpine, as recommended.
.. code:: docker
FROM arm64v8/alpine:3.12
3. Next, we issue some commands to setup the filesystem. Here we update repositories and install required system packages.
.. code:: docker
RUN apk update
RUN apk add tini curl
4. Next, we add the cross-compiled binary of ``hello-world`` to ``/usr/local/bin/`` and add the ``docker_entrypoint.sh`` file from the project root. Then, we set permissions for ``docker_entrypoint.sh``.
.. code:: docker
ADD ./hello-world/target/aarch64-unknown-linux-musl/release/hello-world /usr/local/bin/hello-world
ADD ./docker_entrypoint.sh /usr/local/bin/docker_entrypoint.sh
RUN chmod a+x /usr/local/bin/docker_entrypoint.sh
5. Next, we set a working directory, and set the location of the entrypoint. Exposing ports is not necessary for EOS, but its often useful to leave this line for quick reference and clarity.
.. code:: docker
WORKDIR /root
EXPOSE 80
ENTRYPOINT ["/usr/local/bin/docker_entrypoint.sh"]
6. **That's it!** Let's take a look at our final ``Dockerfile``:
.. code:: docker
FROM arm64v8/alpine:3.12
RUN apk update
RUN apk add tini
ADD ./hello-world/target/aarch64-unknown-linux-musl/release/hello-world /usr/local/bin/hello-world
ADD ./docker_entrypoint.sh /usr/local/bin/docker_entrypoint.sh
RUN chmod a+x /usr/local/bin/docker_entrypoint.sh
WORKDIR /root
# not necessary for EmbassyOS, but often left for quick reference and clarity
EXPOSE 80
ENTRYPOINT ["/usr/local/bin/docker_entrypoint.sh"]
7. Finally, add the following code to the ``docker_entrypoint.sh``:
.. code:: bash
#!/bin/sh
exec tini hello-world
This is a script that defines how the service starts, and often acts as an init system. It will need to complete any environment setup (such as folder substructure), set any environment variables, and execute the run command. It's also PID 1 in the Docker container, so should do all of the signal handling for container exits.

257
site/source/developer-docs/build-package-example/05_create-manifest.rst Normal file
View File

@@ -0,0 +1,257 @@
.. _packaging-create-manifest:
========================
Step 5 - Create Manifest
========================
The Manifest file specifies the details EmbassyOS needs to operate a service. It is the connection point between your service and EmbassyOS.
In this file, values and actions exist for:
- Displaying the service in the marketplace
- Specifying the project assets (eg. icon, instructions, license)
- Defining the docker mount points
- Specifying how to configure the service
- Relaying how to run health checks, backups, and other custom actions
- Outlining dependency relationships (if applicable) and configuration rules for dependencies
- Denoting copy to display in EmbassyUI elements, alerts, descriptions
This file can be written in:
- yaml
- toml
- json
Let's create a yaml manifest file for our hello-world project:
.. code:: bash
touch manifest.yaml
And populate it with the following example manifest (see the line comments for a description of each key and view the full :ref:`type specification here <service_manifest>`):
.. code:: yaml
# The package identifier used by the OS
id: hello-world
# A human readable service title
title: "Hello World"
# Service version - accepts up to four digits, where the last confirms to revisions necessary for EmbassyOS - see documentation: https://github.com/Start9Labs/emver-rs
version: 0.3.0
# Release notes for the update - can be a string, paragraph or URL
release-notes: "Upgrade to EmbassyOS v0.3.0"
# The type of license for the project. Include the LICENSE in the root of the project directory. A license is required for a Start9 package.
license: mit
# The Start9 wrapper repository URL for the package. This repo contains the manifest file (this), any scripts necessary for configuration, backups, actions, or health checks (more below). This key must exist. But could be embedded into the source repository.
wrapper-repo: "https://github.com/Start9Labs/hello-world-wrapper"
# The original project repository URL. There is no upstream repo in this example
upstream-repo: "https://github.com/Start9Labs/hello-world-wrapper"
# URL to the support site / channel for the project. This key can be omitted if none exists, or it can link to the original project repository issues.
support-site: "https://docs.start9.com/"
# URL to the marketing site for the project. This key can be omitted if none exists, or it can link to the original project repository.
marketing-site: "https://start9.com/"
# The series of commands to build the project into an s9pk for arm64/v8. In this case we are using a Makefile with the simple build command "make".
build: ["make"]
# Minimum required version of EmbassyOS
min-os-version: "0.3.0"
# Human readable descriptors for the service. These are used throughout the EmbassyOS user interface, primarily in the marketplace.
description:
# This is the first description visible to the user in the marketplace.
short: Example service
# This description will display with additional details in the service's individual marketplace page
long: |
Hello World is a simple example of a service wrapper that launches a web interface to say hello and nothing more.
# These assets are static files necessary for packaging the service for Start9 (into an s9pk). Each value is a path to the specified asset. If an asset is missing from this list, or otherwise denoted, it will be defaulted to the values denoted below.
assets:
# Default = LICENSE.md
license: LICENSE
# Default = icon.png
icon: icon.png
# Default = INSTRUCTIONS.md
instructions: instructions.md
# Default = image.tar
docker-images: image.tar
# The main action for initializing the service. Currently, the only type of action available is docker.
main:
# Docker is currently the only action implementation
type: docker
# Identifier for the main image volume, which will be used when other actions need to mount to this volume.
image: main
# The executable binary for starting the initialization action. For docker actions, this is typically a "docker_entrypoint.sh" file. See the Dockerfile and the docker_entrypoint.sh in this project for additional details.
entrypoint: "docker_entrypoint.sh"
# Any arguments that should be passed into the entrypoint executable
args: []
# Specifies where to mount the data volume(s), if there are any. Mounts for pointer dependency volumes are also denoted here. These are necessary if data needs to be read from / written to these volumes.
mounts:
# Specifies where on the service's file system its persistence directory should be mounted prior to service startup
main: /root
# Health checks
health-checks:
main:
name: Web Interface
description: Ensures the network interface is accessible via HTTP.
type: docker
image: main
entrypoint: "sh"
args: ["-c", "curl --silent --show-error --fail http://loacalhost:80"]
# When `inject` is true, the health check will use the main image to run the health check. This is faster as there is no need to spin up an additional docker container
# When `inject` is false, the health check will use whatever image is specified. This is useful when using a system image with additional utilities to run a health check. If inject=false, then system must equal true
inject: true
# Optional if false - indicates if an image that is preloaded onto the system will be used
system: false
# Required - valid values are yaml, toml, json
io-format: json
# Specifies how to get and set configuration file values for the service. NOTE: This stanza can be left empty (null) if the service has no configuration options.
config:
# The config action to run to get the specified config file (default is config.yaml)
get:
# The type of implementation to run this action (currently, only Docker is available)
type: docker
# The Docker image to run the action command in. This could be the service's main image, or an image that is preloaded onto the system, like compat (which holds compatible helper functions for default functionality)
image: compat
# Indicates if an image that is preloaded onto the system will be used
system: true
# The initial run command to execute the config get action
entrypoint: compat
# Any arguments that need to be passed into the run command
args:
- config
- get
- /root
- "/mnt/assets/config_spec.yaml"
# The locations at which to mount the specified Docker images
mounts:
compat: /mnt/assets
main: /root
# Required - valid values are yaml, toml, json
io-format: yaml
# The config action to run to set the specified config file (default is config.yaml). Details for the keys below are the same as above.
set:
type: docker
image: compat
system: true
entrypoint: compat
args:
- config
- set
- hello-world
- /root
- "/mnt/assets/config_rules.yaml"
mounts:
compat: /mnt/assets
main: /root
io-format: yaml
# This is a key value map specifying dependent services that this service needs in order to function. The keys are the package id's on which you depend. NOTE: if developing a standalone service, you may leave this stanza as an empty object (the key dependencies is required)
dependencies:
# Key must be the package id of another service that exists in the marketplace
filebrowser:
# The version range that is acceptable for this dependency
version: "^2.14.1.1"
# Describes if the dependency is critical to the service functioning. If the dependency is critical, the service will stop if this dependency is stopped.
critical: false
# Specifies the requirement type of the dependency
requirement:
# "Opt-out" means the dependency will be required according to the default config. "Opt-in" means the dependency may be required if you change the config. And "required" just means it's always required.
type: "opt-out"
# An explanation of how to opt-in or opt-out. This value is optional for type=required
how: Optionally use the selected dependency
# Description of the dependency relationship
description: A dependency that demonstrates the way to configure a dependent service
# This is a list of rules that levies requirements on the configuration of the dependency and suggests ways to remedy any incompatibilities. Documentation of this feature is outside the scope of this example.
config: ~
# This denotes any data, asset, or pointer volumes that should be connected when the "docker run" command is invoked
volumes:
# This is the image where files from the project asset directory will go
main:
type: data
# This is an example of an asset volume
compat:
type: assets
# This specifies how to configure the port mapping for exposing the service over TOR and LAN (if applicable). Many interfaces can be specified depending on the needs of the service. If it can be launched over a Local Area Network connection, specify a `lan-config`. Otherwise, at minimum, a `tor-config` must be specified.
interfaces:
# This key is the internal name that the OS will use to configure the interface
main:
# A human readable name for display in the UI
name: Network Interface
# A descriptive description of what the interface does
description: Specifies the interface to listen on for HTTP connections.
tor-config:
# Port mappings are from the external port to the internal container port
port-mapping:
80: "80"
# Port mappings are from the external port to the internal container port
lan-config:
80:
ssl: false
internal: 80
# Denotes if the service has a user interface to display
ui: true
# Denotes the protocol specifications used by this interface
protocols:
- tcp
- http
# Alerts: omitting these will result in using the default alerts in EmbassyOS, except for start, which has no default.
alerts:
install-alert: This is an alert that will present before the user installs this service
uninstall-alert: This is an alert that will present before the user uninstalls this service
restore-alert: This is an alert that will present before the user restores this service from Embassy backup
start-alert: This is an alert that will present before the user starts this service
# Specifies how backups should be run for this service. The default EmbassyOS provided option is to use the duplicity backup library on a system image (compat)
backup:
create:
# Currently, only docker actions are supported.
type: docker
# The docker image to use. In this case, a pre-loaded system image called compat
image: compat
# Required if the action uses a system image. The default value is false.
system: true
# The executable to run the command to begin the backup create process
entrypoint: compat
# Arguments to pass into the entrypoint executable. In this example, the full command run will be: `compat duplicity hello-world /mnt/backup /root/data`
args:
- duplicity
- hello-world
- /mnt/backup
# For duplicity, the backup mount point needs to be something other than `/root`, so we default to `/root/data`
- /root/data
mounts:
# BACKUP is the default volume that is used for backups. This is whatever backup drive is mounted to the device, or a network filesystem.
# The value here donates where the mount point will be. The backup drive is mounted to this location.
BACKUP: "/mnt/backup"
main: "/root"
# The action to execute the backup restore functionality. Details for the keys below are the same as above.
restore:
type: docker
image: compat
system: true
entrypoint: compat
args:
- duplicity
- hello-world
- /root/data
- /mnt/backup
mounts:
BACKUP: "/mnt/backup"
main: "/root"
# Commands that can be issued from the UI. NOTE: if no actions are required, this section can be left as an empty object
actions:
hello-world-action:
name: Hello World Action
description: A description that describes what the action will accomplish.
warning: |
A warning message indicating and potential dangers associated with the action
# Indicates what state the service can be in while executing the action
allowed-statuses:
- running
# Defines how the action is run
implementation:
type: docker
image: main
entrypoint: sh
args: ["-c", "echo 'hello-world'"]
# Same as note on health-checks
inject: true
# Required - valid values are yaml, toml, json
io-format: json
**That's it!** You now have the structure for your service's Manifest.

52
site/source/developer-docs/build-package-example/06_create-assets.rst Normal file
View File

@@ -0,0 +1,52 @@
.. _packaging-create-assets:
======================
Step 6 - Create Assets
======================
Each ``s9pk`` package contains a set of assets. They include:
- Instructions
- License
- Icon
Instructions
------------
An instructions file is a convenient way to share any steps users should take to setup or interact with your service. This file gets displayed within an EmbassyUI component and should be written in `Markdown <https://www.markdownguide.org/>`_ language.
Let's add instructions to our hello world project:
.. code:: bash
touch instructions.md
And add the following code to the file:
.. code:: bash
# Instructions for Hello World
Instructions go here. These appear to the user in the UI on the Service page under 'Instructions.'
License
-------
Start9 ensures that the proper license is displayed for all open source software running on an EmbassyOS platform. Let's make sure to include the full open source license so users can view the distribution permissions of your service, among other licensing details.
The name and location of this file should be specified in the ``assets.license`` section of the Manifest. The default value if not specified is ``LICENSE``, located in the root of the project folder.
.. code:: bash
touch ./hello-world/LICENSE
Icon
----
Icons are displayed throughout the EmbassyUI to reference to your service.
Simply add the icon file to the root of the project directory. The icon file can be named anything, but this must be specified in the ``assets.icon`` section of the Manifest. The default filename the SDk looks for when packaging the service assets is ``icon.png``.
.. code:: bash
mv /local/path/to/icon ./hello-world/icon.png

23
site/source/developer-docs/build-package-example/07_create-s9pk.rst Normal file
View File

@@ -0,0 +1,23 @@
.. _packaging-create-s9pk:
==========================
Step 7 - Package into s9pk
==========================
We now have all of the necessary components to package the service into the format needed for the OS. This format is a custom filetype with an extension of ``.s9pk``, short for Start9 Package.
To package all components into an ``.s9pk``, run the following command from the root of your project directory:
.. code:: bash
embassy-sdk pack
Let's also make sure to verify the validity of the package:
.. code:: bash
embassy-sdk verify s9pk /path/to/hello-world.s9pk
If anything goes wrong, an error message will indicate the missing component or other failure.
**That's it!**

92
site/source/developer-docs/build-package-example/08_create-makefile.rst Normal file
View File

@@ -0,0 +1,92 @@
.. _packaging-makefile:
========================
Step 8 - Create Makefile
========================
For convenience and repeatability, let's combine all of these commands into a Makefile. Then, we can use `make <https://www.gnu.org/software/make/>`_ to rebuild our project quickly.
.. code:: bash
touch Makefile
1. Add the build rule with the target executable as the key, including a list of dependencies needed to build the target file. In this case, the ``hello-world`` binary compiled for aarch is the target, and the dependencies are the hello-world source files needed to compile this binary:
.. code:: bash
HELLO_WORLD_SRC := $(shell find ./hello-world/src) hello-world/Cargo.toml hello-world/Cargo.lock
hello-world/target/aarch64-unknown-linux-musl/release/hello-world: $(HELLO_WORLD_SRC)
docker run --rm -it -v ~/.cargo/registry:/root/.cargo/registry -v "$(shell pwd)"/hello-world:/home/rust/src start9/rust-musl-cross:aarch64-musl cargo build --release
2. Add the step to build the Docker image. Here, the target is the Docker `image.tar` artifact, and the dependencies are the Dockerfile, docker_entrypoint.sh, and the aarch64 compiled hello-world executable:
.. code:: bash
image.tar: Dockerfile docker_entrypoint.sh hello-world/target/aarch64-unknown-linux-musl/release/hello-world
DOCKER_CLI_EXPERIMENTAL=enabled docker buildx build --tag start9/hello-world/main:$(VERSION) --platform=linux/arm64 -o type=docker,dest=image.tar .
3. Next, add the step for building the ``s9pk`` package, with the ``hello-world.s9pk`` as the target, and all the component files as the dependencies:
.. code:: bash
ASSETS := $(shell yq e '.assets.[].src' manifest.yaml)
hello-world.s9pk: manifest.yaml assets/compat/config_spec.yaml assets/compat/config_rules.yaml image.tar instructions.md $(ASSET_PATHS)
embassy-sdk pack
4. Then, add the step to verify the package:
.. code:: bash
S9PK_PATH=$(shell find . -name hello-world.s9pk -print)
verify: hello-world.s9pk $(S9PK_PATH)
embassy-sdk verify s9pk $(S9PK_PATH)
5. Add steps to clean up the Makefile build artifacts when you want to build from a fresh slate:
.. code:: bash
clean:
rm -f image.tar
rm -f hello-world.s9pk
6. Finally, add the ``all`` make target.
.. code:: bash
all: verify
This serves as the entrypoint to build multiple targets, which we have in this case. When the ``make`` command is invoked here, it looks for the "verify" target. Since the "verify" target depends on the "hello-world.s9pk" target, make then runs this target. It continues down this graph until the first target and its dependencies are satisfied, then works its way back up. The final output of this Makefile is the ``image.tar`` and ``hello-world.s9pk`` files.
**That's it!** Our completed Makefile looks like this:
.. code:: make
ASSETS := $(shell yq e '.assets.[].src' manifest.yaml)
ASSET_PATHS := $(addprefix assets/,$(ASSETS))
VERSION := $(shell yq e ".version" manifest.yaml)
HELLO_WORLD_SRC := $(shell find ./hello-world/src) hello-world/Cargo.toml hello-world/Cargo.lock
S9PK_PATH=$(shell find . -name hello-world.s9pk -print)
# delete the target of a rule if it has changed and its recipe exits with a nonzero exit status
.DELETE_ON_ERROR:
all: verify
verify: hello-world.s9pk $(S9PK_PATH)
embassy-sdk verify s9pk $(S9PK_PATH)
clean:
rm -f image.tar
rm -f hello-world.s9pk
hello-world.s9pk: manifest.yaml assets/compat/config_spec.yaml assets/compat/config_rules.yaml image.tar instructions.md $(ASSET_PATHS)
embassy-sdk pack
image.tar: Dockerfile docker_entrypoint.sh hello-world/target/aarch64-unknown-linux-musl/release/hello-world
DOCKER_CLI_EXPERIMENTAL=enabled docker buildx build --tag start9/hello-world/main:$(VERSION) --platform=linux/arm64 -o type=docker,dest=image.tar .
hello-world/target/aarch64-unknown-linux-musl/release/hello-world: $(HELLO_WORLD_SRC)
docker run --rm -it -v ~/.cargo/registry:/root/.cargo/registry -v "$(shell pwd)"/hello-world:/home/rust/src start9/rust-musl-cross:aarch64-musl cargo build --release

65
site/source/developer-docs/build-package-example/09_create-wrapper.rst Normal file
View File

@@ -0,0 +1,65 @@
.. _packaging-create-wrapper:
============================
Step 9 - Create Wrapper Repo
============================
In order for the Start9 team to review your package for submission to the Start9 Marketplace, we ask that you create a wrapper repository for the project and its components. This repo should contain everything you need to build the service.
Let's do that for our hello-world service.
For a quick start convenience, Start9 has made the finalized version of the `hello-world-wrapper <https://github.com/Start9Labs/hello-world-wrapper>`_ available as a *GitHub template*. Clicking "Use this template" in that repository will clone the entire contents to a specified location. Each file will still have to be manually edited to reflect the changes necessary for your service.
If you want to proceed from scratch, follow these steps:
1. In GitHub, create a new public repository with the name "hello-world-wrapper" under your user profile. Go ahead and select the options to include a README file and a .gitignore file. You can always add these files later too.
2. Once the hosted repository is created, select the "Code" dropdown to copy the https or ssh URL for the repository. If you do not have git setup locally, follow the :ref:`setup steps <environment-setup-git>` first.
.. code:: bash
git clone https://github.com/<username>/hello-world-wrapper.git
cd hello-world-wrapper
3. Include the ``hello-world`` project in the wrapper repo. It can either be included directly, or it can be hosted separately. If it is hosted separately, it should be included as a `git submodule <https://git-scm.com/book/en/v2/Git-Tools-Submodules>`_ within the wrapper repository:
.. code:: bash
git submodule add <link_to_source_project>
4. Edit the ``.gitignore`` file to include the ``.s9pk`` file and ``image.tar`` bundle. This will exclude these files from being published remotely, as they can be large or binary representations.
.. code:: bash
hello-world.s9pk
image.tar
5. Move the ``Dockerfile``, ``docker_entrypoint.sh``, ``LICENSE``, ``icon``, and ``Manifest`` to the root of the wrapper repository. At the end, your project structure should look similar to this:
.. code:: bash
├── Dockerfile
├── LICENSE
├── Makefile
├── README.md
├── assets
│ └── compat
│ ├── config_rules.yaml
│ └── config_spec.yaml
├── docker_entrypoint.sh
├── hello-world
│ ├── Cargo.lock
│ ├── Cargo.toml
│ ├── src
│ │ ├── index.html
│ │ └── main.rs
│ └── target
│ ├── aarch64-unknown-linux-musl
│ ├── debug
│ └── release
├── hello-world.s9pk
├── icon.png
├── image.tar
├── instructions.md
└── manifest.yaml

65
site/source/developer-docs/build-package-example/10_install-package.rst Normal file
View File

@@ -0,0 +1,65 @@
.. _packaging-install:
=================
Step 10 - Install
=================
Now that we have a process for iterating on producing a valid package for EmbassyOS, let's load it onto an Embassy!
1. First, generate an ssh key for the Embassy:
.. code:: bash
ssh-keygen -t ed25519
# Press Enter to leave filename as default
# Press Enter to leave password empty
# Press Enter to confirm password is empty
# Copy file contents to clipboard. This is your ssh pubkey.
pbcopy .ssh/id_ed25519.pub
2. On an Embassy device, enter the ssh pubkey into your SSH settings:
#. Click on Embassy in the menu
#. Click on SSH under SETTINGS
#. Click on + Add new key
#. Paste pubkey from clipboard
3. Copy the ``hello-world.s9pk`` to the Embassy device:
.. code:: bash
# Confirm you can ssh into your Embassy
ssh root@<lan-url>
# Log out of Embassy SSH session
exit
scp <package-id>.s9pk ssh root@<lan-url>:/working/directory/path
eg. scp hello-world.s9pk ssh root@embassy-12345678.local:/root
4. Finally, install the package on an Embassy device:
.. code:: bash
ssh root@<lan-url>
# log in to the command line interface using the Embassy password
embassy-cli auth login
embassy-cli package install hello-world.s9pk
**Congratulations!** You have successfully created and installed a package you created onto EmbassyOS. The package should now be viewable in the "Services" tab in EmbassyUI.
From here, you can play with viewing the results of your Manifest file settings, such as config, actions, interfaces, health checks, etc. You can also view the logs of your service right in the UI!
In order to verify your service is functioning as expected:
- Ensure your service is in "Running" state
- Make sure there are no apparent errors or warnings in the logs
- Ensure each UI component renders as expected:
- Instructions
- Config
- Properties
- Actions
- Interfaces
- Marketplace listing
- Donation
- Launch or use your service in the intended way and make sure all aspects function

122
site/source/developer-docs/build-package-example/index.rst Normal file
View File

@@ -0,0 +1,122 @@
.. _build-package-example:
========================
Build Your First Package
========================
This guide will take you through the fundamentals of packaging a service for EmbassyOS by creating a real service. Let's get started!
.. raw:: html
<div class="topics-grid grid-container full">
<div class="grid-x grid-margin-x">
.. topic-box::
:title: Step 1
:link: 01_setup
:class: large-4
:anchor: View
Context and environment setup
.. topic-box::
:title: Step 2
:link: 02_create-service
:class: large-4
:anchor: View
Create a service
.. topic-box::
:title: Step 3
:link: 03_hardware-compile
:class: large-4
:anchor: View
Compile for Embassy hardware
.. topic-box::
:title: Step 4
:link: 04_docker-build
:class: large-4
:anchor: View
Build service with Docker
.. topic-box::
:title: Step 5
:link: 05_create-manifest
:class: large-4
:anchor: View
Create Manifest file
.. topic-box::
:title: Step 6
:link: 06_create-assets
:class: large-4
:anchor: View
Create package assets
.. topic-box::
:title: Step 7
:link: 07_create-s9pk
:class: large-4
:anchor: View
Package service components together
.. topic-box::
:title: Step 8
:link: 08_create-makefile
:class: large-4
:anchor: View
Create Makefile with build steps (optional)
.. topic-box::
:title: Step 9
:link: 09_create-wrapper
:class: large-4
:anchor: View
Create wrapper repository to host code for review
.. topic-box::
:title: Step 10
:link: 10_install-package
:class: large-4
:anchor: View
Install packaged service on EmbassyOS
.. topic-box::
:title: Resources
:link: resources
:class: large-4
:anchor: View
Get help, learn how to submit your package to a marketplace, and example references
.. raw:: html
</div></div>
.. This is for the side navigation display
.. toctree::
:maxdepth: 1
:hidden:
01_setup
02_create-service
03_hardware-compile
04_docker-build
05_create-manifest
06_create-assets
07_create-s9pk
08_create-makefile
09_create-wrapper
10_install-package
resources

31
site/source/developer-docs/build-package-example/resources.rst Normal file
View File

@@ -0,0 +1,31 @@
.. _packaging-resources:
=========
Resources
=========
Get help
--------
If you get stuck or are having issues debugging why your service is not packaging or running as expected, reach out to our community `dev chat <https://matrix.to/#/#community-dev:matrix.start9labs.com>`_ with questions.
Submission Process
------------------
When you have built and tested your project for EmbassyOS, please send Start9 a submission to dev@start9labs.com with a link to the wrapper repository. After being reviewed for security and compatibility, the service will be deployed to the Start9 Marketplace and available for all EmbassyOS users to download.
If you are deploying to an alternative marketplace, please shout it out in our community channels!
References
----------
Below are links to working examples of more advanced configurations for current Embassy services. They might serve as a starting point or reference during your development process:
- Detailed ``docker_entrypoint.sh`` - `Filebrowser <https://github.com/Start9Labs/filebrowser-wrapper/blob/master/docker_entrypoint.sh>`_
- Optional dependencies - `BTCPay Server <https://github.com/Start9Labs/btcpayserver-wrapper/blob/master/assets/compat/dependencies.yaml>`__
- Config spec - `BTCPay Server <https://github.com/Start9Labs/btcpayserver-wrapper/blob/master/assets/compat/config_spec.yaml>`__
- Config rules - `BTC RPC Proxy <https://github.com/Start9Labs/btc-rpc-proxy-wrapper/blob/master/assets/compat/config_rules.yaml>`__
- Multiple dependencies - `BTCPay Server <https://github.com/Start9Labs/btcpayserver-wrapper/blob/master/manifest.yaml#L172-L187>`__
- Actions - `BTCPay Server <https://github.com/Start9Labs/btcpayserver-wrapper/blob/master/actions/btcpay-admin.sh>`__
- Health checks - `BTCPay Server <https://github.com/Start9Labs/btcpayserver-wrapper/blob/master/assets/utils/health_check.sh>`__
- Alerts - `BTCPay Server <https://github.com/Start9Labs/btcpayserver-wrapper/blob/master/manifest.yaml#L207-L218>`__

95
site/source/developer-docs/getting-started/environment-setup.rst Normal file
View File

@@ -0,0 +1,95 @@
.. _environment-setup:
===========================
Packaging Environment Setup
===========================
To get started packaging a service for EmbassyOS, some basic knowledge of software development is required. Don't worry if you are inexperienced, we will provide enough context to get you started, and you can always reach out with questions.
If you are already an experienced developer, :ref:`jump ahead <environment-quick-start>`.
The only system requirements are `Docker <https://docs.docker.com/get-docker>`_ and `Cargo <https://doc.rust-lang.org/cargo/>`_ (Rust package manager).
Of course, there are additional recommendations that would streamline your development experience, but they are not required.
Code Editor
-----------
A code editor is recommended since you will, after all, be writing code! We are fans of `Visual Studio Code <https://code.visualstudio.com/>`_ . Visual Studio Code is a free, batteries-included text editor made by Microsoft.
Terminal / CLI
--------------
We recommend using your computer's built-in terminal as a command line interface (CLI) for your operating system. For Windows users, we recommend the built-in command line (cmd) *Command Prompt* or the Powershell CLI, running in Administrator mode. For macOS and Linux, the native *Terminal* is recommended, but virtually any terminal will work.
If you are new to the command line, Ionic has published this awesome `guide <https://ionicframework.com/blog/new-to-the-command-line/>`_ to get you started.
.. _environment-setup-git:
Git
---
Although not required, the version control system Git is highly recommended.
Git is often accompanied by a Git Host, such as `GitHub <https://github.com/>`_, in which case additional setup is required. Follow the tutorial from the GitHub `here <https://docs.github.com/en/get-started/quickstart/set-up-git>`_ to set up your environment.
To verify the installation, open a new terminal window and run:
.. code:: bash
git --version
Docker
------
`Docker <https://docs.docker.com/get-docker>`_ must be installed to your computer platform. It is needed to build an image for your package, which will be used to create the running instance of your package on EmbassyOS. In essence, it declares the necessary environment and building stages for your package to run.
We also recommend installing and using `Docker buildx <https://docs.docker.com/buildx/working-with-buildx/>`_, as this adds desirable new features to the Docker build experience. It is included by default with Docker Desktop for Windows and macOS.
Cargo
-----
Cargo is the package management solution for the Rust programming language. It will install Rust to your system, and provide the required environment to build the Embassy SDK for packaging your service into the required format needed by EmbassyOS.
Installation instructions for Cargo can be found `here <https://doc.rust-lang.org/cargo/getting-started/installation.html>`__.
To verify the installation, open a terminal window and run:
.. code:: bash
cargo --version
SDK
---
EmbassyOS provides a :ref:`software development kit <sdk>` embedded in its environment. You do not need a running instance of EmbassyOS to use this component, it can be installed on any computer platform.
To install the SDK:
.. code:: bash
git clone https://github.com/Start9Labs/embassy-os.git
cd backend
./install-sdk.sh
To verify the installation, open a terminal window and run:
.. code:: bash
embassy-sdk --version
EmbassyOS
---------
While not strictly necessary, having a running instance of EmbassyOS is recommended to test installing, running, configuring, and using your package. Without this component, you will have to coordinate with Start9's internal development team to test.
.. _environment-quick-start:
Quick Start Environment Setup
-----------------------------
- `Docker <https://docs.docker.com/get-docker>`_
- `Docker buildx <https://docs.docker.com/buildx/working-with-buildx/>`_
- `Cargo <https://doc.rust-lang.org/cargo/getting-started/installation.html>`__
- `EmbassyOS <https://github.com/Start9Labs/embassy-os>`_
- `Embassy SDK <https://github.com/Start9Labs/embassy-os/blob/master/backend/install-sdk.sh>`_

63
site/source/developer-docs/getting-started/index.rst Normal file
View File

@@ -0,0 +1,63 @@
.. _packaging-getting-started:
========================
Getting Started Overview
========================
Select an option below to get started with packing a service for EmbassyOS.
.. raw:: html
<div class="topics-grid grid-container full">
<div class="grid-x grid-margin-x">
.. topic-box::
:title: Environment Setup
:link: environment-setup
:icon: scylla-icon scylla-icon--integrations
:class: large-4
:anchor: View
Get your environment setup to optimize service packaging
.. topic-box::
:title: Build your first Package
:link: build-package-example/index
:icon: scylla-icon scylla-icon--open-source
:class: large-4
:anchor: Begin
Follow along to learn packaging fundamentals with an example
.. topic-box::
:title: Packaging Quick Start
:link: packaging-quick-start
:icon: scylla-icon scylla-icon--roadmap
:class: large-4
:anchor: Begin
Quickly get started with concise packaging steps
.. topic-box::
:title: SDK
:link: sdk
:icon: scylla-icon scylla-icon--apps
:class: large-4
:anchor: View
Discover details about Start9's Software Development Kit
.. raw:: html
</div></div>
.. This is for the side navigation display
.. toctree::
:maxdepth: 1
:hidden:
environment-setup
sdk
quick-start

33
site/source/developer-docs/getting-started/quick-start.rst Normal file
View File

@@ -0,0 +1,33 @@
.. _packaging-quick-start:
=====================
Packaging Quick Start
=====================
This guide outlines a checklist of steps to complete in order to package a service for EmbassyOS. For a more in depth example, visit the :ref:`build your first package <build-package-example>` guide.
Packaging
---------
1. Create or select project
2. Build project
3. Cross compile for armv8/aarch64 if necessary
4. Create Dockerfile and docker entrypoint file
5. Add build steps, compiled executables, assets, etc in Dockerfile and specify run command in docker entrypoint, handling signal exiting if the service does not already do this gracefully
6. Create Manifest file
7. Create instructions file
8. Create icon file
9. Add license
10. Package all components into s9pk using embassy-sdk
11. Verify package components using embassy-sdk
12. Create a wrapper repository on GitHub to host all package assets for review
13. (Optional) Add all package build steps to a Makefile for replicability
Testing:
--------
1. Sideload ``<package-id>.s9pk`` onto an Embassy
2. Install package using embassy-cli
3. Start package using embassy-cli, or in the UI
4. Check logs to see if errors
5. Ensure service is reachable/launchable

97
site/source/developer-docs/getting-started/sdk.rst Normal file
View File

@@ -0,0 +1,97 @@
.. _sdk:
===========
Embassy SDK
===========
Embassy SDK, or Software Development Kit, is a CLI (Command Line Interface) tool that aids in building and packaging services you wish to deploy to the Embassy.
It mainly helps you validate that the necessary components of your package exist, and package all of those components into a special file type that is understood by EmbassyOS.
To install, checkout the SDK step in :ref:`setting up your environment <environment-setup>`.
Commands
========
To see a list of all available commands provided, run the following from an terminal window:
.. code:: bash
embassy-sdk --help
embassy-sdk init
----------------
Initialize the developer key for interacting with the SDK.
By default, this creates the developer key at `/etc/embassy`. You might need to change ownership of this folder depending on your system permissions.
.. code:: bash
chown <user> /etc/embassy
Alternatively, you can write a config file with your desired developer location, it simply needs the following format:
.. code:: yaml
developer-key-path: /desired/path/to/key
And load it by running:
.. code:: bash
embassy-sdk -c /path/to/config init
embassy-sdk pack
----------------
This command takes the necessary package components and assembles them into the `s9pk` file format needed to install a service on EmbassyOS. It expects the following files to exist:
- Manifest
- Instructions
- License
- Icon
If this command fails, the error response will indicate which component is missing.
embassy-sdk verify
-------------------
This command verifies aspects about the components assembled into the `s9pk`, such as:
- Ensures that all mounts are real volumes in the manifest
- Ensures all cert volumes point to real interfaces in the manifest
- Ensures all actions refer to real images in the manifest
- Ensures all images are tagged correctly in the manifest
- Ensures the icon is less than 100KB
It should be run _after_ `embassy-sdk pack` in order to verify the validity of each component.
If this command fails, the error message will indicate the mismatched details.
embassy-sdk git-info
--------------------
This command outputs the git commit hash of the SDK version installed on your platform.
embassy-sdk inspect
-------------------
This command contains several utilities for reading components once packaged into the `s9pk`. In development, it can be helpful to determine if each component is successfully included in the `s9pk` package.
It contains the following subcommands, and requires the path to the `<pacakge-id>.s9pk` file as the last argument:
- docker-images
- hash
- icon
- instructions
- license
- manifest
For example:
.. code:: bash
embassy-sdk inspect instructions /path/to/<package-id>.s9pk

89
site/source/developer-docs/index.rst Normal file
View File

@@ -0,0 +1,89 @@
.. _developer-docs:
==============
Developer Docs
==============
Welcome to Service Packaging for EmbassyOS!
If you are here, you are interested in becoming part of the mission to change the future of personal computing.
The guides below provide the fundamentals that will take you through the process of packing a service for EmbassyOS. Services are any open source project (application) that can be run on a self-hosted platform, independent of third parties.
By configuring and packaging a project according to these guides, it can be installed on EmbassyOS so that users can interact with the service without needing any technical expertise.
Let's get started!
.. raw:: html
<div class="topics-grid grid-container full">
<div class="grid-x grid-margin-x">
.. topic-box::
:title: Getting Started
:link: getting-started
:icon: scylla-icon scylla-icon--nsql-guides
:class: large-4
:anchor: View
Set up your environment and follow along with an example
.. topic-box::
:title: Build your first Package
:link: build-package-example
:icon: scylla-icon scylla-icon--open-source
:class: large-4
:anchor: Begin
Follow along to learn packaging fundamentals with an example
.. topic-box::
:title: Packaging Quick Start
:link: getting-started/quick-start
:icon: scylla-icon scylla-icon--roadmap
:class: large-4
:anchor: Begin
Quickly get started with concise packaging steps
.. topic-box::
:title: SDK
:link: getting-started/sdk
:icon: scylla-icon scylla-icon--apps
:class: large-4
:anchor: View
Discover details about the Start9 Software Development Kit
.. topic-box::
:title: Full Specification
:link: specification
:icon: scylla-icon scylla-icon--glossary
:class: large-4
:anchor: View
Detailed service packaging specification and advanced features
.. topic-box::
:title: Advanced Guides
:link: advanced
:icon: scylla-icon scylla-icon--integrations
:class: large-4
:anchor: View
Guides for implementing advanced service configurations
.. raw:: html
</div></div>
.. This is for the side navigation display
.. toctree::
:maxdepth: 1
:hidden:
getting-started/index
build-package-example/index
specification/index
advanced/index

41
site/source/developer-docs/specification/backups.rst Normal file
View File

@@ -0,0 +1,41 @@
.. _service_backups:
=======
Backups
=======
Backups are defined by the service developer in the backups section of the Manifest. By default, this action is a command executed within a Docker container.
EmbassyOS provides a system utility for creating backups. This system utility is the `duplicity <https://en.wikipedia.org/wiki/Duplicity_(software)>`_ library that is run inside the Docker image entitled _compat_.
The path to be backed up can be specified in the Manifest. In the create backup example below, it is the last argument:
.. code:: yaml
...
# create backup
entrypoint: compat
# Arguments to pass into the entrypoint executable. In this example, the full command run will be: `compat duplicity <package-id> /mnt/backup /root/data`
args:
- duplicity
- <package-id>
- /mnt/backup
# For duplicity, the backup mount point needs to be something other than `/root`, so we default to `/root/data`
- /root/data
...
For restoration purposes, it might be beneficial to ignore certain files or folders. For instance, ignore the shared/public folder that is mounted for dependencies that expose this feature as it causes data inconsistencies on restore.
In this case, create a ``.backupignore`` file in the mounted volume directory. This file contains a list of relative paths to the ignored files.
Example
-------
The `BTCPay Server wrapper <https://github.com/Start9Labs/btcpayserver-wrapper/blob/master/configurator/src/templates/.backupignore.templates>`_ demonstrates a good use of a backupignore template.
Ultimately, ``/datadir/.backupignore`` gets populated with:
.. code::
/root/volumes/btcpayserver/start9/public
/root/volumes/btcpayserver/start9/shared

21
site/source/developer-docs/specification/compat/index.rst Normal file
View File

@@ -0,0 +1,21 @@
.. _compat:
============
Compat Image
============
With the release of EmbassyOS v0.3.0, system utility Docker images are preloaded for service packager convenience.
The ``Compat`` image is a backwards compatible Docker image that hosts EmbassyOS features used in the v0.2.x series. It was created as a convenience for service packagers to help migrate their service configurations from 0.2.x to 0.3.0.
It exposes functionality to make use of:
- ``ConfigRules`` language
- Duplicity backups
.. This is for the side navigation display
.. toctree::
:hidden:
rules

111
site/source/developer-docs/specification/compat/rules.rst Normal file
View File

@@ -0,0 +1,111 @@
.. _config_rules:
============
Config Rules
============
This file defines the configuration rules, or the rule-set that defines dependencies between config variables. In practice, config rules are for auto-configuring self dependencies. Self dependencies are internal dependencies of a service, such as if the setting of one config variable informs the option of another setting. These "dependencies" are configured as rules.
A rule is a boolean expression that we demand to be true. It is not true if the expression fails the rule parser. Set of predicates. xt
They follow the `Backus–Naur <https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_form>`_ meta-syntax for writing rules.
Rules are composed of two main concepts:
* Variables - accessor into a configuration
* Terms - either a variable or type literal (ie. a boolean term is a boolean variable, a boolean expression, or a comparison operation between numbers or strings)
Variables can be booleans, numbers, or strings, and have a different syntax depending on the type. These type annotations check your config rules against your config spec and throw an error if invalid.
- ``?`` - Casts to boolean value. If the value is not a boolean, this notes whether or not the value is null.
- ``#`` - Treat the value as a number. If it is not a number, the value will be parsed as NaN. String numbers are not currently supported.
- ``'`` - Cast the value into a string. Applies to any value except for an object or a list.
- ``!`` - Equals not.
.. note::
Config rules are processed in order.
If application does not satisfy a rule, a set of suggestions should be provided. These suggestions are in the form of the operation to preform:
- ``Set`` - set the value
- ``Push`` - add to the value (such as to a list)
- ``Delete`` - delete the value
.. code:: typescript
enum SuggestionVariant = Set | Delete | Push
interface Set {
var: String, // fully qualified path without typecast
// one of the following three variants are required
to: Option<String> // a string expression, use when tying another config value
to-value: Option<String>
to-entropy: Option<{
charset: String (eg. 'a-z,A-Z,0-9')
len: Number
}>
}
interface Delete {
src: String, // path to key - removes if in a list
}
interface Push {
to: String,
value: String, // string literal of value to be set
}
SET Examples
------------
.. code:: yaml
- SET:
# the key in config you want to set
var: 'users.[first(item => ''item.name = "c-lightning")].password'
# the value in config that you will set
to-entropy:
charset: "a-z,A-Z,0-9"
len: 22
- SET:
var: 'users.[first(item => ''item.name = "c-lightning")].fetch-blocks'
to-value: true
PUSH Examples
-------------
.. code:: yaml
- PUSH:
to: "users"
value:
name: c-lightning
allowed-calls: []
- PUSH:
to: 'users.[first(item => ''item.name = "c-lightning")].allowed-calls'
value: "getnetworkinfo"
Full example from `c-lightning manifest <https://github.com/Start9Labs/c-lightning-wrapper/blob/master/manifest.yaml>`_:
.. code:: yaml
config:
- rule: '''users.*.name = "c-lightning"'
description: 'Must have an RPC user named "c-lightning"'
suggestions:
- PUSH:
to: "users"
value:
name: c-lightning
allowed-calls: []
- SET:
var: 'users.[first(item => ''item.name = "c-lightning")].password'
to-entropy:
charset: "a-z,A-Z,0-9"
len: 22

490
site/source/developer-docs/specification/config-spec.rst Normal file
View File

@@ -0,0 +1,490 @@
.. _config_spec:
===========
Config Spec
===========
Introduction
============
Most self-hosted applications require the user to tell the application how to behave using a config file in a specific format, environment variables, command-line arguments, or some combination of these inputs. One of the coolest features of EmbassyOS is that the services' configuration will be available to the user as a slick GUI that always produces a valid configuration no matter how little experience or skill the user has.
With EmbassyOS, this means a services' configuration requires a file to define the particular format to ensure it integrates smoothly with the user interface. This format enables clean handling of improper values and dependency management.
This file defines the *structure* of the service's native config and should be curated according to the ``ConfigSpec`` type, which is a detailed mapping of the configuration options with acceptable values, defined patterns, and defaults.
For example, if the user chooses config option A, then config option B must be between 5 and 10. This enables a simple GUI configuration experience, complete with validations and protections, for users. They do not have to worry about the consequences of a wrong value or manually editing a config file.
Purpose
=======
The ``ConfigSpec`` exists primarily an input specification and secondarily for input validation.
.. figure:: /_static/images/services/service5.png
:width: 80%
:alt: Synapse Config
The file containing the ``ConfigSpec`` defines the structure of configuration options your service depends on to run. It additionally can include configuration options that users might want to enable for more advanced or customized usage. Ultimately, these values influence the UI elements for a user to interact with. Specifically, they evaluate to the options available when managing a service, such as:
- Prior to service installation when the user needs to be made aware of any necessary dependency configurations
- When the user installs a service and the service is in the "Needs Config" state
- Whenever a user edits a service config
- When config pointers get updated
The neat part about this file is that each ``ValueSpec`` type gets translated into a specific front end component. For instance, boolean values display as a toggle button, such as in the Synapse example of ``Enable Registration``.
.. figure:: /_static/images/services/synapseconfig.png
:width: 80%
:alt: Example boolean toggle
Another advantage is the ability to define default values. These values automatically get populated if the user selects the ``Default`` option when setting up a service in ``Needs Config`` state. This is incredibly convenient for users who want to get up and running quickly.
Implementation
==============
The following section contains implementation specifications for the structure of the file containing the ``ConfigSpec``. This config specification if of the type:
.. code-block::
:caption: ConfigSpec
key: ValueSpec
ValueSpec Type: Boolean | Enum | List | Number | Object | String | Union | Pointer
- All keys are ``kebab-case`` strings, which correspond to the package id
- All values are one the following specs (ie. ``ValueSpec`` type):
- :ref:`boolean <boolean>`
- :ref:`enum <enum>`
- :ref:`list <list>`
- :ref:`number <number>`
- :ref:`object <object>`
- :ref:`string <string>`
- :ref:`union <union>`
- :ref:`pointer <pointer>`
- In the examples for each value spec type below, ``Option`` means the key is optional. Otherwise, the key is required.
- Descriptions are optional, but recommended
- Name corresponds to the name of the config field
- Find a complete example :ref:`here <example_config_spec>`
- Change warning text displays when the value is altered
Specs
=====
These are the possible value types with examples for each key of the config specification.
.. _boolean:
Boolean
-------
Config value specification denoted as a boolean value. A default value is required.
.. code-block::
:caption: ValueSpec Type
type: boolean
name: String
description: Option<String>
warning: Option<String>
default: Boolean
Example
^^^^^^^^
.. code:: yaml
fetch-blocks:
type: boolean
name: Fetch Blocks
description: Fetch blocks from the network if pruned from disk
default: true
.. _enum:
Enum
----
Config value specification denoted as an enum value. Enums values must be a unique set. If no default is provided, ``null`` will be the assumed value.
.. code-block::
:caption: ValueSpec Type
type: enum
name: String
description: Option<String>
warning: Option<String>
default: Option<Enum>
values: Set<String>
Example
^^^^^^^
.. code:: yaml
theme-mode:
type: enum
name: Theme Mode
values:
- NIGHT
- DAY
valueNames:
NIGHT: Night
DAY: Day
default: NIGHT
.. _list:
List
----
The list type describes an array of values. The values must consist of the same subtype, which can be any of the ValueSpec types available in the EmbassyOS config specification.
Lists of any type do not contain the default for each item in list. The list *itself* can have a default. If no default is provided, ``null`` will be the assumed value.
Range is loosely based off mathematical range syntax, with infinity replaced with ``*``:
``[ || ]`` = inclusive
``( || )`` = noninclusive
``*`` = infinity on either end
eg:
.. code::
[0,*) - all numbers to infinity including 0
.. code-block::
:caption: ValueSpec Type
type: list
name: String
description: Option<String>
subtype: Enum<enum | number | object | string | union>
range: NumRange<unsigned integer>
spec: ValueSpec
default: ValueSpec
Example:
^^^^^^^^
.. code:: yaml
allowed-calls:
type: list
name: Allowed Calls
description: The list of all RPC methods this user is allowed to make
subtype: enum
range: "[0, *)"
spec:
type: enum
values:
- item
- item
.. _number:
Number
------
A number value within an optionally defined range. Nullable field is required. If ``nullable`` is true, the default is assumed to be ``null`` if it is not provided.
.. code-block::
:caption: ValueSpec Type
type: number
name: String
description: Option<String>
change-warning: Option<String>
default: Boolean
nullable: Boolean
range: Option<NumRange<64 bit floating point>>
integral: Boolean
units: Option<String>
Example
^^^^^^^
.. code:: yaml
type: number
name: Peer Message Timeout
description: How long to wait for a response from a peer before failing
nullable: false
integral: true
units: Seconds
range: "[0, *)"
default: 30
.. _object:
Object Type
-----------
A nested representation of a ``ConfigSpec``. The object type takes the same structure under the ``spec`` key as a ``ConfigSpec``: a key indicates the field name, and the value denotes the ``ValueSpec`` type for that field.
.. code-block::
:caption: ValueSpec Type
type: object
name: String
description: Option<String>
warning: Option<String>
display-as: Option<String>
# indicates whether duplicates can be permitted
unique-by: UniqueBy
spec: ConfigSpec
type UniqueBy = null | string | { any: UniqueBy[] } | { all: UniqueBy[] }
Example
^^^^^^^
.. code:: yaml
type: object
name: Advanced
description: Advanced settings for Bitcoin Proxy
nullable: false
spec:
tor-only:
type: boolean
name: Only Tor Peers
description: Use Tor for all peer connections
default: false
peer-timeout:
type: number
name: Peer Message Timeout
description: How long to wait for a response from a peer before failing
nullable: false
integral: true
units: Seconds
range: "[0, *)"
default: 30
max-peer-age:
type: number
name: Maximum Peer Age
description: How long to wait before refreshing the peer list
nullable: false
integral: true
units: Seconds
range: "[0, *)"
default: 300
max-peer-concurrency:
type: number
name: Maximum Peer Concurrency
description: How many peers to reach out to concurrently for block data
nullable: true
integral: true
range: "[1, *)"
default: 1
.. _string:
String
------
There are various options for string values. They can optionally be marked as copyable or masked, such as for passwords, which will reflect the UI element display. A pattern, expressed in regex, can be denoted. If it exists, this field requires both the pattern type (ie. Regex) and pattern description (ie. an explanation of the pattern requirements).
If the default type is ``Entropy``, the charset can optionally specify an inclusive ranged character set (ie. "a-f,0-9").
.. code-block::
:caption: ValueSpec Type
type: string
name: String
description: Option<String>
warning: Option<String>
masked: Option<boolean>
copyable: Option<boolean>
# Placeholder text in UI input box
placeholder: Option<String>
nullable: Boolean
default: String | Entropy
pattern: Option<Regex>
pattern-description: Option<String>
.. code-block::
:caption: Entropy Type
charset: Option<String>
len: integer
Examples
^^^^^^^^
.. code::
color:
type: string
name: Color
description: Color value for the Lightning Network
nullable: false
pattern: "[0-9a-fA-F]{6}"
pattern-description: |
Must be a valid 6 digit hexadecimal RGB value. The first two digits are red, middle two are green and final two are
blue
default:
charset: "a-f,0-9"
len: 6
password:
type: string
name: Password
description: The password for the RPC User
nullable: false
copyable: true
masked: true
default:
charset: "a-z,A-Z,0-9"
len: 22
.. _pointer:
Pointer
--------
The pointer type *points* to a config value on another service installed on EmbassyOS (ie. package subtype) or to the EmbassyOS system (ie. system subtype). When pointing to another service, the ``index`` field indicates the path to the desired config variable.
.. code-block::
:caption: ValueSpec Type
type: pointer
name: String
description: Option<String>
warning: Option<String>
subtype: Enum< package | system>
package-id: String (*always* kebab case)
target: AppPointerSpecVariants | SystemPointerSpecVariants
index: Option<String> (dependent on target being AppPointerSpecVariants)
AppPointerSpecVariants = TorAddress | TorKey | Config
SystemPointerSpecVariants = HostIp
Example
^^^^^^^
.. code::
user:
type: pointer
name: RPC Username
description: The username for the RPC user for Bitcoin Core
subtype: package
package-id: bitcoind
target: config
index: "rpc.username"
.. _union:
Union
-----
This type describes a necessary dependency. Multiple variants can be expressed to enable the user the option to connect to another service (internal dependency) or outside source (external dependency).
For example, the Bitcoin Proxy service is united with an instance of Bitcoin. Three variants are defined: internal, external, and a quick connect. In this case, internal refers to the Bitcoin Core instance running on EmbassyOS, and defines the RPC credentials necessary for connecting; external refers to a Bitcoin Core node running on a different device, and defines the RPC credentials necessary for connecting; quick connect refers to yet another method of connecting to a Bitcoin Core node, optimized for convenience.
Default is required and corresponds to one of the variants.
``Tag`` is the key that will be rendered on the UI element.
.. code-block::
:caption: ValueSpec Type
type: union
name: String
description: Option<String>
change-warning: Option<String>
default: Boolean
tag: Tag
variants: Map<String, ConfigSpec>
display-as: Option<String>
unique-by: any | all | exactly | notUnique
.. code-block::
:caption: Tag Type
id: String
name: String
description: Option<String>
variant-names: Map<String, String>
.. _example_config_spec:
Example
^^^^^^^
.. code:: yaml
bitcoind:
type: union
name: Bitcoin Core
description: The Bitcoin Core node to connect to
tag:
id: type
name: Type
description: |
- Internal: The Bitcoin Core service installed to your Embassy
- External: A Bitcoin Core node running on a different device
- Quick Connect: A Quick Connect URL for an unpruned Bitcoin Core node
variant-names:
internal: Internal
external: External
quick-connect: Quick Connect
default: internal
variants:
internal:
user:
type: pointer
name: RPC Username
description: The username for the RPC user for Bitcoin Core
subtype: app
app-id: bitcoind
target: config
index: "rpc.username"
password:
type: pointer
name: RPC Password
description: The password for the RPC user for Bitcoin Core
subtype: app
app-id: bitcoind
target: config
index: "rpc.password"
external:
addressext:
type: string
name: Public Address
description: The public address of your Bitcoin Core RPC server
nullable: false
userext:
type: string
name: RPC Username
description: The username for the RPC user on your Bitcoin Core RPC server
nullable: false
passwordext:
type: string
name: RPC Password
description: The password for the RPC user on your Bitcoin Core RPC server
nullable: false
quick-connect:
quick-connect-url:
type: string
name: Quick Connect URL
description: The Quick Connect URL for your Bitcoin Core RPC server
nullable: false
pattern: 'btcstandup://[^:]*:[^@]*@[a-zA-Z0-9.-]+:[0-9]+(/(\?(label=.+)?)?)?'
patternDescription: Must be a valid Quick Connect URL. For help, check out https://github.com/BlockchainCommons/Gordian/blob/master/Docs/Quick-Connect-API.md

142
site/source/developer-docs/specification/dependencies.rst Normal file
View File

@@ -0,0 +1,142 @@
.. _dependencies-spec:
============
Dependencies
============
Background
----------
When it comes to running your own server, managing dependencies is perhaps the most difficult part. The term "dependency hell" emerged from this sentiment. Even the best developers have lost significant amounts of time trying to find, correct, or clarify documentation for dependencies or dependency configuration. Individuals who manage their own servers have specific technical skills and are willing to devote the time and effort necessary to maintain them. Companies have entire departments dedicated to this task.
Some other personal server products aimed at non-technical individuals tackle this problem by simply hard coding dependencies. Since EmbassyOS is a platform for running all open-source, self-hosted software, it is not possible to hard code every possible service, service dependency, and service dependency configuration forever. Instead, Start9 built a new, unprecedented operating system that touts a generalized, intuitive approach to dependency management and service configuration. EmbassyOS enables users to easily and selectively install, uninstall, and update any service they want without getting stuck in dependency hell.
This may sound cool or neat, but it is more than that: *it's novel*. This has never been done before.
The key to making the system work is a new, domain-specific-language (DSL) and set of standards that are used by developers to define the rules and requirements of their services. Run in the context of EmbassyOS, these rules and requirements appear as simple UI elements, such as inputs, toggles, and drop downs, and they are enforced by validations and clear user instructions. Using this system, what previously required serious time and expertise, can now be done by anyone in a matter of seconds.
This DSL is utilized in the :ref:`config rules <config_rules>` and :ref:`dependencies <service-dependencies>` key in the service manifest.
Context
-------
Many services depend on other libraries and services on EmbassyOS (such as Bitcoin), sometimes even a particular version of those services. These need to be specified by the developers so that EmbassyOS can handle installing these dependencies under the hood.
Requirement
-----------
Dependencies are defined in the Manifest.
The key of each field in the dependencies object is the lowercase, kebab-case package id of the service that is depended on.
Only the type of requirement must be defined. Requirement types include:
- "Required" - which means it's always required.
- "Opt-in" - which means the dependency may be required if you change the service's config ie. you can opt into the selection
- "Opt-out" - which means the dependency will be required according to the default service config ie. you can opt out of the selection
Advanced Configuration
----------------------
If a dependency requires a more advanced configuration, rule checks and auto configurations can be implemented. These functions utilize a preloaded Docker image within EmbassyOS entitled _compat_. This system utility provides the mechanisms for checking dependency configuration compatibility by providing a file with the defined rules as input.
In this case, each dependency contains a set of rules that need to be fulfilled as true if the dependency is to be properly installed. The "config rules" here are for auto-configuring dependencies - the action defined by the rule will be executed if the service is auto configured with defaults during initial setup. This simplifies and streamlines the user experience. The interface should provide suggestions for the behavior if the denoted rule cannot be met with previous configurations.
Example
-------
Here is an example from BTCPay Server.
First, an except from the dependencies section of the Manifest. We see here that ``bitcoind`` is a dependency of BTCPay Server. It is required to be installed at at least version 0.21.1.2.
There also exists a rule that states the ``advanced.peers.listen`` config option in bitcoind must be equal to ``true``. If this value is not set as such, ``SET`` the value to true.
These actions all take place during the initial service configuration, before the service is started for the first time.
.. code:: yaml
...
dependencies:
# Key must be the package id of another service that exists in the marketplace
bitcoind:
# The version range that is acceptable for this dependency
version: "^0.21.1.2"
# Describes if the dependency is critical to the service functioning. If the dependency is critical, the service will stop if this dependency is stopped.
critical: false
requirement:
# "Opt-out" means the dependency will be required according to the default config. "Opt-in" means the dependency may be required if you change the config. And "required" just means it's always required.
type: "opt-in"
# An explanation of how to opt-in or opt-out. This value is optional for type=required
how: Can alternatively configure an external bitcoind node.
# Description of the necessity of the dependency relationship
description: Used to fetch validated blocks.
# This defines the actions to check dependency rules and auto configure them if possible
config:
# This levies requirements on the configuration of the dependency and suggests ways to remedy any incompatibilities.
check:
type: docker
image: compat
system: true
entrypoint: compat
args:
- dependency
- check
- btcpayserver
- "bitcoind"
- /datadir
- "/mnt/assets/bitcoind_config_rules.yaml"
mounts:
main: /datadir
compat: /mnt/assets
io-format: yaml
# This implements default values on the configuration of the dependency
auto-configure:
type: docker
image: compat
system: true
entrypoint: compat
args:
- dependency
- "auto-configure"
- btcpayserver
- "bitcoind"
- /datadir
- "/mnt/assets/bitcoind_config_rules.yaml"
mounts:
main: /datadir
compat: /mnt/assets
io-format: yaml
...
Secondly, an except from it's dependency's configuration rules file:
.. code:: yaml
- rule: "advanced.peers.listen?"
description: Peer port must be listening on the network.
suggestions:
- SET:
var: advanced.peers.listen
to-value: true
.. note::
Dependency config rules are processed in order.
Optional Dependencies
---------------------
Configuring a service to have multiple optional dependencies is also possible. This can be done by defining a file that specifies the condition under which the dependency should become activated.
For example, in BTCPay Server's config, a user can select either LND or C-lightning as an internal lightning node implementation. To do this, we define the following file:
.. code:: yaml
lnd:
condition: '''lightning.type = "lnd"'
health_checks: []
c-lightning:
condition: '''lightning.type = "c-lightning"'
health_checks: []
This file gets passed in as an argument during BTCPay Server's config set function in its Manifest, shown `here <https://github.com/Start9Labs/btcpayserver-wrapper/blob/master/manifest.yaml#L86>`__.

107
site/source/developer-docs/specification/docker.rst Normal file
View File

@@ -0,0 +1,107 @@
.. _service_docker:
==========
Dockerfile
==========
This purpose of the Dockerfile is to define how to build the image for the service. It declares the environment and building stages.
The Dockerfile is responsible for mounting the service application to whatever volume the manifest specifies, typically ``/root/.<service-id>``.
Since the build requires specific arm runtime environments, these base images can be imported into the Dockerfile as a starting point so the base system does not need to be completely redefined. This enables importing a specific build environment version that encapsulates any necessary libraries for running the service build, eg. golang, rust.
For instance:
``FROM alpine:3.12``
``FROM arm32v7/alpine``
``FROM arm32v7/golang:alpine``
We recommended using ``alpine`` since it produces the smallest image. We try to keep the image under 100MB when possible.
Image
-----
Each service boils down to a Docker image. We're not going to dive into Docker specifics in this guide, since there exists tons of `resources <https://docs.docker.com/>`__ for developing with this containerization tool.
The image is immutable, meaning that when the service is updated, the image is replaced with a completely new image containing the updated features.
When installed, the service image is mounted to the EmbassyOS image. This path is defined in the :ref:`manifest <service_manifest>` configuration file:
.. code:: yaml
...
main:
type: docker
image: main
entrypoint: "docker_entrypoint.sh"
args: []
mounts:
# Specified where to mount the volume in the container
main: /root
io-format: yaml
...
In EmbassyOS, the mount specified in the Manifest gets bound to ``/embassy-data/package-data/volumes/main/<service-id>``.
Volumes
-------
Volumes are the mechanism for persisting data generated by and used by Docker containers. In EmbassyOS, services can declare as one or many volumes in the Manifest:
.. code:: yaml
volumes:
# This key correlates to the image name specified above. The default is "main".
main:
type: data
Volume Types
^^^^^^^^^^^^
1. ``data`` - A volume to persist application data. All service application data is stored in this directory and is persisted across updates.
2. ``assets`` - A volume to store miscellaneous assets that do not need to be in the main image.
If an asset volume is specified, there must exist a folder entitled assets, and a subfolder entitled the name of the asset volume:
.. code-block::
:emphazize-lines: 9,10,11,16,17
properties:
type: docker
image: compat
system: true
entrypoint: compat
args:
- properties
- /root
mounts:
main: /root
compat: /mnt/assets
volumes:
# This is the image where files from the project asset directory will go
main:
type: data
compat:
type: assets
# required project folder structure
├── assets
│ └── compat
.. warning:: Any files in the mounted volume path will be overwritten when a backup restore occurs.
Entrypoint
----------
The ``docker_entrypoint.sh`` defines what to do when the service application starts.
It consists of a bash script that completes any environment setup (eg. creating folder substructure), sets any environment variables, and executes the run command. The only required feature of this file is to execute the run command for the service.
Example
-------
The `LND wrapper <https://github.com/Start9Labs/lnd-wrapper/blob/master/Dockerfile>`_ features a well defined Dockerfile, for example.

113
site/source/developer-docs/specification/index.rst Normal file
View File

@@ -0,0 +1,113 @@
.. _service-packaging-spec:
=============
Specification
=============
The following guides provide an in depth overview of the full capabilities available for packaging a service.
.. raw:: html
<div class="topics-grid grid-container full">
<div class="grid-x grid-margin-x">
.. topic-box::
:title: Docker
:link: docker
:class: large-4
:anchor: View
Learn how to setup the main Docker image for your service.
.. topic-box::
:title: Manifest
:link: manifest
:class: large-4
:anchor: Begin
Understand the function of a Manifest file and its type.
.. topic-box::
:title: Config Spec
:link: config-spec
:class: large-4
:anchor: Begin
Learn the purpose and utility of a config specification.
.. topic-box::
:title: Properties
:link: properties
:class: large-4
:anchor: View
Understand the purpose and requirements of service properties.
.. topic-box::
:title: Dependencies
:link: dependencies
:class: large-4
:anchor: View
Learn how to configure dependency options.
.. topic-box::
:title: Backups
:link: advanced
:class: large-4
:anchor: View
Learn how to configure backup options.
.. topic-box::
:title: Instructions
:link: instructions
:class: large-4
:anchor: View
Understand how an instructions file is relevant to a service.
.. topic-box::
:title: Package
:link: package
:class: large-4
:anchor: View
Learn how to package service components into a single file format.
.. topic-box::
:title: Wrapper
:link: wrapper
:class: large-4
:anchor: View
Understand the purpose of a wrapper repository.
.. topic-box::
:title: Submission
:link: submission
:class: large-4
:anchor: View
Learn about how to submit a package for review to a marketplace.
.. raw:: html
</div></div>
.. toctree::
:hidden:
docker
manifest
config-spec
properties
dependencies
backups
instructions
package
wrapper
makefile
submission
compat/index

61
site/source/developer-docs/specification/instructions.rst Normal file
View File

@@ -0,0 +1,61 @@
.. _service_instructions:
============
Instructions
============
Within each wrapper repository, a file should exist that includes instructions for a service. This file should include any pertinent documentation, instructions, external integrations, or other details about the service that users or developers might find relevant.
This markdown formatted file will be rendered in the "Instructions" section within the service page on EmbassyOS:
.. figure:: /_static/images/services/service-instruct.png
:width: 80%
:alt: Embassy Pages Instructions
Below is the markdown file for Embassy Pages' instructions, as shown above:
.. code-block::
# Embassy Pages
## About
Embassy Pages is a simple web server that uses directories inside File Browser to serve Tor websites. Your website could be a blog, a resume, a portfolio, a business landing page, a product brochure, or just a set of static folders and/or files that you want to share with the world.
When you first install Embassy Pages, there will be a default Homepage hosted at the root, <tor-address>.onion. You can change the behavior of this page, and you can also create Subdomain websites. For example, one site could be hello.<tor-address>.onion and another could be goodbye.<tor-address>.onion. What is served from the Homepage and each Subdomain is totally up to you.
Self-hosting Tor websites using Embassy Pages is easy, permissionless, and censorship-resistant; there are no trusted third parties involved.
Anyone can do it. No one can stop it.
## Instructions
1. Inside `Config`, you can easily change the behavior of your Homepage to serve:
a. A list of hyperlinks to all your Subdomains
b. A personal web page
c. An automatic redirect to a Subdomain
d. A static web page that tells anyone visiting it to fuck off, politely.
2. Inside `Config`, you can create one or more Subdomains, giving each a unique name.
3. To serve a personal website, simply upload the website directory to File Browser. Then, inside the settings for a particular page (either your Homepage or a Subdomain), enter the path to that directory. For example, a path of websites/blog would tell Embassy Pages that it can find the blog website inside the websites directory in File Browser.
4. A list of all your Subdomains can be found inside the Properties section of your Embassy Pages service.
## Restore from Backup
1. In `Config`, select "Restore from Backup".
2. After the restoration completes, navigate to the Marketplace.
3. Downgrade Embassy Pages.
4. Upgrade Embassy Pages to the latest version.
5. Start Embassy Pages.
6. Embassy Pages is now successfully restored!
Example
-------
The `bitcoind wrapper <https://github.com/Start9Labs/bitcoind-wrapper/tree/master/docs>`_ demonstrates a good use of instructions and external integrations.
.. role:: raw-html(raw)
:format: html
:raw-html:`<br />`

28
site/source/developer-docs/specification/makefile.rst Normal file
View File

@@ -0,0 +1,28 @@
.. _service_makefile:
========
Makefile
========
.. note:: *This file is optional*
A Makefile serves as a convenience for outlining the build. This helps streamline additional developer contributions to the project. Please reference the GNU `documentation <https://www.gnu.org/software/make/manual/html_node/Introduction.html>`_ for implementation details.
An alternative to using ``make`` is to use the `nix <https://nixos.wiki/wiki/Nix>`_ specification.
This purpose of this file is to:
- Read the docker container and build the project
- Build all prerequisites for running the docker file
- Build all dependencies
- Package assets and docker image using ``embassy-sdk pack``
Example
-------
The `LND wrapper <https://github.com/Start9Labs/lnd-wrapper/blob/master/Makefile>`_ features a well defined Makefile, for example.
.. role:: raw-html(raw)
:format: html
:raw-html:`<br />`

299
site/source/developer-docs/specification/manifest.rst Normal file
View File

@@ -0,0 +1,299 @@
.. _service_manifest:
========
Manifest
========
Overview
--------
This file describes the service and it's requirements. It is used to:
- create a listing in the marketplace
- denote any installation considerations
- define dependency requirements
- define actions such as health checks, backups, and configuration
- define alerts and other messaging / descriptions for the user interface
Each time a service is updated, the Manifest should be updated to include the new version, release notes, and any pertinent updates to the install, uninstall, or restoration flows. All this information is displayed in the marketplace listing, and the optionally denoted alerts will be displayed when appropriate to provide the user with more information.
There is nothing you need to do as a developer to set up Tor for running a service. This is *completely* handled by EmbassyOS - a Tor address will be automatically generated when the service is installed. Just define an interface with a tor config in the Manifest file. You do, however, need to ensure the service is in fact capable of running over Tor.
The Manifest is also responsible for outlining service :ref:`dependencies <dependencies-spec>`. By defining rules using the :ref:`EmbassyOS DSL specification <config_rules>`, users can easily and selectively install, uninstall, and update any service without getting stuck in dependency hell. EmbassyOS presents this information in a polished install/uninstall/update wizard, so there's no need for editing configuration files or jumping into the command line. For you as a developer, this simply means populating this key in the manifest!
Formatting
----------
- Serialization language:``yaml | toml | json``
- Case style: ``kebab-case``
Type definitions
----------------
Below are the types and sub-type definitions, with necessary elaborations. Any item that contains ``Option<>`` is an optional field.
.. code:: yaml
# The package identifier used by the OS
id: String
# A human readable service title
title: String
# Service version - accepts up to four digits, where the last confirms to revisions necessary for EmbassyOS - see documentation: https://github.com/Start9Labs/emver-rs. This value will change with each release of the service.
version: Number
# Release notes for the update - can be a string, paragraph or URL
release-notes: String
# The type of license for the project. Include the LICENSE in the root of the project directory. A license is required for a Start9 package.
license: String
# The Start9 wrapper repository URL for the package. This repo contains the manifest file (this), any scripts necessary for configuration, backups, actions, or health checks (more below). This key must exist. But could be embedded into the source repository.
wrapper-repo: String
# The original project repository URL. There is no upstream repo in this example
upstream-repo: String
# URL to the support site / channel for the project. This key can be omitted if none exists, or it can link to the original project repository issues.
support-site: String
# URL to the marketing site for the project. This key can be omitted if none exists, or it can link to the original project repository.
marketing-site: String
# The series of commands to build the project into an s9pk for arm64/v8. In this case we are using a Makefile with the simple build command "make".
build: List<String>
# Minimum required version of EmbassyOS
min-os-version: Number
# Human readable descriptors for the service. These are used throughout the EmbassyOS user interface, primarily in the marketplace.
description:
# This is the first description visible to the user in the marketplace.
short: String
# This description will display with additional details in the service's individual marketplace page
long: String
# These assets are static files necessary for packaging the service for Start9 (into an s9pk). Each value is a path to the specified asset. If an asset is missing from this list, or otherwise denoted, it will be defaulted to the values denoted below.
assets:
# Default = LICENSE.md
license: String
# Default = icon.png
icon: String
# Default = INSTRUCTIONS.md
instructions: String
# Default = image.tar
docker-images: String
# The main action for initializing the service. Currently, the only type of action available is docker.
main:
# Docker is currently the only action implementation
type: String
# Identifier for the main image volume, which will be used when other actions need to mount to this volume.
image: String
# The executable binary for starting the initialization action. For docker actions, this is typically a "docker_entrypoint.sh" file. See the Dockerfile and the docker_entrypoint.sh in this project for additional details.
entrypoint: String
# Any arguments that should be passed into the entrypoint executable
args: List<String>
# Specifies where to mount the data volume(s), if there are any. Mounts for pointer dependency volumes are also denoted here. These are necessary if data needs to be read from / written to these volumes.
mounts:
# Specifies where on the service's file system its persistence directory should be mounted prior to service startup
main: String
# Health checks
health-checks:
main:
name: String
description: String
type: String
image: String
entrypoint: String
args: List<String>
# When `inject` is true, the health check will use the main image to run the health check. This is faster as there is no need to spin up an additional docker container
# When `inject` is false, the health check will use whatever image is specified. This is useful when using a system image with additional utilities to run a health check. If inject=false, then system must equal true
inject: Boolean
# Optional if false - indicates if an image that is preloaded onto the system will be used
system: Boolean
# Required - valid values are yaml, toml, json
io-format: Enum<json|yaml|toml>
# Specifies how to get and set configuration file values for the service. NOTE: This stanza can be left empty (null) if the service has no configuration options.
config:
# The config action to run to get the specified config file (default is config.yaml)
get:
# The type of implementation to run this action (currently, only Docker is available)
type: String
# The Docker image to run the action command in. This could be the service's main image, or an image that is preloaded onto the system, like compat (which holds compatible helper functions for default functionality)
image: String
# Indicates if an image that is preloaded onto the system will be used
system: Boolean
# The initial run command to execute the config get action
entrypoint: String
# Any arguments that need to be passed into the run command
args: List<String>
# The locations at which to mount the specified Docker images
mounts:
compat: String
main: String
# Required - valid values are yaml, toml, json
io-format: Enum<yaml|json|toml>
# The config action to run to set the specified config file (default is config.yaml). Details for the keys below are the same as above.
set:
type: String
image: String
system: Boolean
entrypoint: String
args: List<String>
mounts:
compat: String
main: String
io-format: Enum<yaml|json|toml>
# This is a key value map specifying dependent services that this service needs in order to function. The keys are the package id's on which you depend. NOTE: if developing a standalone service, you may leave this stanza as an empty object (the key dependencies is required)
dependencies:
# Key must be the package id of another service that exists in the marketplace
filebrowser:
# The version range that is acceptable for this dependency
version: Emver
# Describes if the dependency is critical to the service functioning. If the dependency is critical, the service will stop if this dependency is stopped.
critical: Boolean
# Specifies the requirement type of the dependency
requirement:
# "Opt-out" means the dependency will be required according to the default config. "Opt-in" means the dependency may be required if you change the config. And "required" just means it's always required.
type: Enum<opt-in|opt-out|required>
# An explanation of how to opt-in or opt-out. This value is optional for type=required
how: String
# Description of the dependency relationship
description: String
# This is a list of rules that levies requirements on the configuration of the dependency and suggests ways to remedy any incompatibilities. Documentation of this feature is outside the scope of this example.
config: ~
# This denotes any data, asset, or pointer volumes that should be connected when the "docker run" command is invoked
volumes:
# This is the image where files from the project asset directory will go
main:
type: Enum<data|asset>
# This is an example of an asset volume
compat:
type: Enum<data|asset>
# This specifies how to configure the port mapping for exposing the service over TOR and LAN (if applicable). Many interfaces can be specified depending on the needs of the service. If it can be launched over a Local Area Network connection, specify a `lan-config`. Otherwise, at minimum, a `tor-config` must be specified.
interfaces:
# This key is the internal name that the OS will use to configure the interface
main:
# A human readable name for display in the UI
name: String
# A descriptive description of what the interface does
description: String
tor-config:
# Port mappings are from the external port to the internal container port
port-mapping:
80: String
# Port mappings are from the external port to the internal container port
lan-config:
80:
ssl: Boolean
internal: Number
# Denotes if the service has a user interface to display
ui: Boolean
# Denotes the protocol specifications used by this interface
protocols: List<String>
# Alerts: omitting these will result in using the default alerts in EmbassyOS, except for start, which has no default.
alerts:
install-alert: String
uninstall-alert: String
restore-alert: String
start-alert: String
# Specifies how backups should be run for this service. The default EmbassyOS provided option is to use the duplicity backup library on a system image (compat)
backup:
create:
type: String
image: String
system: Boolean
entrypoint: String
# Arguments to pass into the entrypoint.
args: List<String>
mounts:
# BACKUP is the default volume that is used for backups. This is whatever backup drive is mounted to the device, or a network filesystem.
# The value here donates where the mount point will be. The backup drive is mounted to this location.
BACKUP: String
main: String
restore:
type: String
image: String
system: Boolean
entrypoint: String
args: List<String>
mounts:
BACKUP: String
main: String
# Commands that can be issued from the UI. NOTE: if no actions are required, this section can be left as an empty object
actions:
hello-world-action:
name: String
description: String
warning: Option<String>
# Indicates what state the service can be in while executing the action
allowed-statuses: List<String>
# Defines how the action is run
implementation:
type: String
image: String
entrypoint: String
args: [List<String>
# Same as note on health-checks
inject: Boolean
# Required - valid values are yaml, toml, json
io-format: Enum<yaml|json|toml>
.. code:: typescript
interface Dependencies [{
serviceId: DepInfo
}]
interface DepInfo {
version: VersionRange // ie. ^0.11.1.1
requirement: Enum<"opt-in"|"opt-out"|"required">,
description?: String,
config: Optional<{
check: ActionImplementation,
auto-configure: ActionImplementation,
}>,
}
interface ActionImplementation {
image: String,
system: Bool,
entrypoint: String,
args: List<String>,
mounts: Map<VolumeId, Path>,
io_format: Option<Enum<json|yaml|toml>>,
inject: Bool,
}
interface ConfigRule {
rule: String, // ie. 'users.*.name = "lnd"
description: String,
suggestions: [SuggestionVariant]
}
interface SuggestionVariant {
SET: {
var: String,
to: SetVariant,
},
DELETE: {
src: String,
},
PUSH: {
to: String,
value: Value,
},
}
interface SetVariant {
to: Option<String>,
to-value: Option<Value>, // ie. true/false
to-entropy: Option<{
charset: String // ie. 'a-z,A-Z,0-9'
len: number
}>
}
.. role:: raw-html(raw)
:format: html
:raw-html:`<br />`
Examples
========
- `Hello World <https://github.com/Start9Labs/hello-world-wrapper/blob/master/manifest.yaml>`__
- `Filebrowser <https://github.com/Start9Labs/filebrowser-wrapper/blob/master/manifest.yaml>`__
- `Embassy-pages <https://github.com/Start9Labs/embassy-pages-wrapper/blob/master/manifest.toml>`__
- `Photoview <https://github.com/Start9Labs/embassyos-photoview-wrapper/blob/master/manifest.yaml>`__
- `BTCPay Server <https://github.com/Start9Labs/btcpayserver-wrapper/blob/master/manifest.yaml>`__
- `Synapse <https://github.com/Start9Labs/synapse-wrapper/blob/master/manifest.yaml>`__

33
site/source/developer-docs/specification/package.rst Normal file
View File

@@ -0,0 +1,33 @@
.. _package:
==============
Package Bundle
==============
All assets get bundled into a custom filetype with an extension of ``.s9pk``, short for Start9 Package. This is the file that will be downloaded from a marketplace. When the user clicks a service's "Install" button, EmbassyOS unpacks and installs the service.
The minimum necessary files for this bundle are:
- image.tar (the result of a Dockerfile build)
- instructions.md
- LICENSE
- icon.png
Each new version release of a service should include the updated version of these files re-bundled into a new ``.s9pk``. .
Building
--------
To package all components into an ``.s9pk``, run the following command from the root of your project directory:
.. code:: bash
embassy-sdk pack
Let's also make sure to verify the validity of the package:
.. code:: bash
embassy-sdk verify s9pk /path/to/hello-world.s9pk
If anything goes wrong, an error message will indicate the missing component or other failure.

46
site/source/developer-docs/specification/properties.rst Normal file
View File

@@ -0,0 +1,46 @@
.. _service_properties:
==========
Properties
==========
The output of this step is a file titled ``stats.yaml``. This file contains a mapping of the values that will be displayed in the ``Properties`` section in a service's menu.
Due to the fact that config variables are only available when the service is running, this file can only be populated at runtime.
.. figure:: /_static/images/services/service-properties.png
:width: 80%
:alt: Service Properties
Service Properties
.. code:: typescript
:caption: Properties Type
interface PropertiesString {
type: 'string'
name: string
value: string
description: string | null
copyable: boolean
qr: boolean
masked: boolean
}
interface PropertiesObject {
type: 'object'
name: string
value: PropertiesObject | PropertiesString
}
Example
-------
A good example of the configurator producing the ``stats.yaml`` file can be found `here <https://github.com/Start9Labs/lnd-wrapper/blob/master/configurator/src/main.rs>`_.
.. role:: raw-html(raw)
:format: html
:raw-html:`<br />`

7
site/source/developer-docs/specification/submission.rst Normal file
View File

@@ -0,0 +1,7 @@
.. _submission:
==================
Submission Process
==================
When you have built and tested your project for EmbassyOS, please send Start9 a submission with the project repository to ``dev@start9labs.com``. After being reviewed for security and compatibility, the service will be deployed to the marketplace and available for all EmbassyOS users to download.

50
site/source/developer-docs/specification/wrapper.rst Normal file
View File

@@ -0,0 +1,50 @@
.. _service_wrapper:
=======
Wrapper
=======
Each service is bound with a wrapper repository, which contains everything you need to build a service.
The purpose of this repo is:
- To define the necessary Manifest and configuration options (ie. config spec and rules)
- To build the project into the ``.s9pk`` format digestible to EmbassyOS
- Link to the source project as a git submodule
- Define the Dockerfile for running the project on EmbassyOS
- Provide documentation for the project, especially user runbook instructions
File Structure
--------------
The project structure should be used as a model:
.. code-block:: bash
├── Dockerfile
├── LICENSE
├── Makefile
├── README.md
├── assets
│ └── compat
│ ├── config_rules.yaml
│ └── config_spec.yaml
├── docker_entrypoint.sh
├── <submodule-project>
├── <package-id>>.s9pk
├── icon.png
├── image.tar
├── instructions.md
└── manifest.yaml
Submodule
---------
`Git submodules <https://www.git-scm.com/book/en/v2/Git-Tools-Submodules>`__ allow use of another project while in the working project directory. Setting up this feature enables linking of the source service repository so that its context is available. The submodule is added into the wrapper so that the wrapper can build the submodule and also track the exact code that's being built.
Run:
.. code:: bash
git submodule add <link_to_source_project>