Start9/documentation
2
0
Fork 0
mirror of https://github.com/Start9Labs/documentation.git synced 2026-03-30 20:14:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

SPG 1st draft complete

Browse Source
This commit is contained in:
kn0wmad
2021-12-07 20:01:46 -07:00
committed by Lucy Cifferello
parent 4d6ae8b0e2
commit ff5871dd9a
3 changed files with 389 additions and 365 deletions
Download Patch File Download Diff File Expand all files Collapse all files

194
source/dev-docs/packaging-example.rst
View File

@@ -1,44 +1,47 @@
.. _packaging-example: .. _packaging-example:
***************** *********************************
Packaging Example EmbassyOS Service Packaging Guide
***************** *********************************
A rough walkthrough of how to package a service using our example "Hello World" wrapper. A rough walkthrough of how to package a service using our example "Hello World" wrapper.
# EmbassyOS Service Packaging Guide
Welcome! The following guide will provide the prerequisites, introduce a brief overview of the packaging process, use an example demonstrating how to package a service, and finally describe the submission process. This essentially describes how you can take an existing app (or one you have written yourself), and wrap it up such that it can be added to an EmbassyOS Marketplace! Welcome! The following guide will provide the prerequisites, introduce a brief overview of the packaging process, use an example demonstrating how to package a service, and finally describe the submission process. This essentially describes how you can take an existing app (or one you have written yourself), and wrap it up such that it can be added to an EmbassyOS Marketplace!
## Prerequisites Pre-requisites
==============
### EmbassyOS (EOS) EmbassyOS (EOS)
---------------
It is **HIGHLY RECOMMENDED** to have a copy of EmbassyOS for testing your packaged service. It is **HIGHLY RECOMMENDED** to have a copy of EmbassyOS for testing your packaged service.
There are 3 options for this: There are 3 options for this:
1. build from [source](https://github.com/Start9Labs/embassy-os/blob/master/BuildGuide.md) #. Build from `source <https://github.com/Start9Labs/embassy-os/build>`_
2. follow the [DIY guide](https://docs.start9.com/getting-started/diy.html#diy) to build on a Raspberry Pi #. Follow the :ref:`DIY guide <diy>` to build on a Raspberry Pi
3. [purchase](https://docs.start9.com/getting-started/purchasing.html#purchasing) a device or copy of the OS #. :ref:`Purchse <purchasing>` a device or copy of the OS
### Development Environment Development Environment
-----------------------
Once you have EOS installed, you'll want to set up your development system with the necessary software. Once you have EOS installed, you'll want to set up your development system with the necessary software.
At minimum you will need the following: At minimum you will need the following:
1. [docker](https://docs.docker.com/get-docker) #. `Docker <https://docs.docker.com/get-docker>`_
2. [docker-buildx](https://docs.docker.com/buildx/working-with-buildx/) #. `Docker-buildx <https://docs.docker.com/buildx/working-with-buildx/>`_
3. ***PLACEHOLDER FOR EOS-SDK*** #. ***PLACEHOLDER FOR EOS-SDK***
The following are recommended: The following are recommended:
4. [cargo](https://doc.rust-lang.org/cargo/) #. `Cargo <https://doc.rust-lang.org/cargo/>`_
5. [yq](https://mikefarah.gitbook.io/yq/) (version 4) #. `yq <https://mikefarah.gitbook.io/yq/>`_ (version 4)
6. [make](https://www.gnu.org/software/make/) #. `make <https://www.gnu.org/software/make/>`_
7. [rust-musl-cross](***PLACEHOLDER FOR NEW MUSL-CROSS REPO***) (For cross compiling Rust to Alpine, not needed otherwise) #. rust-musl-cross (***PLACEHOLDER FOR NEW MUSL-CROSS REPO***) (For cross compiling Rust to Alpine, not needed otherwise)
## Overview Overview
========
### Components Components
----------
Simply, the package is made up of the following parts: Simply, the package is made up of the following parts:
1. Image - Each service is running in a Docker image. Best results will come from an arm based linux; [Alpine](https://www.alpinelinux.org/) is highly recommended. 1. Image - Each service is running in a Docker image. Best results will come from an arm based linux; [Alpine](https://www.alpinelinux.org/) is highly recommended.
@@ -48,35 +51,44 @@ Simply, the package is made up of the following parts:
5. Config - EOS makes a service's configuration available to the user in the GUI and must be valid regardless of user skill. 5. Config - EOS makes a service's configuration available to the user in the GUI and must be valid regardless of user skill.
6. .s9pk Bundle - The image, config, manifest, and icon files get bundled into a .s9pk package. This is the file a user downloads from the Marketplace, at which point EOS un-packages and installs the service. 6. .s9pk Bundle - The image, config, manifest, and icon files get bundled into a .s9pk package. This is the file a user downloads from the Marketplace, at which point EOS un-packages and installs the service.
Check [here](https://docs.start9.com/contributing/services/overview.html) for a detailed overview of package components. Check :ref:`here <service_package_overview>` for a detailed overview of package components.
### Service Wrapper Repo and Submodules Service Wrapper Repo and Submodules
-----------------------------------
See [here](https://docs.start9.com/contributing/services/wrapper.html) for how to structure your service wrapper's git repository. See :ref:`here <service_wrapper>` for how to structure your service wrapper's git repository.
Git submodules allow the use of another project while in the working project directory. In this case, you can use an existing app's git repo in order to source its code into your service wrapper. Git submodules allow the use of another project while in the working project directory. In this case, you can use an existing app's git repo in order to source its code into your service wrapper.
Simply run: Simply run:
```git submodule add <link_to_source_project>```
## Example - Hello World .. code:: bash
Okay, let's actually package a service! For this example, we're going to use an example service [Hello World](https://github.com/Start9Labs/hello-world). This repository can also be used as a template to quickly get started with your service. This will give a good overview of service packaging, but obviously your app will be different. This will assume a Linux development environment with all the recommended dependencies listed above. To get started quickly, we'll use Start9's wrapper template. git submodule add <link_to_source_project>
### Clone the Template Repo and Edit the Manifest Example - Hello World
=====================
Okay, let's actually package a service! For this example, we're going to use an example service `Hello World <https://github.com/Start9Labs/hello-world>`_. This repository can also be used as a template to quickly get started with your service. This will give a good overview of service packaging, but obviously your app will be different. This will assume a Linux development environment with all the recommended dependencies listed above. To get started quickly, we'll use Start9's wrapper template.
Clone the Template Repo and Edit the Manifest
---------------------------------------------
1. Clone and rename the repo (or alternatively, use the template generation button found on the github `repo <https://github.com/Start9Labs/hello-world-wrapper>`_)
.. code-block:: bash
1. Clone and rename the repo (or alternatively, use the template generation button found on the github [repo](https://github.com/Start9Labs/hello-world-wrapper))
```
git clone https://github.com/Start9Labs/hello-world-wrapper git clone https://github.com/Start9Labs/hello-world-wrapper
cd hello-world-wrapper cd hello-world-wrapper
```
2. Edit the `README.md` to explain what the service is, what dependencies are required, build/install/contribute instructions, and any other information you'd like. 2. Edit the ``README.md`` to explain what the service is, what dependencies are required, build/install/contribute instructions, and any other information you'd like.
3. Edit the `manifest` file. This must be in `.json`, `.toml`, or `.yaml` format and in `kebab-case` style. You can see descriptions of each key (and some notes) in our 'Hello World' example `manifest.yaml` below: 3. Edit the ``manifest`` file. This must be in ``.json``, ``.toml``, or ``.yaml`` format and in ``kebab-case`` style. You can see descriptions of each key (and some notes) in our 'Hello World' example ``manifest.yaml`` below:
``` Manifest example
# v0.3.0 and up Manifest example written in .yaml (.toml and .json are also acceptable) ----------------
.. code-block:: yaml
id: hello-world id: hello-world
title: "Hello World" title: "Hello World"
@@ -229,50 +241,53 @@ actions: {} # Commands that can be issued from the UI. None for hello-world, bu
# args: ["reset-root-user"] # args: ["reset-root-user"]
# mounts: # mounts:
# main: "/root" # main: "/root"
```
Note the `dependencies` and `volumes` sections, which may access another service, for example, File Browser, such that files stored on a user's Embassy can be accessed in your service. Note the ``dependencies`` and ``volumes`` sections, which may access another service, e.g. File Browser, such that files stored on a user's Embassy can be accessed in your service.
For details on all the different possible dependency, type, and subtype definitions available for the `manifest` file, please see [here](https://docs.start9.com/contributing/services/manifest.html). For details on all the different possible dependency, type, and subtype definitions available for the ``manifest`` file, please see :ref:`here <service_manifest>`.
### Edit the Dockerfile and Entrypoint Edit the Dockerfile and Entrypoint
----------------------------------
Next, it's time to edit the `Dockerfile`. This defines how to build the image for the package by declaring the environment, building stages, and mounting the package to the volume specified in the `Manifest`. Next, it's time to edit the ``Dockerfile``. This defines how to build the image for the package by declaring the environment, building stages, and mounting the package to the volume specified in the ``manifest``.
1. We start by importing a base image, in this case Alpine, as recommended. 1. We start by importing a base image, in this case Alpine, as recommended.
`FROM arm64v8/alpine:3.12` .. code:: docker
FROM arm64v8/alpine:3.12
2. Next we issue some commands, which in this example simply updates repositories, installs required software, and finally creates a directory for nginx. 2. Next we issue some commands, which in this example simply updates repositories, installs required software, and finally creates a directory for nginx.
``` .. code:: docker
RUN apk update RUN apk update
RUN apk add tini RUN apk add tini
RUN mkdir /run/nginx RUN mkdir /run/nginx
```
3. Next we will add the cross-compiled binary of `hello-world` to `/usr/local/bin/` and add the `docker_entrypoint.sh` file from the repository. Then we set permissions for `docker_entrypoint.sh`. 3. Next we will add the cross-compiled binary of ``hello-world`` to ``/usr/local/bin/`` and add the ``docker_entrypoint.sh`` file from the repository. Then we set permissions for ``docker_entrypoint.sh``.
.. code:: docker
```
ADD ./hello-world/target/aarch64-unknown-linux-musl/release/hello-world /usr/local/bin/hello-world ADD ./hello-world/target/aarch64-unknown-linux-musl/release/hello-world /usr/local/bin/hello-world
ADD ./docker_entrypoint.sh /usr/local/bin/docker_entrypoint.sh ADD ./docker_entrypoint.sh /usr/local/bin/docker_entrypoint.sh
RUN chmod a+x /usr/local/bin/docker_entrypoint.sh RUN chmod a+x /usr/local/bin/docker_entrypoint.sh
```
4. Next we set a working directory, expose a port, and set the location of the entrypoint. 4. Next we set a working directory, expose a port, and set the location of the entrypoint.
``` .. code:: docker
WORKDIR /root WORKDIR /root
EXPOSE 80 EXPOSE 80
ENTRYPOINT ["/usr/local/bin/docker_entrypoint.sh"] ENTRYPOINT ["/usr/local/bin/docker_entrypoint.sh"]
```
5. Great, let's take a look at our final Embassy Pages `Dockerfile`: 5. Great, let's take a look at our final Embassy Pages ``Dockerfile``:
.. code:: docker
```
FROM arm64v8/alpine:3.12 FROM arm64v8/alpine:3.12
RUN apk update RUN apk update
@@ -287,29 +302,30 @@ WORKDIR /root
EXPOSE 80 EXPOSE 80
ENTRYPOINT ["/usr/local/bin/docker_entrypoint.sh"] ENTRYPOINT ["/usr/local/bin/docker_entrypoint.sh"]
```
6. Okay, let's move on to our `docker_entrypoint.sh` file. This is a script that defines what to do when the service starts. It will need to complete any environment setup (such as folder substructure), sets any environment variables, and executes the run command. If you have built a `configurator`, it will also execute here. Let's take a look at our (extremely basic) Hello World example: 6. Okay, let's move on to our ``docker_entrypoint.sh`` file. This is a script that defines what to do when the service starts. It will need to complete any environment setup (such as folder substructure), sets any environment variables, and executes the run command. If you have built a ``configurator``, it will also execute here. Let's take a look at our (extremely basic) Hello World example:
.. code:: bash
```
#!/bin/sh #!/bin/sh
export HOST_IP=$(ip -4 route list match 0/0 | awk '{print $3}') export HOST_IP=$(ip -4 route list match 0/0 | awk '{print $3}')
exec tini hello-world exec tini hello-world
```
7. We've defined the file, exported the IP address, and run the program. 7. We've defined the file, exported the IP address, and run the program.
For a more detailed `docker_entrypoint.sh`, please check out the [filebrowser-wrapper](https://github.com/Start9Labs/filebrowser-wrapper/blob/master/docker_entrypoint.sh). Additional details on the `Dockerfile` and `entrypoint` can be found [here](https://docs.start9.com/contributing/services/docker.html). For a more detailed ``docker_entrypoint.sh``, please check out the `filebrowser-wrapper <https://github.com/Start9Labs/filebrowser-wrapper/blob/master/docker_entrypoint.sh>`_. Additional details on the ``Dockerfile`` and ``docker_entrypoint`` can be found `here <https://docs.start9.com/contributing/services/docker.html>`_.
### Makefile (Optional) Makefile (Optional)
-------------------
Here, we will create a `Makefile`, which is optional, but recommended as it outlines the build and streamlines additional developer contributions. Alternatively, you could use any other build orchestration tool, such as `nix`, `bash`, `python`, `perl`, `ruby`, etc instead of `make`. Here, we will create a ``Makefile``, which is optional, but recommended as it outlines the build and streamlines additional developer contributions. Alternatively, you could use any other build orchestration tool, such as ``nix``, ``bash``, ``python``, ``perl``, ``ruby``, etc instead of ``make``.
Our example `Makefile` is agin fairly simple for Hello World. Let's take a look: Our example ``Makefile`` is agin fairly simple for Hello World. Let's take a look:
.. code-block:: Makefile
```
ASSETS := $(shell yq e '.assets.[].src' manifest.yaml) ASSETS := $(shell yq e '.assets.[].src' manifest.yaml)
ASSET_PATHS := $(addprefix assets/,$(ASSETS)) ASSET_PATHS := $(addprefix assets/,$(ASSETS))
VERSION := $(shell toml get hello-world/Cargo.toml package.version) VERSION := $(shell toml get hello-world/Cargo.toml package.version)
@@ -337,31 +353,35 @@ hello-world/target/aarch64-unknown-linux-musl/release/hello-world: $(HELLO_WORLD
manifest.yaml: hello-world/Cargo.toml manifest.yaml: hello-world/Cargo.toml
yq e -i '.version = $(VERSION)' manifest.yaml yq e -i '.version = $(VERSION)' manifest.yaml
```
1. The first 5 lines set environment variables. 1. The first 5 lines set environment variables.
2. The next line simply removes the progress of a `make` process if it fails. 2. The next line simply removes the progress of a ``make`` process if it fails.
`.DELETE_ON_ERROR:`
3. The `all` step is run when the `make` command is issued. This attempts the `verify` step, which requires that the `hello-world.s9pk` must first be built, which first requires the `image.tar`, and so on. Meaning each step essentially requires the next . .. code-block:: Makefile
4. So the `.s9pk` is created with the `embassy-sdk pack` command, supplied with the `manifest`, `config_spec`, previously created `image.tar`, and `instructions.md`. Your project may likely also contain a `config_rules` file. Some of these files we have not yet edited, but that will come shortly. .DELETE_ON_ERROR:
5. The `image.tar` is built below this, the cross-compiled `hello-world` source code, and `manifest` at the bottom. 1. The ``all`` step is run when the ``make`` command is issued. This attempts the ``verify`` step, which requires that the ``hello-world.s9pk`` must first be built, which first requires the ``image.tar``, and so on. Meaning each step essentially requires the next .
For more details on creating a `Makefile` for your project, please check [here](https://docs.start9.com/contributing/services/makefile.html). 2. So the ``.s9pk`` is created with the ``embassy-sdk pack`` command, supplied with the ``manifest``, ``config_spec``, previously created ``image.tar``, and ``instructions.md``. Your project may likely also contain a ``config_rules`` file. Some of these files we have not yet edited, but that will come shortly.
### Service Config Specification and Rules 3. The ``image.tar`` is built below this, the cross-compiled ``hello-world`` source code, and ``manifest`` at the bottom.
For more details on creating a ``Makefile`` for your project, please check :ref:`here <service_makefile>`.
Service Config Specification and Rules
--------------------------------------
Most self-hosted packages require a configuration. With EmbassyOS, these config options are provided to the user in a friendly GUI, and invalid configs are not permitted. This allows the user to manage their software without a lot of technical skill, and minimal risk of borking their software. Two files are created in this process: Most self-hosted packages require a configuration. With EmbassyOS, these config options are provided to the user in a friendly GUI, and invalid configs are not permitted. This allows the user to manage their software without a lot of technical skill, and minimal risk of borking their software. Two files are created in this process:
`config_spec.yaml` for specifying all the config options your package depends on to run ``config_spec.yaml`` for specifying all the config options your package depends on to run
`config_rules.yaml` for defining the ruleset that defines dependencies between config variables ``config_rules.yaml`` for defining the ruleset that defines dependencies between config variables
These are stored in `assets/compat/` for 0.2.x compatibility, and in `/assets/` for anything built for v0.3.0 and up (almost certainly what you're doing). These files contain a detailed mapping of configuration options with acceptable values, defaults, and relational rule-sets. Hello World has no configuration, as you can see [here](https://github.com/Start9Labs/hello-world-wrapper/blob/0.3.0/assets/compat/config_spec.yaml). Instead, let's take a look at our `config_spec` for Embassy Pages, which actually has some config options: These are stored in ``assets/compat/`` for 0.2.x compatibility, and in ``/assets/`` for anything built for v0.3.0 and up (almost certainly what you're doing). These files contain a detailed mapping of configuration options with acceptable values, defaults, and relational rule-sets. Hello World has no configuration, as you can see `here <https://github.com/Start9Labs/hello-world-wrapper/blob/0.3.0/assets/compat/config_spec.yaml>`_. Instead, let's take a look at our ``config_spec`` for Embassy Pages, which actually has some config options:
.. code-block:: yaml
```
homepage: homepage:
name: Homepage name: Homepage
description: The page that will be displayed when your Embassy Pages .onion address is visited. Since this page is technically publicly accessible, you can choose to which type of page to display. description: The page that will be displayed when your Embassy Pages .onion address is visited. Since this page is technically publicly accessible, you can choose to which type of page to display.
@@ -442,44 +462,48 @@ subdomains:
pattern: '^[a-z-]+$' pattern: '^[a-z-]+$'
pattern-description: May contain only lowercase characters and hyphens. pattern-description: May contain only lowercase characters and hyphens.
nullable: false nullable: false
```
We essentially have 2 config options (homepage and subdomains), with all of their specifications nested below them. Looking at the homepage, it contains a `union` type, which is a necessary dependency, which can be of 5 variants (welcome, index, filebrowser, redirect, or fuck-off). The below images show how this is displayed in the UI. We essentially have 2 config options (homepage and subdomains), with all of their specifications nested below them. Looking at the homepage, it contains a ``union`` type, which is a necessary dependency, which can be of 5 variants (welcome, index, filebrowser, redirect, or fuck-off). The below images show how this is displayed in the UI.
***IMAGE PLACEHODLER*** ***IMAGE PLACEHODLER***
***IMAGE PLACEHODLER*** ***IMAGE PLACEHODLER***
For all the possible types, please check our detailed documentation [here](https://docs.start9.com/contributing/services/config.html#types). For all the possible types, please check our :ref:`Service Config Spec <service_config>`.
In our example, there is *no need* for a `config_rules` file. This is because there is not a rule-set required to define dependencies between config variables. An example of when this would be required would be the following code, from the [LND wrapper](https://github.com/Start9Labs/lnd-wrapper/blob/master/config_rules.yaml): In our example, there is *no need* for a ``config_rules`` file. This is because there is not a rule-set required to define dependencies between config variables. An example of when this would be required would be the following code, from the [LND wrapper](https://github.com/Start9Labs/lnd-wrapper/blob/master/config_rules.yaml):
.. code-block:: yaml
```
--- ---
- rule: '!(max-chan-size?) OR !(min-chan-size?) OR (#max-chan-size > #min-chan-size)' - rule: '!(max-chan-size?) OR !(min-chan-size?) OR (#max-chan-size > #min-chan-size)'
description: "Maximum Channel Size must exceed Minimum Channel Size" description: "Maximum Channel Size must exceed Minimum Channel Size"
```
Here we see that a Maximum Channel Size MUST be one of 3 possible options in order to be a valid config. Here we see that a Maximum Channel Size **MUST** be one of 3 possible options in order to be a valid config.
### Properties Properties
----------
Next we need to create the Properties section for our package, to display any relevant info. The result of this step is a `stats.yaml` file, which is only populated at runtime. These commands will be issued in the `docker_entrypoint` file (or `configurator`, if you are using one). Next we need to create the Properties section for our package, to display any relevant info. The result of this step is a ``stats.yaml`` file, which is only populated at runtime. These commands will be issued in the ``docker_entrypoint`` file (or ``configurator``, if you are using one).
***STATS.YAML IS APPARENTLY BEING DEPRECATED, THIS SECTION NEEDS COMMENT*** ***STATS.YAML IS APPARENTLY BEING DEPRECATED, THIS SECTION NEEDS COMMENT***
### Instructions Instructions
------------
Instructions are the basic directions or any particular details that you would like to convey to the user to help get them on their way. Each wrapper repo should contain a `docs` directory which can include anything you'd like, but specifically if you include an `instructions.md` file, formatted in Markdown language, it will be displayed simply for the user as shown below. Instructions are the basic directions or any particular details that you would like to convey to the user to help get them on their way. Each wrapper repo should contain a ``docs`` directory which can include anything you'd like, but specifically if you include an ``instructions.md`` file, formatted in Markdown language, it will be displayed simply for the user as shown below.
***PLACEHOLDER FOR IMAGE*** ***PLACEHOLDER FOR IMAGE***
You can find the `instructions.md` file for Embassy Pages [here](https://github.com/Start9Labs/embassy-pages-wrapper/tree/master/docs) if you are interested. You can find the ``instructions.md`` file for Embassy Pages `here <https://github.com/Start9Labs/embassy-pages-wrapper/tree/master/docs>`_ if you are interested.
### Backups Backups
-------
Everything in the root folder of the mounted system directory will be stored in an EOS backup. If you want to ignore any particular files for backup, you can create a `.backupignore` file and add the relative paths of any directories you would like ignored. Everything in the root folder of the mounted system directory will be stored in an EOS backup. If you want to ignore any particular files for backup, you can create a ``.backupignore`` file and add the relative paths of any directories you would like ignored.
## Submission Process Submission Process
------------------
When you have built and tested your project for EmbassyOS, please send Start9 a submission with the project repository to dev@start9labs.com. After being reviewed for security and compatibility, the service will be deployed to the marketplace and available for all EmbassyOS users to download. When you have built and tested your project for EmbassyOS, please send Start9 a submission with the project repository to dev@start9labs.com. After being reviewed for security and compatibility, the service will be deployed to the marketplace and available for all EmbassyOS users to download.

2
source/dev-docs/service-packaging/manifest.rst
View File

@@ -1,4 +1,4 @@
.. _manifest: .. _service_manifest:
**************** ****************
Service Manifest Service Manifest

6
source/user-manual/walkthrough/index.rst
View File

@@ -1,6 +1,6 @@
************ ***********
Waltkthrough Walkthrough
************ ***********
An overview of EmbassyOS general capabilities. An overview of EmbassyOS general capabilities.