diff --git a/site/source/about/roadmap.rst b/site/source/about/roadmap.rst index ede2531..4ea38d4 100644 --- a/site/source/about/roadmap.rst +++ b/site/source/about/roadmap.rst @@ -12,63 +12,6 @@ Please pardon our dust... .. .. raw:: html -.. -.. -.. -.. -.. -.. -.. Start9 - Roadmap -.. -.. -.. -.. -.. -.. -.. -.. -.. -.. -.. -.. -.. -.. - ..
..
..

Roadmap

@@ -307,57 +250,3 @@ Please pardon our dust... ..
.. ..
- -.. - -.. -.. -.. -.. -.. -.. diff --git a/site/source/support/dev-docs/dev-tools/embassy-cli.rst b/site/source/service-packaging/advanced/dev-tools/embassy-cli.rst similarity index 100% rename from site/source/support/dev-docs/dev-tools/embassy-cli.rst rename to site/source/service-packaging/advanced/dev-tools/embassy-cli.rst diff --git a/site/source/support/dev-docs/dev-tools/embassy-sdk.rst b/site/source/service-packaging/advanced/dev-tools/embassy-sdk.rst similarity index 100% rename from site/source/support/dev-docs/dev-tools/embassy-sdk.rst rename to site/source/service-packaging/advanced/dev-tools/embassy-sdk.rst diff --git a/site/source/support/dev-docs/dev-tools/embassyd.rst b/site/source/service-packaging/advanced/dev-tools/embassyd.rst similarity index 100% rename from site/source/support/dev-docs/dev-tools/embassyd.rst rename to site/source/service-packaging/advanced/dev-tools/embassyd.rst diff --git a/site/source/support/dev-docs/dev-tools/index.rst b/site/source/service-packaging/advanced/dev-tools/index.rst similarity index 100% rename from site/source/support/dev-docs/dev-tools/index.rst rename to site/source/service-packaging/advanced/dev-tools/index.rst diff --git a/site/source/support/dev-docs/dev-tools/service-container.rst b/site/source/service-packaging/advanced/dev-tools/service-container.rst similarity index 100% rename from site/source/support/dev-docs/dev-tools/service-container.rst rename to site/source/service-packaging/advanced/dev-tools/service-container.rst diff --git a/site/source/service-packaging/advanced/index.rst b/site/source/service-packaging/advanced/index.rst new file mode 100644 index 0000000..a7db8d9 --- /dev/null +++ b/site/source/service-packaging/advanced/index.rst @@ -0,0 +1,36 @@ +.. _advanced-packaging: + +=========================== +Advanced Packaging Overview +=========================== + +This section included advanced commands for working with packages in EmbassyOS. + +.. topic-box:: + :title: Developer Tools + :link: dev-tools/index + :icon: scylla-icon scylla-icon--developers-blog + :class: large-4 + :anchor: View + + EmbassyOS developer CLI tools + +.. topic-box:: + :title: Service Containers + :link: dev-tools/service-container + :icon: scylla-icon scylla-icon--overview + :class: large-4 + :anchor: View + + How to interact with containers on the Embassy + +.. raw:: html + + + +.. This is for the side navigation display +.. toctree:: + :maxdepth: 1 + :hidden: + + dev-tools/index \ No newline at end of file diff --git a/site/source/service-packaging/build-package-example/index.rst b/site/source/service-packaging/build-package-example/index.rst new file mode 100644 index 0000000..f2114ed --- /dev/null +++ b/site/source/service-packaging/build-package-example/index.rst @@ -0,0 +1,750 @@ +.. _build-package-example: + +======================== +Build Your First Package +======================== + +This guide will take you through the fundamentals of packaging a service for EmbassyOS by creating a real service. Let's get started! + +What we'll build +================ + +We'll create a web application that produces a hello world page. This web application will be the service we package for the Embassy. During this guide we will: + +1. Create a simple hello world web service +2. Create a Dockerfile +3. Create a Package Manifest +4. Create a hosted repository which will contain all the service components +5. Create the packaged service file + +You can find the complete service code referenced in this guide on `GitHub `_. + +Download required tools +======================= + +If necessary, download any of the system requirements: + +- Docker +- EmbassySDK +- A code editor +- Terminal + +For more details and installation steps for these tools, visit the :ref:`environment setup ` page. + + +Create Service +============== + +For this example, we are going to create a simple Rust project that serves a static web page. + +.. code:: bash + + mkdir hello-world + cargo init + touch src/index.html + +In `index.html` add: + +.. code:: html + + + Hello World! + + +

Hello World!

+ + + +In `main.rs` add: + +.. code:: rust + + use hyper::service::{make_service_fn, service_fn}; + use hyper::{body::Bytes, Body, Request, Response, Server}; + use std::convert::Infallible; + use std::net::SocketAddr; + + async fn handle(_req: Request) -> Result, Infallible> { + Ok(Response::new(Body::from(Bytes::from_static( + include_bytes!("index.html"), + )))) + } + + #[tokio::main] + async fn main() { + // Construct our SocketAddr to listen on... + let addr = SocketAddr::from(([0, 0, 0, 0], 80)); + + // And a MakeService to handle each connection... + let make_service = make_service_fn(|_conn| async { Ok::<_, Infallible>(service_fn(handle)) }); + + // Then bind and serve... + let server = Server::bind(&addr).serve(make_service); + + // And run forever... + if let Err(e) = server.await { + eprintln!("server error: {}", e); + } + } + + +That's it! We have the code for our service. + +Let's build and run it! + +.. code:: bash + + # build the project + cargo build + # start the executable + target/debug/hello-world + +Visit `localhost:80` to see your running web page! + +Build for RaspberryPi +====================== + +EmbassyOS is run on the arm-v8 architecture, specifically the aarch64 state, for the RaspberryPi. + +Depending on the programming language or libraries used in a project, you might need to set up an environment to *cross compile* the executable for this runtime environment. + +We were able to easily build and run our project locally. However, Rust is one of those programming languages that needs to be cross compiled. This step can be skipped if it does not apply to your project. + +Since Start9 uses Rust throughout its service ecosystem, the team has built a helper to cross compile Rust projects for the Embassy. + +This Rust cross compiler for aarch64 can be downloaded and build from `GitHub `_: + +.. code:: bash + + git clone https://github.com/Start9Labs/rust-musl-cross.git + cd rust-musl-cross + chmod a+x ./build.sh + ./build.sh + +This actually builds a Docker container for us to use locally so we can emulate the environment we need to compile our project for aarch64! + + +Next, we enable cross-arch emulated builds in Docker + +.. code:: bash + + docker run --privileged --rm linuxkit/binfmt:v0.8 + +Finally, we run the following command to build the project, replacing `/absolute/path/to` with the output of `pwd` as necessary: + +.. code:: bash + + docker run --rm -it -v ~/.cargo/registry:/root/.cargo/registry -v "/absolute/path/to"/hello-world:/home/rust/src start9/rust-musl-cross:aarch64-musl cargo build --release + +Awesome! We can now see the compiled executable here: `./hello-world/target/aarch64-unknown-linux-musl/release/hello-world` + +Build with Docker +================= + +Now that we have our code properly built/compiled, we can create a Dockerfile. This file defines how to build the Docker image for the service by declaring the environment, building stages, and copying any binaries or assets needed to run the service to the Docker image filesystem. + +In other words, the Dockerfile serves as a recipe for creating a Docker image, from which Docker containers are spun up. This is ultimately what runs an instance of your service on the Embassy. + +#. Create the necessary Docker files: + + .. code:: bash + + touch Dockerfile + touch docker_entrypoint.sh + +#. We start by importing a base image, in this case Alpine, as recommended. + + .. code:: docker + + FROM arm64v8/alpine:3.12 + +#. Next, we issue some commands to setup the filesystem. Here we update repositories and install required system packages. + + .. code:: docker + + RUN apk update + RUN apk add tini + +#. Next, we add the cross-compiled binary of ``hello-world`` to ``/usr/local/bin/`` and add the ``docker_entrypoint.sh`` file from the project root. Then, we set permissions for ``docker_entrypoint.sh``. + + .. code:: docker + + ADD ./hello-world/target/aarch64-unknown-linux-musl/release/hello-world /usr/local/bin/hello-world + ADD ./docker_entrypoint.sh /usr/local/bin/docker_entrypoint.sh + RUN chmod a+x /usr/local/bin/docker_entrypoint.sh + +#. Next, we set a working directory, and set the location of the entrypoint. Exposing ports is not necessary for EOS, but its often useful to leave this line for quick reference and clarity. + + .. code:: docker + + WORKDIR /root + + EXPOSE 80 + + ENTRYPOINT ["/usr/local/bin/docker_entrypoint.sh"] + +#. That's it! Let's take a look at our final ``Dockerfile``: +.. code:: docker + + FROM arm64v8/alpine:3.12 + + RUN apk update + RUN apk add tini + + ADD ./hello-world/target/aarch64-unknown-linux-musl/release/hello-world /usr/local/bin/hello-world + ADD ./docker_entrypoint.sh /usr/local/bin/docker_entrypoint.sh + RUN chmod a+x /usr/local/bin/docker_entrypoint.sh + + WORKDIR /root + + # not necessary for EmbassyOS, but often left for quick reference and clarity + EXPOSE 80 + + ENTRYPOINT ["/usr/local/bin/docker_entrypoint.sh"] + +#. Finally, add the following code to the `docker_entrypoint.sh`: + +.. code:: bash + + #!/bin/sh + + exec tini hello-world + +This is a script that defines how the service starts, and often acts as an init system. It will need to complete any environment setup (such as folder substructure), set any environment variables, and execute the run command. It's also PID 1 in the Docker container, so should do all of the signal handling for container exits. + +Manifest +======== + +The Manifest file specifies the details EmbassyOS needs to operate a service. It is the connection point between your service and EmbassyOS. + +In this file, values and actions exist for: + +- Displaying the service in the marketplace +- Specifying the project assets (eg. icon, instructions, license) +- Defining the docker mount points +- Specifying how to configure the service +- Relaying how to run health checks, backups, and other custom actions +- Outlining dependency relationships (if applicable) and configuration rules for dependencies +- Denoting copy to display in EmbassyUI elements, alerts, descriptions + +This file can be written in: + +- yaml +- toml +- json + +Let's create a yaml manifest file for our hello-world project: + +.. code:: bash + + touch manifest.yaml + +And populate it with the following example manifest: + +.. code:: yaml + + # The package identifier used by the OS + id: hello-world + # A human readable service title + title: "Hello World" + # Service version - accepts up to four digits, where the last confirms to revisions necessary for EmbassyOS - see documentation: https://github.com/Start9Labs/emver-rs + version: 0.3.0 + # Release notes for the update - can be a string, paragraph or URL + release-notes: "Upgrade to EmbassyOS v0.3.0" + # The type of license for the project. Include the LICENSE in the root of the project directory. A license is required for a Start9 package. + license: mit + # The Start9 wrapper repository URL for the package. This repo contains the manifest file (this), any scripts necessary for configuration, backups, actions, or health checks (more below). This key must exist. But could be embedded into the source repository. + wrapper-repo: "https://github.com/Start9Labs/hello-world-wrapper" + # The original project repository URL. There is no upstream repo in this example + upstream-repo: "https://github.com/Start9Labs/hello-world-wrapper" + # URL to the support site / channel for the project. This key can be omitted if none exists, or it can link to the original project repository issues. + support-site: "https://docs.start9.com/" + # URL to the marketing site for the project. This key can be omitted if none exists, or it can link to the original project repository. + marketing-site: "https://start9.com/" + # The series of commands to build the project into an s9pk for arm64/v8. In this case we are using a Makefile with the simple build command "make". + build: ["make"] + # Minimum required version of EmbassyOS + min-os-version: "0.3.0" + # Human readable descriptors for the service. These are used throughout the EmbassyOS user interface, primarily in the marketplace. + description: + # This is the first description visible to the user in the marketplace. + short: Example service + # This description will display with additional details in the service's individual marketplace page + long: | + Hello World is a simple example of a service wrapper that launches a web interface to say hello and nothing more. + # These assets are static files necessary for packaging the service for Start9 (into an s9pk). Each value is a path to the specified asset. If an asset is missing from this list, or otherwise denoted, it will be defaulted to the values denoted below. + assets: + # Default = LICENSE.md + license: LICENSE + # Default = icon.png + icon: icon.png + # Default = INSTRUCTIONS.md + instructions: instructions.md + # Default = image.tar + docker-images: image.tar + # The main action for initializing the service. Currently, the only type of action available is docker. + main: + # Docker is currently the only action implementation + type: docker + # Identifier for the main image volume, which will be used when other actions need to mount to this volume. + image: main + # The executable binary for starting the initialization action. For docker actions, this is typically a "docker_entrypoint.sh" file. See the Dockerfile and the docker_entrypoint.sh in this project for additional details. + entrypoint: "docker_entrypoint.sh" + # Any arguments that should be passed into the entrypoint executable + args: [] + # Specifies where to mount the data volume(s), if there are any. Mounts for pointer dependency volumes are also denoted here. These are necessary if data needs to be read from / written to these volumes. + mounts: + # Specifies where on the service's file system its persistence directory should be mounted prior to service startup + main: /root + # Health checks + health-checks: + main: + name: Web Interface + description: Ensures the network interface is accessible via HTTP. + type: docker + image: main + entrypoint: "sh" + args: ["-c", "curl --silent --show-error --fail http://loacalhost:80"] + # When `inject` is true, the health check will use the main image to run the health check. This is faster as there is no need to spin up an additional docker container + # When `inject` is false, the health check will use whatever image is specified. This is useful when using a system image with additional utilities to run a health check. If inject=false, then system must equal true + inject: true + # Optional if false - indicates if an image that is preloaded onto the system will be used + system: false + # Required - valid values are yaml, toml, json + io-format: json + # Specifies how to get and set configuration file values for the service. NOTE: This stanza can be left empty (null) if the service has no configuration options. + config: + # The config action to run to get the specified config file (default is config.yaml) + get: + # The type of implementation to run this action (currently, only Docker is available) + type: docker + # The Docker image to run the action command in. This could be the service's main image, or an image that is preloaded onto the system, like compat (which holds compatible helper functions for default functionality) + image: compat + # Indicates if an image that is preloaded onto the system will be used + system: true + # The initial run command to execute the config get action + entrypoint: compat + # Any arguments that need to be passed into the run command + args: + - config + - get + - /root + - "/mnt/assets/config_spec.yaml" + # The locations at which to mount the specified Docker images + mounts: + compat: /mnt/assets + main: /root + # Required - valid values are yaml, toml, json + io-format: yaml + # The config action to run to set the specified config file (default is config.yaml). Details for the keys below are the same as above. + set: + type: docker + image: compat + system: true + entrypoint: compat + args: + - config + - set + - hello-world + - /root + - "/mnt/assets/config_rules.yaml" + mounts: + compat: /mnt/assets + main: /root + io-format: yaml + # This is a key value map specifying dependent services that this service needs in order to function. The keys are the package id's on which you depend. NOTE: if developing a standalone service, you may leave this stanza as an empty object (the key dependencies is required) + dependencies: + # Key must be the package id of another service that exists in the marketplace + filebrowser: + # The version range that is acceptable for this dependency + version: "^2.14.1.1" + # Describes if the dependency is critical to the service functioning. If the dependency is critical, the service will stop if this dependency is stopped. + critical: false + # Specifies the requirement type of the dependency + requirement: + # "Opt-out" means the dependency will be required according to the default config. "Opt-in" means the dependency may be required if you change the config. And "required" just means it's always required. + type: "opt-out" + # An explanation of how to opt-in or opt-out. This value is optional for type=required + how: Optionally use the selected dependency + # Description of the dependency relationship + description: A dependency that demonstrates the way to configure a dependent service + # This is a list of rules that levies requirements on the configuration of the dependency and suggests ways to remedy any incompatibilities. Documentation of this feature is outside the scope of this example. + config: ~ + # This denotes any data, asset, or pointer volumes that should be connected when the "docker run" command is invoked + volumes: + # This is the image where files from the project asset directory will go + main: + type: data + # This is an example of an asset volume + compat: + type: assets + # This specifies how to configure the port mapping for exposing the service over TOR and LAN (if applicable). Many interfaces can be specified depending on the needs of the service. If it can be launched over a Local Area Network connection, specify a `lan-config`. Otherwise, at minimum, a `tor-config` must be specified. + interfaces: + # This key is the internal name that the OS will use to configure the interface + main: + # A human readable name for display in the UI + name: Network Interface + # A descriptive description of what the interface does + description: Specifies the interface to listen on for HTTP connections. + tor-config: + # Port mappings are from the external port to the internal container port + port-mapping: + 80: "80" + # Port mappings are from the external port to the internal container port + lan-config: + 80: + ssl: false + internal: 80 + # Denotes if the service has a user interface to display + ui: true + # Denotes the protocol specifications used by this interface + protocols: + - tcp + - http + # Alerts: omitting these will result in using the default alerts in EmbassyOS, except for start, which has no default. + alerts: + install-alert: This is an alert that will present before the user installs this service + uninstall-alert: This is an alert that will present before the user uninstalls this service + restore-alert: This is an alert that will present before the user restores this service from Embassy backup + start-alert: This is an alert that will present before the user starts this service + # Specifies how backups should be run for this service. The default EmbassyOS provided option is to use the duplicity backup library on a system image (compat) + backup: + create: + type: docker + image: compat + system: true + entrypoint: compat + # Arguments to pass into the entrypoint. In this example, the full command run will be: `compat duplicity hello-world /mnt/backup /root/data` + args: + - duplicity + - hello-world + - /mnt/backup + # For duplicity, the backup mount point needs to be something other than `/root`, so we default to `/root/data` + - /root/data + mounts: + # BACKUP is the default volume that is used for backups. This is whatever backup drive is mounted to the device, or a network filesystem. + # The value here donates where the mount point will be. The backup drive is mounted to this location. + BACKUP: "/mnt/backup" + main: "/root" + restore: + type: docker + image: compat + system: true + entrypoint: compat + args: + - duplicity + - hello-world + - /root/data + - /mnt/backup + mounts: + BACKUP: "/mnt/backup" + main: "/root" + # Commands that can be issued from the UI. NOTE: if no actions are required, this section can be left as an empty object + actions: + hello-world-action: + name: Hello World Action + description: A description that describes what the action will accomplish. + warning: | + A warning message indicating and potential dangers associated with the action + # Indicates what state the service can be in while executing the action + allowed-statuses: + - running + # Defines how the action is run + implementation: + type: docker + image: main + entrypoint: sh + args: ["-c", "echo 'hello-world'"] + # Same as note on health-checks + inject: true + # Required - valid values are yaml, toml, json + io-format: json + + +Instructions +============ + +An instructions file is a convenient way to share any steps users should take to setup or interact with your service. This file gets displayed within an EmbassyUI component and should be written in `Markdown `_ language. + +Let's add instructions to our hello world project: + +.. code:: bash + + touch instructions.md + +And add the following code to the file: + +.. code:: bash + + # Instructions for Hello World + + Instructions go here. These appear to the user in the UI on the Service page under 'Instructions.' + +License +======= + +Start9 ensures that the proper license is displayed for all open source software running on an EmbassyOS platform. Let's make sure to include the full open source license so users can view the distribution permissions of your service, among other licensing details. + +The name and location of this file should be specified in the `assets.license` section of the Manifest. The default value if not specified is `LICENSE`, located in the root of the project folder. + +.. code:: bash + + touch ./hello-world/LICENSE + +Icon +==== + +Icons are displayed throughout the EmbassyUI to reference to your service. + +Simply add the icon file to the root of the project directory. The icon file can be named anything, but this must be specified in the `assets.icon` section of the Manifest. The default filename the SDk looks for when packaging the service assets is `icon.png`. + +.. code:: bash + + mv /local/path/to/icon ./hello-world/icon.png + +Package into s9pk +================= + +We now have all of the necessary components to package the service into the format needed for the OS. This format is a custom filetype with an extension of `.s9pk`, short for Start9 Package. + +To package all components into an `.s9pk`, run the following command from the root of your project directory: + +.. code:: bash + + embassy-sdk pack + +Let's also make sure to verify the validity of the package: + +.. code:: bash + + embassy-sdk verify s9pk /path/to/hello-world.s9pk + +If anything goes wrong, an error message will indicate the missing component or other failure. + +That's it! + +Wrapper Repo +============ + +In order for the Start9 team to review your package for submission to the Start9 Marketplace, we ask that you create a wrapper repository for the project and its components. Let's do that for our hello-world service. + +For a quick start convenience, Start9 has made the finalized version of the `hello-world-wrapper `_ available as a *GitHub template*. Clicking "Use this template" in that repository will clone the entire contents to a specified location. Each file will still have to be manually edited to reflect the changes necessary for your service. + +If you want to proceed from scratch, follow these steps: + +1. In GitHub, create a new public repository with the name "hello-world-wrapper" under your user profile. Go ahead and select the options to include a README file and a .gitignore file. You can always add these files later too. + +2. Once the hosted repository is created, select the "Code" dropdown to copy the https or ssh URL for the repository. If you do not have git setup locally, follow the :ref:`setup steps ` first. + + .. code:: bash + + git clone https://github.com//hello-world-wrapper.git + cd hello-world-wrapper + + +3. Include the `hello-world` project in the wrapper repo. It can either be included directly, or it can be hosted separately. If it is hosted separately, it should be included as a `git submodule `_ within the wrapper repository: + + .. code:: bash + + git submodule add + +4. Edit the `.gitignore` file to include the `.s9pk` file and `image.tar` bundle. This will exclude these files from being published remotely, as they can be large or binary representations. + + .. code:: bash + + hello-world.s9pk + image.tar + +5. Move the Dockerfile, docker_entrypoint.sh, LICENSE, icon, and Manifest to the root of the wrapper repository. At the end, your project structure should look similar to this: + + .. code:: bash + + ├── Dockerfile + ├── LICENSE + ├── Makefile + ├── README.md + ├── assets + │ └── compat + │ ├── config_rules.yaml + │ └── config_spec.yaml + ├── docker_entrypoint.sh + ├── hello-world + │ ├── Cargo.lock + │ ├── Cargo.toml + │ ├── src + │ │ ├── index.html + │ │ └── main.rs + │ └── target + │ ├── aarch64-unknown-linux-musl + │ ├── debug + │ └── release + ├── hello-world.s9pk + ├── icon.png + ├── image.tar + ├── instructions.md + └── manifest.yaml + +Makefile +======== + +For convenience and repeatability, let's combine all of these commands into a Makefile. Then, we can use `make `_ to rebuild our project quickly. + +.. code:: bash + + touch Makefile + +1. Add the build rule with the target executable as the key, including a list of dependencies needed to build the target file. In this case, the `hello-world` binary compiled for aarch is the target, and the dependencies are the hello-world source files needed to compile this binary: + + .. code:: bash + + HELLO_WORLD_SRC := $(shell find ./hello-world/src) hello-world/Cargo.toml hello-world/Cargo.lock + + hello-world/target/aarch64-unknown-linux-musl/release/hello-world: $(HELLO_WORLD_SRC) + docker run --rm -it -v ~/.cargo/registry:/root/.cargo/registry -v "$(shell pwd)"/hello-world:/home/rust/src start9/rust-musl-cross:aarch64-musl cargo build --release + +2. Add the step to build the Docker image. Here, the target is the Docker `image.tar` artifact, and the dependencies are the Dockerfile, docker_entrypoint.sh, and the aarch64 compiled hello-world executable: + + .. code:: bash + + image.tar: Dockerfile docker_entrypoint.sh hello-world/target/aarch64-unknown-linux-musl/release/hello-world + DOCKER_CLI_EXPERIMENTAL=enabled docker buildx build --tag start9/hello-world/main:$(VERSION) --platform=linux/arm64 -o type=docker,dest=image.tar . + +3. Next, add the step for building the `s9pk` package, with the `hello-world.s9pk` as the target, and all the component files as the dependencies: + + .. code:: bash + + ASSETS := $(shell yq e '.assets.[].src' manifest.yaml) + + hello-world.s9pk: manifest.yaml assets/compat/config_spec.yaml assets/compat/config_rules.yaml image.tar instructions.md $(ASSET_PATHS) + embassy-sdk pack + +4. Then, add the step to verify the package: + + .. code:: bash + + S9PK_PATH=$(shell find . -name hello-world.s9pk -print) + + verify: hello-world.s9pk $(S9PK_PATH) + embassy-sdk verify s9pk $(S9PK_PATH) + +5. Add steps to clean up the Makefile build artifacts when you want to build from a fresh slate: + + .. code:: bash + + clean: + rm -f image.tar + rm -f hello-world.s9pk + +6. Finally, add the `all` make target. + + .. code:: bash + + all: verify + + This serves as the entrypoint to build multiple targets, which we have in this case. When the `make` command is invoked here, it looks for the "verify" target. Since the "verify" target depends on the "hello-world.s9pk" target, make then runs this target. It continues down this graph until the first target and its dependencies are satisfied, then works its way back up. The final output of this Makefile is the `image.tar` and `hello-world.s9pk` files. + +That's it! Our completed Makefile looks like this: + +.. code:: make + + ASSETS := $(shell yq e '.assets.[].src' manifest.yaml) + ASSET_PATHS := $(addprefix assets/,$(ASSETS)) + VERSION := $(shell yq e ".version" manifest.yaml) + HELLO_WORLD_SRC := $(shell find ./hello-world/src) hello-world/Cargo.toml hello-world/Cargo.lock + S9PK_PATH=$(shell find . -name hello-world.s9pk -print) + + # delete the target of a rule if it has changed and its recipe exits with a nonzero exit status + .DELETE_ON_ERROR: + + all: verify + + verify: hello-world.s9pk $(S9PK_PATH) + embassy-sdk verify s9pk $(S9PK_PATH) + + clean: + rm -f image.tar + rm -f hello-world.s9pk + + hello-world.s9pk: manifest.yaml assets/compat/config_spec.yaml assets/compat/config_rules.yaml image.tar instructions.md $(ASSET_PATHS) + embassy-sdk pack + + image.tar: Dockerfile docker_entrypoint.sh hello-world/target/aarch64-unknown-linux-musl/release/hello-world + DOCKER_CLI_EXPERIMENTAL=enabled docker buildx build --tag start9/hello-world/main:$(VERSION) --platform=linux/arm64 -o type=docker,dest=image.tar . + + hello-world/target/aarch64-unknown-linux-musl/release/hello-world: $(HELLO_WORLD_SRC) + docker run --rm -it -v ~/.cargo/registry:/root/.cargo/registry -v "$(shell pwd)"/hello-world:/home/rust/src start9/rust-musl-cross:aarch64-musl cargo build --release + +Install on EmbassyOS +==================== + +Now that we have a process for iterating on producing a valid package for EmbassyOS, let's try to load it onto an Embassy device! If you do not have one, you can either :ref:`purchase ` a device or build one using our :ref:`DIY guide `. + +1. First, generate an ssh key for the Embassy: + + .. code:: bash + + ssh-keygen -t ed25519 + # Press Enter to leave filename as default + # Press Enter to leave password empty + # Press Enter to confirm password is empty + # Copy file contents to clipboard. This is your ssh pubkey. + pbcopy .ssh/id_ed25519.pub + +2. On an Embassy device, enter the ssh pubkey into your SSH settings: + + #. Click on Embassy in the menu + #. Click on SSH under SETTINGS + #. Click on + Add new key + #. Paste pubkey from clipboard + +3. Copy the `hello-world.s9pk` to the Embassy device: + + .. code:: bash + + # Confirm you can ssh into your Embassy + ssh root@ + # Log out of Embassy SSH session + exit + + scp .s9pk ssh root@:/working/directory/path + + eg. scp hello-world.s9pk ssh root@embassy-12345678.local:/root + +4. Finally, install the package on an Embassy device: + + .. code:: bash + + ssh root@ + # log in to the command line interface using the Embassy password + embassy-cli auth login + embassy-cli package install hello-world.s9pk + +Congratulations! You have successfully created and installed a package you created onto EmbassyOS. The package should now be viewable in the "Services" tab in EmbassyUI. + +From here, you can play with viewing the results of your Manifest file settings, such as config, actions, interfaces, health checks, etc. You can also view the logs of your service right in the UI! + +In order to verify your service is functioning as expected: + +- Ensure your service is in "Running" state +- Make sure there are no apparent errors or warnings in the logs +- Ensure each UI component renders as expected: + - Instructions + - Config + - Properties + - Actions + - Interfaces + - Marketplace listing + - Donation +- Launch or use your service in the intended way and make sure all aspects function + +Get help +======== + +If you get stuck or are having issues debugging why your service is not packaging or running as expected, reach out to our community `dev chat `_ with questions. + +Submission Process +================== + +When you have built and tested your project for EmbassyOS, please send Start9 a submission to dev@start9labs.com with a link to the wrapper repository. After being reviewed for security and compatibility, the service will be deployed to the Start9 Marketplace and available for all EmbassyOS users to download. + +If you are deploying to an alternative marketplace, please shout it out in our community channels! diff --git a/site/source/service-packaging/build-package-example/references.rst b/site/source/service-packaging/build-package-example/references.rst new file mode 100644 index 0000000..16c1401 --- /dev/null +++ b/site/source/service-packaging/build-package-example/references.rst @@ -0,0 +1,18 @@ +.. _packaging-references: + +==================== +Packaging References +==================== + +Below are links to working examples of more advanced configurations for current Embassy services. They might serve as a starting point or reference during your development process: + +TODO update links when merged to master + +- Detailed `docker_entrypoint.sh` - `filebrowser-wrapper `_ +- Optional dependencies - `btcpayserver <>`_ +- Config spec - `btcpayserver <>`_ +- Config rules - `btcpayserver <>`_ +- Multiple dependencies - `btcpayserver <>`_ +- Actions - `btcpayserver <>`_ +- Health checks - `btcpayserver <>`_ +- Alerts - `lnd `_ diff --git a/site/source/service-packaging/getting-started/environment-setup.rst b/site/source/service-packaging/getting-started/environment-setup.rst new file mode 100644 index 0000000..fb3b68d --- /dev/null +++ b/site/source/service-packaging/getting-started/environment-setup.rst @@ -0,0 +1,93 @@ +.. _environment-setup: + +=========================== +Packaging Environment Setup +=========================== + +To get started packaging a service for EmbassyOS, some basic knowledge of software development is required. Don't worry if you are inexperienced, we will provide enough context to get you started, and you can always reach out with questions. + +If you are already an experienced developer, `jump ahead ` + +The only system requirements are `Docker `_ and `Cargo `_ (Rust package manager). + +Of course, there are additional recommendations that would streamline your development experience, but they are not required. + +Code Editor +=========== + +A code editor is recommended since you will, after all, be writing code! We are fans of`Visual Studio Code `_ . Visual Studio Code is a free, batteries-included text editor made by Microsoft. + +Terminal / CLI +============== + +We recommend using the build in terminal as a command line interface (CLI) for your operating system. For Windows users, we recommend the built-in command line (cmd) *Command Prompt* or the Powershell CLI, running in Administrator mode. For macOS and Linux, the native *Terminal* is recommended, but virtually any terminal will work. + +If you are new to the command line, Ionic has published this awesome `guide `_ to get you started. + +Git +=== + +Although not required, the version control system Git is highly recommended. + +Git is often accompanied by a Git Host, such as `GitHub `_, in which case additional setup is required. Follow the tutorial from the GitHub `here `_ to set up your environment. + +To verify the installation, open a new terminal window and run: + +.. code:: bash + + git --version + +Docker +====== + +`Docker `_ must be installed to your computer platform. It is needed to build an image for your package, which will be used to create the running instance of your package on EmbassyOS. In essence, it declares the necessary environment and building stages for your package to run. + +We also recommend installing and using `Docker buildx `_, as this adds desirable new features to the Docker build experience. It is included by default with Docker Desktop for Windows and macOS. + + +Cargo +===== + +Cargo is the package management solution for the Rust programming language. It will install Rust to your system, and provide the required environment to build the Embassy SDK for packaging your service into the required format needed by EmbassyOS. + +Installation instructions for Cargo can be found `here `_. + +To verify the installation, open a terminal window and run: + +.. code:: bash + cargo --version + +SDK +==== + +EmbassyOS provides a :ref:`software development kit ` embedded in its environment. You do not need a running instance of EmbassyOS to use this component, it can be installed on any computer platform. + +To install the SDK: + +.. code:: bash + + git clone https://github.com/Start9Labs/embassy-os.git + cd backend + ./install-sdk.sh + +To verify the installation, open a terminal window and run: + +.. code:: bash + + embassy-sdk --version + +EmbassyOS +========= + +While not strictly necessary, having a running instance of EmbassyOS is recommended to test installing, running, configuring, and using your package. Without this component, you will have to coordinate with Start9's internal development team to test. ... TODO add more + +You can acquire EmbassyOS by :ref:`Purchase ` or following the :ref:`DIY guide `. + +.. _quick-start: +Quick Start Environment Setup +============================= +- `Docker `_ +- `Docker buildx `_ +- `Cargo `_ +- `EmbassyOS ` +- `Embassy SDK `_ \ No newline at end of file diff --git a/site/source/service-packaging/getting-started/index.rst b/site/source/service-packaging/getting-started/index.rst new file mode 100644 index 0000000..9d85237 --- /dev/null +++ b/site/source/service-packaging/getting-started/index.rst @@ -0,0 +1,63 @@ +.. _packaging-getting-started: + +======================== +Getting Started Overview +======================== + + Select an option below to get started with packing a service for EmbassyOS. + +.. raw:: html + +
+ +
+ +.. topic-box:: + :title: Environment Setup + :link: environment-setup + :icon: scylla-icon scylla-icon--integrations + :class: large-4 + :anchor: View + + Get your environment setup to optimize service packaging + +.. topic-box:: + :title: Build your first Package + :link: build-package-example/index + :icon: scylla-icon scylla-icon--open-source + :class: large-4 + :anchor: Begin + + Follow along to learn packaging fundamentals with an example + +.. topic-box:: + :title: Packaging Quick Start + :link: packaging-quick-start + :icon: scylla-icon scylla-icon--roadmap + :class: large-4 + :anchor: Begin + + Quickly get started with concise packaging steps + +.. topic-box:: + :title: SDK + :link: sdk + :icon: scylla-icon scylla-icon--apps + :class: large-4 + :anchor: View + + Discover details about Start9's Software Development Kit + + +.. raw:: html + +
+ +.. This is for the side navigation display +.. toctree:: + :maxdepth: 1 + :hidden: + + environment-setup + sdk + quick-start \ No newline at end of file diff --git a/site/source/service-packaging/getting-started/quick-start.rst b/site/source/service-packaging/getting-started/quick-start.rst new file mode 100644 index 0000000..367e67f --- /dev/null +++ b/site/source/service-packaging/getting-started/quick-start.rst @@ -0,0 +1,33 @@ +.. _packaging-quick-start: + +===================== +Packaging Quick Start +===================== + +This guide outlines a checklist of steps to complete in order to package a service for EmbassyOS. For a more in depth example, visit the :ref:`build your first package ` guide. + +Packaging +--------- + +1. Create or select project +2. Build project +3. Cross compile for armv8/aarch64 if necessary +4. Create Dockerfile and docker entrypoint file +5. Add build steps, compiled executables, assets, etc in Dockerfile and specify run command in docker entrypoint, handling signal exiting if the service does not already do this gracefully +6. Create Manifest file +7. Create instructions file +8. Create icon file +9. Add license +10. Package all components into s9pk using embassy-sdk +11. Verify package components using embassy-sdk +12. Create a wrapper repository on GitHub to host all package assets for review +13. (Optional) Add all package build steps to a Makefile for replicability + +Testing: +-------- + +1. Sideload s9pk onto an Embassy +2. Install package using embassy-cli +3. Start package using embassy-cli, or in the UI +4. Check logs to see if errors +5. Ensure service is reachable/launchable \ No newline at end of file diff --git a/site/source/service-packaging/getting-started/sdk.rst b/site/source/service-packaging/getting-started/sdk.rst new file mode 100644 index 0000000..7417c01 --- /dev/null +++ b/site/source/service-packaging/getting-started/sdk.rst @@ -0,0 +1,97 @@ +.. _sdk: + +=========== +Embassy SDK +=========== + +Embassy SDK, or Software Development Kit, is a CLI (Command Line Interface) tool that aids in building and packaging services you wish to deploy to the Embassy. + +It mainly helps you validate that the necessary components of your package exist, and package all of those components into a special file type that is understood by EmbassyOS. + +To install, checkout the SDK step in :ref:`setting up your environment `. + +Commands +======== + +To see a list of all available commands provided, run the following from an terminal window: + +.. code:: bash + + embassy-sdk --help + + +embassy-sdk init +---------------- + +Initialized the developer key for interacting with the SDK + +By default, this creates the developer key at `/etc/embassy`. You might need to change ownership of this folder depending on your system permissions. + +.. code:: bash + + chown /etc/embassy + +Alternatively, you can write a config file with your desired developer location, it simply needs the following format: + + .. code:: yaml + + developer-key-path: /desired/path/to/key + +And load it by running: + + .. code:: bash + + embassy-sdk -c /path/to/config init + + +embassy-sdk pack +---------------- + +This command takes the necessary package components and assembles them into the `s9pk` file format needed to install a service on EmbassyOS. It expects the following files to exist: + +- Manifest +- Instructions +- License +- Icon + +If this command fails, the error response will indicate which component is missing. + +embassy-sdk verify +------------------- + +This command verifies aspects about the components assembled into the `s9pk`, such as: + +- Ensures that all mounts are real volumes in the manifest +- Ensures all cert volumes point to real interfaces in the manifest +- Ensures all actions refer to real images in the manifest +- Ensures all images are tagged correctly in the manifest +- Ensures the icon is less than 100KB + +It should be run _after_ `embassy-sdk pack` in order to verify the validity of each component. + +If this command fails, the error message will indicate the mismatched details. + +embassy-sdk git-info +-------------------- + +This command outputs the git commit hash of the SDK version installed on your platform. + +embassy-sdk inspect +------------------- + +This command contains several utilities for reading components once packaged into the `s9pk`. In development, it can be helpful to determine if each component is successfully included in the `s9pk` package. + +It contains the following subcommands, and requires the path to the `.s9pk` file as the last argument: + +- docker-images +- hash +- icon +- instructions +- license +- manifest + +For example: + +.. code:: bash + + embassy-sdk inspect instructions /path/to/.s9pk \ No newline at end of file diff --git a/site/source/service-packaging/index.rst b/site/source/service-packaging/index.rst new file mode 100644 index 0000000..b618b5f --- /dev/null +++ b/site/source/service-packaging/index.rst @@ -0,0 +1,89 @@ +.. _service-packaging: + +========================== +Service Packaging Overview +========================== + + Welcome to Service Packaging! + + If you are here, you are interested in becoming part of the mission to change the future of personal computing. + + The guides below provide the fundamentals that will take you through the process of packing a service for EmbassyOS. Services are any open source project (application) that can be run on a self-hosted platform, independent of third parties. + + By configuring and packaging a project according to these guides, it can be installed on EmbassyOS so that users can interact with the service without needing any technical expertise. + + Let's get started! + +.. raw:: html + +
+ +
+ +.. topic-box:: + :title: Getting Started + :link: packaging-getting-started + :icon: scylla-icon scylla-icon--nsql-guides + :class: large-4 + :anchor: View + + Set up your environment and follow along with an example + +.. topic-box:: + :title: Build your first Package + :link: build-package-example + :icon: scylla-icon scylla-icon--open-source + :class: large-4 + :anchor: Begin + + Follow along to learn packaging fundamentals with an example + +.. topic-box:: + :title: Packaging Quick Start + :link: packaging-quick-start + :icon: scylla-icon scylla-icon--roadmap + :class: large-4 + :anchor: Begin + + Quickly get started with concise packaging steps + +.. topic-box:: + :title: SDK + :link: sdk + :icon: scylla-icon scylla-icon--apps + :class: large-4 + :anchor: View + + Discover details about Start9's Software Development Kit + +.. topic-box:: + :title: Full Specification + :link: packaging-specification + :icon: scylla-icon scylla-icon--glossary + :class: large-4 + :anchor: View + + Detailed service packaging specification and advanced features + +.. topic-box:: + :title: Advanced Guides + :link: advanced/index + :icon: scylla-icon scylla-icon--integrations + :class: large-4 + :anchor: View + + Guides for implementing advanced service configurations + +.. raw:: html + +
+ +.. This is for the side navigation display +.. toctree:: + :maxdepth: 1 + :hidden: + + getting-started/index + build-package-example/index + specification/index + advanced/index diff --git a/site/source/support/dev-docs/service-packaging/backups.rst b/site/source/service-packaging/specification/backups.rst similarity index 94% rename from site/source/support/dev-docs/service-packaging/backups.rst rename to site/source/service-packaging/specification/backups.rst index aba21d9..0ff64e3 100644 --- a/site/source/support/dev-docs/service-packaging/backups.rst +++ b/site/source/service-packaging/specification/backups.rst @@ -22,8 +22,3 @@ Ultimately, ``/datadir/.backupignore`` gets populated with: /root/volumes/btcpayserver/start9/public /root/volumes/btcpayserver/start9/shared - -.. role:: raw-html(raw) - :format: html - -:raw-html:`
` diff --git a/site/source/support/dev-docs/service-packaging/config.rst b/site/source/service-packaging/specification/config.rst similarity index 100% rename from site/source/support/dev-docs/service-packaging/config.rst rename to site/source/service-packaging/specification/config.rst diff --git a/site/source/support/dev-docs/service-packaging/docker.rst b/site/source/service-packaging/specification/docker.rst similarity index 95% rename from site/source/support/dev-docs/service-packaging/docker.rst rename to site/source/service-packaging/specification/docker.rst index 27384bd..5f7d504 100644 --- a/site/source/support/dev-docs/service-packaging/docker.rst +++ b/site/source/service-packaging/specification/docker.rst @@ -33,8 +33,3 @@ Example ------- The `LND wrapper `_ features a well defined Dockerfile, for example. - -.. role:: raw-html(raw) - :format: html - -:raw-html:`
` diff --git a/site/source/service-packaging/specification/emver.rst b/site/source/service-packaging/specification/emver.rst new file mode 100644 index 0000000..e69de29 diff --git a/site/source/service-packaging/specification/index.rst b/site/source/service-packaging/specification/index.rst new file mode 100644 index 0000000..0e28af7 --- /dev/null +++ b/site/source/service-packaging/specification/index.rst @@ -0,0 +1,21 @@ +.. _service-packaging-spec: + +=============================== +Service Packaging Specification +=============================== + +The following guides provide an in depth overview of the full capabilities available for packaging a service. + +.. toctree:: + :hidden: + + overview + wrapper + manifest + docker + makefile + config + properties + instructions + backups + submission diff --git a/site/source/support/dev-docs/service-packaging/instructions.rst b/site/source/service-packaging/specification/instructions.rst similarity index 100% rename from site/source/support/dev-docs/service-packaging/instructions.rst rename to site/source/service-packaging/specification/instructions.rst diff --git a/site/source/support/dev-docs/service-packaging/makefile.rst b/site/source/service-packaging/specification/makefile.rst similarity index 100% rename from site/source/support/dev-docs/service-packaging/makefile.rst rename to site/source/service-packaging/specification/makefile.rst diff --git a/site/source/service-packaging/specification/manifest.rst b/site/source/service-packaging/specification/manifest.rst new file mode 100644 index 0000000..1ed82f8 --- /dev/null +++ b/site/source/service-packaging/specification/manifest.rst @@ -0,0 +1,360 @@ +.. _service_manifest: + +======== +Manifest +======== + +Overview +-------- + +This file describes the service and it's requirements. It is used to: + +- create a listing in the marketplace +- denote any installation considerations +- define dependency requirements +- define actions such as health checks, backups, and configuration +- define alerts and other messaging / descriptions for the user interface + +Each time a service is updated, the Manifest should be updated to include the new version, release notes, and any pertinent updates to the install, uninstall, or restoration flows. All this information is displayed in the marketplace listing, and the optionally denoted alerts will be displayed when appropriate to provide the user with more information. + +There is nothing you need to do as a developer to set up Tor for running a service. This is *completely* handled by EmbassyOS - a Tor address will be automatically generated when the service is installed. Just define an interface with a tor config in the Manifest file. You do, however, need to ensure the service is in fact capable of running over Tor. + +The Manifest is also responsible for outlining service :ref:`dependencies `. By defining rules using the :ref:`EmbassyOS DSL specification `, users can easily and selectively install, uninstall, and update any service without getting stuck in dependency hell. EmbassyOS presents this information in a polished install/uninstall/update wizard, so there's no need for editing configuration files or jumping into the command line. For you as a developer, this simply means populating this key in the manifest! + +Formatting +---------- + +- Serialization language:``.yaml`` +- Case style: ``kebab-case`` + +Type definitions +---------------- + +Below are the types and sub-type definitions, with necessary elaborations. Any item that contains ``Option<>`` is an optional field. + +.. code:: yaml + + # The package identifier used by the OS + id: String + # A human readable service title + title: String + # Service version - accepts up to four digits, where the last confirms to revisions necessary for EmbassyOS - see documentation: https://github.com/Start9Labs/emver-rs. This value will change with each release of the service. + version: Number + # Release notes for the update - can be a string, paragraph or URL + release-notes: String + # The type of license for the project. Include the LICENSE in the root of the project directory. A license is required for a Start9 package. + license: String + # The Start9 wrapper repository URL for the package. This repo contains the manifest file (this), any scripts necessary for configuration, backups, actions, or health checks (more below). This key must exist. But could be embedded into the source repository. + wrapper-repo: String + # The original project repository URL. There is no upstream repo in this example + upstream-repo: String + # URL to the support site / channel for the project. This key can be omitted if none exists, or it can link to the original project repository issues. + support-site: String + # URL to the marketing site for the project. This key can be omitted if none exists, or it can link to the original project repository. + marketing-site: String + # The series of commands to build the project into an s9pk for arm64/v8. In this case we are using a Makefile with the simple build command "make". + build: List + # Minimum required version of EmbassyOS + min-os-version: Number + # Human readable descriptors for the service. These are used throughout the EmbassyOS user interface, primarily in the marketplace. + description: + # This is the first description visible to the user in the marketplace. + short: String + # This description will display with additional details in the service's individual marketplace page + long: String + # These assets are static files necessary for packaging the service for Start9 (into an s9pk). Each value is a path to the specified asset. If an asset is missing from this list, or otherwise denoted, it will be defaulted to the values denoted below. + assets: + # Default = LICENSE.md + license: String + # Default = icon.png + icon: String + # Default = INSTRUCTIONS.md + instructions: String + # Default = image.tar + docker-images: String + # The main action for initializing the service. Currently, the only type of action available is docker. + main: + # Docker is currently the only action implementation + type: String + # Identifier for the main image volume, which will be used when other actions need to mount to this volume. + image: String + # The executable binary for starting the initialization action. For docker actions, this is typically a "docker_entrypoint.sh" file. See the Dockerfile and the docker_entrypoint.sh in this project for additional details. + entrypoint: String + # Any arguments that should be passed into the entrypoint executable + args: List + # Specifies where to mount the data volume(s), if there are any. Mounts for pointer dependency volumes are also denoted here. These are necessary if data needs to be read from / written to these volumes. + mounts: + # Specifies where on the service's file system its persistence directory should be mounted prior to service startup + main: String + # Health checks + health-checks: + main: + name: String + description: String + type: String + image: String + entrypoint: String + args: List + # When `inject` is true, the health check will use the main image to run the health check. This is faster as there is no need to spin up an additional docker container + # When `inject` is false, the health check will use whatever image is specified. This is useful when using a system image with additional utilities to run a health check. If inject=false, then system must equal true + inject: Boolean + # Optional if false - indicates if an image that is preloaded onto the system will be used + system: Boolean + # Required - valid values are yaml, toml, json + io-format: Enum + # Specifies how to get and set configuration file values for the service. NOTE: This stanza can be left empty (null) if the service has no configuration options. + config: + # The config action to run to get the specified config file (default is config.yaml) + get: + # The type of implementation to run this action (currently, only Docker is available) + type: String + # The Docker image to run the action command in. This could be the service's main image, or an image that is preloaded onto the system, like compat (which holds compatible helper functions for default functionality) + image: String + # Indicates if an image that is preloaded onto the system will be used + system: Boolean + # The initial run command to execute the config get action + entrypoint: String + # Any arguments that need to be passed into the run command + args: List + # The locations at which to mount the specified Docker images + mounts: + compat: String + main: String + # Required - valid values are yaml, toml, json + io-format: Enum + # The config action to run to set the specified config file (default is config.yaml). Details for the keys below are the same as above. + set: + type: String + image: String + system: Boolean + entrypoint: String + args: List + mounts: + compat: String + main: String + io-format: Enum + # This is a key value map specifying dependent services that this service needs in order to function. The keys are the package id's on which you depend. NOTE: if developing a standalone service, you may leave this stanza as an empty object (the key dependencies is required) + dependencies: + # Key must be the package id of another service that exists in the marketplace + filebrowser: + # The version range that is acceptable for this dependency + version: Emver + # Describes if the dependency is critical to the service functioning. If the dependency is critical, the service will stop if this dependency is stopped. + critical: Boolean + # Specifies the requirement type of the dependency + requirement: + # "Opt-out" means the dependency will be required according to the default config. "Opt-in" means the dependency may be required if you change the config. And "required" just means it's always required. + type: Enum + # An explanation of how to opt-in or opt-out. This value is optional for type=required + how: String + # Description of the dependency relationship + description: String + # This is a list of rules that levies requirements on the configuration of the dependency and suggests ways to remedy any incompatibilities. Documentation of this feature is outside the scope of this example. + config: ~ + # This denotes any data, asset, or pointer volumes that should be connected when the "docker run" command is invoked + volumes: + # This is the image where files from the project asset directory will go + main: + type: Enum + # This is an example of an asset volume + compat: + type: Enum + # This specifies how to configure the port mapping for exposing the service over TOR and LAN (if applicable). Many interfaces can be specified depending on the needs of the service. If it can be launched over a Local Area Network connection, specify a `lan-config`. Otherwise, at minimum, a `tor-config` must be specified. + interfaces: + # This key is the internal name that the OS will use to configure the interface + main: + # A human readable name for display in the UI + name: String + # A descriptive description of what the interface does + description: String + tor-config: + # Port mappings are from the external port to the internal container port + port-mapping: + 80: String + # Port mappings are from the external port to the internal container port + lan-config: + 80: + ssl: Boolean + internal: Number + # Denotes if the service has a user interface to display + ui: Boolean + # Denotes the protocol specifications used by this interface + protocols: List + # Alerts: omitting these will result in using the default alerts in EmbassyOS, except for start, which has no default. + alerts: + install-alert: String + uninstall-alert: String + restore-alert: String + start-alert: String + # Specifies how backups should be run for this service. The default EmbassyOS provided option is to use the duplicity backup library on a system image (compat) + backup: + create: + type: String + image: String + system: Boolean + entrypoint: String + # Arguments to pass into the entrypoint. + args: List + mounts: + # BACKUP is the default volume that is used for backups. This is whatever backup drive is mounted to the device, or a network filesystem. + # The value here donates where the mount point will be. The backup drive is mounted to this location. + BACKUP: String + main: String + restore: + type: String + image: String + system: Boolean + entrypoint: String + args: List + mounts: + BACKUP: String + main: String + # Commands that can be issued from the UI. NOTE: if no actions are required, this section can be left as an empty object + actions: + hello-world-action: + name: String + description: String + warning: Option + # Indicates what state the service can be in while executing the action + allowed-statuses: List + # Defines how the action is run + implementation: + type: String + image: String + entrypoint: String + args: [List + # Same as note on health-checks + inject: Boolean + # Required - valid values are yaml, toml, json + io-format: Enum + +.. _dependencies-spec: + +Dependencies +------------ + +Many services depend on other libraries and services on EmbassyOS (such as Bitcoin), sometimes even a particular version of those services, which need to be specified by the developers so that EmbassyOS can handle installing these dependencies under the hood. + +The key of each field in the dependencies object is the lowercase, kebab-case app ID of the service that is depended on. Each dependency contains a set of rules that need to be fulfilled as true if the dependency is to be properly installed. The "config rules" here are for auto-configuring dependencies - the action defined by the rule will be executed if the service is auto configured with defaults during initial setup. This simplifies and streamlines the user experience. The interface should provide suggestions for the behavior if the denoted rule cannot be met with previous configurations. + +Let's take this snippet for example: + +.. code:: yaml + + ... + btc-rpc-proxy: + version: ">=0.3.2.1 <0.4.0" + requirement: + type: "opt-in" + how: Can alternatively configure an external bitcoind node. + critical: false + description: Used to fetch validated blocks. + config: + check: + type: docker + image: compat + system: true + entrypoint: compat + args: + - dependency + - check + - btcpayserver + - "btc-rpc-proxy" + - /datadir + - "/mnt/assets/btc-rpc-proxy_config_rules.yaml" + mounts: + main: /datadir + compat: /mnt/assets + io-format: yaml + auto-configure: + type: docker + image: compat + system: true + entrypoint: compat + args: + - dependency + - "auto-configure" + - btcpayserver + - "btc-rpc-proxy" + - /datadir + - "/mnt/assets/btc-rpc-proxy_config_rules.yaml" + mounts: + main: /datadir + compat: /mnt/assets + io-format: yaml + ... + +.. role:: raw-html(raw) + :format: html + +:raw-html:`
` + +TODO update for accuracy + +The service ``btc-rpc-proxy`` is a dependency of the service ``c-lightning``. ``c-lightning`` requires it to be installed at a version >=0.3.2.1 but <0.4.0. There exists a rule that states the config option ``user.name`` must be equal to "c-lightning". If this value does not exist for ``user.name`` when accessed, ``PUSH`` the value "c-lighting" to the field. This all takes place during the initial service configuration, before the service is started for the first time. + +.. note:: + Dependency config rules are processed in order. + +Type definitions +================ + +Types for ``manifest.yaml`` fields: TODO check accuracy + +.. code:: typescript + + interface Dependencies [{ + serviceId: DepInfo + }] + + interface DepInfo { + version: String // ie. 0.11.1.1 + optional?: String, + description?: String, + config: [ConfigRule], + ], + } + + interface ConfigRule { + rule: String, // ie. 'users.*.name = "lnd" + description: String, + suggestions: [SuggestionVariant] + } + + interface SuggestionVariant { + SET: { + var: String, + to: SetVariant, + }, + DELETE: { + src: String, + }, + PUSH: { + to: String, + value: Value, + }, + } + + interface SetVariant { + to: Option, + to-value: Option, // ie. true/false + to-entropy: Option<{ + charset: String // ie. 'a-z,A-Z,0-9' + len: number + }> + } + +.. role:: raw-html(raw) + :format: html + +:raw-html:`
` + +Examples +======== + +- `filebrowser <>`_ +- `embassy-pages <>`_ +- `photoview <>`_ +- `btcpayserver <>`_ +- `lnd <>`_ +- `synapse <>`_ diff --git a/site/source/support/dev-docs/service-packaging/overview.rst b/site/source/service-packaging/specification/overview.rst similarity index 100% rename from site/source/support/dev-docs/service-packaging/overview.rst rename to site/source/service-packaging/specification/overview.rst diff --git a/site/source/support/dev-docs/service-packaging/properties.rst b/site/source/service-packaging/specification/properties.rst similarity index 100% rename from site/source/support/dev-docs/service-packaging/properties.rst rename to site/source/service-packaging/specification/properties.rst diff --git a/site/source/support/dev-docs/service-packaging/submission.rst b/site/source/service-packaging/specification/submission.rst similarity index 100% rename from site/source/support/dev-docs/service-packaging/submission.rst rename to site/source/service-packaging/specification/submission.rst diff --git a/site/source/support/dev-docs/service-packaging/wrapper.rst b/site/source/service-packaging/specification/wrapper.rst similarity index 100% rename from site/source/support/dev-docs/service-packaging/wrapper.rst rename to site/source/service-packaging/specification/wrapper.rst diff --git a/site/source/support/dev-docs/dev-faq/index.rst b/site/source/support/dev-docs/dev-faq/index.rst deleted file mode 100644 index 3e4876d..0000000 --- a/site/source/support/dev-docs/dev-faq/index.rst +++ /dev/null @@ -1,13 +0,0 @@ -.. _dev-faq: - -============= -Developer FAQ -============= - -Frequently Asked Questions by and for developers. If you have a suggestion, please use the Github link at the top of the page. - -.. toctree:: - :maxdepth: 3 - - faq-contributing - faq-service-packaging diff --git a/site/source/support/dev-docs/index.rst b/site/source/support/dev-docs/index.rst deleted file mode 100644 index 35684bb..0000000 --- a/site/source/support/dev-docs/index.rst +++ /dev/null @@ -1,24 +0,0 @@ -.. _dev-docs: - -============== -Developer Docs -============== - -.. toctree:: - :maxdepth: 3 - :hidden: - - dev-tools/index - packaging-example - service-packaging/index - dev-faq/index - -.. panel-box:: - :title: Overview - :id: "dev-docs" - :class: my-panel - - * :doc:`Developer Tools ` - An overview of tools for those interested in contributing to the OS or packaging a service. - * :doc:`Package Example ` - A walkthrough of how to package a service using an example. - * :doc:`Service Packaging ` - The full specification for packaging a service. - * :doc:`Developer FAQ ` - Frequently asked questions by developers in regard to EmbassyOS and packaging services. diff --git a/site/source/support/dev-docs/packaging-example.rst b/site/source/support/dev-docs/packaging-example.rst deleted file mode 100644 index e4e4419..0000000 --- a/site/source/support/dev-docs/packaging-example.rst +++ /dev/null @@ -1,543 +0,0 @@ -.. _packaging-example: - -========================= -Service Packaging Example -========================= - -.. contents:: - :depth: 4 - :local: - - -Welcome! The following guide will provide the prerequisites, introduce a brief overview of the packaging process, use an example demonstrating how to package a service, and finally, describe the submission process. This essentially describes how you can take an existing app (or one you have written yourself), and wrap it up into an ``s9pk`` such that it can be added to an EmbassyOS (EOS) Marketplace! The ``s9pk`` is the final product, which is the portable version of a package that is understood by EOS, and can be distributed to any EOS users either directly, or via a Marketplace. - -Pre-requisites --------------- - -EmbassyOS (EOS) -=============== - -It is **HIGHLY RECOMMENDED** to have a copy of EmbassyOS for testing your packaged service. - -There are 3 options for this: - #. Build from `source `_ - #. Follow the :ref:`DIY guide ` to build on a Raspberry Pi - #. :ref:`Purchase ` a device or copy of the OS - -Development Environment -======================= - -Once you have EOS installed, you'll want to get your development system set up with the necessary software. - -At minimum you will need the following: - #. `Docker `_ - #. `Docker-buildx `_ - #. `Embassy-SDK `_ - -The following are recommended: - #. `Cargo `_ - #. `yq `_ (version 4) - #. `make `_ - #. `rust-musl-cross `_ (For cross compiling Rust to Alpine, not needed otherwise) - -Overview --------- - -Components -========== - -The package is made up of the following parts: - - #. Container Image - Each service is running in a container (typically Docker). Best results will come from an arm based linux; [Alpine](https://www.alpinelinux.org/) is highly recommended. - #. Volume(s) - Each service gets one or more volumes, allocated by EOS. This is the filesystem where the service data will be stored and mounted by the container. The volume directory within EOS (for seeding data into the volume) is located at `/embassy-data/package-data/volumes/` - #. Dependencies - Rules and requirements of your service, which appear as UI elements, such as inputs, toggles, and drop-downs. These are enforced by validations and clear user instructions. EmbassyOS has a unique and powerful system for managing dependencies which allows anyone to have the power of systems administrators without an advanced skillset. - #. Manifest - Describes the service and its requirements. This file describes the marketplace listing, installation considerations, configuration and dependency requirements, health checks, backups and additional info. - #. Config - EOS makes a service's configuration available to the user in the GUI and must be valid regardless of user skill. - #. Instructions & Docs - Directions for setup, use, external integrations, etc. - #. License - The Open Source license of the service wapper. - #. Icon - Image used to identify the service in the UI. - -The ``.s9pk`` is the image, manifest, license, icon, and instructions files bundled into a package. Optional additional assets for use with system images can also be bundled. This is the file a user downloads from the Marketplace, at which point EOS uses to unpack assets and install the service. - -Check :ref:`here ` for a detailed overview of package components. - -Service Wrapper Repo and Submodules -=================================== - -See :ref:`here ` for how to structure your service wrapper's git repository. - -Git submodules allow the use of another project while in the working project directory. In this case, you can use an existing app's git repo in order to source its code in your service wrapper. You can add the submodule into the wrapper so that you can build the submodule and also track the exact code you're building. - -Simply run: - - .. code:: bash - - git submodule add - -Example - Hello World ---------------------- - -Okay, let's actually package a service! For this example, we're going to use an example service `Hello World `_. This repository can also be used as a template to quickly get started with your service. The guide will provide good overview of service packaging, but obviously your app will be different, so don't hesitate to reach out to our community `dev chat `_ with questions. - -The commands below assume a Linux development environment with all the recommended dependencies listed above installed. To get started quickly, we'll use Start9's wrapper template. - -Clone the Template Repo and Edit the Manifest -============================================= - -#. Clone and rename the repo (or alternatively, use the template generation button found on the github `repo `_) - - .. code-block:: bash - - git clone https://github.com/Start9Labs/hello-world-wrapper - cd hello-world-wrapper - -#. Edit the ``README.md`` to explain what the service is, what dependencies are required, build/install/contribute instructions, and any other information you'd like. - -#. Edit the ``manifest`` file. This must be in ``.json``, ``.toml``, or ``.yaml`` format and in ``kebab-case`` style. You can see descriptions of each key (and some notes) in our 'Hello World' example ``manifest.yaml`` below: - -Manifest example -================ - -.. code-block:: yaml - - id: hello-world - title: "Hello World" - version: 0.2.0 # Service version - release-notes: "Upgrade to EmbassyOS 2.16.0 and then to v0.3.0" - license: mit - wrapper-repo: "https://github.com/Start9Labs/hello-world-wrapper" - upstream-repo: "https://github.com/Start9Labs/hello-world-wrapper" # There is no upstream repo in this example - support-site: "https://docs.start9.com/" - marketing-site: "https://start9.com/" - build: ["make"] # Series of commands to build into an s9pk, in this case we are using make and all the build commands are in the makefile - min-os-version: "0.3.0" # Minimum required version of EmbassyOS - description: - short: Example service - long: | - Hello World is a simple example of a service wrapper that launches a web interface to say hello and nothing more. - assets: - license: LICENSE # default = LICENSE.md - icon: icon.png # default = icon.png - instructions: docs/instructions.md # default = INSTRUCTIONS.md - docker-images: image.tar # default = image.tar - main: - type: docker - image: main - entrypoint: "/usr/local/bin/docker_entrypoint.sh" - args: [] - mounts: {} # Specifies where to put volumes, if there are any. Empty in this example - health-checks: {} # Health check config would go here - config: ~ # Configuration options, none for hello-world, but see below example for format: - # get: - # type: docker - # image: compat - # entrypoint: compat - # args: - # - "config" - # - "get" - # - "/root/.hello-world/start9/config.yaml" - # - "/mnt/assets/config_spec.yaml" - # mounts: - # compat: "/mnt/assets" - # main: "/root" - # io-format: yaml - # set: - # type: docker - # image: compat - # entrypoint: compat - # args: - # - "config" - # - "set" - # - "/root/.hello-world/start9/config.yaml" - # mounts: - # main: "/root" - # io-format: yaml - dependencies: {} # Service pre-requisites, none for hello-world, but see below example (which would make BTC Proxy a dependency) for format: - # btc-rpc-proxy: - # version: ">=0.3.2.1 <0.4.0" - # recommended: true - # critical: false - # optional: Can alternatively configure an external bitcoin node. - # description: Used to fetch validated blocks. - # config: - # check: - # type: docker - # image: compat - # system: true - # # the compat image will contain a tool to check the config rules against the specified dependency - # entrypoint: compat - # args: - # - "dependency" - # - "check" - # - "/mnt/assets/btc-rpc-proxy.rules.yaml" - # mounts: - # compat: "/mnt/assets" - # auto-configure: - # type: docker - # image: compat - # # if true, the denoted image is prebuilt and comes stock with EOS - # # uncommon: if you want something not prebuilt with EOS, you can bundle multiple docker images into the `image.tar` during the `make` build process - # system: true - # entrypoint: compat - # args: - # - "dependency" - # - "auto-configure" - # - "/mnt/assets/btc-rpc-proxy.rules.yaml" - # mounts: - # compat: "/mnt/assets" - volumes: # this is the image where data will go from 0.2.x - main: - type: data # this is the image where files from the project asset directory will go - compat: - type: assets - interfaces: - main: - name: Network Interface - description: Specifies the interface to listen on for HTTP connections. - tor-config: - port-mapping: - 80: "80" - lan-config: - 80: - ssl: false - mapping: 80 - ui: true - protocols: - - tcp - - http - alerts: {} - backup: - create: - type: docker - image: compat # default backup process of the compat docker image is duplicity - EOS will have access to the image defined here - system: true - entrypoint: compat # command to run the backup executable, in this case, duplicity - args: # arguments to pass into the entrypoint, in this case duplicity - in this example, the full command run will be: `duplicity hello-world file:///mnt/backup /root` - - duplicity - - hello-world - - /mnt/backup - - /root - mounts: - # BACKUP is the default volume that is used for backups. This is whatever backup drive is mounted to the defice, or a network filesystem. - # The value here donates where the data mount point will be. Backup drive is mounted to this location, which contains previous backups. - BACKUP: "/mnt/backup" - main: "/root" - restore: - type: docker - image: compat - system: true - entrypoint: compat - args: - - duplicity - - hello-world - - /root - - /mnt/backup - mounts: - # See above comments under `backup: -> mounts:` - BACKUP: "/mnt/backup" - main: "/root" - actions: {} # Commands that can be issued from the UI. None for hello-world, but see below example (resetting a root user) for format: - # reset-root-user: - # name: Reset Root User - # description: Resets your root user (the first user) to username "admin" and a random password; restores any lost admin privileges. - # warning: This will invalidate existing sessions and password managers if you have them set up. - # allowed-statuses: - # - stopped - # implementation: - # type: docker - # image: main - # system: true - # entrypoint: docker_entrypoint.sh - # args: ["reset-root-user"] - # mounts: - # main: "/root" - -Note the ``dependencies`` and ``volumes`` sections, which may access another service, e.g. File Browser, such that files stored on a user's Embassy can be accessed in your service. - -For details on all the different possible dependency, type, and subtype definitions available for the ``manifest`` file, please see :ref:`here `. - -Edit the Dockerfile and Entrypoint -================================== - -Next, it's time to edit the ``Dockerfile``. This defines how to build the image for the package by declaring the environment, building stages, and copying any binaries or assets needed to run the service or its health checks to the image filesystem. - -#. We start by importing a base image, in this case Alpine, as recommended. - - .. code:: docker - - FROM arm64v8/alpine:3.12 - -#. Next we issue some commands, which in this example simply updates repositories, installs required software, and finally creates a directory for nginx. - - .. code:: docker - - RUN apk update - RUN apk add tini - - RUN mkdir /run/nginx - -#. Next we will add the cross-compiled binary of ``hello-world`` to ``/usr/local/bin/`` and add the ``docker_entrypoint.sh`` file from the repository. Then we set permissions for ``docker_entrypoint.sh``. - - .. code:: docker - - ADD ./hello-world/target/aarch64-unknown-linux-musl/release/hello-world /usr/local/bin/hello-world - ADD ./docker_entrypoint.sh /usr/local/bin/docker_entrypoint.sh - RUN chmod a+x /usr/local/bin/docker_entrypoint.sh - -#. Next, we set a working directory, and set the location of the entrypoint. Exposing ports is not necessary for EOS, but its often useful to leave this line for clarity. - - .. code:: docker - - WORKDIR /root - - EXPOSE 80 - - ENTRYPOINT ["/usr/local/bin/docker_entrypoint.sh"] - -#. Great, let's take a look at our final Hello World ``Dockerfile``: - - .. code:: docker - - FROM arm64v8/alpine:3.12 - - RUN apk update - RUN apk add tini - - ADD ./hello-world/target/aarch64-unknown-linux-musl/release/hello-world /usr/local/bin/hello-world - ADD ./docker_entrypoint.sh /usr/local/bin/docker_entrypoint.sh - RUN chmod a+x /usr/local/bin/docker_entrypoint.sh - - WORKDIR /root - - EXPOSE 80 - - ENTRYPOINT ["/usr/local/bin/docker_entrypoint.sh"] - -Docker Entrypoint -================= - -#. Okay, let's move on to our ``docker_entrypoint.sh`` file. This is a script that defines what to do when the service starts, and often acts as an init system. It will need to complete any environment setup (such as folder substructure), set any environment variables, and execute the run command. It's also PID 1 in the docker container, so it does all of the signal handling and container exits when it is stopped/exits. If you have built a "configurator," this will also need to be called to execute here. More on configurators below. Let's take a look at our (extremely basic) Hello World example: - - .. code:: bash - - #!/bin/sh - - export HOST_IP=$(ip -4 route list match 0/0 | awk '{print $3}') - - exec tini hello-world - -#. We've defined the file, exported the IP address of the Embassy (host), and run the program. - -For a more detailed ``docker_entrypoint.sh``, please check out the `filebrowser-wrapper `_. Additional details on the ``Dockerfile`` and ``docker_entrypoint`` can be found `here `_. - -Configurators -............. - -- Broadly, a "configurator" is any code that translates the configuration coming from the OS to a format the service can understand. Technically all services with a config have one of these (so, most services on Embassy). The configurator also writes the stats.yaml file, which is used in properties. - -- Narrowly, a configurator is a piece of code separate from the docker_entrypoint.sh script, that does the same as the task above. If you can configure the service in bash, inside the docker_entrypoint.sh script, then you don't need a separate piece of code called "configurator", as we have in LND, CL, and synapse, for example. You might want to create a separate configurator if configuring your service would be complicated or impossible in bash. In the case of CL and LND (and bitcoind, where the configurator is actually called a "manager"), the configurator actually has a long-running element that runs concurrently with the service itself. - -There's really no reason a service has to be engineered in this manner. It's really up to the package maintainer how they want to accomplish translation of the config and implementing the properties action, which doesn't even require a stats.yaml file as of 0.3. - -You can check out an example configurator in the `BitcoinD Wrapper `_, which in this case is called a manager. - -Makefile (Optional) -=================== - -Here, we will create a ``Makefile``, which is optional, but recommended as it outlines the build and streamlines additional developer contributions. Alternatively, you could use ``nix``, ``bash``, ``python``, ``perl``, ``ruby``, etc instead of ``make`` for build orchestration. - -Our example ``Makefile`` is again fairly simple for Hello World. Let's take a look: - -.. code-block:: Makefile - - ASSETS := $(shell yq e '.assets.[].src' manifest.yaml) - ASSET_PATHS := $(addprefix assets/,$(ASSETS)) - VERSION := $(shell toml get hello-world/Cargo.toml package.version) - HELLO_WORLD_SRC := $(shell find ./hello-world/src) hello-world/Cargo.toml hello-world/Cargo.lock - S9PK_PATH=$(shell find . -name hello-world.s9pk -print) - - .DELETE_ON_ERROR: - - all: verify - - verify: hello-world.s9pk $(S9PK_PATH) - embassy-sdk verify $(S9PK_PATH) - - # embassy-sdk pack errors come from here, check your manifest, config, instructions, and icon - hello-world.s9pk: manifest.yaml assets/compat/config_spec.yaml config_rules.yaml image.tar docs/instructions.md $(ASSET_PATHS) - embassy-sdk pack - - image.tar: Dockerfile docker_entrypoint.sh hello-world/target/aarch64-unknown-linux-musl/release/hello-world - DOCKER_CLI_EXPERIMENTAL=enabled docker buildx build --tag start9/hello-world --platform=linux/arm64 -o type=docker,dest=image.tar . - - hello-world/target/aarch64-unknown-linux-musl/release/hello-world: $(HELLO_WORLD_SRC) - docker run --rm -it -v ~/.cargo/registry:/root/.cargo/registry -v "$(shell pwd)"/hello-world:/home/rust/src start9/rust-musl-cross:aarch64-musl cargo +beta build --release - docker run --rm -it -v ~/.cargo/registry:/root/.cargo/registry -v "$(shell pwd)"/hello-world:/home/rust/src start9/rust-musl-cross:aarch64-musl musl-strip target/aarch64-unknown-linux-musl/release/hello-world - - manifest.yaml: hello-world/Cargo.toml - yq e -i '.version = $(VERSION)' manifest.yaml - -#. The first 5 lines set environment variables. - -#. The next line simply removes the progress of a ``make`` process if it fails. - - .. code-block:: Makefile - - .DELETE_ON_ERROR: - -#. The ``all`` step is run when the ``make`` command is issued. This attempts the ``verify`` step, which requires that the ``hello-world.s9pk`` must first be built, which first requires the ``image.tar``, and so on. In this case, each step requires the next and each step indicates the necessary existence of a file. If an indicated file has been altered, such as the `docker_entrypoint.sh`, then any step that contains this file will be rebuilt. - -#. So the ``.s9pk`` is created with the ``embassy-sdk pack`` command, supplied with the ``manifest``, ``config_spec``, previously created ``image.tar``, and ``instructions.md``. Your project may likely also contain a ``config_rules`` file. Some of these files we have not yet edited, but that will come shortly. - -#. The ``image.tar`` is built below this, the cross-compiled ``hello-world`` source code, and ``manifest`` at the bottom. - -For more details on creating a ``Makefile`` for your project, please check :ref:`here `. - -Service Config Specification and Rules -====================================== - -Most self-hosted packages require a configuration. With EmbassyOS, these config options are provided to the user in a friendly GUI, and invalid configs are not permitted. This allows the user to manage their software without a lot of technical skill, and minimal risk of borking their software. - -In the config section of the `manifest`, you can -Two files are created in this process: - -``config_spec.yaml`` for specifying all the config options your package depends on to run - -``config_rules.yaml`` for defining the ruleset that defines dependencies between config variables - -These are stored in ``assets/compat/`` for 0.2.x compatibility, and in ``/assets/`` for anything built for v0.3.0 and up (almost certainly what you're doing). These files contain a detailed mapping of configuration options with acceptable values, defaults, and relational rule-sets. Hello World has no configuration, as you can see `here `_. Instead, let's take a look at our ``config_spec`` for Embassy Pages, which actually has some config options: - -.. code-block:: yaml - - homepage: - name: Homepage - description: The page that will be displayed when your Embassy Pages .onion address is visited. Since this page is technically publicly accessible, you can choose to which type of page to display. - type: union - default: welcome - tag: - id: type - name: Type - variant-names: - welcome: Welcome - index: Subdomain Index - filebrowser: Web Page - redirect: Redirect - fuck-off: Fuck Off - variants: - welcome: {} - index: {} - filebrowser: - directory: - type: string - name: Directory Path - description: The path to the directory in File Browser that contains the static files of your website. For example, a value of "websites/resume_site" would tell Embassy Pages to look for that directory in File Browser. - pattern: "^(\\.|[a-zA-Z0-9_ -][a-zA-Z0-9_ .-]*|([a-zA-Z0-9_ .-][a-zA-Z0-9_ -]+\\.*)+)(/[a-zA-Z0-9_ -][a-zA-Z0-9_ .-]*|/([a-zA-Z0-9_ .-][a-zA-Z0-9_ -]+\\.*)+)*/?$" - pattern-description: Must be a valid relative file path - nullable: false - redirect: - target: - type: string - name: Target Subdomain - description: The name of the subdomain to redirect users to. This must be a valid subdomain site within your Embassy Pages. - pattern: '^[a-z-]+$' - pattern-description: May contain only lowercase characters and hyphens. - nullable: false - fuck-off: {} - subdomains: - type: list - name: Subdomains - description: The websites you want to serve. - default: [] - range: '[0, *)' - subtype: object - spec: - unique-by: name - display-as: "{{name}}" - spec: - name: - type: string - nullable: false - name: Subdomain name - description: The subdomain of your Embassy Pages .onion address to host the website on. For example, a value of "me" would produce a website hosted at http://me.myaddress.onion. - pattern: "^[a-z-]+$" - pattern-description: "May contain only lowercase characters and hyphens" - settings: - type: union - name: Settings - description: The desired behavior you want to occur when the subdomain is visited. You can either redirect to another subdomain, or load a web page from File Browser. - default: filebrowser - tag: - id: type - name: Type - variant-names: - filebrowser: Web Page - redirect: Redirect - variants: - filebrowser: - directory: - type: string - name: Directory Path - description: The path to the directory in File Browser that contains the static files of your website. For example, a value of "websites/resume_site" would tell Embassy Pages to look for that directory in File Browser. - pattern: "^(\\.|[a-zA-Z0-9_ -][a-zA-Z0-9_ .-]*|([a-zA-Z0-9_ .-][a-zA-Z0-9_ -]+\\.*)+)(/[a-zA-Z0-9_ -][a-zA-Z0-9_ .-]*|/([a-zA-Z0-9_ .-][a-zA-Z0-9_ -]+\\.*)+)*/?$" - pattern-description: Must be a valid relative file path - nullable: false - redirect: - target: - type: string - name: Target Subdomain - description: The subdomain of your Embassy Pages .onion address to redirect to. This should be the name of another subdomain on Embassy Pages. Leave empty to redirect to the homepage. - pattern: '^[a-z-]+$' - pattern-description: May contain only lowercase characters and hyphens. - nullable: false - -We essentially have 2 config options (homepage and subdomains), with all of their specifications nested below them. Looking at the homepage, it contains a ``union`` type, which is a necessary dependency, which can be of 5 variants (welcome, index, filebrowser, redirect, or fuck-off). The below images show how this is displayed in the UI. - - .. figure:: /_static/images/dev/pages0.png - :width: 60% - :alt: Pages Config - - .. figure:: /_static/images/dev/pages1.png - :width: 60% - :alt: Pages Union - -For all the possible types, please check our :ref:`Service Config Spec `. - -In our example, there is *no need* for a ``config_rules`` file. This is because there is not a rule-set required to define dependencies between config variables. An example of when this would be required would be the following code, from the [LND wrapper](https://github.com/Start9Labs/lnd-wrapper/blob/master/config_rules.yaml): - -.. code-block:: yaml - - --- - - rule: '!(max-chan-size?) OR !(min-chan-size?) OR (#max-chan-size > #min-chan-size)' - description: "Maximum Channel Size must exceed Minimum Channel Size" - -Here we see that a Maximum Channel Size **MUST** be one of 3 possible options in order to be a valid config. - -Properties -========== - -Next we need to create the Properties section for our package, to display any relevant info. The result of this step is a ``stats.yaml`` file, which is only populated at runtime. These commands will be issued in the ``docker_entrypoint`` file (or ``configurator``, if you are using one). - -.. ***STATS.YAML IS APPARENTLY BEING DEPRECATED, THIS SECTION NEEDS COMMENT*** Possibly this is not actually the case? - -Instructions -============ - -Instructions are the basic directions or any particular details that you would like to convey to the user to help get them on their way. Each wrapper repo should contain a ``docs`` directory which can include anything you'd like, but specifically if you include an ``instructions.md`` file, formatted in Markdown language, it will be displayed simply for the user as shown below. - - .. figure:: /_static/images/dev/instructions.png - :width: 60% - :alt: Instructions - -You can find the ``instructions.md`` file for Embassy Pages `here `_ if you are interested. - -Backups -======= - -Everything in the root folder of the mounted system directory will be stored in an EOS backup. If you want to ignore any particular files for backup, you can create a ``.backupignore`` file and add the relative paths of any directories you would like ignored. - -Submission Process ------------------- - -When you have built and tested your project for EmbassyOS, please send Start9 a submission with the project repository to dev@start9labs.com. After being reviewed for security and compatibility, the service will be deployed to the marketplace and available for all EmbassyOS users to download. - -If you are deploying to an alternative marketplace, please shout it out in our community channels! diff --git a/site/source/support/dev-docs/service-packaging/index.rst b/site/source/support/dev-docs/service-packaging/index.rst deleted file mode 100644 index 4450628..0000000 --- a/site/source/support/dev-docs/service-packaging/index.rst +++ /dev/null @@ -1,35 +0,0 @@ -.. _service-packaging-spec: - -====================== -Service Packaging Spec -====================== - -Welcome! If you are here, you are interested in becoming part of the mission to change the future of personal computing. This guide will take you through the process of packaging a service for EmbassyOS, a novel, self-hosted, sovereign computing platform. - -A service in this context is any open source project that has been bundled into the appropriate format to run on EmbassyOS. By configuring and packaging a project according to this guide, it can be installed on EmbassyOS with no command line or technical expertise required. This opens up an entire platform for self-hosted software to run independent of third parties in a completely private and sovereign way for all individuals. - -This guide is technical, but breaks down the steps for deployment. If you have any feedback or questions concerning this guide, please don't hesitate to `reach out `_ or submit a pull request with alterations. - -To start, you will need to acquire EmbassyOS for testing the packaged service. This can be done by: - -- building from `source `_ -- following the :ref:`DIY ` guide -- :ref:`purchasing ` the ready to run personal server - -While you are waiting to receive or assemble a device, you can begin the process of packaging a project. The next sections outline these steps in detail. - -Happy building! - -.. toctree:: - :hidden: - - overview - wrapper - manifest - docker - makefile - config - properties - instructions - backups - submission diff --git a/site/source/support/dev-docs/service-packaging/manifest.rst b/site/source/support/dev-docs/service-packaging/manifest.rst deleted file mode 100644 index 4f37955..0000000 --- a/site/source/support/dev-docs/service-packaging/manifest.rst +++ /dev/null @@ -1,353 +0,0 @@ -.. _service_manifest: - -======== -Manifest -======== - -Overview --------- - -This file describes the service and it's requirements. It is used to: - -- create a listing in the marketplace -- denote any installation considerations -- define dependency requirements - -Each time a service is updated, the manifest should be updated to include the new version, release notes, and any pertinent updates to the install, uninstall, or restoration flows. All this information is displayed in the marketplace listing, and the optionally denoted alerts will be displayed when appropriate to provide the user with more information. - -For instance, `LND displays alerts `_ around restoration since data will be overwritten. - -There is nothing you need to do as a developer to set up Tor for running a service. This is *completely* handled by EmbassyOS - a Tor address will be automatically generated when the service is installed. Just define the port and which version of Tor the service needs in this manifest file! You do, however, need to ensure the service is in fact capable of running over Tor. - -The manifest is also responsible for outlining service :ref:`dependencies `. By defining rules using the :ref:`EmbassyOS DSL specification `, users can easily and selectively install, uninstall, and update any service without getting stuck in dependency hell. EmbassyOS presents this information in a polished install/uninstall/update wizard, so there's no need for editing configuration files or jumping into the command line. For you as a developer, this simply means populating this key in the manifest! - -Formatting ----------- - -- Serialization language:``.yaml`` -- Case style: ``kebab-case`` - -Type definitions ----------------- - -Below are the types and sub-type definitions, with necessary elaborations. Any item that contains ``Option<>`` is an optional field. - -.. code:: yaml - - # manifest version compatibility ie. v0 (this is currently the only option) - compat: v0 - # service id, used for uniqueness and BE identification ie. bitcoind - id: String - # version number of the release conforming to the semantic versioning scheme - version: String - # display name of service - title: String - # an object containing a short and long description of the service. TODO character lengths - description: - short: String - long: String - # a link to the release tag notes in GitHub, or a short description TODO character length - release-notes: String - # a notification message that should caution the user with any service particularities, eg. beta tech - install-alert: Option - # a notification message warning users of potential problems with uninstalling, such as dependency failures or data loss - uninstall-alert: Option - # a notification message containing warning or details about backup restoration - restore-alert: Option - # denoting the existence of instructions.md - has-instructions: Boolean - # required semver version range of EmbassyOS to run service eg. ">=1.0.0" - os-version-required: VersionReq - # recommended semver version range of EmbassyOS to run service eg."^1.0.0" - os-version-recommended: VersionReq - # a list of objects of ports to run the service on localhost and tor - ports: - - internal: String - tor: String - # currently only tar is supported - image: - type: String - # shared memory container size - shm-size-mb: Option - # path to mount the image on the volume, ie: /root/bitcoind - mount: String - # read only data exposed to dependencies (path is relevant to mount) - public: Option - # shared filesystem segment with each of your dependencies (path is relevant to mount) - shared: Option - # deprecated - will default to an empty vector - assets: [] - # version of tor support, eg. v2, v3 - hidden-service-version: String - # A map of dependent services, see below for more details - dependencies: Dependencies - -.. _dependencies-spec: - -Dependencies ------------- - -Many services depend on other libraries and services on EmbassyOS (such as Bitcoin), sometimes even a particular version of those services, which need to be specified by the developers so that EmbassyOS can handle installing these dependencies under the hood. - -The key of each field in the dependencies object is the lowercase, kebab-case app ID of the service that is depended on. Each dependency contains a set of rules that need to be fulfilled as true if the dependency is to be properly installed. The "config rules" here are for auto-configuring dependencies - the action defined by the rule will be executed if the service is auto configured with defaults during initial setup. This simplifies and streamlines the user experience. The interface should provide suggestions for the behavior if the denoted rule cannot be met with previous configurations. - -Let's take this snippet for example: - -.. code:: yaml - - ... - dependencies: - btc-rpc-proxy: - version: "^0.1.0" - optional: Can configure an external bitcoin node. - description: Required for fetching validated blocks. - config: - - rule: '''users.*.name = "c-lightning"' - description: 'Must have an RPC user named "c-lightning"' - suggestions: - - PUSH: - to: 'users' - value: - name: c-lightning - ... - -.. role:: raw-html(raw) - :format: html - -:raw-html:`
` - -The service ``btc-rpc-proxy`` is a dependency of the service ``c-lightning``. ``c-lightning`` requires it to be installed at a version >=0.1.0 <0.2.0. There exists a rule that states the config option ``user.name`` must be equal to "c-lightning". If this value does not exist for ``user.name`` when accessed, ``PUSH`` the value "c-lighting" to the field. This all takes place during the initial service configuration, before the service is started for the first time. - -.. note:: - Dependency config rules are processed in order. - -Type definitions -================ - -Types for ``manifest.yaml`` fields: - -.. code:: typescript - - interface Dependencies [{ - serviceId: DepInfo - }] - - interface DepInfo { - version: String // ie. 0.11.1.1 - optional?: String, - description?: String, - config: [ConfigRule], - ], - } - - interface ConfigRule { - rule: String, // ie. 'users.*.name = "lnd" - description: String, - suggestions: [SuggestionVariant] - } - - interface SuggestionVariant { - SET: { - var: String, - to: SetVariant, - }, - DELETE: { - src: String, - }, - PUSH: { - to: String, - value: Value, - }, - } - - interface SetVariant { - to: Option, - to-value: Option, // ie. true/false - to-entropy: Option<{ - charset: String // ie. 'a-z,A-Z,0-9' - len: number - }> - } - -.. role:: raw-html(raw) - :format: html - -:raw-html:`
` - -Examples -======== - -Actual ``manifest.yaml`` files for existing services: - -LND -... - -.. code:: yaml - - compat: v0 - id: lnd - version: 0.11.1.1 - title: Lightning Network Daemon - description: - short: "A complete implementation of a Lightning Network node by Lightning Labs" - long: "LND fully conforms to the Lightning Network specification (BOLTs). BOLT stands for: Basis of Lightning Technology. In the current state lnd is capable of: creating channels, closing channels, managing all channel states (including the exceptional ones!), maintaining a fully authenticated+validated channel graph, performing path finding within the network, passively forwarding incoming payments, sending outgoing onion-encrypted payments through the network, updating advertised fee schedules, and automatic channel management (autopilot)." - release-notes: https://github.com/lightningnetwork/lnd/releases/tag/v0.11.1-beta - ports: - - internal: 8080 - tor: 8080 - - internal: 9735 - tor: 9735 - - internal: 9911 - tor: 9911 - - internal: 10009 - tor: 10009 - image: - type: tar - mount: /root/.lnd - public: public - has-instructions: true - os-version-required: ">=0.2.8" - os-version-recommended: ">=0.2.8" - install-alert: | - READ CAREFULLY! LND and the Lightning Network are considered beta software. Please use with caution and do not risk more money than you are willing to lose. We encourage frequent backups. If for any reason, you need to restore LND from a backup, your on-chain wallet will be restored, but all your channels will be closed and their funds returned to your on-chain wallet, minus fees. It may also take some time for this process to occur. - uninstall-alert: "READ CAREFULLY! Uninstalling LND will result in permanent loss of data, including its private keys for its on-chain wallet and all channel states. Please make a backup if you have any funds in your on-chain wallet or in any channels. Recovering from backup will restore your on-chain wallet, but due to the architecture of the Lightning Network, your channels cannot be recovered. All your channels will be closed and their funds returned to your on-chain wallet, minus fees. \n" - restore-alert: | - Restoring LND will overwrite its current data, including its on-chain wallet and channels. Any channels opened since the last backup will be forgotten and may linger indefinitely, and channels contained in the backup will be closed and their funds returned to your on-chain wallet, minus fees. - assets: [] - hidden-service-version: v3 - dependencies: - btc-rpc-proxy: - version: "^0.2.4" - optional: Can alternatively configure an external bitcoin node. - description: Used to fetch validated blocks. - config: - - rule: '''users.*.name = "lnd"' - description: 'Must have an RPC user named "lnd"' - suggestions: - - PUSH: - to: "users" - value: - name: lnd - allowed-calls: [] - - SET: - var: 'users.[first(item => ''item.name = "lnd")].password' - to-entropy: - charset: "a-z,A-Z,0-9" - len: 22 - - rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getinfo"' - description: 'RPC user "lnd" must have "getinfo" enabled' - suggestions: - - PUSH: - to: 'users.[first(item => ''item.name = "lnd")].allowed-calls' - value: "getinfo" - - rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getbestblockhash"' - description: 'RPC user "lnd" must have "getbestblockhash" enabled' - suggestions: - - PUSH: - to: 'users.[first(item => ''item.name = "lnd")].allowed-calls' - value: "getbestblockhash" - - rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "gettxout"' - description: 'RPC user "lnd" must have "gettxout" enabled' - suggestions: - - PUSH: - to: 'users.[first(item => ''item.name = "lnd")].allowed-calls' - value: "gettxout" - - rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getblockchaininfo"' - description: 'RPC user "lnd" must have "getblockchaininfo" enabled' - suggestions: - - PUSH: - to: 'users.[first(item => ''item.name = "lnd")].allowed-calls' - value: "getblockchaininfo" - - rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "sendrawtransaction"' - description: 'RPC user "lnd" must have "sendrawtransaction" enabled' - suggestions: - - PUSH: - to: 'users.[first(item => ''item.name = "lnd")].allowed-calls' - value: "sendrawtransaction" - - rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getblockhash"' - description: 'RPC user "lnd" must have "getblockhash" enabled' - suggestions: - - PUSH: - to: 'users.[first(item => ''item.name = "lnd")].allowed-calls' - value: "getblockhash" - - rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getblock"' - description: 'RPC user "lnd" must have "getblock" enabled' - suggestions: - - PUSH: - to: 'users.[first(item => ''item.name = "lnd")].allowed-calls' - value: "getblock" - - rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getblockheader"' - description: 'RPC user "lnd" must have "getblockheader" enabled' - suggestions: - - PUSH: - to: 'users.[first(item => ''item.name = "lnd")].allowed-calls' - value: "getblockheader" - - rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "estimatesmartfee"' - description: 'RPC user "lnd" must have "estimatesmartfee" enabled' - suggestions: - - PUSH: - to: 'users.[first(item => ''item.name = "lnd")].allowed-calls' - value: "estimatesmartfee" - - rule: '''users.[first(item => ''item.name = "lnd")].allowed-calls.* = "getnetworkinfo"' - description: 'RPC user "lnd" must have "getnetworkinfo" enabled' - suggestions: - - PUSH: - to: 'users.[first(item => ''item.name = "lnd")].allowed-calls' - value: "getnetworkinfo" - - rule: 'users.[first(item => ''item.name = "lnd")].fetch-blocks?' - description: 'RPC user "lnd" must have "Fetch Blocks" enabled' - suggestions: - - SET: - var: 'users.[first(item => ''item.name = "lnd")].fetch-blocks' - to-value: true - bitcoind: - version: "^0.21.0" - optional: Can alternatively configure an external bitcoin node. - description: Used to subscribe to new block events. - config: - - rule: "zmq-enabled?" - description: "Must have an ZeroMQ enabled" - suggestions: - - SET: - var: "zmq-enabled" - to-value: true - -Cups -.... - -.. code:: yaml - - compat: v0 - id: cups - version: "0.3.6" - title: "Cups Messenger" - description: - short: "Real private messaging" - long: "Cups is a private, self-hosted, peer-to-peer, Tor-based, instant messenger. Unlike other end-to-end encrypted messengers, with Cups on the Embassy there are no trusted third parties." - release-notes: | - Features - - Adds instructions defined by EmbassyOS 0.2.4 instructions feature - ports: - - internal: 59001 - tor: 59001 - - internal: 80 - tor: 80 - image: - type: tar - mount: /root - has-instructions: true - os-version-required: ">=0.2.4" - os-version-recommended: ">=0.2.4" - assets: - - src: httpd.conf - dst: "." - overwrite: true - - src: www - dst: "." - overwrite: true - hidden-service-version: v3 - -.. role:: raw-html(raw) - :format: html - -:raw-html:`
` diff --git a/site/source/support/dev-docs/dev-faq/faq-contributing.rst b/site/source/support/faq/faq-contributing.rst similarity index 59% rename from site/source/support/dev-docs/dev-faq/faq-contributing.rst rename to site/source/support/faq/faq-contributing.rst index ab7647c..623244d 100644 --- a/site/source/support/dev-docs/dev-faq/faq-contributing.rst +++ b/site/source/support/faq/faq-contributing.rst @@ -11,16 +11,3 @@ Please follow the guide `here ` us for ideas if you're not sure where to begin. - - - -.. Expound the following into FAQs: -.. -------------------------------- - -.. - Release notes need to be in quotations? As it is a string? (they aren’t by default) – this is a yaml thing and SOMETIMES they are required and SOMETIMES NOT -.. - Alerts in LND example are not wrapped in quotes, same with CUPS release notes – yaml thing -.. - Ports in manifest – can leave as default (80), as it is a docker container?? - Sure - -.. - Type definitions – is this just for dependencies? Can a blurb be added here for extra clarity? -.. - Config spec – Admin pass? - diff --git a/site/source/support/dev-docs/dev-faq/faq-service-packaging.rst b/site/source/support/faq/faq-service-packaging.rst similarity index 81% rename from site/source/support/dev-docs/dev-faq/faq-service-packaging.rst rename to site/source/support/faq/faq-service-packaging.rst index 3212e29..65bba38 100644 --- a/site/source/support/dev-docs/dev-faq/faq-service-packaging.rst +++ b/site/source/support/faq/faq-service-packaging.rst @@ -39,10 +39,6 @@ How can my service access the data directory of another service? Check out `LND `_ and `RTL `_ as an example. -The guide says that a Makefile is optional, why is this? --------------------------------------------------------- -The ``Makefile`` simplifies the development process by putting all your build steps into one place so that you can simply use the ``make`` to build with. A ``Makefile`` is not required however, and you may instead choose to use something like ``nix``, ``bash``, ``perl``, ``python``, or ``ruby`` for your build orchestration. - Why am I getting the error "No rule to make target yq, needed by manifest.yaml?" -------------------------------------------------------------------------------- -This is the message you get when you're missing the executable on your ``$PATH``. +This is the message you get when you're missing the yq executable on your ``$PATH``. You can find details to `install yq here `_.` \ No newline at end of file