Feature/new registry (#2612)

* wip

* overhaul boot process

* wip: new registry

* wip

* wip

* wip

* wip

* wip

* wip

* os registry complete

* ui fixes

* fixes

* fixes

* more fixes

* fix merkle archive
This commit is contained in:
Aiden McClelland
2024-05-06 10:20:44 -06:00
committed by GitHub
parent 8a38666105
commit 9b14d714ca
167 changed files with 6297 additions and 3190 deletions

View File

@@ -1,107 +0,0 @@
# Building StartOS
⚠️ The commands given assume a Debian or Ubuntu-based environment. _Building in
a VM is NOT yet supported_ ⚠️
## Prerequisites
1. Install dependencies
- Avahi
- `sudo apt install -y avahi-daemon`
- Installed by default on most Debian systems - https://avahi.org
- Build Essentials (needed to run `make`)
- `sudo apt install -y build-essential`
- Docker
- `curl -fsSL https://get.docker.com | sh`
- https://docs.docker.com/get-docker
- Add your user to the docker group: `sudo usermod -a -G docker $USER`
- Reload user environment `exec sudo su -l $USER`
- Prepare Docker environment
- Setup buildx (https://docs.docker.com/buildx/working-with-buildx/)
- Create a builder: `docker buildx create --use`
- Add multi-arch build ability:
`docker run --rm --privileged linuxkit/binfmt:v0.8`
- Node Version 12+
- snap: `sudo snap install node`
- [nvm](https://github.com/nvm-sh/nvm#installing-and-updating):
`nvm install --lts`
- https://nodejs.org/en/docs
- NPM Version 7+
- apt: `sudo apt install -y npm`
- [nvm](https://github.com/nvm-sh/nvm#installing-and-updating):
`nvm install --lts`
- https://docs.npmjs.com/downloading-and-installing-node-js-and-npm
- jq
- `sudo apt install -y jq`
- https://stedolan.github.io/jq
- yq
- snap: `sudo snap install yq`
- binaries: https://github.com/mikefarah/yq/releases/
- https://mikefarah.gitbook.io/yq
2. Clone the latest repo with required submodules
> :information_source: You chan check latest available version
> [here](https://github.com/Start9Labs/start-os/releases)
```
git clone --recursive https://github.com/Start9Labs/start-os.git --branch latest
```
## Build Raspberry Pi Image
```
cd start-os
make embassyos-raspi.img ARCH=aarch64
```
## Flash
Flash the resulting `embassyos-raspi.img` to your SD Card
We recommend [Balena Etcher](https://www.balena.io/etcher/)
## Setup
Visit http://start.local from any web browser - We recommend
[Firefox](https://www.mozilla.org/firefox/browsers)
Enter your product key. This is generated during the build process and can be
found in `product_key.txt`, located in the root directory.
## Troubleshooting
1. I just flashed my SD card, fired up StartOS, bootup sounds and all, but my
browser is saying "Unable to connect" with start.local.
- Try doing a hard refresh on your browser, or opening the url in a
private/incognito window. If you've ran an instance of StartOS before,
sometimes you can have a stale cache that will block you from navigating to
the page.
2. Flashing the image isn't working with balenaEtcher. I'm getting
`Cannot read property 'message' of null` when I try.
- The latest versions of Balena may not flash properly. This version here:
https://github.com/balena-io/etcher/releases/tag/v1.5.122 should work
properly.
3. Startup isn't working properly and I'm curious as to why. How can I view logs
regarding startup for debugging?
- Find the IP of your device
- Run `nc <ip> 8080` and it will print the logs
4. I need to ssh into my server to fix something, but I cannot get to the
console to add ssh keys normally.
- During the Build step, instead of running just
`make embassyos-raspi.img ARCH=aarch64` run
`ENVIRONMENT=dev make embassyos-raspi.img ARCH=aarch64`. Flash like normal,
and insert into your server. Boot up StartOS, then on another computer on
the same network, ssh into the the server with the username `start9` password
`embassy`.
4. I need to reset my password, how can I do that?
- You will need to reflash your device. Select "Use Existing Drive" once you are
in setup, and it will prompt you to set a new password.

View File

@@ -1,76 +0,0 @@
# Release Process
## `embassyos_0.3.x-1_amd64.deb`
- Description: debian package for x86_64 - intended to be installed on pureos
- Destination: GitHub Release Tag
- Requires: N/A
- Build steps:
- Clone `https://github.com/Start9Labs/embassy-os-deb` at `master`
- Run `make TAG=master` from that folder
- Artifact: `./embassyos_0.3.x-1_amd64.deb`
## `eos-<version>-<git hash>-<date>_amd64.iso`
- Description: live usb image for x86_64
- Destination: GitHub Release Tag
- Requires: `embassyos_0.3.x-1_amd64.deb`
- Build steps:
- Clone `https://github.com/Start9Labs/eos-image-recipes` at `master`
- Copy `embassyos_0.3.x-1_amd64.deb` to
`overlays/vendor/root/embassyos_0.3.x-1_amd64.deb`
- Run `./run-local-build.sh byzantium` from that folder
- Artifact: `./results/eos-<version>-<git hash>-<date>_amd64.iso`
## `eos.x86_64.squashfs`
- Description: compressed embassyOS x86_64 filesystem image
- Destination: GitHub Release Tag, Registry @
`resources/eos/<version>/eos.x86_64.squashfs`
- Requires: `eos-<version>-<git hash>-<date>_amd64.iso`
- Build steps:
- From `https://github.com/Start9Labs/eos-image-recipes` at `master`
- `./extract-squashfs.sh results/eos-<version>-<git hash>-<date>_amd64.iso`
- Artifact: `./results/eos.x86_64.squashfs`
## `eos.raspberrypi.squashfs`
- Description: compressed embassyOS raspberrypi filesystem image
- Destination: GitHub Release Tag, Registry @
`resources/eos/<version>/eos.raspberrypi.squashfs`
- Requires: N/A
- Build steps:
- Clone `https://github.com/Start9Labs/embassy-os` at `master`
- `make embassyos-raspi.img`
- flash `embassyos-raspi.img` to raspberry pi
- boot raspberry pi with ethernet
- wait for chime
- you can watch logs using `nc <ip> 8080`
- unplug raspberry pi, put sd card back in build machine
- `./build/raspberry-pi/rip-image.sh`
- Artifact: `./eos.raspberrypi.squashfs`
## `lite-upgrade.img`
- Description: update image for users coming from 0.3.2.1 and before
- Destination: Registry @ `resources/eos/<version>/eos.img`
- Requires: `eos.raspberrypi.squashfs`
- Build steps:
- From `https://github.com/Start9Labs/embassy-os` at `master`
- `make lite-upgrade.img`
- Artifact `./lite-upgrade.img`
## `eos-<version>-<git hash>-<date>_raspberrypi.tar.gz`
- Description: pre-initialized raspberrypi image
- Destination: GitHub Release Tag (as tar.gz)
- Requires: `eos.raspberrypi.squashfs`
- Build steps:
- From `https://github.com/Start9Labs/embassy-os` at `master`
- `make eos_raspberrypi.img`
- `tar --format=posix -cS -f- eos-<version>-<git hash>-<date>_raspberrypi.img | gzip > eos-<version>-<git hash>-<date>_raspberrypi.tar.gz`
- Artifact `./eos-<version>-<git hash>-<date>_raspberrypi.tar.gz`
## `embassy-sdk`
- Build and deploy to all registries

View File

@@ -34,7 +34,6 @@ network-manager
nvme-cli
nyx
openssh-server
podman
postgresql
psmisc
qemu-guest-agent

View File

@@ -5,44 +5,104 @@ if [ "$UID" -ne 0 ]; then
exit 1
fi
POSITIONAL_ARGS=()
while [[ $# -gt 0 ]]; do
case $1 in
--no-sync)
NO_SYNC=1
shift
;;
--create)
ONLY_CREATE=1
shift
;;
-*|--*)
echo "Unknown option $1"
exit 1
;;
*)
POSITIONAL_ARGS+=("$1") # save positional arg
shift # past argument
;;
esac
done
set -- "${POSITIONAL_ARGS[@]}" # restore positional parameters
if [ -z "$NO_SYNC" ]; then
echo 'Syncing...'
rsync -a --delete --force --info=progress2 /media/embassy/embassyfs/current/ /media/embassy/next
umount -R /media/startos/next 2> /dev/null
rm -rf /media/startos/upper /media/startos/next
mkdir /media/startos/upper
mount -t tmpfs tmpfs /media/startos/upper
mkdir -p /media/startos/upper/data /media/startos/upper/work /media/startos/next
mount -t overlay \
-olowerdir=/media/startos/current,upperdir=/media/startos/upper/data,workdir=/media/startos/upper/work \
overlay /media/startos/next
fi
mkdir -p /media/embassy/next/run
mkdir -p /media/embassy/next/dev
mkdir -p /media/embassy/next/sys
mkdir -p /media/embassy/next/proc
mkdir -p /media/embassy/next/boot
mount --bind /run /media/embassy/next/run
mount --bind /tmp /media/embassy/next/tmp
mount --bind /dev /media/embassy/next/dev
mount --bind /sys /media/embassy/next/sys
mount --bind /proc /media/embassy/next/proc
mount --bind /boot /media/embassy/next/boot
if [ -n "$ONLY_CREATE" ]; then
exit 0
fi
mkdir -p /media/startos/next/run
mkdir -p /media/startos/next/dev
mkdir -p /media/startos/next/sys
mkdir -p /media/startos/next/proc
mkdir -p /media/startos/next/boot
mount --bind /run /media/startos/next/run
mount --bind /tmp /media/startos/next/tmp
mount --bind /dev /media/startos/next/dev
mount --bind /sys /media/startos/next/sys
mount --bind /proc /media/startos/next/proc
mount --bind /boot /media/startos/next/boot
if [ -z "$*" ]; then
chroot /media/embassy/next
chroot /media/startos/next
CHROOT_RES=$?
else
chroot /media/embassy/next "$SHELL" -c "$*"
chroot /media/startos/next "$SHELL" -c "$*"
CHROOT_RES=$?
fi
umount /media/embassy/next/run
umount /media/embassy/next/tmp
umount /media/embassy/next/dev
umount /media/embassy/next/sys
umount /media/embassy/next/proc
umount /media/embassy/next/boot
umount /media/startos/next/run
umount /media/startos/next/tmp
umount /media/startos/next/dev
umount /media/startos/next/sys
umount /media/startos/next/proc
umount /media/startos/next/boot
if [ "$CHROOT_RES" -eq 0 ]; then
if [ -h /media/startos/config/current.rootfs ] && [ -e /media/startos/config/current.rootfs ]; then
echo 'Pruning...'
current="$(readlink -f /media/startos/config/current.rootfs)"
needed=$(du -s --bytes /media/startos/next | awk '{print $1}')
while [[ "$(df -B1 --output=avail --sync /media/startos/images | tail -n1)" -lt "$needed" ]]; do
to_prune="$(ls -t1 /media/startos/images/*.rootfs | grep -v "$current" | tail -n1)"
if [ -e "$to_prune" ]; then
echo " Pruning $to_prune"
rm -rf "$to_prune"
else
>&2 echo "Not enough space and nothing to prune!"
exit 1
fi
done
echo 'done.'
fi
echo 'Upgrading...'
touch /media/embassy/config/upgrade
time mksquashfs /media/startos/next /media/startos/images/next.squashfs -b 4096 -comp gzip
hash=$(start-cli util b3sum /media/startos/images/next.squashfs | head -c 32)
mv /media/startos/images/next.squashfs /media/startos/images/${hash}.rootfs
ln -rsf /media/startos/images/${hash}.rootfs /media/startos/config/current.rootfs
sync
reboot
fi
fi
umount -R /media/startos/next
rm -rf /media/startos/upper /media/startos/next

View File

@@ -1,98 +0,0 @@
# Local filesystem mounting -*- shell-script -*-
#
# This script overrides local_mount_root() in /scripts/local
# and mounts root as a read-only filesystem with a temporary (rw)
# overlay filesystem.
#
. /scripts/local
local_mount_root()
{
echo 'using embassy initramfs module'
local_top
local_device_setup "${ROOT}" "root file system"
ROOT="${DEV}"
# Get the root filesystem type if not set
if [ -z "${ROOTFSTYPE}" ]; then
FSTYPE=$(get_fstype "${ROOT}")
else
FSTYPE=${ROOTFSTYPE}
fi
local_premount
# CHANGES TO THE ORIGINAL FUNCTION BEGIN HERE
# N.B. this code still lacks error checking
modprobe ${FSTYPE}
checkfs ${ROOT} root "${FSTYPE}"
ROOTFLAGS="$(echo "${ROOTFLAGS}" | sed 's/subvol=\(next\|current\)//' | sed 's/^-o *$//')"
if [ "${FSTYPE}" != "unknown" ]; then
mount -t ${FSTYPE} ${ROOTFLAGS} ${ROOT} ${rootmnt}
else
mount ${ROOTFLAGS} ${ROOT} ${rootmnt}
fi
echo 'mounting embassyfs'
mkdir /embassyfs
mount --move ${rootmnt} /embassyfs
if ! [ -d /embassyfs/current ] && [ -d /embassyfs/prev ]; then
mv /embassyfs/prev /embassyfs/current
fi
if ! [ -d /embassyfs/current ]; then
mkdir /embassyfs/current
for FILE in $(ls /embassyfs); do
if [ "$FILE" != current ]; then
mv /embassyfs/$FILE /embassyfs/current/
fi
done
fi
mkdir -p /embassyfs/config
if [ -f /embassyfs/config/upgrade ] && [ -d /embassyfs/next ]; then
mv /embassyfs/current /embassyfs/prev
mv /embassyfs/next /embassyfs/current
rm /embassyfs/config/upgrade
fi
if ! [ -d /embassyfs/next ]; then
if [ -d /embassyfs/prev ]; then
mv /embassyfs/prev /embassyfs/next
else
mkdir /embassyfs/next
fi
fi
mkdir /lower /upper
mount -r --bind /embassyfs/current /lower
modprobe overlay || insmod "/lower/lib/modules/$(uname -r)/kernel/fs/overlayfs/overlay.ko"
# Mount a tmpfs for the overlay in /upper
mount -t tmpfs tmpfs /upper
mkdir /upper/data /upper/work
# Mount the final overlay-root in $rootmnt
mount -t overlay \
-olowerdir=/lower,upperdir=/upper/data,workdir=/upper/work \
overlay ${rootmnt}
mkdir -p ${rootmnt}/media/embassy/config
mount --bind /embassyfs/config ${rootmnt}/media/embassy/config
mkdir -p ${rootmnt}/media/embassy/next
mount --bind /embassyfs/next ${rootmnt}/media/embassy/next
mkdir -p ${rootmnt}/media/embassy/embassyfs
mount -r --bind /embassyfs ${rootmnt}/media/embassy/embassyfs
}

View File

@@ -3,8 +3,8 @@
ARGS=
for ARG in $@; do
if [ -d "/media/embassy/embassyfs" ] && [ "$ARG" = "/" ]; then
ARG=/media/embassy/embassyfs
if [ -d "/media/startos/root" ] && [ "$ARG" = "/" ]; then
ARG=/media/startos/root
fi
ARGS="$ARGS $ARG"
done

View File

@@ -0,0 +1,114 @@
# Local filesystem mounting -*- shell-script -*-
#
# This script overrides local_mount_root() in /scripts/local
# and mounts root as a read-only filesystem with a temporary (rw)
# overlay filesystem.
#
. /scripts/local
local_mount_root()
{
echo 'using startos initramfs module'
local_top
local_device_setup "${ROOT}" "root file system"
ROOT="${DEV}"
# Get the root filesystem type if not set
if [ -z "${ROOTFSTYPE}" ]; then
FSTYPE=$(get_fstype "${ROOT}")
else
FSTYPE=${ROOTFSTYPE}
fi
local_premount
# CHANGES TO THE ORIGINAL FUNCTION BEGIN HERE
# N.B. this code still lacks error checking
modprobe ${FSTYPE}
checkfs ${ROOT} root "${FSTYPE}"
echo 'mounting startos'
mkdir /startos
ROOTFLAGS="$(echo "${ROOTFLAGS}" | sed 's/subvol=\(next\|current\)//' | sed 's/^-o *$//')"
if [ "${FSTYPE}" != "unknown" ]; then
mount -t ${FSTYPE} ${ROOTFLAGS} ${ROOT} /startos
else
mount ${ROOTFLAGS} ${ROOT} /startos
fi
if [ -d /startos/images ]; then
if [ -h /startos/config/current.rootfs ] && [ -e /startos/config/current.rootfs ]; then
image=$(readlink -f /startos/config/current.rootfs)
else
image="$(ls -t1 /startos/images/*.rootfs | head -n1)"
fi
if ! [ -f "$image" ]; then
>&2 echo "image $image not available to boot"
exit 1
fi
else
if [ -f /startos/config/upgrade ] && [ -d /startos/next ]; then
oldroot=/startos/next
elif [ -d /startos/current ]; then
oldroot=/startos/current
elif [ -d /startos/prev ]; then
oldroot=/startos/prev
else
>&2 echo no StartOS filesystem found
exit 1
fi
mkdir -p /startos/config/overlay/etc
mv $oldroot/etc/fstab /startos/config/overlay/etc/fstab
mv $oldroot/etc/machine-id /startos/config/overlay/etc/machine-id
mv $oldroot/etc/ssh /startos/config/overlay/etc/ssh
mkdir -p /startos/images
mv $oldroot /startos/images/legacy.rootfs
rm -rf /startos/next /startos/current /startos/prev
ln -rsf /startos/images/old.squashfs /startos/config/current.rootfs
image=$(readlink -f /startos/config/current.rootfs)
fi
mkdir /lower /upper
if [ -d "$image" ]; then
mount -r --bind $image /lower
elif [ -f "$image" ]; then
modprobe squashfs
mount -r $image /lower
else
>&2 echo "not a regular file or directory: $image"
exit 1
fi
modprobe overlay || insmod "/lower/lib/modules/$(uname -r)/kernel/fs/overlayfs/overlay.ko"
# Mount a tmpfs for the overlay in /upper
mount -t tmpfs tmpfs /upper
mkdir /upper/data /upper/work
mkdir -p /startos/config/overlay
# Mount the final overlay-root in $rootmnt
mount -t overlay \
-olowerdir=/startos/config/overlay:/lower,upperdir=/upper/data,workdir=/upper/work \
overlay ${rootmnt}
mkdir -p ${rootmnt}/media/startos/config
mount --bind /startos/config ${rootmnt}/media/startos/config
mkdir -p ${rootmnt}/media/startos/images
mount --bind /startos/images ${rootmnt}/media/startos/images
mkdir -p ${rootmnt}/media/startos/root
mount -r --bind /startos ${rootmnt}/media/startos/root
mkdir -p ${rootmnt}/media/startos/current
mount -r --bind /lower ${rootmnt}/media/startos/current
}

View File

@@ -63,7 +63,7 @@ sudo unsquashfs -f -d $TMPDIR startos.raspberrypi.squashfs
REAL_GIT_HASH=$(cat $TMPDIR/usr/lib/startos/GIT_HASH.txt)
REAL_VERSION=$(cat $TMPDIR/usr/lib/startos/VERSION.txt)
REAL_ENVIRONMENT=$(cat $TMPDIR/usr/lib/startos/ENVIRONMENT.txt)
sudo sed -i 's| boot=embassy| init=/usr/lib/startos/scripts/init_resize\.sh|' $TMPDIR/boot/cmdline.txt
sudo sed -i 's| boot=startos| init=/usr/lib/startos/scripts/init_resize\.sh|' $TMPDIR/boot/cmdline.txt
sudo cp ./build/raspberrypi/fstab $TMPDIR/etc/
sudo cp ./build/raspberrypi/init_resize.sh $TMPDIR/usr/lib/startos/scripts/init_resize.sh
sudo umount $TMPDIR/boot