mirror of
https://github.com/Start9Labs/start-os.git
synced 2026-04-04 22:39:46 +00:00
Feature/lxc container runtime (#2514)
* wip: static-server errors * wip: fix wifi * wip: Fix the service_effects * wip: Fix cors in the middleware * wip(chore): Auth clean up the lint. * wip(fix): Vhost * wip: continue manager refactor Co-authored-by: J H <Blu-J@users.noreply.github.com> * wip: service manager refactor * wip: Some fixes * wip(fix): Fix the lib.rs * wip * wip(fix): Logs * wip: bins * wip(innspect): Add in the inspect * wip: config * wip(fix): Diagnostic * wip(fix): Dependencies * wip: context * wip(fix) Sorta auth * wip: warnings * wip(fix): registry/admin * wip(fix) marketplace * wip(fix) Some more converted and fixed with the linter and config * wip: Working on the static server * wip(fix)static server * wip: Remove some asynnc * wip: Something about the request and regular rpc * wip: gut install Co-authored-by: J H <Blu-J@users.noreply.github.com> * wip: Convert the static server into the new system * wip delete file * test * wip(fix) vhost does not need the with safe defaults * wip: Adding in the wifi * wip: Fix the developer and the verify * wip: new install flow Co-authored-by: J H <Blu-J@users.noreply.github.com> * fix middleware * wip * wip: Fix the auth * wip * continue service refactor * feature: Service get_config * feat: Action * wip: Fighting the great fight against the borrow checker * wip: Remove an error in a file that I just need to deel with later * chore: Add in some more lifetime stuff to the services * wip: Install fix on lifetime * cleanup * wip: Deal with the borrow later * more cleanup * resolve borrowchecker errors * wip(feat): add in the handler for the socket, for now * wip(feat): Update the service_effect_handler::action * chore: Add in the changes to make sure the from_service goes to context * chore: Change the * refactor service map * fix references to service map * fill out restore * wip: Before I work on the store stuff * fix backup module * handle some warnings * feat: add in the ui components on the rust side * feature: Update the procedures * chore: Update the js side of the main and a few of the others * chore: Update the rpc listener to match the persistant container * wip: Working on updating some things to have a better name * wip(feat): Try and get the rpc to return the correct shape? * lxc wip * wip(feat): Try and get the rpc to return the correct shape? * build for container runtime wip * remove container-init * fix build * fix error * chore: Update to work I suppose * lxc wip * remove docker module and feature * download alpine squashfs automatically * overlays effect Co-authored-by: Jade <Blu-J@users.noreply.github.com> * chore: Add the overlay effect * feat: Add the mounter in the main * chore: Convert to use the mounts, still need to work with the sandbox * install fixes * fix ssl * fixes from testing * implement tmpfile for upload * wip * misc fixes * cleanup * cleanup * better progress reporting * progress for sideload * return real guid * add devmode script * fix lxc rootfs path * fix percentage bar * fix progress bar styling * fix build for unstable * tweaks * label progress * tweaks * update progress more often * make symlink in rpc_client * make socket dir * fix parent path * add start-cli to container * add echo and gitInfo commands * wip: Add the init + errors * chore: Add in the exit effect for the system * chore: Change the type to null for failure to parse * move sigterm timeout to stopping status * update order * chore: Update the return type * remove dbg * change the map error * chore: Update the thing to capture id * chore add some life changes * chore: Update the loging * chore: Update the package to run module * us From for RpcError * chore: Update to use import instead * chore: update * chore: Use require for the backup * fix a default * update the type that is wrong * chore: Update the type of the manifest * chore: Update to make null * only symlink if not exists * get rid of double result * better debug info for ErrorCollection * chore: Update effects * chore: fix * mount assets and volumes * add exec instead of spawn * fix mounting in image * fix overlay mounts Co-authored-by: Jade <Blu-J@users.noreply.github.com> * misc fixes * feat: Fix two * fix: systemForEmbassy main * chore: Fix small part of main loop * chore: Modify the bundle * merge * fixMain loop" * move tsc to makefile * chore: Update the return types of the health check * fix client * chore: Convert the todo to use tsmatches * add in the fixes for the seen and create the hack to allow demo * chore: Update to include the systemForStartOs * chore UPdate to the latest types from the expected outout * fixes * fix typo * Don't emit if failure on tsc * wip Co-authored-by: Jade <Blu-J@users.noreply.github.com> * add s9pk api * add inspection * add inspect manifest * newline after display serializable * fix squashfs in image name * edit manifest Co-authored-by: Jade <Blu-J@users.noreply.github.com> * wait for response on repl * ignore sig for now * ignore sig for now * re-enable sig verification * fix * wip * env and chroot * add profiling logs * set uid & gid in squashfs to 100000 * set uid of sqfs to 100000 * fix mksquashfs args * add env to compat * fix * re-add docker feature flag * fix docker output format being stupid * here be dragons * chore: Add in the cross compiling for something * fix npm link * extract logs from container on exit * chore: Update for testing * add log capture to drop trait * chore: add in the modifications that I make * chore: Update small things for no updates * chore: Update the types of something * chore: Make main not complain * idmapped mounts * idmapped volumes * re-enable kiosk * chore: Add in some logging for the new system * bring in start-sdk * remove avahi * chore: Update the deps * switch to musl * chore: Update the version of prettier * chore: Organize' * chore: Update some of the headers back to the standard of fetch * fix musl build * fix idmapped mounts * fix cross build * use cross compiler for correct arch * feat: Add in the faked ssl stuff for the effects * @dr_bonez Did a solution here * chore: Something that DrBonez * chore: up * wip: We have a working server!!! * wip * uninstall * wip * tes --------- Co-authored-by: J H <dragondef@gmail.com> Co-authored-by: J H <Blu-J@users.noreply.github.com> Co-authored-by: J H <2364004+Blu-J@users.noreply.github.com>
This commit is contained in:
Aiden McClelland
@@ -1,23 +1,178 @@
|
||||
use std::ffi::OsStr;
|
||||
use std::path::Path;
|
||||
use std::sync::Arc;
|
||||
|
||||
use imbl_value::InternedString;
|
||||
use models::{mime, DataUrl, PackageId};
|
||||
use tokio::fs::File;
|
||||
|
||||
use crate::prelude::*;
|
||||
use crate::s9pk::manifest::Manifest;
|
||||
use crate::s9pk::merkle_archive::file_contents::FileContents;
|
||||
use crate::s9pk::merkle_archive::sink::Sink;
|
||||
use crate::s9pk::merkle_archive::source::{ArchiveSource, FileSource, Section};
|
||||
use crate::s9pk::merkle_archive::MerkleArchive;
|
||||
use crate::s9pk::merkle_archive::source::multi_cursor_file::MultiCursorFile;
|
||||
use crate::s9pk::merkle_archive::source::{ArchiveSource, DynFileSource, FileSource, Section};
|
||||
use crate::s9pk::merkle_archive::{Entry, MerkleArchive};
|
||||
use crate::ARCH;
|
||||
|
||||
const MAGIC_AND_VERSION: &[u8] = &[0x3b, 0x3b, 0x02];
|
||||
|
||||
pub struct S9pk<S>(MerkleArchive<S>);
|
||||
pub mod compat;
|
||||
pub mod manifest;
|
||||
|
||||
/**
|
||||
/
|
||||
├── manifest.json
|
||||
├── icon.<ext>
|
||||
├── LICENSE.md
|
||||
├── instructions.md
|
||||
├── javascript.squashfs
|
||||
├── assets
|
||||
│ └── <id>.squashfs (xN)
|
||||
└── images
|
||||
└── <arch>
|
||||
├── <id>.env (xN)
|
||||
└── <id>.squashfs (xN)
|
||||
*/
|
||||
|
||||
fn priority(s: &str) -> Option<usize> {
|
||||
match s {
|
||||
"manifest.json" => Some(0),
|
||||
a if Path::new(a).file_stem() == Some(OsStr::new("icon")) => Some(1),
|
||||
"LICENSE.md" => Some(2),
|
||||
"instructions.md" => Some(3),
|
||||
"javascript.squashfs" => Some(4),
|
||||
"assets" => Some(5),
|
||||
"images" => Some(6),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn filter(p: &Path) -> bool {
|
||||
match p.iter().count() {
|
||||
1 if p.file_name() == Some(OsStr::new("manifest.json")) => true,
|
||||
1 if p.file_stem() == Some(OsStr::new("icon")) => true,
|
||||
1 if p.file_name() == Some(OsStr::new("LICENSE.md")) => true,
|
||||
1 if p.file_name() == Some(OsStr::new("instructions.md")) => true,
|
||||
1 if p.file_name() == Some(OsStr::new("javascript.squashfs")) => true,
|
||||
1 if p.file_name() == Some(OsStr::new("assets")) => true,
|
||||
1 if p.file_name() == Some(OsStr::new("images")) => true,
|
||||
2 if p.parent() == Some(Path::new("assets")) => {
|
||||
p.extension().map_or(false, |ext| ext == "squashfs")
|
||||
}
|
||||
2 if p.parent() == Some(Path::new("images")) => p.file_name() == Some(OsStr::new(&*ARCH)),
|
||||
3 if p.parent() == Some(&*Path::new("images").join(&*ARCH)) => p
|
||||
.extension()
|
||||
.map_or(false, |ext| ext == "squashfs" || ext == "env"),
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct S9pk<S = Section<MultiCursorFile>> {
|
||||
manifest: Manifest,
|
||||
manifest_dirty: bool,
|
||||
archive: MerkleArchive<S>,
|
||||
size: Option<u64>,
|
||||
}
|
||||
impl<S> S9pk<S> {
|
||||
pub fn as_manifest(&self) -> &Manifest {
|
||||
&self.manifest
|
||||
}
|
||||
pub fn as_manifest_mut(&mut self) -> &mut Manifest {
|
||||
self.manifest_dirty = true;
|
||||
&mut self.manifest
|
||||
}
|
||||
pub fn as_archive(&self) -> &MerkleArchive<S> {
|
||||
&self.archive
|
||||
}
|
||||
pub fn as_archive_mut(&mut self) -> &mut MerkleArchive<S> {
|
||||
&mut self.archive
|
||||
}
|
||||
pub fn size(&self) -> Option<u64> {
|
||||
self.size
|
||||
}
|
||||
}
|
||||
|
||||
impl<S: FileSource> S9pk<S> {
|
||||
pub async fn new(archive: MerkleArchive<S>, size: Option<u64>) -> Result<Self, Error> {
|
||||
let manifest = extract_manifest(&archive).await?;
|
||||
Ok(Self {
|
||||
manifest,
|
||||
manifest_dirty: false,
|
||||
archive,
|
||||
size,
|
||||
})
|
||||
}
|
||||
|
||||
pub async fn icon(&self) -> Result<(InternedString, FileContents<S>), Error> {
|
||||
let mut best_icon = None;
|
||||
for (path, icon) in self
|
||||
.archive
|
||||
.contents()
|
||||
.with_stem("icon")
|
||||
.filter(|(p, _)| {
|
||||
Path::new(&*p)
|
||||
.extension()
|
||||
.and_then(|e| e.to_str())
|
||||
.and_then(mime)
|
||||
.map_or(false, |e| e.starts_with("image/"))
|
||||
})
|
||||
.filter_map(|(k, v)| v.into_file().map(|f| (k, f)))
|
||||
{
|
||||
let size = icon.size().await?;
|
||||
best_icon = match best_icon {
|
||||
Some((s, a)) if s >= size => Some((s, a)),
|
||||
_ => Some((size, (path, icon))),
|
||||
};
|
||||
}
|
||||
best_icon
|
||||
.map(|(_, a)| a)
|
||||
.ok_or_else(|| Error::new(eyre!("no icon found in archive"), ErrorKind::ParseS9pk))
|
||||
}
|
||||
|
||||
pub async fn icon_data_url(&self) -> Result<DataUrl<'static>, Error> {
|
||||
let (name, contents) = self.icon().await?;
|
||||
let mime = Path::new(&*name)
|
||||
.extension()
|
||||
.and_then(|e| e.to_str())
|
||||
.and_then(mime)
|
||||
.unwrap_or("image/png");
|
||||
DataUrl::from_reader(mime, contents.reader().await?, Some(contents.size().await?)).await
|
||||
}
|
||||
|
||||
pub async fn serialize<W: Sink>(&mut self, w: &mut W, verify: bool) -> Result<(), Error> {
|
||||
use tokio::io::AsyncWriteExt;
|
||||
|
||||
w.write_all(MAGIC_AND_VERSION).await?;
|
||||
self.0.serialize(w, verify).await?;
|
||||
if !self.manifest_dirty {
|
||||
self.archive.serialize(w, verify).await?;
|
||||
} else {
|
||||
let mut dyn_s9pk = self.clone().into_dyn();
|
||||
dyn_s9pk.as_archive_mut().contents_mut().insert_path(
|
||||
"manifest.json",
|
||||
Entry::file(DynFileSource::new(Arc::<[u8]>::from(
|
||||
serde_json::to_vec(&self.manifest).with_kind(ErrorKind::Serialization)?,
|
||||
))),
|
||||
)?;
|
||||
dyn_s9pk.archive.serialize(w, verify).await?;
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn into_dyn(self) -> S9pk<DynFileSource> {
|
||||
S9pk {
|
||||
manifest: self.manifest,
|
||||
manifest_dirty: self.manifest_dirty,
|
||||
archive: self.archive.into_dyn(),
|
||||
size: self.size,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl<S: ArchiveSource> S9pk<Section<S>> {
|
||||
#[instrument(skip_all)]
|
||||
pub async fn deserialize(source: &S) -> Result<Self, Error> {
|
||||
use tokio::io::AsyncReadExt;
|
||||
|
||||
@@ -36,6 +191,46 @@ impl<S: ArchiveSource> S9pk<Section<S>> {
|
||||
"Invalid Magic or Unexpected Version"
|
||||
);
|
||||
|
||||
Ok(Self(MerkleArchive::deserialize(source, &mut header).await?))
|
||||
let mut archive = MerkleArchive::deserialize(source, &mut header).await?;
|
||||
|
||||
archive.filter(filter)?;
|
||||
|
||||
archive.sort_by(|a, b| match (priority(a), priority(b)) {
|
||||
(Some(a), Some(b)) => a.cmp(&b),
|
||||
(Some(_), None) => std::cmp::Ordering::Less,
|
||||
(None, Some(_)) => std::cmp::Ordering::Greater,
|
||||
(None, None) => std::cmp::Ordering::Equal,
|
||||
});
|
||||
|
||||
Self::new(archive, source.size().await).await
|
||||
}
|
||||
}
|
||||
impl S9pk {
|
||||
pub async fn from_file(file: File) -> Result<Self, Error> {
|
||||
Self::deserialize(&MultiCursorFile::from(file)).await
|
||||
}
|
||||
pub async fn open(path: impl AsRef<Path>, id: Option<&PackageId>) -> Result<Self, Error> {
|
||||
let res = Self::from_file(tokio::fs::File::open(path).await?).await?;
|
||||
if let Some(id) = id {
|
||||
ensure_code!(
|
||||
&res.as_manifest().id == id,
|
||||
ErrorKind::ValidateS9pk,
|
||||
"manifest.id does not match expected"
|
||||
);
|
||||
}
|
||||
Ok(res)
|
||||
}
|
||||
}
|
||||
|
||||
async fn extract_manifest<S: FileSource>(archive: &MerkleArchive<S>) -> Result<Manifest, Error> {
|
||||
let manifest = serde_json::from_slice(
|
||||
&archive
|
||||
.contents()
|
||||
.get_path("manifest.json")
|
||||
.or_not_found("manifest.json")?
|
||||
.read_file_to_vec()
|
||||
.await?,
|
||||
)
|
||||
.with_kind(ErrorKind::Deserialization)?;
|
||||
Ok(manifest)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user