Feature/lxc container runtime (#2514)

* wip: static-server errors

* wip: fix wifi

* wip: Fix the service_effects

* wip: Fix cors in the middleware

* wip(chore): Auth clean up the lint.

* wip(fix): Vhost

* wip: continue manager refactor

Co-authored-by: J H <Blu-J@users.noreply.github.com>

* wip: service manager refactor

* wip: Some fixes

* wip(fix): Fix the lib.rs

* wip

* wip(fix): Logs

* wip: bins

* wip(innspect): Add in the inspect

* wip: config

* wip(fix): Diagnostic

* wip(fix): Dependencies

* wip: context

* wip(fix) Sorta auth

* wip: warnings

* wip(fix): registry/admin

* wip(fix) marketplace

* wip(fix) Some more converted and fixed with the linter and config

* wip: Working on the static server

* wip(fix)static server

* wip: Remove some asynnc

* wip: Something about the request and regular rpc

* wip: gut install

Co-authored-by: J H <Blu-J@users.noreply.github.com>

* wip: Convert the static server into the new system

* wip delete file

* test

* wip(fix) vhost does not need the with safe defaults

* wip: Adding in the wifi

* wip: Fix the developer and the verify

* wip: new install flow

Co-authored-by: J H <Blu-J@users.noreply.github.com>

* fix middleware

* wip

* wip: Fix the auth

* wip

* continue service refactor

* feature: Service get_config

* feat: Action

* wip: Fighting the great fight against the borrow checker

* wip: Remove an error in a file that I just need to deel with later

* chore: Add in some more lifetime stuff to the services

* wip: Install fix on lifetime

* cleanup

* wip: Deal with the borrow later

* more cleanup

* resolve borrowchecker errors

* wip(feat): add in the handler for the socket, for now

* wip(feat): Update the service_effect_handler::action

* chore: Add in the changes to make sure the from_service goes to context

* chore: Change the

* refactor service map

* fix references to service map

* fill out restore

* wip: Before I work on the store stuff

* fix backup module

* handle some warnings

* feat: add in the ui components on the rust side

* feature: Update the procedures

* chore: Update the js side of the main and a few of the others

* chore: Update the rpc listener to match the persistant container

* wip: Working on updating some things to have a better name

* wip(feat): Try and get the rpc to return the correct shape?

* lxc wip

* wip(feat): Try and get the rpc to return the correct shape?

* build for container runtime wip

* remove container-init

* fix build

* fix error

* chore: Update to work I suppose

* lxc wip

* remove docker module and feature

* download alpine squashfs automatically

* overlays effect

Co-authored-by: Jade <Blu-J@users.noreply.github.com>

* chore: Add the overlay effect

* feat: Add the mounter in the main

* chore: Convert to use the mounts, still need to work with the sandbox

* install fixes

* fix ssl

* fixes from testing

* implement tmpfile for upload

* wip

* misc fixes

* cleanup

* cleanup

* better progress reporting

* progress for sideload

* return real guid

* add devmode script

* fix lxc rootfs path

* fix percentage bar

* fix progress bar styling

* fix build for unstable

* tweaks

* label progress

* tweaks

* update progress more often

* make symlink in rpc_client

* make socket dir

* fix parent path

* add start-cli to container

* add echo and gitInfo commands

* wip: Add the init + errors

* chore: Add in the exit effect for the system

* chore: Change the type to null for failure to parse

* move sigterm timeout to stopping status

* update order

* chore: Update the return type

* remove dbg

* change the map error

* chore: Update the thing to capture id

* chore add some life changes

* chore: Update the loging

* chore: Update the package to run module

* us From for RpcError

* chore: Update to use import instead

* chore: update

* chore: Use require for the backup

* fix a default

* update the type that is wrong

* chore: Update the type of the manifest

* chore: Update to make null

* only symlink if not exists

* get rid of double result

* better debug info for ErrorCollection

* chore: Update effects

* chore: fix

* mount assets and volumes

* add exec instead of spawn

* fix mounting in image

* fix overlay mounts

Co-authored-by: Jade <Blu-J@users.noreply.github.com>

* misc fixes

* feat: Fix two

* fix: systemForEmbassy main

* chore: Fix small part of main loop

* chore: Modify the bundle

* merge

* fixMain loop"

* move tsc to makefile

* chore: Update the return types of the health check

* fix client

* chore: Convert the todo to use tsmatches

* add in the fixes for the seen and create the hack to allow demo

* chore: Update to include the systemForStartOs

* chore UPdate to the latest types from the expected outout

* fixes

* fix typo

* Don't emit if failure on tsc

* wip

Co-authored-by: Jade <Blu-J@users.noreply.github.com>

* add s9pk api

* add inspection

* add inspect manifest

* newline after display serializable

* fix squashfs in image name

* edit manifest

Co-authored-by: Jade <Blu-J@users.noreply.github.com>

* wait for response on repl

* ignore sig for now

* ignore sig for now

* re-enable sig verification

* fix

* wip

* env and chroot

* add profiling logs

* set uid & gid in squashfs to 100000

* set uid of sqfs to 100000

* fix mksquashfs args

* add env to compat

* fix

* re-add docker feature flag

* fix docker output format being stupid

* here be dragons

* chore: Add in the cross compiling for something

* fix npm link

* extract logs from container on exit

* chore: Update for testing

* add log capture to drop trait

* chore: add in the modifications that I make

* chore: Update small things for no updates

* chore: Update the types of something

* chore: Make main not complain

* idmapped mounts

* idmapped volumes

* re-enable kiosk

* chore: Add in some logging for the new system

* bring in start-sdk

* remove avahi

* chore: Update the deps

* switch to musl

* chore: Update the version of prettier

* chore: Organize'

* chore: Update some of the headers back to the standard of fetch

* fix musl build

* fix idmapped mounts

* fix cross build

* use cross compiler for correct arch

* feat: Add in the faked ssl stuff for the effects

* @dr_bonez Did a solution here

* chore: Something that DrBonez

* chore: up

* wip: We have a working server!!!

* wip

* uninstall

* wip

* tes

---------

Co-authored-by: J H <dragondef@gmail.com>
Co-authored-by: J H <Blu-J@users.noreply.github.com>
Co-authored-by: J H <2364004+Blu-J@users.noreply.github.com>
This commit is contained in:
Aiden McClelland
2024-02-17 11:14:14 -07:00
committed by GitHub
parent 65009e2f69
commit fab13db4b4
326 changed files with 31708 additions and 13987 deletions

View File

@@ -1,21 +1,25 @@
use std::fs::Metadata;
use std::future::Future;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::time::UNIX_EPOCH;
use async_compression::tokio::bufread::GzipEncoder;
use color_eyre::eyre::eyre;
use axum::body::Body;
use axum::extract::{self as x, Request};
use axum::response::Response;
use axum::routing::{any, get, post};
use axum::Router;
use digest::Digest;
use futures::FutureExt;
use futures::future::ready;
use futures::{FutureExt, TryFutureExt};
use http::header::ACCEPT_ENCODING;
use http::request::Parts as RequestParts;
use hyper::{Body, Method, Request, Response, StatusCode};
use http::{HeaderMap, Method, StatusCode};
use include_dir::{include_dir, Dir};
use new_mime_guess::MimeGuess;
use openssl::hash::MessageDigest;
use openssl::x509::X509;
use rpc_toolkit::rpc_handler;
use rpc_toolkit::Server;
use tokio::fs::File;
use tokio::io::BufReader;
use tokio_util::io::ReaderStream;
@@ -25,11 +29,10 @@ use crate::core::rpc_continuations::RequestGuid;
use crate::db::subscribe;
use crate::hostname::Hostname;
use crate::install::PKG_PUBLIC_DIR;
use crate::middleware::auth::{auth as auth_middleware, HasValidSession};
use crate::middleware::cors::cors;
use crate::middleware::db::db as db_middleware;
use crate::middleware::diagnostic::diagnostic as diagnostic_middleware;
use crate::net::HttpHandler;
use crate::middleware::auth::{Auth, HasValidSession};
use crate::middleware::cors::Cors;
use crate::middleware::db::SyncDb;
use crate::middleware::diagnostic::DiagnosticMode;
use crate::{diagnostic_api, install_api, main_api, setup_api, Error, ErrorKind, ResultExt};
static NOT_FOUND: &[u8] = b"Not Found";
@@ -40,10 +43,6 @@ static EMBEDDED_UIS: Dir<'_> = include_dir!("$CARGO_MANIFEST_DIR/../../web/dist/
const PROXY_STRIP_HEADERS: &[&str] = &["cookie", "host", "origin", "referer", "user-agent"];
fn status_fn(_: i32) -> StatusCode {
StatusCode::OK
}
#[derive(Clone)]
pub enum UiMode {
Setup,
@@ -63,180 +62,123 @@ impl UiMode {
}
}
pub async fn setup_ui_file_router(ctx: SetupContext) -> Result<HttpHandler, Error> {
let handler: HttpHandler = Arc::new(move |req| {
let ctx = ctx.clone();
let ui_mode = UiMode::Setup;
async move {
let res = match req.uri().path() {
path if path.starts_with("/rpc/") => {
let rpc_handler = rpc_handler!({
command: setup_api,
context: ctx,
status: status_fn,
middleware: [
cors,
]
});
rpc_handler(req)
.await
.map_err(|err| Error::new(eyre!("{}", err), crate::ErrorKind::Network))
}
_ => alt_ui(req, ui_mode).await,
};
match res {
Ok(data) => Ok(data),
Err(err) => Ok(server_error(err)),
}
}
.boxed()
});
Ok(handler)
pub fn setup_ui_file_router(ctx: SetupContext) -> Router {
Router::new()
.route_service(
"/rpc/*path",
post(Server::new(move || ready(Ok(ctx.clone())), setup_api()).middleware(Cors::new())),
)
.fallback(any(|request: Request| async move {
alt_ui(request, UiMode::Setup)
.await
.unwrap_or_else(server_error)
}))
}
pub async fn diag_ui_file_router(ctx: DiagnosticContext) -> Result<HttpHandler, Error> {
let handler: HttpHandler = Arc::new(move |req| {
let ctx = ctx.clone();
let ui_mode = UiMode::Diag;
async move {
let res = match req.uri().path() {
path if path.starts_with("/rpc/") => {
let rpc_handler = rpc_handler!({
command: diagnostic_api,
context: ctx,
status: status_fn,
middleware: [
cors,
diagnostic_middleware,
]
});
rpc_handler(req)
.await
.map_err(|err| Error::new(eyre!("{}", err), crate::ErrorKind::Network))
}
_ => alt_ui(req, ui_mode).await,
};
match res {
Ok(data) => Ok(data),
Err(err) => Ok(server_error(err)),
}
}
.boxed()
});
Ok(handler)
pub fn diag_ui_file_router(ctx: DiagnosticContext) -> Router {
Router::new()
.route(
"/rpc/*path",
post(
Server::new(move || ready(Ok(ctx.clone())), diagnostic_api())
.middleware(Cors::new())
.middleware(DiagnosticMode::new()),
),
)
.fallback(any(|request: Request| async move {
alt_ui(request, UiMode::Diag)
.await
.unwrap_or_else(server_error)
}))
}
pub async fn install_ui_file_router(ctx: InstallContext) -> Result<HttpHandler, Error> {
let handler: HttpHandler = Arc::new(move |req| {
let ctx = ctx.clone();
let ui_mode = UiMode::Install;
async move {
let res = match req.uri().path() {
path if path.starts_with("/rpc/") => {
let rpc_handler = rpc_handler!({
command: install_api,
context: ctx,
status: status_fn,
middleware: [
cors,
]
});
rpc_handler(req)
.await
.map_err(|err| Error::new(eyre!("{}", err), crate::ErrorKind::Network))
}
_ => alt_ui(req, ui_mode).await,
};
match res {
Ok(data) => Ok(data),
Err(err) => Ok(server_error(err)),
}
}
.boxed()
});
Ok(handler)
pub fn install_ui_file_router(ctx: InstallContext) -> Router {
Router::new()
.route("/rpc/*path", {
let ctx = ctx.clone();
post(Server::new(move || ready(Ok(ctx.clone())), install_api()).middleware(Cors::new()))
})
.fallback(any(|request: Request| async move {
alt_ui(request, UiMode::Install)
.await
.unwrap_or_else(server_error)
}))
}
pub async fn main_ui_server_router(ctx: RpcContext) -> Result<HttpHandler, Error> {
let handler: HttpHandler = Arc::new(move |req| {
let ctx = ctx.clone();
async move {
let res = match req.uri().path() {
path if path.starts_with("/rpc/") => {
let auth_middleware = auth_middleware(ctx.clone());
let db_middleware = db_middleware(ctx.clone());
let rpc_handler = rpc_handler!({
command: main_api,
context: ctx,
status: status_fn,
middleware: [
cors,
auth_middleware,
db_middleware,
]
});
rpc_handler(req)
pub fn main_ui_server_router(ctx: RpcContext) -> Router {
Router::new()
.route("/rpc/*path", {
let ctx = ctx.clone();
post(
Server::new(move || ready(Ok(ctx.clone())), main_api())
.middleware(Cors::new())
.middleware(Auth::new())
.middleware(SyncDb::new()),
)
})
.route(
"/ws/db",
any({
let ctx = ctx.clone();
move |headers: HeaderMap, ws: x::WebSocketUpgrade| async move {
subscribe(ctx, headers, ws)
.await
.map_err(|err| Error::new(eyre!("{}", err), crate::ErrorKind::Network))
.unwrap_or_else(server_error)
}
"/ws/db" => subscribe(ctx, req).await,
path if path.starts_with("/ws/rpc/") => {
match RequestGuid::from(path.strip_prefix("/ws/rpc/").unwrap()) {
}),
)
.route(
"/ws/rpc/*path",
get({
let ctx = ctx.clone();
move |headers: HeaderMap,
x::Path(path): x::Path<String>,
ws: axum::extract::ws::WebSocketUpgrade| async move {
match RequestGuid::from(&path) {
None => {
tracing::debug!("No Guid Path");
Ok::<_, Error>(bad_request())
bad_request()
}
Some(guid) => match ctx.get_ws_continuation_handler(&guid).await {
Some(cont) => match cont(req).await {
Ok::<_, Error>(r) => Ok::<_, Error>(r),
Err(err) => Ok::<_, Error>(server_error(err)),
},
_ => Ok::<_, Error>(not_found()),
Some(cont) => ws.on_upgrade(cont),
_ => not_found(),
},
}
}
path if path.starts_with("/rest/rpc/") => {
match RequestGuid::from(path.strip_prefix("/rest/rpc/").unwrap()) {
}),
)
.route(
"/rest/rpc/*path",
any({
let ctx = ctx.clone();
move |request: x::Request| async move {
let path = request
.uri()
.path()
.clone()
.strip_prefix("/rest/rpc/")
.unwrap_or_default();
match RequestGuid::from(&path) {
None => {
tracing::debug!("No Guid Path");
Ok::<_, Error>(bad_request())
bad_request()
}
Some(guid) => match ctx.get_rest_continuation_handler(&guid).await {
None => Ok::<_, Error>(not_found()),
Some(cont) => match cont(req).await {
Ok::<_, Error>(r) => Ok::<_, Error>(r),
Err(e) => Ok::<_, Error>(server_error(e)),
},
None => not_found(),
Some(cont) => cont(request).await.unwrap_or_else(server_error),
},
}
}
_ => main_embassy_ui(req, ctx).await,
};
match res {
Ok(data) => Ok(data),
Err(err) => Ok(server_error(err)),
}
}
.boxed()
});
Ok(handler)
}),
)
.fallback(any(move |request: Request| async move {
main_embassy_ui(request, ctx)
.await
.unwrap_or_else(server_error)
}))
}
async fn alt_ui(req: Request<Body>, ui_mode: UiMode) -> Result<Response<Body>, Error> {
async fn alt_ui(req: Request, ui_mode: UiMode) -> Result<Response, Error> {
let (request_parts, _body) = req.into_parts();
match &request_parts.method {
&Method::GET => {
@@ -266,20 +208,21 @@ async fn alt_ui(req: Request<Body>, ui_mode: UiMode) -> Result<Response<Body>, E
async fn if_authorized<
F: FnOnce() -> Fut,
Fut: Future<Output = Result<Response<Body>, Error>> + Send + Sync,
Fut: Future<Output = Result<Response, Error>> + Send + Sync,
>(
ctx: &RpcContext,
parts: &RequestParts,
f: F,
) -> Result<Response<Body>, Error> {
if let Err(e) = HasValidSession::from_request_parts(parts, ctx).await {
) -> Result<Response, Error> {
if let Err(e) = HasValidSession::from_header(parts.headers.get(http::header::COOKIE), ctx).await
{
un_authorized(e, parts.uri.path())
} else {
f().await
}
}
async fn main_embassy_ui(req: Request<Body>, ctx: RpcContext) -> Result<Response<Body>, Error> {
async fn main_embassy_ui(req: Request, ctx: RpcContext) -> Result<Response, Error> {
let (request_parts, _body) = req.into_parts();
match (
&request_parts.method,
@@ -291,21 +234,7 @@ async fn main_embassy_ui(req: Request<Body>, ctx: RpcContext) -> Result<Response
.split_once('/'),
) {
(&Method::GET, Some(("public", path))) => {
if_authorized(&ctx, &request_parts, || async {
let sub_path = Path::new(path);
if let Ok(rest) = sub_path.strip_prefix("package-data") {
FileData::from_path(
&request_parts,
&ctx.datadir.join(PKG_PUBLIC_DIR).join(rest),
)
.await?
.into_response(&request_parts)
.await
} else {
Ok(not_found())
}
})
.await
todo!("pull directly from s9pk")
}
(&Method::GET, Some(("proxy", target))) => {
if_authorized(&ctx, &request_parts, || async {
@@ -322,19 +251,27 @@ async fn main_embassy_ui(req: Request<Body>, ctx: RpcContext) -> Result<Response
.iter()
.any(|bad| h.as_str().eq_ignore_ascii_case(bad))
})
.map(|(h, v)| (h.clone(), v.clone()))
.flat_map(|(h, v)| {
Some((
reqwest::header::HeaderName::from_lowercase(
h.as_str().as_bytes(),
)
.ok()?,
reqwest::header::HeaderValue::from_bytes(v.as_bytes()).ok()?,
))
})
.collect(),
)
.send()
.await
.with_kind(crate::ErrorKind::Network)?;
let mut hres = Response::builder().status(res.status());
let mut hres = Response::builder().status(res.status().as_u16());
for (h, v) in res.headers().clone() {
if let Some(h) = h {
hres = hres.header(h, v);
hres = hres.header(h.to_string(), v.as_bytes());
}
}
hres.body(Body::wrap_stream(res.bytes_stream()))
hres.body(Body::from_stream(res.bytes_stream()))
.with_kind(crate::ErrorKind::Network)
})
.await
@@ -368,7 +305,7 @@ async fn main_embassy_ui(req: Request<Body>, ctx: RpcContext) -> Result<Response
}
}
fn un_authorized(err: Error, path: &str) -> Result<Response<Body>, Error> {
fn un_authorized(err: Error, path: &str) -> Result<Response, Error> {
tracing::warn!("unauthorized for {} @{:?}", err, path);
tracing::debug!("{:?}", err);
Ok(Response::builder()
@@ -378,7 +315,7 @@ fn un_authorized(err: Error, path: &str) -> Result<Response<Body>, Error> {
}
/// HTTP status code 404
fn not_found() -> Response<Body> {
fn not_found() -> Response {
Response::builder()
.status(StatusCode::NOT_FOUND)
.body(NOT_FOUND.into())
@@ -386,28 +323,28 @@ fn not_found() -> Response<Body> {
}
/// HTTP status code 405
fn method_not_allowed() -> Response<Body> {
fn method_not_allowed() -> Response {
Response::builder()
.status(StatusCode::METHOD_NOT_ALLOWED)
.body(METHOD_NOT_ALLOWED.into())
.unwrap()
}
fn server_error(err: Error) -> Response<Body> {
fn server_error(err: Error) -> Response {
Response::builder()
.status(StatusCode::INTERNAL_SERVER_ERROR)
.body(err.to_string().into())
.unwrap()
}
fn bad_request() -> Response<Body> {
fn bad_request() -> Response {
Response::builder()
.status(StatusCode::BAD_REQUEST)
.body(Body::empty())
.unwrap()
}
fn cert_send(cert: &X509, hostname: &Hostname) -> Result<Response<Body>, Error> {
fn cert_send(cert: &X509, hostname: &Hostname) -> Result<Response, Error> {
let pem = cert.to_pem()?;
Response::builder()
.status(StatusCode::OK)
@@ -499,12 +436,12 @@ impl FileData {
let (len, data) = if encoding == Some("gzip") {
(
None,
Body::wrap_stream(ReaderStream::new(GzipEncoder::new(BufReader::new(file)))),
Body::from_stream(ReaderStream::new(GzipEncoder::new(BufReader::new(file)))),
)
} else {
(
Some(metadata.len()),
Body::wrap_stream(ReaderStream::new(file)),
Body::from_stream(ReaderStream::new(file)),
)
};
@@ -519,7 +456,7 @@ impl FileData {
})
}
async fn into_response(self, req: &RequestParts) -> Result<Response<Body>, Error> {
async fn into_response(self, req: &RequestParts) -> Result<Response, Error> {
let mut builder = Response::builder();
if let Some(mime) = self.mime {
builder = builder.header(http::header::CONTENT_TYPE, &*mime);