Feature/lxc container runtime (#2514)

* wip: static-server errors

* wip: fix wifi

* wip: Fix the service_effects

* wip: Fix cors in the middleware

* wip(chore): Auth clean up the lint.

* wip(fix): Vhost

* wip: continue manager refactor

Co-authored-by: J H <Blu-J@users.noreply.github.com>

* wip: service manager refactor

* wip: Some fixes

* wip(fix): Fix the lib.rs

* wip

* wip(fix): Logs

* wip: bins

* wip(innspect): Add in the inspect

* wip: config

* wip(fix): Diagnostic

* wip(fix): Dependencies

* wip: context

* wip(fix) Sorta auth

* wip: warnings

* wip(fix): registry/admin

* wip(fix) marketplace

* wip(fix) Some more converted and fixed with the linter and config

* wip: Working on the static server

* wip(fix)static server

* wip: Remove some asynnc

* wip: Something about the request and regular rpc

* wip: gut install

Co-authored-by: J H <Blu-J@users.noreply.github.com>

* wip: Convert the static server into the new system

* wip delete file

* test

* wip(fix) vhost does not need the with safe defaults

* wip: Adding in the wifi

* wip: Fix the developer and the verify

* wip: new install flow

Co-authored-by: J H <Blu-J@users.noreply.github.com>

* fix middleware

* wip

* wip: Fix the auth

* wip

* continue service refactor

* feature: Service get_config

* feat: Action

* wip: Fighting the great fight against the borrow checker

* wip: Remove an error in a file that I just need to deel with later

* chore: Add in some more lifetime stuff to the services

* wip: Install fix on lifetime

* cleanup

* wip: Deal with the borrow later

* more cleanup

* resolve borrowchecker errors

* wip(feat): add in the handler for the socket, for now

* wip(feat): Update the service_effect_handler::action

* chore: Add in the changes to make sure the from_service goes to context

* chore: Change the

* refactor service map

* fix references to service map

* fill out restore

* wip: Before I work on the store stuff

* fix backup module

* handle some warnings

* feat: add in the ui components on the rust side

* feature: Update the procedures

* chore: Update the js side of the main and a few of the others

* chore: Update the rpc listener to match the persistant container

* wip: Working on updating some things to have a better name

* wip(feat): Try and get the rpc to return the correct shape?

* lxc wip

* wip(feat): Try and get the rpc to return the correct shape?

* build for container runtime wip

* remove container-init

* fix build

* fix error

* chore: Update to work I suppose

* lxc wip

* remove docker module and feature

* download alpine squashfs automatically

* overlays effect

Co-authored-by: Jade <Blu-J@users.noreply.github.com>

* chore: Add the overlay effect

* feat: Add the mounter in the main

* chore: Convert to use the mounts, still need to work with the sandbox

* install fixes

* fix ssl

* fixes from testing

* implement tmpfile for upload

* wip

* misc fixes

* cleanup

* cleanup

* better progress reporting

* progress for sideload

* return real guid

* add devmode script

* fix lxc rootfs path

* fix percentage bar

* fix progress bar styling

* fix build for unstable

* tweaks

* label progress

* tweaks

* update progress more often

* make symlink in rpc_client

* make socket dir

* fix parent path

* add start-cli to container

* add echo and gitInfo commands

* wip: Add the init + errors

* chore: Add in the exit effect for the system

* chore: Change the type to null for failure to parse

* move sigterm timeout to stopping status

* update order

* chore: Update the return type

* remove dbg

* change the map error

* chore: Update the thing to capture id

* chore add some life changes

* chore: Update the loging

* chore: Update the package to run module

* us From for RpcError

* chore: Update to use import instead

* chore: update

* chore: Use require for the backup

* fix a default

* update the type that is wrong

* chore: Update the type of the manifest

* chore: Update to make null

* only symlink if not exists

* get rid of double result

* better debug info for ErrorCollection

* chore: Update effects

* chore: fix

* mount assets and volumes

* add exec instead of spawn

* fix mounting in image

* fix overlay mounts

Co-authored-by: Jade <Blu-J@users.noreply.github.com>

* misc fixes

* feat: Fix two

* fix: systemForEmbassy main

* chore: Fix small part of main loop

* chore: Modify the bundle

* merge

* fixMain loop"

* move tsc to makefile

* chore: Update the return types of the health check

* fix client

* chore: Convert the todo to use tsmatches

* add in the fixes for the seen and create the hack to allow demo

* chore: Update to include the systemForStartOs

* chore UPdate to the latest types from the expected outout

* fixes

* fix typo

* Don't emit if failure on tsc

* wip

Co-authored-by: Jade <Blu-J@users.noreply.github.com>

* add s9pk api

* add inspection

* add inspect manifest

* newline after display serializable

* fix squashfs in image name

* edit manifest

Co-authored-by: Jade <Blu-J@users.noreply.github.com>

* wait for response on repl

* ignore sig for now

* ignore sig for now

* re-enable sig verification

* fix

* wip

* env and chroot

* add profiling logs

* set uid & gid in squashfs to 100000

* set uid of sqfs to 100000

* fix mksquashfs args

* add env to compat

* fix

* re-add docker feature flag

* fix docker output format being stupid

* here be dragons

* chore: Add in the cross compiling for something

* fix npm link

* extract logs from container on exit

* chore: Update for testing

* add log capture to drop trait

* chore: add in the modifications that I make

* chore: Update small things for no updates

* chore: Update the types of something

* chore: Make main not complain

* idmapped mounts

* idmapped volumes

* re-enable kiosk

* chore: Add in some logging for the new system

* bring in start-sdk

* remove avahi

* chore: Update the deps

* switch to musl

* chore: Update the version of prettier

* chore: Organize'

* chore: Update some of the headers back to the standard of fetch

* fix musl build

* fix idmapped mounts

* fix cross build

* use cross compiler for correct arch

* feat: Add in the faked ssl stuff for the effects

* @dr_bonez Did a solution here

* chore: Something that DrBonez

* chore: up

* wip: We have a working server!!!

* wip

* uninstall

* wip

* tes

---------

Co-authored-by: J H <dragondef@gmail.com>
Co-authored-by: J H <Blu-J@users.noreply.github.com>
Co-authored-by: J H <2364004+Blu-J@users.noreply.github.com>
This commit is contained in:
Aiden McClelland
2024-02-17 11:14:14 -07:00
committed by GitHub
parent 65009e2f69
commit fab13db4b4
326 changed files with 31708 additions and 13987 deletions

536
core/startos/src/lxc/mod.rs Normal file
View File

@@ -0,0 +1,536 @@
use std::collections::BTreeSet;
use std::ops::Deref;
use std::path::Path;
use std::sync::{Arc, Weak};
use std::time::Duration;
use clap::Parser;
use futures::{AsyncWriteExt, FutureExt, StreamExt};
use imbl_value::{InOMap, InternedString};
use rpc_toolkit::yajrc::{RpcError, RpcResponse};
use rpc_toolkit::{
from_fn_async, AnyContext, CallRemoteHandler, GenericRpcMethod, Handler, HandlerArgs,
HandlerExt, ParentHandler, RpcRequest,
};
use rustyline_async::{ReadlineEvent, SharedWriter};
use serde::{Deserialize, Serialize};
use tokio::fs::File;
use tokio::io::{AsyncBufReadExt, BufReader};
use tokio::process::Command;
use tokio::sync::Mutex;
use tokio::time::Instant;
use crate::context::{CliContext, RpcContext};
use crate::core::rpc_continuations::{RequestGuid, RpcContinuation};
use crate::disk::mount::filesystem::bind::Bind;
use crate::disk::mount::filesystem::block_dev::BlockDev;
use crate::disk::mount::filesystem::idmapped::IdMapped;
use crate::disk::mount::filesystem::overlayfs::OverlayGuard;
use crate::disk::mount::filesystem::ReadWrite;
use crate::disk::mount::guard::{GenericMountGuard, TmpMountGuard};
use crate::disk::mount::util::unmount;
use crate::prelude::*;
use crate::util::rpc_client::UnixRpcClient;
use crate::util::{new_guid, Invoke};
const LXC_CONTAINER_DIR: &str = "/var/lib/lxc";
const RPC_DIR: &str = "media/startos/rpc"; // must not be absolute path
pub const CONTAINER_RPC_SERVER_SOCKET: &str = "service.sock"; // must not be absolute path
pub const HOST_RPC_SERVER_SOCKET: &str = "host.sock"; // must not be absolute path
pub struct LxcManager {
containers: Mutex<Vec<Weak<InternedString>>>,
}
impl LxcManager {
pub fn new() -> Self {
Self {
containers: Default::default(),
}
}
pub async fn create(self: &Arc<Self>, config: LxcConfig) -> Result<LxcContainer, Error> {
let container = LxcContainer::new(self, config).await?;
let mut guard = self.containers.lock().await;
*guard = std::mem::take(&mut *guard)
.into_iter()
.filter(|g| g.strong_count() > 0)
.chain(std::iter::once(Arc::downgrade(&container.guid)))
.collect();
Ok(container)
}
pub async fn gc(&self) -> Result<(), Error> {
let expected = BTreeSet::from_iter(
self.containers
.lock()
.await
.iter()
.filter_map(|g| g.upgrade())
.map(|g| (&*g).clone()),
);
for container in String::from_utf8(
Command::new("lxc-ls")
.arg("-1")
.invoke(ErrorKind::Lxc)
.await?,
)?
.lines()
.map(|s| s.trim())
{
if !expected.contains(container) {
let rootfs_path = Path::new(LXC_CONTAINER_DIR).join(container).join("rootfs");
if tokio::fs::metadata(&rootfs_path).await.is_ok() {
unmount(Path::new(LXC_CONTAINER_DIR).join(container).join("rootfs")).await?;
if tokio_stream::wrappers::ReadDirStream::new(
tokio::fs::read_dir(&rootfs_path).await?,
)
.count()
.await
> 0
{
return Err(Error::new(
eyre!("rootfs is not empty, refusing to delete"),
ErrorKind::InvalidRequest,
));
}
}
Command::new("lxc-destroy")
.arg("--force")
.arg("--name")
.arg(container)
.invoke(ErrorKind::Lxc)
.await?;
}
}
Ok(())
}
}
pub struct LxcContainer {
manager: Weak<LxcManager>,
rootfs: OverlayGuard,
guid: Arc<InternedString>,
rpc_bind: TmpMountGuard,
config: LxcConfig,
exited: bool,
}
impl LxcContainer {
async fn new(manager: &Arc<LxcManager>, config: LxcConfig) -> Result<Self, Error> {
let guid = new_guid();
let container_dir = Path::new(LXC_CONTAINER_DIR).join(&*guid);
tokio::fs::create_dir_all(&container_dir).await?;
tokio::fs::write(
container_dir.join("config"),
format!(include_str!("./config.template"), guid = &*guid),
)
.await?;
// TODO: append config
let rootfs_dir = container_dir.join("rootfs");
tokio::fs::create_dir_all(&rootfs_dir).await?;
Command::new("chown")
.arg("100000:100000")
.arg(&rootfs_dir)
.invoke(ErrorKind::Filesystem)
.await?;
let rootfs = OverlayGuard::mount(
&IdMapped::new(
BlockDev::new("/usr/lib/startos/container-runtime/rootfs.squashfs"),
0,
100000,
65536,
),
&rootfs_dir,
)
.await?;
tokio::fs::write(rootfs_dir.join("etc/hostname"), format!("{guid}\n")).await?;
Command::new("sed")
.arg("-i")
.arg(format!("s/LXC_NAME/{guid}/g"))
.arg(rootfs_dir.join("etc/hosts"))
.invoke(ErrorKind::Filesystem)
.await?;
Command::new("mount")
.arg("--make-rshared")
.arg(rootfs.path())
.invoke(ErrorKind::Filesystem)
.await?;
let rpc_dir = rootfs_dir.join(RPC_DIR);
tokio::fs::create_dir_all(&rpc_dir).await?;
let rpc_bind = TmpMountGuard::mount(&Bind::new(rpc_dir), ReadWrite).await?;
Command::new("chown")
.arg("-R")
.arg("100000:100000")
.arg(rpc_bind.path())
.invoke(ErrorKind::Filesystem)
.await?;
Command::new("lxc-start")
.arg("-d")
.arg("--name")
.arg(&*guid)
.invoke(ErrorKind::Lxc)
.await?;
Ok(Self {
manager: Arc::downgrade(manager),
rootfs,
guid: Arc::new(guid),
rpc_bind,
config,
exited: false,
})
}
pub fn rootfs_dir(&self) -> &Path {
self.rootfs.path()
}
pub fn rpc_dir(&self) -> &Path {
self.rpc_bind.path()
}
#[instrument(skip_all)]
pub async fn exit(mut self) -> Result<(), Error> {
self.rpc_bind.take().unmount().await?;
self.rootfs.take().unmount(true).await?;
let rootfs_path = self.rootfs_dir();
let err_path = rootfs_path.join("var/log/containerRuntime.err");
if tokio::fs::metadata(&err_path).await.is_ok() {
let mut lines = BufReader::new(File::open(&err_path).await?).lines();
while let Some(line) = lines.next_line().await? {
let container = &**self.guid;
tracing::error!(container, "{}", line);
}
}
if tokio::fs::metadata(&rootfs_path).await.is_ok() {
if tokio_stream::wrappers::ReadDirStream::new(tokio::fs::read_dir(&rootfs_path).await?)
.count()
.await
> 0
{
return Err(Error::new(
eyre!("rootfs is not empty, refusing to delete"),
ErrorKind::InvalidRequest,
));
}
}
Command::new("lxc-destroy")
.arg("--force")
.arg("--name")
.arg(&**self.guid)
.invoke(ErrorKind::Lxc)
.await?;
self.exited = true;
Ok(())
}
pub async fn connect_rpc(&self, timeout: Option<Duration>) -> Result<UnixRpcClient, Error> {
let started = Instant::now();
let sock_path = self.rpc_dir().join(CONTAINER_RPC_SERVER_SOCKET);
while tokio::fs::metadata(&sock_path).await.is_err() {
if timeout.map_or(false, |t| started.elapsed() > t) {
return Err(Error::new(
eyre!("timed out waiting for socket"),
ErrorKind::Timeout,
));
}
tokio::time::sleep(Duration::from_millis(100)).await;
}
Ok(UnixRpcClient::new(sock_path))
}
}
impl Drop for LxcContainer {
fn drop(&mut self) {
if !self.exited {
tracing::warn!(
"Container {} was ungracefully dropped. Cleaning up dangling containers...",
&**self.guid
);
let rootfs = self.rootfs.take();
let guid = std::mem::take(&mut self.guid);
if let Some(manager) = self.manager.upgrade() {
tokio::spawn(async move {
if let Err(e) = async {
let err_path = rootfs.path().join("var/log/containerRuntime.err");
if tokio::fs::metadata(&err_path).await.is_ok() {
let mut lines = BufReader::new(File::open(&err_path).await?).lines();
while let Some(line) = lines.next_line().await? {
let container = &**guid;
tracing::error!(container, "{}", line);
}
}
Ok::<_, Error>(())
}
.await
{
tracing::error!("Error reading logs from crashed container: {e}");
tracing::debug!("{e:?}")
}
rootfs.unmount(true).await.unwrap();
drop(guid);
if let Err(e) = manager.gc().await {
tracing::error!("Error cleaning up dangling LXC containers: {e}");
tracing::debug!("{e:?}")
} else {
tracing::info!("Successfully cleaned up dangling LXC containers");
}
});
}
}
}
}
#[derive(Default, Serialize)]
pub struct LxcConfig {}
pub fn lxc() -> ParentHandler {
ParentHandler::new()
.subcommand(
"create",
from_fn_async(create).with_remote_cli::<CliContext>(),
)
.subcommand(
"list",
from_fn_async(list)
.with_custom_display_fn::<AnyContext, _>(|_, res| {
use prettytable::*;
let mut table = table!([bc => "GUID"]);
for guid in res {
table.add_row(row![&*guid]);
}
table.printstd();
Ok(())
})
.with_remote_cli::<CliContext>(),
)
.subcommand(
"remove",
from_fn_async(remove)
.no_display()
.with_remote_cli::<CliContext>(),
)
.subcommand("connect", from_fn_async(connect_rpc).no_cli())
.subcommand("connect", from_fn_async(connect_rpc_cli).no_display())
}
pub async fn create(ctx: RpcContext) -> Result<InternedString, Error> {
let container = ctx.lxc_manager.create(LxcConfig::default()).await?;
let guid = container.guid.deref().clone();
ctx.dev.lxc.lock().await.insert(guid.clone(), container);
Ok(guid)
}
pub async fn list(ctx: RpcContext) -> Result<Vec<InternedString>, Error> {
Ok(ctx.dev.lxc.lock().await.keys().cloned().collect())
}
#[derive(Deserialize, Serialize, Parser)]
pub struct RemoveParams {
pub guid: InternedString,
}
pub async fn remove(ctx: RpcContext, RemoveParams { guid }: RemoveParams) -> Result<(), Error> {
if let Some(container) = ctx.dev.lxc.lock().await.remove(&guid) {
container.exit().await?;
}
Ok(())
}
#[derive(Deserialize, Serialize, Parser)]
pub struct ConnectParams {
pub guid: InternedString,
}
pub async fn connect_rpc(
ctx: RpcContext,
ConnectParams { guid }: ConnectParams,
) -> Result<RequestGuid, Error> {
connect(
&ctx,
ctx.dev.lxc.lock().await.get(&guid).ok_or_else(|| {
Error::new(eyre!("No container with guid: {guid}"), ErrorKind::NotFound)
})?,
)
.await
}
pub async fn connect(ctx: &RpcContext, container: &LxcContainer) -> Result<RequestGuid, Error> {
use axum::extract::ws::Message;
let rpc = container.connect_rpc(Some(Duration::from_secs(30))).await?;
let guid = RequestGuid::new();
ctx.add_continuation(
guid.clone(),
RpcContinuation::ws(
Box::new(|mut ws| {
async move {
if let Err(e) = async {
loop {
match ws.next().await {
None => break,
Some(Ok(Message::Text(txt))) => {
let mut id = None;
let result = async {
let req: RpcRequest =
serde_json::from_str(&txt).map_err(|e| RpcError {
data: Some(serde_json::Value::String(
e.to_string(),
)),
..rpc_toolkit::yajrc::PARSE_ERROR
})?;
id = req.id;
rpc.request(req.method, req.params).await
}
.await;
ws.send(Message::Text(
serde_json::to_string(&RpcResponse::<GenericRpcMethod> {
id,
result,
})
.with_kind(ErrorKind::Serialization)?,
))
.await
.with_kind(ErrorKind::Network)?;
}
Some(Ok(_)) => (),
Some(Err(e)) => {
return Err(Error::new(e, ErrorKind::Network));
}
}
}
Ok::<_, Error>(())
}
.await
{
tracing::error!("{e}");
tracing::debug!("{e:?}");
}
}
.boxed()
}),
Duration::from_secs(30),
),
)
.await;
Ok(guid)
}
pub async fn connect_cli(ctx: &CliContext, guid: RequestGuid) -> Result<(), Error> {
use futures::SinkExt;
use tokio_tungstenite::tungstenite::Message;
let mut ws = ctx.ws_continuation(guid).await?;
let (mut input, mut output) =
rustyline_async::Readline::new("> ".into()).with_kind(ErrorKind::Filesystem)?;
async fn handle_message(
msg: Option<Result<Message, tokio_tungstenite::tungstenite::Error>>,
output: &mut SharedWriter,
) -> Result<bool, Error> {
match msg {
None => return Ok(true),
Some(Ok(Message::Text(txt))) => match serde_json::from_str::<RpcResponse>(&txt) {
Ok(RpcResponse { result: Ok(a), .. }) => {
output
.write_all(
(serde_json::to_string(&a).with_kind(ErrorKind::Serialization)? + "\n")
.as_bytes(),
)
.await?;
}
Ok(RpcResponse { result: Err(e), .. }) => {
let e: Error = e.into();
tracing::error!("{e}");
tracing::debug!("{e:?}");
}
Err(e) => {
tracing::error!("Error Parsing RPC response: {e}");
tracing::debug!("{e:?}");
}
},
Some(Ok(_)) => (),
Some(Err(e)) => {
return Err(Error::new(e, ErrorKind::Network));
}
};
Ok(false)
}
loop {
tokio::select! {
line = input.readline() => {
let line = line.with_kind(ErrorKind::Filesystem)?;
if let ReadlineEvent::Line(line) = line {
input.add_history_entry(line.clone());
if serde_json::from_str::<RpcRequest>(&line).is_ok() {
ws.send(Message::Text(line))
.await
.with_kind(ErrorKind::Network)?;
} else {
match shell_words::split(&line) {
Ok(command) => {
if let Some((method, rest)) = command.split_first() {
let mut params = InOMap::new();
for arg in rest {
if let Some((name, value)) = arg.split_once("=") {
params.insert(InternedString::intern(name), if value.is_empty() {
Value::Null
} else if let Ok(v) = serde_json::from_str(value) {
v
} else {
Value::String(Arc::new(value.into()))
});
} else {
tracing::error!("argument without a value: {arg}");
tracing::debug!("help: set the value of {arg} with `{arg}=...`");
continue;
}
}
ws.send(Message::Text(match serde_json::to_string(&RpcRequest {
id: None,
method: GenericRpcMethod::new(method.into()),
params: Value::Object(params),
}) {
Ok(a) => a,
Err(e) => {
tracing::error!("Error Serializing Request: {e}");
tracing::debug!("{e:?}");
continue;
}
})).await.with_kind(ErrorKind::Network)?;
if handle_message(ws.next().await, &mut output).await? {
break
}
}
}
Err(e) => {
tracing::error!("{e}");
tracing::debug!("{e:?}");
}
}
}
} else {
ws.send(Message::Close(None)).await.with_kind(ErrorKind::Network)?;
}
}
msg = ws.next() => {
if handle_message(msg, &mut output).await? {
break;
}
}
}
}
Ok(())
}
pub async fn connect_rpc_cli(
handle_args: HandlerArgs<CliContext, ConnectParams>,
) -> Result<(), Error> {
let ctx = handle_args.context.clone();
let guid = CallRemoteHandler::<CliContext, _>::new(from_fn_async(connect_rpc))
.handle_async(handle_args)
.await?;
connect_cli(&ctx, guid).await
}