mirror of
https://github.com/Start9Labs/start-os.git
synced 2026-03-30 20:14:49 +00:00
Feature/lxc container runtime (#2514)
* wip: static-server errors * wip: fix wifi * wip: Fix the service_effects * wip: Fix cors in the middleware * wip(chore): Auth clean up the lint. * wip(fix): Vhost * wip: continue manager refactor Co-authored-by: J H <Blu-J@users.noreply.github.com> * wip: service manager refactor * wip: Some fixes * wip(fix): Fix the lib.rs * wip * wip(fix): Logs * wip: bins * wip(innspect): Add in the inspect * wip: config * wip(fix): Diagnostic * wip(fix): Dependencies * wip: context * wip(fix) Sorta auth * wip: warnings * wip(fix): registry/admin * wip(fix) marketplace * wip(fix) Some more converted and fixed with the linter and config * wip: Working on the static server * wip(fix)static server * wip: Remove some asynnc * wip: Something about the request and regular rpc * wip: gut install Co-authored-by: J H <Blu-J@users.noreply.github.com> * wip: Convert the static server into the new system * wip delete file * test * wip(fix) vhost does not need the with safe defaults * wip: Adding in the wifi * wip: Fix the developer and the verify * wip: new install flow Co-authored-by: J H <Blu-J@users.noreply.github.com> * fix middleware * wip * wip: Fix the auth * wip * continue service refactor * feature: Service get_config * feat: Action * wip: Fighting the great fight against the borrow checker * wip: Remove an error in a file that I just need to deel with later * chore: Add in some more lifetime stuff to the services * wip: Install fix on lifetime * cleanup * wip: Deal with the borrow later * more cleanup * resolve borrowchecker errors * wip(feat): add in the handler for the socket, for now * wip(feat): Update the service_effect_handler::action * chore: Add in the changes to make sure the from_service goes to context * chore: Change the * refactor service map * fix references to service map * fill out restore * wip: Before I work on the store stuff * fix backup module * handle some warnings * feat: add in the ui components on the rust side * feature: Update the procedures * chore: Update the js side of the main and a few of the others * chore: Update the rpc listener to match the persistant container * wip: Working on updating some things to have a better name * wip(feat): Try and get the rpc to return the correct shape? * lxc wip * wip(feat): Try and get the rpc to return the correct shape? * build for container runtime wip * remove container-init * fix build * fix error * chore: Update to work I suppose * lxc wip * remove docker module and feature * download alpine squashfs automatically * overlays effect Co-authored-by: Jade <Blu-J@users.noreply.github.com> * chore: Add the overlay effect * feat: Add the mounter in the main * chore: Convert to use the mounts, still need to work with the sandbox * install fixes * fix ssl * fixes from testing * implement tmpfile for upload * wip * misc fixes * cleanup * cleanup * better progress reporting * progress for sideload * return real guid * add devmode script * fix lxc rootfs path * fix percentage bar * fix progress bar styling * fix build for unstable * tweaks * label progress * tweaks * update progress more often * make symlink in rpc_client * make socket dir * fix parent path * add start-cli to container * add echo and gitInfo commands * wip: Add the init + errors * chore: Add in the exit effect for the system * chore: Change the type to null for failure to parse * move sigterm timeout to stopping status * update order * chore: Update the return type * remove dbg * change the map error * chore: Update the thing to capture id * chore add some life changes * chore: Update the loging * chore: Update the package to run module * us From for RpcError * chore: Update to use import instead * chore: update * chore: Use require for the backup * fix a default * update the type that is wrong * chore: Update the type of the manifest * chore: Update to make null * only symlink if not exists * get rid of double result * better debug info for ErrorCollection * chore: Update effects * chore: fix * mount assets and volumes * add exec instead of spawn * fix mounting in image * fix overlay mounts Co-authored-by: Jade <Blu-J@users.noreply.github.com> * misc fixes * feat: Fix two * fix: systemForEmbassy main * chore: Fix small part of main loop * chore: Modify the bundle * merge * fixMain loop" * move tsc to makefile * chore: Update the return types of the health check * fix client * chore: Convert the todo to use tsmatches * add in the fixes for the seen and create the hack to allow demo * chore: Update to include the systemForStartOs * chore UPdate to the latest types from the expected outout * fixes * fix typo * Don't emit if failure on tsc * wip Co-authored-by: Jade <Blu-J@users.noreply.github.com> * add s9pk api * add inspection * add inspect manifest * newline after display serializable * fix squashfs in image name * edit manifest Co-authored-by: Jade <Blu-J@users.noreply.github.com> * wait for response on repl * ignore sig for now * ignore sig for now * re-enable sig verification * fix * wip * env and chroot * add profiling logs * set uid & gid in squashfs to 100000 * set uid of sqfs to 100000 * fix mksquashfs args * add env to compat * fix * re-add docker feature flag * fix docker output format being stupid * here be dragons * chore: Add in the cross compiling for something * fix npm link * extract logs from container on exit * chore: Update for testing * add log capture to drop trait * chore: add in the modifications that I make * chore: Update small things for no updates * chore: Update the types of something * chore: Make main not complain * idmapped mounts * idmapped volumes * re-enable kiosk * chore: Add in some logging for the new system * bring in start-sdk * remove avahi * chore: Update the deps * switch to musl * chore: Update the version of prettier * chore: Organize' * chore: Update some of the headers back to the standard of fetch * fix musl build * fix idmapped mounts * fix cross build * use cross compiler for correct arch * feat: Add in the faked ssl stuff for the effects * @dr_bonez Did a solution here * chore: Something that DrBonez * chore: up * wip: We have a working server!!! * wip * uninstall * wip * tes --------- Co-authored-by: J H <dragondef@gmail.com> Co-authored-by: J H <Blu-J@users.noreply.github.com> Co-authored-by: J H <2364004+Blu-J@users.noreply.github.com>
This commit is contained in:
Aiden McClelland
272
core/startos/src/upload.rs
Normal file
272
core/startos/src/upload.rs
Normal file
@@ -0,0 +1,272 @@
|
||||
use std::path::PathBuf;
|
||||
use std::pin::Pin;
|
||||
use std::sync::Arc;
|
||||
use std::task::Poll;
|
||||
use std::time::Duration;
|
||||
|
||||
use axum::body::Body;
|
||||
use axum::response::Response;
|
||||
use clap::Parser;
|
||||
use futures::{FutureExt, StreamExt};
|
||||
use http::header::CONTENT_LENGTH;
|
||||
use http::StatusCode;
|
||||
use tokio::fs::File;
|
||||
use tokio::io::{AsyncRead, AsyncWrite, AsyncWriteExt};
|
||||
use tokio::sync::{watch, OwnedMutexGuard};
|
||||
|
||||
use crate::context::RpcContext;
|
||||
use crate::core::rpc_continuations::{RequestGuid, RpcContinuation};
|
||||
use crate::prelude::*;
|
||||
use crate::s9pk::merkle_archive::source::multi_cursor_file::{FileSectionReader, MultiCursorFile};
|
||||
use crate::s9pk::merkle_archive::source::ArchiveSource;
|
||||
use crate::util::io::TmpDir;
|
||||
|
||||
pub async fn upload(ctx: &RpcContext) -> Result<(RequestGuid, UploadingFile), Error> {
|
||||
let guid = RequestGuid::new();
|
||||
let (mut handle, file) = UploadingFile::new().await?;
|
||||
ctx.add_continuation(
|
||||
guid.clone(),
|
||||
RpcContinuation::rest(
|
||||
Box::new(|request| {
|
||||
async move {
|
||||
let headers = request.headers();
|
||||
let content_length = match headers.get(CONTENT_LENGTH).map(|a| a.to_str()) {
|
||||
None => {
|
||||
return Response::builder()
|
||||
.status(StatusCode::BAD_REQUEST)
|
||||
.body(Body::from("Content-Length is required"))
|
||||
.with_kind(ErrorKind::Network)
|
||||
}
|
||||
Some(Err(_)) => {
|
||||
return Response::builder()
|
||||
.status(StatusCode::BAD_REQUEST)
|
||||
.body(Body::from("Invalid Content-Length"))
|
||||
.with_kind(ErrorKind::Network)
|
||||
}
|
||||
Some(Ok(a)) => match a.parse::<u64>() {
|
||||
Err(_) => {
|
||||
return Response::builder()
|
||||
.status(StatusCode::BAD_REQUEST)
|
||||
.body(Body::from("Invalid Content-Length"))
|
||||
.with_kind(ErrorKind::Network)
|
||||
}
|
||||
Ok(a) => a,
|
||||
},
|
||||
};
|
||||
|
||||
handle
|
||||
.progress
|
||||
.send_modify(|p| p.expected_size = Some(content_length));
|
||||
|
||||
let mut body = request.into_body().into_data_stream();
|
||||
while let Some(next) = body.next().await {
|
||||
if let Err(e) = async {
|
||||
handle
|
||||
.write_all(&next.map_err(|e| {
|
||||
std::io::Error::new(std::io::ErrorKind::Other, e)
|
||||
})?)
|
||||
.await?;
|
||||
Ok(())
|
||||
}
|
||||
.await
|
||||
{
|
||||
handle.progress.send_if_modified(|p| p.handle_error(&e));
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
Response::builder()
|
||||
.status(StatusCode::NO_CONTENT)
|
||||
.body(Body::empty())
|
||||
.with_kind(ErrorKind::Network)
|
||||
}
|
||||
.boxed()
|
||||
}),
|
||||
Duration::from_secs(30),
|
||||
),
|
||||
)
|
||||
.await;
|
||||
Ok((guid, file))
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct Progress {
|
||||
expected_size: Option<u64>,
|
||||
written: u64,
|
||||
error: Option<Error>,
|
||||
}
|
||||
impl Progress {
|
||||
fn handle_error(&mut self, e: &std::io::Error) -> bool {
|
||||
if self.error.is_none() {
|
||||
self.error = Some(Error::new(eyre!("{e}"), ErrorKind::Network));
|
||||
true
|
||||
} else {
|
||||
false
|
||||
}
|
||||
}
|
||||
fn handle_write(&mut self, res: &std::io::Result<usize>) -> bool {
|
||||
match res {
|
||||
Ok(a) => {
|
||||
self.written += *a as u64;
|
||||
true
|
||||
}
|
||||
Err(e) => self.handle_error(e),
|
||||
}
|
||||
}
|
||||
async fn expected_size(watch: &mut watch::Receiver<Self>) -> Option<u64> {
|
||||
watch
|
||||
.wait_for(|progress| progress.error.is_some() || progress.expected_size.is_some())
|
||||
.await
|
||||
.ok()
|
||||
.and_then(|a| a.expected_size)
|
||||
}
|
||||
async fn ready_for(watch: &mut watch::Receiver<Self>, size: u64) -> Result<(), Error> {
|
||||
if let Some(e) = watch
|
||||
.wait_for(|progress| progress.error.is_some() || progress.written >= size)
|
||||
.await
|
||||
.map_err(|_| {
|
||||
Error::new(
|
||||
eyre!("failed to determine upload progress"),
|
||||
ErrorKind::Network,
|
||||
)
|
||||
})?
|
||||
.error
|
||||
.as_ref()
|
||||
.map(|e| e.clone_output())
|
||||
{
|
||||
Err(e)
|
||||
} else {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
fn complete(&mut self) -> bool {
|
||||
match self {
|
||||
Self {
|
||||
expected_size: Some(size),
|
||||
written,
|
||||
..
|
||||
} if *written == *size => false,
|
||||
Self {
|
||||
expected_size: Some(size),
|
||||
written,
|
||||
error,
|
||||
} if *written > *size && error.is_none() => {
|
||||
*error = Some(Error::new(
|
||||
eyre!("Too many bytes received"),
|
||||
ErrorKind::Network,
|
||||
));
|
||||
true
|
||||
}
|
||||
Self { error, .. } if error.is_none() => {
|
||||
*error = Some(Error::new(
|
||||
eyre!("Connection closed or timed out before full file received"),
|
||||
ErrorKind::Network,
|
||||
));
|
||||
true
|
||||
}
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct UploadingFile {
|
||||
tmp_dir: Arc<TmpDir>,
|
||||
file: MultiCursorFile,
|
||||
progress: watch::Receiver<Progress>,
|
||||
}
|
||||
impl UploadingFile {
|
||||
pub async fn new() -> Result<(UploadHandle, Self), Error> {
|
||||
let progress = watch::channel(Progress::default());
|
||||
let tmp_dir = Arc::new(TmpDir::new().await?);
|
||||
let file = File::create(tmp_dir.join("upload.tmp")).await?;
|
||||
let uploading = Self {
|
||||
tmp_dir,
|
||||
file: MultiCursorFile::open(&file).await?,
|
||||
progress: progress.1,
|
||||
};
|
||||
Ok((
|
||||
UploadHandle {
|
||||
file,
|
||||
progress: progress.0,
|
||||
},
|
||||
uploading,
|
||||
))
|
||||
}
|
||||
pub async fn delete(self) -> Result<(), Error> {
|
||||
if let Ok(tmp_dir) = Arc::try_unwrap(self.tmp_dir) {
|
||||
tmp_dir.delete().await?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
#[async_trait::async_trait]
|
||||
impl ArchiveSource for UploadingFile {
|
||||
type Reader = <MultiCursorFile as ArchiveSource>::Reader;
|
||||
async fn size(&self) -> Option<u64> {
|
||||
Progress::expected_size(&mut self.progress.clone()).await
|
||||
}
|
||||
async fn fetch(&self, position: u64, size: u64) -> Result<Self::Reader, Error> {
|
||||
Progress::ready_for(&mut self.progress.clone(), position + size).await?;
|
||||
self.file.fetch(position, size).await
|
||||
}
|
||||
}
|
||||
|
||||
#[pin_project::pin_project(PinnedDrop)]
|
||||
pub struct UploadHandle {
|
||||
#[pin]
|
||||
file: File,
|
||||
progress: watch::Sender<Progress>,
|
||||
}
|
||||
#[pin_project::pinned_drop]
|
||||
impl PinnedDrop for UploadHandle {
|
||||
fn drop(self: Pin<&mut Self>) {
|
||||
let this = self.project();
|
||||
this.progress.send_if_modified(|p| p.complete());
|
||||
}
|
||||
}
|
||||
impl AsyncWrite for UploadHandle {
|
||||
fn poll_write(
|
||||
self: Pin<&mut Self>,
|
||||
cx: &mut std::task::Context<'_>,
|
||||
buf: &[u8],
|
||||
) -> Poll<Result<usize, std::io::Error>> {
|
||||
let this = self.project();
|
||||
match this.file.poll_write(cx, buf) {
|
||||
Poll::Ready(res) => {
|
||||
this.progress
|
||||
.send_if_modified(|progress| progress.handle_write(&res));
|
||||
Poll::Ready(res)
|
||||
}
|
||||
Poll::Pending => Poll::Pending,
|
||||
}
|
||||
}
|
||||
fn poll_flush(
|
||||
self: Pin<&mut Self>,
|
||||
cx: &mut std::task::Context<'_>,
|
||||
) -> Poll<Result<(), std::io::Error>> {
|
||||
let this = self.project();
|
||||
match this.file.poll_flush(cx) {
|
||||
Poll::Ready(Err(e)) => {
|
||||
this.progress
|
||||
.send_if_modified(|progress| progress.handle_error(&e));
|
||||
Poll::Ready(Err(e))
|
||||
}
|
||||
a => a,
|
||||
}
|
||||
}
|
||||
fn poll_shutdown(
|
||||
self: Pin<&mut Self>,
|
||||
cx: &mut std::task::Context<'_>,
|
||||
) -> Poll<Result<(), std::io::Error>> {
|
||||
let this = self.project();
|
||||
match this.file.poll_shutdown(cx) {
|
||||
Poll::Ready(Err(e)) => {
|
||||
this.progress
|
||||
.send_if_modified(|progress| progress.handle_error(&e));
|
||||
Poll::Ready(Err(e))
|
||||
}
|
||||
a => a,
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user