Multipart Upload Pipeline

For files larger than 50 MiB, Lioran S3 provides a resumable multipart upload pipeline supporting concurrent part chunking, automatic per-part retries, and atomic server-side assembly.

---

1. High-Level Automated Upload (uploadMultipart)

The easiest way to upload large files is using bucket.uploadMultipart():

import { BastionClient } from "@liorans3/driver";

const client = new BastionClient("bastion://admin:YOUR_PASSWORD@127.0.0.1:27118");
const bucket = client.bucket("datasets");

const metadata = await bucket.uploadMultipart("large-dataset.tar.gz", "./data/dataset.tar.gz", {
  contentType: "application/gzip",
  partSize: 32 * 1024 * 1024, // 32 MiB chunks (minimum 5 MiB)
  concurrency: 4,              // 4 parallel worker threads
  maxRetries: 3,               // Retry failed chunks up to 3 times
  customMetadata: {
    dataset_version: "v2.4",
    pipeline_run: "nightly-104",
  },
  onProgress: (p) => {
    console.log(`Progress: ${p.percent}% (${p.bytesUploaded}/${p.totalBytes} bytes, Part ${p.partNumber}/${p.totalParts})`);
  },
});

console.log(`Committed ${metadata.key} (${metadata.size_bytes} bytes, SHA-256: ${metadata.sha256_checksum})`);

---

2. Multipart Options Reference

Option	Type	Default	Description
partSize	number	33554432 (32 MiB)	Part size in bytes. Must be at least 5 MiB (5242880 bytes).
concurrency	number	4	Number of simultaneous chunk uploads.
contentType	string	"application/octet-stream"	MIME content type of the assembled object.
fileName	string	—	Original filename metadata.
customMetadata	Record<string, string>	—	Custom key-value metadata saved in RocksDB.
maxRetries	number	3	Maximum retry attempts per part before aborting.
onProgress	(p: MultipartUploadProgress) => void	—	Callback reporting bytes, percentage, and chunk metrics.
signal	AbortSignal	—	Signal to cancel in-flight part uploads.

---

3. Low-Level Multipart Session Management

For specialized workflows requiring manual part coordination:

// 1. Initialize session
const handle = await bucket.createMultipartUpload("custom-upload.bin", {
  partSize: 10 * 1024 * 1024, // 10 MiB
  contentType: "application/octet-stream",
});

console.log("Upload ID:", handle.uploadId);

// 2. Upload individual parts (1-indexed)
const part1Stream = fs.createReadStream("./part1.bin");
const p1 = await handle.uploadPart(1, part1Stream, { contentLength: 10 * 1024 * 1024 });
console.log(`Part 1 ETag: ${p1.etag}`);

const part2Stream = fs.createReadStream("./part2.bin");
const p2 = await handle.uploadPart(2, part2Stream, { contentLength: 5 * 1024 * 1024 });

// 3. Check session status
const status = await handle.status();
console.log("Verified Parts:", Object.keys(status.parts).length);

// 4. Finalize and assemble
const committed = await handle.complete();
console.log("Committed:", committed.key);

Aborting a Session

To cancel an incomplete multipart upload and purge staged parts:

await handle.abort();

---

4. Resuming Interrupted Uploads

If a large transfer disconnects or fails mid-stream, you can resume without re-uploading completed chunks:

import { MultipartUploadHandle, executeMultipartUpload } from "@liorans3/driver";

// Create handle for existing session
const handle = new MultipartUploadHandle(
  {
    upload_id: "0192a7b8-c9d0-7000-8000-000000000001",
    bucket: "datasets",
    key: "large-dataset.tar.gz",
    recommended_part_size: 32 * 1024 * 1024,
    max_parallel_parts: 4,
    expires_at: "",
  },
  client["http"]
);

// Resumes upload: queries server for verified parts and uploads only missing parts
const committed = await executeMultipartUpload(handle, "./data/dataset.tar.gz", {
  concurrency: 4,
  onProgress: (p) => console.log(`Resume progress: ${p.percent}%`),
});