Skip to main content

Core Concepts & Architecture

This document details the fundamental primitives, memory model, and storage safety guarantees implemented in Lioran S3.

1. Storage Primitives​

┌────────────────────────────────────────────────────────────────────────┐
│                              BUCKET                                    │
│  (e.g., 'assets', quota: 50 GiB, versioning: false, owner: 'admin')    │
│                                                                        │
│   ┌───────────────────────────┐    ┌───────────────────────────┐       │
│   │         OBJECT            │    │         OBJECT            │       │
│   │ Key: "images/logo.png"    │    │ Key: "data/report.pdf"    │       │
│   │ Size: 48,291 bytes        │    │ Size: 1,048,576 bytes     │       │
│   │ ETag: "a4f8b9..."         │    │ ETag: "7c2e10..."         │       │
│   │ SHA-256: "8e9a2b..."      │    │ SHA-256: "f1c4a0..."      │       │
│   │ Type: "image/png"         │    │ Type: "application/pdf"   │       │
│   └───────────────────────────┘    └───────────────────────────┘       │
└────────────────────────────────────────────────────────────────────────┘

Buckets​

A bucket is a top-level container for objects.

  • Naming Rules: 1 to 63 characters consisting of lowercase alphanumeric characters, dots (.), and hyphens (-).
  • Quota (quota_bytes): An optional hard ceiling in bytes. Uploads attempting to exceed the bucket's cumulative stored bytes are rejected with HTTP 507 Insufficient Storage (quota_exceeded).
  • Versioning: Buckets maintain a versioning_enabled boolean flag.

Objects​

An object represents stored data alongside its system metadata.

  • Object Key: A path-like string (e.g. raw/2026/january/data.parquet). Leading slashes (/) are automatically stripped.
  • Content Type: Standard MIME type guessed from filename or explicitly supplied via header.
  • ETag: Entity tag string generated from content digest.
  • SHA-256 Checksum: Cryptographic hash computed incrementally during stream ingestion and stored in RocksDB metadata.

2. Bounded-Memory Streaming Model​

Many traditional storage servers buffer whole incoming or outgoing files in heap memory, risking Out-Of-Memory (OOM) crashes under heavy concurrency or multi-gigabyte transfers.

Lioran S3 enforces bounded memory streaming:

Incoming Socket
      │
      ▼  (Chunk: BASTION_STREAM_CHUNK_KIB = 256 KiB)
[ Fixed Buffer Ring ] ───► Incremental SHA-256 Hash
      │
      ▼
Direct Storage Volume Disk Descriptor
  • Fixed Buffer Size: Streaming buffer chunk size is configured via BASTION_STREAM_CHUNK_KIB (default: 256 KiB, configurable between 64 and 4096 KiB).
  • Constant Memory Footprint: A 10 GB file upload uses the same bounded RAM allocation (~256 KiB per active request connection) as a 10 KB file upload.
  • Byte-Range Requests: Range downloads seek directly to the target offset on disk and stream only the requested byte slice without buffering neighboring content.

3. Atomic Object Lifecycle & Durability​

Every mutating write adheres to a five-stage commit protocol:

1. Stream Ingestion  ───►  Staged to temporary UUID v7 file (.staging/...)
2. Hash Finalization ───►  Verify stream length and compute SHA-256 digest
3. Physical Disk Sync───►  Flush buffer and execute fsync (in strict mode)
4. Metadata Commit   ───►  Atomically write metadata record to RocksDB
5. Promotion         ───►  Atomically rename staging file into object volume

Crash Invariants​

  • No Partial Exposure: Interrupted transfers leave temporary files in .staging/, which are invisible to GET, HEAD, or LIST operations and swept during garbage collection.
  • No Orphaned References: A metadata record is never committed unless the underlying physical payload has been successfully flushed to disk.

4. Cursor-Based Pagination​

Traditional offset-based pagination degrades rapidly as bucket sizes scale into millions of records. Lioran S3 uses opaque cursor pagination:

  • Parameters:
    • prefix: Filter keys matching a prefix string (e.g. users/1024/).
    • limit: Maximum records to return per batch (1 to 1000, default: 100).
    • cursor: Opaque token representing the scan position in the RocksDB key index.
  • Response:
    • objects: Array of object metadata.
    • next_cursor: Token for the next page, or null if exhausted.
    • has_more: Boolean indicating whether additional items remain.

5. Storage Safety Guardrails​

Bucket Quota Enforcement​

Calculates current bucket byte volume against the configured quota_bytes. Quota limits are verified before standard single PUT uploads and during multipart upload assembly.

Disk Low-Watermark Protection​

The server inspects available host filesystem capacity before accepting mutating writes:

  • BASTION_MIN_FREE_SPACE_BYTES: Minimum required free host storage in bytes (default: 512 MiB).
  • BASTION_MIN_FREE_SPACE_PERCENT: Optional percentage threshold (e.g. 5.0 for 5% free).
  • If host storage drops below the watermark, mutating operations are rejected with HTTP 507 (storage_capacity_exhausted), protecting system stability.