Skip to main content

Durability & Crash Safety

Lioran S3 provides deterministic write durability guarantees engineered to prevent data corruption during unexpected server restarts or hardware power failures.

1. Durability Modes​

Write durability is controlled via the BASTION_DURABILITY environment variable:

# Strict durability (Default, recommended for production)
BASTION_DURABILITY=strict

# Balanced writeback caching (High-throughput for battery-backed storage)
BASTION_DURABILITY=balanced

Mode Comparison​

ModeDisk Sync GuaranteeCrash SafetyTarget Workloads
strict (Default)Calls explicit fsync (sync_all) on physical payload files before committing metadata to RocksDB.Guaranteed crash consistency. In the event of sudden power failure, committed objects are fully preserved on disk.General production, critical archives, user documents, transactional data.
balancedOmits per-write synchronous disk flush, delegating sync operations to OS filesystem writeback caching.Data committed to metadata may be lost if power fails before OS flushes dirty pages.Ephemeral media transcoding, high-IOPS scratch storage, systems with battery-backed RAID/NVMe controllers.

2. The 5-Stage Atomic Commit Protocol​

To ensure that partially uploaded objects never become visible to readers, all writes execute through a 5-stage pipeline:

1. Stream Ingestion  ───►  Streamed to isolated temporary staging file (.staging/<uuid-v7>.tmp)
2. Checksum Compute  ───►  Incremental SHA-256 hash calculated during transfer
3. Disk Flush & Sync ───►  Flush buffer and call sync_all() (in strict mode)
4. Metadata Commit   ───►  Atomically insert metadata record into RocksDB
5. Promotion         ───►  Atomically rename staging file into /objects/ hierarchy
6. Acknowledgement   ───►  Return HTTP 200 OK / 201 Created to client

Invariants Maintained:​

  • A PUT request is never acknowledged with HTTP 200/201 until both payload bytes and metadata are committed.
  • Interrupted or aborted uploads leave files in .staging/, which are invisible to GET, HEAD, or LIST queries.
  • RocksDB metadata never references a nonexistent payload on disk.

3. Graceful Shutdown​

Upon receiving SIGINT (Ctrl+C) or SIGTERM (Docker stop):

  1. The server stops accepting new HTTP connections.
  2. In-flight requests drain within the configured stop_grace_period (default: 30s).
  3. Open file descriptors are flushed and closed.
  4. RocksDB handles execute clean memtable flushes before process exit.