Skip to main content

Engineering Proof & Chaos Benchmark

This report documents the official empirical soak and chaos-resilience benchmark run for Lioran S3 (Lioran Bastion).

To provide complete transparency and reproducibility, all benchmark metrics, telemetry logs, crash sequences, and integrity verification audits are published below with direct access to raw immutable log artifacts.

Benchmark Executive Summary​

┌───────────────────────────┬───────────────────────────┬───────────────────────────┐
│        100.52 GiB         │        5 / 5 Crashes      │          0 Mismatches     │
│       Total Ingested      │     Recovered Seamlessly  │    188 Verified Samples   │
├───────────────────────────┼───────────────────────────┼───────────────────────────┤
│        87.3 MB/s          │        281.7 MB/s         │         924.2 MB          │
│    Avg Write Throughput   │    Avg Read Throughput    │         Peak RAM RSS      │
└───────────────────────────┴───────────────────────────┴───────────────────────────┘
  • Run Identifier: 20260930T111856-d14352
  • Engine Version: Lioran S3 V1 Pre-Alpha (bastion-server v0.1.0)
  • Organization: Lioran Developer Solutions (LDS) / Lioran Group
  • Founder & CTO: Swaraj Puppalwar
  • Benchmark Date: September 30, 2026
  • Workload Volume: 100.52 GiB (107,931,062,250 bytes)
  • Total Objects Ingested: 6,050 objects across 15 buckets
  • Reference Payload: test2.mp4 (17.84 MB / 17,840,175 bytes, SHA-256: d6617a009c0c6c9aebf7398d43cad6d1985ddc1b9ab0479e2ea977362b8af5b0)
  • Worker Concurrency: 32 concurrent write workers, 32 concurrent read workers
  • Chaos Disruptions: 5 SIGKILL hard process kills injected during active streaming I/O
  • Post-Chaos Integrity Verification: 188 random samples across all 15 buckets; 0 missing objects, 0 bit rot errors, 0 SHA-256 checksum mismatches.

Detailed Performance Telemetry​

1. Ingestion Phase (Write Workload)​

During the write phase, 32 worker threads streamed 100.52 GiB of video payload data into 15 isolated buckets. During this ingestion, the storage daemon was subjected to 5 unannounced process crashes.

MetricMeasured ValueNotes
Total Ingestion Time1236.1s (20m 36s)Includes crash recovery and worker reconnection latencies
Successful Object Writes6,050Fully committed to disk and RocksDB metadata
Unexpected Write Failures0Zero unexplained data corruptions or dropouts
Chaos-Expected Drops155Transient HTTP connection resets during process kills
Average Write Throughput87.3 MB/sAggregate across all 32 workers throughout full run
Healthy Ingest Throughput88.5 MB/sSteady-state throughput excluding restart delays
Peak 5-Second Burst663.2 MB/sPeak ingestion during optimal memory pipeline caching
Latency p504,316.5 msMedian duration for 17.84 MB object upload
Latency p9012,823.8 msUpload duration under high I/O saturation
Latency p9513,386.9 msUpload duration under background compaction
Latency p9915,016.0 msUpload duration spanning a crash/reconnect event

:::info Chaos Connection Handling The 155 write drops recorded were strictly due to inflight HTTP socket disconnections during the 5 unannounced process termination signals (SIGKILL). The client driver correctly detected broken pipes, initiated exponential backoff, re-authenticated, and resumed writes with zero uncommitted orphan artifacts. :::

2. Egress Phase (Read Workload) & Performance Honesty​

Following write ingestion, 32 worker threads executed a randomized read audit across the full 6,050-object dataset.

MetricMeasured ValueAnalysis
Total Read Duration383.1s (6m 23s)6,050 read operations completed
Unexpected Read Errors0All 6,050 requested objects read with 200 OK
Average Read Throughput281.7 MB/sSustained concurrent read throughput
Peak 5-Second Read Burst414.1 MB/sPeak read speed during warm OS page cache
Latency p501,879.5 msMedian retrieval time for 17.84 MB object
Latency p902,488.8 ms90th percentile retrieval time
Latency p952,738.2 ms95th percentile retrieval time
Latency p998,779.2 ms99th percentile retrieval time under heavy thread contention

:::warning Performance Honesty: 500 MB/s Read Target Missed We explicitly targetted a sustained read throughput of 500 MB/s for this single-node benchmark. The measured average was 281.7 MB/s (peak 414.1 MB/s), falling short of our 500 MB/s target.

Root Cause Analysis:

  1. Thread Contention in Range Seeking: Under 32 concurrent read streams, file descriptor seek locks and simultaneous async file handle cloning on the storage mount introduced micro-contention.
  2. Buffer Copying: Streaming from LocalObjectStore into Axum response bodies traversed an intermediate Tokio memory buffer before socket delivery.

Optimizing zero-copy socket streaming via splice/sendfile and lockless file descriptors is scheduled for the upcoming releases. :::

Chaos Engineering & Crash Recovery​

To validate the crash consistency guarantees of Lioran S3's two-phase atomic write pipeline, 5 SIGKILL hard kills were injected at precise milestone byte thresholds during active writes.

sequenceDiagram
    autonumber
    actor W as Write Workers (x32)
    participant S as Staging Area (.staging/)
    participant F as Payload Store (objects/)
    participant R as RocksDB (Metadata)
    participant C as Chaos Engine

    W->>S: Stream payload bytes into .staging/tmp
    Note over S: Continuous SHA-256 computation
    C--xS: Injected Crash #1 (at 10.00 GB)
    Note over S: Incomplete staging file left uncommitted
    Note over R: Daemon restarts & recovers RocksDB WAL
    W->>S: Retry upload -> Stream to new .staging/tmp
    S->>S: fsync (sync_all) completed
    S->>F: Atomic filesystem rename (promote)
    F->>R: Atomically insert ObjectRecord into RocksDB
    R-->>W: 200 OK Upload Successful

Crash Events Timeline​

  1. Crash #1 (10.00 GB Ingested)
    • Trigger: Ingest threshold reached 10,737,418,240 bytes.
    • Impact: Daemon terminated abruptly with SIGKILL. Active HTTP requests dropped.
    • Recovery: RocksDB WAL replayed in 142 ms. Staging files isolated. Daemon resumed service in < 1.2s.
  2. Crash #2 (27.02 GB Ingested)
    • Trigger: Ingest threshold reached 29,012,852,736 bytes.
    • Impact: 32 write streams interrupted mid-packet.
    • Recovery: Clean restart; 0 partially written files registered in metadata.
  3. Crash #3 (45.01 GB Ingested)
    • Trigger: Ingest threshold reached 48,328,294,400 bytes.
    • Impact: Heavy compaction and flush in progress during kill.
    • Recovery: RocksDB column families reconciled; zero SST corruption.
  4. Crash #4 (68.00 GB Ingested)
    • Trigger: Ingest threshold reached 73,014,444,032 bytes.
    • Impact: 4 workers completing atomic renames during kill.
    • Recovery: Filesystem and metadata remained 100% synchronized.
  5. Crash #5 (85.00 GB Ingested)
    • Trigger: Ingest threshold reached 91,268,055,040 bytes.
    • Impact: Final pre-completion stress interruption.
    • Recovery: All previously committed objects verified readable.

Cryptographic Data Integrity Verification​

Following all 5 crashes and the completion of 100.52 GiB of ingestion, an automated cryptographic integrity verification suite was executed against the storage engine.

======================================================================
                 INTEGRITY AUDIT REPORT (188 SAMPLES)
======================================================================
Total Buckets Sampled:               15 / 15
Total Objects Inspected:             188
Verification Method:                 Full Body Streaming SHA-256 Digest
Expected SHA-256:                    d6617a009c0c6c9aebf7398d43cad6d...
Payload Byte Matches:                188 / 188 (100.0%)
Missing Objects:                     0
Corrupt / Truncated Objects:         0
Checksum Mismatches:                 0
Bit Rot Detected:                    0
======================================================================
FINAL INTEGRITY STATUS:              PASS (ZERO DEFECTS)
======================================================================

Memory Utilization & Stability​

Lioran S3 enforces bounded streaming memory buffers across all worker threads. Payload bytes are streamed directly through fixed-size chunks (256 KiB) rather than loaded into memory.

  • Initial RSS: 64.2 MB
  • Peak RSS Under 32 Concurrent Streams + Heavy Compaction: 924.2 MB (969,080,832 bytes)
  • Final Settled RSS: 231.7 MB (243,003,392 bytes)
  • Memory Leaks Detected: Zero heap growth after garbage collection and buffer reuse.

Raw Proof Artifacts & Verification Logs​

To verify these results independently, inspect the complete set of raw log files, JSON metric lines, crash traces, and system metrics published from this benchmark run:

Artifact NameDescriptionDirect View / Download
summary.txtComplete human-readable benchmark summary and audit logsummary.txt
summary.jsonMachine-readable JSON summary of all aggregate metricssummary.json
crashes.logExact timestamped log of all 5 injected SIGKILL crash eventscrashes.log
errors.logDetailed error trace of transient connection drops during chaoserrors.log
integrity.logSHA-256 cryptographic verification logs for all 188 object samplesintegrity.log
manifest.jsonlJSON Lines manifest of all 6,050 ingested objects across 15 bucketsmanifest.jsonl
metrics.jsonlHigh-frequency time-series telemetry recorded throughout the runmetrics.jsonl
reads.logLatency and throughput breakdown for all 6,050 read requestsreads.log
recovery.logDaemon recovery, WAL replay, and database reconciliation logsrecovery.log
server.logFull standard output and trace logs emitted by bastion-serverserver.log
test.logComplete orchestrator execution log for the chaos test runnertest.log
writes.logPer-object write durations, latency quantiles, and status codeswrites.log

Next Steps​