Flume: LSM-Tree Storage Engine

Flume is an enterprise-grade LSM-tree storage engine built from scratch to handle heavy write amplification challenges under significant continuous disk load. The architecture leverages a Write-Ahead Log (WAL) to guarantee durability, combined with multi-layered L0 to L6 leveled compaction mechanics to manage high-throughput data volumes and reduce read amplification bottlenecks over time.

The core challenge centered on eliminating deep race conditions between active user readers and asynchronous background compaction engines. By implementing an MVCC snapshot strategy, sequence-number recency tracking, and fine-grained reference-counted SSTable lifetimes, I resolved multiple hidden concurrency deadlocks. The final system incorporates Prometheus observability, structured slog instances, and a native HTTP JSON API.