Logflow: Distributed Log Aggregator

Logflow is a fault-tolerant, high-throughput distributed log aggregation engine built to ingest, queue, and dispatch massive event streams reliably. The architecture utilizes pluggable gRPC sources and sinks to maintain reliable streaming contexts across distributed node topologies under severe burst loads.

To ensure durability against unexpected node crashes, I built a persistent FileChannel featuring a custom CRC32 Write-Ahead Log (WAL) that enables at-least-once delivery guarantees. This is backed by a spillable memory channel that gracefully offloads up to 98,000 burst events to disk without dropping packets. The edge topology includes priority-based failover hooks, load-balancing sink groups, dynamic token authentication via mTLS, and graceful staged shutdown handling using SIGTERM traps.