systems · sub-star

nexus_db

embedded LSM key-value store in C++17, with a live telemetry dashboard.

readme

An embedded key-value store written in C++17, taking its architecture from LevelDB and RocksDB: writes land in an in-memory skip list, get durability from a write-ahead log, and are flushed to immutable SSTables on disk. On top of the engine sits a FastAPI layer talking to the C++ core over a ctypes FFI, and a React dashboard showing live telemetry.

signal log

• MemTable is a probabilistic skip list, giving O(log n) insert and point lookup with no tree-rebalancing overhead.
• write-ahead log appends every operation to disk before the MemTable is touched, recording intent ahead of effect.
• a full MemTable is handed to a worker thread, which writes it to an immutable SSTable in one sequential pass while the writer carries on against a fresh one.
• every file carries a Bloom filter and a sparse index, so a lookup rules out the files that cannot hold the key and seeks straight into the one that can: a miss inside the key range went from 33 ms to 0.7 us.
• levelled compaction merges files down the tree, keeping only the newest version of each key. 200k keys written five times over sit in 25.5 MB across 15 files rather than 69.2 MB across 55, at 2.9x write amplification.
• a MANIFEST names every live file and its level, written through a temp file and a rename, so a crash mid compaction leaves outputs nothing points at and inputs that are still listed.
• reads hold the lock only to look at the memtables and take references to the files they will search, then scan with no lock held: 82k reads/s on one thread, 662k on eight, where the old design got slower with every thread added.
• deletes write a record with a tombstone flag rather than removing data, and the space comes back only when compaction reaches it. a level thick with tombstones earns a compaction of its own, or a delete can sit above its value indefinitely.
• SSTables are re-adopted on open and the file counter resumes past the highest index, so reopening a database does not shadow or overwrite what is already on disk.
• crash recovery: the WAL is replayed on open, and a torn final record is truncated so writes appended after it stay reachable. tested by forking a child that exits without running destructors.
• 1M writes at 180k ops/s into 30 files over 3 levels, a random read at 12.6 us, and a missing key at 0.2 us.
• three-layer stack: C++17 engine, Python/FastAPI REST API over ctypes FFI, React/TypeScript dashboard.
• not yet built: range scans, a block cache, checksums, and an fsync on the log rather than a flush.

built with