Doolta
Metarc
GitHub →An experimental archive format that compresses a code corpus's structure before its bytes, with gains measured on real open-source repositories.
Project Details
- Real problemClassic compressors (gzip, zstd) operate at the byte level and miss semantic redundancy across files in a corpus: duplicated licenses, boilerplate, repetitive JSON or logs, recurring code tokens.
- Structural choicesA chain of reversible, versioned transforms (dedup, normalization), driven by a cost/benefit heuristic that can explicitly choose to do nothing rather than degrade. Files are grouped into zstd blocks by extension to exploit cross-file redundancy, with a SQLite catalog kept separate from the blob store.
- What it proves todayAn actively developed, already functional and tested project (CI, coverage, continuous audit), with measured and reproducible gains on real open-source repositories (Kubernetes, React, Redis, NumPy): 3 to 9% over tar+zstd, up to 22% on a highly redundant personal corpus.
- Associated offerBespoke engagement. Get in touch.
Compress the structure before the bytes: rather than treating a file corpus as an undifferentiated byte stream, Metarc identifies redundancy across files before handing off to a classic compressor, with gains measured on real open-source repositories rather than just claimed.
This project demonstrates rigorous algorithmic exploration: every optimization is benchmarked before being adopted, and the system knows how to say no to a transform that doesn’t pay off.