What happens between uploading a file and trusting it to be there?
A Go file-storage project exploring content-defined chunking, content-addressable storage, streaming RPCs, and event-driven replica placement.
File storage becomes a systems problem surprisingly quickly. Repeated content wastes space. Large files put pressure on memory. Metadata needs to stay connected to bytes, and node placement has to adapt when the available machines change.
DFMS separates the streaming data path from metadata and background work. The implementation combines a Go chunk service, PostgreSQL manifests, Redis caching, S3-compatible object storage, and Kafka-driven placement logic.
I built the streaming storage pipeline and connected it to metadata, caching, and background replica-placement logic. The project brings together service communication, content-addressable storage, and the control-plane model for deciding where chunks belong.
Objects are stored by content. A separate ordered manifest turns them back into the file the user uploaded.
a8f2Ac31dBa8f2Ae90bCa8f2AREFERENCED 2xc31dBREFERENCED 1xe90bCREFERENCED 1xThe chunk service feeds a gRPC upload stream into a Rabin-based content-defined chunker through io.Pipe. Chunks get SHA-256 content addresses; the service checks whether each object already exists before uploading missing content with a bounded worker pool.
The tradeoff. Content-defined chunking can preserve matching regions when file contents shift. It also adds boundary-detection work and makes chunking configuration part of the deduplication behavior.
PostgreSQL records file metadata, chunk references, and their order. Redis caches file metadata and manifests. On download, the chunk service retrieves content in manifest order and emits it in streaming frames rather than assembling the entire file in memory.
The tradeoff. Separating metadata and objects makes their responsibilities clear, but introduces consistency boundaries. Partial persistence and event-publication failures need explicit handling.
A weighted consistent-hash ring uses virtual nodes and selects distinct healthy nodes for a chunk. Kafka health events update node eligibility and trigger replacement placement. This is the control-plane side of replication.
The tradeoff. Computing placement is not the same as copying and verifying bytes on independent storage nodes. The current manager records assignments against a shared object store; physical per-node replication remains simulated.
The clearest boundary is replication: the code implements node selection and replacement placement, but not physical copies across separate node endpoints. The chunker also generates a Rabin polynomial per instance, so stable boundaries across service restarts are not guaranteed by the current implementation.
The next engineering steps would be persistent chunking parameters, real per-node transfers with integrity checks, and stronger recovery guarantees around metadata writes and event publication. Storage savings, latency, and durability would need measurement before making numerical claims.
The repository includes setup instructions, implementation details, and the surrounding architecture. Follow the data path, explore a service, or run the project locally.