The problem
Why this system needed to exist.
Moving operational data into object storage is easy in the happy path. The real design work appears when a process stops mid-batch, a resume token gets ahead of the data, a historical range needs to be replayed, or a handful of records need targeted repair.
A useful ingestion platform needs one operational model for snapshot, streaming, scheduled batch, validation, restart, backfill, and patch workflows.
The system
How the pieces work together.
The service supports initial snapshots, continuous MongoDB change streams, scheduled collection loads, and targeted patch commands. Records are buffered into partitioned GZIP JSONL objects and written either to Amazon S3 or a local object store.
Validation metadata and SHA-256 checksums are written with each object. Recovery state advances only after both the data object and its metadata are durable, so a restart can replay from the previous safe boundary.