Selected work

Public reference · Data platform

MongoDB to PostgreSQL CDC with SCD2 History

A queue-first snapshot and change-data-capture service that preserves event order, durable resume state, complete SCD2 history, and a current PostgreSQL view.

Type
CDC reference implementation
Role
Architect / engineer
Status
Public and runnable
Focus
Ordering, durability, history, recovery
PythonMongoDB change streamsPostgreSQLSCD2DockerIdempotencyResume stateAutomated tests
System map

The operating path.

A deliberately simplified architecture view. The case study below explains where the important guarantees and decisions live.

  1. 01MongoDB
  2. 02Snapshot boundary
  3. 03Snapshot + CDC
  4. 04Durable queue
  5. 05Ordered SCD2
  6. 06History + current view

The problem

Why this system needed to exist.

Taking a snapshot and then starting a change stream creates a race: writes can occur between the two operations. Resume tokens can also be acknowledged too early, leaving a gap after a crash.

The destination needs more than a current copy. It must preserve ordered history, deletes, replay safety, and a clean current-state interface for downstream consumers.

The system

How the pieces work together.

The service captures an operation-time boundary, performs the initial snapshot, and continues from the corresponding MongoDB change stream. Snapshot rows and live events land in a durable PostgreSQL queue.

Resume state advances only after PostgreSQL accepts an event. A deterministic transform consumes the queue in MongoDB event order, writes SCD2 versions, closes prior versions, handles deletes, and exposes the current record through a view.

01

Queue before acknowledgement

An event is considered durable only after the PostgreSQL transaction commits. A restart can replay safely instead of skipping data that never reached the destination.

02

Ordered SCD2

Events are processed in source order so history versions close and open deterministically, including update and delete behavior.

03

Runnable proof

The public repository includes synthetic records, Docker Compose, demo mutations, inspection SQL, and a test suite without exposing the original production schema.

Source boundary

This public repository is an anonymized reference implementation based on a production pipeline. It uses synthetic customer records and contains no client data, credentials, internal schema, or proprietary configuration.

A system like this on your roadmap?

Let’s make the hard parts explicit.

Discuss the project