Integrations and data migration

A migration is not a script. It is a plan with a rehearsal, a verification step and a way back. The script is the smallest part of it, and the part most likely to be rewritten twice before the real run.

The rehearsal happens on a full copy of production, timed end to end, because the duration decides the shape of the cutover: a job that takes twenty minutes allows a maintenance window, one that takes eleven hours does not, and the difference has to be known before a date is promised rather than discovered on the night. Verification is per row, not per table count: totals match easily while individual records are silently truncated, re-encoded or reassigned to the wrong parent.

The rollback plan is what makes the cutover a decision instead of a leap. It means the old system stays readable, the new one can be abandoned without losing what was written to it, and someone has agreed in advance which symptom triggers that call.

The largest migration behind this practice moved 40 million records with no data loss. Integrations follow the same logic at a smaller scale: idempotent operations so a retry cannot create a duplicate, explicit handling of the partial failure, and a queue that survives the other side being down for an afternoon. Systems that talk to each other fail in ways neither of them fails alone, and that is the part worth designing.

The order of operations is where most integrations quietly go wrong. Two systems that each behave correctly on their own can still produce a state neither of them would produce alone, because the message arrived twice, or arrived out of order, or arrived while the record it referred to was still being written. Designing for that is not pessimism, it is the normal case at any real volume.