Update to Headcount: Technology

{
  "description": "Moves data from one system to another without losing it or corrupting it — profiling the source before mapping, deciding between big-bang and parallel-run cutover, reconciling counts and values rather than assuming, handling the records that will not map cleanly, and planning a rollback that is actually executable. Use this to plan or run a migration, assess a migration plan someone else wrote, work out why a completed migration is producing wrong numbers, or size how long one will really take.",
  "included_files": [],
  "name": "data-migration",
  "skill_md_contents": "---\nname: data-migration\ndescription: Moves data from one system to another without losing it or corrupting it — profiling the source before mapping, deciding between big-bang and parallel-run cutover, reconciling counts and values rather than assuming, handling the records that will not map cleanly, and planning a rollback that is actually executable. Use this to plan or run a migration, assess a migration plan someone else wrote, work out why a completed migration is producing wrong numbers, or size how long one will really take.\n---\n\n# Data migration\n\nMigrations are estimated as a data-movement problem and turn out to be a data-quality problem. The\ntransfer is the easy part; discovering what the old system actually contains is where the time goes.\n\n## Profile the source before you map anything\n\nLook at the real data, not the schema and not what anyone tells you it contains. Every legacy\nsystem has fields used for something other than their name, free-text where an enumeration was\nintended, duplicates that both look canonical, and records predating a rule everyone believes has\nalways applied.\n\nCount distinct values, null rates, format variance, and outliers on every field you intend to move.\nThis is the single highest-return activity in a migration and the one most often skipped in favor\nof starting the mapping.\n\n**Decide what not to move.** Migrating everything is the default and rarely the right answer.\nArchive what has no live use and move a clean subset — it shrinks the mapping, the reconciliation,\nand the cutover window all at once.\n\n## Write the mapping down field by field, including the ugly parts\n\nFor each target field: source field, transformation, what happens when the source is empty, and\nwhat happens when it does not fit. That last column is the one that decides how the migration goes.\n\n**Records that will not map cleanly need a decision, not a default.** A row silently dropped, a\nrequired field filled with a placeholder, or a truncated value is a defect discovered months later\nby someone who trusted the number. Route exceptions to a list a human works through, and count\nthem.\n\n## Rehearse on a full copy, more than once\n\nA rehearsal on a sample proves the mapping compiles. A rehearsal on a full copy proves the timing,\nfinds the pathological records, and gives you a real number for the cutover window.\n\nExpect several rehearsals. Each one should end with a reconciliation and a defect list, and the\nlast one should be clean and timed. Going into cutover having never completed a full run at\nproduction volume means the cutover is the first full run.\n\n## Choose the cutover style deliberately\n\n- **Big-bang** — stop, migrate, start on the new system. Simplest to reason about, and the whole\n  risk lands in one window. Viable when the window is affordable and rollback is real.\n- **Parallel run** — both systems live, one authoritative. Safer and much more expensive, because\n  something has to keep them in step and someone has to reconcile the divergence daily.\n- **Phased** — by entity, region, or business unit. Reduces blast radius and creates the hardest\n  problem in migration: records that reference each other across the boundary while it exists.\n\nWhichever you pick, name the point of no return and what the rollback is on each side of it. **A\nrollback that has never been executed is not a rollback**, and after the new system has taken live\nwrites, reverting means merging them back — which is a second migration nobody planned.\n\n## Reconcile on counts and on values\n\nRow counts alone catch a truncated load and nothing else. Reconcile control totals for anything\nfinancial, distributions for key fields, and a sample compared record by record. Then have the\npeople who use the data check the records they know by heart — they find things no query will.\n\nReconcile again a week after cutover. Problems in the ongoing integrations that replaced the\nmigration usually show up then rather than on the day.\n\n## Plan for the long tail\n\nMigrations are declared done and then produce findings for months. Keep the source system readable\nfor longer than feels necessary, keep the mapping and the exception lists, and keep someone\nassigned. The alternative is a question about a historical record that nobody can answer because\nthe old system was decommissioned on schedule.\n\n## Never\n\n- Map from the schema rather than from the data.\n- Let a record fail to map and be dropped without landing on a list someone works.\n- Cut over without a full-volume rehearsal that completed and reconciled.\n- Decommission the source system before the reconciliation window has closed.\n"
}