Contacts, tickets, deals, conversations and marketing tags had to move from Freshdesk and SalesManago into HubSpot. Half a million records, three source shapes, one target schema, and no tolerance for silent loss.
Rather than one long-running function per batch, SQS orchestrated the whole migration. Each message was a small, idempotent unit that could fail and be retried without touching anything already done. The Lambda timeout stopped being a limit on how much could be migrated at once.
More moving parts and more infrastructure to reason about, in exchange for a job that could survive being interrupted at any point.
Nothing went straight from source to target. Every record landed in Atlas first, was validated against the target schema, then pushed onward. That gave a queryable record of what had moved, what had failed, and why.
An extra write per record and a second datastore to operate, worth it for being able to answer "did record X make it?" without calling three APIs.
Rate limits, malformed source data and transient API errors were expected rather than exceptional. Each failure was recorded with its cause and re-queued under a backoff, so the run degraded in throughput instead of stopping.
I would build the reconciliation reporting before the migration logic next time. I built it last, and it was the first thing people asked about.