A DNS change that looked routine at 2:14 PM can take a production API offline by 2:16 PM, and the rollback procedure you have in place at that moment determines whether the outage lasts two minutes or two hours. This article walks through what an effective DNS rollback process looks like, why TTL values wreck naive rollback assumptions, and how to structure change management so reverting a bad record is a five-minute task instead of a fire drill.
Why DNS Rollbacks Are Harder Than They Look
Rolling back a DNS change sounds simple: change the record back to what it was. In practice, three things get in the way.
First, DNS is cached at every layer between your authoritative nameserver and the end user — recursive resolvers, ISP caches, browser DNS caches, even corporate proxy servers. A record change propagates outward, and a rollback has to propagate the same way, at the mercy of whatever TTL was set on the original (bad) record. If someone pushed a change with a 24-hour TTL instead of the usual 300 seconds, your “instant” rollback won’t actually reach a meaningful share of resolvers for up to a day.
Second, most teams don’t actually know what the previous record looked like. Registrar dashboards like GoDaddy, Cloudflare, or Route 53 show current state, not history. Unless someone screenshotted the zone file or exported it before the change, “revert” becomes “guess.”
Third, a single DNS change is rarely atomic. Migrating a subdomain to a new CDN might touch an A record, a CNAME, and a TXT record for domain verification in the same session. Rolling back the A record while leaving the TXT record in place can leave a service in a half-configured, worse-than-broken state.
Building a Rollback-Ready Change Process
The fix isn’t a faster rollback button — it’s a change process that makes rollback boring. Five steps cover most environments:
1. Export the zone before every change. A `dig axfr` where zone transfers are enabled, or a full zone export from your DNS provider’s API (Route 53’s `ListResourceRecordSets`, Cloudflare’s `GET /zones/{id}/dns_records`), gives you a timestamped snapshot. Store it somewhere outside the DNS provider itself — a git repo or S3 bucket — because if the provider account is the thing that’s compromised or misconfigured, you don’t want your only backup living there too.
2. Lower the TTL 24-48 hours before a planned change, not during it. If a record currently has a 3600-second TTL and you’re migrating it Friday, drop it to 300 seconds on Wednesday. That way both the change and any rollback propagate fast. Raise it back to normal afterward — permanently low TTLs increase query load on your authoritative nameservers and can add latency for CDN-fronted domains.
3. Change one record type per deployment window where possible. If you need to change A, CNAME, and TXT records together, document the dependency explicitly so a partial rollback isn’t attempted by someone who doesn’t know the TXT record was load-bearing.
4. Set a rollback trigger before you make the change, not after something breaks. “If error rate on /api exceeds 2% for 5 minutes, revert” is a decision made with a clear head, not during an incident.
5. Verify the rollback the same way you’d verify the original change — query the authoritative nameserver directly with `dig @ns1.yourprovider.com yourdomain.com`, not just your local resolver, which may be serving a cached answer that makes it look like the rollback worked when it hasn’t propagated at all.
What Actually Happens During a Bad Change: A Timeline
A mid-size SaaS company migrates `app.example.com` from an old load balancer to a new one. The engineer updates the CNAME at 10:02 AM and forgets that the old CNAME had a 14400-second (4-hour) TTL, inherited from a config nobody had touched since 2021. At 10:04 the new backend starts returning 502s under production load — it hadn’t been warmed up. The engineer reverts the CNAME at 10:06.
Here’s the part that catches teams off guard: reverting the record at 10:06 does almost nothing for the next several hours. Resolvers that cached the old CNAME before 10:02 are fine. Resolvers that cached the new (broken) CNAME between 10:02 and 10:06 will keep serving it until their individual TTL expires — which, at 4 hours, means some users are still hitting the broken backend at 2 PM. The only real fix at that point is making the “new” backend actually work, because the rollback can’t out-run the TTL.
This is the myth worth busting directly: a DNS rollback is not instant, and treating it as an emergency stop button is a mistake that costs teams hours during live incidents. The actual emergency stop is a low TTL set in advance, combined with keeping the old infrastructure running and reachable until the new one is proven stable — don’t decommission the old load balancer the moment you cut over.
Common Mistakes in DNS Rollback Planning
A few patterns show up repeatedly in postmortems. Teams decommission the old target immediately after cutover, so even a fast rollback points to nothing. Teams trust the registrar’s UI history instead of an independent export, then discover the UI only retains 30 days of change logs when they need something from 90 days back. And teams test the rollback plan only in theory — nobody has actually run `dig` against the authoritative server during a drill, so the first real test happens during a live outage. Keeping a baseline snapshot of known-good DNS state solves the first two problems at once, and running periodic drills solves the third.
FAQ
How long does a DNS rollback take to fully propagate?
It depends entirely on the TTL of the record being replaced, not the TTL of the new one. A record cached with a 3600-second TTL at the moment of the bad change will keep serving stale data for up to an hour after rollback, regardless of how quickly you revert it. Anything relying on CNAME chains compounds this, since each hop in the chain can carry its own TTL.
Should we always keep TTLs low as a safety net?
Not universally. Very low TTLs (under 60 seconds) increase authoritative query volume and can add latency for some resolver configurations. The practical middle ground most teams land on is 300 seconds for records that change occasionally, dropped further only in the 24-48 hours around a planned migration.
What’s the fastest way to confirm a rollback actually worked?
Query the authoritative nameserver directly, not a local or public resolver, since those may still be serving cached results. If DNSVigil or a similar monitoring setup already has a documented process for responding to a critical DNS alert, follow the same verification steps for a rollback as for the original incident.
A rollback plan that only exists as a mental note falls apart under pressure. Export zone state before changes, lower TTLs ahead of planned migrations, and keep old infrastructure alive until the new setup is proven — that combination turns a bad DNS change from a multi-hour incident into a footnote in the next standup.