DEVOPS-05-004-P · Primer — Mutable vs Immutable Infrastructure

The idea in one sentence

Instead of logging into a running server to change it — patching, upgrading, tweaking a config until it works — you build a brand-new server from a known image, deploy that, and throw the old one away, so the thing running in production is always something you can rebuild from scratch instead of something that slowly drifted into a state no one fully understands.

Why it exists

Picture a server that has been running for three years. In year one someone SSH'd in to bump a memory limit. In year two a different engineer applied a security patch by hand at 2 a.m. during an incident. Somewhere along the way a library got upgraded, a cron job was added, a config file was edited "just to test something" and never reverted. Nobody wrote any of it down. That server now works — but no one can tell you why it works, and no one dares reboot it because they're not sure it will come back the same. Martin Fowler calls this a snowflake server: unique, delicate, and impossible to reproduce.

The deep problem is configuration drift. Every server you touch by hand drifts a little further from every other server, and from the fresh install it started as. Two machines meant to be identical stop being identical. A fix that works on one fails on the other, and you burn an afternoon on "but it works on box A" — the infrastructure version of "works on my machine."

The mutable approach — modify servers in place over their lifetime — is how systems administration worked for decades, and it's not wrong so much as fragile at scale. When you have three servers you can hold their quirks in your head. When you have three hundred, hand-tuning is a source of outages, not stability. Immutable infrastructure exists to kill drift at the root: if you never modify a running server, it can never drift. Any change means a new server from a new image. The old one is destroyed, quirks and all.

How it works

The whole idea rests on one hard rule and a small set of moving parts.

The rule: servers are never changed after they're launched. No SSH-in-and-edit. No hand-applied patches. If a running instance needs to be different, you don't fix it — you build a new one and replace it. This is the heart of replace vs. modify in place. Mutable = modify the thing that's running. Immutable = replace it wholesale.

Golden images. A golden image is a pre-built, fully-configured machine template — an OS with your app, dependencies, security patches, and settings already baked in and frozen. Tools like HashiCorp Packer build these: you write one template, Packer boots a temporary machine, installs and configures everything, then snapshots the result into a reusable image (an AWS AMI, a Docker image, a VM template). That image is versioned and immutable. Every server launched from image v42 is byte-for-byte identical to every other server from image v42. There is no per-server hand-tuning to drift.

The change cycle. To ship a change — a patch, a new app version, a config tweak — you:

1. Rebuild the golden image with the change baked in, producing a new version.
2. Launch fresh servers from that new image.
3. Shift traffic to the new servers (often via a load balancer).
4. Destroy the old servers entirely.

Fowler's name for a server built this way is a phoenix server — it dies and rises from ashes on a regular schedule, so rebuilding from scratch stays a routine, boring operation rather than a terrifying one.

Where state goes. If servers are disposable, they can't hold anything you care about. So data — databases, uploaded files, session state — is deliberately pushed off the compute and into durable, managed stores (a managed database, object storage like S3, a cache). Servers become cattle, not pets: interchangeable, nameless, replaced without ceremony. The classic contrast: a pet gets a name and a vet visit when it's sick; cattle get a number, and a sick one is replaced.

Rollback becomes trivial. Because the old image still exists, "undo the deploy" is just "launch the previous image again." There's no un-patching, no reconstructing what the box looked like before — the previous known-good state is a first-class, reproducible artifact.

The one gotcha

Immutability moves the cost, it doesn't erase it — and the trap is a slow build-and-deploy loop. If rebuilding a golden image and rolling out fresh servers takes 45 minutes, then a one-character config typo becomes a 45-minute round trip, and every engineer will feel the gravitational pull to "just SSH in and fix it quickly this once." The moment someone does, immutability is broken: that server is now a snowflake, and worse, it's a lie — it no longer matches the image it claims to be running, so your next rebuild will silently wipe out the undocumented fix and reintroduce the bug. The whole guarantee collapses the first time a hand-edit sticks. Making immutability actually work therefore demands fast, automated image builds and painless deploys, plus the discipline to route every change — even the tiny urgent ones — back through the pipeline. If your emergency fix can't go through the image, your emergency process is broken, not your rule.

See it — mutable drift versus immutable replacement

flowchart TB
    subgraph MUT["Mutable (modify in place)"]
        M0["Fresh server v1"]
        M1["Hotfix patch<br/>applied by hand"]
        M2["Config edited<br/>during an incident"]
        M3["Library upgraded,<br/>not written down"]
        M4["Snowflake:<br/>no one can rebuild it"]
        M0 --> M1 --> M2 --> M3 --> M4
    end
    subgraph IMM["Immutable (replace)"]
        I0["Golden image v42"]
        I1["Need a change?<br/>Rebuild image v43"]
        I2["Launch fresh<br/>servers from v43"]
        I3["Shift traffic,<br/>destroy old servers"]
        I4["Rollback =<br/>relaunch v42"]
        I0 --> I1 --> I2 --> I3
        I1 -.rebuild.-> I0
        I3 -.undo.-> I4
    end

The one-line takeaway

Don't fix the server that's running — replace it with a freshly built one from a known image, so every machine in production is something you can rebuild, and drift never gets a foothold.

Where you'll meet it

You'll meet this the first time you inherit a server nobody wants to reboot, or spend a day chasing a bug that only reproduces on one of five "identical" machines — that's configuration drift billing you for someone else's untracked edit. You'll meet it again the moment you touch containers, because a Docker image is immutable infrastructure by construction: you don't patch a running container, you build a new image and redeploy. It shows up in auto-scaling groups (every new instance boots from the same AMI), in blue-green and canary deploys (stand up the new version alongside the old, then cut over), and all over infrastructure-as-code, where the image is the source of truth. Interviewers ask it plainly: "what's the difference between mutable and immutable infrastructure, and why would you choose immutable?" The strong answer names golden images and Packer, explains how replacing servers kills configuration drift and snowflakes, moves state off the compute, and honestly notes the cost — you need a fast build-and-deploy pipeline, or the discipline breaks the first time someone SSHes in to fix something quickly.

Named concepts: immutable infrastructure, mutable infrastructure, replace vs modify in place, golden images and configuration drift