Ideas & Writing Michael Orlando

The most expensive way to learn infrastructure is to learn it in production.

The issue isn’t that teams are careless. The issue is that production has real stakes: customer impact, incident pressure, and a low tolerance for experimentation.

That’s why I like having a smaller proving ground.

What I mean by “prototype infrastructure”

I’m not talking about building a perfect mini-version of your production environment.

I mean creating a place where you can:

  • test an architecture choice without betting the business on it
  • expose hidden dependencies before they become outages
  • practice upgrades, backups, restores, and failure modes on purpose
  • learn the operating costs (not just the feature set)

Homelabs, rehearsal environments, and small internal sandboxes can all serve this purpose.

The mechanism (what gets better when you rehearse)

Proving grounds work because they shorten the feedback loop on the things that usually hurt later:

  • unclear networking assumptions
  • identity and access friction
  • backup/restore reality (what actually works vs what you hoped)
  • “it works on my machine” deployment gaps
  • monitoring that only exists after the first incident
  • upgrades that are easy in theory and painful in practice

In other words: they make infrastructure legible before it becomes mission-critical.

The trade-off (proving grounds can become a trap)

A lab can also turn into a hobby that never feeds back into production decisions.

The way I avoid that is to keep the proving ground scoped to a specific question:

  • “Can we operate this reliably?”
  • “What breaks first?”
  • “What’s the smallest setup that teaches us the real constraints?”

Practical next step

If you want to use this pattern without over-building it:

  1. Pick one infrastructure decision you’re about to make (k8s vs ECS, Postgres vs Dynamo, Nginx vs Traefik, etc.).
  2. Name the failure mode you’re most afraid of (restore time, security model, upgrade pain, cost blowout).
  3. Build the smallest sandbox that lets you test that failure mode intentionally.
  4. Write down what you learned and carry it into the production design.

Prototyping infrastructure is how you buy understanding before the stakes get expensive.