Spend your novelty budget on the product

Every team has a fixed budget for novelty. Not money — attention. The number of things you can afford to be uncertain about at once, before uncertainty starts compounding and you can no longer tell which of your unknowns is the one hurting you.

Most teams spend that budget on infrastructure, because infrastructure is where the interesting-looking problems are. Then they wonder why the product feels ordinary.

We spend ours on the product, and run infrastructure that is almost aggressively unremarkable.

The stack, in full

• Postgres for the system of record. Not a document store, not an event-sourcing framework. A relational database with constraints that actually enforce the invariants.
• Go for services. Static binaries, a scheduler that handles our concurrency shape, and no runtime to tune.
• Containers built once and promoted through environments unchanged.
• Terraform for every environment, including the one you'd be tempted to click together.
• Kubernetes, used as a boring scheduler and nothing more. No service mesh, no operators we wrote ourselves.

There is nothing on that list a competent engineer hasn't seen before. That is the point.

What "boring" actually buys

A partition-tolerant, eventually-consistent, multi-primary datastore is a fascinating thing to operate. It is also a thing that will, at some point, hand you a conflict-resolution bug that only manifests under a network condition you cannot reproduce. You will spend a week on it. That week comes out of the product.

Postgres will not do that to you. It will fail in ways that are documented, in ways that thousands of people have already hit, and in ways where the top Stack Overflow answer is usually correct. When it does something surprising, the surprise has a name.

Where we do spend it

The novelty in our products is real, and it is all above the infrastructure line:

• Mesh's Raft-backed sequencer, which makes message ordering a property of the system rather than a hope. That was genuinely hard, and it was worth being hard.
• Spanzo's grounded-inference pipeline, where every number a model reports is computed in code first.

Both of those are sitting on Postgres, Go, and Kubernetes. The interesting part is the idea, not the substrate it runs on.

The test

Here is the question we ask before adopting anything new into the infrastructure layer:

If this breaks at 3am, will the person on call be debugging our problem, or the vendor's?

If the answer is "the vendor's," we need a very good reason to be there — because that is a night we cannot spend on the thing our users are actually paying for.