Sanitation Is 80% of Brewing
A short meditation on mops, microbes, and the promotion cycle
The part of the craft nobody photographs
Brewing photographs beautifully. Copper vessels, steam curling through a shaft of light, a bearded person holding a hydrometer aloft with the expression of someone receiving a prophecy. That is the brochure. That is the tap handle, the label art, the Instagram grid.
The job is a person in rubber boots at half past five in the morning, elbow-deep in a tank with a caustic solution and a bad attitude, scrubbing the underside of a valve that nobody will ever see. Then rinsing. Then acid. Then rinsing. Then dismantling every hose fitting, because a hose fitting is where good intentions go to ferment.
Ask a working brewer what the job actually is. The honest ones say they clean stainless steel for a living and beer occasionally happens as a by-product. The proportion varies depending on who is complaining - seventy percent, eighty, ninety on a bottling week - but the shape never changes. The creative act is a rounding error on the janitorial one.
Wort does not want to be beer
Here is the thing that makes sanitation non-negotiable rather than merely virtuous. Wort is a warm, sweet, nutrient-dense, entirely defenseless sugar solution. It is the most appealing thing in the building to every microorganism in a five mile radius. Lactobacillus wants it. Pediococcus wants it. Brettanomyces has plans. Acetobacter is already drafting a proposal.
So you are not making beer. You are rigging a competition. You are ensuring that your chosen yeast enters the arena with such an overwhelming numerical advantage that nothing else gets a vote. Sanitation is not the preparation for brewing. Sanitation is the brewing. Recipe design, hop selection, the poetry on the can - that is garnish on a decision that was made with a mop.
Every mature engineering organization is also a warm bath of nutrients waiting for something opportunistic to move in.
The infection tastes fine on Tuesday
The cruelty of a sanitation failure is its patience. Gravity readings look normal. The samples taste clean. Packaging goes off without incident. Everybody goes home pleased with themselves.
Four weeks later a bottle in a customer's fridge has turned into sour cider with notes of nail polish, and the review does not say "inadequate clean-in-place protocol on vessel three." It says "this brewery is bad." The feedback arrives late, arrives to the wrong address, and arrives in a form that makes root cause analysis socially impossible.
The dependency you did not update does not hurt you in the sprint you skipped it. It hurts you eleven months later, on the day you urgently needed to move fast, in a form that looks like general organizational incompetence rather than one deferred decision. The flaky test does not fail. It erodes belief in all tests, slowly, until nobody reads the pipeline output and the pipeline becomes an expensive ritual.
The unloved load-bearing list
The engineering equivalents of scrubbing a valve are well known and universally deferred.
- Continuous integration that is fast and believed
- Dependency and runtime upgrades taken in small doses rather than as a five year gastric event
- On-call quality, which means alerts that mean something, runbooks that are true, and ownership that has a name attached
- Documentation written for the person who arrives after you leave
- Test suites without flakes, because a flaky suite is worse than no suite and considerably more expensive
- Deleting things, reducing surface area, finishing migrations instead of leaving two systems and a shrug
They share three properties. They are invisible when done well. They are catastrophic when neglected. And when they save you, nobody can prove they saved you.
The counterfactual has no résumé
You cannot put an outage that did not occur into a promotion packet. There is no artifact. There is no demo, no launch email, no graph angling encouragingly upward, no Slack thread with fourteen rocket emoji and a director saying "huge."
Feature work is generous in this respect. It leaves fossils. Maintenance work leaves an absence, and absences do not survive contact with a calibration meeting. The committee is not malicious. It is illiterate in one specific and fatal way - it can only read things that exist.
So the story writes itself. Two engineers, same team, same year. One shipped the thing on the slide. The other spent four months making the build ten minutes faster for sixty people, retiring two internal services, and turning a paging rota that ate three nights a week into one that eats none. At review time, the first one has a narrative and the second one has a feeling.
An excellent reward function, for producing dirty tanks
Every company will tell you it values reliability, quality, craftsmanship, the long term. Then it promotes the launch. When the poster on the wall and the rubric in the spreadsheet disagree, believe the spreadsheet. A values statement is an aspiration. A promotion rubric is a contract, and unlike the poster, it is enforceable.
What makes this genuinely pernicious rather than merely disappointing is that sloppiness usually works. Most rushed batches turn out fine. Most skipped upgrades cause nothing. The punishment is rare, delayed, and diffuse, while the reward is immediate and personal. Behavioral psychology has a name for the reinforcement schedule that produces the most stubbornly durable habits, and it is precisely this one. You are not training people to gamble occasionally. You are training them, thoroughly and at expense, to believe that hygiene is a personality quirk and that they are simply skilled.
The market for fires
We hand out medals to firefighters. We hand out nothing at all to the person who spent a joyless quarter installing sprinklers, and mildly resent them for the disruption.
Follow that incentive to its logical end. Heroism requires an emergency. Emergencies require decay. If the fastest route to visibility runs through a severity one incident, an organization has quietly opened a market for severity one incidents. Nobody lights the fire, obviously. People are not villains. They simply feel no great urgency about clearing the kindling, and they notice that the person who restored service at three in the morning got a spot bonus and a shout-out, while the person who made three in the morning impossible got a calendar with nothing in it and a slightly awkward review conversation about impact.
Making the invisible visible, which is harder than saying it
The obvious counter-argument is that maintenance is unmeasurable, and this is half right. Measure cleaning directly and you will receive cleaning theatre with striking speed. A dashboard about dependency freshness is not fresh dependencies. A documentation initiative produces documents, which is not the same as anyone being less lost.
The trick is to reward the change in other people's costs rather than the volume of one's own effort.
- Give it a row in the rubric with the same weight as delivery. If it is not in the spreadsheet, it is a hobby, and people with mortgages do not pursue hobbies at work.
- Demand the counterfactual in writing. What did you prevent, and how do you know? Force an estimate. Wide error bars are acceptable. Silence is not. An engineer who cannot articulate the value of their maintenance work has a communication problem worth fixing, and one who can should not be penalized for the answer being a subtraction.
- Measure derivatives, not activity. Build time down. Flake rate down. Pages per on-call week down. Time to restore down. Onboarding time to first commit down. Services retired, code deleted, runbook steps eliminated. All of these are somebody else's cost falling, which is the only definition of leverage that survives scrutiny.
- Budget it as work. "Twenty percent time for tech debt" is a way of saying zero percent with more syllables, because the twenty percent is where deadlines go to feed. Put hygiene on the roadmap with a name, a date, and a person who is accountable for it in exactly the way they would be for a feature.
- Rotate the mop. A permanent platform-and-hygiene team becomes a caste with a visible career ceiling, and the feature teams learn that mess is somebody else's department, which is the actual root cause of the mess.
- Promote somebody for it, loudly, at least once. One legible precedent outperforms a hundred all-hands sermons. Nobody reads the policy. Everybody reads the promotion list, forensically, and updates their behavior by the end of the week.
- Celebrate boredom out loud. A quarter in which nothing happened is an achievement with an author. Name the author, in the meeting, in front of everyone, and watch how quickly the definition of impact starts to shift.
The virtue you punish
None of this is complicated. It is merely uncomfortable, because it requires an organization to admit that its stated preferences and its revealed preferences have been living separate lives for years and only one of them pays rent.
You can ask for cleanliness. You can reward shipping. You cannot do both and receive cleanliness. What you receive instead is a slow filter, applied over several promotion cycles, which selects out precisely the people who were keeping the tanks clean - some of them by leaving, most of them by simply learning, sensibly and permanently, to stop.
The beer is the only evidence anyone ever tastes. And the beer is downstream of the mop.