subscribe for upcoming articles and periodical summaries of the most read ones

Random Thoughts on Leadership & Technology

Envelope Protection

cockpit

or, how to build a workforce that has never once met the edge

In Normal law, an Airbus A320 will not let you stall it. You can haul the sidestick fully aft and hold it there with the sincerity of a man wrestling a bear, and the aeroplane will decline. It will fly at maximum lift, mushing along like an offended cat, and it will not depart. Somewhere in three redundant flight control computers there is a rule, and the rule outranks you. You are not the pilot in command. You are a well-regarded source of suggestions.

This is one of the great engineering achievements of the twentieth century, and I mean that without irony, which for me is a considerable sacrifice. Envelope protection did not reduce a category of accident. It deleted several. Stall on approach. Overbank into a spiral. Pulling the wings off in a recovery you initiated yourself in a panic. Whole species of catastrophe, once responsible for enormous quantities of human misery, now largely extinct in the fleets that adopted it.

And it was bought with a currency that nobody put on the invoice.

The envelope is a shape, and the interesting part is the corners

Ask an aerodynamicist for the flight envelope and you get a chart with stall speed running up one side, maximum operating speed down the other, and load factor limits capping the top and bottom. Clean configuration, roughly plus two and a half g and minus one g. Flaps out, less. It is a fenced field, and the fence is not arbitrary. It is where aluminium stops being a wing and starts being a lesson.

Almost all flying happens in the middle of that field. Cruise is boring by design. The corners exist for the days that are not boring - the windshear escape, the terrain pull-up, the upset recovery, the go-around that begins two hundred feet lower than anybody planned. Capability at the edges is worthless ninety-nine point nine percent of the time, and on the remaining occasion it is the only thing in the world that matters.

Guardrails in an engineering platform work the same way and are sold on the same premise. Nobody needs to run a destructive migration against production by hand. Nobody needs to bypass the deploy pipeline. Nobody needs privileged shell on a node. And that is true, right up until the day the pipeline itself is the outage, and then it is the only thing in the world that matters, and the person holding the keyboard has never done it and the runbook was last verified during an administration that has since left office.

Who is pilot in command, and why the argument is basically theological

There are two houses. Airbus builds hard protections - the limit is a wall, and you may push on the wall as hard as you like. Boeing has traditionally built soft ones - heavy artificial resistance, a stick shaker, a very strong opinion, all of it overridable by a sufficiently determined human with sufficiently strong forearms. Two philosophies about the same question. Is the last word held by the engineer who wrote the rule years ago in a quiet office, or by the person currently strapped to the front of the problem with their life on the line and thirty seconds of context nobody else has.

The genuinely deflating fact is that both fleets have superb and broadly comparable safety records. Decades of argument in bars near flight schools, and the data politely declines to declare a winner. What the data does seem to say is that coherence matters more than doctrine. A system whose authority model is consistent, clearly annunciated, and matched by the training around it works. A system that is a mosaic of overrides, exceptions, and modes that behave differently on Tuesdays kills people regardless of which house designed it.

Which is an uncomfortable finding for anyone whose platform strategy is fourteen years of accreted policy written by nine different teams, half of whom have left, in four different enforcement engines, with an escape hatch documented in a wiki page whose last editor is now a founder of something.

When the rule has bad information

October 2008, Qantas Flight 72, an A330 over Western Australia. A faulty air data unit began emitting nonsense angle-of-attack values. The envelope protection, being an honest and diligent system, believed them and did exactly what it was built to do, which was to save an aircraft that was not in danger by pitching the nose down hard. Twice. Passengers who were not wearing belts hit the ceiling. More than a hundred injuries, a dozen of them serious. The protection was not defective. The protection worked perfectly on data that was a lie.

Then MCAS, and two hulls and three hundred and forty six lives, and a system with more authority than the crew, driven by a single sensor, repeatedly, with no clear way to argue. Not envelope protection exactly, but the same family and the same lethal genetic defect - a guardrail whose confidence in its own inputs vastly exceeded the reliability of those inputs, and whose designers had reasoned about failure of the mechanism rather than corruption of the premise.

Every platform engineer reading this has already thought of theirs. The autoscaler that read a broken metric and scaled to zero. The circuit breaker that tripped on a health check probing the wrong port and took down the healthy fleet to protect it from itself. The policy engine that denied the emergency rollback because the emergency had also broken the service the policy engine used to check identity. Guardrails do not fail by being too weak. Guardrails fail by being confidently, mechanically, tirelessly correct about a world that has stopped existing.

The accident lives in the handover

June 2009, Air France 447. Iced pitot tubes, unreliable airspeed, autopilot drops out, and the flight control system degrades from Normal law to Alternate law. In doing so it sheds low speed stability protection. The aeroplane that had spent the crew's entire careers being physically incapable of stalling became, in that instant and without any of them internalising it, an aeroplane that could stall. One pilot held the stick back. It stalled from thirty eight thousand feet and fell into the Atlantic in under four minutes with the stall warning sounding for most of them.

The protection did not kill them. The removal of the protection killed them, at the exact moment they most needed the skill the protection had made it unnecessary to maintain. The system was safe, and then it handed them an aircraft they had never actually flown, at night, in weather, with a couple of minutes of budget.

This is the finding that should be tattooed somewhere visible on every platform team. The risk is not the guardrail. The risk is the boundary of the guardrail's competence and whether anybody is told, loudly, in words, that they have just crossed it. Degraded mode is where people die. Degraded mode is where your platform hands a team full manual control of infrastructure they have only ever addressed through a form with three fields and a dropdown, and the handover message is a yellow banner nobody reads.

Our envelope is made of scars, not physics

Here is where the analogy earns its keep by breaking.

An aircraft's envelope is derived. Someone computed the loads, built the airframe, put it in a rig, bent it until it failed, and then drew the fence with margin. The limit is a property of the metal. It is knowable, static, and true whether or not anyone believes in it.

Software has no V-n diagram. There is no coupon test for a distributed system. So we build our envelope the only way available to us, which is by archaeology. Every rule in your policy engine is a headstone. That deploy freeze commemorates a Friday in 2019. That mandatory two-reviewer gate is a memorial to one specific bad afternoon. That ban on a particular class of query exists because of a person whose name is still spoken quietly.

Which means our envelope is a map of things that have already gone wrong, drawn at the scale of a company that no longer exists, and confidently mistaken for a map of physics. Worse, it ratchets in one direction only. Adding a constraint after an incident is career-enhancing and requires no evidence beyond the incident. Removing one requires a named human being to volunteer their signature next to a hypothetical future disaster. Guardrails therefore accumulate at the rate of institutional trauma and are never, ever paid off, and the person who writes the rule is structurally insulated from the cost of the rule, because friction lands on the feature teams and glory lands on the platform.

Aviation at least has an incentive to keep the envelope honest - a fence drawn too tight makes an aircraft that cannot escape windshear, and that shows up in the accident record. Nothing shows up in your record. Velocity lost to a rule that never needed to exist is invisible, unattributed, and permanent.

Children of the magenta line

The phrase belongs to a legendary American Airlines training talk, and it describes crews who manage the automation beautifully and fly the aircraft poorly, because the second skill has been rendered ceremonial. Regulators noticed. There are now advisory circulars gently encouraging airlines to let their pilots hand-fly the thing occasionally, which is a remarkable sentence to have to write down and publish.

Aviation, to its enormous credit, hedged the bet. It paired the removal of authority with a training regime of ferocious seriousness. Recurrent simulator sessions. Practised degraded modes. Upset recovery. Type ratings. A licensing structure with teeth, and a culture in which practising the thing you hope never to do is a professional obligation rather than a nice-to-have that slips when the quarter gets tight.

We took the protections and skipped the type rating.

Our version of simulator time is the game day that keeps getting rescheduled, the chaos experiment that runs in staging where nothing rhymes with production, and the break-glass procedure that exists in a document rather than in anybody's hands. An untested escape hatch is not an escape hatch. It is a comment. It is a prayer with a ticket number.

A rough field guide, offered with appropriate humility

Constrain hard when the harm is irreversible, the limit is genuinely known rather than inferred from one bad memory, and no legitimate emergency requires crossing it. Deleting the only copy of customer data qualifies. Very little else does.

Constrain softly - warn, log, require a second pair of eyes, make it awkward but possible - when the edge has real operational value and the operator on the day may know things your rule cannot. Awkward but possible is a wildly underrated setting and almost nobody uses it, because it is harder to build and impossible to put on a slide.

Never let a hard limit fire on a single unverified input. If Qantas 72 and MCAS did not teach the industry that, it is because the industry has not been paying attention, which is the base case.

Announce degradation like an adult. Not a banner. Not a log line at info level. State what protections have gone, what the operator now owns, and what mode they are actually in. Mode confusion is the recurring villain of this entire genre.

And if you take the edges away, buy back the skill deliberately, on a schedule, with budget, in an environment that resembles reality. Otherwise you have not eliminated the risk of the edges. You have deferred it, concentrated it, and arranged for it to arrive all at once, on your worst day, in the hands of somebody who has only ever read about it.

The bet

Envelope protection is not a substitute for airmanship. It is a wager that airmanship will not be needed, backed by an institution that spends fortunes making sure it is there anyway, precisely because it believes its own wager might lose.

Your guardrails are the same wager. The question is not whether you should have them. You should, and the argument is over, and the accident statistics won it decisively. The question is what you have posted as collateral.

Aviation posted recurrent training, a licensing regime, a hard-won culture of practising failure, and a genuine institutional belief that the human at the front might one day be the last functioning component in the system.

Most engineering organisations have posted a Confluence page.

The fence holds nearly always. Nearly always is a very long time, and it is not forever, and the bill for the difference is not paid by whoever drew the fence.

The refused input - Studio photograph of a single aircraft sidestick pulled fully aft, a pale hand gripping it, shot in hard low side light against near-black. Shallow depth of field, cinematic teal and amber grade, faint cockpit reflections. Empty negative space on the right for a headline. Editorial, restrained, no text in the image. The fence and the corner - Minimalist technical illustration of a flight envelope diagram, thin white lines on deep navy, the safe interior filled with soft glowing gradient, one small human figure standing alone in the far top corner beyond the boundary. Swiss editorial poster aesthetic, generous white space, no labels. Alternate law - A darkened glass cockpit at night, most instruments unlit, a single amber caution message glowing on the centre display, rain streaking the windscreen, faint lightning behind. Photorealistic, high dynamic range, unsettling calm. No readable text on the screens. The forgotten hands - Conceptual still life of two objects side by side on a concrete surface, a worn leather pilot glove and a pristine unopened one still in plastic, top-down, cold north-facing daylight, subtle shadow. Wes Anderson symmetry with a muted industrial palette. Quiet, ironic, magazine cover framing. Guardrails as headstones - Wide isometric illustration of a vast server room where the racks are rendered as neat rows of pale stone markers, thin blue floor lighting, one small figure with a torch walking between them. Muted colours, fine line work, New Yorker technical illustration feel.