Engineering Failures and What We Have Learned from Them
Engineering is the only profession that names its failures after the places where they happened. Doctors have "complications". Economists have "corrections". Engineers have Tacoma Narrows, Chernobyl, Challenger, the Hyatt Regency walkway. There is nowhere to hide when your mistake is 300 metres long and visible from the highway.
That brutal public accountability produced something unusual - a body of hard-won knowledge about how complicated things go wrong. It turns out that bridges, marriages, startups, cities and dinner parties fail in remarkably similar ways. Here are ten lessons that cost lives and fortunes to learn, offered here for free.
1. Things break at the seams, not in the middle
In September 1999, the Mars Climate Orbiter arrived at Mars after nine months and roughly 125 million dollars of flawless flight, dipped far too low into the atmosphere, and stopped existing.
The cause was not a faulty part. Lockheed Martin's software reported thruster impulse in pound-force seconds. NASA's navigation software at JPL read those numbers as newton-seconds. Both programs were correct. Both teams were competent. Nobody was in charge of the handshake between them, and the handshake was off by a factor of about 4.45.
The components almost always work. It is the space between them that kills you - between departments, between contractors, between what you said and what your partner heard. Your organization's version of pound-force seconds is the word "done".
2. Shrink the problem before you get clever
Ferdinand de Lesseps had built the Suez Canal, which made him the most celebrated engineer alive and therefore the most dangerous man in Panama. From 1881 he set out to dig a sea level canal straight through a mountain range, because that is what he had done in the flat sand of Egypt and it had made him immortal. The jungle killed roughly 20,000 workers, the company collapsed in scandal, and France got a national humiliation instead of a canal.
The Americans succeeded largely by refusing to solve the same problem. They stopped trying to cut the mountains down to sea level and built locks to lift the ships up over them instead. They also noticed, thanks to William Gorgas, that the actual adversary was a mosquito, and spent years on drainage and fumigation before the serious digging began.
De Lesseps' fatal error was not technical. It was that his reputation had become attached to the harder version of the problem. Requirements are almost always more negotiable than physics, which means most heroic engineering is a failure of nerve at the requirements meeting.
3. Bad estimates come from unknowns, not laziness
The Sydney Opera House was estimated in 1957 at seven million Australian pounds, finishing in 1963. It opened in 1973 and cost 102 million. That is not a rounding error. That is an order of magnitude and a decade.
Here is the part everyone forgets. Jørn Utzon's beautiful shells were sketched before anyone on Earth knew how to build them, and construction on the podium started before the roof had been solved, because a nervous government wanted visible progress. Years of analysis followed before the breakthrough - every shell could be cut from the surface of one single imaginary sphere, which meant they could be cast from repeated moulds instead of individually.
Padding your estimate by twenty percent does not make an unknown known. It merely makes you wrong more expensively and with more confidence. The cure for an unknown is a cheap experiment, a prototype, a test article, one awkward phone call. Buy information, not comfort.
4. Measure first, because your intuition is a confident liar
In the summer of 1854, cholera was tearing through Soho in London, and every respectable authority knew the cause was bad air. John Snow (yeah - really - just with an additional h - his parents must really loved GOT 😉, being tiresome, went door to door and drew dots on a map of where the dead were. The dots clustered around a single water pump on Broad Street. He got the handle removed.
The delicious footnote is that Joseph Bazalgette then rebuilt London's sewers on the entirely wrong theory, aiming to banish the smell rather than the bacteria, and saved the city anyway. He also quietly doubled the diameter of every pipe on the grounds that engineers always underestimate. His sewers were still serving a city several times the size a century and a half later, which makes this history's single most successful fudge factor.
Guessing where the time goes, where the money goes, where the heat goes, where your evening went - the guess is wrong far too often to be worth trusting. Count something.
5. Prefer a simple thing you understand to a clever thing you do not
Fifty-five hours into Apollo 13, an oxygen tank ruptured and the mission became a survival problem. The lunar module was designed to keep two men alive for two days. It now had to keep three men alive for four, and its carbon dioxide scrubbers were the wrong shape (literally the square peg-round hole thing) - the command module had square lithium hydroxide canisters, the lunar module had round receptacles.
So a team on the ground built a solution from an inventory of exactly what was aboard. Duct tape, cardboard from a flight manual cover, a hose from a spacesuit, a sock. The crew read the instructions aloud and built it. It worked.
That improvisation was only possible because the spacecraft was simple enough for tired, frightened people to reason about from first principles. Sophistication that you do not fully understand is a loan against your worst future day, and the interest rate is cruel. You will be debugging this thing at 3 in the morning with someone waiting.
6. Sort your decisions by whether you can take them back
Standard railway gauge is 4 feet 8.5 inches, a number no committee ever blessed and no physicist ever derived. It descends from the wagonways of English collieries, where the wheel spacing suited the horses and the ruts (it also is indirectly the reason for the maximum width ot the nozzle of the F-1 engine of Saturn V, but that's a different story). George Stephenson used it because it was what he knew.
Isambard Kingdom Brunel (an engineering giant), who did think about it, chose seven feet for the Great Western Railway and was arguably right. Broad gauge was faster, steadier and more comfortable. It lost anyway, because by the 1840s there were simply more miles of the other thing, and a network decision only has to be made once to be permanent. In 1892 the Great Western surrendered and converted its remaining broad gauge line to standard over a single weekend using an army of labourers. The gauge of most of the world's railways is still an accident of horse-drawn Northumbrian coal carts.
Some decisions are doors you can walk back through and some are one-way. Almost nobody spends their care in proportion. We agonise for six weeks over a logo, which is reversible, and pick a database, a city, or a co-founder over a pleasant lunch.
7. Write down why, not just what
NASA never lost the blueprints for the Saturn V. This is the popular myth and it is false. The drawings for the aforementioned F-1 engine, the largest single chamber liquid-fuelled engine ever flown, were archived and available.
It did not help much. When engineers at Marshall Space Flight Center wanted to understand the F-1 again in the 2010s, they had to drag engines out of museums and storage, CT-scan the components, and reverse-engineer the thing from the metal outward. The drawings recorded what had been built. They did not record why the injector had that particular geometry, which welders could produce a joint that held, what had been tuned by hand and hearing on a test stand in Alabama, or which of the strange choices were essential and which were Tuesday afternoon.
Rationale evaporates in months and is the most expensive thing in the world to reconstruct. Documentation is not bureaucracy. It is a letter to an exhausted stranger who will inherit your decisions, and that stranger is usually you, eighteen months from now, with no memory of being this clever.
8. If people keep using it wrong, the design is wrong, not the ones using it
During the Second World War, American pilots kept landing B-17s beautifully and then retracting the landing gear while still on the runway, dropping the aircraft onto its belly. The Army Air Forces did what institutions do and concluded that pilots needed to be better, more disciplined, more careful.
Alphonse Chapanis, a psychologist rather than an engineer, went and looked at the cockpit. The landing gear lever and the flap lever were identical in shape and sat side by side. Tired men reaching without looking had a coin flip to make. He glued a small rubber wheel onto the gear lever and a wedge shape onto the flap lever, so the controls could be identified by touch alone. The accidents essentially stopped.
An entire discipline, human factors, was born from a man putting funny shapes on knobs. The lesson generalises viciously. "User error" is nearly always a design decision wearing a false moustache, and this applies to your onboarding process, your kitchen, and your household rules about the dishwasher.
9. Success is the worst teacher you will ever have
On the morning of 28 January 1986, Challenger's solid rocket booster joints did what they had been quietly doing for years. Rubber O-rings, which were supposed to be untouched by hot gas, had shown erosion and blow-by on previous flights. Each time, because the vehicle came home, the anomaly was reviewed and reclassified as an acceptable flight risk. Evidence of a system behaving outside its design intent was steadily converted into proof that it was fine.
The sociologist Diane Vaughan gave the pattern its name, the normalization of deviance. Richard Feynman (my favourite physicist - yeah I have a favourite physicist 😊 - watch or read his lectures on physics if you haven't - he is one of the gratest science communicators of all time) later put a piece of O-ring rubber in a glass of ice water in front of the cameras and let it stop being springy. He also noted the estimated odds of losing a vehicle - management said roughly one in 100,000, the working engineers said roughly one in 100.
Nothing bad has happened yet is not data about tomorrow. It is a description of your luck so far. Every organisation, family and body has one thing that has been fine, so far.
10. There is never one cause, so stop hunting for a villain
At about 2 in the afternoon on 14 August 2003, a transmission line in Ohio sagged in the summer heat into a tree that should have been pruned. Roughly seven minutes of cascade later, 50 million people across the northeastern United States and Ontario had no electricity.
The tree is the part everyone remembers, and the tree is the least interesting part. An alarm system at the utility had failed silently over an hour earlier, so operators were watching a screen that had stopped telling them the truth and did not know it. A regional monitoring tool was running on stale data. Nobody had a picture of the whole grid. Reserve margins, communications, training and the physics of load transfer all lined up. Any one hole in the cheese slices would have been survivable. They happened to align.
Complex systems almost never fail because a person was bad. They fail because several ordinary weaknesses, each individually tolerable, arranged themselves into a channel. If your incident review concludes with somebody's name, you have found a scapegoat, not a cause, and you have learned anything nor purchased safety whatsoever.
The lesson underneath all the lessons
You will be wrong. Not hypothetically, not occasionally, but as a standing condition of doing anything ambitious. The competent are not distinguished by being right more often. They are distinguished by finding out sooner, while it is still cheap.
So build the feedback loop before you build the thing. Instrument it. Invite the pruning of trees and the reading of alarms and the person who says the erosion is not fine.
And then keep paying for it, which is the part nobody puts in the founding myth. The Roman aqueducts did not last because the concrete was magic. They lasted because for centuries somebody employed the curatores aquarum, an unglamorous permanent staff whose entire job was walking the channels and fixing things. When the payroll stopped, the miracle stopped. It always does.