You Can Only Build Telescopes for Questions You've Already Thought Of
The mirror was flawless. It was also the wrong shape.
In 1990 we lifted a telescope above the atmosphere, which is the single greatest upgrade in the history of looking at things, and discovered that its primary mirror was ground to the wrong curve. Not wildly wrong. Wrong by roughly two micrometres at the edge, a deviation you could not detect by staring, licking, or praying, but more than enough to smear every point of light into a fuzzy halo.
The error is not the interesting part. Manufacturing errors are boring. The interesting part is how the error passed inspection.
The mirror's shape had been verified with a device called a null corrector, an instrument whose entire purpose in life is to say "yes, this is the shape you asked for." The null corrector had been assembled with one internal component sitting slightly out of position. So it did its job beautifully. It confirmed, with extraordinary precision, a shape nobody wanted. Simpler tests were available and reportedly hinted that something was off, and they were discounted, because they were the less sophisticated instruments and this was the good one.
The most accurate measuring device in the building was wrong, and the only things capable of catching it were outranked by it.
Your dashboard is a null corrector.
An instrument is a frozen hypothesis with a hardware budget
Nobody builds a general purpose detector, because there is no such object. You build a detector for a band, a range, a waveform, an event type. Every choice narrows. A photomultiplier responds to certain wavelengths and is majestically indifferent to the rest of the universe. A dark matter experiment is tuned to a guessed mass range, which means thirty years of null results could mean "no dark matter here" or could mean "we guessed the mass wrong and built a very cold net with the wrong hole size."
Gravitational wave astronomy makes the point almost too cleanly. The primary detection method is matched filtering, which compares the data stream against a bank of pre-computed waveform templates. It is exquisite. It can pull a signal out of noise that dwarfs it by orders of magnitude. It can also, by construction, only find shapes somebody already sat down and calculated. To catch the unmodelled things, the ones nobody had a template for, you need entirely separate pipelines built on the humiliating premise that your imagination is incomplete.
That premise is the whole discipline. Almost nobody applies it to their metrics.
Your dashboard is an argument you have stopped remembering that you made
Here is what happened. At some point, a group of reasonable people held a theory of how the business created value. They encoded it. They shipped it. Then they left the company, or moved teams, or simply forgot, and the encoding stayed on the wall in the lobby, glowing, above the beanbags.
Every tile is a buried claim.
- Counting daily active users claims that presence is value and that a user who logs in ten times is worth more than one who logged in once, got what they needed, and got on with their life.
- A funnel claims the journey is linear, ordered, and completed in a single identity across a single device.
- Net Promoter Score claims that sentiment is scalar, that eleven points of resolution is the right amount, and that the people who answer surveys resemble the people who do not.
- Average session duration claims that time spent is a good, which is a theory the entertainment industry holds and the tax preparation industry emphatically should not.
- Any ratio claims a denominator, and the denominator is where the bodies are buried, because someone decided who counts as a customer and everyone excluded became structurally invisible in a single afternoon.
None of these are wrong. That would be easy. They are all locally reasonable, which is much worse, because a locally reasonable measurement generates confidence, and confidence is the thing you were supposed to be spending carefully.
The instrumented world is shaped like your engineering convenience
There is a second selection pressure nobody puts in the strategy deck. You instrument what is cheap to instrument.
Client side events are cheap, so you have a rich picture of people whose browser successfully loaded your JavaScript. Error tracking is cheap, so you know a great deal about crashes in a running application and nothing whatsoever about the user for whom the application never booted. High cardinality is expensive, so you aggregate, and aggregation is a machine for converting individuals into averages, which is a problem, because the anomaly you need lives in an individual and dies in an average.
Then there is sampling rate, and here the physicists have already done the work and named the trap. If your instrument samples more slowly than the phenomenon oscillates, the phenomenon does not vanish. It is worse than that. It reappears disguised as a slower, smoother, entirely fictional trend. Aliasing is not blindness. Aliasing is confident hallucination in the shape of a clean line going up.
So the picture on the wall is not a picture of the business. It is a picture of the intersection of the business with what was convenient to measure in an earlier era, resampled at whatever interval fit the query cost, then smoothed.
Displayed in red and green, so you know how to feel.
A short field guide to the things your instrumentation structurally cannot see
Not "does not currently show." Cannot. The distinction matters, because the first is a backlog ticket and the second is an epistemological condition.
The people who never became a row. Every analytics system is a survivorship engine. Abraham Wald worked this out in the 1940s while people were putting armour on returning bombers where the bullet holes were, which is precisely the wrong place, because the planes that got hit elsewhere were not available for inspection. Your churn analysis is built exclusively from planes that came back.
Everything before the first event fires. The app store page nobody converted on. The load that timed out. The signup form that failed on a name with an apostrophe in it. There is no funnel step for people who never entered the funnel, and no alert fires when zero of something happens, because zero looks exactly like a quiet day.
Whatever fell into the bucket labelled Other. A taxonomy is a theory of what kinds of things exist. New phenomena are, by definition, things your taxonomy did not anticipate, so they all arrive wearing the same disguise. The Other category is where novelty goes to be politely ignored. When it grows, that is not noise. That is your ontology expiring.
Anything filtered out for being implausible. The great cautionary tale here is the ozone hole. Satellite processing routines flagged extremely low readings as suspect, because values that low were outside the range anyone considered physically reasonable. The hole was found from the ground, with older and cruder instruments, by people who had not yet automated their expectations. Your data quality rules are a formal statement of what you have decided reality is allowed to do.
Successes that look identical to nothing happening. A user who solved their problem in eight seconds and left, and a user who bounced in disgust, can produce byte-identical telemetry. If your instrument cannot distinguish satisfaction from abandonment, no amount of dashboard polish will separate them.
The counterfactual, always and forever. No instrument on earth reports what would have happened. This is why attribution is less a measurement discipline than a genre of persuasive fiction.
Slow things. Trust, reputation, brand, craft, the accumulating cost of a thousand small indignities. These move on timescales longer than your reporting cadence and your average tenure, which means they are invisible right up until they are the only thing anyone can see.
The audit question
One question, asked on a schedule, out loud, in a room, by someone senior enough that the answer cannot be a shrug.
What could be happening right now that our instrumentation structurally cannot see?
Not "what are our data gaps," which produces a list of dashboards people want. Not "what should we measure next," which produces more tiles and therefore more confidence. The question is deliberately unpleasant. It asks a group of competent people to describe the shape of their own ignorance, which is the one shape nobody is built to perceive.
Run it quarterly. Give it an hour. Write the answers down, because the value compounds only if you can compare this quarter's blind spots to last quarter's and notice which ones you have been comfortably re-nominating for two years.
Useful ways to make people actually answer it.
- Invert an alert. For each critical alert, ask what failure mode produces silence instead of a spike. Then ask what monitors the silence.
- Interrogate a denominator. Pick one ratio on the main dashboard. Reconstruct who was excluded from the bottom of it and why. Do this with the person who wrote the query, if they still work here.
- Name the theory. Go tile by tile and state the belief about value creation each one encodes. Then ask whether the company still holds that belief. Roughly one in five will turn out to be a fossil from a strategy that was abandoned in a reorg.
- Watch ten raw sessions. Unaggregated, unsampled, uncomfortable. Aggregation is how you get precision, and precision is how you lose texture, and texture is where the unknown unknowns keep their belongings.
- Read the free text. Every survey has a box at the end that nobody analyses because it does not fit in a chart. That box is your unmodelled burst search pipeline. It is the only part of the instrument capable of returning an answer to a question you did not ask.
- Talk to a human who does not work here. Insufficiently rigorous, embarrassingly effective.
- Build one thing you expect to find nothing with. Every serious observational programme budgets for surveys with no hypothesis attached. Yours should too, and yes, it will be the first thing cut, because it has no defined success metric, which is the entire point of it.
The uncomfortable arithmetic of adding more charts
Intuition says more instrumentation means fewer blind spots. Sometimes. But coverage and felt coverage grow at very different rates, and only one of them affects decisions.
Add a chart, and the sense of visibility rises immediately. Actual visibility rises by whatever thin slice of reality that chart intersects, minus the attention it steals from everything unmeasured. Past a certain density, a dashboard stops being an observation deck and becomes an environment - somewhere you live, with weather and moods, complete enough that the question of what lies outside it stops occurring to anyone. Nobody feels ignorant while scrolling. That feeling is the failure mode.
This is not an argument for measuring less. Groups that abandon measurement do not become wiser, they become louder, and decisions migrate to whoever has the most stamina in meetings. Robert McNamara's error was never that he counted. It was that he treated the countable as the complete.
The instrument is not the problem. The instrument's silence about its own edges is the problem.
Coda, with pigeons
In 1964 two radio astronomers at Bell Labs found excess noise in their antenna that they could not eliminate. They checked everything. They eventually removed a pair of nesting pigeons and cleaned out the material the pigeons had contributed to the apparatus, which they described in print with magnificent scientific euphemism. The noise remained. That residual, the thing left after every available explanation had been exhausted, was the cosmic microwave background, which is to say the leftover glow of the beginning of everything, arriving uninvited in equipment built to study telephony.
They did not find it because they were looking for it. They found it because they had a number they could not explain and refused to round it away.
Most of your unexplained residuals really will be pigeons. Check the pigeons. But keep a standing appointment with the question of what your telescope was never built to catch, because the alternative is a perfectly polished mirror, a certifying instrument that agrees with it, and a lobby full of green tiles quietly reporting on a business you used to have.