Tetlock Built a Neural Network Out of Humans

And the forecasting industry still doesn’t have an engine under the bonnet

Something funny happened this week.

I had a conversation with someone from RAND’s Forecasting Initiative. A Superforecaster, one of the top-ranked forecasters in the world. She wanted to talk about my model and how it might plug into a forecasting platform.

So I did what any reasonable person would do. I went and looked at what forecasting platforms actually do.

Sliders. They use sliders.

I need you to sit with that for a moment. The cutting edge of geopolitical prediction, the methodology that beat intelligence analysts with access to classified information, is smart people moving a slider between 0% and 100%.

That’s it. That’s the product.

The greatest guessing game ever played

Philip Tetlock’s insight was genuinely brilliant, and I don’t want to undersell it. In 2011, IARPA — the intelligence community’s version of DARPA — launched a massive forecasting tournament. Thousands of volunteers were asked to predict geopolitical events. Would Assad still be in power in six months? Would the EU impose new sanctions on Iran?

Most people were terrible at it. This surprised nobody.

But some people were consistently, measurably better than everyone else. Better than the crowd average. Better than prediction markets. Better than professional intelligence analysts sitting in Langley with access to classified briefings. Tetlock called them Superforecasters, and the name stuck.

The methodology that made them good is well-documented. Think probabilistically. Update your estimates when new information arrives. Work in teams to challenge each other’s reasoning. Keep score relentlessly. Stay calibrated, when you say 70%, it should happen about 70% of the time.

This is valuable. This works. It produces measurably better predictions than virtually any alternative.

But here’s the thing nobody says out loud.

There’s no model underneath.

Pattern recognition all the way down

When a Superforecaster assigns 73% probability to an event, they’re not running a calculation. They’re not deriving the probability from a structural model of how political systems work. They’re synthesising information, cross-referencing analogies, adjusting for base rates, and arriving at a number that feels right to their well-calibrated judgment.

It’s expert intuition with exceptionally good hygiene.

The calibration training is real. The bias reduction is real. The team deliberation process genuinely improves accuracy. But the underlying mechanism is still: smart humans processing information and producing a probability estimate based on pattern recognition and experience.

Sound familiar?

It should. Because that’s exactly how a neural network works.

The biological neural network

Think about what Tetlock actually built.

You take a large pool of human nodes, thousands of volunteers. Each one has different training data (life experience, domain knowledge, reading habits). Each one has different weightings (political priors, analytical style, risk tolerance). You put information into the network and each node produces an output.

Then you score the outputs. Ruthlessly. The nodes with the lowest error rates get promoted. The best ones get clustered into teams — layers, if you like — where they can process information through deliberation before producing a refined output.

The scoring system is the loss function.
The team structure is the network architecture.
The calibration training is gradient descent.

And the Superforecasters themselves are the nodes that survived pruning because they consistently minimised prediction error.

Tetlock built a neural network. He just built it out of people instead of silicon.

We’ve seen this architecture before

Once you start looking for it, the pattern shows up elsewhere.

During Russia’s invasion of Ukraine, a loose online community known as NAFO (you may have heard about them), emerged on social media. On the surface it looked like memes and cartoon dogs.

Underneath, it behaved like something much more interesting.

Thousands of participants were constantly scanning open-source information, sharing links, flagging propaganda, and rapidly distributing corrections. Claims would be stress-tested in public threads. Good information propagated. Bad information was mocked, challenged, or discarded.

There was no central command. No formal hierarchy. No institutional funding.

Just a distributed network of humans processing information in parallel.

Sound familiar?

Tetlock’s forecasting tournaments built a formalised human neural network.
NAFO functioned as a spontaneous one.

Different purpose. Same architecture.

Nodes with different training data.
Rapid information exchange.
Feedback through social scoring.
Outputs that converge toward a collective judgement.

In machine learning terms, NAFO looked less like a carefully engineered model and more like a chaotic but highly effective swarm system.

The Ceiling

And just like a silicon neural network, these systems share the same fundamental limitation.

They work.

But they can’t tell you why they work.

When a Superforecaster assigns 73% probability to an event, they’re not running a structural model. They’re synthesising information, adjusting for base rates, comparing historical analogies, and arriving at a number that feels right to their trained judgement.

It’s expert intuition with exceptionally good hygiene.

Which is exactly what neural networks do.

They detect patterns.

They minimise error.

But they don’t explain the causal mechanisms generating the patterns.

The missing engine

The Superforecasters know this, by the way. The smartest ones have been writing about it.

The RAND Forecasting Initiative has published work acknowledging that AI struggles with causal reasoning and counterfactual thinking, the “what would happen if...” questions that require understanding *mechanisms*, not just patterns. And they’re right. AI is excellent at correlation. It’s poor at causation.

But here’s what’s strange: the Superforecaster methodology has exactly the same limitation. It can tell you *that* something will probably happen. It can’t tell you *why* the incentive structures make it likely. It can produce a well-calibrated probability. It can’t derive that probability from a formal model of how political actors respond to incentive changes.

The forecasting industry has spent a decade optimising the driver. Nobody’s built the car.

What would an engine look like?

A structural model of political behaviour would change the game completely. Instead of asking “what do the smart people reckon?”, you’d ask “what do the incentive structures predict?”

Not replacing the Superforecasters, complementing them. Giving them a causal framework to anchor their probabilistic judgments. A model that says: here’s why actors in this kind of institutional environment, facing these incentive structures, are likely to behave this way. Now calibrate your probability around that structural prediction instead of around your gut.

That’s the difference between weather forecasting before and after we understood atmospheric physics. Before the physics, we had experienced sailors looking at the sky and saying “I reckon it’ll rain.” Some of them were remarkably good at it. But the forecasts got dramatically better once we understood the causal mechanisms driving weather systems.

The sailors didn’t become obsolete. They became better, because they had a model underneath their intuition.

Pro slider movers

I don’t say any of this to denigrate the Superforecasters. Tetlock’s work is genuine science and the people who’ve risen to the top of the forecasting tournaments are extraordinarily talented. Being consistently well-calibrated across hundreds of diverse geopolitical questions is a rare and valuable cognitive skill.

But let’s be honest about what it is. It’s professional slider-moving. Extremely well-calibrated, rigorously scored, team-refined slider-moving. The best guessing game ever designed.

The next step isn’t better guessing. It’s building the structural models that can tell us *why* the slider should be where it is.

And that’s a conversation I’m looking forward to having.

---

*Matthew Pearce is an economist whose framework, “The Rent Theory of Political Identity,” models political identity as a rent-bearing asset under contemporary attention markets. The paper is currently in peer review at Constitutional Political Economy.*

*All proceeds from this Substack are donated to Ukrainian causes.*