The Computational Deliberation Lab

The Computational Deliberation Lab

An independent research labEvaluation research underway

AI can make our thinking far better and faster. I'm learning how to tell when it does.

An independent research lab designing evaluations for open-ended deliberation, and building the systems to test them on - from business strategy to science, philosophy and policy.

I use AI in my own deliberation every day and can feel it getting better. The questions that matter most have no answer key, though, so feeling it isn't the same as knowing it.

Pressure-test a decisionRead the open questions
FIG. 1 · THE MEASUREMENT PROBLEMILLUSTRATIVE
DECISIONMONTHSYEARS LUCK, AND EVERYTHING ELSE PATH NOT TAKEN · NEVER OBSERVED PATH TAKEN OUTCOME · LATE, NOISY, OR NEVER DECISIONExpand to the USbefore the Series A? Was the thinking good?THE ANSWER IS NEEDED HERE, NOT YEARS LATER
A deliberation has to be judged when the decision is made. The outcome that might vindicate it arrives late and tangled with luck, and the path not taken is never seen at all.
§01 · The problem

We are about to hand our hardest questions to systems we have no way to grade.

i.

There's no answer key.

Whether to enter a market, which theory to fund, how to regulate a technology: nobody can look up the right answer and check the reasoning against it.

ii.

Outcomes arrive late, and mixed with luck.

Good reasoning can end badly and careless reasoning can get lucky, so waiting for results teaches less than it seems to, and the road not taken is never seen.

iii.

Fluency looks like rigor.

The better models get at making a case, the harder it becomes to tell a well-tested conclusion from a well-written one. Judging their output gets harder exactly as it gets more persuasive.

§02 · Why now

AI made ideas cheap and left checking them as expensive as ever.

Every consequential decision runs on two budgets: one for building the case and one for attacking it. The second is almost always the smaller one, and I call the gap the refutation deficit.

Models now draft well-specified claims, plans and strategies almost for free, so the first budget keeps growing while the second hasn't moved. The gap is where AI either makes our deliberation better or just makes it louder, and we can only close it if we can measure which is happening.

Read the essay: The Refutation Deficit
Effort to find out whether a claim is true Effort to produce a well-specified claim THE GAP produced faster than it can be checked BEFORENOW EFFORT PER CLAIM
FIG. 2 · SCHEMATIC, NOT MEASURED
§03 · What I'm building

Ways to measure deliberation, and machines to measure it on.

The core work · Evaluations

Telling a strong deliberation from a fluent one

Ways to judge the quality of open-ended reasoning, by people or machines, before the outcome comes back. This is the hard part, and the part I think will matter most as models improve.

Testbeds · Deliberation systems

Systems I build to test the evaluations on

Including a deliberation calculator: a machine that maps an open-ended problem, tests what the map contains and shows what survives. I run them on real decisions to learn their limits and what's possible.

FIG. 3 · DELIBERATION CALCULATOR OUTPUT, AS A LEDGERILLUSTRATIVE
FACTTwo US pilots are signed; neither is paid.Contracts in hand
Every entry carries a status, so the reasoning can be inspected - and, eventually, measured.
POSSIBILITY · CONSIDEREDWin one paying reference at home first.Kept as the live alternative
Alternatives stay on the map after they lose, so you can see what the choice was made against.
BET · SURVIVEDA US reference unlocks the Series A.Attacked twice, still standing
Survived means attacked and still standing. It does not mean proven.
ASSUMPTION · RULED OUTUS buyers close inside a quarter.Evidence: both pilots stalled in procurement
Whatever falls carries the evidence that felled it, so you can disagree with the evidence instead of the conclusion.
NEVER EXAMINEDWhat it takes to hire a US seller.Flagged, not tested
The map marks where it's thin. Knowing what nobody thought to ask is still an open research problem.
VERDICT[ left to you ]
The system builds the map. The verdict stays yours.
§04 · Open questions

What I'm trying to find out, in the order I'm trying to find it.

These are questions, not results. When one gets an answer I trust, it gets published here, including the answers that go against me.

Q1Can a deliberation be judged well before its outcome arrives?UNDERWAY
Q2Can a well-tested conclusion be told apart from a well-written one?OPEN
Q3When ten runs of the same model agree, is that ten opinions or one?OPEN
Q4Which methods for thinking well still work when a machine runs them?OPEN
Q5How do you find what nobody thought to ask?OPEN
Q6Does a measure built on business decisions travel to science and policy?LATER
§05 · How I work

A lab that studies judgment has to hold its own claims to the same bar.

01 · Falsifiers

Every claim carries the sentence that would prove it wrong.

02 · Bars first

What counts as success is written down before the results come in.

03 · Losses published

Results that kill my own ideas get published too.

04 · Human verdict

Machines build the map; a person makes the call.

SEE IT APPLIED · Lab note No. 0 shows how one research program was run, and everything it did not show

§06 · Where this goes

The work starts in strategy and is headed for science, philosophy and policy.

A measure of good deliberation is only worth having if it travels. Business strategy comes first because the decisions are real and the feedback arrives relatively fast. The aim is the questions where better deliberation would matter most.

Now

Business strategy

Real decisions, feedback within months, and founders and investors willing to have their thinking attacked.

Next

Science

Rival explanations, ambiguous evidence, and results that can take years to settle.

Next

Philosophy

No decision at the end, only understanding - and arguments that have to survive their strongest objection.

Next

Policy and shared problems

Many stakeholders, values in conflict, and outcomes nobody will ever observe cleanly.

§08 · Work with the lab

Bring one real decision.

The research runs on real cases. A small number of founders and investors get a live decision pressure-tested with the systems the lab builds, and every case is anonymized before anything about it is published.

For founders

A decision you can't stop re-litigating.

A market, a raise, a hire, a pivot. You get back a tested map of it: what it rests on, what was attacked, what survived, and what nobody examined.

Pressure-test a decision
For investors

One investment, reviewed before the money moves.

A fixed-fee, five-day adversarial review of one live decision: what survives, what doesn't, three questions to put to the founder, and the condition that should make you walk away.

Request a pre-commit review
NOWComputational Deliberation Lab
A DECADEFounder and CEO, Joyned
BEFOREMission control, Israeli Air Force
ANDWilderness guide
TRAINED INPhysics and cognitive science
§09 · Who runs the lab

For a decade I ran a company built on one question: how do people decide together?

That company was Joyned. When it wound down, the question stayed with me. Now I use AI in my own deliberation every day and can feel it getting better and faster, and the part I can't stop working on is how to show that it actually is.

I'm Michael Levinson. Before Joyned: physics and cognitive science, mission control in the Israeli Air Force, and years as a wilderness guide, where a decision that doesn't survive contact with the terrain finds out quickly.

More about the lab
§10 · The question behind all of this

How do you know a deliberation was good when its answer never comes back?

Follow along as it gets worked out: each note and each result as it lands, including the ones that go against me.

Computational Deliberation LabBetter deliberation, and a way to know it
Essays
AboutArtifacts