News
What Is TasteVal? The AI “Research Taste” Benchmark, Explained
TasteVal is a new benchmark that tests whether AI can design and judge its own experiments. What it measures, how Opus 5.5 beat experts, and why it matters.
TasteVal is the benchmark behind this week’s headlines that Anthropic co-founder Jack Clark now thinks AI could start improving itself in 2027. If you saw the name and wondered what a “taste” test has to do with AI, you are in the right place.
Here is what TasteVal measures, who made it, what the results say, and what the numbers do and do not prove.
What is TasteVal?
TasteVal (full title: “TasteVal: Measuring the Experimental Research Taste of AI Systems Against Human Experts”) is a test for one specific skill. Can an AI decide which experiments to run, and then draw the right conclusions from what happens?
Think of two cooks given the same kitchen and the same dish to make. A good cook wastes fewer ingredients getting there because they make smarter choices. TasteVal asks the same thing of AI models, except the “ingredients” are computer time and the “dish” is a better AI system.
The paper splits research taste into three parts: picking good problems, designing experiments, and interpreting results. TasteVal only tests the last two. The problem is fixed, and the model has to work out how to solve it.
How does TasteVal work?
There are 8 open-ended tasks taken from real AI development work, such as choosing training data, pretraining and fine-tuning language models, training on human preferences, and making a model resist adversarial prompts.
The model being tested plays the “researcher.” It never writes the code itself. A fixed coding agent builds and runs whatever experiments the researcher asks for on a single Nvidia H100 GPU. That way, a model with great coding skills but poor judgment does not get a free boost. Each attempt stops after 40 GPU-hours or 120 hours of real time.
The score is called a compute multiplier. If a model reaches the expert’s score using half the experiment compute, it has a multiplier of 2x, meaning twice the taste. The human bar is the best attempt from 24 experts (at least two per task) who recently worked at places like OpenAI, Google DeepMind, Nvidia and Microsoft Research.
TasteVal results: what the numbers say
The authors tested 20 models released between 2023 and 2026. These are the headline findings from the paper:
| Finding | Number |
|---|---|
| Best model | Claude Opus 5.5 |
| Opus 5.5 compute multiplier vs best human expert | 2.3x (95% range: 1.15x to 4.37x) |
| Opus 5.5 cost per attempt | About 1/30 of the average expert attempt |
| How fast taste improves since December 2025 | Doubles about every 3.0 months (range 1.7 to 5.0) |
| How fast it improved from 2023 to December 2025 | Doubled about every 14 months |
| Tasks in the benchmark | 8, kept private to avoid contamination |
| Human experts in the baseline | 24 |
The big claim is the speed-up. Before December 2025, models got better at research taste slowly. Since then, the doubling time has dropped to about three months. One detail worth knowing: when the authors measured raw final scores instead of compute savings, they found no such jump, just a steady doubling about every 14.6 months.
Opus 5.5 is the same model we covered in What is Claude Opus 5.5?
Why Jack Clark cited TasteVal
Jack Clark co-founded Anthropic and writes the Import AI newsletter. In Import AI 476, published October 12, 2026, he pointed to TasteVal and recent AI math results and wrote: “I now think it’s more likely we’ll see full RSI become possible in 2027.” In a May essay he had leaned toward 2028.
RSI means recursive self-improvement: AI systems that do the research to build their own, better successors. Research taste is one piece of that, since a model that can choose good experiments needs fewer humans steering it. The Decoder covered the forecast on October 25, and many outlets picked it up on October 26 and 27.
The Decoder also noted the obvious tension: Clark co-founded the company whose model tops the benchmark. TasteVal itself was not made by Anthropic.
What TasteVal does not prove
The authors are upfront about the weak spots, and they are worth reading before you believe any headline:
- It may overstate progress. The tasks are quick and cheap to check, which makes them easy for AI labs to train toward. The human group did not include the very top researchers, who may be much better.
- It may understate progress. The models got far less effort per attempt than the humans did, so better setups could score higher.
- It is narrow. Eight tasks, one GPU, a fixed problem. Real research also means picking which problem matters, which TasteVal leaves out.
The authors call their long-range extrapolations naive, not forecasts. Treat the 2027 talk as one informed person’s guess, not a measured fact.
TasteVal vs Anthropic’s TASTE benchmark
The names are easy to mix up. Anthropic’s Alignment Science team published a different benchmark called TASTE (The AI Safety Taste Evaluation) on August 28, 2026. TASTE asks models to pick the better of two AI safety research proposals, across 92 pairs. Experienced human researchers agree with its labels about 77% of the time, while the best model, Fable 5, scored 60%.
| TasteVal | TASTE | |
|---|---|---|
| Made by | P-Zero Research (Jaffe, Sherburn) | Anthropic Alignment Science |
| Published | October 5, 2026 | August 28, 2026 |
| What the model does | Runs real experiments and reads results | Judges pairs of written proposals |
| Did AI beat humans? | Yes, Opus 5.5 at 2.3x | No, Fable 5 at 60% vs about 77% |
What this means for you
You will never run TasteVal yourself. But the skill it measures is the same one that makes an AI agent useful in daily life: breaking a goal into steps, trying something, looking at what happened, and adjusting. That loop is what separates an agentic AI from a chatbot that just answers once.
If models really are getting better at that loop every few months, the practical win is simple: you can hand off bigger jobs. Not “write me a sentence,” but “compare these three apartments, check the commute, and draft the email to the landlord.”
That is what our app is for. Agentic AI gives you ready-made expert agents on iPhone and iPad, running on frontier models. You can put up to 4 agents on one task, there is no account to create, and your first 3 tasks are free.
New to terms like RSI or compute? The agentic AI glossary explains them in plain English.
For a fresh example of AI doing research, see OpenAI’s claim that its models proved the Exponential Time Hypothesis false.
Quick answers
What is TasteVal? A benchmark that measures how well AI models design experiments and interpret results in AI research, scored by how much compute they save compared with expert humans.
Who made TasteVal? Oliver Jaffe and Dane Sherburn of P-Zero Research. The paper went on arXiv on October 5, 2026. It is not an Anthropic benchmark.
Which AI model scored best on TasteVal? Claude Opus 5.5, with a 2.3x compute multiplier over the best human expert, at about 1/30 of the cost per attempt.
Can I run TasteVal myself? No. The paper is free to read, but the 8 tasks are kept private so labs cannot train on them.
What does TasteVal have to do with AI self-improvement? Jack Clark cited it when he moved his forecast for recursive self-improvement, AI building better versions of itself, from 2028 to 2027.
Is TasteVal the same as Anthropic TASTE? No. TASTE is Anthropic’s separate benchmark where models judge AI safety research proposals. Models still score below humans on it.
Sources: arXiv: TasteVal paper · LessWrong: TasteVal summary by the authors · Import AI 476 · The Decoder · Anthropic Alignment Science: TASTE
