RealityBench: Simulated Worlds to Evaluate a Vision Language Model’s Ability to Document Human Physical Work

Can models fully perceive physical work? Not yet.

Guiding workers in the physical world has been a decades-long goal for AI agents. A key capability is understanding what the worker is doing, seen through their camera or smart glasses, and helping them document it. How well vision language models do this is uncertain and hard to benchmark: most physical work data is proprietary, and field tasks such as rooftop solar installation and vehicle repair are unconstrained and high variance.

We model real-world physical work in Blender across eleven environments, with two tasks. Acceptability asks whether a view would be suitable proof of work. Documentation asks whether an agent can understand the task-relevant state of the environment from that view.

From our evaluation of 35 vision-language models, OpenAI’s Astra, Anthropic’s Fable and Gemini Flash perform well on documentation, but are weak at reading measurements and at judging small parts and spatial relations. On-device models such as Google’s Gemma and Liquid’s LFM reach about 80% of the top documentation scores, which is promising for these workloads.

arXiv paper HF datasets

2:33

A benchmark you can extend, not a fixed image set.
We provide eleven simulated environments across rooftop solar and vehicle repair that you can modify and use to generate grounded evals. Each environment has programmatic scorers for acceptability (validated against human raters) and documentation.
Models read well but measure and judge poorly.
We evaluated 35 vision-language models on our environments and observed that top models largely solve documentation when the answer is text in frame, but are weak at reading measurements and at judging small parts and spatial relations.
Simulation predicts reality.
On three questions with matched real photographs, model rankings on rendered frames carry over (Spearman ρ 0.63–0.79). Released: 133 tape-measure readings on 84 photos.
Models fail at seeing, not at understanding the rule.
Where errors can be traced, wrong rejections are mostly misread scenes, not misread requirements: 97% for the tape gap, 74% for the slope gauge, 63% for the rail attachment.

Where is there room to grow?

Models have all but solved the tasks that come down to reading text in the photo: license plates, equipment labels and breaker panels. The gap is in measuring and in judging small parts and spatial relations: no model reads the number on a tape measure right even half the time.

Leaderboard

From the RealityBench dataset on Hugging Face. 35 models, one query per item, each at its provider’s or model card’s default settings, 13–23 September 2026 (models marked † in October 2026). Sorted by acceptability F1; best model score in each column in bold.

Model	Acceptability			Documentation		Cost ($)
Model	F1	Precision	Recall	Score	In grammar (%)	Cost ($)
GPT-6 Astra	0.61	0.55	0.68	0.83	100	30.05
Gemini 4 Argon, added after the paper	0.60	0.49	0.78	0.86	100	4.96
GPT-6.1 Sol, added after the paper	0.59	0.52	0.70	0.82	100	6.22
Claude Opus 5.5	0.58	0.51	0.67	0.83	100	16.36
GPT-5.6 Sol	0.57	0.49	0.69	0.77	100	17.65
Claude Sonnet 5.5, added after the paper	0.57	0.56	0.58	0.79	100	7.70
GLM-5.3 Flash	0.57	0.49	0.68	0.75	99	1.30
Gemini 3.5 Flash	0.56	0.49	0.66	0.80	97	7.36
Claude Fable 5.1	0.56	0.51	0.63	0.83	100	41.35
Cloudflare Clef, added after the paper, a decision model	0.56	0.47	0.68	not scored	not scored	0.40
Claude Opus 5	0.56	0.54	0.57	0.75	100	23.32
Qwen 3.8 Max	0.55	0.43	0.78	0.82	100	38.19
GPT-6 Sol	0.55	0.50	0.61	0.78	100	8.13
Muse Spark 1.3	0.55	0.49	0.62	0.79	100	13.22
Qwen 3.5 397B-A17B	0.55	0.49	0.61	0.76	99	15.03
GPT-5.6 Terra	0.55	0.45	0.70	0.76	100	8.49
GPT-6.1 Terra, added after the paper	0.54	0.54	0.55	0.84	96	3.60
Qwen 3.8 27B	0.54	0.45	0.68	0.74	99	12.36
Grok 4.6	0.54	0.43	0.72	0.78	100	24.88
Gemini 3.8 Flash	0.54	0.48	0.62	0.84	100	7.16
Muse Wonder, added after the paper	0.54	0.51	0.56	0.79	100	45.99
Elorian Glimpse, added after the paper	0.53	0.43	0.70	0.76	99	18.73
Kimi K3	0.53	0.42	0.70	0.77	99	39.31
GPT-5.6 Luna	0.53	0.46	0.62	0.71	100	1.18
Gemma 4 31B	0.52	0.44	0.63	0.79	100	0.70
DeepSeek V4.1 Flash	0.50	0.39	0.68	0.68	95	8.67
Gemma 4 26B-A4B	0.48	0.39	0.64	0.77	99	1.24
Claude Sonnet 5	0.48	0.45	0.51	0.73	100	4.80
TypeSafe Jev 1.14, added after the paper, a decision model	0.47	0.57	0.40	not scored	not scored	0.15
Gemini 3.1 Flash Lite	0.47	0.39	0.58	0.73	98	0.73
Claude Haiku 5.5, added after the paper	0.46	0.49	0.44	0.75	100	0.56
LFM2.5-VL-3B	0.43	0.30	0.79	0.68	99	no list price
GPT-6 Luna	0.43	0.46	0.41	0.62	100	0.69
Gemma 4 E4B	0.42	0.30	0.70	0.69	98	no list price
Qwen3.5-4B	0.39	0.31	0.50	0.66	95	no list price
Human rater	0.59
Always “adequate”	0.33	0.19	1.00

A single non-author rater against the scorer, on 274 rated frames rather than this set (95 % CI 0.50–0.67).

Cost: the developer’s list price times the tokens used for all 1,378 items, thinking at the output rate, no cache discount; – for models we served ourselves, which have no list price.

Not in the paper: run in October 2026 on the same frames with the same prompts, settings and scoring; prices read 13–26 October 2026; Elorian Glimpse at OpenRouter’s price (no developer list price).

Cloudflare Clef and TypeSafe Jev 1.14 are decision models (OpenRouter’s Decisions API): they answer typed questions with probabilities and write no text, so they are scored on acceptability only, asked as a yes/no question with the same requirement (adequate at probability 0.5 or more). Priced at their developers’ own endpoints on OpenRouter, input tokens only (output is free). Intervals, per-environment scores and every model’s answers are in results/.

A benchmark you can extend, not a fixed image set.

Models read well but measure and judge poorly.

Simulation predicts reality.

Models fail at seeing, not at understanding the rule.

Where is there room to grow?

Leaderboard