CRAFT Benchmark
Multi-agent coordination under partial information · 15 director models · 20 structures · 3 directors + 1 builder · oracle-assisted evaluation
| # |
Model |
Type |
Task progress ↓ |
IOU |
Oracle adh. |
Pos. acc. |
Complexity split |
| # |
Model | Type |
Task progress |
IOU |
Oracle adh. |
Pos. acc. |
Fraction of oracle-turn budget by failure category. Hover bars for exact values. Sorted by correct rate descending.
Beyond task progress — how do these models behave as teammates? Sorted by Teammate Score.
Teammate Score = low chatter × low panic × low repetition ·
Overthink Ratio = thinking length / message length ·
Grounding = spatial coordinate references in reasoning