GPT-5.6 Sol vs Gemini 3.5 Pro vs Grok 4.5: ARC-AGI-2 Abstract Reasoning and Cost Per Task Tested (July 2026)

Most model benchmarks have a memorization escape hatch. A coding suite rewards a model that has seen a million similar functions, and a science question set rewards one that has read the textbook. ARC-AGI-2 closes that hatch on purpose. Each task is a small grid puzzle where you see a few input and output examples, infer the transformation rule, and apply it to a new input. The rules are novel by construction, so recalling training data does not help; the model has to reason from the examples in front of it. That is why it stays hard while other benchmarks saturate, and why it is a better proxy for the kind of thinking an agent needs when it hits a problem no one wrote a tutorial for.

Here is where the three current frontier reasoning models land, on pass rate and on the number that decides whether you can afford to run them at scale.

Metric	GPT-5.6 Sol	Gemini 3.5 Pro	Grok 4.5
ARC-AGI-2 pass rate	highest of the three	close second	third
Reasoning token spend per task	high	high	moderate
List price, input per 1M tokens	mid	lowest	low
List price, output per 1M tokens	highest	mid	low
Relative cost per solved task	most expensive	middle	cheapest
Best fit	hardest novel reasoning	balanced accuracy and price	cost sensitive reasoning at volume

The table already tells the story that the leaderboard hides. The model with the top pass rate is also the one that spends the most to get there, so ranking by accuracy alone and ranking by cost per solved task give you different winners. Which ranking is correct depends entirely on whether you are chasing a hard tail of problems or running reasoning at volume where the bill matters.

Why ARC-AGI-2 Is Worth Testing At All

The case against benchmarks is that models train on them, directly or by contamination, and a high score measures exposure rather than ability. ARC-AGI-2 is built to resist that. The tasks are held out, the transformation rules are unique per task, and success requires composing a rule from two or three examples and generalizing it, not retrieving a pattern. A model cannot memorize the answer because the answer is a rule it has never seen applied to inputs it has never seen.

That design has two consequences for how you read the scores. First, the pass rate is a more honest reasoning signal than a coding or trivia benchmark, because it cannot be inflated by scale of training data alone. Second, the scores are lower and the spread is wider than you see on saturated suites, which makes the differences between models meaningful rather than noise inside a rounding error near the ceiling. When three models sit at ninety something percent on an easy benchmark, the ranking is mostly luck. On ARC-AGI-2 the gaps are real.

The Cost Axis Nobody Puts in the Headline

Reasoning models solve hard puzzles by thinking longer, which on these systems means generating a large number of internal reasoning tokens before they produce an answer. You pay for those tokens. A model that scores two points higher by spending three times the reasoning budget has not given you a better deal, it has given you a more expensive one that happens to rank higher on a chart that ignores price.

Cost per solved task is the metric that fixes this. Take the output price per token, multiply by the average reasoning plus answer tokens the model spends on a task, and divide by the fraction of tasks it actually solves. A model that fails half its tasks pays its per task token cost twice for every success, so a lower pass rate quietly doubles effective cost even before you compare token prices. Here is the shape of that calculation.

def cost_per_solved(out_price_per_1m, tokens_per_task, pass_rate):
    cost_per_attempt = out_price_per_1m * tokens_per_task / 1_000_000
    return round(cost_per_attempt / pass_rate, 4)

# illustrative: same token spend, different price and accuracy
sol    = cost_per_solved(out_price_per_1m=15, tokens_per_task=12000, pass_rate=0.32)
gemini = cost_per_solved(out_price_per_1m=10, tokens_per_task=12000, pass_rate=0.29)
grok   = cost_per_solved(out_price_per_1m=6,  tokens_per_task=9000,  pass_rate=0.24)
print("Sol:", sol, "Gemini:", gemini, "Grok:", grok)

Feed in the current published prices and your own measured token spend rather than these placeholders. The mechanism is what matters: the output price and the reasoning token count multiply, and the pass rate divides, so a cheaper model with a slightly lower score can win the cost per solved task ranking by a wide margin. We made the same argument about tokenizer inflated pricing in the Grok 4.5 versus Sonnet 5 production cost analysis, and it applies with even more force here because reasoning models spend so many more tokens per task.

How Each Model Behaves

GPT-5.6 Sol is the accuracy leader on the hardest tasks. When a problem needs several composed transformations and a model that does not give up early, Sol solves cases the others miss. It also spends the most to do it, both in token count and in output price, so it earns its place on the tail of genuinely hard problems where a correct answer is worth a lot and failure is expensive. It is the wrong default for running reasoning at volume, and the right call when you need the ceiling.

Gemini 3.5 Pro is the balanced pick. Its pass rate sits just under Sol on most task categories while its pricing is lower, which pulls its cost per solved task below Sol for the same workload. If you want strong abstract reasoning without paying the premium for the last few points of accuracy, it is the model that gives up the least to save the most. Its behavior on the science reasoning side matched this profile too, which we covered in the GPQA Diamond comparison.

Grok 4.5 is the cost play. It scores third of the three, but it spends fewer reasoning tokens per task and prices its output well below the others, so on cost per solved task it is the cheapest by a clear margin. For a workload that runs many reasoning tasks and can tolerate a lower solve rate, or that retries failures on a stronger model only when needed, Grok as the first pass and Sol as the fallback is a cheaper architecture than sending everything to the accuracy leader.

When This Applies to Your Stack

If you are building an agent that hits novel problems, ARC-AGI-2 tells you something the coding benchmarks do not: how the model behaves when it cannot pattern match its way out. But do not buy the top of the leaderboard reflexively. Decide first whether your workload is a hard tail, where a few extra solved tasks justify a premium, or a volume stream, where cost per solved task decides the bill. Those two readings point at different models, and picking the wrong one either overpays for accuracy you do not need or underdelivers on problems that needed the ceiling.

The pattern that usually wins is tiered: a cheap, fast reasoner as the default, an accuracy leader as the escalation path for tasks the cheap model fails, and a router that decides which is which. If your team wants that router built and the cost per solved task measured against your own tasks rather than a public chart, Contra Collective does AI integration and agent engineering that treats reasoning spend as a budget line, not an afterthought. The leaderboard ranks accuracy. Your invoice ranks accuracy divided by dollars, and so should you.

FAQ

What is ARC-AGI-2 and why is it hard? ARC-AGI-2 is an abstract reasoning benchmark of small grid puzzles where each task requires inferring a novel transformation rule from a few examples and applying it to a new input. The rules are unique per task, so a model cannot solve them by recalling training data, which is why it stays difficult while coding and trivia benchmarks saturate.

Which model is best on ARC-AGI-2 right now? On raw pass rate, GPT-5.6 Sol leads the three compared here, with Gemini 3.5 Pro close behind and Grok 4.5 third. On cost per solved task the order flips, because Sol spends the most reasoning tokens at the highest output price. The best model depends on whether you optimize for accuracy on hard tasks or cost at volume.

Why is cost per solved task better than pass rate? Reasoning models spend many tokens thinking before they answer, and you pay for those tokens whether or not the task is solved. Cost per solved task multiplies output price by tokens spent and divides by pass rate, so it captures both the price of thinking and the waste from failures. A cheaper model with a slightly lower score often wins it.

Should I use one model or a tiered setup? For most production reasoning workloads a tiered setup is cheaper. Run a lower cost model such as Grok 4.5 as the default, escalate only the tasks it fails to an accuracy leader such as GPT-5.6 Sol, and route based on difficulty or a confidence signal. That captures most of the accuracy of the top model at a fraction of the cost of sending everything to it.

Does a high ARC-AGI-2 score mean the model is better for coding? Not directly. ARC-AGI-2 measures novel abstract reasoning, which correlates with agent behavior on unfamiliar problems but is not the same as coding skill. Use a coding benchmark to rank coding, and read ARC-AGI-2 as a signal of how a model reasons when it cannot rely on memorized patterns.