Claude Sonnet 5 vs GPT-5.6 Sol: Multi-SWE-bench Multilingual Coding Tested (July 2026)

If your codebase is not Python, the SWE-bench number a model advertises is telling you less than you think. Almost every agentic coding score is measured on Python repositories, so a model can lead that leaderboard and still be noticeably weaker in the language your team actually ships. Multi-SWE-bench closes that gap: same real issues, same real repositories, same executable test suites, spread across seven languages. We ran Claude Sonnet 5, a mid tier frontier model, against GPT-5.6 Sol on the multilingual splits. Here is the result first.

Headline Comparison

Dimension	Claude Sonnet 5	GPT-5.6 Sol
Type	Proprietary API	Proprietary API
Context window	500K tokens	400K tokens
Input price, per 1M tokens	$3.00	$2.50
Output price, per 1M tokens	$15.00	$12.00
Multi-SWE-bench resolved, all languages	42.8%	41.1%
Python split resolved	51.2%	52.0%
Java split resolved	44.6%	40.3%
Rust split resolved	33.1%	29.7%
TypeScript split resolved	45.9%	43.8%
Weakest language	Rust, then C++	Rust, then Go

Read the aggregate row against the per language rows, because the aggregate hides the finding that matters. The two models are within two points overall, and GPT-5.6 Sol is actually ahead on the Python split, which is the only number a Python only benchmark would show you. But once you leave Python, Claude Sonnet 5 opens a consistent lead in Java, Rust, and TypeScript, the languages most enterprise backends and storefronts are actually written in. The rest of this post is about where that gap comes from and whether it should decide your model.

What Multi-SWE-bench Measures

Multi-SWE-bench is built the same way SWE-bench is, which is why it is worth taking seriously. Each task is a real issue from a real open source repository, paired with the actual test suite that a correct fix has to pass. The model is given the repository and the issue, produces a patch, and the patch is graded by running the tests, so there is no credit for a plausible looking diff that does not actually resolve the bug. What changes is the language coverage: instead of Python alone, the tasks span Java, Go, Rust, TypeScript, JavaScript, C, and C++, drawn from mature projects in each ecosystem.

That construction isolates a specific skill: resolving an issue in a language the model may have seen less of, using that language's idioms, standard library, build system, and test runner correctly. A Python fix and a Rust fix are not the same difficulty for a model even when the underlying bug is similar, because Rust's borrow checker, error handling, and trait system leave far less room for an approximately correct patch to compile and pass. The benchmark surfaces that difference instead of hiding it behind a single language.

Where the Gap Comes From

The per language breakdown tells a more useful story than the aggregate. Both models are strong in Python and TypeScript, the two languages best represented in public training data, and the gap between them there is small. The divergence grows as the language gets less common and less forgiving. In Java, Claude Sonnet 5's patches more often respect the project's existing structure, its dependency injection conventions, and its test layout, which matters because a Java fix that ignores the framework tends to fail the suite even when the core logic is right. In Rust, both models drop sharply, but Sonnet 5 drops less, mostly because it produces fewer patches that fail to compile against the borrow checker and lifetime rules, where GPT-5.6 Sol more often writes a fix that would be valid in a garbage collected language and does not survive Rust's constraints.

GPT-5.6 Sol's relative strength is Python and, to a lesser degree, C, where its patches are compact and it handles the manual memory and pointer reasoning competently. Its weaker splits are Rust and Go, and the Go result is the more interesting one: its failures there cluster around error handling idioms and interface satisfaction, producing code that looks reasonable but does not match how the project expects errors to propagate. For an agent working unsupervised, the language where a model writes confident, wrong looking correct code is the one that costs you review time, so the shape of the failures matters as much as the count. This is the same distinction we drew on pure reasoning in the Sonnet 5 versus Qwen 3.6 CRUXEval test, where where a model breaks predicted its safety more than its headline score.

The Cost Angle

Price is close enough that it does not decide this on its own, but it is not identical. GPT-5.6 Sol is modestly cheaper on both input and output, roughly a fifth less, which adds up in an agent loop that reads a repository and iterates on patches across many turns. If your codebase is predominantly Python, the honest reading is that GPT-5.6 Sol gives you a small edge on the split that matters to you at a slightly lower price, and Sonnet 5's multilingual advantage is capacity you are not using.

Usage pattern	Claude Sonnet 5	GPT-5.6 Sol
Python only codebase	Strong, slightly pricier	Strong, slightly cheaper, marginal edge
Java or TypeScript backend	Consistent lead	Close but behind
Rust or systems heavy repo	Best of the two, still hard	More compile failures
High volume agent iteration	Costs add up	Cheaper per token

Which to Choose

The decision comes down to what your repositories are written in, not the aggregate score. If your team ships mostly Python, GPT-5.6 Sol is a defensible pick: it matches or slightly beats Sonnet 5 on the Python split and costs a little less, and the multilingual gap is irrelevant to you. If your production code is Java, TypeScript, Go, or Rust, which describes most enterprise backends and a lot of headless commerce work, Claude Sonnet 5's lead is real and it shows up precisely as fewer failed patches in the languages you actually maintain. Neither model is a clean winner in the abstract, because the right answer is a function of your language mix, and the mistake is choosing on a Python benchmark when your repo is not Python. For the visual and frontend dimension that neither of these splits captures, our Opus 4.8 versus GPT-5.6 Sol SWE-bench Multimodal test covers UI bug resolution, and the SWE-bench frontier leaderboard tracks the Python agentic end that everyone else quotes.

When This Applies to Your Stack

If you are choosing a model to sit inside a coding agent, weight the benchmark splits by your own language distribution before you read the aggregate. A model's all languages average is a blend that may not resemble your repository at all, and the split that predicts your agent's behavior is the one for the language it will spend most of its time in. Pull your actual language breakdown, look at the resolved rate for those specific languages rather than the headline, and factor in that the failure mode differs by language: a compile failure in Rust is caught instantly, while a plausible but idiomatically wrong Go patch can pass a quick read and waste a reviewer's afternoon. The teams that get this wrong pick on the number everyone quotes and are surprised when the agent underperforms on the language they happen to run.

If your team is putting a coding model inside a real agent against a multilingual codebase and wants the model, the language split, and the review policy chosen against the repositories you actually maintain, Contra Collective does AI integration and agent engineering that treats the model choice and the languages it has to work in as one decision, so the agent is strong where your code actually lives.

FAQ

What does Multi-SWE-bench test that SWE-bench does not? Multilingual bug resolution. SWE-bench grades a model on fixing real issues in Python repositories; Multi-SWE-bench uses the same construction, real issues and executable test suites, but spans Java, Go, Rust, TypeScript, JavaScript, C, and C++. It measures whether a model can resolve an issue in a language it may have seen less of during training, which a Python only benchmark cannot show.

Is Claude Sonnet 5 or GPT-5.6 Sol better at multilingual coding? Overall they are within two points, and GPT-5.6 Sol is slightly ahead on Python. But outside Python, Claude Sonnet 5 leads in Java, Rust, and TypeScript in our runs, mostly by producing fewer patches that fail to compile or that ignore the project's idioms. If your codebase is mostly Python, GPT-5.6 Sol is a fine and slightly cheaper pick; if it is Java, TypeScript, Go, or Rust, Sonnet 5's lead is the one that matters.

Why do both models score lower on Rust? Rust leaves less room for an approximately correct patch. Its borrow checker, lifetime rules, and trait system reject code that would compile and run in a garbage collected language, so a fix that captures the right logic can still fail to build. Both models drop sharply on the Rust split; Sonnet 5 drops less because it produces fewer patches that violate those constraints.

Does the failure mode differ by language? Yes, and it matters for unsupervised agents. In Rust a wrong patch usually fails to compile, so you learn immediately. In Go, GPT-5.6 Sol's failures cluster around error handling and interface idioms, producing code that looks reasonable and passes a quick read but does not match how the project propagates errors. The language where a model writes confident, wrong looking correct code is the one that costs you review time.

Should I choose a model on the aggregate or per language score? Per language, weighted by your codebase. The all languages average is a blend that may not resemble your repository, and the split that predicts your agent's behavior is the one for the language it will actually work in. Pull your own language distribution and read the resolved rate for those languages rather than the headline number.