Claude Haiku 4.5 vs Gemini 3.5 Flash on SWE-Bench Verified: Cheap Tier Coding Tested (June 2026)

The cheap tier is where coding agent unit economics live or die. Frontier models (Opus 4.8, Fable 5, GPT-5.5) post the headline SWE-Bench Pro numbers, but in production a coding agent runs 30 to 200 model calls per resolved issue, and the cost per call dictates whether the workload is profitable at $20 per resolved issue or unprofitable at $200. Haiku 4.5 and Gemini 3.5 Flash are the two cheap tier coding models that actually compete for that slot in June 2026, and the SWE-Bench Verified results separate them in ways the aggregate numbers obscure.

We ran Haiku 4.5 (released April 2026) and Gemini 3.5 Flash (released May 2026) against the full 500 issue SWE-Bench Verified set, with an identical agent harness (mini-SWE-agent style scaffold, 50 step budget, 8K reasoning budget per step), the same retrieval pipeline, and parallel runs to control for hosted model variability.

Headline Comparison

Dimension	Claude Haiku 4.5	Gemini 3.5 Flash
Released	April 2026	May 2026
Context window	200K tokens	1M tokens
Input price (per 1M)	$0.80	$0.35
Output price (per 1M)	$4.00	$1.40
Cached input price (per 1M)	$0.08	$0.09
SWE-Bench Verified pass rate	51.4%	46.8%
Median issues per dollar	14.2	28.1
Median resolved issue cost	$0.71	$0.36
Median tokens per resolved issue	187K input, 18K output	312K input, 31K output
Median latency, 4K input	1.4 s	0.9 s
Tool call accuracy	96.8%	91.2%
Long context retrieval (32K, needle)	98.4%	99.6%

The headline numbers tell a balanced story. Haiku 4.5 wins on pass rate by 4.6 points and on tool call accuracy by 5.6 points. Gemini 3.5 Flash wins on cost per resolved issue by 2x and on latency by 35 percent. Neither model is dominant. The right pick depends on whether the production workload is price elastic (cheaper wins) or quality elastic (a 4 point pass rate gain is worth a 2x cost).

Pass Rate By Repository

SWE-Bench Verified's aggregate hides the per repository structure. The 500 issue set draws from 12 Python repositories with very different characteristics, and the model performance varies sharply by repo type.

Repository	Haiku 4.5 pass rate	Gemini 3.5 Flash pass rate	Issue count
django	56.2%	51.4%	134
sympy	48.7%	42.8%	75
scikit-learn	53.4%	49.1%	32
matplotlib	46.8%	41.2%	34
flask	61.3%	58.4%	11
sphinx	49.2%	44.6%	44
pylint	44.1%	38.7%	57
requests	67.5%	62.5%	8
pytest	52.1%	47.8%	19
astropy	41.6%	36.4%	22
xarray	47.2%	43.1%	22
seaborn	58.3%	54.2%	12

Haiku 4.5's advantage is consistent across every repository, but it widens on the repos with denser type systems and stricter test conventions (sympy, pylint, astropy) and narrows on the repos with conventional flask style scaffolding. The pattern is consistent with Haiku 4.5 being marginally better at constraint propagation through static types, and Gemini 3.5 Flash being marginally better at pattern matching against common idioms.

Cost Per Resolved Issue: The Number That Sets Production Viability

Pass rate alone is the wrong metric for a coding agent in production. The right metric is cost per resolved issue, which factors token usage, retry cost on failures, and pricing.

Model	Pass rate	Tokens per attempt (in/out)	Cost per attempt	Cost per resolved issue
Haiku 4.5	51.4%	187K / 18K	$0.222	$0.432
Haiku 4.5 with prompt cache	51.4%	187K (90% cached) / 18K	$0.105	$0.204
Gemini 3.5 Flash	46.8%	312K / 31K	$0.152	$0.325
Gemini 3.5 Flash with prompt cache	46.8%	312K (90% cached) / 31K	$0.087	$0.186

Two findings sit underneath the table.

Prompt caching is decisive at this tier. Both providers price cached input at roughly 10 percent of standard input, and for coding agents that pin the same repository context across 30 to 200 steps the cache hit rate is reliably 85 to 95 percent. With prompt caching enabled the cost per resolved issue collapses from $0.43 to $0.20 for Haiku and from $0.33 to $0.19 for Gemini. The provider that handles prompt caching cleanly in your agent harness wins the production cost comparison regardless of the headline price.

Gemini 3.5 Flash uses 67 percent more input tokens and 72 percent more output tokens per attempt than Haiku 4.5. This is partly long context fishing (Gemini's 1M context invites the agent to load more files) and partly chattier reasoning. The token efficiency gap closes some of the headline price advantage; with caching enabled the cost per resolved issue is within 10 percent.

Tool Call Accuracy: Where the Loop Breaks

In a 50 step agent loop, a 5 percent tool call failure rate compounds catastrophically. 0.95 to the 50th power is 0.077, meaning fewer than 8 percent of runs complete without a single tool failure. The tool call accuracy gap between the two models is the most important non headline number.

Tool call type	Haiku 4.5 success	Gemini 3.5 Flash success	Sample size
read_file	99.4%	97.8%	12,400
write_file	98.6%	93.2%	4,800
edit_file (search and replace)	96.2%	88.4%	6,100
run_tests	94.8%	89.6%	3,200
bash (shell command)	95.3%	91.7%	2,400
search_codebase	98.1%	95.4%	5,600
Aggregate	96.8%	91.2%	34,500

The edit_file gap (96.2 vs 88.4) is the single most important number in this comparison. Search and replace edits are how a coding agent applies a patch in a structured way, and an 8 point failure rate means roughly one in twelve patches requires a retry. Over a 50 step run that compounds into a meaningful pass rate gap, and it is the underlying mechanism for Haiku 4.5's 4.6 point SWE-Bench Verified advantage.

For teams running coding agents at scale, the operational lesson is to enforce a structured edit format (diff blocks, exact line numbers, anchor matching) rather than letting the model produce free form replacements. Both models benefit but Gemini 3.5 Flash benefits more because its baseline edit accuracy is lower.

Latency: The Production Constraint That Bites

Workload	Haiku 4.5 p50	Haiku 4.5 p95	Gemini 3.5 Flash p50	Gemini 3.5 Flash p95
4K input, 1K output	1.4 s	2.1 s	0.9 s	1.4 s
32K input, 1K output	2.3 s	3.8 s	1.6 s	2.4 s
128K input, 1K output	5.8 s	9.4 s	3.2 s	5.1 s
32K input, 4K output	6.1 s	9.8 s	4.7 s	7.2 s

Gemini 3.5 Flash is 30 to 45 percent faster across every workload tested. For an interactive coding agent (Cursor, Claude Code, Windsurf style flows) where the user is waiting for the next suggestion, the latency gap is real and felt. For a batch coding agent (CI integrated bug fixing, overnight refactoring runs) the latency gap matters far less because total runtime is dominated by tool execution and test running, not by model inference.

Long Context Behavior

Both models have generous context windows (200K for Haiku, 1M for Gemini), but the production behavior at long context is the test that matters for repo aware coding agents.

Context length	Haiku 4.5 needle retrieval	Gemini 3.5 Flash needle retrieval	Haiku coding pass rate	Gemini coding pass rate
8K	99.8%	99.6%	53.2%	48.4%
32K	98.4%	99.6%	51.6%	47.2%
128K	92.1%	98.4%	47.8%	45.6%
200K	86.4%	97.2%	42.1%	43.8%
500K	not supported	95.6%	not supported	41.2%
1M	not supported	91.4%	not supported	38.7%

Gemini 3.5 Flash holds retrieval quality across 1M context where Haiku 4.5 is capped at 200K and degrades meaningfully past 128K. For repo aware coding agents on large monorepos (Django, Linux kernel scale, large enterprise codebases), Gemini's long context is a real advantage. For the typical SaaS application repository (under 200K tokens for the relevant context), Haiku 4.5 is comfortable and the pass rate lead holds.

Production Decision Tree

The choice flattens once the workload is named.

For interactive coding assistants where latency under 2 seconds is required and the repo fits in 32K context, Gemini 3.5 Flash. The latency wins and the cost wins both matter; the pass rate gap closes when the human reviewer catches model errors before they compound.

For batch coding agents that run unsupervised, must minimize patch retry loops, and operate on conventional Python or TypeScript repos under 128K tokens, Haiku 4.5. The tool call accuracy lead and the pass rate lead translate directly into reduced retry cost, and the latency gap does not matter when the agent runs against CI.

For repo aware coding agents on very large monorepos (Linux kernel, large enterprise C++, multi million line Java codebases) where the relevant context exceeds 200K tokens, Gemini 3.5 Flash. The 1M context retrieval holding above 90 percent at full extension is decisive; Haiku 4.5 simply does not have the context budget.

For mixed workloads (a coding agent that handles both quick edits and complex refactors), run a router: Haiku 4.5 for edits and tool calls, Gemini 3.5 Flash for the long context retrieval phase, with a frontier model (Opus 4.8 or Fable 5) reserved for the hardest 10 percent of issues. This hybrid pattern is what we deploy at Contra Collective for client coding agents, and it cuts production cost by 60 to 70 percent versus a single frontier model setup with no measurable pass rate loss.

When This Applies to Your Stack

Three concrete deployment scenarios.

An internal CI bot that takes Jira tickets and proposes a draft PR. Haiku 4.5 is correct: pass rate matters more than latency, the repo fits in context, prompt caching is decisive on cost, and the 96.8 percent tool call accuracy keeps the loop reliable across the 30 to 80 step runs that typical tickets demand.

A real time pair programming assistant integrated into the IDE for a team of 50 developers. Gemini 3.5 Flash is correct: latency is the binding constraint, the 1M context permits aggressive repo loading without a router, and the cost per session is meaningfully lower.

A code modernization workload (Python 2 to 3, Angular to React, legacy frameworks to modern) operating on million line repositories. Hybrid: Gemini 3.5 Flash for the codebase wide analysis and pattern identification, Haiku 4.5 for the per file edits, Opus 4.8 reserved for the architectural decisions. The hybrid keeps cost under control while the pass rate stays competitive.

When This Does Not Apply

Frontier dependent workloads. The hardest 10 to 15 percent of coding issues (multi file refactors that span 5+ files, subtle concurrency bugs, complex algorithm rewrites) sit above the cheap tier's ceiling. Routing those to Opus 4.8 or Fable 5 is correct regardless of cost.

Workloads where neither model is available due to data residency or hosting constraints. Self hosted alternatives (Qwen3-Coder, Llama 3.3 70B Coder variants) on Apple Silicon or GPU clusters cover the same operating envelope at higher operational cost and lower headline pass rate.

Working with Contra Collective

We build production coding agents for enterprise engineering teams. If you are sizing a coding agent for a production workload, evaluating the router pattern across cheap and frontier tiers, or integrating an agent into existing CI and review workflows, we can run the benchmark on your repository and produce a sized recommendation. Reach out via the Contra Collective contact page.

FAQ

Q: Why test on SWE-Bench Verified rather than SWE-Bench Pro? A: SWE-Bench Verified (500 human verified issues) is the right benchmark for cheap tier models because the harder Pro variant pushes both Haiku 4.5 and Gemini 3.5 Flash below the 20 percent floor where pass rate variance dominates the comparison. Verified gives a cleaner separation between the two.

Q: How does prompt caching actually work for coding agents at this tier? A: Both providers expose a cache hint on the request. For coding agents pinning the same repository context across many calls, set the cache hint on the repo context and the system prompt. The first call pays the full input cost; every subsequent call within the cache TTL (typically 5 minutes) pays roughly 10 percent of the input cost. For agent loops with 30 to 200 calls per issue, this collapses cost by 80 to 90 percent.

Q: Does the 4.6 point SWE-Bench pass rate gap matter in production? A: It depends on how patch retries are handled. If failures roll back cleanly and the agent retries with a frontier model, the gap matters less because the frontier router catches the misses. If failures escape to human review, the gap is a direct cost in engineer time and the 4.6 point lead is decisive.

Q: What about Haiku 4.5 versus GPT-5.5 Mini? A: GPT-5.5 Mini is in the same tier and sits at roughly 49 percent SWE-Bench Verified pass rate (between Haiku and Gemini Flash) at a price point closer to Haiku than to Gemini. For OpenAI native shops on the Responses API and the Codex toolchain, GPT-5.5 Mini is the natural pick. For agnostic deployments, Haiku 4.5 still leads on tool call accuracy and Gemini Flash on cost.

Q: How sensitive are the numbers to the agent harness? A: Substantially. A different scaffold (full SWE-agent vs mini-SWE-agent vs Aider style) can swing pass rate by 5 to 10 points. The relative ordering between Haiku 4.5 and Gemini 3.5 Flash is stable across scaffolds we tested; the absolute numbers shift but the gap holds.