Disclaimer: This post is machine-translated from the original Chinese article: https://ai-coding.wiselychen.com/qwen-3-8-27b-open-weights-local-security/
The original work is written in Chinese; the English version is translated by AI.

Translation Note

This article is translated from the Chinese original. The author writes primarily in Traditional Chinese, and the original version is the canonical source — including the latest updates, comments, and follow-up discussions.

Read the Chinese original: Qwen3.8-27B 開源：SWE-bench Pro 61.7 贏過 Opus 4.6 Max，「那就用地端」第一次不是妥協

If you spot any translation errors or have feedback, please refer to the Chinese version as the source of truth.

SWE-bench Pro: Qwen3.8-27B 61.7 vs Claude Opus 4.6 Max 53.4

Table of Contents

TL;DR

This is the most anticipated small model I’ve ever seen — the community literally held a countdown party.

When Junyang Lin left Alibaba in March, everyone feared Qwen would go closed-source. Qwen 3.7 only shipped as API-only Max with no open 27B weights, and the fear was not unreasonable. Then Kimi, DeepSeek, and GLM all pushed ahead with open large models — but none that fit on consumer GPUs. The open-source community spent six months building an entire ecosystem on Qwen 3.6-27B.

After getting outpaced by competitors on the large-model front, Alibaba remembered: their strongest card was never Max. It was the 27B that people can actually run.

On 8/14, Qwen3.8-27B weights dropped. Apache 2.0, 27B dense, native multimodal, 262K context. The headline: SWE-bench Pro 61.7. Claude Opus 4.6 Max sits at 53.4.

A model that runs on your own GPU, matching my favorite cloud model on paper. Today is security day on this blog. This release belongs on security day because enterprise teams choosing on-prem models always faced the same awkward tradeoff: on-prem is secure, but it’s also a downgrade. Qwen3.8-27B is the first time — in official numbers — that a 27B open model beats a frontier closed model on agentic tasks. “Just use on-prem” is no longer a compromise.

The 6-Month Wait

Spec Qwen3.8-27B
Architecture 27B dense, 64 layers hybrid (48 Gated DeltaNet + 16 full attention)
Modality Native multimodal: text, image, video input
Context 262K native, YaRN-extendable to 1M
License Apache 2.0
Weight size BF16 51.76 GiB / FP8 28.76 GiB / third-party Q4 ~17 GiB
Also released Qwen3.8-2.4T-A95B (Max tier) weights also open-sourced

Win/Loss Distribution Matters More Than Any Single Score

Spread the official benchmark table against Opus 4.6 Max and a clear structure emerges.

27B wins:

Benchmark Qwen3.8-27B Opus 4.6 Max
SWE-bench Pro (agentic coding) 61.7 53.4
LiveCodeBench v6 (competitive programming) 90.3 88.8
CoWorkBench (long-horizon office tasks) 70.7 68.2
IFBench (instruction following) 79.5 62.5
OSWorld-Verified (computer use) 84.3 72.7
AndroidWorld (mobile use) 81.9 62.0

27B loses:

Benchmark Qwen3.8-27B Opus 4.6 Max
Terminal Bench 2.1 73.0 78.2
GPQA Diamond (science reasoning) 89.2 91.3
HLE (cross-domain reasoning) 30.8 40.0
NL2Repo-Bench (full repo generation) 42.3 47.6

The distribution is not random. Wins are all agentic workflow tasks: fixing code, running processes, operating computers and phones, following instructions. Losses are all knowledge-ceiling tasks: science reasoning, cross-domain challenges, generating entire repos from scratch.

Win/loss distribution: all agentic wins, all knowledge-ceiling losses

This distribution maps directly to enterprise deployment needs. An enterprise agent’s daily work is “read a ticket, modify three files, run tests, fill out forms, operate internal systems” — not solving GPQA-level science problems. Losing a few points on ceiling tasks has limited impact on this workload; winning on agentic tasks is the difference you use every day.

The generational jump also shows where the effort went: SWE-bench Pro went from 53.5 (Qwen3.6-27B) to 61.7, DeepSWE from 13.3 to 42.2, OSWorld from 63.9 to 84.3. This is not uniform improvement across the board — it’s concentrated firepower aimed at agent scenarios.

Architecture Unchanged — It’s All Post-Training

Why does the win/loss distribution look like this? Dig through Qwen3.8-27B’s config.json, weight index, and Transformers implementation, and the answer is clear: the underlying architecture is virtually unchanged from Qwen3.6-27B.

model_type is still qwen3_5. 64-layer decoder, 5120 hidden size, 17408 FFN, 48 Gated DeltaNet layers + 16 full attention layers, 24 query heads, 4 KV heads, 262K context, same vision tower, same MTP config, same vocabulary — every key parameter is identical.

This means the agentic benchmark jumps don’t come from architectural innovation. The more likely explanation: Alibaba has stabilized the Qwen3.5 hybrid architecture and focused entirely on weight optimization, agent trajectory training, coding environment interaction data, reinforcement learning, and tool-calling behavior.

Qwen3.8 is not “3.8 in architecture.” It’s “3.8 in agent behavior.”

Zero architecture changes — the leap comes from post-training

This also explains why GPQA Diamond only improved by 1.4 points and HLE barely moved, while OSWorld jumped 20.4 points and DeepSWE jumped 28.9 points — the former depends on model capacity and pre-training knowledge, the latter on post-training and environment interaction data. Same architecture, concentrated post-training = the agent metrics are what jump.

For production systems, this is actually good news. Architecture stability means inference engines like vLLM and SGLang don’t need operator re-adaptation, quantization schemes carry over, existing kernel optimizations can be reused, and enterprise migration costs are lower.

Deployment Reality: Hybrid Architecture Saves KV Cache by Design

The hybrid architecture has a very practical benefit for on-prem deployment: only 16 of 64 layers need to store KV cache that grows linearly with context.

Hybrid architecture memory advantage: 48 Gated DeltaNet layers use fixed memory, 16 Full Attention layers grow with context

The 48 Gated DeltaNet layers maintain a fixed-size recurrent state matrix — memory usage stays constant regardless of context length. A traditional pure Transformer stores KV cache at all 64 layers; Qwen3.8 only needs it at 16. This makes it inherently better suited for long conversations, coding agents, and multi-turn tool-calling scenarios where context keeps growing.

But don’t mythologize it — those 16 full-attention layers still have KV cache. At BF16, each token costs about 64 KiB; 262K fully loaded is ~16 GiB, 1M context is ~61 GiB. Quadratic complexity went from 64 layers to 16 — it didn’t disappear.

Regular readers of this security-day series know that last month I ran a week-long Tier 1 on-prem experiment on an RTX Pro 6000: a GLM 5.2-class 744B model couldn’t fit in 96GB VRAM and had to offload to system RAM via MoE, capping out at about ten tokens per second — painful for interactive agent use.

27B dense + hybrid cache is a completely different world:

  • FP8 official: 28.76 GiB — a single 48GB card can comfortably hold the weights plus working KV cache
  • Third-party Q4 quant: ~17 GiB — a 24GB consumer card can run medium-context workloads
  • The caveat is long-context math: at full 262K, just the 16 full-attention layers’ KV cache consumes ~16 GiB. A 24GB card runs “Q4 + restrained context,” not the full-spec experience

Two days ago I wrote in the memory price article that you should “pick the model first, then decide what hardware to buy; the 27B tier lets you skip the entire hardware procurement cycle designed for large models.” At the time, Qwen3.8-27B weights hadn’t dropped yet and I could only write about it as futures. Now the weights are here, and this path has its most critical piece: the 27B tier has its first option that beats a frontier model in official numbers.

One old debt to remember about quantization, though. The Bonsai 27B lesson: compression losses are uneven. The previous Qwen3.6-27B compressed to 1-bit kept MATH 500 at 98 (from 99.4) but TauBench (tool calling) dropped from 82.9 to 61.3. The official benchmarks are BF16/FP8 numbers — how much your Q4 version on a 24GB card loses, and whether the loss happens to hit agentic capability, nobody has measured. After getting a quantized version, test tool calling first, then decide how much to trust.

My RTX 5090 Numbers

Hardware and Model Configuration

Item Setting
GPU NVIDIA RTX 5090 32 GB
Model Qwen3.8-27B UD-Q4_K_XL
Model size 17.92 GB
Context limit 262,144 tokens
Longest actual input ~237,100 tokens
KV Cache Q4_0 (K/V)
Flash Attention On
MTP On, draft max 2, p-min 0.75
Parallel 1
CPU threads 8
Thinking On
Thinking budget 1,536 tokens
Answer limit 4,096 tokens
Temperature 0
Prompt cache On
Vision projection Not loaded; text-only test

Actual VRAM Usage

  • Model serving: ~26 GB
  • With background processes: ~29 GB
  • Headroom at long context: ~3 GB
  • Q8_0 KV: OOMs at ~67K tokens during long-text prefill
  • Q4_0 KV: Completes ~237K token full-text input

On a 32GB card, running full 262K context requires Q4 KV Cache — Q8 won’t make it.

Generation Speed

Scenario Speed
7K–28K context (daily conversation, short text) 100–126 tok/s
Thinking average across all prompts 106.7 tok/s
Thinking off average 112.8 tok/s
237K context (near maximum) 49–55 tok/s
MTP off baseline ~72 tok/s
  • 237K first-pass prompt prefill: ~238 seconds
  • Prompt cache hit: ~12–83 seconds per question
  • Average total time per question: ~20 seconds

Thinking has minimal impact on generation speed (106.7 vs 112.8) — the main cost is the extra thinking tokens. Context above 200K is where attention computation becomes the bottleneck, dropping from 100+ tok/s to 49–55 tok/s.

Last month, the same machine running GLM 5.2 with offloading was grinding at ten-ish tok/s. 27B dense at 100+ tok/s for daily context — real-time conversation, long-text summarization, high-frequency reasoning all work fine. Interactive agents are completely viable.

Launch Parameters

Server-side (llama.cpp):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

llama-server \
  -m Qwen3.8-27B-UD-Q4_K_XL.gguf \
  -ngl 99 \
  --flash-attn on \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --ctx-size 262144 \
  --spec-type draft-mtp \
  --spec-draft-p-min 0.75 \
  --spec-draft-n-max 2 \
  --parallel 1 \
  --reasoning on \
  --reasoning-format deepseek \
  --reasoning-budget 1536 \
  --host 0.0.0.0 \
  --port 8001 \
  -t 8

API parameters:

1
2
3
4
5
6
7
8
9

{
  "temperature": 0,
  "max_tokens": 4096,
  "cache_prompt": true,
  "reasoning_effort": "medium",
  "chat_template_kwargs": {
    "enable_thinking": true
  }
}

What actually limits thinking length is the server-side --reasoning-budget 1536; setting reasoning_effort alone won’t prevent Thinking from consuming the entire output budget.

RTX 5090 vs DGX Spark

Community numbers for the same model on DGX Spark are in — the gap is stark:

Hardware Quant / Engine Speed Source
RTX 5090 32GB Q4_K_XL + MTP (daily context) 100–126 tok/s My test
RTX 5090 32GB Q4_K_XL + MTP (237K context) 49–55 tok/s My test
RTX 5090 SGLang NVFP4 206.1 tok/s SGLang
DGX Spark SGLang NVFP4 38.28 tok/s SGLang
DGX Spark Q4 basic config ~40 tok/s sudoingX
DGX Spark 256K context ~19 tok/s ivanfioravanti

Same model, same inference engine: 5090 at 206 vs Spark at 38 — a 5.4x gap. The reason is not compute; it’s memory bandwidth. The 5090’s GDDR7 delivers ~1,792 GB/s; the Spark’s LPDDR5X tops out at ~273 GB/s. Dense models read ALL weights for every generated token — low bandwidth means slow, period. MoE models only read active expert weights per token, so unified memory’s lower bandwidth is tolerable. But 27B dense fits entirely in discrete VRAM, so the Spark’s large-capacity advantage goes unused — only the bandwidth disadvantage remains. Pick discrete GPU for 27B dense, pick unified memory for large MoE — match hardware to model architecture, not parameter count.

The Honest Version of “On-Prem Is More Secure”

“On-prem is more secure” — engineers can now say this with a straight face without feeling guilty about the capability gap.

But today is security day, so this claim itself needs a security audit.

What on-prem deployment actually eliminates is data exfiltration risk: prompts, source code, and customer data never leave your premises. No vendor data-retention clauses to review, no cross-border transfer compliance to handle. For Taiwan’s financial, medical, and government-contract scenarios, this class of risk is often a “single veto” dealbreaker, and on-prem removes it entirely. That’s real.

But on-prem does not eliminate agent behavior risk. An agent that can modify code, run commands, and operate internal systems — running on your own hardware — is still vulnerable to prompt injection, still capable of dropping databases if given excessive permissions. This class of risk has nothing to do with where the model lives and everything to do with how the harness is designed.

Honest security boundary: on-prem eliminates data exfiltration, but agent behavior risk remains

So the honest version is: on-prem solves the “who sees your data” problem, not the “what can the agent do” problem. The former is a procurement and compliance question; the latter is an engineering question. “On-prem is more secure” is true, but only half-true — the engineering work on permission scoping and tool-chain isolation is exactly the same.

The Counterargument: These Are Qwen’s Own Numbers

Let’s put the strongest rebuttal on the table first: the entire benchmark table was published by Qwen.

And the testing methodology is not apples-to-apples. The official notes state that on SWE-bench Pro, only Opus 4.6 Max uses its officially reported score — all other models were tested by Qwen using a Claude Code harness, temp=1.0, 256K context. That’s not a controlled experiment under identical conditions. The two items with the largest margin (QwenSWEBench 79.0 vs 63.8, CoWorkBench 70.7 vs 68.2) happen to be Qwen’s own benchmarks. One day after release, no independent third-party reproduction exists. The four losses, on the other hand, are not in question — the official table admits them.

What’s left after all these discounts?

What’s left is actually enough. This argument doesn’t need “27B comprehensively beats Opus 4.6 Max” to hold — that claim was never true to begin with; the four ceiling-task losses are crystal clear. It only needs “27B is close enough to frontier on enterprise agent workloads that the capability gap is no longer a reason to veto on-prem.” SWE-bench Pro is a public benchmark, and the generational jump from 53.5 to 61.7 was measured under the same harness and conditions. OSWorld from 63.9 to 84.3 — even at a 30% haircut — is still a tier change. The credibility of the direction is higher than the precision of any single data point.

Reality Check

  • All numbers come from the official release. One day after launch, no third-party reproduction exists. This article is “a breakdown of official claims,” not a test report.
  • I have my own speed numbers (RTX 5090 + Q4_K_XL + MTP: daily context 100–126 tok/s, 237K long-text 49–55 tok/s; Q8 KV OOMs at 67K, full 262K on a 32GB card requires Q4 KV). But systematic agent-quality evaluation hasn’t been done — especially Q4 quantized tool-calling degradation. Benchmark-based judgments still carry the caveat “if the official numbers are trustworthy.”
  • The “wins are all agentic” distribution can also be read backwards as “Qwen cherry-picked items that favor themselves for the table.” OSWorld and AndroidWorld comparison numbers are from official announcements; I couldn’t find public details on the testing conditions for the Opus side.
  • The most consistent community complaint from early testers is “thinks too much.” The official default is reasoning_effort=xhigh. Someone ran the same Tetris task on both: 3.6 thought for ~3,000 tokens and started coding; 3.8 was still thinking at 15,000 tokens. The output was more polished (pause menu, high-score board, retro sound effects), but wall-clock time was several times longer. The open chat template shows that reasoning_effort is essentially prompt steering (system prompt with varying intensity of “think carefully”), not dynamic network depth. Turning it to medium or disabling thinking largely solves this, but the official benchmarks were almost certainly run at xhigh. Whether lowering effort for speed trades away capability is still unknown.
  • 27B is competing against Opus 4.6 Max, this generation. Frontier models are also moving. How long this “close enough” window stays open is anyone’s guess.

Key Takeaways

  1. “Just use on-prem” is no longer a compromise. Choosing on-prem used to mean accepting a downgrade. Now a 27B model beats Opus 4.6 Max on agentic tasks in official numbers — the capability-gap argument against on-prem just lost its teeth.

  2. Read benchmark tables by win/loss distribution, not single scores. 27B wins on agentic workflows, loses on knowledge ceiling. Which category your workload falls into determines whether this table is signal or noise for you. Most enterprise agent workloads fall in the former.

  3. After quantizing, test tool calling first. Official numbers are BF16/FP8. The Q4 version on a 24GB card is effectively a different, untested model. The Bonsai lesson is still warm: compression hits agent capability first.

  4. “On-prem is more secure” is only half the story. Data exfiltration risk drops to zero; agent behavior risk stays exactly the same. After winning the procurement meeting, harness permission design is where the security work actually begins.

Further Reading

Primary Sources