52% Fewer Attended Tokens: The Model Declares Where It Will Look

The Paper

"Language Models Can Control Their Own Attention" was published in September 2026 by Namgyu Ho, Huzama Ahmad, Woosung Koh and Se-Young Yun of KAIST AI, with Tal Schuster and Cicero Nogueira dos Santos of Google DeepMind. The central claim is that a model already knows which parts of a long context it needs next, and that if you give it a syntax for saying so, the inference engine can read the attention mask straight off the generated text instead of approximating one from activations. Their protocol, Declarative Attention, partitions generation into three declared modes - <global>, <focus> and <local> - and running zero-shot on off-the-shelf Gemma-4-31B and Qwen-3.6-27B across 15 long-context tasks, it cuts attended tokens per response by 52.0% and 31.1% for accuracy drops of 1.27 and 2.75 percentage points, with no parameter updates anywhere in the pipeline.

Read the Paper on arXiv →

The Problem Before This Paper

A transformer reads its entire KV cache at every decode step, whether or not the tokens in it matter to the token being produced. The authors put a number on what that costs at the frontier: serving Qwen-3.5-397B-A17B at a 1M-token context means loading roughly 15 GB of KV cache per sequence per step, a memory bandwidth demand comparable to loading the model's 17B active parameters, and it repeats for every token of the reply. The empirical literature has said for years that this is mostly waste, since attention weight concentrates on a small subset of context tokens (Child et al., 2019; Zhang et al., 2023; Tang et al., 2024). The obstacle is that the true attention scores only exist after you compute the full attention matrix, so the mask has to be predicted rather than measured. Two families of prediction have been tried and both have a structural ceiling. Static heuristics such as H2O's historical-magnitude scoring and StreamingLLM's attention sinks plus recency window (Xiao et al., 2024) are free per step but cannot anticipate what a future query will need, and long-context accuracy degrades accordingly. Query-aware methods such as Quest and DeepSeek's lightweight indexer fix the accuracy problem by rescoring a compact surrogate of every context token at each step, which lowers the constant factor but leaves per-step complexity at O(N) - the scan itself still touches the whole cache. There was one prior attempt at eliciting the selection from the model rather than inferring it, Self-Selected Attention Span (Jin et al., 2024), but it fine-tuned separately per task on hand-designed context partitions inside 2K-token contexts. Nobody had shown a single task-agnostic protocol that a stock model follows at 100K-token scale without training.

What They Built

Declarative Attention is three pieces: a prompt that teaches the protocol, a delivery format that makes context addressable, and a state machine that turns the generated tags into a KV cache mask at decode time. The prompt separates a scaffold - system instruction, question, and the DA instruction itself - which stays attended in every mode, from the context, whose visibility the state machine controls. The system instruction exists partly to occupy the attention sink so that context never lands in it. The context is delivered as addressable segments the authors call magic chunks, targeting 2048 tokens each, with a segmenter that splits only units exceeding the cap and cuts at the coarsest boundary available: paragraph breaks, then single newlines, then sentence ends, then clause ends, then word boundaries, so a segment edge never falls inside a word. The presentation detail is the clever part. Each segment is rendered as a simulated tool-use transcript, an assistant turn appearing to call a get_magic_chunk tool declared through the model's native tool format, answered by a tool response headed Magic Chunk N. No tool is ever executed and every segment is in place before generation starts. The point is that segment boundaries then sit on the special tokens that delimit user, assistant and tool messages, boundaries the model tracked fluently throughout post-training, rather than on novel delimiters it has never seen.

A response then looks like a plan that names its own working set, alternating freely between the three modes with no restriction on how, when, or how often each is used:

<global>I need the founding year and the IPO year. The company
history in Magic Chunk 2 should state the founding.</global>
<focus magic_chunks="2">"Acme Corp was founded in 2003."</focus>
<global>The IPO year is still missing. Magic Chunk 7 covers
Acme's financial milestones.</global>
<focus magic_chunks="7">"Acme went public on the NYSE in 2011."</focus>
<local>2011 - 2003 = 8 years.</local>
<answer>8 years</answer>

The state machine starts in global and transitions on the closing > of an opening <focus magic_chunks="K"> or <local> tag, reverting on the matching close. The <global> tag itself produces no transition at all - global is already the default state between declared spans, and the tag survives only because writing it keeps the model's reasoning organised into contiguous spans with a declared scope. The diagram below is one decode trace under that protocol, using the token counts from the paper's own worked example.

One Declarative Attention response, animated. Token counts are the paper's worked example on a 25,466-token prompt: global attends everything, focus attends the single named magic chunk, local attends no context at all. The scaffold - system instruction, question, DA instruction - stays attended in all three.

The masking itself is deliberately unglamorous, which is why it works. vLLM stores the KV cache in fixed 16 to 32 token blocks and its kernels read whole blocks, so dropping scattered individual tokens would save nothing at all. The state machine therefore applies the mask at block granularity, rounding the kept spans outward to block boundaries so that no token the model declared is ever dropped, at a cost of at most one extra block per span edge against 2048-token segments. What comes out is an ordinary block list, so FlashAttention runs unmodified, following the block-sparse principle of Native Sparse Attention. The vLLM integration is a hook on the attention metadata builder that rewrites the request's KV cache block table each decode step, with no kernel changes and no scheduler changes. One scope limit matters for reading the results: DA applies only to global attention layers. Sliding window attention and Gated DeltaNet layers already have per-step costs bounded by a window or a recurrent state, so there is nothing for a mask to save there and DA leaves them alone.

Key Findings

• The mask is the saving, not the prompt format. The DA-no-mask ablation runs the identical chunked tool-use prompt under full causal attention, and it is nearly free on accuracy (87.01% vs 87.01% on Gemma, within 0.69pp on Qwen) while attending 66.2% more tokens than vanilla on Gemma, because the protocol generates 15-35% more decode steps. Turning the mask on cuts attended tokens 71.1% relative to that ablation, moving Gemma from 66.2% above vanilla to 52.0% below. The chunked format costs almost nothing and buys almost nothing; every token of the saving comes from the mask.
• Accuracy converges toward vanilla as the backbone scales, and adherence is why. Relative accuracy rises monotonically in both families: 29%, 91%, 99% for Gemma-4-E4B, 12B and 31B, and 64%, 89%, 97% for Qwen-3.5-4B, 3.5-9B and 3.6-27B. The collapse at the small end is mostly protocol failure rather than reasoning failure - focus parse success is 58% on Gemma-4-E4B against 99% on the two largest models. Focus attempts per response stay flat at 1.4 to 1.9 across all six, so stronger models succeed by parsing their calls correctly, not by issuing fewer.
• Cheap modes carry the tokens, expensive mode carries the cost. On Gemma-4-31B, <focus> and <local> account for about 73% of generated tokens and save 76-99% of the per-token attention read, rising with context length. <global> takes the remaining 27% of tokens and saves nothing by construction, yet accounts for over 80% of everything DA still attends, and its share climbs to about 45% of tokens in the longest bucket.
• Absolute savings scale with context, which is where they are needed. Per-step masking removes a roughly constant fraction, near 50% on five of six models, so the absolute saving tracks raw context length: about 1M fewer attended tokens per response at short context and about 21M fewer in the longest bin. The two heaviest tasks dominate - code_repo saves 41.8M tokens on Gemma and 52.0M on Qwen, dialogue_history 22.1M and 39.1M.
• The accuracy cost is concentrated in multi-span reasoning. Losses are roughly triple in multi-span tasks versus single-span retrieval on Gemma (2.28pp vs 0.78pp) and larger on Qwen (3.59pp vs 2.34pp), which is what you would expect from a protocol that names one segment at a time. Relative accuracy holds within about a point of vanilla up to 32K tokens and falls to roughly 96% in the longest bin, a decline absent from the maskless arm.

Results

Across the 15 sources - drawn from RULER, LongBench v1 and v2, LooGLE and ZeroSCROLLS, with contexts from 6.4K to a 1.07M-token mean on code_repo - DA takes Gemma-4-31B from 87.01% to 85.74% accuracy while cutting mean attended tokens from 13.43M to 6.45M per response, and Qwen-3.6-27B from 85.31% to 82.56% while cutting 22.54M to 15.52M. Aggregates hide real per-task spread. DA matches or beats vanilla on 7 of 15 tasks on Gemma and 5 of 15 on Qwen, with the clearest gains on longdep_qa (+3.1pp, Gemma) and code_repo (+5.6pp, Qwen), while the worst single result is shortdep_cloze on Qwen falling 73.4% to 68.8%. On five Qwen sources, most visibly qmsum and the two LongBench v2 QA sets, DA's longer generations push attended tokens above vanilla outright. Evaluation used an LLM judge with a per-question acceptance rubric generated by Gemini-3-Flash and applied by a thinking-enabled Qwen-3.5-4B that correlates with Gemini-3.1-Pro at Pearson r = 0.99, on up to 128 examples per source, with thinking disabled on the models under test because they failed to follow the protocol inside thinking traces. The efficiency claim is where care is needed: the authors do not report measured wall-clock time. They report a roofline projection on a single B200 in bf16 at an assumed 40% model FLOPs utilisation and 70% model bandwidth utilisation, excluding prefill on the assumption of phase-disaggregated serving.

Table 3 of the paper, redrawn. The saving is entirely in the blue segment. On Gemma the sliding window attention floor is 42% of DA's remaining attention time and caps the gain even though the global read itself more than halves; on Qwen the Gated DeltaNet state is small enough that the reduction passes through almost undiluted.

Why This Matters for AI and Automation

• Agentic contexts are the regime this was built for, even though it was not tested there. The authors make the argument directly: retrieval decides what enters the context, DA decides what is attended among what already entered, and retrieval makes DA's problem worse rather than better. Every tool result stays in context for the rest of the episode while being relevant only to the steps that requested it. On a long agent run, the transcript is already segmented by tool and turn boundaries, which is exactly the addressable structure the paper had to manufacture with magic chunks.
• It points at reversible context compaction. Compaction as practised today is destructive: a summary replaces the original tokens, their KV cache is discarded, and getting the content back means re-prefilling it from text. Under DA the sequence is never edited, the attended set changes only at span boundaries, and each change is announced in plain text before the tokens that read it. That is precisely the access pattern KV offloading needs, so out-of-focus segments could sit in host memory and be prefetched the moment a declaration names them. Sparse attention methods reselect every step and leave nothing to prefetch; serving-level offloading moves KV only at request boundaries.
• Sparse attention becomes an instruction, not a kernel. Because the selection policy lives in language, you change it by changing the prompt, and it improves with model scale without touching the protocol. The mechanism ships as a block-table rewrite with no kernel or scheduler changes, which is a far smaller integration surface than any learned-mask method.
• Nothing here is deployable this quarter, and the paper does not pretend otherwise. It needs the custom vLLM integration, it runs in non-thinking mode only, and the headline efficiency figure is a roofline projection rather than a measurement. The result to take away is directional: attention selection is now something a stock model can be asked to do in text, and it gets better as models get better.

My Take

The framing is the contribution, and it is a good one, but the number in the title is doing more work than it should. A 52% reduction in attended tokens is not a 52% reduction in anything a user or an invoice observes. Table 3 converts it to 0.71x decode time on Gemma and 0.77x on Qwen, and those are roofline projections at assumed utilisations with prefill excluded, not measurements from a serving stack. The gap between the two figures is where the honest reading lives: the protocol runs 31 to 35% more decode steps, and every one of those extra steps pays full matmul cost and full efficient-layer cost. On Gemma the sliding-window floor eats 42% of DA's remaining attention time, which is why a global read that more than halves turns into a 29% wall-clock gain. If the field keeps replacing global layers with local and linear ones, the ceiling on this technique falls with them, and the authors' own appendix analysis - attention at 56 to 97% of decode time on the newest sparse designs, 94% on Kimi-K3 which kept vanilla global attention - is the load-bearing argument that it will not fall to zero. The limitation I would fix first is the non-thinking constraint, because every deployment that actually has a 200K-token context is running a thinking model, and "we disabled thinking because the models could not follow the protocol inside thinking traces" is a larger asterisk than its one paragraph suggests. The most interesting idea in the paper is one the authors do not implement: an in-context index, a short description of each segment that global mode surveys instead of the raw context. Global is 27% of generated tokens and over 80% of what DA still attends, so the index is where the next factor of two lives, and it is a prompt-construction change rather than a systems change. What I keep coming back to is that the accuracy gap closes monotonically with scale on both families while the token saving stays roughly flat. That combination is unusual and it is the real finding. It means this is not a technique that trades quality for cost at a fixed exchange rate, it is one whose exchange rate improves for free every time somebody ships a better backbone.

Discussion question: DA works because the model can name a region it has not read at this step and be right about what is in it, which is a claim about the model's memory of its own context rather than about attention at all. In an agent loop where tool results accumulate over hundreds of turns and the useful segment may be forty turns back, would you trust a model's declaration of where to look more or less than a scan-based scorer that rereads the whole cache every step, and what would you need to measure to find out?