DeepSeek-V4.1 Flash:
Pushing the Limits of KV Cache Compression

TL;DR

When DeepSeek-V4.1 Flash was released, I thought it might just be a post-training iteration version... but after using it for a while, I found it reached nearly 420 Tokens/s in speed, and then Cui said all DeepSeek-V4 Pro models would be taken offline... suddenly I felt this was no small matter... until the Technical Report 《DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression》[1] was fully released, only then did I realize it should be called DeepSeek-V5 Flash...

As the paper title states, the purpose of DeepSeek-V4.1 Flash is to push KVCache compression to the extreme. The main reason is that Long-horizon Agent Workflows cause the Context to grow longer and longer, while various tool calls bring heavy prefill computation pressure. The storage pressure of KVCache in HBM and external SSD is very high, all of which are reasons that make Scaling impossible. Therefore, a series of optimizations were made on the model architecture, especially in the compression of KVCache and the computation optimization of Prefill.

• Prefill computation optimization: Drawing on YOCO, the entire model has 40 layers, and only 20 layers are needed during Prefill. Therefore, the Prefill activated parameters are only 8B, and the Decode activated parameters are 16B
• KVCache compression: Engineering-wise, KVCache compression is divided into several dimensions: head count compression similar to GQA, then block-based compression like CSA, and the cross-layer compression of CSA2 in this paper. At the same time, the indexer computation of Sparse Attention is also optimized. Finally, there are some numerical precision optimizations, for example DS41F adopts FP4 KVCache.

Finally, under the premise of maintaining high-quality task completion by the model, KVCache is further compressed by 4x:

In addition, the original writing of the paper is somewhat complex, especially the description of CED. In fact, if we redraw a diagram centered on KVCache and combined with the perspective of computer architecture, it seems to become clear all at once. It can be seen as a kind of Recursive Transformer architecture, a way of modifying Q and reusing KV during the recursive process.

Regarding the Recursive Transformer architecture, you can refer to 《On the Future Transformer: Loops Are Not What You Need》. Next, we will conduct a detailed interpretation and analysis according to the chapter structure of the technical report. This article is the first in this series, analyzing the model architecture in detail, and the more critical content is in Chapter 3.

1. Overview

1.1 Why KVCache compression is needed

First, the report states that in recent years Long-horizon Agents have made ultra-long-context processing an increasingly important model workload. Supporting this type of workload not only requires efficient processing of long sequences, but also requires persistent storage, reuse, and transfer of large KVCache. Therefore, KVCache management has become a fundamental capability of model deployment, while also bringing significant challenges in computation, storage, and communication.

Then it goes on to introduce the DeepSeek-V4 architecture, which processes by combining a Sparse Attention that fully covers the context with a Sliding Window Attention (SWA) that covers the local window. Although advances related to Sparse Attention have significantly reduced the computational cost of long sequence processing, persistent storage and data movement have gradually become more prominent bottlenecks. In long contexts, the usage of the Global KV Cache will dominate, and being persisted for prefix reuse, it will heavily occupy Host memory capacity and SSD capacity, and will also place high demands on the interconnect bandwidth for KVCache movement. These limit service throughput, increase deployment cost, and ultimately hinder the deployment and promotion of agents toward longer task spans and broader application scenarios.

Therefore, further reducing the key-value cache footprint is crucial for alleviating storage and communication bottlenecks and reducing long-context serving costs. DeepSeek-V4.1-Flash is a multimodal mixture-of-experts model designed for more aggressive KVCache compression. DeepSeek-V4.1-Flash has a parameter scale of 552B, natively supports multimodal input, and supports contexts of up to 1 million tokens. It adopts a Causal Encoder-Decoder (CED) architecture, in which the Decoder's Global KVCache is obtained by projecting the Encoder's final hidden states. This design makes the model activate 8B parameters per token during the Prefill stage and 16B parameters during the Decoding stage, which is especially cost-effective for input-dominated Agent scenarios. Although DeepSeek-V4.1-Flash is significantly larger than DeepSeek-V4-Flash, at the same sequence length, its required runtime KVCache storage is only about 1/4 of the latter, and its persistent KVCache storage is only about 1/8 of the latter. In addition, the overall performance of DeepSeek-V4.1-Flash is superior to DeepSeek-V4-Flash.

These compressions for KVCache mainly come from the joint optimization of model architecture, cache precision, and deployment strategy. For DSv4, it is a model with SWA as the backbone and enhanced by global compressed attention (CSA/HCA). Based on this perspective, the DeepSeek team carried out a series of optimizations. First, it is worth noting that they abandoned the block-based high-compression-ratio structure like HCA, and instead carried out more optimizations on CSA, forming CSA2. The main optimizations compress the KV Cache from three dimensions:

• In the channel dimension, a 512-dimensional latent vector is used to share the representation of the keys and values required by each attention head;
• In the sequence dimension, the Encoder merges 2 adjacent positions into 1 cache entry through channel-wise learned weights, while the Decoder retains per-position entries;
• In the layer dimension, multiple layers share the same global KV, and the whole network retains only 3 copies of Encoder cache and 1 copy of Decoder cache.

Combined with FP4 quantization, the storage growth of the global main KV and the Indexer is about 890 bytes per token.

1.2 Overview of model architecture

The overall model architecture is as follows:

The paper reports that the backbone parameters are about , the Engram parameters are about , and the activated parameters per token for prefill and decode are about and respectively. The core is to use CED to reduce long-context Prefill computation, use CSA2 to reduce attention and KV cache overhead, and then combine Engram conditional memory with DSpark speculative decoding.

The model has layers in total, hidden dimension , and vocabulary size . Each layer contains attention and MoE, organized through mHC residual connections. The entire attention mechanism is divided into two modules, Encoder and Decoder, forming a Causal Encoder-Decoder (CED) architecture. The key of CED is that the Decoder's global KV comes from the Encoder's end representation, and subsequent Decoder layers share these KVs. Therefore, most positions of a long prompt only need to pass through the first 20 layers,

The key CSA2 among them adopts a mechanism of local sliding window + global sparse retrieval + cross-layer KVCache reuse. The sliding window size is 128, attention uses Q heads, sharing a -dimensional KV latent, of which RoPE is -dimensional and NoPE is -dimensional. Q uses a low-rank projection of rank , and the output projection is divided into groups, each of rank . The relevant parameters of the entire model are as follows:

dim

n_layers

n_mtp_layers

vocab_size

n_heads

head_dim

rope_head_dim

q_lora_rank

o_lora_rank

o_groups

window_size

compress_ratios

[0,0, 2×18, 1×20, 0,0,0]

kv_source_layers

[2,8,14,20]

index_source_layers

[2,8,14,20,24,28,32,36]

index_n_heads

index_head_dim

index_topk

candidate_source_layer

candidate_topk_blocks

candidate_block_size

original_seq_len

rope_factor

rope_theta

compress_rope_theta

n_routed_experts

n_activated_experts

moe_inter_dim

score_func

sqrtsoftplus

route_scale

swiglu_limit

hc_mult

hc_sinkhorn_iters

engram_layer_ids

[1, 14]

engram_num_embeddings

[384006168, 384016682]

engram_max_ngram_size

engram_n_heads

engram_head_dim

engram_vocab_size

dspark_block_size

dspark_target_layer_ids

[37,38,39]

dspark_n_routed_experts

vision_n_layers

vision_dim

vision_patch_size

vision_downsample_ratio

vision_max_n_token

Among them:

• mHC: Each token maintains residual streams of dimensions. The residual mixing matrix is constrained to an approximately doubly stochastic matrix through Sinkhorn iterations. The key of Single-Pass is to use the input mixing coefficients produced by the previous sub-layer, releasing the dependency of the current coefficient computation, facilitating kernel fusion and reducing memory read/write.
• DSpark: An additional SWA-128 draft blocks, each layer adopts a small-scale MoE with Top-3 out of routed experts. It reads the mean of the four residual streams at the entry of backbone network layers , computes draft positions in parallel at once, cooperates with a Markov head to model dependencies, and a confidence head assists in deciding the verification length.
• Vision branch: A ViT with layers and hidden dimension , patch size . Features go through pixel-unshuffle, reducing the number of tokens to of the original, then mapped to dimensions by an MLP and inserted into the text sequence, with a maximum of visual tokens per single image.

Regarding CED and CSA2, we will introduce them in detail in Chapter 2. Finally, as the Context grows, the required computation of DeepSeek V4.1 Flash grows almost linearly within the 1M range, and the computation overhead is far less than that of previous generations of models

2. Model Architecture

2.1 Multimodal architecture

The visual path of DeepSeek-V4.1-Flash can be summarized as: complete visual encoding on a finer image patch grid, rearrange adjacent features into fewer wide vectors, and then project them into the input space of the language backbone. Among them:

• vision_patch_size=14

  : is the patch size of the original image

• vision_downsample_ratio=3 

  : is the merging range of the ViT output feature grid

The entire processing flow is shown in the figure below. The ViT first completes intra-image interaction on high-resolution features, and then feeds them into the LLM through spatial rearrangement and compression projection.

The above figure takes a 1008 x 1008 pixel square image after preprocessing and padding as an example:

Image preprocessing: It should be noted that it does not perform text recognition like OCR. After the file is loaded, it is directly decoded via

load_image