Grouped-Query Attention (GQA) Explained: MHA vs MQA vs GQA

Grouped-query attention (GQA) is a variant of transformer attention in which several query heads share a single key head and a single value head. Instead of giving every query head its own keys and values, as standard multi-head attention does, GQA splits the query heads into groups and lets each group read from one shared key/value pair. The point is memory: it shrinks the key-value cache that dominates the cost of serving long-context language models, while keeping almost all of the model quality that full multi-head attention provides.

A quick refresher on multi-head attention

In self-attention, each token is projected into three vectors: a query, a key, and a value. A token’s query is compared against every other token’s key to produce attention weights, and those weights mix the values into an output.

Multi-head attention (MHA) runs this process several times in parallel. A model with 32 heads has 32 independent sets of query, key, and value projections, each working in a smaller subspace. Different heads learn to track different relationships — syntax, coreference, position, and so on — and their outputs are concatenated and projected back to the model dimension.

During training this is fine. The problem shows up at inference time.

Why the KV cache is the bottleneck

When a model generates text one token at a time, it doesn’t recompute keys and values for the whole prompt at every step. It stores them in a KV cache and only computes the new token’s entries. That cache grows with four things multiplied together:

• the number of layers
• the number of key/value heads
• the dimension of each head
• the sequence length (and the batch size, if serving many requests)

For long contexts the KV cache can grow larger than the model weights themselves. Worse, every decoding step has to read the entire cache from GPU memory. Token generation is usually limited by memory bandwidth rather than raw compute, so a bigger cache means slower tokens and fewer concurrent users per GPU. This is a big part of why accelerator vendors fight over memory bandwidth, from HBM to GDDR.

Of the four factors above, the number of key/value heads is the one an architecture designer can cut without shortening the context window or making the model shallower. That’s exactly the lever MQA and GQA pull.

Multi-query attention: the extreme option

Multi-query attention (MQA) keeps all the query heads but uses just one key head and one value head, shared by every query head. If a model has 32 query heads, MQA cuts the KV cache for attention by a factor of 32 compared with MHA.

The savings are dramatic, but there’s a cost. With only one set of keys and values, the heads lose much of their ability to specialize. Models trained with MQA can show measurable quality drops, and training can be less stable. MQA proved the idea worked; it just went too far for many use cases.

Grouped-query attention: the middle ground

GQA sits between the two extremes. You choose a number of key/value groups G:

• G = number of query heads — every query head has its own key/value head. That’s plain MHA.
• G = 1 — all query heads share a single key/value head. That’s MQA.
• 1 < G < number of query heads — query heads are split into G groups, and each group shares one key/value head. That’s GQA.

A typical configuration might pair 32 query heads with 8 key/value heads, so each group of four query heads reads from the same keys and values. The KV cache shrinks by 4x relative to MHA, and in practice quality stays very close to the full multi-head baseline.

Mechanically, nothing exotic happens. During the attention computation, each key/value head is simply broadcast (repeated) across the query heads in its group. Optimized attention kernels avoid physically copying the tensors, but conceptually that’s all it is.

MHA vs MQA vs GQA compared

	Multi-head (MHA)	Multi-query (MQA)	Grouped-query (GQA)
Key/value heads	One per query head	One total	One per group
KV cache size	Largest (baseline)	Smallest	In between, tunable
Decoding memory bandwidth	Highest	Lowest	Moderate
Model quality	Baseline	Noticeable drop possible	Close to baseline
Head specialization	Full	Limited	Mostly preserved
Typical use today	Smaller or older models	Latency-critical niches	Default for many modern LLMs

Converting an existing model to GQA

One practical attraction of GQA is that you don’t have to train from scratch to get it. A model trained with standard multi-head attention can be “uptrained” into a GQA model: the key and value heads within each group are combined (commonly by averaging their projection weights) into one head, and the model then continues training for a small fraction of its original compute budget so it adapts to the shared heads.

This made GQA an easy upgrade path for labs that already had strong MHA checkpoints and wanted cheaper inference without a full retraining run.

How GQA fits with other inference optimizations

GQA is one of several techniques that attack the same memory problem from different angles, and they stack:

• KV cache quantization stores the cached keys and values at lower precision, cutting bytes per entry.
• Paged or block-based cache management reduces fragmentation so more requests fit in GPU memory.
• Speculative decoding produces several tokens per expensive forward pass, amortizing each read of the cache.
• Sparse or sliding-window attention limits how many past tokens each layer attends to.
• Latent compression schemes project keys and values into a smaller shared space before caching them.

GQA reduces the number of heads; quantization reduces the size of each entry; paging reduces waste. Production serving stacks commonly use several at once. GQA is also independent of choices like mixture-of-experts, which changes the feed-forward layers rather than attention.

What GQA means for cost

For anyone paying for inference, smaller KV caches translate into higher throughput per GPU and better behavior on long prompts. That’s one of the quiet reasons long context windows became economically practical. If you want to see how context length drives a bill, the LLM token cost calculator makes the relationship between input size and price concrete.

The takeaway

Grouped-query attention lets groups of query heads share key and value heads. It keeps most of the expressiveness of multi-head attention while cutting the KV cache — and the memory bandwidth needed to read it on every generated token — by the ratio of query heads to key/value heads. MQA is the extreme version with one shared head; MHA is the other extreme with none shared. GQA’s tunable middle ground is why it became a standard building block in modern LLM architectures.