Articles

Matryoshka Embeddings: Shrinking Vectors, Keeping Meaning

Matryoshka embeddings pack the most important information into the first dimensions, so you can shorten vectors to cut storage and search cost.

Chisato Chisato · · 5 min read
Abstract neural network fibers in purple

Matryoshka embeddings are vector embeddings trained so that their leading dimensions carry the most important information, which means you can chop a vector down to its first 256, 128, or even 64 numbers and still get a useful representation. The name comes from Russian nesting dolls: a large embedding contains a smaller, coarser embedding inside it, which contains a smaller one still.

The technique, introduced in research as Matryoshka Representation Learning, gives you one model and one set of vectors that you can resize after the fact to trade accuracy for storage, memory, and search speed.

Why embedding size matters

A vector embedding represents a piece of text, an image, or other data as a list of floating-point numbers. Similar items end up close together in that space, which is what powers semantic search, recommendation, clustering, and the retrieval step of retrieval-augmented generation.

Embedding models commonly produce vectors with hundreds or a few thousand dimensions. Every dimension costs something:

  • Storage. A 1,024-dimension vector of 32-bit floats takes 4 KB. Across a hundred million documents, that is roughly 400 GB before any index overhead.
  • Memory. Approximate nearest neighbour indexes perform best when the vectors fit in RAM.
  • Compute. Each similarity calculation is a dot product across every dimension, so search cost scales with vector length.

With an ordinary embedding model, you cannot simply throw dimensions away. Information is spread across all of them in no particular order, so dropping the last half of the vector destroys meaning more or less at random.

How Matryoshka training works

A standard embedding model is trained with a loss function computed on the full vector. Matryoshka training changes one thing: it computes the loss on several nested prefixes of the vector at once and adds them together.

Suppose the full vector has 1,024 dimensions. During training, the model is evaluated using:

  • the first 64 dimensions,
  • the first 128 dimensions,
  • the first 256 dimensions,
  • the first 512 dimensions,
  • and all 1,024 dimensions.

Each prefix must independently do a good job on the training objective, such as placing a query close to its matching document. To satisfy every prefix simultaneously, the model learns to put the most broadly useful, coarse-grained information in the earliest dimensions and progressively finer detail in later ones.

The result is a single embedding in which any prefix is itself a reasonable embedding. Accuracy declines gracefully as you truncate, instead of collapsing.

Using a truncated embedding

Using a smaller version is straightforward:

  1. Generate the full embedding as usual.
  2. Keep the first k dimensions and discard the rest.
  3. Re-normalise the truncated vector to unit length if you are using cosine similarity or a dot product on normalised vectors.

The re-normalisation step is easy to forget. The full vector was normalised as a whole, so its prefix will generally have a smaller magnitude, and skipping normalisation skews similarity scores.

Some embedding services and open-source models trained this way let you request a specific output size directly, which performs the truncation and normalisation for you. If a model’s documentation does not say it was trained with a Matryoshka objective, assume truncation will hurt quality badly.

Adaptive retrieval: small vectors first, big vectors second

The most powerful pattern Matryoshka embeddings enable is adaptive retrieval, a two-stage search:

  1. Shortlist with short vectors. Search the whole corpus using, say, the first 128 dimensions. This is fast and memory-cheap, and it retrieves a candidate set of a few hundred items that almost certainly contains the true best matches.
  2. Rerank with full vectors. Recompute similarity for just those candidates using all dimensions and reorder them.

Because the full vector contains the short one, you store one embedding per item and simply read different lengths of it at each stage. The expensive comparisons happen only on a small candidate set, so you approach full-vector accuracy at a fraction of the cost. A vector database can hold the short prefix in its in-memory index and keep full vectors on disk for the rerank step.

Matryoshka vs other ways to shrink vectors

Matryoshka truncationScalar or binary quantizationDimensionality reduction (e.g. PCA)
What it reducesNumber of dimensionsBits per dimensionNumber of dimensions
Requires special trainingYesNoNo, but needs a fitting step
Extra computationNone, just sliceEncode onceProject every vector
Reversible within the same vectorYes, keep the full vectorOnly if you keep originalsOnly if you keep originals
Combinable with othersYesYesRarely needed together

These techniques stack. Truncating to 256 dimensions and then applying vector quantization can shrink storage by an order of magnitude or more, with a rerank step against higher-precision vectors recovering much of the lost accuracy.

Choosing a dimension

There is no universal right size. The practical approach is empirical:

  • Build a small evaluation set of real queries with known relevant results.
  • Measure retrieval quality, such as recall at 10, at several truncation sizes.
  • Plot quality against cost and pick the knee of the curve.

You will often find that quality drops only slightly over the first few halvings and then falls off more sharply at very small sizes. Domain matters too: a corpus with many near-duplicate documents needs finer distinctions, and therefore more dimensions, than one with clearly separated topics.

Also consider how your index behaves. Graph-based indexes such as HNSW do more distance computations per query than the corpus size alone would suggest, so shorter vectors speed up both building the index and searching it.

Limitations

  • Model-dependent. Only models trained with the nested objective truncate well. You cannot retrofit the property onto an existing model without retraining or fine-tuning.
  • Not free accuracy. Short prefixes are coarser. For high-precision tasks, plan on a rerank stage.
  • Re-embedding on model change. As with any embedding, switching models means re-embedding the corpus. Matryoshka flexibility applies within one model’s vectors, not across models.

The takeaway

Matryoshka embeddings are trained so every prefix of the vector is a usable embedding, with the most important information packed into the first dimensions. That lets you store one vector and read as much of it as each task needs: short prefixes for fast, memory-efficient candidate search, full vectors for precise reranking. Confirm the model was trained this way, re-normalise after truncating, and pick the dimension by measuring recall on your own data.

Chisato Chisato · · 3 min read

What Are Vector Embeddings? Meaning as Numbers

A vector embedding turns text, images, or audio into numbers where similar meanings land close together — the foundation of semantic search and RAG.

#AI #Machine Learning #Databases
Chisato Chisato · · 5 min read

Prefill vs Decode: The Two Phases of LLM Inference

LLM inference runs in two phases: prefill processes the prompt in parallel, decode generates tokens one at a time. Why they behave so differently.

#AI #LLMs #Machine Learning