Dense Retrieval - How It Actually Works: Embeddings, Contrastive Training, and Approximate Nearest Neighbor Search

November 10, 2025 · 15 min read · retrieval · search · IR · dense-retrieval

Lexical retrieval like BM25 has a blind spot built into its design: it can only match words that actually appear. A user who searches for “cheap flights to Tokyo” will not find a page about “low-cost airlines serving Japan” unless the two happen to share terms. This is the vocabulary mismatch problem, and it has been the central weakness of keyword search for as long as keyword search has existed. Query expansion, synonyms and stemming chip away at it, but they work word by word, without understanding the query as a whole.

Dense retrieval approaches the problem from the other direction. Instead of matching terms, it maps every query and every document to a point in a continuous vector space, trained so that texts with related meaning land close together. Retrieval becomes geometry: find the document vectors nearest to the query vector. When Karpukhin et al. introduced Dense Passage Retrieval (DPR) in 2020, it beat a strong BM25 system by 9–19 percentage points in top-20 passage retrieval accuracy on open-domain question answering, and dense retrieval quickly became a standard component of search, question answering and, later, retrieval-augmented generation.

That simple description, “embed and find the nearest neighbours”, hides most of what determines whether a dense retriever works. This article looks at what the vectors represent, how the encoders are trained, how search over millions of vectors is made fast, and where dense retrieval fails.

The basic setup

A dense retriever has two parts:

An encoder $E$ that maps text to a fixed-size vector, $E(\text{text}) \in \mathbb{R}^k$ , with $k$ typically between 384 and 1,024 (sometimes more).
A vector index over the document embeddings that can quickly return the vectors most similar to a query vector.

At index time, every document (or chunk of a document) is passed through the encoder once, and its vector is stored in the index. This is the expensive, offline part. At query time, the query is encoded with the same, or a paired, encoder, and the index returns the top- $k$ nearest document vectors.

Similarity functions

The score between a query embedding $\mathbf{q}$ and a document embedding $\mathbf{d}$ is usually the dot product or cosine similarity:

s_{\text{dot}}(q, d) = \mathbf{q}^\top \mathbf{d}, \qquad s_{\cos}(q, d) = \frac{\mathbf{q}^\top \mathbf{d}}{\lVert \mathbf{q} \rVert\, \lVert \mathbf{d} \rVert}.

Cosine similarity ignores vector length and compares only direction. The dot product does not, so a document with a larger-norm embedding can score higher regardless of direction. Some models deliberately use the norm to encode something like confidence or “generality”. If embeddings are L2-normalized, the two are identical, and they are also equivalent to Euclidean distance, because

\lVert \mathbf{q} - \mathbf{d} \rVert^2 = \lVert \mathbf{q} \rVert^2 + \lVert \mathbf{d} \rVert^2 - 2\,\mathbf{q}^\top \mathbf{d} = 2 - 2\,s_{\cos}(q, d) \quad \text{when } \lVert \mathbf{q} \rVert = \lVert \mathbf{d} \rVert = 1.

So “nearest neighbour by L2” and “highest cosine” give the same ranking for normalized vectors. This matters in practice, because many vector indexes only support one metric. The important rule is to use the similarity function the model was trained with. A model trained with dot product and queried with cosine (or the reverse) loses accuracy for no reason.

Where the vectors come from

Most dense retrievers are built on a pretrained transformer encoder (BERT-family models in early work, and today many purpose-trained embedding models). The encoder produces a contextual vector for every token. A pooling step turns those into a single vector:

[CLS] pooling uses the representation of a special first token. DPR did this.
Mean pooling averages all token vectors, weighted by the attention mask:

\mathbf{e} = \frac{\sum_{t} m_t\, \mathbf{h}_t}{\sum_t m_t}.

Sentence-BERT and many later models use it, and it tends to be more robust.

A pretrained language model is not a good retriever out of the box. Its token representations are trained for masked-word prediction, not for making related texts close in vector space. Raw BERT sentence embeddings are known to be anisotropic: they occupy a narrow cone of the space, so nearly every pair of sentences has high cosine similarity (Ethayarajh, 2019, studied this for contextual embeddings). Retrieval quality comes from the next step, contrastive fine-tuning.

Training: contrastive learning

The training data consists of queries paired with relevant documents (“positives”). The model is trained to score the positive above other documents (“negatives”). The standard objective is the InfoNCE (contrastive cross-entropy) loss. For a query $q$ , its positive $d^+$ and a set of negatives $\{d^-_j\}$ :

\mathcal{L}(q) = -\log \frac{\exp\big(s(q, d^+)/\tau\big)}{\exp\big(s(q, d^+)/\tau\big) + \sum_{j} \exp\big(s(q, d^-_j)/\tau\big)}.

This is a softmax classification: out of all the candidates, pick the right one. The temperature $\tau$ (often around 0.01–0.05 for cosine similarity) sharpens the distribution, which makes the model focus on separating the positive from the most similar negatives.

In-batch negatives

Encoding thousands of negatives per query would be expensive. In-batch negatives reuse what is already there: in a batch of $B$ query–positive pairs, each query treats the other $B - 1$ positives as negatives. The full score matrix is one matrix multiplication:

import torch
import torch.nn.functional as F

def in_batch_contrastive_loss(q_emb, d_emb, temperature=0.05):
    """q_emb, d_emb: (B, k). Row i of d_emb is the positive for query i."""
    q = F.normalize(q_emb, dim=-1)
    d = F.normalize(d_emb, dim=-1)
    scores = q @ d.T / temperature                  # (B, B): every query vs every doc
    labels = torch.arange(q.size(0), device=q.device)   # positives on the diagonal
    return F.cross_entropy(scores, labels)

This makes batch size an important hyperparameter: a larger batch gives more negatives per query for free, and contrastive retrievers often train with batches in the thousands, using techniques such as gradient caching or sharing negatives across GPUs.

Hard negatives

Random in-batch negatives are mostly easy. A query about KV caches is trivially different from a document about gardening, and the model learns little from it. The model improves most when it has to separate the positive from documents that are almost relevant: same topic, shared keywords, wrong answer. These are hard negatives.

DPR added one BM25 hard negative per query: a passage with high lexical overlap that does not contain the answer.
ANCE (Xiong et al., 2020) mined negatives from the retriever itself, using an index refreshed periodically during training, so the model is always trained on its own current mistakes.
RocketQA (Qu et al., 2021) pointed out a serious problem with hard negatives: many of them are actually relevant but unlabelled (false negatives), and training against them pushes good documents away. It used a cross-encoder to filter likely false negatives.

Distillation from cross-encoders

A powerful way to get better training labels is to use a stronger but slower model as a teacher. A cross-encoder (described below) scores query–document pairs very accurately, and the bi-encoder is trained to reproduce those scores, or the margins between positive and negative scores (Margin-MSE, Hofstätter et al., 2020). This transfers fine-grained relevance judgements rather than binary labels.

Pretraining for retrieval

Modern embedding models typically go through several stages: large-scale contrastive pretraining on weakly paired text from the web (title–body, question–answer, query–click pairs, or, in Contriever’s case, two random crops of the same document with no labels at all), then fine-tuning on smaller, high-quality labelled datasets with hard negatives. Some models expect task prefixes or instructions. E5, for example, is trained with "query: " and "passage: " prefixes, and omitting them noticeably degrades results. Instruction-tuned embedding models take a natural-language description of the task, which lets one model produce embeddings for different notions of similarity.

Matryoshka representation learning (Kusupati et al., 2022) trains embeddings so that the first $m$ dimensions are themselves a useful embedding, for several values of $m$ . A 1,024-dimensional vector can then be truncated to 256 dimensions for a cheap first-pass search, with a modest loss in quality, and the full vector used for rescoring.

Bi-encoders, cross-encoders, and late interaction

How the query and document interact inside the model determines both quality and cost.

Bi-encoder (dense retrieval). Query and document are encoded independently. Document vectors can be precomputed, and search is a vector similarity. The weakness is that all information about the document has to be compressed into one vector before the query is known, and the only interaction is a single dot product.

Cross-encoder. The query and document are concatenated and passed through the transformer together, so every query token can attend to every document token, and a classification head outputs a relevance score. This is far more accurate, but nothing can be precomputed: scoring a query against a million documents would mean a million transformer forward passes. Cross-encoders are therefore used as rerankers over a small candidate set.

Late interaction (ColBERT). Khattab and Zaharia (2020) proposed a middle ground. Keep one vector per token for both query and document, precompute the document token vectors, and score with the MaxSim operator: for each query token, take its best match among the document tokens, and sum:

s(q, d) = \sum_{i \in q} \max_{j \in d}\; \mathbf{q}_i^\top \mathbf{d}_j.

This keeps much of the token-level matching precision of a cross-encoder while allowing documents to be precomputed. The cost is storage: one vector per token instead of one per document. ColBERTv2 (Santhanam et al., 2021) reduced that substantially with residual compression.

	Bi-encoder	Late interaction	Cross-encoder
Document precomputation	One vector per document	One vector per token	None possible
Query-time cost	One encoding + ANN search	One encoding + token-level search	One forward pass per document
Storage	Small	Large (tens of times larger)	None
Accuracy	Good	Better	Best
Typical role	First-stage retrieval	First stage or reranking	Reranking the top 50–200

Diagram comparing bi-encoder, cross-encoder and late-interaction retrieval architectures

Figure 1. Where query and document interact. Bi-encoders compare two precomputed vectors, cross-encoders read both texts together, and late interaction compares token vectors.

Approximate nearest neighbor search

With $N$ documents of dimension $k$ , exact search computes $N$ dot products, so $O(Nk)$ per query. For 10 million 768-dimensional vectors that is 7.7 billion multiply-adds per query. That is feasible on a GPU for small query volumes, but too slow and expensive for a high-traffic system on CPUs. Memory is a problem too: 10 million float32 vectors of 768 dimensions take about 30.7 GB.

Approximate nearest neighbour (ANN) indexes trade a small amount of recall for large gains in speed and memory. The main families:

IVF: partitioning the space

An inverted file index (IVF) clusters the document vectors with k-means into $C$ cells (for example a few thousand). Each vector is stored in the list of its nearest centroid. At query time, the query is compared with the centroids, and only the vectors in the nprobe closest cells are searched. With $C = 4096$ and nprobe = 16, about 0.4% of the vectors are scored. The risk is that a true neighbour lies in a cell just outside the probed set, so nprobe is the main recall–speed knob.

Two-dimensional illustration of IVF search: k-means cells, a query, and the three probed cells whose vectors are scored

Figure 2. IVF in two dimensions. Vectors are grouped into k-means cells, and only the cells closest to the query are searched. A true neighbour just across a cell boundary can be missed.

Product quantization: compressing the vectors

Product quantization (PQ) (Jégou et al., 2011) compresses each vector. The $k$ dimensions are split into $m$ subvectors, each subspace gets its own small k-means codebook (typically 256 centroids, so one byte per subvector), and each subvector is replaced by its centroid’s ID. A 768-dimensional float32 vector (3,072 bytes) encoded with $m = 96$ becomes 96 bytes, a 32× reduction, so the 10-million-vector example fits in under 1 GB. Distances are computed quickly from precomputed tables of query-to-centroid distances. Because quantization is lossy, a common pattern is to retrieve candidates with PQ and rerank them with the full-precision vectors. IVF and PQ are usually combined (IVF-PQ).

HNSW: navigating a graph

Hierarchical Navigable Small World graphs (Malkov and Yashunin, 2016) connect each vector to a set of near neighbours, in a hierarchy of layers where upper layers are sparse “express lanes” and the bottom layer contains everything. Search starts at the top, greedily moves toward the query, and descends layer by layer, keeping a candidate list of size efSearch. Search cost grows roughly logarithmically with $N$ .

HNSW gives excellent recall–latency trade-offs and is the default in many vector databases. Its costs are memory, for the graph links (with M = 32 neighbours, the bottom layer alone needs on the order of 256 bytes per vector for 4-byte IDs, on top of the vectors themselves), slow index construction, and awkward deletions.

Other approaches

ScaNN (Guo et al., 2020) uses anisotropic quantization, which preserves the components of a vector that matter most for inner-product ranking. DiskANN (Subramanya et al., 2019) builds a graph designed for SSD-resident indexes, for collections too large for RAM.

import faiss
import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("intfloat/e5-base-v2")
docs = ["passage: " + text for text in corpus]                 # model-specific prefix
doc_emb = model.encode(docs, normalize_embeddings=True).astype(np.float32)

d = doc_emb.shape[1]
index = faiss.IndexHNSWFlat(d, 32, faiss.METRIC_INNER_PRODUCT)  # M = 32
index.hnsw.efConstruction = 200
index.add(doc_emb)

index.hnsw.efSearch = 64                                        # recall / latency knob
q = model.encode(["query: how does a kv cache work"], normalize_embeddings=True)
scores, ids = index.search(q.astype(np.float32), 10)

Measuring ANN quality is a separate question from measuring retrieval quality. ANN recall@k compares the approximate top- $k$ against the exact top- $k$ from a flat index. A system can have 99% ANN recall and still return poor results if the embedding model is poor, and a great embedding model can be undermined by an index tuned for speed at 80% recall.

Evaluating dense retrievers

Retrieval metrics: recall@k (did the relevant documents appear in the top $k$ ?) for first-stage retrieval, and nDCG@10 or MRR when the order of the first results matters.
Benchmarks: BEIR (Thakur et al., 2021) tests zero-shot retrieval on 18 diverse datasets. MTEB (Muennighoff et al., 2022) covers retrieval along with other embedding tasks. Leaderboard positions are a useful starting point but often transfer poorly to a specific domain, and some benchmark data has leaked into model training sets.
Your own data: the most important evaluation is a few hundred real queries from the target system, labelled for relevance. Comparing the dense retriever against tuned BM25 on that set is the fastest way to find out whether it helps.

Limitations

Out-of-domain generalization. BEIR’s most cited finding was that several dense retrievers which clearly beat BM25 in-domain fell behind it zero-shot on specialized collections, such as biomedical, scientific and argument retrieval. Embedding models learn the notion of similarity in their training data, and that notion does not always transfer. Newer models trained on much broader data have narrowed this gap considerably, but it is still worth testing for.

Exact matching. Rare entities, product codes, error messages, numbers and new terminology are poorly represented in an embedding space. Two documents about different versions of a library can be nearly identical vectors. For “find the document that mentions ERR_CONN_RESET”, BM25 is more reliable.

Compression into one vector. A single vector has limited capacity to represent a long document with many topics, which is why documents are split into chunks before embedding. Weller et al. (2025) also argued that, for a fixed embedding dimension, there are combinations of documents that a single-vector retriever cannot return as a top- $k$ set for any query, and built a simple dataset (LIMIT) on which strong embedding models do poorly.

Negation and fine logical distinctions. “Hotels that allow pets” and “hotels that do not allow pets” are very close in most embedding spaces.

Operational cost. Every document must be embedded, which needs GPU time and a pipeline to keep it in sync. Changing or upgrading the embedding model means re-embedding the entire corpus, because vectors from different models are not comparable. Vector indexes also use more memory than inverted indexes, and handle updates and deletions less gracefully.

Opacity. When BM25 returns a document, the matching terms explain why. When a dense retriever returns one, the reason is a dot product between two learned vectors, which makes failures harder to diagnose.

When dense retrieval wins

Semantic similarity where the wording differs: “cheap flight” against “low-cost airline”.
Paraphrases and questions phrased very differently from the answer text.
Cross-lingual retrieval, with multilingual encoders that map different languages into a shared space.
Queries that describe intent rather than exact keywords: “a laptop for travelling that lasts all day” rather than “laptop battery life 14 hours”.
Natural-language questions against answer-bearing passages, which is exactly the setting DPR was trained for.

Practical notes

Keep keyword search. In most production systems dense and sparse retrieval are complementary: dense handles meaning, sparse handles exact terms, and hybrid search usually beats either alone.
Don’t choose a model by dimension. Training data and objective matter far more than vector size. A good 384-dimensional model can beat a weak 1,536-dimensional one at a quarter of the memory.
Don’t use fixed similarity thresholds. Scores are only meaningful relative to other scores from the same model, and different models have very different score distributions.
Don’t expect the vector database to fix relevance. The index decides how fast and how approximately nearest neighbours are found. Whether those neighbours are relevant depends on the embedding model.

Dense retrieval in production

A typical high-quality retrieval stack today combines the pieces above:

Chunk and embed documents with a model evaluated on the target domain, using its required prefixes or instructions, and normalize the vectors if the model was trained with cosine similarity.
Index with HNSW for latency-sensitive, moderate-size collections, or IVF-PQ for very large ones, with full-precision rescoring of the top candidates.
Retrieve in parallel with BM25, and merge the two result lists with reciprocal rank fusion, which needs no score calibration.
Rerank the merged top 50–200 with a cross-encoder.
Monitor retrieval quality on a fixed set of labelled queries, and re-run it whenever the embedding model, chunking or index parameters change.