Vector Databases and Embeddings Interview Prep

Vector databases and embeddings for AI engineers: semantic search, HNSW vs IVF indexing, chunking strategy, hybrid search, reranking, pgvector, and vector-vs-graph trade-offs.

Quick answer

Vector databases and embeddings are the storage and indexing substrate behind semantic search.

Retrieval quality is the biggest lever on RAG answer quality, and vector databases plus embeddings are the retrieval substrate.

Editorial review

Written by

CompoundLearn editorial team

Wireless / RF / hardware engineering

Reviewed by

CompoundLearn editorial team

Wireless / RF / hardware engineering

Last reviewed

Built from curated topic maps, editorial validation, and subject-matter review so the page stays aligned with the interview intent and the current content pipeline.

Key points

  • Chunking often moves retrieval quality more than index tuning does — get chunk size and overlap right before fine-tuning HNSW parameters.
  • HNSW is a common low-latency default but its memory cost scales with the graph link count (M); IVF fits when memory is constrained or the corpus makes HNSW build cost prohibitive.
  • Hybrid search — BM25 + dense embeddings, often fused via Reciprocal Rank Fusion — is a common default for serious retrieval; pure semantic search misses exact tokens and rare nouns.
  • On L2-normalised vectors, cosine similarity and dot-product give identical rankings; the metric choice rarely shifts recall@K by more than a point or two.
  • Store the embedding-model version as metadata on every vector so upgrades are detectable and re-indexing can be triggered — the durable guard against embedding drift.
  • Bigger embedding dimensions are not automatically better; architecture, training data, and domain match dominate — measure recall@K on your own query distribution.
  • Flat vector retrieval answers "passages about X"; escalate to graph traversal once multi-hop and relationship queries become a measurable fraction of traffic.

What it is

Vector databases and embeddings are the storage and indexing substrate behind semantic search. An embedding is a dense vector representation of text — produced by models such as OpenAI text-embedding-3, Voyage AI, or BGE — that places semantically similar content near each other in a high-dimensional space. A vector database (pgvector, Qdrant, Pinecone, Weaviate, Milvus) stores those embeddings with metadata and runs fast approximate-nearest-neighbor (ANN) search over them. The gain over keyword search: you no longer need exact word overlap, because meaning-similar text has nearby vectors, so a query finds relevant content even when the phrasing differs. Two ANN algorithms dominate the indexing layer. HNSW (Hierarchical Navigable Small World) builds a multi-layer proximity graph; queries navigate from coarse upper layers to fine lower layers, giving very low latency and high recall at the cost of more memory — the extra cost comes from the neighbor links each node stores, governed by the graph's link count, not from a flat per-vector tax. IVF (Inverted File) partitions vectors into clusters and probes only a subset per query — typically lower memory and slightly lower recall, which suits very large corpora where HNSW build cost is prohibitive. Neither vector search alone nor keyword search (BM25) is usually optimal on its own. Hybrid search — combining BM25 for exact-match precision with dense embeddings for semantic recall, often fused via Reciprocal Rank Fusion — is a common default for serious retrieval systems, though weighted fusion and learned ranking are also used. A reranker (cross-encoder) then refines the top-k candidates by scoring query and document together, adding precision at a bounded latency cost. Chunking — how source documents are split before embedding — is an underrated tuning lever. Chunks that are too short lose context; chunks that are too long bury the relevant sentence. A common working range is 256–1,024 tokens with 20–50 token overlap, but the right value depends on document structure and is worth measuring against your actual query distribution rather than copying a default. On production fit: pgvector (Postgres HNSW + halfvec storage) handles many RAG workloads up to tens of millions of vectors at moderate query rates with no extra service. Whether you stay on it or move to a purpose-built database is a sizing question — hardware, vector dimension, metadata-filter selectivity, write rate, recall target, and latency SLO decide it, not the brand. Dedicated engines earn their complexity at higher write throughput, very large scale, or when the pipeline needs vector and graph traversal in one engine (Neo4j 5+ supports both). Flat vector retrieval has a ceiling worth naming, because it marks where this substrate stops being enough. A vector index answers one question shape well: "which passages are about X?" It ranks by semantic proximity, so a query and its answer must sit close in embedding space. Multi-hop and relationship questions break that assumption — "which components depend on the auth service AND shipped a breaking change last quarter?" chains two relations that are rarely stated together in any single passage, so co-occurring text alone usually cannot satisfy it. That is the escalation boundary: once relationship and multi-hop queries become a measurable fraction of traffic, the fix is typically graph traversal over a knowledge graph (GraphRAG for global or thematic questions, PageRank-style graph retrieval for multi-hop factoids), not a bigger embedding model — usually layered as a hybrid where vector search finds the entry-point nodes and graph traversal expands them along explicit edges.

Why interviewers ask

Retrieval quality is the biggest lever on RAG answer quality, and vector databases plus embeddings are the retrieval substrate. Interviewers probe this area because it separates engineers who have shipped retrieval systems from those who have only read about them. The questions that distinguish strong candidates turn on trade-offs and production judgment, not definitions. Strong answers show four things: why chunking often moves retrieval quality more than index-parameter tuning; how to pick between HNSW and IVF from scale, memory, and latency constraints; when hybrid search (BM25 + dense) beats pure semantic search; and where a reranker fits and what it costs. The vector-versus-graph trade-off carries higher signal. Interviewers want to hear that vector retrieval answers "find a passage about X" well but cannot traverse relationships — and that this becomes a bottleneck once multi-hop queries ("what depends on X, and what does that in turn depend on?") are a measurable fraction of traffic. Knowing when to escalate from flat vector retrieval to a hybrid knowledge-graph plus vector setup, and what that costs operationally, is a senior-level tell. Embedding drift, re-indexing strategy, and metric tracking (recall@K, mean reciprocal rank) test whether a candidate treats retrieval quality as a production engineering problem with monitoring and alerting — not a one-time setup task.

Common mistakes

The most common mistake is treating embedding-model selection as a one-time decision. Generic models trained on general text often underperform on specialised domains (legal, medical, RF/wireless, code). When retrieval quality stalls, audit first whether the model's training distribution matches your corpus; a domain-matched model, or contrastive fine-tuning on in-domain pairs, can lift recall@K materially — measure the gain on your own data rather than assuming a fixed number. The second systemic mistake is neglecting chunking. Engineers spend days on HNSW parameters and minutes on chunk size and overlap, even though chunk size frequently affects retrieval quality more than index tuning does. Chunks that are too short (under roughly 100 tokens) fragment sentences and produce misleading embeddings; chunks that are too long (over roughly 2,000 tokens) bury the relevant sentence in noise. Other frequent pitfalls: not measuring retrieval quality at all and trusting demo impressions; using pure vector search when queries carry exact product codes, acronyms, or rare proper nouns that BM25 catches more reliably; assuming bigger embedding dimensions mean better quality (they do not — architecture and training data matter more than dimension count); and failing to store the embedding-model version with each vector, which makes embedding-drift regressions hard to debug. At the architecture level, teams confuse when to use vector retrieval versus knowledge-graph retrieval. Flat vector search cannot follow explicit relationships across entities, so teams that stay on it as multi-hop queries grow hit a hard-to-diagnose quality ceiling. The fix is usually hybrid — vector search for entry-point discovery, graph traversal for relationship expansion — rather than trying to encode relationship structure into embedding space.

Vector store selection — honest, vendor-neutral trade-offs (capabilities evolve; verify current docs before committing)

OptionIndex typesPractical scale fitOps complexityHybrid (BM25 + dense)Choose when
pgvector (Postgres extension)HNSW + IVF; halfvec half-precision storageOften comfortable to tens of millions of vectors at moderate QPS (sizing-dependent)Lowest — no new service; reuses Postgres ops, backups, RLS, ACIDVia Postgres full-text search alongside vectors (manual fusion)You already run Postgres and want vectors in the same transactional stack
QdrantHNSW (default); payload filteringTens to hundreds of millionsModerate — dedicated service to run and monitorBuilt-in sparse + dense with fusionYou want a dedicated engine with strong metadata filtering
WeaviateHNSW; supports object cross-referencesTens to hundreds of millionsModerateBuilt-in hybrid (BM25 + dense) with fusionYou want vector search plus graph-style cross-references in one engine
MilvusHNSW, IVF, and GPU-accelerated indexesHundreds of millions to billionsHigher — distributed components to operateSupported, more assemblyVery large corpora or GPU-accelerated indexing at extreme scale
Pinecone (managed)Proprietary serverless ANNLarge; the managed service handles scaling for youLow operationally, but a managed dependency (cost, vendor coupling)Supported via sparse-dense vectorsYou want managed serverless retrieval and accept the managed-service trade-off

Sample interview questions

  1. You are choosing an ANN index for a 200M-vector corpus on memory-constrained nodes, where p99 latency can tolerate a few extra milliseconds but RAM is the binding constraint. Which index choice and reasoning is correct?
    • A. IVF, because partitioning vectors into clusters and probing only a subset keeps the resident memory footprint low and the build cost tractable at 200M vectors, trading a little recall for the memory headroom the deployment actually needs.
    • B. HNSW, because its multi-layer graph always delivers both the lowest latency and the lowest memory footprint, so it is the correct default regardless of corpus size or RAM budget.
    • C. A flat (brute-force) index, because at 200M vectors exact search is still fast enough and avoids any recall loss from approximation.
    • D. IVF, because it is an exact index that guarantees 100% recall while using less memory than HNSW.

    Option A is correct because the binding constraint here is memory, not latency. IVF (Inverted File) partitions the vector space into clusters and at query time probes only the nprobe nearest clusters; it builds faster than HNSW on a very large corpus and avoids HNSW's per-node graph-link overhead, so for a given precision it typically carries a smaller memory footprint (smaller still when paired with PQ/SQ compression). The cost is a recall reduction relative to a well-tuned HNSW graph — workload-dependent, but acceptable when the deployment can spend a few extra milliseconds yet cannot spend the memory. Option B is incorrect because HNSW does not minimize memory. HNSW gives very low latency and high recall, but its multi-layer proximity graph stores extra neighbor links per node — memory that scales with the link count (M) on top of the stored vectors — so it usually consumes more RAM per vector than IVF and is the wrong default precisely when memory is the constraint. Option C is incorrect because brute-force search scans every vector, so query cost grows linearly with corpus size. At 200M vectors a flat scan is far too slow for interactive retrieval, which is why ANN indexes exist in the first place. Option D is incorrect on its premise: IVF is an approximate index, not an exact one. Probing a subset of clusters means a query can miss neighbors that fell into an unprobed cluster, so IVF does not guarantee 100% recall — recall is traded against the number of clusters probed.

  2. For a text-retrieval workload whose embedding model returns L2-normalised vectors, an engineer claims switching the distance metric from cosine similarity to dot-product will materially change which documents rank in the top-k. What is the correct assessment?
    • A. On L2-normalised vectors, cosine similarity and dot-product produce the same ranking, because cosine is dot-product divided by both magnitudes and the magnitudes are 1 — so the metric switch does not change the top-k.
    • B. Dot-product will always rank longer documents higher than cosine, because it rewards larger vector magnitude regardless of normalisation.
    • C. L2 (Euclidean) distance is interchangeable with cosine for any embedding model, so all three metrics are equivalent in every case.
    • D. Cosine similarity is exact while dot-product is approximate, so the switch will lower recall by definition.

    Option A is correct because cosine similarity is the dot-product of two vectors divided by the product of their magnitudes. When both vectors are L2-normalised, each magnitude is exactly 1, so the denominator is 1 and cosine reduces to plain dot-product. The two metrics therefore induce the same ordering on normalised vectors, and most modern text-embedding APIs return normalised vectors — so the metric switch is a no-op for ranking. This is why production systems often configure dot-product on normalised embeddings: it is cheaper to compute and gives identical results. Option B is incorrect because the magnitude-rewarding behaviour of dot-product only appears on un-normalised vectors. Once vectors are L2-normalised, every vector has magnitude 1, so dot-product cannot favour "longer" documents — there is no length signal left to reward. Option C is incorrect because L2 distance and cosine are only monotonically related when vectors are normalised, and even then the relationship is a transformation of cosine, not an unconditional equivalence. For un-normalised vectors the two can rank results differently; the right metric depends on the loss the embedding model was trained with. Option D is incorrect because neither metric is "approximate" — the approximation in ANN retrieval comes from the index structure (HNSW, IVF), not from the choice between cosine and dot-product. Both are exact similarity functions; swapping them on normalised vectors does not change recall.

  3. A team running a production RAG system on roughly 8M vectors and a few hundred QPS, already operating Postgres, is debating moving to a purpose-built vector database for "scalability." What is the strongest engineering judgment?
    • A. Stay on pgvector: at single-digit-million vectors and moderate QPS it performs comparably to purpose-built engines while keeping vectors inside the existing SQL stack — ACID, row-level security, and one fewer service to operate. Purpose-built engines earn their keep at tens of thousands of QPS or hundreds of millions of vectors.
    • B. Migrate immediately, because pgvector cannot build HNSW indexes and therefore cannot do approximate-nearest-neighbor search at all.
    • C. Migrate immediately, because purpose-built vector databases are always faster than pgvector at every scale, so there is never a reason to stay on Postgres for vectors.
    • D. Stay on pgvector only if the corpus is under 100K vectors; above that, Postgres cannot store embeddings.

    Option A is correct because the decision is driven by scale and QPS, not by branding. pgvector adds HNSW and IVF indexing plus half-precision storage to Postgres, and at single-digit-million vectors with moderate query rates it performs comparably to dedicated engines. Staying put keeps embeddings inside the existing transactional stack — ACID guarantees, Postgres row-level security, and no extra service to deploy, monitor, and back up. The trade-off flips toward purpose-built systems only at very high QPS (tens of thousands) or very large corpora (hundreds of millions of vectors), where dedicated query paths and write throughput matter. Option B is incorrect because pgvector does support HNSW (and IVF) indexes — that is precisely what makes it viable for million-scale retrieval. The claim that it cannot do ANN search is simply false. Option C is incorrect because "always faster at every scale" ignores the operational reality. At moderate scale the latency difference is negligible, and the cost of running and securing a second data system can outweigh any marginal speed gain. Engineering judgment weighs total cost, not peak benchmark numbers. Option D is incorrect because Postgres with pgvector comfortably stores and indexes millions of vectors; the 100K figure is fabricated. Corpus size influences which index and storage settings to use, not whether Postgres can hold the data at all.

  4. Retrieval quality on a technical-documentation corpus is poor: retrieved chunks pass the similarity threshold but answers are vague and miss the relevant sentence. The team has been tuning HNSW ef_search. What is the highest-leverage fix?
    • A. Revisit chunking: chunks that are too long bury the answer sentence in noise so the chunk matches semantically but the LLM cannot find the fact, while chunks that are too short fragment context — fixing chunk size and overlap typically moves retrieval quality more than index-parameter tuning.
    • B. Increase the embedding dimension count, because higher-dimensional vectors always capture more meaning and resolve vague retrieval.
    • C. Lower the similarity threshold so more chunks are returned, because returning more candidates is always better than returning fewer.
    • D. Switch from HNSW to IVF, because IVF inherently returns more relevant chunks for technical prose.

    Option A is correct because the symptom — chunks that pass the similarity check but still produce vague answers — is the classic signature of a chunking problem, not an index problem. When a chunk is too long, the relevant sentence is diluted among unrelated text: the chunk embedding is close enough to the query to be retrieved, but the LLM then has to sift the answer out of noise. When chunks are too short, sentences fragment and embeddings lose context. Chunk size and overlap usually have a larger effect on end-to-end retrieval quality than index-parameter tuning, which is why chunking is the most-overlooked production lever. The fix is to tune chunk size (commonly 256–1,024 tokens with 20–50 token overlap) against the actual query distribution. Option B is incorrect because bigger embedding dimensions do not reliably improve quality. Model architecture and training data dominate; beyond a point, extra dimensions add storage and compute cost without improving recall, and they do nothing about a chunk that buries its answer sentence. Option C is incorrect because lowering the threshold returns more chunks but also more irrelevant ones, increasing noise in the LLM context window — which is the very failure mode being observed. More candidates is not unconditionally better; precision matters. Option D is incorrect because IVF is a different ANN partitioning strategy, not a relevance booster. It generally trades some recall for lower memory; it does not address the chunk-too-long problem, so swapping the index leaves the root cause untouched.

  5. A legal-search RAG system must reliably retrieve documents that contain exact statute numbers and rare proper nouns, as well as paraphrased concepts. Pure dense (semantic) retrieval is missing some exact-match queries. What is the production-standard fix?
    • A. Hybrid search: run BM25 (lexical) and dense embedding retrieval in parallel and fuse the two ranked lists with Reciprocal Rank Fusion, because BM25 catches exact tokens like statute numbers and rare nouns that dense vectors blur, while dense retrieval catches paraphrases that BM25 misses.
    • B. Switch entirely to BM25, because lexical matching is strictly superior to embeddings for all retrieval tasks.
    • C. Increase the dense top-k to a very large number, because exact-match queries will eventually appear in the candidate set if you retrieve enough vectors.
    • D. Add a reranker in front of the first-stage retriever so the cross-encoder performs the initial corpus-wide search and finds the exact matches.

    Option A is correct because the failure — missing exact statute numbers and rare proper nouns while still needing paraphrase recall — is exactly what hybrid search exists to solve. Dense embeddings generalise across phrasings but smear exact, low-frequency tokens; BM25 scores precise lexical overlap and reliably surfaces an exact statute number. Running both in parallel and merging their ranked lists — Reciprocal Rank Fusion is a common choice, though weighted or learned fusion also works — gives a single list with lexical precision and semantic recall. This is why regulated-domain retrieval (legal, medical, finance) commonly runs hybrid rather than pure semantic. Option B is incorrect because BM25 alone reintroduces the opposite gap: it misses synonyms and paraphrases that have no lexical overlap with the query. "Strictly superior for all tasks" is false — each method covers the other's blind spot, which is the whole rationale for fusing them. Option C is incorrect because inflating dense top-k does not fix a representation problem. If the exact token is not well separated in embedding space, it may not appear in the dense candidate set at any reasonable k, and pushing k very high floods the reranker and the context window with noise. Option D is incorrect because a cross-encoder reranker is an O(k) second-stage model that scores a small candidate set; it cannot scan the full corpus as a first-stage retriever. Rerankers refine the top candidates after first-stage retrieval (vector and/or BM25); they do not replace it.

  6. Where does a cross-encoder reranker belong in a retrieval pipeline, and what governs its cost?
    • A. After first-stage retrieval, as a second stage: first-stage retrieval (vector and/or BM25) cheaply narrows the corpus to a candidate set of, say, the top-100, then the reranker scores query and each candidate jointly to pick the final top-k — so its cost is bounded by the candidate-set size, not the full corpus.
    • B. Before first-stage retrieval, as the primary search over the entire corpus, because joint query-document scoring is the most accurate way to scan every document.
    • C. In place of the embedding model, because a reranker eliminates the need to store any vectors.
    • D. Only at index-build time, because reranking is a one-time preprocessing step applied to documents before they are stored.

    Option A is correct because a cross-encoder reranker reads the query and a candidate document together and produces a precise joint relevance score, which is materially more accurate than the independent (query-alone, document-alone) comparison that first-stage vector or BM25 search uses. That accuracy is expensive: it requires one forward pass per candidate. The standard design therefore runs a cheap first stage to fetch a bounded candidate set (e.g. top-100 via hybrid search), then reranks that set to the final top-k. The cost scales with the candidate-set size, not the corpus, which keeps it affordable. Option B is incorrect because running a cross-encoder over the entire corpus means one forward pass per document on every query — computationally infeasible at any real corpus size. The two-stage retrieve-then-rerank pattern exists precisely to avoid this. Option C is incorrect because the reranker does not replace embeddings or storage. It needs a first-stage retriever to produce candidates; without stored vectors (or a BM25 index) there is nothing to rerank. Option D is incorrect because reranking is a query-time operation: the relevance of a document depends on the specific query, so it cannot be precomputed once at index-build time independent of the query.

  7. A production retrieval system shows a slow, sustained decline in recall@K and click-through on retrieved results over several weeks, not explained by any query-distribution shift. The embedding model was upgraded last month. What is the most likely diagnosis and the right operational guard?
    • A. Embedding drift / stale vectors after the model change: the safest guard is to store the embedding-model version as metadata on each vector and trigger a re-embedding (re-index) job when the model is upgraded or when tracked retrieval metrics fall below threshold.
    • B. The vector database is running out of disk, so the fix is simply to add storage.
    • C. The similarity metric silently changed from cosine to L2; reverting the metric will restore recall with no re-indexing.
    • D. Recall cannot be monitored in production, so the decline is unmeasurable and the only option is to rebuild the entire system from scratch.

    Option A is correct because a sustained recall and engagement decline that is not explained by changing queries, following a model upgrade, is the signature of stale embeddings — newly served queries are embedded with the new model while stored document vectors still reflect the old one (or the corpus meaning has drifted). The durable guard is to tag every vector with the embedding-model version, so a model upgrade is detectable and a re-embedding job can be triggered for the affected vectors; and to monitor retrieval metrics (recall@K, mean reciprocal rank, click-through proxies) over time so a slow regression raises an alert rather than going unnoticed. Option B is incorrect because disk pressure produces write failures or hard errors, not a gradual recall decline. Adding storage does not reconcile a query embedded with a new model against documents embedded with an old one. Option C is incorrect because a metric change would cause an immediate step change in ranking, not a slow multi-week drift, and there is no indication the metric changed. Reverting a metric that did not change fixes nothing. Option D is incorrect because retrieval quality is exactly what you should monitor in production — recall@K, MRR, and user-signal proxies are tracked over time precisely to catch this kind of regression. Treating it as unmeasurable, and rebuilding wholesale, ignores the targeted re-index that actually addresses the cause.

  8. A retrieval system answers single-passage "find documentation about X" queries well, but fails on questions like "which services depend on the billing module AND were changed in the last release?" Increasing top-k and reranking has not helped. What is the correct escalation?
    • A. Escalate from flat vector retrieval to graph-structured retrieval over a knowledge graph: multi-hop relationship questions require traversing explicit entity-to-entity edges that no single passage co-states, so the fix is hybrid — vector search finds entry-point nodes, graph traversal expands them along relations — not a bigger embedding model or higher top-k.
    • B. Increase the embedding dimension count, because higher-dimensional vectors can encode multi-hop relationships directly into a single vector.
    • C. Raise top-k far higher, because every multi-hop answer is guaranteed to appear once enough chunks are retrieved.
    • D. Fine-tune the embedding model on the corpus, because fine-tuning lets a single dense vector represent transitive dependency chains.

    Option A is correct because the failing query is multi-hop: it chains two relations ("depends on the billing module" and "changed in the last release") that are unlikely to be stated together in any one passage. Flat vector retrieval ranks by semantic proximity to a single passage, so it does not follow explicit relationships across documents — that is its structural ceiling. (Some multi-hop questions can be handled by query decomposition, metadata filters, or multi-step retrieval, but a relationship-heavy workload is the case graph retrieval targets directly.) The common fix is graph-structured retrieval: build a knowledge graph of entities and relations, and answer relationship and multi-hop questions by traversing edges. In practice this is hybrid: vector search locates the entry-point nodes by semantic similarity, then graph traversal expands those nodes along the relevant relations. Escalate when such multi-hop or relationship queries become a measurable fraction of traffic. Option B is incorrect because adding dimensions does not give a flat index the ability to traverse relations. A dependency chain is a property of edges between entities, not a feature that a larger single-vector representation can encode and then retrieve by proximity. Option C is incorrect because the answer to a multi-hop query is not a single chunk that fails to surface — it must be assembled across multiple documents by following relations. Raising top-k returns more loosely-related passages and more noise; it does not perform the traversal. Option D is incorrect because fine-tuning improves how well embeddings separate in-domain content, lifting recall on "passages about X" queries. It does not let one dense vector represent a transitive dependency chain; the relational structure still has to live in a graph.

  9. When comparing two text-embedding models for a retrieval task, a candidate argues "the 3072-dimension model must retrieve better than the 768-dimension one because it has more dimensions." How should you evaluate this claim?
    • A. Reject the dimension-count heuristic: retrieval quality is driven mainly by the model architecture, training data, and how well the training distribution matches your corpus — bigger embeddings cost more memory and compute without guaranteeing better recall, so the only sound decision is to measure recall@K on your own query distribution.
    • B. Accept it: embedding dimension count is the single best predictor of retrieval quality, so the 3072-dimension model is the correct choice without further testing.
    • C. Accept it for technical corpora but reject it for general text, because dimension count only matters in specialised domains.
    • D. Reject it because higher-dimensional embeddings always retrieve worse due to the curse of dimensionality, so the 768-dimension model is guaranteed superior.

    Option A is correct because "more dimensions equals better" is a well-known fallacy. What determines retrieval quality is the model's architecture and training data, and crucially whether its training distribution matches your corpus — a generic model can underperform a smaller domain-matched one on specialised content (legal, medical, RF/wireless, code). Larger embeddings also carry real costs: more storage per vector and more compute per comparison. Because the relationship between dimension count and quality is not monotonic, the only defensible way to choose is to measure recall@K (and related metrics) on your actual query distribution rather than reasoning from the dimension number. Option B is incorrect because dimension count is not the best predictor of quality; it is a weak proxy at best. Choosing a model on dimension count alone, without measuring, is exactly the mistake the question is probing. Option C is incorrect because the dimension-count heuristic is unreliable for both general and specialised corpora. There is no clean split where "more dimensions" becomes a valid rule for technical text — domain match and training data dominate in every case. Option D is incorrect because it overcorrects into the opposite absolute. Higher-dimensional embeddings do not always retrieve worse; the curse of dimensionality is a real concern but it does not guarantee the smaller model wins. The honest answer is empirical: measure on your data.

Frequently asked questions

What is an embedding and how is it used in retrieval?
An embedding is a dense vector representation of text (or code or images) that places semantically similar items near each other in a high-dimensional space. Production embeddings typically come from models such as OpenAI text-embedding-3, Voyage AI, or BGE, usually producing 768–3072-dimensional vectors (for example 1536 or 3072 for OpenAI text-embedding-3). In retrieval, the system encodes an incoming query into a vector and compares it against stored document vectors with a distance metric (cosine similarity is the most common). The nearest vectors come back as the most semantically relevant results, even when the query wording differs from the document. This is what lets RAG pipelines fetch relevant context without requiring keyword overlap.
What is a vector database and which OSS options exist in 2025–2026?
A vector database stores embeddings alongside metadata and supports approximate-nearest-neighbor (ANN) search at scale. Rather than scanning every vector, it uses index structures — HNSW (Hierarchical Navigable Small World) or IVF (Inverted File Index) — to find the top-k nearest neighbors in milliseconds. Widely used options include: pgvector (Postgres extension, HNSW + IVF, good for teams already on Postgres), Pinecone (managed, serverless), Qdrant (open-source, Rust, HNSW-default), Weaviate (open-source, supports both vector and graph-style cross-references), and Milvus (open-source, GPU-accelerated). For teams that need vector and graph traversal in one engine, Neo4j 5+ has a native vector index alongside Cypher queries.
What is the difference between HNSW and IVF indexing?
HNSW (Hierarchical Navigable Small World) builds a multi-layer graph where each node links to nearby nodes; search navigates from coarse upper layers to fine lower layers. It delivers very low query latency and high recall but uses more memory. The overhead is driven by the graph's link count (the M parameter — how many neighbor links each node stores), added on top of the stored vector itself, so raising M improves recall but costs memory; it is not a flat per-dimension tax. IVF (Inverted File) partitions vectors into clusters (Voronoi cells) and probes only a subset of clusters per query, trading recall for speed and a lower memory footprint. HNSW is a common default for low-latency retrieval; IVF is preferred when memory is constrained or when the dataset size makes HNSW build time prohibitive. Both are approximate — recall@10 of 95–99% is typical for well-tuned parameters.
What is chunking and why does it matter more than most teams expect?
Chunking is splitting source documents into pieces before embedding, and it is often the most-overlooked tuning lever in retrieval. Chunks that are too short (under roughly 100 tokens) lose sentence context and produce fragmented embeddings, so the LLM sees fragments and confabulates. Chunks that are too long (over roughly 2,000 tokens) bury the relevant sentence: the retrieved chunk passes the similarity check, but answer quality drops because the LLM must sift through noise. A common working range is 256–1,024 tokens with 20–50 token overlap between adjacent chunks, but it is a heuristic, not a fixed standard. The right value depends on document structure — dense technical prose tends to want smaller chunks, narrative or legal text larger ones. Measure recall@K on your actual query distribution rather than copying a benchmark default.
When does hybrid search (BM25 + dense embeddings) outperform pure semantic search?
Hybrid search — combining BM25 (lexical) and dense embeddings, often fused via Reciprocal Rank Fusion — outperforms pure semantic search when queries carry exact keywords, product IDs, model numbers, or rare proper nouns. Dense embeddings generalise well to paraphrases but can miss exact-match signals, which BM25 catches; BM25 alone misses synonyms and paraphrases, which embeddings handle. A typical recipe runs BM25 and vector search in parallel, merges the ranked lists (RRF is common, though weighted fusion and learned ranking are also used), and optionally reranks with a cross-encoder for the final top-k. Regulated-domain retrieval (legal, medical, finance) commonly leans on hybrid search because keyword precision matters as much as semantic recall there.
What is a reranker and where does it fit in the retrieval pipeline?
A reranker (cross-encoder) is a second-stage model that reads the query and each candidate document together and produces a refined relevance score. First-stage retrieval (vector ANN or BM25) trades precision for speed by comparing query and document independently — its cost depends on the index, its parameters, any metadata filters, and how many candidates it visits, not a constant per query. The reranker adds precision by scoring the pair jointly: one model forward pass per candidate, so its cost grows with the candidate-set size (O(k)) but is bounded to that set rather than the full corpus. That extra work materially improves how well the top-k separates high-quality from low-quality matches. Common choices include Cohere Rerank, BGE-Reranker, or cross-encoder models from Hugging Face. A typical pipeline retrieves the top-100 via hybrid search, then reranks to the top-5 before passing them to the LLM.
How does vector retrieval compare to knowledge-graph retrieval and when should you use each?
Vector retrieval finds documents by semantic similarity — it answers "which passages are about X?" quickly and cheaply. Knowledge-graph (KG) retrieval traverses structured entity-relation edges — it answers "how does X relate to Y through Z?" and "what are the prerequisites for X?", questions that require connecting facts across multiple sources. The typical failure mode of flat vector retrieval is multi-hop queries: "which engineers worked on the auth system AND shipped to production last quarter?" chains two relations that rarely appear together in one passage, so co-occurring text alone usually cannot satisfy it. A common answer is hybrid (vector + graph): vector search finds entry-point nodes by semantic similarity, then graph traversal expands those nodes along relations. Use pure vector retrieval for open-ended "find a passage about X" queries, and escalate to a KG once multi-hop or cross-document relationship questions become a measurable, recurring fraction of traffic — the threshold is workload-specific, so set it from your own query mix rather than a fixed percentage.
What is cosine similarity and when should you use dot-product or L2 distance instead?
Cosine similarity measures the angle between two vectors, ignoring their magnitude. It is the usual choice when embeddings come from models trained with a cosine objective, which covers many sentence-transformer and text-embedding APIs. Dot-product is equivalent to cosine when vectors are L2-normalised — many embedding APIs return normalised vectors, but confirm in the provider's docs rather than assuming. L2 (Euclidean) distance accounts for magnitude and fits models trained with an L2 objective, such as some image embeddings or certain fine-tuned models. In practice, use cosine or dot-product on normalised vectors for text retrieval; on normalised vectors the choice between them rarely shifts recall@K by more than a point or two.
What is pgvector and what trade-offs does it make versus a purpose-built vector database?
pgvector is a Postgres extension that adds vector storage and ANN search (HNSW and IVF) to an existing Postgres instance. It supports halfvec (half-precision storage, roughly halving memory versus float32) and HNSW indexes for million-doc-scale retrieval. The trade-off: pgvector keeps vectors inside your existing SQL stack — familiar ops, ACID transactions, Postgres row-level security, and no extra service to manage. Purpose-built databases (Pinecone, Qdrant, Weaviate, Milvus) offer higher write throughput, more index-tuning knobs, and dedicated vector query paths that matter at high query rates or very large corpora. Where the line falls is a sizing decision, not a fixed cutoff: hardware, vector dimension, metadata-filter selectivity, write rate, recall target, and latency SLO together determine whether pgvector still holds up. At moderate scale and query rates it often performs comparably to dedicated options while keeping operational complexity low — benchmark on your own workload before committing.
What is embedding drift and how should teams detect it in production?
Embedding drift occurs when the meaning of terms in your corpus shifts over time and the stored vectors no longer accurately represent current document semantics. This is distinct from two other cases: the underlying content changing (which requires re-embedding the new documents) and swapping or upgrading the embedding model (which requires a full re-index). Detection: track retrieval quality metrics — recall@K, mean reciprocal rank, or user-signal proxies like click-through-rate on retrieved results — over time. A sustained drop that is not explained by query-distribution shift is a signal that embeddings are stale relative to corpus evolution. The safest production pattern: store the embedding-model version as metadata on each vector; trigger a re-embedding job when the model is upgraded or when retrieval quality drops below threshold.

Related topics

Essential AI-Native Skills for Vector Databases and Embeddings

Modern engineering work increasingly uses AI tools for design and code review, debugging, documentation, test and testbench generation, and workflow automation. The goal is not to let AI replace engineering judgment — it is to move faster while keeping verification discipline.

  • Use AI to explain unfamiliar code, logs, waveforms, datasheets, or test failures.
  • Break large problems into small, reviewable steps you can verify independently.
  • Ask AI for hypotheses, then validate them against tests, measurements, simulations, or lab data.
  • Version-control your analysis scripts, testbenches, and configs — keep changes small and reviewable.
  • Document your assumptions, design tradeoffs, and debugging decisions.
  • Verify AI output before trusting it: run the checks that fit the domain — unit tests, linters, simulations, or bench/lab measurements.
  • Review AI output for correctness, edge cases, and real-world consequences.

Next up: Vector Databases and Embeddings practice

The adaptive practice engine is already live for core wireless, RF, and ML domains. Vector Databases and Embeddings questions — covering production realities, not just framework syntax — are in active development. Join the early-access list to get them first.

One email when this topic launches. Nothing else. Unsubscribe in one click.