Build a Vector Index

Goal

Build a deterministic in-memory vector index, return top-k chunk IDs with scores, and preserve the mapping from vector rows back to source metadata.

A vector index stores retrieval vectors and enough identity information to return the corresponding chunks. For a tiny index:

row 0 → chunk A → [0.9, 0.1]
row 1 → chunk B → [0.2, 0.8]
row 2 → chunk C → [0.7, 0.3]

A query vector is compared with each stored vector, the scores are sorted, and the highest-scoring chunk IDs are returned. Large systems use specialized indexes so they do not scan every vector. The small brute-force version is still the right place to learn the contract.

Index rows are not source identities

Suppose row 1 is chunk B today. After rebuilding the index, row ordering may change. If evaluation records only:

retrieved row = 1

the result is ambiguous later. Store stable chunk IDs beside vectors:

chunk_id
source_id
vector
metadata

The row is an implementation detail. The chunk ID is part of the evidence identity.

Top-k is a candidate budget

If k=3, the retriever returns three candidate chunks. Increasing k may improve recall because more evidence is included. It can also add noise and prompt cost. A larger k is not always better. For one query:

k=1 → exact relevant chunk
k=5 → exact relevant chunk + 4 distractors

If downstream generation is sensitive to distractors, adding candidates can hurt. This is why retrieval evaluation should inspect both recall and the later answer behavior.

Tie-breaking should be deterministic

Two chunks can receive the same score. A stable implementation should define how ties are ordered, for example:

sort by descending score
then by chunk_id

Deterministic tie-breaking makes tests reproducible. Without it, two runs can return the same set in a different order and create confusing evaluation diffs.

Index build and query configuration belong together

Record:

• embedding model/version;
• normalization policy;
• similarity function;
• chunking version;
• index build version;
• top-k policy.

If the query code changes similarity without rebuilding compatible index data, results can become meaningless.

Exact search is a useful reference

Approximate nearest-neighbor systems trade some exactness for speed. Before tuning an approximate index, a small exact/brute-force reference can answer:

Is the desired neighbor actually nearest under our embeddings and score?

That separates representation problems from indexing-approximation problems.

An index is a versioned derived artifact

The source documents are not the index. The index is produced from:

source snapshot
+ chunker version
+ embedding model/version
+ normalization policy
+ index configuration

If any of those change, the derived artifact may need rebuilding. Keep that lineage visible so an answer can be traced back to the source representation that retrieval actually searched.

Updates and deletions need explicit behavior

Suppose a source paragraph is deleted but its old vector remains searchable. The RAG system can still retrieve information that no longer exists in the source of truth. Likewise, updating a document without replacing its old chunks can create contradictory versions in the same index. A production index therefore needs rules for insert, update, delete, and rebuild—not only nearest-neighbor lookup.

An index is a reproducible snapshot of representations

A vector index is not just a container of numbers. Each vector should remain connected to a stable chunk ID, source ID, embedding-model version, and corpus snapshot. If the document text changes but the old vector remains in the index, retrieval is now operating on stale evidence even though search still returns plausible scores.

Small exact search is ideal for learning because every score can be recomputed directly. Larger systems may use approximate nearest-neighbor methods that trade some exactness for speed and memory. That optimization should not erase the reference point: keep a small exact benchmark or known-query set so you can detect when indexing or search settings reduce retrieval quality beyond an acceptable threshold.

Predict

Why should retrieval results return stable chunk IDs rather than only vector-row numbers?

Chunk IDs preserve evidence identity across index rebuildsVector rows are always identical to source document IDsStable IDs make similarity scores unnecessary

Build the index in the Lab

The index holds four chunk vectors. Notice that chunk-c and chunk-d have the same vector, so they will always tie.

1. Click Run once. top2: [] is empty and the checks fail.
2. Complete the TODO in search: score every chunk with dot, sort by score from high to low, break ties by chunk ID in alphabetical order, and return the first k results as (chunk_id, score) pairs.
3. Click Run again. You should see top2: [('chunk-a', 0.9), ('chunk-c', 0.7)].
4. Change search(index, query, 2) to search(index, query, 3). Before running, predict which chunk appears third.
5. Click Run. chunk-d appears third with the same score 0.7. Without a tie rule, chunk-c and chunk-d could swap places from run to run; with the ID tie-break, the order is always the same.

Loading lab…

Quick Check

1. What is top-k in retrieval?

The number of source documents before chunkingThe number of candidate results returned after rankingThe number of embedding dimensions kept after scoring

2. Why can increasing k hurt a RAG answer?

Similarity scores become less accurate for later ranksMore distractor context can compete with the relevant evidenceIncreasing k improves answer quality monotonically because more evidence cannot hurt a grounded generator

3. Why keep a brute-force exact search for small tests?

It removes the need to store source metadataExact search is useful because it reproduces production latency while avoiding approximate-index configurationIt helps distinguish embedding/score problems from approximate-index behavior

0 of 3 questions answered.

Explain it back

Describe the fields you would store for one vector-index row. Then explain how top-k and deterministic tie-breaking affect reproducibility.

Key Takeaways

• A vector index maps retrieval vectors back to stable chunk/source identities.
• Top-k is a candidate budget, not a quality guarantee.
• More candidates can increase both recall and noise.
• Deterministic tie-breaking improves reproducibility.
• Exact search is a valuable reference before approximate indexing.

Next Lesson

Next, add lexical matching so exact names and identifiers can complement semantic retrieval.

References

• Johnson et al., Billion-scale similarity search with GPUs.