glucolte

AI Knowledge Bases and Retrieval

RAG stands for retrieval-augmented generation: retrieve relevant information, augment the model context, then generate an answer. It can supply current, specific evidence when the sources and retrieval pipeline are current and reliable.

Part 4 of 7: Models and providers → Knowledge bases and retrieval → AI Agents. This page owns the quality of evidence selection and RAG answers; agent-loop and platform-release metrics come later.

Use a fictional Apple battery-service policy for the running example. Ingestion prepares that policy when it changes. The query path runs each time someone asks, “Can I replace my worn-out iPhone battery in Australia?” The application—not the LLM—finds permitted evidence and supplies it to the LLM.

Building the Searchable Knowledge Base

Before a question can retrieve anything, source material needs to become searchable. This is one continuous pipeline:

For the vector-based implementation in this walkthrough: Raw file → Parse → Structure → Chunk → Embed → Store / Index. A lexical-only index does not require embedding.

Parsing

Parsing converts a source format into a representation the application can reliably process. For the fictional Apple PDF, that means preserving the document, title, heading hierarchy, sections, paragraphs and tables, plus source, version, and ACL/security metadata.

The useful output is not merely a wall of text. It can retain a path such as Battery Service > Australia > Eligibility, alongside product=iPhone, region=AU, and status=current.

Chunking

A full document is usually too large and too broad to retrieve as one unit. Chunking makes small evidence units that can be found, cited, and fit into a model’s context. The boundary matters: splitting a sentence can separate a condition from its qualification.

Common strategies are deliberately different tools:

Section-aware chunking is often a strong default for policies because the heading path becomes both context and retrieval metadata. It is not automatically best: tables, transcripts, and long narrative text may need another approach.

Embeddings: representing meaning as vectors

During ingestion, an embedding model converts each prepared chunk’s text to numbers. An embedding is a numerical representation useful for comparing semantic similarity—not a human-readable list of labels.

At query time, use a compatible model and its prescribed query/document encoding settings. Search compares vectors using a supported metric such as cosine similarity, dot product, or Euclidean distance. Match the metric and normalization to the model: these scores are not generally interchangeable. With unit-normalized vectors, dot product equals cosine similarity and squared Euclidean distance produces equivalent rankings. Faiss metric guidance.

How Can We Retrieve Relevant Information?

Retrieval is the act of finding evidence. Vector search is only one possible method.

Method Best At Example
Metadata / structured filter Known attributes region=AU AND product=iPhone
Lexical / BM25 Exact words, IDs, errors POL-BAT-AU-2026
Vector / semantic Meaning despite different wording “worn-out phone battery”
Hybrid Exact + semantic requirements policy ID + natural-language question

Metadata filtering

If the application already knows structured attributes, filtering can dramatically reduce the search space:

region = AU
product = iPhone
status = current

These example fields are relevance filters, not permission. region=AU may make a policy relevant. Authorization must instead use trusted caller identity and document ACLs or equivalent policy. A caller-supplied user_has_access=true flag is not proof of permission. Enforce access before evidence reaches a model, external reranker, or user-visible result.

Lexical / BM25

Lexical search is best when the actual word matters: policy IDs, error codes, product names, exact phrases, and acronyms. Searching for POL-BAT-AU-2026 should strongly favour that literal ID. Vector similarity may not be the best tool for this; lexical search is excellent.

BM25 ranks indexed terms; it does not by itself guarantee an exact identifier or phrase match. Tokenization/analyzers can split IDs and punctuation. Use a keyword field with an exact term filter for an identifier, or an appropriate phrase query for ordered text. OpenSearch exact-term queries.

Vector / semantic search

Vector search is best when meaning matters. A user might ask, “When will Apple service my worn-out battery?” while the policy says, “An iPhone battery qualifies for service when tested capacity is below 80%.” The words differ, but the intent is related. This is where embeddings help.

Hybrid retrieval

Real systems often combine signals instead of betting on one technique. Metadata can constrain either or both searches; BM25 and vector search can each produce candidates; then a fusion method combines their ranks.

BM25 candidates ────────┐
                        ├── Fuse / RRF ──→ Candidates
Vector candidates ──────┘

Reciprocal Rank Fusion (RRF) is a simple option: a result earns more credit when it ranks highly in either list, without requiring the BM25 and vector score scales to match. Hybrid retrieval can also combine normalized scores; RRF is one fusion method. OpenSearch rank fusion.

How Do We Search Millions of Vectors?

Now that vector search has a job, we can ask how it scales. Start with a brute-force exact nearest-neighbour baseline, which compares the query with every stored vector and selects the top K:

ANN means Approximate Nearest Neighbour. Methods such as HNSW and IVF reduce search work in exchange for possibly missing exact nearest neighbours. The speed/recall tradeoff depends on data and settings; it is not guaranteed to be small. Here, ANN recall means recovering neighbours from an exact-search baseline, which differs from retrieving documents a human judges relevant. Exact vector search is exact about the chosen metric, not about factual or semantic correctness. Faiss index comparison.

HNSW: Navigate Through Neighbours

HNSW stands for Hierarchical Navigable Small World. Its mental model is: navigate through vector space toward increasingly similar neighbours. It is graph-based: upper layers make larger jumps, and lower layers refine the route.

IVF: Find the Right Neighbourhood

IVF means Inverted File Index. Its mental model is: find the right neighbourhood first, then search the houses. It divides vector space into partitions (clusters), then searches the most promising ones.

Searching more partitions (often called probes) generally improves recall, but costs more work. It is a knob, not a guarantee.

Neither index is universally better. Choose based on corpus size, latency and recall targets, update behaviour, memory budget, and the retrieval system around it.

IVF commonly needs representative data to train its partition centroids; HNSW builds a neighbour graph. Query-time search effort (nprobe for IVF, often efSearch for HNSW) is separate from index-construction settings. Faiss index details.

Reconnecting the Query-Time Pipeline

This is the retrieval layer’s final mental model. The LLM is downstream of retrieval; it does not independently search HNSW, IVF, or the vector store.

For the battery policy, use region=AU, product=iPhone, and status=current to narrow results; verify the caller may access the policy; fuse lexical and semantic candidates; then send the best permitted excerpts, section paths, and citation IDs to the LLM.

Reranking: Find Broadly, Judge Narrowly

First-stage vector/BM25 retrieval is a fast candidate finder. A reranker is a more expensive relevance judge that takes a question and a small candidate set, then reorders it.

A typical dense-retrieval system uses a bi-encoder to encode queries and chunks independently, allowing document vectors to be precomputed. A cross-encoder reranker reads a query and candidate together; this is usually too expensive for a large corpus, so it scores a shortlist. Other rerankers exist, and improved relevance must be measured. Sentence Transformers retrieval and reranking.

Evaluation: Does the System Retrieve and Answer Well?

Evaluate retrieval and generation separately. A polished answer may still be based on poor evidence.

Precision@K and Recall@K are a good starting pair. Suppose the corpus contains four relevant battery-policy chunks, and the top five returned are:

1. Relevant ✓
2. Relevant ✓
3. Irrelevant ✗
4. Relevant ✓
5. Irrelevant ✗

Precision@5 = 3 / 5 = 60% — “Was the stuff I found actually relevant?”

Recall@5 = 3 / 4 = 75% — “Did I find the relevant stuff?”

For the underlying TP/FP/FN definitions, F1, and why accuracy can mislead on imbalanced data, see classification metrics. Here, K is the number of retrieved results; it is unrelated to the model’s next-token top-k sampling control. AWS-specific query and click trends are covered under Kendra Search Analytics.

MRR rewards placing the first useful result early; NDCG also accounts for graded relevance and ranking position. Use retrieval and answer metrics together, but never substitute one for the other:

Layer Measure What it reveals
Retrieval Precision@K, Recall@K, MRR, NDCG Whether permitted, relevant evidence was found and ranked high enough.
Grounded answer faithfulness/groundedness, citation entailment, source validity Whether each answer claim is supported by the supplied evidence and valid citation.
Task answer correctness, relevance, completeness, refusal/uncertainty, schema validity Whether the user received an appropriate answer, including a safe no-answer when evidence is absent.

Agent task/tool metrics belong on AI Agents. Release gates, human review, judge calibration, latency, and cost monitoring belong on AI Infrastructure and Evaluation.

Check your understanding

Continue to AI Agents for using evidence during tool execution. For AWS implementations, see AWS AI Services.

×
Contents