glucolte

AI Knowledge Bases and Retrieval

RAG stands for retrieval-augmented generation: retrieve relevant information, augment the model context, then generate an answer. It can supply current, specific evidence when the sources and retrieval pipeline are current and reliable.

Part 4 of 7: Models and providers → Knowledge bases and retrieval → AI Agents. This page owns the quality of evidence selection and RAG answers; agent-loop and platform-release metrics come later.

QUICK INTERACTIVE PRACTICE Find the first broken RAG boundary

Choose an answer get feedback inspect the flow.

Identify the decisive requirement, choose the smallest useful change, and explain why the nearest alternative fails. Use the worked examples and references below to check your reasoning.

Enable JavaScript to try the scenarios and inspect each decision boundary.

Use Northstar Device Care as the running example. Northstar has a knowledge base containing millions of policy chunks covering products, countries, warranty versions, and repair rules. We will repeatedly ask: “Can I replace my worn-out iPhone battery in Australia?” One policy might be about Apple battery service, but the important idea is the scale: the application must find a small, permitted shortlist before an expensive relevance judge and the LLM see anything. The application—not the LLM—finds permitted evidence and supplies it to the LLM.

Building the Searchable Knowledge Base

Before a question can retrieve anything, source material needs to become searchable. This is one continuous pipeline:

A knowledge base is primarily useful for unstructured or semi-structured evidence such as documents, notes, manuals, and transcripts. For authoritative rows, amounts, dates, and transactions, query the governed database/API directly; use retrieval and generation only where interpretation of text is needed.

For the vector-based implementation in this walkthrough: Raw file → Parse → Structure → Chunk → Embed → Store / Index. A lexical-only index does not require embedding.

Parsing

Parsing converts a source format into a representation the application can reliably process. For the fictional Apple PDF, that means preserving the document, title, heading hierarchy, sections, paragraphs and tables, plus source, version, and ACL/security metadata.

The useful output is not merely a wall of text. It can retain a path such as Battery Service > Australia > Eligibility, alongside product=iPhone, region=AU, and status=current.

Chunking

A full document is usually too large and too broad to retrieve as one unit. Chunking makes small evidence units that can be found, cited, and fit into a model’s context. The boundary matters: splitting a sentence can separate a condition from its qualification.

Common strategies are deliberately different tools:

Failure in a mock question Better choice
A precise fact such as a torque value is buried in a broad chunk Use smaller, context-focused chunks and preserve the heading/unit; test whether the fact enters retrieved candidates.
A clause is cut at an arbitrary token or sentence count Use structural boundaries (headings/paragraphs) or semantic topic changes. Overlap helps but does not guarantee a complete idea.
A narrative article changes topic without clear headings Evaluate semantic chunking; variable-length chunks can follow topic boundaries, but inspect resulting coherence and retrieval recall.
A retrieved child lacks the full clause Parent-child retrieval can supply the parent at answer time; it does not repair poor boundaries in the indexed child.

Section-aware chunking is often a strong default for policies because the heading path becomes both context and retrieval metadata. It is not automatically best: tables, transcripts, and long narrative text may need another approach.

Implementation boundary: parent substitution is a retriever feature, not a universal property of a hierarchical document. Bedrock vector retrieval and Neptune GraphRAG differ here. A lost table layout needs a parser change before any chunking adjustment. See Knowledge Bases configuration layers.

Embeddings: representing meaning as vectors

During ingestion, an embedding model converts each prepared chunk’s text to numbers. An embedding is a numerical representation useful for comparing semantic similarity—not a human-readable list of labels.

At query time, use a compatible model and its prescribed query/document encoding settings. Search compares vectors using a supported metric such as cosine similarity, dot product, or Euclidean distance. Match the metric and normalization to the model: these scores are not generally interchangeable. With unit-normalized vectors, dot product equals cosine similarity and squared Euclidean distance produces equivalent rankings. Faiss metric guidance.

Changing the embedding space

A new visual style or vocabulary can reveal a domain coverage gap. First separate representation failure from missing ingestion, poor captions, incorrect filters, and low first-stage recall. Hybrid metadata matching can recover tagged items without changing the model. If the embedding model itself must adapt, use a supported customization path and evaluate held-out image–text or document–query pairs.

New embedding version or dimension setting
                  ↓
Re-embed corpus → separate compatible index → evaluate recall + answers
                  ↑                                  ↓
        compatible query encoder            switch both together
                                            retain rollback pair

Do not mix unrelated model versions or dimensions in one similarity space. Re-embedding only new documents leaves the old catalogue incompatible unless the system explicitly maintains separate indexes and routing. Lower dimensions reduce raw vector bytes but can reduce retrieval quality; corpus size alone does not determine the right dimension. Titan customization and dimension controls.

How Can We Retrieve Relevant Information?

Retrieval is the act of finding evidence. Vector search is only one possible method.

Method Best At Example
Metadata / structured filter Known attributes region=AU AND product=iPhone
Lexical / BM25 Exact words, IDs, errors POL-BAT-AU-2026
Vector / semantic Meaning despite different wording “worn-out phone battery”
Hybrid Exact + semantic requirements policy ID + natural-language question

Metadata filtering

If the application already knows structured attributes, filtering can dramatically reduce the search space:

region = AU
product = iPhone
status = current

These example fields are relevance filters, not permission. region=AU may make a policy relevant. Authorization must instead use trusted caller identity and document ACLs or equivalent policy. A caller-supplied user_has_access=true flag is not proof of permission. Enforce access before evidence reaches a model, external reranker, or user-visible result.

Lexical / BM25

Lexical search is best when the actual word matters: policy IDs, error codes, product names, exact phrases, and acronyms. Searching for POL-BAT-AU-2026 should strongly favour that literal ID. Vector similarity may not be the best tool for this; lexical search is excellent.

BM25 ranks indexed terms; it does not by itself guarantee an exact identifier or phrase match. Tokenization/analyzers can split IDs and punctuation. Use a keyword field with an exact term filter for an identifier, or an appropriate phrase query for ordered text. OpenSearch exact-term queries.

Vector / semantic search

Vector search is best when meaning matters. A user might ask, “When will Apple service my worn-out battery?” while the policy says, “An iPhone battery qualifies for service when tested capacity is below 80%.” The words differ, but the intent is related. This is where embeddings help.

Hybrid retrieval

Real systems often combine signals instead of betting on one technique. Metadata can constrain either or both searches; BM25 and vector search can each produce candidates; then a fusion method combines their ranks.

BM25 candidates ────────┐
                        ├── Fuse / RRF ──→ Candidates
Vector candidates ──────┘

Reciprocal Rank Fusion (RRF) is a simple option: a result earns more credit when it ranks highly in either list, without requiring the BM25 and vector score scales to match. Hybrid retrieval can also combine normalized scores; RRF is one fusion method. OpenSearch rank fusion.

Query preparation before retrieval

Query problem Transformation Check
Colloquial term versus clinical vocabulary, such as “arthritis” versus “osteoarthritis” Expand with validated synonyms or related terms. Keep original wording and evaluate false positives.
One question asks about side effects and interactions Decompose into atomic retrieval queries, then combine authorized evidence. Verify both sub-answers and citations; Bedrock Knowledge Bases also supports query decomposition where configured.
SKU, error code, or proprietary part name Detect and preserve the exact identifier; use a keyword field, phrase query, or metadata filter alongside semantic search. Do not assume embeddings distinguish near-identical codes.
Domain-specific named parts appear in documents but are not reliably extracted Consider Amazon Comprehend Custom Entity Recognition; train with annotated examples or a plaintext entity list, then verify extracted entities before indexing/filtering. A recognizer is probabilistic; it does not guarantee exact matches. Use exact fields for hard constraints.

Knowledge Base query decomposition · Comprehend custom entities.

How Do We Search Millions of Vectors?

Now that vector search has a job, we can ask how it scales. Start with a brute-force exact nearest-neighbour baseline, which compares the query with every stored vector and selects the top K:

ANN means Approximate Nearest Neighbour. Methods such as HNSW and IVF reduce search work in exchange for possibly missing exact nearest neighbours. The speed/recall tradeoff depends on data and settings; it is not guaranteed to be small. Here, ANN recall means recovering neighbours from an exact-search baseline, which differs from retrieving documents a human judges relevant. Exact vector search is exact about the chosen metric, not about factual or semantic correctness. Faiss index comparison.

HNSW: Navigate Through Neighbours

HNSW stands for Hierarchical Navigable Small World. Its mental model is: navigate through vector space toward increasingly similar neighbours. It is graph-based: upper layers make larger jumps, and lower layers refine the route.

Tune the stage that consumes resources: ef_search changes query exploration; ef_construction changes graph-building effort; m changes connectivity and graph memory. More search effort can improve recall but adds latency. Shard distribution and available memory must support the index; adding shards without adding capacity can increase overhead. OpenSearch HNSW controls.

IVF: Find the Right Neighbourhood

IVF means Inverted File Index. Its mental model is: find the right neighbourhood first, then search the houses. It divides vector space into partitions (clusters), then searches the most promising ones.

Searching more partitions (often called probes) generally improves recall, but costs more work. It is a knob, not a guarantee.

Neither index is universally better. Choose based on corpus size, latency and recall targets, update behaviour, memory budget, and the retrieval system around it.

IVF commonly needs representative data to train its partition centroids; HNSW builds a neighbour graph. Query-time search effort (nprobe for IVF, often efSearch for HNSW) is separate from index-construction settings. Faiss index details.

Reconnecting the Query-Time Pipeline

This is the retrieval layer’s final mental model. The LLM is downstream of retrieval; it does not independently search HNSW, IVF, or the vector store.

For the battery policy, use region=AU, product=iPhone, and status=current to narrow results; verify the caller may access the policy; fuse lexical and semantic candidates; then send the best permitted excerpts, section paths, and citation IDs to the LLM.

Worked example: searching millions of Northstar policies

Imagine Northstar stores 20 million policy chunks. A customer asks the battery question. We do not send all 20 million chunks to a reranker or to Bedrock. Each stage has a different job:

Northstar: find broadly, judge narrowly
Question“Can I replace my worn-out iPhone battery in Australia?”
Scope + accessregion=AU · product=iPhone · current policy · caller ACL
Fast retrievalBM25 + vector ANN + metadata
20M → Top 100 candidates
RerankerReads question + each candidate together
Top 100 → Top 8
Bedrock answerOnly the best permitted chunks enter context
Answer with citations
Retrieval optimizes recall and speed; reranking spends more computation on a small shortlist; generation receives only the final evidence.
Stage What Northstar does Why it exists Typical trade-off
1. Scope and authorization Applies trusted filters such as country, product, version, and caller permissions Prevents irrelevant or forbidden chunks from becoming candidates Narrowing too aggressively can remove the correct policy; authorization must come from trusted identity/ACL data
2. First-stage retrieval Runs metadata filters, lexical/BM25 search for exact policy IDs and terms, vector ANN for meaning, or hybrid search Searches millions of chunks quickly and protects recall It is fast, but its ranking is approximate and may include noisy candidates
3. Candidate Top-K Keeps, for example, the best 100 candidates after fusion Makes the expensive next step affordable K=5 may be too small and lose evidence; K=1000 raises latency and cost
4. Reranking Sends the question and each of the 100 candidates to a more expensive reranker Judges the relationship between the exact question and each passage more deeply Better relevance per candidate, but more compute, latency, and cost
5. Context pruning Keeps the best 5–8 permitted chunks within the token budget Prevents irrelevant text from reaching the generator A threshold must be calibrated; reranker scores are not universal probabilities
6. Generation Sends the final excerpts, metadata, and citation IDs to Bedrock Produces a conversational answer grounded in selected evidence The LLM cannot recover a policy that retrieval failed to find

The key distinction: first-stage retrieval is the fast librarian scanning the catalogue; reranking is the senior librarian reading the shortlisted books side by side. We do not ask the senior librarian to read all 20 million books because that would be too slow and expensive. We also do not ask the fast catalogue search to make the final judgment because it is optimized for recall and speed, not deep passage-level relevance.

If the correct battery policy never appears in the Top 100, the reranker cannot recover it. If it appears at rank 73, reranking may move it into the final Top 8. If it is retrieved but the user is not authorized, it must be excluded before the passage reaches the reranker, model context, or answer.

Choose and maintain the vector store

Requirement or symptom Choice and boundary
Transactions need atomic writes of business data, embeddings, and metadata, plus SQL filters and vector ranking Aurora PostgreSQL with pgvector can keep them in one database transaction and query with WHERE plus vector-distance ORDER BY. A foreign key only checks a reference; it does not make separate writes atomic. Validate query plans and vector-index/filter behavior at scale. Aurora pgvector
Infrequent vector queries, little operations work, no continuously provisioned vector cluster S3 Vectors offers managed vector buckets, indexes, metadata filters, and usage-based storage/request charges. It can reduce idle compute spend, but storage and requests still cost money; benchmark latency, features, and total bill for the workload. S3 Vectors
OpenSearch index has many deleted documents after frequent updates Lucene marks deletions until segments merge. A force merge can expunge them, but is resource-intensive; schedule it for suitable quiet/read-only indexes and measure the effect. EventBridge/Lambda can automate a measured maintenance trigger. ISM rollover addresses time-series index lifecycle, not this deletion problem. Force Merge API · ISM
Tens of millions of vectors make full-scan exact k-NN too slow Benchmark HNSW or another ANN index against exact-search recall and latency. ANN trades possible recall for speed; do not assume a fixed O(log N) bound or guaranteed millisecond latency. Pre-filtering helps only when a selective, valid filter exists. OpenSearch HNSW
OpenSearch must generate embeddings at ingest and query time A model connector plus ingest text_embedding processor and neural query can avoid custom middleware. Hybrid BM25/vector score fusion still needs a search pipeline (for example normalization-processor); the ingest pipeline does not blend query scores. Text embedding processor · Hybrid search

Reranking: Find Broadly, Judge Narrowly

First-stage vector/BM25 retrieval is a fast candidate finder. A reranker is a more expensive relevance judge that takes a question and a small candidate set, then reorders it.

A typical dense-retrieval system uses a bi-encoder to encode queries and chunks independently, allowing document vectors to be precomputed. A cross-encoder reranker reads a query and candidate together; this is usually too expensive for a large corpus, so it scores a shortlist. Other rerankers exist, and improved relevance must be measured. Sentence Transformers retrieval and reranking.

Dynamic context pruning keeps a broad candidate search, then drops low-relevance candidates after reranking and before prompt construction. Calibrate a score threshold and token budget against recall and answer quality; reranker scores are not universal probabilities. Summarizing every irrelevant chunk preserves noise in compressed form, while a recency-only window can discard an older but decisive source. AWS reranking integration.

Evaluation: Does the System Retrieve and Answer Well?

Evaluate retrieval and generation separately. A polished answer may still be based on poor evidence.

Precision@K and Recall@K are a good starting pair. Suppose the corpus contains four relevant battery-policy chunks, and the top five returned are:

1. Relevant ✓
2. Relevant ✓
3. Irrelevant ✗
4. Relevant ✓
5. Irrelevant ✗

Precision@5 = 3 / 5 = 60% — “Was the stuff I found actually relevant?”

Recall@5 = 3 / 4 = 75% — “Did I find the relevant stuff?”

For the underlying TP/FP/FN definitions, F1, and why accuracy can mislead on imbalanced data, see classification metrics. Here, K is the number of retrieved results; it is unrelated to the model’s next-token top-k sampling control. AWS-specific query and click trends are covered under Kendra Search Analytics.

MRR rewards placing the first useful result early; NDCG also accounts for graded relevance and ranking position. Use retrieval and answer metrics together, but never substitute one for the other:

Layer Measure What it reveals
Retrieval Precision@K, Recall@K, MRR, NDCG Whether permitted, relevant evidence was found and ranked high enough.
Grounded answer faithfulness/groundedness, citation entailment, source validity Whether each answer claim is supported by the supplied evidence and valid citation.
Task answer correctness, relevance, completeness, refusal/uncertainty, schema validity Whether the user received an appropriate answer, including a safe no-answer when evidence is absent.

Agent task/tool metrics belong on AI Agents. Release gates, human review, judge calibration, latency, and cost monitoring belong on AI Infrastructure and Evaluation.

Check your understanding

Continue to AI Agents for using evidence during tool execution. For AWS implementations, see AWS AI Services.

×
Contents