Your RAG system did not fail because the model hallucinated.

It failed because the only “facts” the model saw were a mangled PDF table, an outdated policy, a chunk with missing context, and three near-duplicate paragraphs that pushed better evidence out of the top-k results.

The LLM was downstream of a retrieval pipeline that had already distorted reality.

This is the part of RAG that is easy to miss. Teams spend a lot of time choosing models, tuning prompts, and debating context windows. But in production, a shocking number of failures happen earlier: during ingestion, chunking, indexing, filtering, ranking, and query transformation. By the time the LLM receives the retrieved context, the answer may already be impossible.

The retrieval pipeline is not a neutral search layer. It decides what the model is allowed to know.

TL;DR

• Most RAG failures are not prompt failures; they are retrieval-pipeline failures.
• Ingestion and parsing can destroy the meaning before embedding begins.
• Chunking is context surgery, not text splitting.
• Vector similarity is not the same as evidence relevance.
• Metadata, permissions, versioning, and time are where production trust lives.
• If you only evaluate final answers, you cannot diagnose retrieval failures.

📋 Table of Contents

• The LLM never sees reality
• 1. Your source of truth is already damaged at ingestion
• 2. Chunking is context surgery not text splitting
• 3. Generic embeddings flatten your domain
• 4. Similarity is not evidence
• 5. Metadata is where trust and permissions live
• 6. Old versions quietly poison the index
• 7. The user query is not the real question
• 8. Duplicates make retrieval confidently wrong
• 9. If you only evaluate final answers you are blind
• A retrieval contract for production RAG

The LLM never sees reality

A retrieval-augmented generation system gives the model a narrow slice of the world: the retrieved chunks.

If that slice is incomplete, outdated, duplicated, unauthorized, or structurally broken, the model cannot reason its way out of that problem. It can only work with what it was given.

That is why the phrase “the retrieval pipeline is lying” is not just dramatic wording. The pipeline creates the model’s operating reality.

A bad prompt can usually be fixed with a better prompt.

A bad retrieval pipeline is worse. It silently supplies bad evidence.

This matters even more now because many teams are moving toward hybrid retrieval, rerankers, metadata filtering, and agentic query planning. Those techniques improve things, but they also add more places where the system can fail before generation starts.

The rest of this article walks through the failure modes I would check first when a RAG system produces wrong answers even though “the document is in the database.”

1. Your source of truth is already damaged at ingestion

Scenario:

Your knowledge base contains the correct answer in a table inside a PDF. The user asks a question that should be easy to answer. The model returns nonsense because the retrieved chunk looks like this:

Plan A 10 20 50 Plan B 15 25 75 Monthly Annual

The table was technically retrieved. But its meaning was destroyed during extraction.

Why it matters:

A lot of RAG teams treat ingestion as a boring preprocessing step: extract text, chunk it, embed it, move on. That is a mistake. If the extraction layer loses structure, the embedding layer embeds garbage, and the generation layer receives garbage with confidence.

Documents are not just bags of words. They contain:

• headings,
• lists,
• tables,
• code blocks,
• captions,
• footnotes,
• page context,
• and document hierarchy.

When parsing flattens all of that into one-dimensional text, the model loses the relationships that make the content meaningful.

Solution:

Make ingestion document-type aware. Preserve structure where it matters, and convert tables into a representation the model can actually read.

A practical ingestion layer should produce structured blocks, not just raw text.

from dataclasses import dataclass

@dataclass
class ParsedBlock:
    block_id: str
    doc_id: str
    kind: str  # "text", "table", "code", "heading"
    section_path: list[str]
    content: str
    source_location: str

For tables, do not dump cells as a single line. Convert them into Markdown, CSV, or a compact JSON structure.

def table_to_markdown(headers: list[str], rows: list[list[str]]) -> str:
    header_line = "| " + " | ".join(headers) + " |"
    separator = "| " + " | ".join(["---"] * len(headers)) + " |"
    body = [
        "| " + " | ".join(row) + " |"
        for row in rows
    ]
    return "\n".join([header_line, separator, *body])

Then store both the searchable text and the structured representation.

Why this works:

The LLM is much better at reading preserved structure than reconstructing it from flattened text. A Markdown table gives the model row and column relationships. A flattened string does not.

🚨 Production warning:

If your corpus contains PDFs, scanned images, slides, or HTML with heavy navigation boilerplate, parsing is not a solved problem. It is one of the highest-impact parts of your RAG system.

2. Chunking is context surgery not text splitting

Scenario:

The retrieved chunk says:

“The limit is 50 per workspace. Exceeding it triggers a soft stop.”

The user asked: “What is the API rate limit for Enterprise?”

The chunk is from the right document. But it does not say what “the limit” refers to, because the previous chunk contained the subject.

Why it matters:

Naive chunking breaks references.

If you split text by fixed token counts, you will routinely cut between:

• a heading and its content,
• a question and its answer,
• a definition and its usage,
• a list and its introduction,
• a pronoun and its antecedent.

The retriever may find the chunk because the words are similar. But the chunk is semantically orphaned.

Solution:

Chunk with context boundaries, not just length boundaries.

A good chunk should usually carry:

• document title,
• section path,
• nearby heading context,
• and enough surrounding text to be self-contained.

from dataclasses import dataclass

@dataclass
class Chunk:
    chunk_id: str
    doc_id: str
    parent_id: str | None
    section_path: tuple[str, ...]
    text: str
    searchable_text: str


def contextualize_chunk(doc_title: "str, chunk: Chunk) -> str:"
    path = " > ".join([doc_title, *chunk.section_path])
    return f"{path}\n\n{chunk.text}"

For retrieval, you can embed the contextualized version while still storing the original text.

Even better, use a parent-child pattern:

• retrieve using small, precise chunks,
• but send the larger parent section to the LLM.

def build_context_from_hits(hits: list[Chunk], chunk_store, parent_store) -> list[str]:
    parent_ids = {hit.parent_id for hit in hits if hit.parent_id}
    parents = [parent_store.get(pid) for pid in parent_ids]
    return [parent.text for parent in parents if parent]

Why this works:

The small chunk gives retrieval precision. The parent chunk gives the LLM enough context to understand what the precise chunk actually means.

💡 Practical note:

If your chunks often begin with “This”, “It”, “The above”, or “As described”, your chunking strategy is probably breaking referential context.

3. Generic embeddings flatten your domain

Scenario:

A user asks about “credit limits”. The retriever returns documents about “credit scores”, “credit cards”, and “credit risk” because they are semantically close. The exact policy about account credit limits is ranked too low to matter.

Why it matters:

Embedding models are powerful, but they are not magical. They encode general semantic similarity. They do not automatically understand the distinctions that matter in your product, your legal language, your codebase, or your internal terminology.

In production, this creates subtle failures:

• “environment” means deployment environment in your docs, but the embedding model leans toward general computing environments;
• “workspace” means tenant container in your product, but the model treats it as a generic UI concept;
• “policy” means insurance policy in one corpus and access policy in another.

The retriever returns plausible text. It is just not the right text.

Solution:

Do not rely on vector similarity alone.

A production retrieval pipeline usually needs at least three layers:

1. keyword search for exact terms, IDs, error codes, and names;
2. vector search for semantic similarity;
3. metadata filtering for source, version, permissions, and product area.

A simple way to combine keyword and vector results is reciprocal rank fusion.

def reciprocal_rank_fusion(
    ranked_lists: list[list[str]],
    k: int = 60,
) -> list[str]:
    scores: dict[str, float] = {}

    for ranked_list in ranked_lists:
        for rank, doc_id in enumerate(ranked_list, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)

    return sorted(scores, key=lambda doc_id: scores[doc_id], reverse=True)

You can then pass the fused list to a reranker.

Why this works:

Keyword search rescues exact matches that embeddings can blur. Vector search rescues paraphrases that keyword search misses. Fusion gives you both.

⚠️ Gotcha:

If your corpus contains product names, error codes, SKUs, ticket IDs, or legal clause numbers, pure vector search will disappoint you. Those are often exact-match problems.

4. Similarity is not evidence

Scenario:

The top retrieved chunk is about refunds. The user asked about refunds. The chunk is relevant. But it does not contain the answer to the specific question: “Can I get a refund after 60 days?”

The model sees relevant text, but not sufficient evidence. It guesses.

Why it matters:

Retrieval systems often optimize for similarity, but the LLM needs evidence.

Those are not the same thing.

A chunk can be:

• topically relevant but answer-irrelevant;
• relevant but outdated;
• relevant but incomplete;
• relevant to a different product tier;
• relevant but contradictory to another chunk.

If your retrieval pipeline stops at “these chunks are similar to the query”, it has not done enough.

Solution:

Add a reranking and evidence-selection stage.

A cross-encoder reranker is a common approach: it scores query-chunk pairs more carefully than bi-encoder vector similarity can.

from dataclasses import dataclass

@dataclass
class CandidateChunk:
    chunk_id: str
    text: str


def rerank_candidates(
    query: str,
    candidates: list[CandidateChunk],
    cross_encoder,
    top_k: int = 6,
    min_score: float = 0.30,
) -> list[CandidateChunk]:
    scored: list[tuple[float, CandidateChunk]] = []

    for candidate in candidates:
        score = cross_encoder.predict(query, candidate.text)
        if score >= min_score:
            scored.append((score, candidate))

    scored.sort(key=lambda item: item[0], reverse=True)
    return [candidate for _, candidate in scored[:top_k]]

The important design decision is this: retrieve broadly, rerank narrowly.

For example:

• vector search returns 100 candidates;
• keyword search returns 100 candidates;
• fusion produces 50 candidates;
• reranker selects 5 to 8 chunks for the LLM.

Why this works:

The first-stage retriever optimizes recall. The reranker optimizes precision. The LLM gets fewer, better chunks instead of a noisy pile.

🔍 Why this matters:

If your answer quality improves when you reduce the number of retrieved chunks, your problem is probably evidence selection, not context length.

5. Metadata is where trust and permissions live

Scenario:

An employee asks the internal assistant about salary bands. The correct document exists. But the assistant retrieves a document from another department, another country, or a draft that was never approved.

Or worse: it retrieves something the user should not be allowed to see.

Why it matters:

Vector databases are often treated as if similarity is the only query dimension. In real systems, retrieval must also respect:

• tenant boundaries,
• user groups,
• document status,
• product version,
• geography,
• customer tier,
• publication date,
• and access control rules.

If metadata is an afterthought, your retrieval pipeline becomes a security and correctness problem.

Solution:

Model retrieval filters as first-class citizens.

Every indexed chunk should carry metadata such as:

{
    "doc_id": "policy-123",
    "tenant_id": "acme",
    "acl_groups": ["hr", "managers"],
    "status": "published",
    "product": "billing",
    "version": 4,
    "effective_at": "2026-01-01T00:00:00Z",
    "superseded_at": None,
}

Then build filters from the user context and request context.

from dataclasses import dataclass
from datetime import datetime


@dataclass(frozen=True)
class UserContext:
    tenant_id: str
    groups: frozenset[str]


def authorized_retrieval_filter(user: UserContext, as_of: datetime) -> dict:
    return {
        "tenant_id": user.tenant_id,
        "acl_groups_overlap": list(user.groups),
        "status": "published",
        "effective_at_lte": as_of.isoformat(),
        "not_superseded_at": as_of.isoformat(),
    }

The exact query syntax depends on your vector database, but the architectural idea is the same: retrieval must be authorized before it is ranked.

Why this works:

It prevents the model from seeing information it should not use. That is better than trying to make the model “be careful” after the fact.

🧠 The important part:

If access control is enforced only in the UI, but not in retrieval, your RAG system can become a permission bypass machine.

6. Old versions quietly poison the index

Scenario:

Your company updated the refund policy in 2026. The old 2024 policy is still in the vector index because nobody removed it. The user asks about refunds. The retriever returns both versions. The model picks the older one because it ranks slightly better.

Now your assistant confidently gives outdated advice.

Why it matters:

Retrieval systems are sensitive to semantic match, not historical truth. If multiple versions of a document exist, the index does not automatically know which one is authoritative.

This problem shows up in:

• policies,
• pricing pages,
• API docs,
• runbooks,
• legal terms,
• release notes,
• internal SOPs.

The failure is especially nasty because the retrieved document may be “right” in a historical sense. It is just wrong for the current question.

Solution:

Version your documents and filter by time.

Use fields like:

• effective_at,
• superseded_at,
• is_current,
• doc_version,
• source_updated_at.

Then query with a point-in-time filter.

def current_version_filter(as_of: str) -> dict:
    return {
        "effective_at_lte": as_of,
        "superseded_at_gt_or_null": as_of,
    }

If your system supports it, prefer explicit version graphs:

{
    "doc_id": "refund-policy",
    "version": 5,
    "supersedes": "refund-policy-v4",
    "effective_at": "2026-02-01T00:00:00Z",
}

When a new version is published, mark the old version as superseded instead of leaving both equally retrievable.

Why this works:

The retrieval pipeline stops treating stale knowledge as equally valid. The LLM sees the version that is active for the relevant time window.

💡 Practical note:

“Delete old documents” is often not enough. You may need historical answers for old incidents, audits, or customer disputes. Versioning beats deletion.

7. The user query is not the real question

Scenario:

A user asks:

“Why did my deploy break?”

The actual logs say:

“Pipeline failed due to container image pull timeout.”

The user’s vocabulary and the system’s vocabulary do not match. The retriever searches for “deploy break” and returns generic deployment docs instead of the relevant incident record.

Why it matters:

Users ask questions using their own mental model. Documents are written using the author’s mental model. Retrieval has to bridge that gap.

If you send the raw user query directly to the index every time, you are assuming the user knows the correct terminology. In production, that assumption fails constantly.

Solution:

Transform the query before retrieval.

This can be as simple as deterministic expansion, or as sophisticated as an LLM-assisted query planner.

A safe starting point is to create a structured query object:

from dataclasses import dataclass, field


@dataclass
class RewrittenQuery:
    canonical_query: str
    expansions: list[str] = field(default_factory=list)
    filters: dict = field(default_factory=dict)

Then build multiple retrieval queries from it.

def rewrite_deploy_question(raw_query: str) -> RewrittenQuery:
    return RewrittenQuery(
        canonical_query=raw_query,
        expansions=[
            "deployment failure",
            "CI pipeline error",
            "release pipeline timeout",
            "container image pull failure",
        ],
        filters={
            "doc_type": ["incident", "runbook", "log_explanation"],
        },
    )

For more advanced systems, you can use the LLM to generate:

• a normalized question,
• likely synonyms,
• missing entities,
• subquestions,
• and metadata hints.

But do not let query rewriting become a black box. Log the rewritten query and use it in evaluation.

Why this works:

It increases recall across vocabulary mismatch. The retriever no longer depends on the user accidentally using the same words as the documentation.

⚠️ Gotcha:

Query rewriting can drift into hallucinated constraints. Always ground rewrites in the user’s original intent and validate them with retrieval evals.

8. Duplicates make retrieval confidently wrong

Scenario:

Your corpus contains the same onboarding guide copied across five product folders. The user asks an onboarding question. The top five results are all near-duplicates of the same paragraph. A better answer from a different document never makes it into the context.

Why it matters:

Duplicate content distorts ranking.

If the same idea appears many times, it can look artificially important. The retrieval system may return multiple variants of the same chunk, reducing diversity and crowding out complementary evidence.

This happens when:

• documents are copied across spaces;
• multiple versions are indexed;
• templates generate repeated text;
• web crawls include repeated headers, footers, and navigation;
• support articles are duplicated for different brands.

The result is not just inefficiency. It is biased evidence selection.

Solution:

Deduplicate at ingestion and retrieval.

At ingestion, exact duplicates can be caught with content hashes.

import hashlib


def chunk_signature(text: str) -> str:
    normalized = " ".join(text.lower().split())
    return hashlib.sha256(normalized.encode("utf-8")).hexdigest()

Store the signature with the chunk and skip or canonicalize duplicates.

def dedupe_chunks(chunks: list[dict]) -> list[dict]:
    seen: set[str] = set()
    unique: list[dict] = []

    for chunk in chunks:
        signature = chunk_signature(chunk["text"])
        if signature in seen:
            continue

        seen.add(signature)
        unique.append(chunk)

    return unique

For near-duplicates, exact hashing is not enough. You may need:

• canonical document IDs,
• section-level deduplication,
• similarity clustering,
• or retrieval diversification.

A simple diversification rule is: do not return more than one chunk from the same document unless they contain clearly different sections.

Why this works:

It improves evidence diversity. The LLM gets a broader set of relevant facts instead of five copies of the same fact.

9. If you only evaluate final answers you are blind

Scenario:

The assistant gives a wrong answer. The team debates whether the prompt is bad, the model is bad, or retrieval is bad. Nobody knows, because the only thing being measured is the final response.

Why it matters:

RAG systems have multiple failure stages. If you only evaluate the final answer, you cannot tell whether:

• the right document was missing;
• the right document was retrieved but ranked too low;
• the right chunk was retrieved but lacked context;
• the reranker removed the correct chunk;
• the prompt ignored the evidence;
• or the model hallucinated despite good evidence.

Those require different fixes.

Solution:

Evaluate retrieval separately from generation.

At minimum, build a dataset where each question has:

• expected answer,
• required evidence chunks or documents,
• and known negative chunks that should not be retrieved.

Then measure retrieval quality directly.

def retrieval_recall_at_k(
    retrieved_ids: list[str],
    relevant_ids: list[str],
    k: int,
) -> float:
    if not relevant_ids:
        return 1.0

    retrieved = set(retrieved_ids[:k])
    relevant = set(relevant_ids)
    return len(retrieved & relevant) / len(relevant)

Other useful metrics include:

• precision@k: how many retrieved chunks were actually useful;
• MRR: how early the first correct chunk appears;
• evidence coverage: whether all required facts are present;
• context contamination: whether wrong or outdated chunks appear;
• permission violations: whether unauthorized chunks were retrievable.

A practical eval case might look like this:

{
    "question": "What is the refund window for Enterprise plans?",
    "relevant_chunk_ids": ["policy-v5-enterprise-refunds"],
    "forbidden_chunk_ids": ["policy-v3-legacy-refunds"],
    "expected_answer_contains": ["90 days"],
}

Why this works:

It separates retrieval failures from generation failures. That makes debugging possible instead of speculative.

🔍 Why this matters:

If your retrieval recall@5 is poor, no amount of prompt engineering will reliably save the system.

A retrieval contract for production RAG

The way to stop being surprised by RAG failures is to define a retrieval contract.

A retrieval contract is the set of guarantees your pipeline must satisfy before the LLM is allowed to see anything.

Before shipping a production RAG system, I would want answers to these questions:

Ingestion

• Are tables preserved in a model-readable format?
• Are headings and section paths retained?
• Are headers, footers, and navigation boilerplate removed?
• Are code blocks preserved as code, not flattened prose?

Chunking

• Can each chunk be understood without reading the previous chunk?
• Does each chunk carry document and section context?
• Are small chunks used for retrieval and larger chunks for generation where appropriate?
• Are chunks connected to parent documents?

Retrieval

• Is vector search combined with keyword search for exact terms?
• Is there a reranking stage?
• Is top-k selected based on evidence quality, not just similarity score?
• Are duplicate and near-duplicate chunks controlled?

Metadata and trust

• Are permissions enforced in retrieval filters?
• Are tenant boundaries enforced?
• Is document status included: draft, published, archived?
• Are effective dates and supersession handled?

Query handling

• Is the raw query transformed or expanded?
• Are filters inferred safely?
• Are rewritten queries logged and evaluated?
• Are ambiguous queries routed to clarification instead of blind retrieval?

Evaluation

• Do we know retrieval recall@k?
• Do we know how often outdated chunks appear?
• Do we know whether forbidden documents are retrievable?
• Do we have golden questions with required evidence chunks?

If you cannot answer those, your RAG system may work in demos, but it is not telling you the truth in production.

The most important mental shift is this:

Do not ask, “Why did the LLM hallucinate?” Ask, “What evidence did the retrieval pipeline allow the LLM to see?”

In many systems, that question reveals the real failure immediately.

Every vector database tutorial opens with pip install pinecone and skips the part where you understand why vector search works. That's fine for shipping fast, but it costs you later — when recall drops and you don't know where to look, or when a tuning parameter turns your latency from 10ms to 800ms.

HNSW (Hierarchical Navigable Small World) is the algorithm behind almost every ANN (Approximate Nearest Neighbor) library in production today: FAISS, Qdrant, Weaviate, and hnswlib all use it internally. This post builds a working HNSW index in pure Python — not production-grade, but real enough to give you a concrete mental model.

What HNSW actually does

The naive approach to nearest neighbor search is brute force: compare your query vector to every stored vector, return the k closest. That's O(n·d) per query — for 1M vectors at 1536 dimensions, expect around 6 seconds per lookup.

HNSW solves this with a layered graph. Vectors live across multiple layers:

• Higher layers: sparse "highway" connections between distant nodes
• Lower layers: dense "local" connections between nearby nodes
• Layer 0: the complete graph with the richest connectivity

At query time:

1. Enter at the top layer via a fixed entry point
2. Greedily walk toward nodes closer to the query
3. Drop to the next layer, repeat
4. At layer 0, collect the k nearest neighbors

Expected complexity drops from O(n) to O(log n). The trade-off: more memory, and approximate results — you may occasionally miss the single true nearest neighbor, but you'll be well within a small margin.

Building the index in Python

Here's a minimal HNSW implementation that captures the core structure:

import numpy as np
import heapq
import math
import random
from dataclasses import dataclass, field
from typing import Optional

@dataclass
class HNSWNode:
    vector: np.ndarray
    neighbors: dict = field(default_factory=dict)  # layer -> list of node ids

class HNSWIndex:
    def __init__(self, dim: int, M: int = 16, ef_construction: int = 200, max_layers: int = 6):
        self.dim = dim
        self.M = M               # max neighbors per layer
        self.M0 = M * 2          # max neighbors at layer 0
        self.ef = ef_construction
        self.max_layers = max_layers
        self.nodes: list[HNSWNode] = []
        self.entry_point: Optional[int] = None
        self.top_layer = 0

    def _distance(self, a: np.ndarray, b: np.ndarray) -> float:
        diff = a - b
        return float(np.dot(diff, diff))  # squared L2

    def _random_level(self) -> int:
        level = 0
        while random.random() < (1.0 / self.M) and level < self.max_layers - 1:
            level += 1
        return level

    def _search_layer(self, query: np.ndarray, entry_id: int, ef: int, layer: int) -> list[int]:
        visited = {entry_id}
        entry_dist = self._distance(query, self.nodes[entry_id].vector)
        candidates = [(entry_dist, entry_id)]
        dynamic_list = [(entry_dist, entry_id)]

        while candidates:
            c_dist, c_id = heapq.heappop(candidates)
            worst_dist = max(d for d, _ in dynamic_list)
            if c_dist > worst_dist:
                break
            for nb_id in self.nodes[c_id].neighbors.get(layer, []):
                if nb_id not in visited:
                    visited.add(nb_id)
                    nb_dist = self._distance(query, self.nodes[nb_id].vector)
                    if nb_dist < worst_dist or len(dynamic_list) < ef:
                        heapq.heappush(candidates, (nb_dist, nb_id))
                        dynamic_list.append((nb_dist, nb_id))
                        if len(dynamic_list) > ef:
                            dynamic_list.remove(max(dynamic_list))

        return [nid for _, nid in sorted(dynamic_list)[:ef]]

    def add(self, vector: np.ndarray) -> int:
        node_id = len(self.nodes)
        node = HNSWNode(vector=vector)
        self.nodes.append(node)
        level = self._random_level()

        if self.entry_point is None:
            self.entry_point = node_id
            self.top_layer = level
            for l in range(level + 1):
                node.neighbors[l] = []
            return node_id

        ep = self.entry_point
        for l in range(self.top_layer, level, -1):
            candidates = self._search_layer(vector, ep, ef=1, layer=l)
            ep = candidates[0]

        for l in range(min(level, self.top_layer) + 1):
            neighbors = self._search_layer(vector, ep, ef=self.ef, layer=l)
            M = self.M0 if l == 0 else self.M
            neighbors = sorted(
                neighbors,
                key=lambda nid: self._distance(vector, self.nodes[nid].vector)
            )[:M]
            node.neighbors[l] = neighbors
            for nb_id in neighbors:
                nb_node = self.nodes[nb_id]
                if l not in nb_node.neighbors:
                    nb_node.neighbors[l] = []
                nb_node.neighbors[l].append(node_id)
                if len(nb_node.neighbors[l]) > M:
                    nb_node.neighbors[l] = sorted(
                        nb_node.neighbors[l],
                        key=lambda nid: self._distance(nb_node.vector, self.nodes[nid].vector)
                    )[:M]
            ep = neighbors[0] if neighbors else ep

        if level > self.top_layer:
            self.top_layer = level
            self.entry_point = node_id
        return node_id

    def search(self, query: np.ndarray, k: int = 10, ef: int = 50) -> list[tuple[float, int]]:
        if self.entry_point is None:
            return []
        ep = self.entry_point
        for l in range(self.top_layer, 0, -1):
            candidates = self._search_layer(query, ep, ef=1, layer=l)
            ep = candidates[0]
        candidates = self._search_layer(query, ep, ef=max(ef, k), layer=0)
        results = sorted(candidates, key=lambda nid: self._distance(query, self.nodes[nid].vector))
        return [(self._distance(query, self.nodes[nid].vector), nid) for nid in results[:k]]

Using the index

import numpy as np
from hnsw import HNSWIndex

# Build an index with 128-dimensional vectors
index = HNSWIndex(dim=128, M=16, ef_construction=200)

# Insert 10k random vectors (documents, embeddings, etc.)
np.random.seed(42)
n = 10_000
vectors = np.random.randn(n, 128).astype(np.float32)

print("Building index...")
for i, vec in enumerate(vectors):
    index.add(vec)
    if (i + 1) % 1000 == 0:
        print(f"  {i + 1}/{n} vectors inserted")

# Query with a random vector
query = np.random.randn(128).astype(np.float32)
results = index.search(query, k=5, ef=50)

print("\nTop 5 nearest neighbors (HNSW):")
for dist, node_id in results:
    print(f"  node_id={node_id}, sq_distance={dist:.4f}")

# Verify against brute force
all_dists = np.sum((vectors - query) ** 2, axis=1)
brute_top5 = sorted(enumerate(all_dists), key=lambda x: x[1])[:5]
print("\nBrute-force top 5:")
for nid, dist in brute_top5:
    print(f"  node_id={nid}, sq_distance={dist:.4f}")

On a 10k-vector dataset, you'll typically see 4 or 5 out of 5 results overlap between HNSW and brute force. That's the "approximate" in ANN: for most real workloads, occasionally missing one true neighbor is an acceptable trade-off for a 50x–100x query speedup.

Key parameters and their effect

M — max connections per node per layer. Higher M means better recall and a denser graph, at the cost of more memory and slower inserts. Start with 16. Go up to 32 if your recall consistently falls below 90%.

ef_construction — candidate pool size during index build. Larger values produce a better-connected graph but slow down insertion. 200 is a solid default; lower it to 100 if insertion throughput matters more than recall.

ef (search-time) — candidate pool size at query time. This is the most important lever for production tuning because you can adjust it without rebuilding the index. Setting ef = k gives the fastest queries; ef = 500 approaches brute-force recall. Plotting ef vs. recall typically reveals a knee around ef = 50–100 for most datasets — that's your sweet spot.

Understanding these parameters matters the moment your AI-powered search starts returning wrong results. Before blaming your embedding model, check ef and whether your index was built with sufficient ef_construction. It's the same diagnostic discipline that applies across security tooling — knowing the internals saves time when something breaks. The security hardening checklists we publish follow the same philosophy: understand the mechanism, then apply the control.

Production considerations

This implementation deliberately omits several things you'd need in production:

• Deletion support: HNSW graphs don't support clean node removal without a full rebuild. Libraries like Qdrant implement soft-deletion with tombstones.
• Persistence: serialize node vectors and neighbor lists to disk (numpy's np.save for vectors, JSON or msgpack for the neighbor maps).
• Thread safety: concurrent inserts will corrupt the neighbor lists. Use a lock or batch inserts single-threaded.
• SIMD distance computation: the Python distance function is the bottleneck. hnswlib's C++ core runs distance in vectorized instructions — easily 100x faster.

For any production workload, use hnswlib directly (pip install hnswlib) or a vector database that wraps it. The API is almost identical to what's shown above, and you get persistence, filtering, and proper concurrency for free.

The takeaway

HNSW is not magic — it's a graph with controlled connectivity. The layered structure lets queries skip large sections of the search space, and the ef parameter gives you a runtime dial between speed and accuracy.

Building a stripped-down version reveals what every production vector database is doing under the hood. That understanding pays off when you're debugging recall issues at 3am — and it's faster than re-reading library documentation when you already know what the parameters mean.

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.

Your MCP server's tool schema is the entire interface a model has. No README, no repo, no idea what you meant — just the JSON from tools/list.

We read that JSON for 4,951 public servers: 87,146 tools and 270,487 parameters. Here's what's in it.

The short version: one tool in six carries a description containing no word that distinguishes it from a sibling tool on the same server. On servers with more than sixty tools, it's nearly one in three.

What this is, and what it isn't

This post is about what models are given. It is not about what they do with it.

We did not run a model against these servers. We never observed a tool selection, an argument, or a retry, so nothing here can tell you how often models actually get it wrong. Every number below is a property of a schema sitting still.

We're drawing that line hard because it's the line the interesting claim sits on. "A model has nothing to discriminate on" is a fact about a schema. "A model therefore picks wrong 30% of the time" is a fact about traffic, and we don't have it.

How we did it

We took the tool schemas from the Smithery registry, which stores the tools/list response for every server it has scanned — inputSchema included. That's byte-for-byte the JSON a model receives.

The obvious alternative was to boot each server in a container and call tools/list ourselves. We didn't, and the reason is the finding underneath the method: a large share of public MCP servers won't start without real credentials. The set that boots cleanly on a machine with no API keys isn't a random subset of anything.

Of 5,123 servers in the frame, 154 detail requests failed, 5 had never been scanned, and 13 published no tools at all. That leaves 4,951 servers with schemas.

The check that nearly ended the study

Before any of this counts for anything, one question has to be answered: does the registry hand back the schema the server actually serves?

We had reason to think it might not. Across the first 1,377 tools we collected, not one carried a top-level required array. Real MCP servers mark parameters required constantly.

So we booted the four official reference servers locally — they need no credentials and no network — took their real tools/list over stdio, and diffed field by field.

Everything survives except required, which is stripped. On one server that's 27 of 37 tools with a required array on the real server, and 0 via the API.

So this study measures nothing about required-versus-optional parameters. If you're doing your own analysis on registry data, that field is not there, and it does not announce itself.

1. One parameter in five has no description at all

59,038 of 270,487 parameters — 21.8% — ship with no description. They appear on 33.7% of servers.

A parameter with no description is a parameter the model guesses at. It has the name, it has the type, and that's the entire brief. Sometimes the name carries it: query on a search tool isn't mysterious. Often it doesn't — we found plenty of bare id, type, mode and filter parameters with nothing to say which of several plausible things they meant.

The striking part is the contrast with tool descriptions. Only 0.4% of tools have no description. Authors describe the tool and forget the arguments — which is understandable, because the tool is the thing you're thinking about when you write it, and the arguments are the thing the model has to fill in.

2. One tool in six has nothing to tell it apart from its neighbour

For every tool, we took the content words in its description and asked how many appear in no other tool's description on the same server. Call it the tool's distinctive share.

• The median tool's description is 26% distinctive. Three-quarters of the words it spends are words its siblings also use.
• 26.7% of tools are under 10% distinctive.
• 17.4% are exactly zero. 23.1% of servers have at least one.

Zero doesn't mean the description is bad. Here are four from one widely-installed Gmail server:

Gmail_DeleteDraftEmail   "Delete a draft email using the Gmail API."
Gmail_SendDraftEmail     "Send a draft email using the Gmail API."
Gmail_ListLabels         "List all the labels in the user's mailbox."
Gmail_SearchThreads      "Search for threads in the user's mailbox."

Every one of those is clear, correct English. Every one is also built entirely from words the other tools use — delete, draft, email, gmail, api, list, search, threads, mailbox all recur across the set. The description tells you what the tool does. It doesn't tell you what this tool does and the others don't, and that second thing is the one a model needs when it's choosing.

The pattern that produces the most extreme cases is shared boilerplate. One server appends the same 51-word context block to all 275 of its tools. "Create a new NFT collection" and "Purchase an NFT from a listing" differ by 8 words out of 59. That block was added deliberately, to help.

3. It gets worse the more tools you ship

Description collision rises monotonically and by a factor of sixty. Some of that is arithmetic — more tools means more chances for two to collide — but it's also the point at which authors start generating descriptions from a template, and a template is a machine for producing tools that read alike.

And notice the column that doesn't move. Missing parameter descriptions sit between 20% and 25% at every size above the smallest bucket. It's not a scale problem; it's a habit. The two failures are independent, which means shipping fewer tools won't fix your undescribed parameters and writing better descriptions won't fix your collisions.

4. The median tool list costs about 1,250 tokens before anyone asks a question

Tool schemas are sent on every connection, whether or not a single tool gets called.

• Median server: 4,991 bytes, roughly 1,250 tokens
• 90th percentile: 32,636 bytes, roughly 8,200 tokens
• Largest in the corpus: 1,145,575 bytes — on the order of 280,000 tokens of schema, from one server, before the conversation starts

Median tools per server is 7 and the mean is 17.6. One server publishes 2,530 tools.

5. Some parameters name their valid values and then don't enforce them

8.0% of all parameters carry an enum. What you can find from outside is the case where the author wrote the values down in prose and left the schema as an open string:

outcome_attribution   "Attribution type for the outcomes.
                       Valid values: "direct", "influenced",
                       "unattributed", "total"."

commitment            "Optional processed|confirmed|finalized commitment"

895 parameters, on 4.6% of servers. That's a floor rather than an estimate: it only catches authors who documented the set.

Worth reporting how we got there. Our first version of this measurement said 108 hits in a 40-server sample. It was matching text like Filter by line (e.g. "1", "A", "F"), which is an illustration, not a closed set. Requiring explicit closed-set language took that sample from 108 to 9, and all nine were real.

One signal we expected to find and didn't: duplicate tool names within a server, on 0.2% of servers. Effectively nobody does this. If it's on your review checklist, take it off.

What we couldn't see

This is static analysis, and we never observed a single real request to any of these servers.

That means we missed everything that only shows up under traffic. The tool that works in isolation but gets called in the wrong order. The parameter that's fine until someone phrases a request unusually. The retry loop that only triggers on a specific error path.

We also can't tell you the thing you most want to know, which is how much of this matters. A zero-distinctive description might cost nothing when the tool name is unambiguous, and a great deal when it isn't.

One more hole, found by a server author after publication: the 21.8% measures absence only. A parameter described as "query: The query" counts as described and passes. Restating the parameter name is the more common failure in his experience, and it passes every linter. So that number is a floor too.

If you maintain a server

Three things, ordered by how common the problem is in the data. None of them changes your server's behaviour — they're all changes to the text a model reads.

Describe every parameter. The most common gap by a distance, and it doesn't get better at any size. If you do one thing, do this one.

Make each description say what the others don't. Not "is it clear" — is it clear which of my tools this is. Read your tool list as one block and ask what a reader with only that block would use to choose.

If you have more than about thirty tools, audit for collisions specifically. Above that size, roughly one tool in six has no distinguishing word, rising to one in three past sixty.

The data

Analysis scripts and raw data: https://github.com/getmcpulse/mcp-schema-study

Every figure is in analysis.json, with the servers and tools behind each one in examples.json. The collector rebuilds the whole corpus from a public API in about twenty minutes.

There's also a browser-based checker that runs these measurements on your own tools/list — paste the JSON, get your numbers scored against the corpus: https://getmcpulse.com/check

We ran this because we're building MCPulse, an SDK that reports what actually happens under real traffic. Everything in this post came from outside the server, which is exactly its limit. A schema can tell you a model has nothing to choose on. Only traffic can tell you whether it chose wrong.

Originally published at getmcpulse.com.

I fine-tuned a small language model to sort messages into safe or unsafe, the kind of check a platform runs before letting a comment through. The whole thing runs on an 8 GB laptop, in a few minutes, for free.

The result was not a clean win, and that was the point: the overall score went up while the model got worse at what mattered most. What I came away with was the method - how to run a fine-tune, and how to tell whether it actually helped.

A learning project, written up for anyone who wants to try fine-tuning without a big machine or a big budget.

Repo: https://github.com/sbezjak/fine-tuning-cm

First: should you even fine-tune?

Most of the time, the answer is no. If a better prompt gets you there, do that. If the model just needs facts it does not have, give it those at question time (that is what retrieval, or RAG, is for).

Fine-tuning earns its place in a narrower spot: one repeated, well-defined task you want done the same way every time, without a long prompt. Sorting messages into two labels is exactly that shape, so it is a good thing to learn on.

The task, and why grading is easy

The model gets a message and answers with one word: safe or unsafe. Because that answer is one word from a fixed set, I can grade it with a plain string check, no second model acting as a judge and no scoring rubric. That keeps the scoring objective and the whole project free to run.

One catch: safe is contained inside unsafe, so a check that just looks for "safe" in the reply would mark every "unsafe" answer correct. Matching whole words only is a one-line fix, and the kind of thing that silently corrupts your numbers if you miss it.

The data

The real training examples come from Civil Comments, a public set of reader comments from news sites. Each comment has a toxicity number, which is simply the share of human reviewers who marked it toxic. I turn that into a single label, calling a comment unsafe when at least 70% of reviewers flagged it.

The comments are real and some are genuinely toxic, so they never go into the public repo. A small download script pulls a tiny, class-balanced slice into a git-ignored folder (it is CC0, public domain; the source is cited in docs/dataset.md). Only the derived labels and the final numbers are ever committed, never the text itself: in a sensitive domain, you publish what you measured, not the corpus you measured it on.

What a fine-tune actually changes

I did not retrain the model. That would need far more memory than a laptop has.

Instead I trained a small add-on called a LoRA adapter. The original model - a small open one, Qwen2.5-1.5B-Instruct - has more than a billion numbers inside it, and they all stay frozen. The adapter is a tiny extra set of numbers, trained from scratch, that sits on top and nudges the model's answers. In this project the adapter was about 0.17% of the model's size, a file of roughly 10 MB.

That is the fact that made fine-tuning click for me: I am not rewriting the model, I am training a small, cheap correction that clips onto it. Take the clip off and you have the original model back. Put it on and you have the tuned one. That is why adapters are so easy to store, swap, and compare.

Training, and knowing when to stop

I used Apple's MLX toolchain, which trains on the Mac's own chip, and a small model kept in a compact 4-bit form to save memory. Every setting is shaped by the 8 GB limit: small batches, short inputs, few trained layers.

Training took about nine minutes and peaked at roughly 1.8 GB of memory. The 8 GB limit barely bit at this size, which was itself worth learning: the ceiling I planned around was not the one that mattered.

It helps to picture what training is actually doing. The adapter starts as a set of numbers that do nothing. The model reads one training message, guesses a label, and the guess is compared to the correct answer. Every number in the adapter is then nudged a tiny amount, in the direction that would have made the guess a little closer - the way a hiker in fog feels which way is downhill and takes a step that way. Then the next message, and the next. One such pass is a step, and I ran hundreds of them. No single step changes much - the learning is thousands of these small nudges accumulating, slowly bending the numbers into a shape that gets the labels right.

The important lesson was about when to stop. As training runs, you watch two numbers. One is how well the model fits the examples it is training on (the training loss). The other is how well it does on a separate set it never trains on (the validation loss). The training loss kept dropping. The validation loss dropped, hit a low point, and then started climbing again. That climb is the model starting to memorize the training examples instead of learning the general pattern, which is called overfitting. So the best adapter was not the last one saved: I trained for 400 steps but kept the checkpoint from step 200, where the validation loss bottomed out, right in the middle of the run.

How to trust a before-and-after

This is the part the whole project is really about. The test set is "held out": a batch of labelled messages set aside at the start and never shown to the model during training, so grading it is a fair check on messages it has not already seen. Training and measuring are not two phases, they are one loop: you never train and just trust the result, you train and immediately re-run that same held-out test to see whether it actually helped.

To measure whether the adapter helped, I run the held-out messages through the model twice: once without the adapter (the "before"), once with it (the "after"). Everything else is identical. Same messages, same settings, and the model is set to be fully repeatable, so the same message always gives the same answer. That way any difference in the score is caused by the adapter and nothing else.

There is a quiet trap here. If you point the code at the wrong adapter folder, it just loads the plain model and reports a difference of zero, and you would never know the fine-tune did nothing. So the code now checks that the adapter is really there and stops loudly if it is not. A before-and-after you cannot trust is worse than no measurement at all.

The result, and the trap in it

First, how to read these numbers, because the direction flips from the last section. Training and validation loss are errors, so lower is better. Everything in the table below is either an accuracy (the share the model got right, from 0 to 1, so higher is better) or a plain count of mistakes (lower is better). So a rising score is good news and more threats let through is bad news, even though both numbers go up.

I ran the whole loop first on a smaller 0.5B model with fabricated data, just to prove it worked end to end. That is not a result, so the numbers here are the real run: the 1.5B model, trained and measured on held-out Civil Comments.

The starting model over-flags badly. It catches only 11% of the safe messages, calling almost everything unsafe, and on 13 of the 90 it does not answer at all: instead of a label it falls back to a stock phrase ("contains a threat of violence"), because the untuned model was never taught to reply with one word from a fixed set. Training fixes that, and the non-answers drop to zero. But the interesting part is what it trades to get there.

Read the overall column and the fine-tune is a clear win: 0.533 up to 0.789, and it almost stops over-flagging safe speech (11% caught up to 89%). But read across the row, not down that one column. Of 54 genuinely unsafe messages, the tuned model now lets 15 through, up from 10. It bought its lower false-alarm rate by catching fewer real threats, and letting a threat through is the one error a moderation model can least afford.

Two things did move it in the right direction. The first was cleaner labels: at the bare 50% cutoff the unsafe pile filled up with genuinely civil comments that only a couple of raters happened to flag, so raising the bar to 70% agreement dropped that noise and pointed "unsafe" at real hostility. The second was better-chosen training examples, aimed at the exact cases the model kept getting wrong. I also tried teaching the model to name the type of harm rather than just safe/unsafe, which helped a little; that is a thread for a future project, not this one.

To check the method did not depend on my particular laptop, I ran the same training on a free cloud Google Colab T4 GPU. It behaved the same way, down to the same over-training climb: the recipe is about the method, not the hardware.

What real systems do that this one doesn't

Worth being clear about the gap between this and production.

When a big lab ships a new model, it trains all of its numbers across huge data and hardware. The LoRA approach I used is the common choice for the other job: adapting an existing model to a narrow task cheaply, and keeping many small adapters over one shared base.

Real moderation systems also do things this one skips: they read a confidence score and send the uncertain cases to a human instead of forcing a yes/no; they weigh a missed threat as more costly than an over-flag; they test against much larger, carefully labeled sets. Those are the next steps, named here rather than built.

Try it yourself

Start with the "smoke" path. It trains on the tiny set of made-up examples that ship with the repo, so it does not measure real moderation, but it turns the whole loop: preparing the data, training the adapter, and measuring the before-and-after. The point is to feel the machinery run end to end, not to get a real number:

uv sync
uv run pytest -m mocked                                   # fast unit tests, no model or network
scripts/train-smoke.sh                                    # prepare + train on the fabricated set
uv run python -m ft_cm.eval                               # before/after on held-out examples
uv run python -m ft_cm.report evidence/smoke-before-after.json

To get a real result, point it at the real data instead. scripts/download-dataset.sh pulls a small, class-balanced slice of Civil Comments into the git-ignored data/real/ folder (no login, CC0 licensed; provenance is in docs/dataset.md), and you train and evaluate on that the same way. The real text stays on your machine and never gets committed.

Either way, the habit is the same one from earlier: change something, retrain, re-run the before-and-after, and read the whole breakdown rather than the headline score. That is the fastest way to feel what a fine-tune does, and what it does not.

Repo: https://github.com/sbezjak/fine-tuning-cm

DailyVox, the voice journal I build, holds no network permission. There is no analytics call to remove and no upload path to audit, because the capability was never granted. That single constraint made most of my architecture decisions for me. This post walks through what it forced.

The reasoning behind the rule is short. A diary is the most honest text a person produces, and I could not bring myself to be honest in an app that stored my entries on someone's server. "We promise not to look" is a policy. "The app has no way to send it" is a property. I wanted the property, so every model in the pipeline has to run on the phone, and every fallback that reaches for a server has to be treated as a bug.

Speech first

Recording is the whole interface. You talk for about 42 seconds, and everything downstream is built on what the recognizer returns.

let request = SFSpeechAudioBufferRecognitionRequest()
request.requiresOnDeviceRecognition = true

That flag is load-bearing. Without it, SFSpeechRecognizer is free to use Apple's servers when it judges the result would be better, and for connected iPhones it usually judges exactly that. With it, transcription runs on the Neural Engine or fails. When the on-device model for a language is missing, DailyVox shows an error that names the setting to fix, because a loud failure is cheaper than a quiet upload.

The trade is real. Server-side models punctuate better and handle rare words better. In exchange you get transcription that works in airplane mode, latency that doesn't depend on your connection, and audio that dies inside the device that heard it.

Sentiment you can check

Each transcript gets a mood score from NLTagger plus a lexicon layer. When I correlated the scorer against self-reported mood labels, it came out at r = +0.663. Middling as correlations go, and about what a lexicon method earns on 42-second spoken entries. I publish the number anyway. A mood feature that has never been measured against ground truth is decoration.

The scores feed a small forecasting model keyed on day of week. Mine found that my low days cluster on Sundays. Twenty years of paper diaries never showed me that, because nobody rereads twenty years of paper.

The name the tagger couldn't see

The Twin builds a knowledge graph of the people in your life, which means named entity recognition. Apple's tagger handled "Sarah" fine and treated "Adyah" as noise. Names from outside its training distribution simply vanished, and on my entity test suite that blind spot cost 34.6% recall.

The fix was to stop trusting the tagger's entity class as the source of truth. Capitalized tokens become entity candidates, and the graph decides over time which candidates are people, by watching how they recur across entries. It recovered the missing names.

It also bought me a new dependency: the speech recognizer must capitalize names in the first place. Apple's does. Whether Android's does on real Samsung and Xiaomi hardware is the question currently standing between the finished Android port and its release.

Search with no server

Semantic search runs on NLEmbedding.sentenceEmbedding. Apple ships those vectors for English, Spanish, French, German and Italian, which is why the interface ships in exactly those five languages. Shipping a language where search quietly returns nothing would be worse than waiting.

Cosine similarity has no sense of time, so a pure vector search happily surfaces a three-year-old entry over last week's. The ranking blends similarity with recency, and below a similarity of roughly 0.37 the app says "no good match" instead of padding results. Teaching a search box to abstain took longer than teaching it to match.

The Twin itself

Everything lands in the model the app is named for: a Digital Twin, meaning baselines for emotional valence and arousal, a communication-style profile, the entity graph, and the mood forecaster. The whole state is Codable JSON inside a single Core Data entity, synced through the user's own iCloud container so a new phone doesn't mean a new stranger.

You can ask it questions. Answers come from Apple's Foundation Models framework, on-device, and every answer must cite the entries it drew from. A deterministic audit checks each claim against its citations and rejects answers that fail. That gate deserves its own post, and it will get one.

What the constraint bought

The App Store privacy label reads "Data Not Collected," and it reads that way as a build output rather than a marketing decision. Demos run in airplane mode. There are no retention pings and no A/B tests, so the only usage number I have is the App Store's install count, which is a strange feeling for someone who has shipped instrumented products for years. I recommend the feeling.

The honest split

The iOS app is MIT. How it records, stores, encrypts and exports your words is readable by anyone, and so is the permission list. The Twin's analysis engine is a closed Swift package that lives inside that app. The part that touches your data is checkable; the part that models you is the product I'm betting on. Some people find that line unsatisfying. I'd rather draw it where you can see it than blur it.

Speak for 42 seconds. Watch your words become stars.

The free voice journal with on-device AI and a Digital Twin that learns who you are — entirely on your phone. iPhone today; Android in development.

_{Free forever  ·  100% on-device  ·  No account, no servers, no analytics  ·  MIT licensed  ·  on Product Hunt}

DailyVox turns your spoken thoughts into a constellation of stars. Every journal entry becomes a point of light in your inner sky — mood-colored, connected by patterns only you can see.

Behind the scenes, an on-device Digital Twin learns how you think, how you feel, and who matters to you. It predicts your mood, answers questions about your patterns, and reveals meaning across months of entries. No accounts. No servers. No data collection.

Your thoughts never leave your device. Ever.

_{Real screenshots from v1.11.0, seeded with a demo journal.}…