Projects · AIForge
RAG pipeline and Qdrant
Retrieval augmented generation end to end: ingestion, parsing, chunking, embedding models, Qdrant collections and HNSW, filtered retrieval, reranking, context assembly, the prompt, citations, evaluation, and every way this pipeline lies to you.
A language model does not know it is wrong. It produces the most plausible continuation of your text, and plausible is not the same as true.
Retrieval augmented generation is the standard fix: put the real answer in the prompt so the model does not have to invent one. It works, and it works well.
It also fails in ways that are much harder to notice than a model simply being wrong, because a RAG answer comes wrapped in the authority of a citation. Sections 10 and 11 are the ones I would read first if you already know what RAG is. The pipeline is easy. Knowing when it is quietly broken is the actual skill.
RAG pipeline and Qdrant
Everything in this volume so far has been infrastructure that does not care what the model says. The engine moves tokens. The lifecycle layer moves versions. The gateway moves requests.
This chapter is the first one where content quality is an infrastructure concern, and that shift is worth naming, because it changes what "working" means. A model server is working when it returns 200. A RAG pipeline can return 200 for a year while giving people confidently wrong answers about their own documents.
By the end of this chapter, documents go in one side and cited answers come out the other, with a Qdrant collection in the middle that enforces tenant isolation at the query level, and an evaluation harness that tells me when retrieval degrades.
1. Why retrieval, and not the alternatives
Three ways to make a model know something it did not learn during training. They are not equivalent and the choice is usually made badly.
| Approach | How it works | Cost to update | When it is right |
|---|---|---|---|
| Fine tuning | Adjust the weights on your data | A training run. Hours to days, plus hardware I do not have | Teaching behaviour: a format, a tone, a domain vocabulary. Poor at teaching facts |
| Everything in the context window | Paste all the documents into every prompt | Zero | Small, stable corpora. Falls apart on cost, latency, and the model's tendency to lose things in the middle of long inputs |
| Retrieval augmented generation | Find the relevant few passages, put only those in the prompt | Re embed the changed documents. Seconds to minutes | Facts that change, large corpora, per tenant data, and any case where you need to cite a source |
Almost every "we should fine tune on our documentation" conversation is actually a retrieval problem wearing a training costume. Fine tuning a model on your company handbook does not reliably make it able to quote the handbook, it makes it sound like the handbook. And when the handbook changes, you train again.
There is also a hard requirement that only retrieval satisfies: a fine tuned model cannot tell you which document an answer came from. The knowledge is smeared across billions of weights. In a regulated or even mildly serious context, an answer without provenance is not an answer.
2. The pipeline, both halves
RAG is two pipelines that meet inside a database. Treating them as one system is the first conceptual mistake, because they run on different schedules, fail differently, and are debugged separately.
Look at the two Embed boxes. They must use the same model, at the same version, with the same normalisation.
Change the embedding model and re embed nothing, and the system keeps working. No error. No warning. Queries return results, scores look like numbers, the answers are simply nonsense, because you are comparing coordinates from two different spaces.
The embedding model is part of the data, not part of the code. In AIForge the model name and revision live in the collection metadata, and the ingestion job refuses to write into a collection built by a different model. That check took ten minutes to write and it removes an entire category of silent failure.
3. Ingestion and parsing
The least glamorous stage and the one that decides your ceiling. Nothing downstream can recover information that the parser destroyed.
What actually matters at this stage:
PDFs are a rendering format, not a document format. A PDF knows where glyphs are painted, not that a paragraph exists. Two column layouts get read across the columns. Headers and footers get interleaved into sentences. Tables become a soup of numbers with no rows.
Tables are where naive pipelines die. A chunk reading 25 30 15 retrieved without its header row is worse than no chunk at all, because the model will confidently attach those numbers to whatever the question asked about. I convert tables to markdown at parse time so structure survives as text.
Scanned documents need OCR, and OCR output needs a quality gate. A page that returns forty characters of gibberish should be flagged, not embedded.
Capture provenance now. Source URI, page number, heading path, last modified date, tenant. Every one of those becomes a citation, a filter, or a freshness signal later. Reconstructing them afterwards is impossible.
I do not evaluate the parser by looking at it. I take twenty random chunks from the output, read them cold, and ask: could I answer a question from this text alone, with no other context?
If a chunk starts mid sentence, or contains a table with no header, or is a page footer repeated four hundred times, the parser is not done. This takes fifteen minutes and it is the highest leverage quarter hour in the whole pipeline.
Retrieval quality is capped by chunk quality, and chunk quality is visible to the naked eye.4. Chunking, the decision everyone underestimates
A chunk is the atomic unit of retrieval. It is what gets embedded, what gets returned, and what ends up in the prompt. Its size is a genuine tradeoff with no universally correct answer.
| Chunk size | What you gain | What you lose |
|---|---|---|
| Small, 128 to 256 tokens | Precise. The vector represents one idea, so similarity scores are sharp | Context is amputated. "It must be approved in advance" without knowing what "it" is |
| Medium, 400 to 600 tokens | Usually the sweet spot. Enough to stand alone, focused enough to embed meaningfully | Nothing dramatic, which is why it is the default |
| Large, 1000 or more tokens | Rich context, fewer chunks, cheaper index | The embedding becomes an average of several topics and matches nothing well. It also eats the prompt budget fast |
The lab uses 512 tokens with 64 tokens of overlap, split on structure first and length second. The overlap exists for one reason: a fact that straddles a boundary would otherwise be split in half and be retrievable by neither chunk.
A raw chunk from page 40 of a policy document might read: "Requests must be submitted at least fourteen days in advance and approved by a line manager."
Requests for what? The chunk does not say. The document said it in a heading twelve paragraphs earlier, and the chunker threw that away.
Prepending Employee Handbook > Leave > Annual leave to the embedded text makes that chunk retrievable by the question "how far ahead do I need to book holiday", which it previously was not. Same text, same model, dramatically better recall.
Note that I embed the breadcrumb but display the body. The text you search on and the text you show do not have to be the same text, and forgetting that is a missed opportunity in most pipelines I have read.
5. Embeddings, concretely
An embedding model maps text to a fixed length vector such that similar meanings land near each other. That is the entire idea, and the useful details are practical rather than mathematical.
| Model | Dimensions | Max input | Notes for a CPU lab |
|---|---|---|---|
all-MiniLM-L6-v2 | 384 | 256 tokens | Tiny and fast. The 256 token limit silently truncates 512 token chunks, which is a trap |
bge-small-en-v1.5 | 384 | 512 tokens | The lab default. Matches the chunk size, strong quality for its size, wants a query prefix |
bge-base-en-v1.5 | 768 | 512 tokens | Better, and twice the storage and CPU. Worth it if quality is the binding constraint |
e5-small-v2 | 384 | 512 tokens | Comparable. Requires explicit query and passage prefixes, and omitting them measurably hurts |
Three practical points that matter more than the leaderboard rankings:
Dimensions cost real bytes. A 384 dimension float32 vector is 1,536 bytes. One hundred thousand chunks is about 150 MB of raw vectors, plus the HNSW graph on top. Going to 768 dimensions doubles that and increases search time. Bigger is not automatically better, especially when your corpus is small enough that a small model separates it cleanly.
Normalise, then use cosine distance. With normalised vectors, cosine similarity and dot product become equivalent, which is fast and predictable. Mixing normalised and unnormalised vectors in one collection produces scores that are meaningless in a way nothing will warn you about.
Asymmetric search is real. Questions and passages do not look alike. The BGE family expects a prefix on the query side, something like Represent this sentence for searching relevant passages:, and the E5 family expects query: and passage: prefixes. Forgetting them is a silent quality regression, and it is one of the most common mistakes in home grown pipelines.
It would be simpler to load the model in process with sentence-transformers and skip the network hop. I deliberately do not.
Routing embeddings through LiteLLM means they are counted, rate limited and attributed like every other call. The RAG service is a tenant of the platform, not an exception to it. When someone asks why the cluster was busy at 3am, "the reindex job embedded 400,000 chunks" is an answer the spend log can give.
For the serving side, Hugging Face Text Embeddings Inference is what runs behind that endpoint. It is genuinely good on CPU, which is rare and worth saying.
6. Qdrant, properly
Qdrant is the vector database. Its job is to store vectors with attached metadata and find the nearest ones to a query vector, fast, with filters applied correctly.
6.1 The vocabulary
| Term | What it is |
|---|---|
| Collection | A named set of points with a fixed vector size and distance metric. Roughly a table |
| Point | One record: an id, one or more vectors, and a payload. Roughly a row |
| Payload | Arbitrary JSON attached to a point. Where tenant id, source, heading and the chunk text live |
| Filter | A condition on payload applied during the vector search, not after it. Section 6.3 explains why that word matters |
| HNSW index | The graph structure that makes approximate nearest neighbour search fast |
6.2 HNSW without the hand waving
Exact nearest neighbour search means comparing the query to every vector. At 100,000 vectors that is fine. At 10 million it is not.
HNSW, hierarchical navigable small world, builds a layered graph. The top layer is sparse with long links, and each layer down is denser. A search enters at the top, greedily walks toward the query, drops a layer, and repeats. It is a skip list for high dimensional space.
HNSW does not guarantee the true nearest neighbours. With default settings recall is typically in the high nineties percent, which is excellent, and it is not 100 percent.
So a relevant chunk can be missed by the index, not by the embedding. Two consequences follow. First, ef is a real quality knob and not just a performance one, so it belongs in your evaluation runs. Second, when a specific document stubbornly refuses to be retrieved, raise ef and try again before blaming the chunker: if it appears, the index was the problem, and if it does not, the embedding was.
6.3 Creating the collection
tenant_id in the payload is what stops tenant A retrieving tenant B's documents. Every single search in AIForge carries that filter, and it is applied by the service from the authenticated identity, never from anything the caller supplied.
The naive alternative, searching without a filter and discarding foreign results afterwards, is broken in two separate ways. It is slower, because you retrieve results you throw away. And it is a data leak one bug away, because the sensitive text was already loaded into the process by the time the filter ran.
Qdrant applies filters during graph traversal rather than after it, which is why this is both the fast option and the safe one. When the secure path is also the fast path, take it and never look back.
Stronger isolation, a collection per tenant or separate Qdrant instances, is a Volume 4 discussion. A payload filter with a payload index is the right default for a shared knowledge base.
status: green means indexing is settled. Yellow means optimisers are still working, and searches during that window can have lower recall than you expect. Benchmarking a yellow collection is how people conclude that Qdrant is inaccurate.
indexed_vectors_count equal to points_count means every point is in the graph. A gap after a large upsert is normal and temporary. A gap that persists means an optimiser is failing, usually on disk or memory.
7. Retrieval
Now the query side, which is short because the hard work happened earlier.
Retrieval and selection are different jobs and should be done by different components.
The vector search is a fast, cheap, approximate filter over 48,000 chunks. It is good at ruling things out and mediocre at ranking the survivors, because a single 384 dimension vector cannot capture everything about a passage.
The reranker in the next section is slow, expensive and accurate, and it only has to look at 20 candidates.
Cast a wide net cheaply, then judge carefully in a small pool. This two stage pattern is the highest impact change most naive RAG pipelines can make, and it costs about thirty lines of code.
A note on hybrid search, since it comes up constantly. Dense vectors are semantic and they are genuinely weak at exact tokens: product codes, error numbers, surnames, ERR_4471. Sparse keyword matching, BM25 style, is excellent at exactly those and useless at paraphrase. Qdrant supports both in one collection with named vectors and fuses the result lists. I have not enabled it in the lab yet, and I am not going to pretend otherwise: it is on the list, and the honest reason it is not done is that my corpus is prose rather than identifiers, so the benefit would be theoretical.
8. Reranking
A cross encoder reads the question and a candidate passage together and scores their relevance. That is the crucial difference from the embedding model, which encoded them separately and never got to compare them directly.
bge-reranker-base scoring 20 candidates against one question on my 4 vCPU node takes 1.4 to 2.2 seconds.
That is not free, and on faster generation hardware it would be a noticeable share of total latency. In this lab, where generation takes 40 seconds, it is 4 percent of the request for a large accuracy gain, so it is obviously worth it.
The tradeoff flips on a GPU. When generation drops to two seconds, a two second reranker doubles your latency. The mitigations are a smaller reranker, fewer candidates, or running it on the accelerator alongside the model. The right architecture at 40 seconds per answer is not the right architecture at 2 seconds per answer, and it is worth knowing that before the hardware changes rather than after.
9. Context assembly, the prompt, and the answer
Now the arithmetic that CPU inference makes impossible to ignore. The model has a 4,096 token context window, and the answer has to fit in it too.
| Budget item | Tokens |
|---|---|
| Total context window | 4,096 |
| Reserved for the answer | 512 |
| System instructions | about 200 |
| The question, plus chat history | about 300 |
| Formatting, citations, separators | about 100 |
| Left for retrieved context | about 2,980 |
Roughly five chunks of 512 tokens, with room to spare. That is not a coincidence, it is why the reranker keeps five.
"ONLY the provided context." Without it the model blends retrieved facts with training data, and you cannot tell which sentence came from where. This is the difference between a grounded answer and a plausible one.
"say I don't know." You have to give the model an acceptable way to fail. Without an escape hatch, a model asked an unanswerable question will invent something, because inventing is what it does when no continuation is well supported. Explicitly permitting refusal is the cheapest hallucination control available.
"say that they disagree." Contradictory sources are common in real corpora, usually because an old document was never deleted. Silently picking one is the worst possible behaviour. Surfacing the conflict turns a wrong answer into a useful signal about your documentation.
The system refused. No citations, no invention, no confident paragraph about a refund policy that does not exist in this tenant's corpus.
A RAG system that never says "I don't know" is not a well tuned RAG system, it is one that has not been asked a hard question yet. The refusal rate on out of scope questions is a metric, and it should not be zero.
Also look at the timings. Retrieval is 73 milliseconds of a 40 second request. On CPU, generation is 96 percent of the latency, which means every optimisation that is not about generation is optimising noise. Knowing that stops you rewriting a search layer that was never the problem.
10. Evaluating it, because otherwise you are guessing
Here is the discipline that separates a RAG demo from a RAG system: evaluate retrieval separately from generation. They fail independently and they are fixed differently. Grading only the final answer tells you something is wrong and nothing about where.
| Stage | Metric | What it means | What a bad score points at |
|---|---|---|---|
| Retrieval | Recall at k | Did the right chunk appear in the top k at all | Chunking, embedding model, missing prefixes, ef too low |
| Retrieval | MRR | How high up the right chunk landed | Reranking, or the lack of it |
| Generation | Faithfulness | Is every claim in the answer supported by the context | Prompt, or a model too small for the task |
| Generation | Answer relevance | Does it answer the question that was asked | Prompt, or context crowded with near misses |
| End to end | Refusal rate on unanswerable questions | Does it decline when it should | Missing escape hatch in the prompt, or a threshold set too low |
The golden set does not need to be large. Fifty question and answer pairs, written by hand against documents you know, will find more real problems than any benchmark.
Reranking moved recall at 5 from 0.72 to 0.88 and MRR from 0.48 to 0.71. That is a large win from one component, and I would not have known its size without measuring it.
Larger chunks made it worse, from 0.88 down to 0.79, which is the chunking tradeoff from section 4 showing up as a number instead of an opinion.
And the retrieval only runs take seconds because no model generates anything. Retrieval evaluation is cheap enough to run on every commit, and it is where most regressions actually live. The expensive end to end run is a weekly job, not a gate.
11. Failure modes, named
Every one of these produces a working system that returns 200 and gives bad answers.
| Failure | What it looks like | Where it actually lives |
|---|---|---|
| Embedding mismatch | Answers became nonsense after a deploy, with no errors anywhere | Query and index built by different models. Section 2. Guard it in code |
| Silent truncation | Long chunks retrieve poorly on their second half | 512 token chunks through a 256 token embedding model. Nothing warns you |
| Missing query prefix | Quality is mediocre and nothing is obviously broken | BGE and E5 both expect prefixes. Costs several points of recall |
| Chunk boundary amputation | A retrieved chunk contains a rule with the condition cut off | Chunking. Overlap plus heading breadcrumbs, section 4 |
| Destroyed tables | Model confidently attaches numbers to the wrong label | Parsing. Convert tables to markdown before chunking |
| Stale index | Answers cite a policy that was updated three months ago | Ingestion scheduling. Surface updated_at in the citation so it is visible |
| Missing tenant filter | Nothing looks wrong. It is a data breach | Retrieval. The filter comes from identity, never from the request, section 6.3 |
| Lost in the middle | The correct chunk was in the prompt and the model ignored it | Too much context. Fewer, better chunks beat more chunks, every time |
| Negation blindness | "Can I expense alcohol" retrieves the passage saying alcohol is not reimbursable, and the model answers yes | Both stages. Embeddings barely encode negation, and small models mishandle it. A named limitation, not a bug to fix |
| Confident synthesis across sources | Two chunks from different documents merged into one plausible false statement | Generation. Per claim citations make it visible, which is most of the fix |
| Recall cliff from low ef | One specific document is never retrieved, everything else is fine | HNSW. Raise hnsw_ef and retest before blaming the chunker |
Negation blindness, because it is not fixable at this model size and pretending otherwise is dishonest. Dense embeddings represent topic far better than they represent polarity, and "alcohol is not reimbursable" sits very close to "alcohol is reimbursable" in vector space. The mitigation is a prompt that forces the model to quote the passage, and an evaluation set that deliberately contains negations.
The missing tenant filter, because it is the only failure in that table with a legal consequence. Everything else produces a bad answer. This one produces someone else's document.
Write the filter into the retrieval function itself so that calling it without a tenant is impossible, rather than into the calling code where a future refactor can drop it.
What I would watch
- Recall at 5 on the golden set, run on every ingestion pipeline change. The single most predictive number in the system.
- Refusal rate. A sudden drop means the model started inventing. A sudden rise means retrieval broke.
- Score distribution of top results. If average top scores fall, the corpus and the questions have drifted apart.
- Time since last successful ingestion run, per source. Stale indexes fail silently and by definition.
Qdrant runs here as a StatefulSet on local-path storage, and that quietly makes it a single point of failure for every RAG tenant.
I did not work that out by reasoning about it. I found it in the node loss drill, where I powered off a worker and Qdrant never came back, because its PersistentVolume physically lives on that machine's disk. The stateless workloads shrugged and rescheduled onto the surviving node. Qdrant sat in Pending forever.
Raising the replica count does nothing when the storage is pinned to a switched off disk. The real answers are replicated storage or Qdrant's own distributed collections, and neither one is built. The full write up is in reliability and failure drills.
12. Where this leaves Volume 2
Everything in this volume is now used in a single request. A tenant application calls the gateway with a virtual key. The gateway authenticates, budgets and routes. The RAG service embeds through that same gateway, searches Qdrant with a tenant filter, reranks, assembles a prompt inside a real token budget, and generates through the gateway again against a model served by vLLM, deployed either as a plain Deployment or as a KServe InferenceService whose version came from MLflow.
That is the serving layer, complete, on hardware that cannot run a large model, doing every job except being fast.
Next
The volume gate is in the overview. Worth rereading now that the pieces exist, because the fourth tab reads differently once you have argued with the tools.
Volume 3, the platform control plane takes everything here and hides it. A developer should not write an InferenceService, a LiteLLM key request and a Qdrant collection. They should say what they want, and a FastAPI service should produce all three. Every abstraction in this volume was chosen with that generator in mind.
Volume 4 then makes it observable, secure and accountable: Langfuse on the traces the gateway is already emitting, Prometheus on the metrics vLLM is already exposing, tenant isolation enforced rather than conventional, and a cost model that finally puts a real number on the token counts sitting in the spend log.