How to Build a Retrieval Layer Without Overengineering
Every RAG tutorial you will find starts the same way: set up Pinecone, install LangChain, design your chunking strategy, build your embedding pipeline, configure your retrieval chain. By the time you have followed the instructions, you have assembled a system with six moving parts before you have answered a single user query.
That architecture is correct for production systems at scale. It is the wrong starting point for a team building their first retrieval layer for a product with 50 users and a document set that fits in a single folder.
This post gives you the lightweight path first. Start with the simplest thing that works, understand exactly when it stops being sufficient, and know what to add and in what order when you hit those limits.
Why Most RAG Tutorials Set You Up to Overbuild
The RAG tutorials that rank well are written by teams who have solved the problem at scale. They teach the architecture they arrived at after hitting limits, not the architecture they should have started with.
The full enterprise RAG pipeline blueprint for Azure environments is the right reference for teams building at enterprise scale with strict data residency and integration requirements. For a developer building their first retrieval layer for a SaaS product or internal tool, that level of infrastructure is premature complexity.
The cost of overbuilding your first retrieval layer is not just wasted setup time. It is the ongoing maintenance overhead of a system with more components than your use case requires. Vector databases need to be kept in sync with your document store. Embedding pipelines need to be re-run when documents change. Chunking strategies need to be tuned when retrieval quality drops. Each of these is the right problem to solve at the right scale. Solving all of them before you have validated that your retrieval layer works at all is the wrong order of operations.
Three common overengineering patterns in first RAG builds:
- Reaching for a managed vector database before the document set is large enough to require one. Chroma running locally or pgvector on an existing PostgreSQL instance handles most first-version retrieval needs without the operational overhead of a managed service.
- Building a complex chunking strategy before understanding how the documents will actually be queried. Fixed-size chunking with a sensible overlap is adequate for most document types until you have evidence that it is not.
- Adding a retrieval chain abstraction before the retrieval logic is stable. LangChain and similar frameworks add significant value at complexity. They add friction at simplicity.
The Simplest Retrieval Layer That Actually Works
The minimum viable retrieval layer for most first-version use cases is: embed your documents, store the embeddings, accept a query, embed the query, find the closest document chunks, pass them to the model as context.
That is the complete system. Everything else is an optimisation on top of it.
How retrieval fits into a production AI workflow system covers the broader pipeline context. Within that pipeline, the retrieval layer’s job is to reduce the search space from all available documents to the three to five most relevant chunks for the current query. How you accomplish that reduction is what varies by scale and use case.
For a document set under approximately 10,000 chunks, the simplest implementation that works:
Step 1: Embed your documents. Use the OpenAI Embeddings API (text-embedding-3-small is the current cost-efficient option) or a locally hosted model like nomic-embed-text if you need to keep data on-premise. Run each chunk through the embedding model and store the resulting vector alongside the chunk text.
Step 2: Store embeddings in the simplest available store. For a prototype or early product, a JSON file or a SQLite database with a vector column is sufficient. For a product already running on PostgreSQL or Supabase, pgvector adds vector similarity search to your existing database with a single extension install. No new infrastructure required.
Step 3: At query time, embed the query and find closest chunks. Compute cosine similarity between the query embedding and all stored document embeddings. Return the top three to five chunks above a similarity threshold. Pass those chunks as context to your model.
Step 4: Pass retrieved chunks as context. Structure your prompt as: system instructions, then retrieved context clearly labelled, then the user query. Tell the model explicitly to answer based on the provided context and to say when the context does not contain the answer.
This four-step implementation handles the majority of first-version retrieval use cases. The total infrastructure required is an embedding API call and whatever database you are already running.
Chunking and Embedding Without a Vector Database
Chunking strategy is where most developers spend too much time too early. The right approach is to start simple, evaluate retrieval quality on real queries, and adjust only when you have evidence that the current approach is failing.
For most document types, fixed-size chunking with overlap is the right starting point. Chunk size of 500 to 800 tokens with 100 to 150 token overlap handles the majority of prose documents adequately. The overlap prevents relevant information from being split across chunk boundaries and lost.
Three chunking decisions that matter more than chunk size:
- Chunk at natural boundaries where possible. Split on paragraph breaks or section headings before falling back to fixed-size splitting. A chunk that ends mid-sentence is harder for the model to use than one that ends at a paragraph boundary, even if both are the same token count.
- Keep metadata with each chunk. Store the source document name, section heading, and page or position alongside each chunk. This lets you include source attribution in responses and filter retrieval by document type when needed.
- Do not chunk short documents. If a document is under 500 tokens, store it as a single chunk. The overhead of splitting and reassembling short documents introduces complexity without retrieval benefit.
On embedding models: the choice matters less than most tutorials suggest at small scale. OpenAI’s text-embedding-3-small produces good retrieval quality at low cost for English-language documents. If your documents are multilingual, multilingual-e5-large is the standard open-source alternative. Do not spend time benchmarking embedding models before you have a working system. Pick one and move on. The chunking and query design decisions have more impact on retrieval quality than embedding model selection at this scale.
Storage without a vector database: FAISS is the standard library for in-memory similarity search without a database. It handles millions of vectors efficiently on a single machine and requires no server infrastructure. For a document set that fits comfortably in memory, FAISS running in your application process is operationally simpler than any database-backed alternative. The trade-off is that it does not persist across restarts without serialising the index to disk, which requires one additional step in your deployment process.
When to Introduce a Vector Database
The right time to introduce a managed vector database is when you have a specific problem that a simpler solution cannot solve. Not before.
Four signals that indicate you have hit the limits of the lightweight approach:
| Signal | What It Means | What to Add |
|---|---|---|
| Query latency increases as document set grows | In-memory similarity search is hitting scale limits | pgvector with an IVFFlat index, or Chroma locally |
| Document set exceeds available memory | FAISS index no longer fits in application memory | Database-backed vector store (pgvector or Chroma) |
| Documents update frequently and sync is a problem | Embedding pipeline needs to be decoupled from application | Dedicated embedding pipeline with change detection |
| Multi-tenancy requires per-tenant document isolation | Single shared index cannot enforce tenant boundaries | Namespaced collections in Chroma or Pinecone |
| Retrieval quality plateaus and re-ranking is needed | Top-k retrieval is not sufficient, order matters | Add a cross-encoder re-ranking step after retrieval |
If none of these signals have appeared, you do not need a vector database yet. Adding one before you hit these limits adds operational complexity without retrieval quality benefit.
When you do introduce a vector database, the decision between options is straightforward for most teams. If you are already running PostgreSQL or Supabase, pgvector is the lowest-friction path: one extension install, no new infrastructure, vector columns sit alongside your existing data. If you need a dedicated vector store and are not already on PostgreSQL, Chroma running locally is the right starting point before graduating to Pinecone or Weaviate if you need managed infrastructure at scale.
Query Design: The Part Most Tutorials Skip
Retrieval quality is determined as much by how you construct the query as by how you store and index the documents. Most RAG tutorials spend significant time on chunking and embedding and almost none on query design. This is the wrong emphasis.
The query passed to your retrieval layer is not necessarily the user’s raw input. In most production use cases, the raw user input is a poor retrieval query for one of three reasons: it is too short and lacks context, it contains pronouns that reference earlier conversation turns, or it asks a question in natural language that does not match the language of the documents.
How query design affected retrieval accuracy in a real knowledge base build covers a concrete example of this problem and how it was resolved. The pattern that emerges consistently: query rewriting before retrieval improves retrieval accuracy more reliably than any change to the embedding or chunking layer.
Three query design patterns that improve retrieval quality without adding infrastructure:
Query expansion. Before retrieval, use the model to rewrite the user’s query into two or three alternative formulations. Retrieve against all formulations and merge the results before deduplication. This addresses the vocabulary mismatch problem where the user’s phrasing does not match the document’s phrasing. A user asking “how do I cancel my subscription” and a document containing “terminating your account” are semantically similar but may not retrieve well on direct embedding comparison.
Conversation context injection. In multi-turn conversations, the user’s current query often relies on context from earlier turns. “What about the pricing?” is not a useful retrieval query without knowing what product was being discussed. Before retrieval, rewrite the query to incorporate relevant context from the conversation history. A simple approach: pass the last two to three conversation turns to the model and ask it to produce a self-contained retrieval query.
Hypothetical document embedding (HyDE). Instead of embedding the query directly, ask the model to generate a short hypothetical document that would answer the query, then embed that document and use it for retrieval. This works well when the query vocabulary is very different from the document vocabulary, such as when users ask questions in plain language about technical documentation.
The Upgrade Path: From Simple to Scale
The retrieval layer upgrade path follows a predictable sequence. Each step adds complexity only when the previous step is insufficient.
Level 1: In-memory FAISS + OpenAI Embeddings. Document set under 50,000 chunks, single-tenant, documents change infrequently. Setup time: one to two days. No new infrastructure.
Level 2: pgvector on existing PostgreSQL. Document set growing beyond comfortable memory limits, or documents update frequently enough that a persistent store is needed. Multi-tenant isolation via row-level filtering. Setup time: half a day if already on PostgreSQL. No new infrastructure.
Level 3: Chroma (local or hosted). Dedicated vector store needed, or team wants vector-specific features like collection management and metadata filtering without managing a PostgreSQL schema. Setup time: one day. Requires one new service.
Level 4: Pinecone or Weaviate (managed). High query volume requiring horizontal scaling, strict SLA requirements, or multi-region deployment. Setup time: two to three days. Requires managed service subscription and operational integration.
Level 5: Full pipeline with re-ranking and query routing. Retrieval quality is the primary product differentiator and warrants the investment. Cross-encoder re-ranking, hybrid search combining dense and sparse retrieval, query routing to different indices by document type. Setup time: one to two weeks. Multiple new components.
Most products that ship a retrieval layer never need to go beyond Level 2 or 3. The teams that end up at Level 5 are typically those building search as a core product feature, not those using retrieval as a supporting layer in a broader AI workflow.
How retrieval layers fit into a broader autonomous agent architecture covers what happens when retrieval becomes one tool among many in an agent system, which is the architecture that makes sense once your retrieval layer is stable and you are ready to extend what the agent can do.
For teams at the point where retrieval quality has become the primary engineering challenge and the upgrade path above is not sufficient, how we design and build production RAG systems for client products covers what a professional architecture engagement looks like at that stage.
Key Takeaways
“The minimum viable retrieval layer for most first-version use cases requires four components: an embedding model, a similarity search mechanism, a document store, and a prompt structure that passes retrieved chunks as context. Every additional component is an optimisation that should be added only when a specific problem requires it.”
“Fixed-size chunking with 500 to 800 token chunks and 100 to 150 token overlap is the correct starting point for most document types. Chunking strategy should only be revisited after retrieval quality has been evaluated on real queries and found to be failing for a specific reason.”
“Query rewriting before retrieval, including query expansion, conversation context injection, and hypothetical document embedding, consistently improves retrieval accuracy more than changes to the embedding or chunking layer. Most RAG tutorials skip query design entirely, which is where the highest-leverage improvements sit.”
“The signal to introduce a vector database is a specific operational problem: document set exceeding memory, query latency degrading at scale, multi-tenancy requirements, or frequent document updates breaking a simple sync approach. In the absence of one of these signals, pgvector on an existing PostgreSQL instance or FAISS in memory is sufficient.”
No. LangChain is useful when you need its abstractions: chain composition, agent tooling, memory management across turns. For a simple retrieval layer, the abstractions add friction rather than removing it. You can build an effective retrieval layer with the OpenAI Python SDK, a similarity search library, and your existing database. Add LangChain when the complexity of your system justifies the abstraction layer, not before.
There is no hard minimum, but RAG adds value when the information you want the model to use is too large to fit in a single context window, changes frequently enough that retraining is not practical, or needs to be kept private and not baked into model weights. For a static document set of five to ten pages that rarely changes, including the full text directly in the prompt is often simpler and equally effective.
Build a small evaluation set of twenty to fifty question and answer pairs drawn from your documents, where you know what the correct source chunk is for each question. Run your retrieval layer against each question and measure what percentage of the time the correct source chunk appears in the top three results. That metric, called recall at k, tells you whether retrieval is working before you integrate the model response layer. Aim for above 80 percent before considering the retrieval layer production-ready.
The model answers based on the wrong context, which produces a confident but incorrect response. This is the primary failure mode of RAG systems and is more dangerous than the model saying it does not know. The mitigations are: improve retrieval quality using the query design techniques above, add a similarity threshold so chunks below a minimum relevance score are excluded, and instruct the model explicitly to say when the retrieved context does not contain the answer rather than attempting to answer from general knowledge.
Start with dense retrieval (embedding-based similarity search). It handles semantic similarity well and is simpler to implement. Sparse retrieval (keyword-based, like BM25) performs better on exact term matching and is less affected by domain-specific vocabulary that the embedding model was not trained on. Hybrid search combining both is the standard approach in production systems where retrieval quality is critical. Start dense, add sparse when you have evidence that exact term matching is failing.
The simplest approach: re-embed the entire document set on a schedule, nightly or weekly depending on update frequency. This works until the document set is large enough that full re-embedding is too slow or expensive. At that point, implement change detection: track a hash of each document, re-embed only documents whose hash has changed since the last run. This is the standard pattern for keeping a retrieval layer in sync with a frequently updated document store without rebuilding the entire index on each update.