RAGLLMsArchitecture

Building RAG Systems With Local LLMs

Architecture decisions behind AI question-answering systems using local models, embeddings, and vector databases.

Mar 15, 20268 min read

Retrieval-Augmented Generation (RAG) has become the de facto architecture for grounding LLM outputs in private or domain-specific knowledge. By combining a retrieval step with a generation step, RAG systems can produce accurate, attributable answers without fine-tuning.

System Architecture

A typical RAG pipeline consists of two phases: indexing and querying. During indexing, documents are split into chunks, embedded into vector representations, and stored in a vector database. During querying, the user question is embedded, similar chunks are retrieved, and the LLM generates an answer conditioned on those chunks.

from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.vectorstores import Chroma
from langchain.chains import RetrievalQA
from langchain_community.llms import Ollama

embeddings = HuggingFaceEmbeddings(
    model_name="BAAI/bge-small-en-v1.5"
)
vectorstore = Chroma(
    collection_name="docs",
    embedding_function=embeddings,
    persist_directory="./chroma_db"
)
llm = Ollama(model="mistral:7b")
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=vectorstore.as_retriever(search_k=4)
)

Choosing an Embedding Model

The embedding model determines how well semantic relationships are captured. For local setups, BGE and E5 models offer strong performance with modest resource requirements. BGE-small-en-v1.5 (384 dimensions) runs comfortably on CPU, while BGE-large-en-v1.5 (1024 dimensions) benefits from GPU acceleration.

Key considerations for embedding model selection:

•Dimensionality: lower dims = faster retrieval, higher dims = better accuracy
•Context window: ensure chunks fit within the model max tokens
•Normalization: cosine similarity requires normalized embeddings
•Domain alignment: some models are fine-tuned for specific domains

Chunking Strategies

Chunk size and overlap significantly impact retrieval quality. Too small and chunks lack context; too large and retrieval precision degrades. A common starting point is 512 tokens with 128 token overlap. RecursiveCharacterTextSplitter handles natural boundaries gracefully.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=128,
    separators=["\n\n", "\n", ".", " ", ""]
)
chunks = splitter.split_documents(raw_documents)

Consider using semantic chunking (e.g., by section or paragraph) rather than fixed token windows for documents with clear structure like Markdown or HTML.

Retrieval Strategies

Beyond naive top-k retrieval, several strategies improve result quality: hybrid search combining vector similarity with keyword BM25, multi-query retrieval that generates variations of the user question, and re-ranking using cross-encoders to refine initial results.

For local deployments, Ollama provides a convenient interface for running models like Mistral, Llama 3, and Qwen on consumer hardware. Quantized versions (GGUF format) reduce memory requirements by 4-8x with minimal quality loss.

Evaluation & Monitoring

RAG evaluation requires measuring both retrieval and generation quality. Hit rate and MRR assess retrieval, while faithfulness, relevance, and answer completeness measure generation. Tools like RAGAS automate this evaluation pipeline using an LLM-as-a-judge approach.

Back to Engineering Notes