Deep Dive: From Beginner to Advanced, Mastering RAG in Large Models

Core Insight

This article addresses the most practical issue when deploying large models: how to enable general-purpose large models to provide reliable answers based on private data and real-time data. Whether you're building a Q&A system for an enterprise knowledge base or encountering the RAG concept for the first time, this article will help you establish a complete understanding framework, ranging from fundamental principles to advanced optimization techniques. The article primarily uses LlamaIndex's implementation as a case study, systematically explaining advanced technologies such as chunking strategies, hybrid retrieval, query transformation, and agent architecture, while also providing feasible evaluation and fine-tuning approaches.

1. Why RAG Has Become a Mandatory Choice for Large Model Applications

Large models have experienced explosive growth over the past two years, but when they are actually integrated into business scenarios, the limitations of the base models become immediately apparent. This gap mainly arises from three levels.

Limitations in knowledge. The training data of mainstream large models (DeepSeek, GPT series, Qwen) all come from the public internet, which means they have no knowledge of real-time information, enterprise private documents, or domain-specific knowledge. A simple example: asking the model about internal company procedures, it can only provide generic, vague answers, because the training data does not contain such content at all.

Hallucination issues. The output of all deep learning models is essentially the result of a series of probability calculations, and large models are no exception. When a model lacks sufficient knowledge about a particular question, it will "seriously fabricate nonsense," a phenomenon known in professional terms as "hallucination." In enterprise scenarios, the consequences of hallucination are far more severe than in consumer applications — an incorrect technical document or a fabricated customer information could lead to real-world losses.

The red line of data security. Almost no enterprise is willing to upload private data to third-party platforms for training or inference. For large companies, data security is an unyielding bottom line. This has led to an awkward situation: relying entirely on the capabilities of general-purpose large models prevents the use of data advantages, while attempting to use the data brings the risk of leakage.

RAG (Retrieval Augmented Generation, retrieval-augmented generation) is a systematic solution proposed specifically to address these three issues. Its idea is straightforward: instead of letting the large model generate answers out of thin air, it is better to first retrieve relevant information from a reliable database and then provide that information as context for the large model to organize its response.

In one sentence: RAG = retrieval technology + LLM prompting. The specific process is as follows: after the user asks a question, the system first retrieves relevant content from various data sources, then injects the retrieval results and the original question into a prompt template, and finally generates the answer using the large model.

The development of this field can be traced back to 2019, when Faiss implemented vector search technology based on embeddings. Since then, several startups such as Chroma, Weaviate, and Pinecone have emerged, offering vector databases. Most of these are built upon open-source indexing engines like Faiss and NMSLib, adding metadata storage and other auxiliary tools on top of vector retrieval. Today, RAG has a wide range of applications — from question-answering services based on web search and LLMs, to fully private data-driven applications.

Currently, the three most well-known open-source RAG frameworks are LangChain, LlamaIndex, and Dify. This article mainly references the implementation of LlamaIndex to explain advanced RAG techniques, which will be the focus of this article.

II. RAG Basic Workflow: From Data Processing to Answer Generation

The core of the RAG architecture can be understood as a combination of "retrieval + generation." The vector database is responsible for efficiently storing and recalling target knowledge, while the large model uses the retrieved knowledge to generate reasonable responses. The complete workflow is divided into two phases: offline data preparation and online application.

Data Preparation Phase

This is the offline data processing phase, with the goal of vectorizing proprietary data and building an index.

Data Extraction addresses the issue of unifying multi-source and heterogeneous data. It involves loading data in various formats (PDF, Word, HTML, database records, etc.), performing filtering, compression, and formatting, while also extracting metadata such as filenames, titles, and timestamps.

Text Segmentation is one of the key steps that affect retrieval performance. Text segmentation must balance two factors: one is the token length limit of the embedding model, as different models can handle varying sequence lengths; the other is the impact of semantic integrity on retrieval quality. Common strategies include sentence segmentation (splitting based on periods, line breaks, etc., preserving the semantic completeness of full sentences) and fixed-length segmentation (such as 512 or 1024 tokens, but this may result in loss of semantic information, which is typically mitigated by adding redundancy at the beginning and end).

Vectorization is the process of converting text into a vector matrix, directly influencing retrieval accuracy. Currently, mainstream embedding models include OpenAI's ChatGPT-Embedding, Baidu's ERNIE-Embedding V1, and open-source models such as the M3E series (Hugging Face: moka-ai/m3e-base) and the BGE series from the Zhuyuan Institute (Hugging Face: BAAI/bge-base-en-v1.5). For vertical scenarios involving rare specialized terms, it is advisable to fine-tune open-source models or train custom embedding models.

Application Phase

The core of the online phase is data retrieval and prompt injection.

Common retrieval methods include Similarity Retrieval (calculating cosine similarity, Euclidean distance, or Manhattan distance between the query vector and stored vectors, and returning the highest-scoring records) and Full-Text Retrieval (building an inverted index based on keywords). In practice, multiple retrieval methods are often combined to improve recall.

Prompt design directly affects the quality of the output. A standard knowledge-based question-answering prompt consists of three parts: task description (telling the model what role to play), background knowledge (the retrieved relevant text), and task instruction (the user's specific question). For example:

【Task Description】
Assume you are a professional customer service chatbot. Please refer to the 【Background Knowledge】 to answer the user's question:
【Background Knowledge】
{content}
【Question】
What is the battery life of the Stone robot vacuum P10?

Prompt design does not have a fixed syntax and heavily relies on personal experience. It requires iterative optimization based on the actual output of the large model.

III. Six Key Technical Directions for Advanced RAG

The basic RAG workflow addresses the issue of "being usable," but in complex scenarios, there are still significant shortcomings in retrieval accuracy, context utilization efficiency, and inference quality. The goal of advanced RAG techniques is to approach the upper limits of these three metrics within an acceptable cost range.

1. Chunking and Vectorization

The input sequence length of Transformer models is fixed. Even with a very large context window, the vector of a single sentence or a few sentences is more representative of its semantic meaning than the vector of several pages of text. The chunk size depends on the Token capacity of the Embedding model: sentence transformers based on BERT support up to 512 Tokens, while OpenAI's ada-002 can handle 8191 Tokens. However, there is a cost—more Tokens mean that the embedding of a single text chunk becomes less specific, which can reduce retrieval accuracy. LlamaIndex provides the NodeParser class, which supports custom text splitters, metadata, and node relationships.

For vectorization, it is recommended to first check the MTEB leaderboard (a retrieval model benchmark platform) and choose an Embedding model optimized for search, such as bge-large or the E5 series.

2. Multiple Forms of Search Indexes

The simplest implementation is a flat index—i.e., calculating the distance between the query vector and all chunk vectors in a brute-force manner. However, when the data scale reaches the tens of thousands or higher, it is necessary to introduce approximate nearest neighbor algorithms (such as clustering, tree structures, HNSW), with implementations including Faiss, NMSLib, and Annoy.

Hierarchical indexing is an effective solution for large databases. It involves creating a two-tier index: the first layer consists of document summaries, and the second layer is the document chunks. During search, the system first filters relevant documents using the summary layer, then performs a detailed search within those relevant documents. This significantly reduces unnecessary computations.

Hypothetical questions and HyDE are two methods that leverage LLMs to improve retrieval quality. The former has the LLM generate a hypothetical question for each document chunk and embed it as a vector. During retrieval, the system searches for the question vector instead of the document chunk vector—typically, the semantic similarity between the query and the hypothetical question is higher than that between the query and the document chunk. HyDE (reverse logic) requires the LLM to generate a hypothetical answer based on the query, and then use the vector of this answer along with the original query vector for the search.

Sentence window retriever and auto-merge retriever address the contradiction between "retrieval accuracy" and "context completeness." The sentence window retriever embeds each sentence in the document individually, achieving very high retrieval accuracy. After identifying the most relevant sentence, it sends that sentence along with the preceding and following k sentences as context to the LLM. The auto-merge retriever follows a similar logic: the document is split into smaller child blocks and larger parent blocks. If multiple child blocks retrieved via Top-K are linked to the same parent node, the system automatically replaces the child blocks with the parent node as context. These methods essentially "fallback" to a larger context window after precise retrieval, providing the LLM with sufficient reasoning basis.

3. Hybrid Search and Re-ranking

Pure vector search overlooks a key fact: exact keyword matching still holds irreplaceable value in many scenarios. Hybrid search combines traditional keyword search (TF-IDF, BM25) with modern semantic search, integrating similarity scores from different sources using the Reciprocal Rank Fusion (RRF) algorithm. LangChain implements this workflow via the Ensemble Retriever class, and LlamaIndex also has a similar implementation. Hybrid search typically delivers superior retrieval results because it considers both semantic similarity and keyword matching.

Re-ranking after retrieval is equally important. You can filter results using similarity scores, keywords, or metadata, or use a cross-encoder (such as sentence-transformer) or a re-ranking interface like Cohere to finely rank the candidate results. This step represents the final calibration opportunity in the entire retrieval pipeline.

4. Query Rewriting

When a user's original query is too complex or ambiguous, direct retrieval often yields poor results. Query rewriting involves using the LLM as a reasoning engine to rewrite and decompose the user's input.

Take a classic example: "Which of the two frameworks, LangChain and LlamaIndex, is more popular on GitHub?" The answer is unlikely to appear directly in the corpus, and direct retrieval may yield nothing. The correct approach is to decompose it into two sub-queries: "How many stars does LangChain have on GitHub?" and "How many stars does LlamaIndex have on GitHub?"—these sub-queries are executed in parallel, and the retrieval results are then merged and sent to the LLM. LangChain's multi-query retriever and LlamaIndex's sub-question query engine implement this functionality.

Another approach is Step-back prompting, where the LLM generates a more general query to retrieve higher-level context, which is then merged with the results of the original query and fed back into the LLM. Query rewriting involves having the LLM directly rewrite the initial query to improve retrieval quality.

5. Chat Engine and Query Routing

To build a RAG system capable of multi-turn conversations, it is necessary to handle context compression—subsequent user questions often contain pronouns or depend on the context of previous messages. The logic of ContextChatEngine is to first retrieve the context relevant to the current query, then send the chat history together to the LLM. A more complex approach, CondensePlusContextMode, compresses the chat history and the last message into a new query, which then proceeds to the index retrieval. LlamaIndex also supports a chat engine based on OpenAI agents, offering more flexible interaction modes.

Query routing is a decision-making step driven by the LLM, determining what the next action should be based on the user's query—whether to perform data index search, call a summary index, or trigger other operations. In multi-document storage scenarios, the summary index and document chunk vector index are two different levels of index systems, and the query router is responsible for selecting the appropriate path. Both LlamaIndex and LangChain provide support for query routing.

6. Agents

The Agent architecture has existed almost since the first LLM API was released—the core idea is to equip the LLM with a set of tools, enabling it to make autonomous decisions and call external functions. These tools can be code functions, external APIs, or even other agents. The name "LangChain" originates from this idea of "linked calling."

In RAG scenarios, the multi-document agent approach is a typical application of Agent architecture: an OpenAIAgent is initialized for each document, and each agent has two tools—vector storage index and summary index—deciding which to use based on query routing. Additionally, a top-level agent is responsible for distributing the query to each document agent and synthesizing the final answer. The advantage of this architecture is that it can compare solutions or entities across different documents, covering common use cases from single-document summarization to cross-document QA.

The drawback is the longer response time—within the agent, multiple round-trip calls between multiple LLMs are required. LLM calls are typically the most time-consuming operation in a RAG pipeline, so for large multi-document scenarios, it's necessary to weigh whether the trade-off of sacrificing speed for this flexibility is worthwhile.

7. Response Synthesis

This is the endpoint of the RAG pipeline, where the final answer is generated based on the complete retrieved context and the user's query. The most straightforward approach is to concatenate all content above the relevance threshold with the query and send it to the LLM. More complex options include: sending the context in chunks and allowing the LLM to optimize the answer segment by segment; first summarizing the retrieved context before generating the response; or generating multiple versions of the answer based on different context blocks and then merging them.

IV. Quality Evaluation and Fine-tuning: How to Make RAG Truly Usable

The evaluation of RAG systems can be conducted along three dimensions: whether the retrieved context is relevant to the question (retrieval quality), whether the generated answer is supported by the context (faithfulness), and whether the answer addresses the user's question (relevance). Among these, the most critical and controllable metric is retrieval quality—most advanced RAG techniques are designed around improving this metric; the faithfulness and relevance of answers mainly depend on LLM capabilities and prompt design.

Evaluation Framework

The Ragas framework evaluates the quality of generated responses using factual accuracy and answer relevance, and assesses retrieval performance using context precision and recall. LlamaIndex and Truelens introduced the "RAG triplet" evaluation model (context relevance, answer groundedness, answer relevance). OpenAI has released a practical evaluation guide, recommending the addition of more refined metrics such as Mean Reciprocal Rank (MRR) in addition to hit rate.

The LangSmith framework provided by LangChain not only allows for custom evaluators but also monitors the entire RAG pipeline during operation, enhancing system transparency. With LlamaIndex, you can directly try the rag_evaluator llama pack.

The Limits of Fine-tuning Benefits

Fine-tuning the embedding encoder yields limited benefits—transformer encoders optimized for search are already highly efficient, and additional fine-tuning is unlikely to provide significant breakthroughs. Fine-tuning the ranker (cross-encoder) is an optional approach.

Quantitative data supports the practical effectiveness of LLM fine-tuning: after fine-tuning GPT-3.5-turbo using LlamaIndex, the faithfulness metric in the Ragas framework improved by 5%, indicating that the fine-tuned model can better utilize the provided context. Meta AI's RA-DIT study demonstrated a more complex method that simultaneously fine-tunes the LLM and the retriever. On knowledge-intensive tasks, the fine-tuned Llama2 65B model showed an improvement of approximately 5% compared to the original RAG approach.

However, these gains come at a cost—fine-tuning a base model using a small synthetic dataset may weaken the model's general capabilities.

V. Applicable Boundaries: RAG Fusion Needs Evaluation

The original text specifically discusses the RAG fusion (using an LLM to generate multiple queries to cover different aspects of a question) and its pros and cons.

Advantages include: obtaining richer and more diverse context, providing an additional layer of control, and automatically correcting spelling errors in user input and supplementing related information through the LLM. In terms of cost, there are also unique benefits—generating multiple queries typically requires only around 100 tokens for an LLM call, whereas the main retrieval process can consume thousands of tokens, with the cost difference ranging by one to two orders of magnitude. Even if the fusion approach can only reduce one follow-up question per ten queries, it is still cost-effective.

Limitations are equally apparent. Latency is the biggest trade-off—each additional LLM call introduces a few hundred milliseconds of waiting time, which significantly impacts the interactive experience. Failure in automatic error correction may occur in scenarios involving internal terminology or jargon—the LLM's training data may not include these terms, leading it to generate completely irrelevant queries that directly skew the retrieval results. For example, when the topic of a paper is the Transformer architecture, the LLM might misinterpret the term "attention" in context and generate queries unrelated to the topic.

Improvement directions include: providing clearer system prompts (explicitly defining application scenarios), using semantic search to identify similar queries, adding a few examples before prompting, or fine-tuning a small LLM specifically for query generation. The last approach, although more costly, is sufficient for query generation tasks under specific templates, even with a small model having only a few million parameters.

The core recommendation is: after building a basic RAG application, immediately establish an evaluation process. Every adjustment to the prompt or search may improve the effectiveness of one type of query but sacrifice the effectiveness of another. The most rational approach is to treat it as a machine learning problem and let the data tell you the answer.

VI. Next Steps for Consideration

RAG is being integrated with more technologies. RAG based on web search has already emerged; exploration of agent-based architectures is still deepening; research on the long-term memory of LLMs is also progressing. The issue of speed remains the biggest challenge for RAG systems in production — ChatGPT and other mainstream assistants use streaming output, essentially to shorten the perceived response time by users. This is also the reason why small-parameter LLMs are receiving significant attention; the emergence of models like Mixtral and Phi-2 is driving this direction.

---

References:

• PAI by Guokr: A Comprehensive Guide to RAG Applications in Large Models (with Practical Cases)
• iyacontrol: Visual Guide to Advanced RAG Techniques
• ketchum: Building a Chatbot Based on RAG (Part 4): RAG Fusion