Why RAG Evaluation Is So Difficult
Evaluating a RAG system is far more difficult than evaluating a traditional classification or regression model. There are three reasons. First, the effectiveness of RAG depends on the synergy of two components: whether the documents retrieved by the retriever are relevant (retrieval quality), and whether the answers generated by the generator based on these documents are accurate (generation quality). If either part goes wrong, the overall effect will be compromised. Second, evaluation criteria are highly subjective—what counts as a "good answer"? Different users and scenarios may have completely different standards. Third, there is a lack of standardized test sets—each RAG application has a different knowledge base, making it difficult for generic evaluation datasets to reflect real-world performance.
However, without measurement, there is no improvement. You need to know in which aspects your RAG system performs well and which aspects need optimization, so you can make targeted improvements. This article will establish a three-level RAG evaluation framework to help you upgrade from "feels good" to "proven by data."
Level 1: Retrieval Quality Evaluation
Retrieval quality evaluation focuses on whether the retriever finds the correct documents. Core metrics include: Recall@k (the proportion of relevant documents found among the top k results), which is the most important retrieval metric—if you can't even find the documents, no matter how good the generation is, it's useless. MRR (Mean Reciprocal Rank) (the average of the reciprocal ranks of the first relevant document), measuring the system's ability to rank the most relevant document at the top. NDCG@k (Normalized Discounted Cumulative Gain) (normalized cumulative gain considering rank position weights), which not only considers relevance but also the degree of relevance (partially relevant vs. fully relevant) and rank position.
To calculate these metrics, you need a labeled test set—for each query, label which documents are relevant. This can be done through human annotation (high quality but costly) or by having an LLM automatically annotate (low cost but may have bias).
import numpy as np
from openai import OpenAI
client = OpenAI(api_key="your-deepseek-api-key", base_url="https://api.deepseek.com")
class RetrievalEvaluator:
def __init__(self):
self.metrics = {}
def recall_at_k(self, retrieved_ids, relevant_ids, k):
"""Recall@k: how many relevant documents are covered in the top k results"""
top_k = set(retrieved_ids[:k])
relevant = set(relevant_ids)
if not relevant:
return 0.0
return len(top_k & relevant) / len(relevant)
def mrr(self, retrieved_ids, relevant_ids):
"""MRR: reciprocal of the rank of the first relevant document"""
relevant = set(relevant_ids)
for i, doc_id in enumerate(retrieved_ids):
if doc_id in relevant:
return 1.0 / (i + 1)
return 0.0
def evaluate_all(self, test_queries):
"""Batch evaluation"""
recalls = {1:[], 3:[], 5:[], 10:[]}
mrrs = []
for q in test_queries:
retrieved = q["retrieved_ids"]
relevant = q["relevant_ids"]
for k in recalls:
recalls[k].append(self.recall_at_k(retrieved, relevant, k))
mrrs.append(self.mrr(retrieved, relevant))
result = {
f"Recall@{k}": np.mean(v) for k, v in recalls.items()
}
result["MRR"] = np.mean(mrrs)
return result
evaluator = RetrievalEvaluator()
test_data = [
{"retrieved_ids":[1,3,5,7,9], "relevant_ids":[3,7]},
{"retrieved_ids":[2,4,6,8,10], "relevant_ids":[4,6,10]},
]
results = evaluator.evaluate_all(test_data)
for metric, value in results.items():
print(f"{metric}: {value:.3f}")Level 2: Generation Quality Evaluation
Generation quality evaluation focuses on how well the model generates answers based on the retrieved documents. Core metrics include: Faithfulness: whether the generated answer is faithful to the retrieved documents, without fabricating non-existent information (hallucination). This is the most important generation quality metric for RAG systems. Answer Relevance: whether the generated answer is directly relevant to the user's question, without going off-topic. Context Relevance: whether the retrieved documents are relevant to the user's question (if the retrieved documents are all irrelevant, the subsequent generation is doomed to fail).
The most practical way to evaluate these metrics is to use LLM-as-a-Judge—using a stronger LLM (such as DeepSeek) to evaluate the output of another LLM. Although this introduces some evaluation bias, it is the most feasible solution in the absence of human annotation. The RAGAS framework is the best tool to systematize this approach.
class GenerationEvaluator: def evaluate_faithfulness(self, answer, context): """Evaluate whether the answer is faithful to the context (whether there is hallucination)""" prompt = f"""Please evaluate whether the following answer is faithful to the provided context. Context: {context} Answer: {answer} Please determine whether each statement in the answer can be supported by the context. Output in JSON format: {{ "score": score from 0 to 1, "hallucinations": ["fabricated statement 1", "fabricated statement 2"], "reasoning": "evaluation reasoning" }}""" response = client.chat.completions.create(model="deepseek-chat", messages=[{"role":"user","content":prompt}], temperature=0.1 ) return response.choices[0].message.content def evaluate_relevance(self, answer, question): """Evaluate whether the answer is directly relevant to the question.""" prompt = f"""Question: {question} Answer: {answer} Evaluate whether the answer directly addresses the question without deviating. JSON: {{"score": 0-1, "reasoning":""}}""" response = client.chat.completions.create( model="deepseek-chat", messages=[{"role":"user","content":prompt}], temperature=0.1 ) return response.choices[0].message.content eval_gen = GenerationEvaluator() faithfulness = eval_gen.evaluate_faithfulness( "Python is an interpreted language.", "Python is a programming language created by Guido van Rossum, supporting multiple programming paradigms." ) Layer 3: End-to-End Evaluation
End-to-end evaluation assesses the performance of the entire RAG system from the user's perspective. Common methods include: A/B testing (randomly assigning users to different versions of the RAG system and comparing key business metrics such as satisfaction and task completion rate), human evaluation (having annotators score according to unified criteria, including four dimensions: accuracy, completeness, fluency, and usefulness), and automated benchmarking (using public RAG evaluation datasets such as RGB, CRUD-RAG, etc. for standardized evaluation). End-to-end evaluation is the ultimate touchstone—no matter how high the retrieval and generation scores are, if users are not satisfied, the system is a failure.
Establishing a Continuous Evaluation System
RAG evaluation is not a one-time activity but should be an ongoing process. It is recommended to establish a three-tier evaluation system: before launch, use offline evaluation (labeled datasets + automated metrics) to ensure basic quality; after launch, use online evaluation (collecting user feedback + sampling manual review) for continuous monitoring; and conduct regular regression evaluations (weekly/monthly) to prevent system degradation. Visualize evaluation results on a dashboard so the team can intuitively see trends in various metrics.
RAGAS Framework in Practice
RAGAS (Retrieval Augmented Generation Assessment) is currently the most popular open-source RAG evaluation framework. It provides a standardized set of evaluation metrics and automated evaluation processes. The core idea of RAGAS is to decompose each evaluation dimension into sub-questions that an LLM can judge, allowing models like DeepSeek to automatically score them. The typical workflow using RAGAS: prepare evaluation data (including questions, ground truth answers, retrieved contexts, and generated responses), call RAGAS's evaluation functions, and obtain metrics such as faithfulness, answer_relevancy, context_precision, and context_recall. It is important to note that LLM-as-Judge, while convenient, is not 100% reliable. It is recommended to periodically (e.g., monthly) extract a sample of evaluation cases for manual review to calibrate the accuracy of automated scoring.Online Evaluation and Production Monitoring: Offline evaluation only reflects the model's performance on the test set and cannot fully represent real-world online effectiveness. In production environments, online evaluation metrics should be set up—user like rate, dislike rate, copy rate (whether users copy the AI's answer), share rate, and whether users continue to ask follow-up questions (if users repeatedly ask similar questions, it indicates the AI did not answer well the first time). Although these metrics are less precise than offline metrics, they reflect user experience in real time and are suitable as north star metrics for daily monitoring. It is recommended to display both offline evaluation trends (updated weekly) and online metric trends (updated in real time) on the operations dashboard, comparing the two to gain a comprehensive understanding of the RAG system's quality.
Want to orchestrate this skill chain yourself?
Open in Skill Chain →