RAG Caching and Performance Optimization

Performance Bottleneck Analysis of RAG Systems

A typical RAG request chain is: Embedding query → Vector retrieval → Document reranking → Context concatenation → LLM generation. In actual tests, vector retrieval and LLM generation each account for about 40% of the latency, while Embedding and reranking account for 20%. Optimization needs to start from each step, but the most valuable optimization point is caching—many queries are repeated or similar, and caching can reduce P99 latency from 3 seconds to less than 100 milliseconds.

Multi-Level Cache Architecture

Three levels of cache are recommended: L1-Exact Match Cache (identical queries → directly return cached result, using Redis String, TTL=1 hour), L2-Semantic Similarity Cache (queries with similarity > 0.95 → reuse cached result, using vector index + semantic deduplication), and L3-Document Fragment Cache (precomputed Embedding cache for high-frequency retrieved document fragments, using Redis Vector Set). L1 hit rate is about 15-25%, L2 hit rate is about 20-30%, and L3 can reduce retrieval latency by 50-70%.

Redis Cache in Practice

import redis, json, hashlib
import numpy as np
from openai import OpenAI

client = OpenAI(api_key="your-deepseek-api-key", base_url="https://api.deepseek.com")
redis_client = redis.Redis(host="localhost", port=6379, decode_responses=True)

class RAGCache:
    def __init__(self, similarity_threshold=0.95):
        self.threshold = similarity_threshold

    def _hash(self, text):
        return hashlib.md5(text.encode()).hexdigest()

    def get_exact(self, query):
        """L1: Exact match cache"""
        return redis_client.get(f"rag:exact:{self._hash(query)}")

    def set_exact(self, query, result, ttl=3600):
        redis_client.setex(f"rag:exact:{self._hash(query)}", ttl,
                           json.dumps(result, ensure_ascii=False))

    def get_semantic(self, query_embedding):
        """L2: Semantic similarity cache—find similar historical queries"""
        # Redis vector similarity search
        results = redis_client.ft("rag_semantic_idx").search(
            redis.commands.search.Query(
                f"*=>[KNN 1 @embedding $vec AS score]")
            .return_fields("query", "result", "score")
            .dialect(2),
            {"vec": np.array(query_embedding, dtype=np.float32).tobytes()}
        )
        if results.docs and float(results.docs[0].score) > self.threshold:
            return json.loads(results.docs[0].result)
        return None

    def embed_query(self, query):
        resp = client.embeddings.create(
            model="text-embedding-3-large", input=query)
        return resp.data[0].embedding

    def retrieve(self, query, retriever_func):
        """Retrieval with cache"""
        # L1 check
        cached = self.get_exact(query)
        if cached:
            return {"source": "L1-cache", "result": json.loads(cached)}
        # L2 check
        emb = self.embed_query(query)
        cached = self.get_semantic(emb)
        if cached:
            return {"source": "L2-cache", "result": cached}
        # Actual retrieval
        result = retriever_func(query)
        self.set_exact(query, result)
        return {"source": "retriever", "result": result}

cache = RAGCache()
# Usage: cache.retrieve("What is RAG?", my_retriever_func)

Batch Optimization and Asynchronous Processing

Batch Embedding: Combine multiple queries into one API call; the Embedding API supports batch input (up to 2048 entries), reducing Embedding stage latency by 90%. Asynchronous Retrieval: Vector retrieval and document reranking are IO-intensive operations; using asyncio to execute concurrently can reduce multi-path retrieval latency from serial N×t to approximately max(t). Connection Pooling: Use connection pools for Redis and vector databases to avoid frequent connection setup overhead. Preloading Hot Data: Based on statistics, preload the top 20% most frequently accessed documents into memory.

Cost Optimization

In addition to performance, caching also significantly reduces API call costs. Taking OpenAI Embedding as an example, text-embedding-3-large costs $0.13 per million tokens—seemingly cheap, but 100,000 queries per day costs over $5,000 annually. Exact cache can reduce Embedding calls by 15-25%, semantic cache additionally reduces by 20-30%, and the combination saves $2,000-3,000 per year. For high-frequency LLM generation calls, if caching the final reply (rather than intermediate retrieval results), the savings are even more significant.

Cache Invalidation Strategy and Consistency Guarantee

After introducing caching, RAG systems face a classic problem: cache consistency and timeliness. When knowledge base documents are updated, related cache entries need to be invalidated. Our solution is based on

Cold Start Optimization for RAG Systems

When a RAG system is newly launched or after cache clearing, the first few thousand queries experience a "cold start"—each requiring the full Embedding + vector retrieval + LLM generation pipeline, with P99 latency potentially 10 times that of steady state. Our cold start optimization strategies: Warm-up script—extract the Top 1000 high-frequency queries from historical query logs (after desensitization), and use a warm-up script to batch execute these queries to populate the cache at service startup or after cache clearing; Progressive traffic ramp-up—after a new Pod starts, it first enters "warm-up mode" (receiving only 10% traffic), and only after cache hit rate reaches above 60% does it receive 100% traffic; Cache persistence—periodically dump the L1 exact-match cache to disk, and restore the cache from disk when a new Pod starts. With the warm-up mechanism, cold start P99 latency drops from 2.8 seconds to 0.5 seconds, significantly reducing the sense of disruption in user experience.

Implementation Details and Tuning of Semantic Cache

The core challenge of semantic cache (L2 cache) is identifying queries that are "similar but not identical." If the similarity threshold is set too high (>0.98), there are few hits; if set too low (<0.8), it may return irrelevant cached results. Our tuning experience: Threshold setting—for factual queries (e.g., "Python version"), a similarity threshold of 0.92 is sufficient; for open-ended queries (e.g., "write me a poem"), even a similarity of 0.98 should not use the cache—poems should be original. In implementation, we attach a "query_type" tag to each query and decide whether to allow semantic cache based on the type. Cache eviction strategy—dual eviction based on TTL + LFU (Least Frequently Used), where popular queries automatically extend TTL, and unpopular queries may be evicted even before TTL expires. This fine-grained strategy reduces the "bad hit rate" (returning irrelevant cache) of semantic cache from 8% to 1.5%.