Prompt Compression and Semantic Caching Engineering in Practice

1. Why Prompt Compression is the Cornerstone of Semantic Caching

In the engineering implementation of DeepSeek models, the longer the prompt, the super-linear growth in latency and cost per request. In practice, a prompt with complete few-shot examples (about 2000 tokens) has a response time over 40% higher than a concise version (about 500 tokens). More critically, the effectiveness of semantic caching heavily depends on the structural stability of prompts—if each request's prompt is semantically equivalent but textually different, the cache hit rate will drop sharply.

Therefore, prompt compression is not only a means to reduce costs and improve efficiency, but also a prerequisite for semantic caching to 'recognize' duplicate requests. What we need to do is not simple truncation, but generating a semantic fingerprint by extracting 'core intent + key constraints'. This fingerprint must be stable and insensitive to irrelevant modifications (such as polite phrases, redundant descriptions).

2. Compression Strategy: From Dirty Text to Semantic Skeleton

I recommend a three-layer compression pipeline: the first layer uses regex and stopword lists to remove symbol noise; the second layer uses TF-IDF or TextRank to extract keywords, but must retain domain entities (such as API names, parameter names); the third layer uses the DeepSeek model itself for 'semantic summarization'—give the model an instruction to output the 'minimal understandable prompt'. This layer is the most effective, but be careful to control the summary length, otherwise it may increase latency.

Below is a practical compression function that calls DeepSeek's chat completion API to compress long prompts into short text carrying the same semantics:

import os
from openai import OpenAI

client = OpenAI(
    api_key="your-deepseek-api-key",
    base_url="https://api.deepseek.com"
)

def compress_prompt(raw_prompt: str) -> str:
    """Use DeepSeek model to compress prompt, preserving core semantics."""
    sys_msg = "You are a prompt compression expert. Please compress the user's prompt to a version of no more than 50 words, preserving all entities, numbers, key constraints, and question types. Output only the compressed text."
    resp = client.chat.completions.create(
        model="deepseek-chat",
        messages=[
            {"role": "system", "content": sys_msg},
            {"role": "user", "content": raw_prompt}
        ],
        temperature=0.2,
        max_tokens=100
    )
    return resp.choices[0].message.content.strip()

In production, this code needs attention: if the compressed prompt still exceeds a preset threshold (e.g., 80 tokens), you can add a 'brute-force truncation' as a fallback. But for semantic completeness, we recommend setting max_tokens=150 and using low randomness with temperature=0.2.

3. Principles of Semantic Caching: Vectorization and Similarity Threshold

Semantic caching differs from traditional key-value caching; it needs to map 'prompts' into vector space. We use DeepSeek's embedding model (if using deepseek-chat, you can temporarily use its hidden layers or call a separate embedding API) to convert the compressed text into a 768-dimensional vector. Then store it in a vector database supporting cosine similarity search (such as FAISS or Milvus).

When a new request arrives, we compress and vectorize it, and retrieve the most similar cached item. The key parameter is the threshold—through extensive experiments, I found that a cosine similarity of 0.92 balances accuracy and hit rate. Below 0.90 causes frequent false positives (semantic deviation), while above 0.95 almost requires identical text, losing the meaning. The table below shows measured results at different thresholds:

From the table, 0.92 is the most cost-effective point. But for finance or law domains, it is recommended to adjust to 0.95.

4. Design of Cache Keys: Not Just Vectors

Using vectors alone as cache keys is insufficient because vector retrieval may return results that are partially similar but have different key parameters. We adopt a 'composite key': first extract 'hard constraints' (such as user ID, model parameters temperature, max_tokens, streaming or not) through a lightweight rule, combine them into a string, then hash this string as a filter condition for the vector database.

For example, the structure of a cache record includes three fields: compressed_text (compressed prompt), embedding (vector), meta_hash (hash value). When querying, first compute meta_hash to filter, then perform vector similarity ranking within the candidate set. This ensures both precision and leverages semantic expansion capabilities.

Below is pseudocode for querying the cache, showing how to combine hash and vector retrieval:

import hashlib
import numpy as np

def get_cache_key(prompt: str, temperature: float, max_tokens: int):
    meta = f"{temperature}|{max_tokens}"
    return hashlib.sha256((prompt[:20] + meta).encode()).hexdigest()[:12]

def search_cache(collection, vector, meta_hash, threshold=0.92):
    # Assume collection is a vector database collection supporting vector search and filtering
    results = collection.search(
        vector, top_k=5, filter={"meta_hash": meta_hash}
    )
    for res in results:
        if res.distance >= threshold:  # Using cosine similarity, distance closer to 1 means more similar
            return res.payload
    return None

Note: Using `prompt[:20]` to truncate the prefix is to speed up hashing, but it may cause false positives. A more robust approach is to use the compressed_text as part of the prefix.

5. Pitfalls and Solutions in Practice

Pitfall 1: Vector drift problem. When the model is updated (e.g., DeepSeek version iteration), the embedding space may change, causing old caches to become invalid. The solution is to periodically perform a 'smooth migration' of the cache—keep the old vector database, create a new one, and gradually eliminate old records during dual-write.

Pitfall 2: Inconsistency between compression and caching. If the compression algorithm has slight randomness (when Temperature is high), the same request may produce different compressed texts, leading to different vectors and thus cache misses. The solution is to set temperature=0.2 or lower, and use a deterministic compression mode (e.g., greedy decoding).

Pitfall 3: Cache invalidation and update. When the underlying data changes (e.g., user permissions or business rules), cached responses may be outdated. The solution is to introduce a version number or timestamp in the cache key, and set an appropriate TTL (time-to-live).

Pitfall 4: Cold start problem. In the initial stage, the cache is empty, and the hit rate is low. The solution is to preload some common prompts or use a hybrid strategy: for requests with high similarity but below threshold, use the cached response but mark it as 'low confidence' for manual review.

In summary, prompt compression and semantic caching are a pair of 'golden partners'. Through the three-layer compression pipeline and composite key design, we can achieve a cache hit rate of 15%-20% in production, reducing average latency by 30%-40% and saving about 20% of API costs. The key is to find the balance point—neither over-compressing to lose semantics, nor over-caching to lose flexibility.

amount changes. Solution: Force temperature=0 during compression, and set a seed (if the API supports it). Additionally, you can perform a "normalization" step on the compressed result, such as lowercasing.