Why Agents Need a Memory System
Have you ever encountered a situation where you chat with an AI assistant for half an hour, and it suddenly forgets key information you told it ten minutes ago? This is not due to insufficient model capability, but rather because it lacks an effective memory system. Human conversations flow smoothly because we are constantly memorizing, updating, and retrieving context—whereas traditional large models start from scratch with every conversation.
The core goal of an Agent memory system is to enable AI to maintain state consistency across multi-turn interactions, remember user preferences, and learn from historical experience. A good memory system can transform an Agent from a "forgetful tool" into a "caring companion," significantly improving user experience and task completion quality. In practical engineering, an Agent memory system needs to handle three types of information: conversation context (short-term memory), user knowledge and preferences (long-term memory), and task history and experience (episodic memory), which together form a complete cognitive loop.
The Three-Layer Architecture of Memory Systems
Through extensive engineering practice, the industry has developed a mature three-layer architecture for Agent memory systems:
- Working memory layer: The context window of the current conversation; the messages array is the simplest form. Capacity is approximately 8K-128K tokens.
- Short-term memory layer: Stores recent conversation summaries or key information, typically implemented with Redis, retaining data for hours to days.
- Long-term memory layer: Persistently stores user profiles, preference settings, and knowledge fragments, using vector databases (Milvus/Pinecone) for permanent storage.
The three layers are coordinated by a memory manager, which decides when to archive, compress, and retrieve relevant information.
RAG-Enhanced Memory: Giving Agents an External Brain
Traditional RAG is mainly used for knowledge base Q&A, but in Agent memory systems, RAG plays a more central role—it serves as an unlimited-capacity external brain that the Agent can retrieve on demand. Workflow: memory write → memory retrieval → memory injection → memory update. A key design decision is retrieval timing—it is recommended to use trigger-based retrieval: trigger when the user mentions a new topic or a new name.
Hands-On Practice: Building a Memory Manager
import json, hashlib, numpy as np
from datetime import datetime
from openai import OpenAI
client = OpenAI(api_key="your-deepseek-api-key", base_url="https://api.deepseek.com")
class MemoryManager:
def __init__(self, user_id):
self.user_id = user_id
self.working_memory = []
self.short_term = {}
self.long_term = {}
def get_embedding(self, text):
resp = client.embeddings.create(model="text-embedding-ada-002", input=text)
return resp.data[0].embedding
def store_long_term(self, key, value):
vec = self.get_embedding(value)
self.long_term[key] = {"content": value, "vector": vec}
def retrieve_relevant(self, query, top_k=5):
if not self.long_term: return []
qv = self.get_embedding(query)
scored = [(np.dot(qv, v["vector"])/(np.linalg.norm(qv)*np.linalg.norm(v["vector"])), k, v["content"]) for k,v in self.long_term.items()]
scored.sort(reverse=True)
return [{"key": k, "content": c, "score": s} for s,k,c in scored[:top_k] if s > 0.6]
def chat(self, user_input):
relevant = self.retrieve_relevant(user_input)
ctx = "\n".join(r["content"] for r in relevant) if relevant else ""
resp = client.chat.completions.create(model="deepseek-chat",
messages=[{"role":"system","content": f"历史记忆：\n{ctx}" if ctx else "你是智能助手。"},
{"role":"user","content": user_input}])
return resp.choices[0].message.content
mem = MemoryManager("u1")
mem.store_long_term("lang", "用户偏好Python")
print(mem.chat("帮我设计推荐算法"))Memory Conflict and Forgetting Strategies
Memory systems face two major challenges: information conflict (e.g., a user prefers Python first then switches to Go) and capacity management. Conflict handling strategies include time priority (latest overwrites old), version retention (keep both old and new with timestamps), confidence weighting, and conflict prompting (actively ask the user). Capacity management uses LRU eviction, importance scoring, semantic deduplication, and tiered storage (hot data in Redis, warm data in vector DB, cold data in object storage).
Evaluation Metrics for Memory Systems
- Memory Recall@K: The proportion of relevant memories included in retrieval results
- Dialogue Coherence Score: The context consistency of the Agent in long conversations
- User Repetition Count: The frequency with which users need to repeat the same information
- Memory Utilization Rate: The proportion of retrieved memories that are actually referenced in responses
It is recommended to start with dialogue coherence and user repetition count—they most directly reflect improvements in user experience. After meeting these targets, optimize retrieval precision and storage efficiency.
Practical Case: Memory System in Customer Service
Taking an intelligent customer service system as an example, the value of the memory system is particularly prominent. When a user first mentions "My order number is 20240715001" during a consultation, the memory system stores this information in short-term memory. The next day, when the user asks again with only "the order I mentioned before," the Agent retrieves the order number from short-term memory and seamlessly continues the conversation. A month later, when the user asks "help me check what I bought in July," the long-term memory
Memory System Selection: Redis vs. Vector Database vs. Graph Database
Different levels of memory suit different storage backends: Working memory can directly use Python's list or deque, no external dependencies needed. Short-term memory is recommended to use Redis—it natively supports TTL, and its List structure is suitable for storing recent conversation summaries, with performance and reliability proven over time. Long-term memory is recommended to use vector databases: Milvus is suitable for large-scale (>1 million vectors) production environments, Pinecone for teams that don't want to manage infrastructure, and Chroma for rapid prototyping. If your memory contains a lot of relational information (e.g., "User A is a member of Team B, and Team B works on Project C"), a graph database (Neo4j) might be more appropriate than a vector database—graph structures directly support relational reasoning, while vectors only do semantic similarity. Our recommendation is a hybrid approach: use a vector database for user preferences and knowledge snippets, and a graph database for entity relationships, exposing both through a unified Memory Manager interface.
Memory System Performance Benchmark and Tuning
In production, memory retrieval latency directly impacts conversation experience—users feel noticeable lag if they wait more than 2 seconds. We conducted comprehensive performance benchmarks on memory systems: Vector retrieval latency (retrieving Top-5 from 1 million vectors, P99 latency should be <50ms—achievable with Milvus + IVF index), Hybrid retrieval fusion time (fusion ranking of vector + keyword + graph three-way retrieval should complete within <20ms), Memory compression quality (when compressing 10 rounds of conversation into a summary, key information retention should be >95%—evaluated using a human-annotated test set). Performance bottlenecks typically occur in embedding computation and vector retrieval—using batch embedding and GPU-accelerated vector retrieval engines can reduce end-to-end latency from 800ms to 120ms. Another easily overlooked optimization point is the logic of the memory manager itself—avoid full retrieval on every conversation turn, and use incremental updates and an event-driven architecture.
Want to orchestrate this skill chain yourself?
Open in Skill Chain →