API Service Architecture Design

A mature large model API service architecture typically includes the following layers:

  1. Access layer: API gateway (e.g., Kong, Nginx), responsible for authentication, rate limiting, routing
  2. Scheduling layer: Request queue and scheduler, managing concurrency and priority
  3. Inference layer: Inference engine cluster such as vLLM/TGI
  4. Cache layer: Redis and other caches, reducing duplicate inference
  5. Monitoring layer: Prometheus+Grafana, real-time monitoring and alerting

API Gateway Configuration

# Nginx reverse proxy configuration
upstream vllm_backend {
    least_conn;
    server 10.0.1.1:8000 weight=1 max_fails=3 fail_timeout=30s;
    server 10.0.1.2:8000 weight=1 max_fails=3 fail_timeout=30s;
    server 10.0.1.3:8000 weight=1 max_fails=3 fail_timeout=30s;
}

server {
    listen 443 ssl;
    server_name api.example.com;

    # Rate limiting: max 10 requests per second per IP
    limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;
    limit_req zone=api_limit burst=20 nodelay;

    location /v1/chat/completions {
        proxy_pass http://vllm_backend;
        proxy_read_timeout 300s;
        proxy_buffering off;

        # Add authentication header
        proxy_set_header X-API-Key $http_x_api_key;
    }
}

Rate Limiting Strategies

Multi-level rate limiting strategies:

  • IP level: N requests per second per IP, preventing single IP abuse
  • User level: Different plan users have different concurrency limits
  • Token level: Limit the number of tokens generated per minute
  • Queue mechanism: Requests exceeding the limit enter a queue for waiting, rather than being directly rejected

Cache Optimization

import hashlib
import redis

r = redis.Redis(host='localhost', port=6379)

def cached_llm_call(model, messages, temperature=0.7):
    # Generate cache key
    cache_key = hashlib.md5(
        f"{model}:{str(messages)}:{temperature}".encode()
    ).hexdigest()

    # Query cache
    cached = r.get(cache_key)
    if cached:
        return cached.decode()

    # Call LLM
    response = call_llm_api(model, messages, temperature)

    # Write cache (TTL=1 hour)
    r.setex(cache_key, 3600, response)

    return response

Security Protection

  • API Key authentication: Generate a unique API Key for each user
  • Content moderation: Review input and output for content safety
  • Prompt injection defense: Detect and filter maliciously injected prompts
  • Data masking: Automatically mask sensitive information (phone numbers, ID numbers, etc.)
  • Audit logs: Record all API calls for traceability and auditing

Cost Control

  • Model tiering: Use small models for simple tasks, large models for complex tasks
  • Cache hit rate: Improve cache hit rate through semantic caching
  • Token limits: Limit the maximum number of tokens per request
  • Dynamic scaling: Automatically adjust the number of inference instances based on load

Summary

Large model API service is not simply "adding an HTTP interface", but requires systematic design from multiple dimensions such as architecture, security, performance, and cost. It is recommended to start with a minimal viable solution and gradually improve based on actual business needs.