Special Characteristics of AI Application Monitoring

Unlike traditional web applications, monitoring AI applications presents unique challenges. Traditional application monitoring focuses on infrastructure metrics such as CPU, memory, and request latency, while AI applications also require monitoring at the model level: token consumption per call (cost monitoring), quality of generated content (whether there are hallucinations or off-topic responses), user behavior metrics (satisfaction, re-ask rate, human handoff rate), and the model's own status (API rate limiting, model version changes, response format changes). Together, these metrics constitute the observability of AI applications—not only seeing whether the system is running, but also how well it is running.

Without a comprehensive monitoring system, an AI application is like flying in the dark. You don't know if users are satisfied, if costs exceed the budget, or if the model is hallucinating—until user complaints erupt or the bill exceeds the limit. This article will build an AI application monitoring system covering four layers: infrastructure, application, model, and business.

Four-Layer Monitoring Architecture

Layer 1: Infrastructure Monitoring. Consistent with traditional application monitoring—CPU, memory, network, disk. Use Prometheus + Node Exporter for collection and Grafana for visualization. For GPU inference services, also monitor GPU utilization, VRAM usage, and GPU temperature.

Layer 2: Application Layer Monitoring. Core metrics include: API request volume (QPS), request latency (P50/P95/P99), error rate (4xx/5xx), and concurrent connections. Use OpenTelemetry for distributed tracing to track the time distribution of a user request across ASR→LLM→TTS stages.

Layer 3: Model Layer Monitoring. This is the monitoring layer unique to AI applications and the most critical. Core metrics: token consumption (input/output tokens per call), model latency (TTFT time to first token, TPOT time per output token), API rate limit hit count, and model response length distribution.

import time, json
from openai import OpenAI
from dataclasses import dataclass, field
from typing import List

client = OpenAI(api_key="your-deepseek-api-key", base_url="https://api.deepseek.com")

@dataclass
class LLMCallMetrics:
    timestamp: float
    model: str
    input_tokens: int
    output_tokens: int
    ttft_ms: float  # Time to first token
    total_latency_ms: float
    status: str  # success/error/rate_limited
    cost_estimate: float

class AIMonitor:
    def __init__(self):
        self.metrics: List[LLMCallMetrics] = []
        self.alert_thresholds = {
            "p99_latency_ms": 5000,
            "error_rate": 0.05,
            "daily_cost_usd": 50,
            "hallucination_rate": 0.10
        }

    def call_with_monitoring(self, messages, model="deepseek-chat", **kwargs):
        """Wrap LLM call, automatically collect metrics"""
        start = time.time()
        ttft = None
        try:
            stream = client.chat.completions.create(
                model=model, messages=messages, stream=True, **kwargs
            )
            result = ""
            usage = None
            for chunk in stream:
                if ttft is None and chunk.choices[0].delta.content:
                    ttft = (time.time() - start) * 1000
                if chunk.choices[0].delta.content:
                    result += chunk.choices[0].delta.content
                if hasattr(chunk, 'usage') and chunk.usage:
                    usage = chunk.usage

            latency = (time.time() - start) * 1000
            metric = LLMCallMetrics(
                timestamp=time.time(), model=model,
                input_tokens=usage.prompt_tokens if usage else 0,
                output_tokens=usage.completion_tokens if usage else len(result)//2,
                ttft_ms=ttft or 0, total_latency_ms=latency,
                status="success",
                cost_estimate=(usage.prompt_tokens if usage else 0)*0.14/1000000
            )
            self.metrics.append(metric)
            self._check_alerts()
            return result
        except Exception as e:
            latency = (time.time() - start) * 1000
            self.metrics.append(LLMCallMetrics(
                timestamp=time.time(), model=model,
                input_tokens=0, output_tokens=0,
                ttft_ms=0, total_latency_ms=latency,
                status=f"error: {str(e)[:50]}", cost_estimate=0
            )
)) raise def _check_alerts(self): """Check alert thresholds""" recent = [m for m in self.metrics if time.time()-m.timestamp < 3600] if not recent: return error_rate = sum(1 for m in recent if m.status!="success") / len(recent) if error_rate > self.alert_thresholds["error_rate"]: print(f"🚨 Alert: Error rate {error_rate:.1%} exceeds threshold {self.alert_thresholds['error_rate']:.1%}") latencies = [m.total_latency_ms for m in recent if m.status=="success"] if latencies and sorted(latencies)[int(len(latencies)*0.99)] > self.alert_thresholds["p99_latency_ms"]: print(f"🚨 Alert: P99 latency exceeds {self.alert_thresholds['p99_latency_ms']}ms") def get_stats(self): """Get statistics summary""" success = [m for m in self.metrics if m.status=="success"] if not success: return "No data available" return { "Total calls": len(self.metrics), "Success rate": f"{len(success)/len(self.metrics):.1%}", "Average latency": f"{sum(m.total_latency_ms for m in success)/len(success):.0f}ms", "Total token consumption": sum(m.input_tokens+m.output_tokens for m in success), "Estimated cost": f"${sum(m.cost_estimate for m in success):.4f}" } monitor = AIMonitor() for i in range(5): result = monitor.call_with_monitoring( [{"role":"user","content":"Introduce DeepSeek in one sentence"}] ) print(json.dumps(monitor.get_stats(), ensure_ascii=False, indent=2))

Layer 4: Business-level monitoring. Ultimately, the value of an AI application is measured by business metrics: user satisfaction (like/dislike rate), task completion rate (whether users complete what they intended to do), re-ask rate (whether users repeatedly ask the same question—indicating the AI didn't answer well), human handoff rate (for customer service AI), daily active users, and session duration. These metrics cannot be automatically obtained from system logs; they need to be collected via in-product instrumentation.

Cost tracking and optimization

The cost of AI applications mainly comes from LLM API calls. Dimensions to track include: cost distribution by model (V3 vs R1), cost distribution by feature (which feature consumes the most tokens), cost distribution by user (whether there is abuse or heavy users), and cost trends (day/week/month over month). It is recommended to set daily budget alerts (e.g., automatically notify if exceeding $50) and automatically circuit-break abnormal consumption (e.g., a user consuming more than 10 times the average in a single day).

Quality monitoring

The quality of AI-generated content is the hardest to monitor but the most important dimension. Practical approaches: random sampling (manually review 1-5 conversations per hour), user feedback (using like/dislike data), automated evaluation (using LLM-as-Judge to automatically assess the faithfulness and relevance of generated content), and regression testing (maintain a set of standard test cases, automatically run after each model or prompt change to ensure key scenarios are not affected). Quality monitoring does not need to cover 100% of traffic; covering 5-10% is sufficient to identify problem trends.

Alert strategy design

With monitoring data, the next step is to design a reasonable alert strategy. Too few alerts lead to problems being ignored; too many alerts lead to "alert fatigue"—people start ignoring all alerts. Recommended alert design principles: layered alerts—P0 (urgent, affects online users, respond within 5 minutes), P1 (important, needs to be handled the same day), P2 (general, can be scheduled into iterations); composite conditions—avoid triggering alerts on a single metric, e.g., trigger only when "error rate > 5%" AND "lasting > 5 minutes"; alert convergence—send only one alert for the same type within 30 minutes to avoid alert storms; alert escalation—if a P1 alert is not acknowledged within 30 minutes, automatically escalate to P0. For AI-specific alerts: abnormal token consumption (single-day consumption exceeding 150% of the same period the previous day), abnormal hallucination rate (automatically evaluated Faithfulness below 0.8), and abnormal model response format (JSON parsing failure rate exceeding 5%).Cost visualization dashboard: Cost monitoring needs a dedicated dashboard so the team can see at a glance "where the money is going." Recommended dashboard layout: top-left shows today's/this week's/this month's total cost and month-over-month change; middle shows cost distribution by model (pie chart); right shows cost by feature/API endpoint (bar chart); bottom shows cost trend (line chart, switchable by day/week/month). Also mark the budget line—yellow warning when current cost approaches 80% of budget, red warning when exceeding budget. Dashboard data updates every hour to ensure timeliness.

Anomaly detection and intelligent diagnosis

In addition to passive alerts, proactive anomaly detection allows the team to discover problems before users complain. It is recommended to use time-series anomaly detection algorithms (such as Prophet or Isolation Forest) to analyze historical trends of key metrics, and automatically trigger alerts when the current value deviates from the predicted value by more than 3 standard deviations. Going further, an intelligent diagnosis system can be built—when an anomaly is detected, automatically correlate and analyze possible causes (e.g., "latency spike" with "GPU utilization spike" may indicate traffic surge rather than model issues), provide preliminary diagnostic conclusions and recommended handling plans, helping operations personnel quickly locate the root cause.

Summary: Observability of AI applications is a continuous building process. It is recommended to start with the most critical metrics (API latency, error rate, token consumption) and gradually expand to a more comprehensive monitoring system. Do not pursue perfection from the start—a simple but actually used monitoring system is far more valuable than a complex but never deployed monitoring solution. Remember the ultimate purpose of monitoring is not to collect data, but to enable the team to discover and resolve issues before they affect users.

Want to orchestrate this skill chain yourself?

Open in Skill Chain →