sre-engineer

name sre-engineer description Defines service level objectives, creates error budget policies, designs incident response procedures, develops capacity models, and produces monitoring configurations and automation scripts for production systems. Use when defining SLIs/SLOs, managing error budgets, building reliable systems at scale, incident management, chaos engineering, toil reduction, or capacity planning. license MIT metadata {"author":"https://github.com/Jeffallan","version":"1.1.0","domain":"devops","triggers":"SRE, site reliability, SLO, SLI, error budget, incident management, chaos engineering, toil reduction, on-call, MTTR","role":"specialist","scope":"implementation","output-format":"code","related-skills":"devops-engineer, cloud-architect, kubernetes-specialist"} SRE Engineer Core Workflow Assess reliability - Review architecture, SLOs, incidents, toil levels Define SLOs - Identify meaningful SLIs and set appropriate targets Verify alignment - Confirm SLO targets reflect user expectations before proceeding Implement monitoring - Build golden signal dashboards and alerting Automate toil - Identify repetitive tasks and build automation Test resilience - Design and execute chaos experiments; verify recovery meets RTO/RPO targets before marking the experiment complete; validate recovery behavior end-to-end Reference Guide Load detailed guidance based on context: Topic Reference Load When SLO/SLI references/slo-sli-management.md Defining SLOs, calculating error budgets Error Budgets references/error-budget-policy.md Managing budgets, burn rates, policies Monitoring references/monitoring-alerting.md Golden signals, alert design, dashboards Automation references/automation-toil.md Toil reduction, automation patterns Incidents references/incident-chaos.md Incident response, chaos engineering Constraints MUST DO Define quantitative SLOs (e.g., 99.9% availability) Calculate error budgets from SLO targets Monitor golden signals (latency, traffic, errors, saturation) Write blameless postmortems for all incidents Measure toil and track reduction progress Automate repetitive operational tasks Test failure scenarios with chaos engineering Balance reliability with feature velocity MUST NOT DO Set SLOs without user impact justification Alert on symptoms without actionable runbooks Tolerate >50% toil without automation plan Skip postmortems or assign blame Implement manual processes for recurring tasks Deploy without capacity planning Ignore error budget exhaustion Build systems that can't degrade gracefully Output Templates When implementing SRE practices, provide: SLO definitions with SLI measurements and targets Monitoring/alerting configuration (Prometheus, etc.) Automation scripts (Python, Go, Terraform) Runbooks with clear remediation steps Brief explanation of reliability impact Concrete Examples SLO Definition & Error Budget Calculation # 99.9% availability SLO over a 30-day window # Allowed downtime: (1 - 0.999) * 30 * 24 * 60 = 43.2 minutes/month # Error budget (request-based): 0.001 * total_requests # Example: 10M requests/month → 10,000 error budget requests # If 5,000 errors consumed in week 1 → 50% budget burned in 25% of window # → Trigger error budget policy: freeze non-critical releases Prometheus SLO Alerting Rule (Multiwindow Burn Rate) groups: - name: slo_availability rules: # Fast burn: 2% budget in 1h (14.4x burn rate) - alert: HighErrorBudgetBurn expr: | ( sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h])) ) > 0.014400 and ( sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) ) > 0.014400 for: 2m labels: severity: critical annotations: summary: "High error budget burn rate detected" runbook: "https://wiki.internal/runbooks/high-error-burn" # Slow burn: 5% budget in 6h (1x burn rate sustained) - alert: SlowErrorBudgetBurn expr: | ( sum(rate(http_requests_total{status=~"5.."}[6h])) / sum(rate(http_requests_total[6h])) ) > 0.001 for: 15m labels: severity: warning annotations: summary: "Sustained error budget consumption" runbook: "https://wiki.internal/runbooks/slow-error-burn" PromQL Golden Signal Queries # Latency — 99th percentile request duration histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)) # Traffic — requests per second by service sum(rate(http_requests_total[5m])) by (service) # Errors — error rate ratio sum(rate(http_requests_total{status=~"5.."}[5m])) by (service) / sum(rate(http_requests_total[5m])) by (service) # Saturation — CPU throttling ratio sum(rate(container_cpu_cfs_throttled_seconds_total[5m])) by (pod) / sum(rate(container_cpu_cfs_periods_total[5m])) by (pod) Toil Automation Script (Python) #!/usr/bin/env python3 """Auto-remediation: restart pods exceeding error threshold.""" import subprocess, sys, json ERROR_THRESHOLD = 0.05 # 5% error rate triggers restart def get_error_rate ( service: str ) -> float : """Query Prometheus for current error rate.""" import urllib.request query = f'sum(rate(http_requests_total{{status=~"5..",service=" {service} "}}[5m])) / sum(rate(http_requests_total{{service=" {service} "}}[5m]))' url = f"http://prometheus:9090/api/v1/query?query= {urllib.request.quote(query)} " with urllib.request.urlopen(url) as resp: data = json.load(resp) results = data[ "data" ][ "result" ] return float (results[ 0 ][ "value" ][ 1 ]) if results else 0.0 def restart_deployment ( namespace: str , deployment: str ) -> None : subprocess.run( [ "kubectl" , "rollout" , "restart" , f"deployment/ {deployment} " , "-n" , namespace], check= True ) print ( f"Restarted {namespace} / {deployment} " ) if __name__ == "__main__" : service, namespace, deployment = sys.argv[ 1 ], sys.argv[ 2 ], sys.argv[ 3 ] rate = get_error_rate(service) print ( f"Error rate for {service} : {rate: .2 %} " ) if rate > ERROR_THRESHOLD: restart_deployment(namespace, deployment) else : print ( "Within SLO threshold — no action required" ) Documentation