agent-platform-eval-flywheel

name agent-platform-eval-flywheel metadata {"category":"AiAndMachineLearning"} description Measures and improves the quality of AI models and agents on Google Cloud using the Eval Quality Flywheel methodology. Use when evaluating an agent or model, building an eval dataset, picking or writing evaluation metrics, analyzing failures, comparing results before and after a fix, or when guidance is needed on Agent Platform eval methodology — including dataset schema, LLM-as-judge scoring, and common failure causes. For fine-tuning, use agent-platform-tuning. For general production deployment, use agent-platform-deploy. Agent Platform Eval Flywheel Skill Help users evaluate and iteratively improve GenAI models and agents using the Agent Platform GenAI Evaluation SDK ( google.genai / agentplatform ). When to use this skill Evaluating GenAI agents or models with the Agent Platform GenAI Evaluation SDK ( client.evals.evaluate() ). Creating evaluation datasets from session traces, pandas DataFrames, or synthetic generation. Selecting, configuring, or writing custom evaluation metrics. Analyzing rubric verdicts, loss patterns, and clustering failures. Suggesting concrete code/prompt improvements based on eval results. Evaluating a model served on an Agent Platform endpoint (BYOM) or a Model-as-a-Service (MaaS) model by ID — including deploying the model first if needed. For this case, follow references/deployment.md and use the endpoint_evaluation.py / maas_evaluation.py scripts. Safety & Confirmation Tiers (CRITICAL) Before executing any commands or scripts on behalf of the user, you MUST adhere to the following safety tiers based on the action requested: Tier R : Read-only ( inspect_results.py , compare_results.py , validate_dataset.py , parse_adk_traces.py , render_html_report.py ) Rule : No confirmation needed. You may execute these helper scripts immediately to inspect data, validate schemas, parse traces, or compare evaluation results. Tier M: Read-only with Compute Costs ( client.evals.run_inference , client.evals.evaluate , client.evals.generate_conversation_scenarios , client.evals.generate_loss_clusters ) Rule : These operations invoke LLMs or remote evaluation services that consume compute resources and incur costs. This requires interactive confirmation with 'Yes'/'No' options. Once granted once, you do not have to prompt for future evaluation. Same-turn restriction : Do not run the evaluation in the same turn as presenting the confirmation prompt. End your turn after asking and wait for the user's reply; only execute after explicit 'Yes' / approval. Printing a preview and then calling the tool before the user can answer does not count as obtaining confirmation. Setup The scripts need vertexai (from google-cloud-aiplatform[evaluation] ), google-genai , pandas , and requests . Do not create a virtual environment — it starts empty and hides packages the environment already provides, forcing a redundant install. Probe, and install only what is missing: python3 -c "import vertexai, google.genai, pandas, requests" \ || pip install 'google-cloud-aiplatform[evaluation]>=1.163.0' 'google-genai>=1.0.0' The version specifiers must stay quoted: unquoted, bash reads >=1.154.0 as a redirect and silently writes an empty file instead of constraining the install. Need GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION . Check env vars first; if missing, ask the user. Newer Gemini models often need location="global" . Correct SDK entrypoints import agentplatform client = agentplatform.Client(project=PROJECT, location=LOCATION) client.evals.run_inference(model=..., src=...) client.evals.evaluate(dataset=..., metrics=...) client.evals.generate_conversation_scenarios(...) Two imports that look plausible and are not: from agentplatform.types import evals -- ModuleNotFoundError . types is a module, not a package; use from agentplatform import types . from vertexai.evaluation import PointwiseMetric, EvalTask -- the superseded SDK. Its classes take different arguments ( PointwiseMetric has no system_instruction ), so code written against it fails with TypeError rather than an import error. Use agentplatform throughout. The Quality Flywheel Five stages, run in order on the first pass, then loop 2 → 5 until quality targets are met. Shortcuts that waste time Shortcut Why it fails "I'll tune the metric threshold down Hides real failures. Fix the agent, : so it passes." : not the bar. : "This case is flaky, I'll skip it." Flakiness reveals non-determinism in : : the agent. Fix with temperature=0 : : : or stricter instructions. : "I just need to fix the eval If expected outputs keep moving, the : dataset, not the agent." : agent has a behavior problem. : "I can tell from the trace it works Self-grading doesn't generalize. : — skip Stage 3." : Always run evaluate() and read : : : scores. : "One iteration is enough." Expect 5–10+ iterations. Stopping : : early leaves regressions on other : : : metrics undetected. : 1. Prepare Data Produce an EvaluationDataset . There are three input shapes, pick the one that matches the data the user already has: EvalCase list (single-turn or multi-turn): from agentplatform import types from google.genai import types as genai_types # prompt/reference/response values are Content, not str. UserContent and # ModelContent wrap a plain string and set the right role. dataset = types.EvaluationDataset(eval_cases=[ types.EvalCase( prompt=genai_types.UserContent( "What is 2+2?" ), responses=[types.ResponseCandidate( response=genai_types.ModelContent( "4" ))], reference=types.ResponseCandidate( response=genai_types.ModelContent( "4" )), ), # For multi-turn agent traces, set agent_data instead of prompt/responses. ]) Multi-turn agent traces wrap each conversation in AgentData → ConversationTurn → AgentEvent . See references/dataset_schema.md for the full type hierarchy. Pandas DataFrame (tabular sources — CSV, BigQuery, Sheets): import pandas as pd from agentplatform import types df = pd.DataFrame({ "prompt" : [ "What is 2+2?" , "Capital of France?" ], "response" : [ "4" , "Paris" ], "reference" : [ "4" , "Paris" ], }) dataset = types.EvaluationDataset(eval_dataset_df=df) Column names must match the fields the chosen metrics expect (see references/dataset_schema.md for the per-metric requirements table). Cold start (no data at all): synthesize scenarios server-side with client.evals.generate_conversation_scenarios(agent=..., config=...) -- the parameter is agent or agent_info , not agents , and config is required. The config class is types.evals.UserScenarioGenerationConfig , not types.UserScenarioGenerationConfig . Set its user_scenario_count (1-100): it defaults to None, the client accepts that, and the server rejects the call with 400 INVALID_ARGUMENT . count is a separate field and does not substitute for it. Stage 2 plays the scenarios out. Managed Agents (Gemini Agents API): evaluate agents created with the Managed Agents API . Use generate_conversation_scenarios to create test scenarios from the agent's configuration, run_inference to execute the agent, and evaluate to score the traces. These functions now accept managed agents and interaction ids as input. You can also evaluate existing interactions recorded via the Interactions API using InteractionsDataSource . See references/sdk_patterns.md Pattern 8 for the full code pattern. For ADK session dumps, use scripts/parse_adk_traces.py instead of writing the conversion by hand. 2. Run Inference Populate responses/traces on the dataset. Skip this stage if traces are already complete (e.g., production logs or replay). # Agent eval — pass a callable wrapping the user's ADK Agent/App. client.evals.run_inference(model=agent_callable, src=dataset) # Model eval — pass a model ID directly. client.evals.run_inference(model= "gemini-2.5-flash" , src=dataset) # Synthesized scenarios — let the simulator drive. client.evals.run_inference( model=agent_callable, src=dataset, user_simulator_config=UserSimulatorConfig(max_turn= 10 ), ) # DataFrame also works as src= — no EvalCase wrapping needed. client.evals.run_inference(model= "gemini-2.5-flash" , src=df) # Managed Agent — pass an agent resource name. AGENT_RESOURCE = f"projects/ {PROJECT_ID} /locations/global/agents/ {AGENT_ID} " client.evals.run_inference( agent=AGENT_RESOURCE, src=scenarios, config={ "user_simulator_config" : { "max_turn" : 3 }}, ) 3. Grade (always run) result = client.evals.evaluate(dataset=dataset, metrics=[...]) result.show() # Interactive HTML report with scores, rubrics, and traces. Pick metrics by what you want to measure. Full catalog in references/metric_registry.md . Agent metrics (multi-turn, adaptive rubrics) — start here for agent eval. Goal Metric Did the agent achieve the user's goal? multi_turn_task_success Was the reasoning path logical and efficient? multi_turn_trajectory_quality Tool/function calling quality across turns multi_turn_tool_use_quality Overall conversational quality multi_turn_general_quality Final response quality (no reference needed) final_response_quality Final response vs. a golden reference final_response_match Single-turn tool use tool_use_quality General quality metrics (single-turn, adaptive rubrics) — for model eval. Goal Metric Overall response quality (recommended starting point) general_quality Linguistic quality (fluency, coherence, grammar) text_quality Adherence to specific constraints / instructions instruction_following Static rubric metrics (fixed criteria) — apply alongside the above. Goal Metric Catch hallucinated claims (RAG, factual answers) hallucination Factuality / consistency against provided context grounding Safety policy compliance safety Domain-specific check no built-in covers: write a custom metric. Predefined: types.RubricMetric.<NAME> — server-side AutoRater, no judge model needed. Custom LLM-as-a-judge: types.LLMMetric with prompt_template or types.MetricPromptBuilder for structured rubrics. Always set judge_model ; it defaults to None and every case then fails with 400 INVALID_ARGUMENT: Error parsing JSON . Custom code: types.CodeExecutionMetric with a custom_function string containing def evaluate(instance: dict) for remote sandboxed execution; or types.Metric with custom_function=<callable> for local execution. Always persist the result so Stage 4 and 5 can read it. Save both JSON (machine-readable, diffable) and HTML (human-readable, linkable): import datetime from pathlib import Path from agentplatform._genai import _evals_visualization out_dir = Path( "artifacts/grade_results" ) out_dir.mkdir(parents= True , exist_ok= True ) ts = datetime.datetime.now().strftime( "%Y%m%d_%H%M%S" ) # fallback=str, or a DataFrame-backed dataset raises PydanticSerializationError. result_json = result.model_dump_json(fallback= str ) (out_dir / f"results_ {ts} .json" ).write_text(result_json) html = _evals_visualization.get_evaluation_html(result_json) (out_dir / f"results_ {ts} .html" ).write_text( str (html)) Or after the fact: scripts/render_html_report.py --type evaluation or scripts/inspect_results.py --save-html . 4. Analyze Failures Read summary_metrics and eval_case_results — never fabricate scores. Use scripts/inspect_results.py --failing-only to filter to failures. For each failed metric, see references/failure_patterns.md for deeper diagnoses. The compact mapping: Failing metric What to change multi_turn_task_success low The agent isn't completing the goal — : : fix orchestration, missing tool calls, : : : premature termination, wrong tool : : : selection. :