Multimodal Agent in Practice: Vision-Driven Automated Operations

Introduction: Why Multimodal Agents Are the Next Paradigm for Automated Operations

Traditional RPA relies on fixed coordinates and DOM structures; once the interface is redesigned or the resolution changes, scripts fail immediately. Humans, however, rely on visual information when operating interfaces—we look at button styles, icon semantics, and the relationship between color and position. Multimodal agents endow machines with this capability: through visual understanding, agents can 'see' the screen, identify actionable elements, and generate action sequences. This article uses DeepSeek's vision-language model as a foundation to demonstrate how to build an operational agent with screen understanding, delving into engineering details.

The reason for choosing DeepSeek is not only that its API is compatible with OpenAI format and has a low integration barrier, but also because the deepseek-chat model performs stably on visual question-answering tasks and supports quick environment switching via base_url. We will implement a closed loop of 'screen reader → intent parsing → action generation' from scratch, and discuss key challenges: how to convert pixels into structured semantics, how to ensure action reliability, and how to handle error recovery.

System Architecture and Data Flow

Our agent consists of four layers: acquisition, perception, decision, and execution. The acquisition layer is responsible for taking screenshots or receiving image input; the perception layer uses the DeepSeek vision model to convert images into text descriptions, such as 'there is a search box at the top of the page, and a blue button on the right labeled Login'; the decision layer generates action sequences in JSON format based on user instructions and perception results using the language model; the execution layer invokes simulated clicks or keyboard input. Throughout the process, the model is called twice for perception and decision, but through prompt engineering, these can be merged into one call to reduce latency and cost.

The key to data flow is the standardization of intermediate representations. We unify perception results into a JSON structure containing element type, text, position (normalized coordinates), and confidence. The decision layer receives this structure, combines it with user natural language instructions, and outputs actions like [{"action": "click", "target": {"text": "Login"}}]. This design makes the agent interpretable and facilitates rollback and retry.

Step 1: Screen Perception—From Pixels to Structured Semantics

The perception layer is the agent's 'eyes'. A naive approach is to send the entire screenshot to the model, but large images increase token consumption and reduce accuracy. In engineering, we typically use OpenCV for preprocessing: removing redundant backgrounds, cropping highlighted areas, adjusting contrast, and resizing the image to the model's recommended dimensions (e.g., 1024x1024). Then, by calling DeepSeek's chat interface, we pass the image and prompt, requiring the model to return a JSON array of elements.

In practice, directly asking 'describe the screen content' yields overly generalized responses. A strongly constrained prompt is necessary, for example: 'You are the visual module for UI automation. Identify all interactive elements on the screen and output a JSON array, each element containing type (button/input/link), text (if visible), bbox (normalized [x,y,w,h]), and confidence. Do not output any extra explanation.' Below is a complete call example:

import base64, requests, json

def encode_image(path):
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode()

response = requests.post(
    "https://api.deepseek.com/chat/completions",
    headers={
        "Authorization": f"Bearer your-deepseek-api-key",
        "Content-Type": "application/json"
    },
    json={
        "model": "deepseek-chat",
        "messages": [{
            "role": "user",
            "content": [
                {"type": "text", "text": "You are the UI automation visual module. Identify interactive elements on the screen and output a JSON array, each element containing type, text, bbox, and confidence."},
                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{encode_image('screen.png')}"}}
            ]
        }],
        "max_tokens": 800
    }
)
result = response.json()["choices"][0]["message"]["content"]
print(json.loads(result))

In practice, we often find that the model occasionally misses static text or mistakenly identifies decorative icons as buttons. The countermeasure is to add 'prioritize text and input, ignore pure image decorations' to the prompt, and after defining the output schema as JSON, use pydantic for validation, and on failure, retry once with a degraded prompt. Additionally, if the page has too many elements (more than 20), it is recommended to recognize them in regions; otherwise, accuracy drops significantly.

Step 2: From Visual Semantics to Action Decision

After obtaining the element list, the agent needs to understand user instructions and generate operations. For example, if the user says 'search for DeepSeek tutorials on Baidu', the agent must find the search box and search button in the element list. The matching strategy can be based on text similarity (fuzzywuzzy) or use the LLM directly for decision-making. We send the element list and instructions together to the model, requiring it to output an action sequence JSON.

The decision prompt is designed as: 'Given the user instruction and screen elements, generate an action sequence. Action types are click/type/scroll, and target is the element index (starting from 0). Only output a JSON array.' Since element indices may change due to recognition jitter, we recommend using the element's semantic text or coordinates as the target, and have the model output confidence. Below is a decision code snippet:

import requests

def plan_actions(user_cmd, elements):
    prompt = f"User instruction: {user_cmd}\nScreen elements: {json.dumps(elements, ensure_ascii=False)}\nPlease output a JSON action array, each action containing action (click/type/scroll), target (element text or coordinates), and value (if type)."
    resp = requests.post(
        "https://api.deepseek.com/chat/completions",
        headers={"Authorization": "Bearer your-deepseek-api-key"},
        json={
            "model": "deepseek-chat",
            "messages": [{"role": "user", "content": prompt}],
            "temperature": 0.1,
            "response_format": {"type": "json_object"}
        }
    )
    return json.loads(resp.json()["choices"][0]["message"]["content"])["actions"]

There are two key points here: first, lower the temperature to 0.1 to avoid randomness causing legitimate actions to be rejected; second, forcing JSON output format (if supported) greatly reduces parsing failures. If the model returns invalid JSON, we implement a robust parser: extract the content between the first '[' and the last ']', parse with the json5 library, and if it still fails, feed the error back to the model and retry.

Step 3: Action Execution and Coordinate Mapping

Actions cannot directly use the pixel coordinates from recognition because the screenshot may differ from the real-time screen. We re-screenshot and locate the target before each action. The localization algorithm first looks for the same element text as in the decision; if it is missing due to dynamic content, it degrades to re-extracting text via OCR (e.g., paddleocr). The execution layer uses pyautogui or win32api to simulate mouse and keyboard, paying attention to screen scaling and multi-monitor issues: ensure coordinates are converted for DPI.

Pitfalls we encountered in engineering include: clicks not working (element obscured), slider dragging imprecise, and scroll position errors. Most issues stem from deviations between the bbox provided by visual recognition and the actual clickable area (the model often treats the entire text block as clickable). We summarize a compensation scheme: for input elements, click 10 pixels to the right of the center; for buttons, click the center of the text. Additionally, if the interface state does not change as expected after an action, the agent should have a verification mechanism—after each operation, compare key regions of before and after screenshots; if inconsistent, roll back and try alternative actions.

Step 4: Error Handling and Self-Healing

Single visual recognition accuracy cannot be 100%, so the agent must have fault tolerance. We design two levels of error handling: local fault tolerance—when an action fails (e.g., target not found), re-perceive the screen, update the element list, and try again; global retry—if multiple consecutive failures occur, modify the decision strategy, such as changing click to key events (Tab to switch focus). Additionally, we leverage DeepSeek's self-explanation capability by feeding error information back to the model, asking it to propose alternative solutions, similar to the ReAct pattern.

An effective technique is 'post-action snapshot difference': take a screenshot after each operation, compute the perceptual hash (pHash) with the before image, and if the difference is below a threshold, consider the operation ineffective and trigger a retry. In experiments, we compared the impact of different thresholds on accuracy: at threshold 0.2, the false positive rate was 8%; at 0.1, it rose to 15%, but recall improved. We finally chose 0.15 as a balance point and configured it as a tunable parameter in the code.

Performance Tuning: Dual Optimization of Latency and Cost

Multimodal calls are slower and more expensive than pure text. We benchmarked 100 operations: the full perception-decision process averaged 2.8 seconds, with image encoding and network transmission accounting for 60%. Optimization options include: 1) Use DeepSeek's streaming output (stream=true) to reduce first token wait; 2) Send only necessary cropped images during perception to reduce size; 3) Use caching for static interfaces, e.g., reuse the previous element list if the window title hasn't changed. In terms of cost, we estimate each operation requires about 1500 input + 300 output tokens, equivalent to less than 0.01 RMB, which is acceptable.

We also implemented a 'fast path': when user instructions match common templates (e.g., 'click a certain button'), skip LLM decision and directly execute based on element text matching, improving speed to 0.4 seconds. In practice, about 30% of operations can take the fast path, significantly improving overall experience. The table below shows a comparison under different configurations:

From Demo to Production: Engineering Essentials

To push the prototype to production, stability and security must be addressed. First, API calls should have concurrency control to avoid rate limiting; we use asyncio with a semaphore. Second, all external inputs (user instructions, images) need sensitive information filtering to prevent prompt injection—for example, users might include 'ignore previous instructions' in their commands. We added content safety checks, limiting instruction length and keywords. Finally, model outputs must be validated against a whitelist, allowing only expected action types.

Another often overlooked point: environment consistency. Different OSs have significant differences in screenshot color spaces and mouse control APIs; we encapsulate an abstraction layer with a unified interface. On Windows, we use pyautogui; on Linux, xdotool; on macOS, Quartz. Although this increases maintenance costs, it brings cross-platform compatibility.

Case Review: Automated PPT Page Turning and Annotation

We take 'annotate slides based on voice commands' as an example. The user says 'underline the title on the third page'. The agent flow: perceive the current slide, identify the title area; the decision layer generates a draw_line action; the execution layer uses OpenCV to draw a line at the corresponding coordinates and verifies via screenshot. The problem encountered was that coordinates become invalid after slide page changes—we force re-perception before each action and record page identifiers (like title text) for localization.

Another lesson: when instructions involve multiple steps (e.g., 'first go to the next page, then click the chart'), generating all actions at once often fails because the second step depends on the interface state after the first. Our solution is iterative operation: generate only the current step, execute, then call the decision layer again, forming a closed loop. Although this increases the number of calls, the success rate improved from 70% to 94%.

Summary and Future Outlook

This article demonstrates the complete path from theory to practice for multimodal agents. The key lies in: accuracy of visual perception, reliability of action decisions, and design of fault tolerance mechanisms. The DeepSeek model performs stably on this task, and the API's flexibility allows us to customize prompts and output formats. Next, we plan to introduce memory mechanisms so the agent can remember layouts of frequently used interfaces, reducing repeated perception; and explore using vision-language models to directly predict action coordinates, eliminating intermediate representations and further reducing latency.

The potential of multimodal agents extends far beyond this; they can be applied to automated testing, accessibility assistance, remote desktop control, and more. We hope the details in this article help you avoid common pitfalls in real projects and build your own automated visual agent.