--- name: agivar-gui-task description: >- Run visual GUI tasks on the user's real desktop through Agivar's MCP server — clicking, typing, opening apps, driving a real browser, and multi-step on-screen workflows from natural language. Use whenever the user wants to automate something on their actual desktop or screen, drive a desktop app, or test a web page through a real browser (web UI testing, end-to-end click-throughs, verifying a frontend change by operating the page). Reach for it especially right after a UI change — "now test it", "check the page still works", "test like a real user." Also trigger on mentions of Agivar, `run_gui_task`, or "do this on my computer / screen for me." --- # Agivar GUI Task Agivar is a desktop app whose built-in **local MCP server** lets you (an AI client such as Claude Code or Cursor) execute **visual GUI tasks on the real desktop**. You describe a task in natural language; Agivar drives the actual screen (moves the mouse, types, opens apps) and reports back what happened. This skill tells you how to connect and — critically — how to drive the asynchronous task workflow correctly. ## Prerequisites Before any task can run: 1. The **Agivar desktop app is running** on the local machine. 2. The user is **logged in** inside Agivar. Tasks fail without a login. The MCP server starts automatically with the app — the user does not start it manually. If you cannot reach it, the most likely cause is that the app is not running. ## Connecting The server speaks **HTTP (Streamable HTTP, stateless / JSON responses)**. | Item | Value | | --- | --- | | URL | `http://127.0.0.1:17342/mcp` | | Server name | `agivar` | | Health check | `GET http://127.0.0.1:17342/mcp` (non-SSE) returns `{"status":"ok","server":"agivar"}` when online | ## Tools | Tool | Purpose | Key args | | --- | --- | --- | | `get_status()` | Check readiness (logged in? busy?) before dispatching | none | | `run_gui_task(task, context?)` | Start a desktop task; returns a `task_id` **immediately** (runs in background) | `task` (natural-language task), `context` (optional background) | | `get_task_result(task_id)` | Poll a task's result | `task_id` | | `send_task_message(task_id, message)` | After a task finished, continue the SAME session with a follow-up (keeps prior context) | `task_id`, `message` | | `cancel_task(task_id)` | Stop a running task | `task_id` | ### Return shapes - `run_gui_task` → `{ status: "started", task_id, message }`. If Agivar is not ready (already running a task, etc.) it returns `{ status: "failed", error }`. - `get_task_result`: - running → `{ status: "running", elapsed_seconds, summary, new_observations, observations_total, message }` - finished → `{ status: "completed" | "failed" | "partial", summary, observations, screenshots, duration, error }` - `summary` — natural-language outcome of the task, including how the agent verified it succeeded (so you can judge completion from the text alone). - `observations` — list of key tool-call summaries from the run. - `screenshots` — list of `{ path, caption }`: each `path` is a **local `.webp` file** (NOT base64) captured during the run, and `caption` describes what was on screen there. Read `summary` together with these captioned screenshots to decide whether the task truly succeeded. Open the paths if you need to inspect what happened. On Windows, paths use forward slashes so Markdown links do not lose backslashes. - The `summary` verifies the task **step by step and cites each screenshot by its full absolute path** (matching one `screenshots[].path`). See "Reporting back to the user" below. - Polling a finished task again returns the same final result (idempotent), so you can re-read it while deciding whether to follow up. - `send_task_message` → `{ status: "started", task_id, message }` (now running again — poll like `run_gui_task`), or `{ status: "failed", error }` if the `task_id` is unknown, its conversation was deleted, or another task is running. - `cancel_task` → `{ status: "stopping" | "already_done" | "failed", task_id, message }`. - `get_status` → `{ logged_in, busy, running_tasks }`. ## The workflow (follow this exactly) `run_gui_task` returns right away; the task runs in the background. You must poll for the result. This is the single most important thing to get right. ``` get_status() # recommended: confirm logged_in=true, busy=false ↓ run_gui_task(task="...") # get task_id back ↓ get_task_result(task_id) # poll every 15–30 seconds ↓ status == "running" → wait, poll again ↓ status in {completed, failed, partial} → done read summary + captioned screenshots → review ↓ needs more work? → send_task_message(task_id, "next instruction") → poll again ↓ satisfied? → stop ``` This is a continuous loop: **dispatch → poll → review the summary and captioned screenshots → decide to finish or continue**. When you need another step, prefer `send_task_message` over a fresh `run_gui_task` so the agent keeps all the context from earlier in the session. ### Rules that matter (and why) 1. **Always poll until a terminal status.** Do not assume the task finished just because `run_gui_task` returned. Call `get_task_result(task_id)` every 15–30 seconds until `status` is `completed`, `failed`, or `partial`. Simple tasks take ~2–5 minutes; complex workflows can take 10–30 minutes. Polling too eagerly wastes turns; not polling at all means you never see the result. 2. **One task at a time — never run two concurrently.** GUI tasks drive the single real desktop; two at once would fight over the mouse and keyboard and corrupt each other. While `get_task_result` still says `running`, do **not** call `run_gui_task` again — it will be rejected with `status:"failed"`. To switch tasks, `cancel_task` the current one (or wait for it) first. 3. **Make the task self-contained and specific.** Put the concrete steps and the expected end state in `task`; put any extra background in `context`. Agivar's own agent decides the fine-grained actions, so a clear goal beats a vague one. This matters most for web UI testing: spell out the URL, the exact interactions, and what "pass" looks like. For example, to verify a frontend change: `task="Open the browser, go to http://localhost:5173/settings, switch the theme toggle to dark, and confirm the page background turns dark"`, `context="The dev server is already running; the theme toggle was just reworked."` 4. **Cancel instead of waiting on a task that's gone wrong.** `cancel_task(task_id)` asks it to stop; it finishes shortly as `partial`, and `get_task_result` then returns whatever was accomplished. Keep polling until that terminal status. 5. **To continue, reuse the session — don't start over.** Once a task is finished, if you want a follow-up step, call `send_task_message(task_id, "...")` instead of `run_gui_task`. It keeps the whole prior context (what was opened, typed, navigated). Only the same one-at-a-time rule applies: send the follow-up *after* the task reached a terminal status, then poll again. Start a brand-new `run_gui_task` only for genuinely unrelated work. Review the `summary` and captioned `screenshots` first so you know whether a follow-up is even needed. ## Reporting back to the user (show the evidence, step by step) When a task finishes, **do not just paraphrase the outcome** — relay the verification so the user can check it themselves. The `summary` already walks through the task step by step and cites each screenshot **by its full absolute path**. In your reply: 1. Present the result **step by step**, mirroring the summary's verification. 2. For each step that cites a screenshot, give **both** a clickable Markdown link **and** the plain absolute path right after it — because in some clients the link isn't clickable, and the plain path still lets the user open the file manually. Use `screenshots[].path` as the authoritative path. 3. End with the overall pass/fail conclusion. So instead of "已完成。点赞成功。", write something like: ``` 任务完成。确认步骤如下： 1. 打开 B 站首页 — 页面正常加载且已登录 截图 [01_gui_sub_task.webp](C:/…/mcp_shots//01_gui_sub_task.webp) 路径：C:/…/mcp_shots//01_gui_sub_task.webp 2. 进入视频《非十科技 - agivar 产品介绍》 — 播放器正常显示 截图 [02_gui_sub_task.webp](C:/…/mcp_shots//02_gui_sub_task.webp) 路径：C:/…/mcp_shots//02_gui_sub_task.webp 3. 点击点赞 — 点赞按钮变为蓝色高亮 截图 [03_gui_sub_task.webp](C:/…/mcp_shots//03_gui_sub_task.webp) 路径：C:/…/mcp_shots//03_gui_sub_task.webp 整体结论：任务成功完成，点赞按钮处于已点赞状态。如需进一步操作，可继续用 send_task_message。 ``` This way every claim is backed by a screenshot the user can click **or** open by path, and they can decide whether to continue the session. ## End-to-end example ```text 1. get_status() → { "logged_in": true, "busy": false, "running_tasks": 0 } 2. run_gui_task(task="Open the system Calculator, compute 23 * 19, and tell me the result") → { "status": "started", "task_id": "a1b2c3d4e5f60718", "message": "..." } 3. get_task_result("a1b2c3d4e5f60718") → { "status": "running", "elapsed_seconds": 12.3, "message": "..." } # wait 15–30s, poll again 4. get_task_result("a1b2c3d4e5f60718") → { "status": "completed", "summary": "Opened Calculator; entered 23 × 19; the result shown is 437, confirming the task.", "observations": ["gui_sub_task: opened Calculator and entered the expression", "..."], "screenshots": [ { "path": ".../mcp_shots//01_gui_sub_task.webp", "caption": "Calculator showing 23 × 19 = 437" } ], "duration": 0, "error": null } 5. # not satisfied? continue in the same session (keeps context): send_task_message("a1b2c3d4e5f60718", "Now also compute 437 + 100 and tell me the total") → { "status": "started", "task_id": "a1b2c3d4e5f60718", "message": "..." } # poll again ``` ## Troubleshooting - **Can't connect / empty tool list** — confirm the Agivar app is running; hit the health check on port `17342`. - **Tasks keep failing** — call `get_status()`; if `logged_in` is false, ask the user to sign in to Agivar. - **`run_gui_task` returns failed saying a task is already running** — poll the current task with `get_task_result`, or `cancel_task` it, then dispatch again. - **Where the user sees the task** — each task appears in Agivar's main window as a conversation titled `MCP: `, with live progress.