ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents

English | 中文

News

• 📄 [2026/4/14] Our paper is available on arXiv: ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents.
• 🔥 [2026/4/13] ClawGUI is released — train with ClawGUI-RL (GiGPO), evaluate with ClawGUI-Eval, deploy with ClawGUI-Agent. ClawGUI-2B, a 2B agent trained end-to-end with this pipeline, hits 17.1 MobileWorld SR vs. the 11.1 baseline. See Quick Start.

Table of Contents

• Overview
• Architecture
• Quick Start
  • ClawGUI-RL — Build
  • ClawGUI-Eval — Evaluate
  • ClawGUI-Agent — Deploy
  • ClawGUI-Skills — Self-Evolving Skills
  • ClawGUI-APP — On-Device Deploy
• Roadmap
• Acknowledgements
• License

💡 Overview

ClawGUI is a research framework for GUI agents, covering the complete lifecycle from online RL training and standardized evaluation to real-device deployment.

Building a capable GUI agent involves three tightly coupled problems that are rarely solved together: you need an environment to train the agent online, rigorous benchmarks to measure what it has learned, and a production system to deploy it on real devices. ClawGUI addresses all three.

🏗️ Architecture

🚀 Quick Start

git clone https://github.com/ZJU-REAL/ClawGUI.git
cd ClawGUI

Each module is independent with its own environment. Click into each one for full installation and usage instructions.

🚀 ClawGUI-RL — Build

📁 clawgui-rl/ · 📖 Full Documentation

ClawGUI-RL trains GUI agents with online reinforcement learning. It runs dozens of Docker-based Android emulators in parallel or trains directly on physical devices — and replaces standard GRPO with GiGPO+PRM for fine-grained step-level rewards that drive stronger policy learning.

• Parallel multi-environment — Dozens of Docker-based virtual Android environments simultaneously
• Real-device training — Physical or cloud Android phones with the same API
• GiGPO + PRM — Fine-grained step-level reward for better policy optimization than standard GRPO
• Spare server rotation — Automatic failover keeps training running without interruption
• Episode visualization — Record and replay any training trajectory

→ Get started with ClawGUI-RL

📊 ClawGUI-Eval — Evaluate

📁 clawgui-eval/ · 📖 Full Documentation · 🤗 Dataset · 🤖 ModelScope

ClawGUI-Eval gives GUI grounding research a reliable measurement baseline. Its three-stage Infer → Judge → Metric pipeline covers 6 benchmarks and 11+ models, with a 95.8% reproduction rate against official results — so numbers across papers are actually comparable.

• 6 benchmarks — ScreenSpot-Pro, ScreenSpot-V2, UIVision, MMBench-GUI, OSWorld-G, AndroidControl
• 11+ models — Qwen3-VL, Qwen2.5-VL, UI-TARS, MAI-UI, GUI-G2, UI-Venus, Gemini, Seed 1.8, and more
• Dual backend — Local GPU (transformers) or remote API (OpenAI-compatible)
• Multi-GPU & multi-thread — Parallel inference with automatic resume
• ClawGUI-Agent integration — Pair with ClawGUI-Agent to run the full pipeline via natural language

→ Get started with ClawGUI-Eval

🤖 ClawGUI-Agent — Deploy

📁 clawgui-agent/ · 📖 Full Documentation · 中文

ClawGUI-Agent closes the loop from training to production. Built on OpenClaw and powered by nanobot, it lets you control Android, HarmonyOS, or iOS devices with natural language from 12+ chat platforms — and trigger the full ClawGUI-Eval benchmark pipeline with a single sentence, no scripts required.

• Cross-platform — Android (ADB), HarmonyOS (HDC), iOS (XCTest)
• Multi-model — AutoGLM, MAI-UI, GUI-Owl, Qwen-VL, UI-TARS via OpenAI-compatible API
• One-command evaluation — Say "benchmark qwen3vl on screenspot-pro" and it handles env check → multi-GPU inference → judging → metrics → result comparison
• Personalized memory — Automatically learns user preferences and injects context across tasks
• Episode recording — Every task saved as structured episodes for replay and dataset building
• Web UI — Gradio interface for device management, task execution, and memory inspection

→ Get started with ClawGUI-Agent

🧩 ClawGUI-Skills — Self-Evolving Skills

📁 clawgui-skills/ · 📖 Full Documentation · 中文

ClawGUI-Skills implements the training-free self-evolving GUI skill architecture proposed and validated in our paper “Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agents.” It stores procedural task knowledge as structured skill packages and lets PhoneAgent retrieve, inject, diagnose, and revise them on demand.

• Four modes — off, trace, reuse, and evolve; disabled by default to avoid extra context cost
• Structured packages — meta_info.json, plan.md, backup.md, recover.md, and failure_examples/
• Instant revision — failed runs are diagnosed by an isolated verifier and mapped to targeted skill-file edits
• Visual inspection — the Web UI shows matched skill name, skill_id, injected context, revisions, and failure examples

→ Get started with ClawGUI-Skills

📱 ClawGUI-APP — On-Device Deploy

📁 clawgui-app/ · 📖 Setup Guide

ClawGUI-APP runs the full ClawGUI "brain + GUI agent" stack directly on one Android phone, removing the old split architecture where a desktop host orchestrates tasks and the phone only executes them. Built on Shizuku for high-privilege, non-root device control.

• Phone-only workflow — No desktop coordinator required; a device with Shizuku is enough
• Two-agent design — Brain LLM handles planning and tool orchestration, phone agent handles screen understanding and actions
• Multi-model support — AutoGLM, MAI-UI, GUI-Owl, Qwen-VL, UI-TARS and more via OpenAI-compatible API
• Voice input (STT) — Tap-to-record microphone with OpenAI-compatible speech-to-text transcription (SiliconFlow, Groq Whisper, etc.)
• Conversation + automation — Sessions, long-term memory, external channels (Feishu), and trace replay
• Built for real usage — Floating overlay status, built-in IME, session persistence, and diagnostics

→ Build ClawGUI-APP

🎯 Roadmap

• ClawGUI-Agent — GUI agent framework for phone control and evaluation via natural language
• ClawGUI-RL — Scalable mobile online RL training infrastructure with GiGPO + PRM
• ClawGUI-Eval — Standardized GUI grounding evaluation suite with 6 benchmarks and 95%+ reproduction rate
• ClawGUI-2B — 2B GUI agent trained with GiGPO, achieving 17.1 MobileWorld SR (vs. 11.1 baseline)
• On-device ClawGUI-Agent (ClawGUI-APP) — Deploy ClawGUI-Agent directly on real phones — no desktop coordinator, paving the way for fully on-device inference (brain/VLM still served via cloud API today)
• Desktop Online RL — Extend ClawGUI-RL to desktop environments for online reinforcement learning
• Web Online RL — Extend ClawGUI-RL to web environments for online reinforcement learning
• More Skills for ClawGUI-Agent — Add more pluggable skills to expand ClawGUI-Agent's capabilities
• Hybrid CLI & GUI Mechanism — Explore hybrid interaction combining command-line and GUI operations
• Real-time RL — Integrate real-time reinforcement learning based on the OPD algorithm for ClawGUI-RL and ClawGUI-Agent

🤝 Contributing

We welcome contributions of all kinds — new model support, new RL environments, bug fixes, and documentation improvements. See CONTRIBUTING.md for how to get started, module-specific guidelines, and PR requirements.

🙏 Acknowledgements

ClawGUI is built upon the following excellent open-source projects. We sincerely thank their contributors:

• verl-agent
• MAI-UI
• MobileWorld
• Mobile-Agent
• nanobot
• Open-AutoGLM

License

This project is licensed under the Apache License 2.0.

📝 Citation

If you find ClawGUI useful in your research, please consider citing our paper:

@article{tang2026clawgui,
  title={ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents},
  author={Tang, Fei and Lu, Zhiqiong and Zhang, Boxuan and Lu, Weiming and Xiao, Jun and Zhuang, Yueting and Shen, Yongliang},
  journal={arXiv preprint arXiv:2604.11784},
  year={2026}
}

Star History

Fixes

• Feishu GUI runner now drives step-by-step. The chat bubble updates with the current action on every step instead of staying frozen on "正在执行任务…". Each step has a 5-minute timeout so a wedged LLM call can't hang the loop forever.
• Clean error surfacing for Feishu replies. SDK errors are decoded to code: message instead of the previous Error@<hash>. Pre-flight failures (missing Vision creds, no device-control auth) now reply to Feishu with a human-readable reason instead of silently bailing. Image-reply failures land in Settings → 外部通道 → 调试日志 so users can see them without logcat.
• @_user_N mention stripping. Was hardcoded to @_user_1 ; now regex-matches @_(user|chat)_N so @bot 发朋友圈 reaches PhoneAgent as just 发朋友圈.
• Live screenshot fallback when image-reply is enabled but trace recording is off — users still get a final-state image instead of silent nothing.
• IME enabled check via the Android Framework API instead of ime list -s. Users who'd already enabled "ClawGUI Input" via system settings would see "未启用" in our panel because the shell command needed Shizuku/wadb auth to succeed. Settings → 输入法 also dropped the manual switcher button — agent handles the switching itself.

Chore

• Repo cleanup: retired the v1 clawgui-app/ (Java client) and promoted clawgui-app-ng/ into the natural clawgui-app/ path. Android applicationId stays com.clawgui.ng so existing v0.3.0 installs upgrade in place.