GUI-HARVEST

Self-improving GUI agents · Under review at ICLR 2027 · arXiv preprint

Geyi Yang, Zikun Qu, Xiang Li, Zhiyong Wang, Min Zhang, Shipei Zeng, and Zhongxiang Dai

First author · Research Assistant at Shenzhen Research Institute of Big Data · advised by Prof. Zhongxiang Dai

arXiv paper HF Paper Code Back to projects

GUI-HARVEST enables a frozen GUI agent to improve its own executable runtime harness from repeated multimodal executions, without updating the backbone model’s weights.

GUI-HARVEST evidence-driven self-improvement loop for frozen GUI agents — Repeated GUI executions support evidence analysis, cross-task failure clustering, bounded harness edits, and score-plus-behavior validation before an update is promoted.

6backbones improved

+12.33Qwen full-suite points

+13.87GPT-5 transfer points

Why GUI harness evolution is different

A GUI harness controls how observations are assembled, actions are executed, and verification, recovery, and termination are handled. Optimizing it requires more than reading textual traces: the system must reconcile model intent with visible interface changes, diagnose failures despite run-to-run execution variability, and turn task-local evidence into reusable runtime changes.

Evidence-driven self-improvement

GUI-HARVEST aligns model outputs and executed actions with before-and-after screenshots, then compares repeated runs of each task as a joint evidence unit. Verified findings are consolidated into recurring cross-task failure patterns. A Harness Engineer maps those patterns to bounded source-code edits and records predicted behavioral effects before evaluation; a Validator promotes an edit only when repeated executions pass both score and behavior checks.

Results and transfer

On OSWorld-Verified, GUI-HARVEST produces consistent held-out gains across six general-purpose open, GUI-specialized open, and proprietary backbones. With Qwen3-VL-32B-Instruct at a 15-step budget, the full-suite score rises from 38.61% to 50.94% (+12.33 points). The frozen OSWorld-derived harness also transfers to WindowsAgentArena without further optimization, improving GPT-5 from 50.88% to 64.75% (+13.87 points) at 50 steps.

The contribution is not a single hand-designed harness. It is an evidence-driven optimization procedure that lets frozen GUI agents convert their own repeated executions into validated, reusable runtime improvements.