GuideWeb: A Benchmark for Automatic In-App Guide Generation on Real-World Web UIs

Digital Adoption Platform (DAP) provide web-based overlays that deliver operation guidance and contextual hints to help users navigate complex websites. Although modern DAP tools enable non-experts to author such guidance, maintaining these guides remains labor-intensive because website layouts and functionalities evolve continuously, which requires repeated manual updates and re-annotation. In this work, we introduce $\mathbf{\text{GuideWeb}}$, a new benchmark for automatic in-app guide generation on real-world web UIs. GuideWeb formulates the task as producing page-level guidance by selecting $\mathbf{\text{guide}}\mathbf{\text{target}}\mathbf{\text{elements}}$ grounded in the webpage and generating concise guide text aligned with user intent. We also propose a comprehensive evaluation suite that jointly measures the accuracy of guide target element selection and the quality of generated intents and guide texts. Experiments show that our proposed $\mathbf{\text{GuideWeb}}\mathbf{\text{Agent}}$ achieves $\mathbf{\text{30.79\%}}$ accuracy in guide target element prediction, while obtaining BLEU scores of $\mathbf{\text{44.94}}$ for intent generation and $\mathbf{\text{21.34}}$ for guide-text generation. Existing baselines perform substantially worse, which highlights that automatic guide generation remains challenging and that further advances are necessary before such systems can be reliably deployed in real-world settings.