VisualWebArena
Freea benchmark designed to assess the performance of multimodal web agents on realistic visually grounded tasks.
About VisualWebArena
VisualWebArena is a benchmark introduced at ACL 2024 to evaluate multimodal web agents on realistic visually grounded tasks. It comprises 910 diverse tasks across three web environments: a new Classifieds site (with real-world data) and existing Shopping and Reddit sites from WebArena. The benchmark requires agents to process image-text inputs, interpret natural language instructions, and execute actions on websites to accomplish user-defined objectives. It introduces visually grounded evaluation metrics as part of an execution-based evaluation paradigm, and proposes a Set-of-Marks (SoM) preprocessing step that annotates interactable elements with bounding boxes and IDs to improve navigability. The project provides code, data, and a leaderboard for systematic benchmarking of multimodal autonomous agents.
Key Features
Pros & Cons
- Provides a realistic, visually grounded benchmark for multimodal web agents
- Includes execution-based evaluation for objective performance assessment
- Introduces Set-of-Marks representation to improve agent navigability
- Open-source with publicly available code, data, and leaderboard
- Comprehensive evaluation of state-of-the-art agents reveals actionable insights
- Limited to three specific web environments (Classifieds, Shopping, Reddit)
- Requires local setup of environments for evaluation, which may add overhead
- May not cover all types of web tasks or real-world complexity
- Benchmark focuses on visual information but still relies on predefined websites