Benchmark complete vision-language systems in a controlled zero-shot evaluation, or help uncover compelling grounding failures that matter for real-world deployment.
Build and evaluate a VLM or a multimodal pipeline on VistaQA in a zero-shot setting. Custom architectures, training on external data, and tool- or agent-augmented systems are welcome, provided VistaQA is used only for final evaluation.
View challenge detailsSubmit compelling image or video cases that expose grounding failures in one or more VLMs, especially failures relevant to real-world use.
View challenge detailsHow well can a VLM answer correctly and ground its answers in the right visual evidence when evaluated zero-shot on VistaQA?
Participants may develop any vision-language model or system and evaluate it on VistaQA in a zero-shot setting. Training or fine-tuning on datasets other than VistaQA is allowed, and systems may use custom architectures, external tools, retrieval components, agents, or other inference-time methods. The challenge uses the official VistaQA evaluation protocol, with GROVE as the primary metric for jointly assessing answer correctness and visual grounding.
Complete the team registration form once per participating team by October 31, 2026.
Use any model architecture, external training data, tools, agents, retrieval, or inference strategy, as long as VistaQA is not used for training, fine-tuning, or development.
Produce answers and visual evidence predictions using the official format.
Submit a system report describing your model or system, training data, external tools, agents, retrieval components, prompting strategy, and any other components used to produce the results. The report should also include links to your submitted evaluation results and, where available, the model, checkpoint, code repository, or system implementation.
VistaQA samples, labels, masks, or derived annotations may not be used for model training, fine-tuning, prompt optimization, model selection, hyperparameter tuning, or any other development step. Training and fine-tuning on external datasets are allowed. Participants may use custom model architectures, external tools, agents, retrieval systems, and other components, but must clearly disclose the complete system and all external resources used in their submission report. VistaQA is reserved for final evaluation only.
Each participating team must submit a system report through OpenReview by November 15, 2026. The report should be written in the workshop paper format and may be up to 4 pages, excluding references and appendix.
The report should clearly describe the complete system used to produce the submitted VistaQA results, including:
Show us where today’s VLMs fail to connect their responses to the right visual evidence.
We invite researchers to submit compelling real-world instances revealing grounding failures in vision-language models. We especially welcome cases that are surprising, systematic, practically important, or reveal risks relevant to real-world deployment.
An image, short video, screenshots, and one-page PDF.
The prompt or question used to elicit the failure.
Affected model family/families and exact versions where available.
A concise explanation of what was grounded incorrectly and why the case matters.
Clear. The grounding failure is easy to understand and supported by the submitted evidence.
Insightful. The case reveals something meaningful about how current VLMs fail.
Relevant. The failure has implications for real-world use, reliability, or safety.
Reproducible. Stronger cases may recur across runs, prompts, or multiple models.
Selected and notable grounding failure cases will be highlighted and recognized during the workshop.
Challenge submission deadlines and result announcement dates will be posted here. All deadlines will follow Anywhere on Earth (AoE) unless otherwise stated.
For questions about eligibility, evaluation, or submissions, please contact the VLM4RWD organizing team.
