ALFRED Challenge

Leaderboard is accepting submissions to the ALFRED Embodied AI Challenge with [EAI22] tags! Humans have a success rate of 91% on unseen environments, but the best models are still behind 😢

Can you do even better? Code, precomputed features, and AI2Thor simulator are all available on Github. For a quick start, checkout FILM – a SoTA agent.

Leaderboard

Leaderboard»

ALFRED Challenge @ CVPR 2022 Embodied AI Workshop

Overview

The focus of this challenge is on embodied visual tasks that require the grounding of language to actions in real-world settings. Specifically, we want to draw focus to challenges like partial observability, continuous state spaces, and irrevocable actions for language-guided agents in visual environments. Such challenges are not captured by current datasets for embodiment [1, 2, 3].

Key Topics

Egocentric and Robotic vision
Language Grounding
Navigation and Path Planning
Interactive/Causal Reasoning
Learning from Demonstration
Task and Symbolic Planning
Deep Reinforcement Learning
Commonsense Reasoning

To encourage research in embodied vision & language, the workshop includes a benchmark challenge based on ALFRED. This benchmark captures real-world complexities like object state changes, and requires long-horizon planning. This workshop exists to bring together Vision, Robotics, and NLP researchers to tackle the unique challenges of this three-field intersection that are often avoided when focusing only on vision-and-language or vision-and-robotics (i.e., 'embodied AI').

Previous versions of this challenge were held at CVPR 2021 and ECCV 2020. The best performing submission by Blukis et al. acheived an unseen success of 16.3%.

Mohit Shridhar

Ishika Singh

Anthony Liang

Tiffany Min

Jesse Thomason

Yonatan Bisk

Roozbeh Mottaghi

Eric Kolve

Challenge Details

Guidelines
Participants are required to upload their model to our evaluation server with [EAI22] in the submission title, e.g. [EAI22] Seq2seq Model. The evaluation server automatically evaluates the models on an unseen test set. Final numbers for the prize challenge will be frozen on May 31. Winning submissions will be required to submit a brief (private) report of technical details for validity checking. We will also conduct a quick code inspection to ensure that the challenge rules weren't violated (e.g. using additional info on test scenes).

Dataset
The challenge is based on the ALFRED Dataset, which contains 25K language annotations of both high-level goals and low-level step-by-step instructions for various tasks set in the AI2THOR simulator. Agents interact with environments through discrete actions and pixelwise masks.

Important Dates

Timeline
Challenge Opens	Feb 14
Leaderboard closes	May 31
Winner announcement	Jun 17

Starter Code
Checkout the FILM repository by So Yeon Min et al.

Evaluation
The leaderboard script records actions taken by a pre-trained agent and dumps them to a JSON file. These deterministic actions in the JSON will be replayed on the leaderboard server for evaluation. This process is model-agnostic, allowing you to use your local resources for test-time inference.

Metric
The submissions will be ranked by Unseen Success Rate.

Rules

Include [EAI22] in the submission title e.g. [EAI22] Seq2seq Model.
Do not exploit the metadata in test-scenes: you should solve the vision-language grounding problem without misusing the metadata from THOR. For leaderboard evaluations, agents should just use RGB input and language instructions (goal & step-by-step). You cannot use additional depth, mask, metadata info etc. from the simulator on Test Seen and Test Unseen scenes. Submissions that use additional info on test scenes will be disqualified. However, during training you are allowed to use additional info for auxiliary losses etc.
During evaluation, agents are restricted to max_steps=1000 and max_fails=10. Do not change these settings in the leaderboard script; these modifications will not be reflected in the evaluation server.
You can publish your results on the leaderboard only once every 7 days.
Do not spam the leaderboard with repeated submissions (under different email accounts) in order to optimize on the test set. Fine-tuning should be done only on the validation set. Violators will be disqualified from the challenge.
Try to solve the ALFRED dataset: all submissions must be attempts to solve the ALFRED dataset.
Answer the following questions: a. Did you use additional sensory information from THOR as input, eg: depth, segmentation masks, class masks, panoramic images etc. during test-time? If so, please report them. b. Did you use the alignments between step-by-step instructions and expert action-sequences for training or testing? (no by default; the instructions are serialized into a single sentence)
Share who you are: you must provide a team name and affiliation.
(Optional) Share how you solved it: if possible, share information about how the task was solved. Link an academic paper or code repository if public.
Only submit your own work: you may evaluate any model on the validation set, but must only submit your own work for evaluation against the test set.

Submissions

Rank	Model	Unseen Success	Seen Success	Unseen PLW	Seen PLW
	Human Performance University of Washington (Shridhar et al. '20)	91.0	-	87.6	-
- 20 Feb, 2022	FILM CMU and Meta AI github.com/soyeonm/FILM	27.8	28.8	11.3	11.3
- 28 Mar, 2020	Seq2seq Baseline University of Washington github.com/askforalfred/alfred	0.4	4.0	0.0	2.0

FAQ

Do you need to submit a report?
Winning submissions will be required to submit a brief (private) report of technical details for validity checking. Also consider submitting a workshop paper to EAI by TBD. See submission guidelines for EAI.

Is there a prize for the winner?
TBD