ALFRED Challenge

🏆 Challenge Winners 🏆

1^st Place 🥇: Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, Yoav Artzi (Cornell University and NVIDIA)
2^nd Place 🥈: Byeonghwi Kim, Suvaansh Bhambri, Kunal Pratap Singh, Roozbeh Mottaghi, Jonghyun Choi (GIST, IIT Roorkee, AI2)

Can you do even better? Code, precomputed features, and AI2Thor simulator are all available for a quick start on GitHub

Leaderboard

Leaderboard»

ALFRED Challenge @ CVPR 2021 Embodied AI Workshop

Overview

The focus of this challenge is on embodied visual tasks that require the grounding of language to actions in real-world settings. Specifically, we want to draw focus to challenges like partial observability, continuous state spaces, and irrevocable actions for language-guided agents in visual environments. Such challenges are not captured by current datasets for embodiment [1, 2, 3].

Key Topics

Egocentric and Robotic vision
Language Grounding
Navigation and Path Planning
Interactive/Causal Reasoning
Learning from Demonstration
Task and Symbolic Planning
Deep Reinforcement Learning
Commonsense Reasoning

To encourage research in embodied vision & language, the workshop includes a benchmark challenge based on ALFRED. This benchmark captures real-world complexities like object state changes, and requires long-horizon planning. This workshop exists to bring together Vision, Robotics, and NLP researchers to tackle the unique challenges of this three-field intersection that are often avoided when focusing only on vision-and-language or vision-and-robotics (i.e., 'embodied AI').

The previous version of this challenge was held at ECCV 2020 where the top scoring submission attained an Unseen Success Rate of 4.5%.

Mohit Shridhar

Jesse Thomason

Yonatan Bisk

Roozbeh Mottaghi

Eric Kolve

Challenge Details

Guidelines
Participants are required to upload their model to our evaluation server with [EAI21] in the submission title, e.g. [EAI21] Seq2seq Model. The evaluation server automatically evaluates the models on an unseen test set. Final numbers for the prize challenge will be frozen on ~~May 31~~ Jun 7. Winning submissions will be required to submit a brief (private) report of technical details for validity checking. We will also conduct a quick code inspection to ensure that the challenge rules weren't violated (e.g. using additional info on test scenes).

Dataset
The challenge is based on the ALFRED Dataset, which contains 25K language annotations of both high-level goals and low-level step-by-step instructions for various tasks set in the AI2THOR simulator. Agents interact with environments through discrete actions and pixelwise masks.

Important Dates

Timeline
~~Leaderboard closes~~	~~May 31~~
~~Leaderboard closes~~	~~Jun 7~~
~~Winner announcement~~	~~Jun 19~~

Evaluation
The leaderboard script records actions taken by a pre-trained agent and dumps them to a JSON file. These deterministic actions in the JSON will be replayed on the leaderboard server for evaluation. This process is model-agnostic, allowing you to use your local resources for test-time inference.

Metric
The submissions will be ranked by Unseen Success Rate.

Rules

Include [EAI21] in the submission title e.g. [EAI21] Seq2seq Model.
Do not exploit the metadata in test-scenes: you should solve the vision-language grounding problem without misusing the metadata from THOR. For leaderboard evaluations, agents should just use RGB input and language instructions (goal & step-by-step). You cannot use additional depth, mask, metadata info etc. from the simulator on Test Seen and Test Unseen scenes. Submissions that use additional info on test scenes will be disqualified. However, during training you are allowed to use additional info for auxiliary losses etc.
During evaluation, agents are restricted to max_steps=1000 and max_fails=10. Do not change these settings in the leaderboard script; these modifications will not be reflected in the evaluation server.
You can publish your results on the leaderboard only once every 7 days.
Try to solve the ALFRED dataset: all submissions must be attempts to solve the ALFRED dataset.
Answer the following questions: a. Did you use additional sensory information from THOR as input, eg: depth, segmentation masks, class masks, panoramic images etc. during test-time? If so, please report them. b. Did you use the alignments between step-by-step instructions and expert action-sequences for training or testing? (no by default; the instructions are serialized into a single sentence)
Share who you are: you must provide a team name and affiliation.
(Optional) Share how you solved it: if possible, share information about how the task was solved. Link an academic paper or code repository if public.
Only submit your own work: you may evaluate any model on the validation set, but must only submit your own work for evaluation against the test set.

Prizes

🏆 1st Place – 1,500 USD Amazon Gift Card

🥈 2nd Place – 500 USD Amazon Gift Card

Submissions

Rank	Model	Unseen Success	Seen Success	Unseen PLW	Seen PLW
	Human Performance University of Washington (Shridhar et al. '20)	91.0	-	87.6	-
🥇 07 Jun, 2021	HLSM Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, Yoav Artzi	16.3	25.1	4.3	6.7
🥈 06 Jun, 2021	ABP Byeonghwi Kim, Suvaansh Bhambri, Kunal Pratap Singh, Roozbeh Mottaghi, Jonghyun Choi. (GIST, IIT Roorkee, AI2)	15.4	44.5	1.1	3.9
3 30 Jan, 2021	HiTUT Yichi Zhang and Joyce Chai (University of Michigan)	13.9	21.3	5.9	11.1
4 04 Jan, 2021	LWIT Van-Quang Nguyen, Masanori Suganuma, Takayuki Okatani	9.4	30.9	5.6	25.9
5 07 Jun, 2021	SRCB-sudoer Xingrui Wang. HaoyuLiu and YangLiu, Samsung Research, China	5.6	21.3	2.7	14.8
6 06 Jun, 2021	EmBERT Anonymous	5.0	31.5	2.2	24.3
- 28 Mar, 2020	Seq2seq Baseline University of Washington github.com/askforalfred/alfred	0.4	4.0	0.0	2.0

FAQ

Do you need to submit a report?
Winning submissions will be required to submit a brief (private) report of technical details for validity checking. Also consider submitting a workshop paper to EAI by May 13. See submission guidelines for EAI.

Is there a prize for the winner?
Yes! 1,500 USD Amazon Gift Card for First Place, and 500 USD Amazon Gift Card for Second Place.

ALFRED Challenge

Leaderboard