Matt Deitke
Matt Deitke
AI Researcher at AI2 and UW CSE
mattd@allenai.org
Seattle, WA
Research Statement
CV

Matt Deitke

Hi! I work on the PRIOR team at the Allen Institute for AI and am also an undergraduate in the Allen School at the University of Washington, Seattle. My research interests are in computer vision, deep learning, and embodied AI. I am interested in building AI systems that are broadly useful and robust. Recently, I have led the development of ProcTHOR , Objaverse, and Phone2Proc.

Previously, I grew up near Chicago and spent several teenage years working on computer graphics, interface design, and visualization in the Department of Athletics at The Ohio State University, the University of Cincinnati, and a variety of other organizations. Later in high school, I studied machine learning and deep learning at Georgia Tech.

News

NeurIPS Outstanding Paper Award
ProcTHOR won the Outstanding Paper Award at NeurIPS 2022 from over 10K submissions!
Nov, 2022
Avatar
Thrilled to announce Objaverse, a massive dataset of 3D objects with broad applications across AI.
Dec, 2022
Avatar
Giving an invited talk at UW's Vision Lunch titled, "Scaling Embodied AI with ProcTHOR: Where We Are and What's Next."
Jun, 2022
Avatar
The 2nd edition of Rick Szeliski's Computer Vision textbook was published! Ecstatic to have contributed!
Jan, 2022
Avatar
Extremely excited about the release of ProcTHOR! We use procedural generation to scale up the diversity of data in Embodied AI and achieve remarkable generalization results.
Jun, 2022
Avatar
Excited to release a retrospectives on the Embodied AI workshops! We cover common approaches in embodied AI, discuss its scope, and discuss future directions.
Oct, 2022
Avatar
Co-Organizing the Embodied AI Workshop and the AI2-THOR Rearrangement Challenge at CVPR 2022 in New Orleans.
Jun, 2022
Avatar
We released an updated revision of the AI2-THOR paper covering its impact and new features!
Aug, 2022
Avatar
Serving as a reviewer for CVPR 2023 and ICLR 2023!
Aug, 2022

Preprints

Objaverse

Objaverse: A Universe of Annotated 3D Objects

Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, Ali Farhadi

Under Review
TLDR
Objaverse is a massive dataset of objects with 800K+ (and growing) 3D models with descriptive captions, tags, and animations. We demonstrate it's potential by training generative models, improving 2D instance segmentation, training open-vocabulary object navigation models, and creating a benchmark for testing the robustness of vision models.
1
PDF
Semantic Scholar
arXiv
Phone2Proc

Phone2Proc: Bringing Robust Robots Into Our Chaotic World

Matt Deitke*, Rose Hendrix*, Luca Weihs, Ali Farhadi, Kiana Ehsani, Aniruddha Kembhavi

Under Review
TLDR
From a 10-minute iPhone scan of any environment, we generated simulated training scenes that semantically match that environment. Training a robot to perform ObjectNav in these scenes dramatically improves sim-to-real performance from 35% to 71% and results in an agent that is remarkably robust to human movement, lighting variations, added clutter, and rearranged objects.
0
PDF
Semantic Scholar
arXiv

Publications

ProcTHOR

🏘️ ProcTHOR: Large-Scale Embodied AI Using Procedural Generation

Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Jordi Salvador, Kiana Ehsani, Winson Han, Eric Kolve, Ali Farhadi, Aniruddha Kembhavi, Roozbeh Mottaghi

NeurIPS 2022 Outstanding Paper Award
TLDR
We built a platform to procedurally generate realistic, interactive, simulated 3D environments to dramatically scale up the diversity and size of training data in Embodied AI. We find that it helps significantly with performance on many tasks.
17
PDF
Semantic Scholar
arXiv
Features in AI2-THOR

Retrospectives on the Embodied AI Workshop

Matt Deitke, Dhruv Batra, Yonatan Bisk, Tommaso Campari, Angel X. Chang, Devendra Singh Chaplot, Changan Chen, Claudia Pérez D'Arpino, Kiana Ehsani, Ali Farhadi, Li Fei-Fei, Anthony Francis, Chuang Gan, Kristen Grauman, David Hall, Winson Han, Unnat Jain, Aniruddha Kembhavi, Jacob Krantz, Stefan Lee, Chengshu Li, Sagnik Majumder, Oleksandr Maksymets, Roberto Martín-Martín, Roozbeh Mottaghi, Sonia Raychaudhuri, Mike Roberts, Silvio Savarese, Manolis Savva, Mohit Shridhar, Niko Sünderhauf, Andrew Szot, Ben Talbot, Joshua B. Tenenbaum, Jesse Thomason, Alexander Toshev, Joanne Truong, Luca Weihs, Jiajun Wu

ArXiv 2022
TLDR
We present a retrospective on the state of Embodied AI research. Our analysis focuses on 13 challenges in visual navigation, rearrangement, and embodied vision-and-language. We discuss the scope of embodied AI research, performance of state-of-the-art models, common modeling approaches, and future directions.
6
PDF
Semantic Scholar
arXiv
Features in AI2-THOR

AI2-THOR: An Interactive 3D Environment for Visual AI

Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, Aniruddha Kembhavi, Abhinav Gupta, Ali Farhadi

ArXiv 2022
TLDR
We introduce The House Of inteRactions (THOR), a framework for visual AI research. AI2-THOR consists of near photo-realistic 3D indoor scenes, where AI agents can navigate in the scenes and interact with objects to perform tasks. It has enabled research in many areas of AI.
561
PDF
Semantic Scholar
GitHub
arXiv
Room Rearrangement in AI2-THOR

Visual Room Rearrangement

Luca Weihs, Matt Deitke, Aniruddha Kembhavi, Roozbeh Mottaghi

CVPR 2021 Oral Presentation
TLDR
We built a pre-training task where the agent's goal is to interactively rearrange objects in a room from one state to another. For instance, the agent may have to open the Fridge and move the Lettuce to the CounterTop. Modern deep-RL struggles.
43
PDF
Semantic Scholar
GitHub
arXiv
Video
RoboTHOR

RoboTHOR: An Open Simulation-to-Real Embodied AI Platform

Matt Deitke*, Winson Han*, Alvaro Herrasti*, Aniruddha Kembhavi*, Eric Kolve*, Roozbeh Mottaghi*, Jordi Salvador*, Dustin Schwenk*, Eli VanderBilt*, Matthew Wallingford*, Luca Weihs*, Mark Yatskar*, Ali Farhadi

CVPR 2020
TLDR
We rent office buildings in Seattle and turn them into apartment studios with many possible furniture and wall layouts. Each apartment layout is then computationally remodeled by hand to enable a simulated robot to interact with it in video-game-like context. We study how well a robot trained purely in the simulated environments can transfer to reality.
113
PDF
Semantic Scholar
GitHub
arXiv

Software

AI2-THOR

AI2-THOR consists of real and simulated environments for interactive robot learning.

allenai/ai2thor
Framework
Interactive Simulated Environments for Embodied AI
Unity
Procedurally Generate Houses for Embodied AI Training
Python
A Framework for Training Embodied-AI Agents
PyTorch
Code for Running the Visual Room Rearrangement task
PyTorch
Python Package for Distributing Datasets and Models
Python
Run AI2-THOR with Google Colab
Colab
The ProcTHOR-10K Houses Dataset
Python
Evaluation tasks for ObjectNav models
Python
The Website for the Embodied AI Workshop at CVPR
React
Explore Trending Papers at CVPR
React

Workshops

Embodied AI

I've co-organized the Embodied AI workshops at CVPR. Our goal is to bring together researchers to share and discuss the current state of intelligent agents that can see, talk, act, and reason.