Wei-Chiu Ma 馬惟九

Assistant Professor @ Cornell CS

Email / Google scholar / Twitter / Misc. / Pro Bono

About Me

I am an Assistant Professor of Computer Science at Cornell University.

My research lies at the intersection of 3D/4D computer vision and robotics. I am interested in building AI systems that can understand, reconstruct, and re-simulate our dynamic world, and leverage these capabilities to enable more robust autonomous systems or advance entertainment applications.

Prior to joining Cornell, I was a Young Investigator/Postdoc at AI2/University of Washington. I received my Ph.D. from MIT, where I worked with Antonio Torralba (aka the Great Torralba) and Raquel Urtasun. Previously, I was a Senior Research Scientist at Uber ATG R&D and Waabi working on self-driving vehicles. I completed my M.S. in Robotics at Carnegie Mellon University (CMU), where I was advised by Kris M. Kitani.

Prospective students: I am always looking for motivated and talented students! If you are interested in collaborating or joining my group as a PhD/MS/Undergrad student or intern, please read this.


 

Recent News

  • NEW Pro bono: I will be hosting pro bono office hours starting 2021. Please check out here for more details.

  • NEW CVPR Workshop: We will be organizing the second workshop on Synthetic Data for Computer Vision at CVPR 2025. Stay tuned for more details!
  • NEW CVPR Workshop: We will be organizing the first workshop on Agent in Interaction, from Humans to Robots at CVPR 2025. Stay tuned for more details!

  • NEW Dec. 2024: We released the 360-1M dataset! Check them out here!
  • NEW Dec. 2024: Can AI system generate 3D worlds from a single image? Yes! All you need is the RIGHT data!
  • NEW Oct. 2024: We organized the 3D Modeling, Reconstruction, and Generation in the Wild workshop at ECCV 2024!
  • Jul. 2024: Can Multimodal LLMs (GPT-4V) perceive the world as humans do? See our ECCV paper for answers!
  • Apr. 2024: How can synthetic data benefit computer vision? Attend our CVPR SynData4CV workshop to find out!
  • Mar. 2024: Check out our latest effort on building realistic, interactive, and game-engine compatible digital twins!
  • Sep. 2023: Two papers on in-the-wild/extreme 3D inverse graphics accepted to NeurIPS 2023!
  • Sep. 2023: I am now Dr. Ma! Thank you, Antonio and Raquel, for your guidance and tremendous support!
  • Apr. 2023: Our work on extreme-view geometry is featured on Vox! Check it out!
  • Apr. 2023: Selected as a Cyber-Physical Systems (CPS) rising star!
  • Mar. 2023: Check out our latest effort on closed-loop sensor simulation, LiDAR generation, and thermal imaging!
  • Oct. 2022: Gave a talk at CMU, Columbia, and UIUC on in-the-wild 3D modeling, generation, and simulation!
  • Sep. 2022: Check out our latest effort on 3D scene generation, sensor simulation, and neural fields for manipulation!
  • Aug. 2022: Selected as a Siebel Scholar!
  • Apr. 2022: Gave a talk at Harvard on exploiting high-level vision for level-vision!
  • Mar. 2022: Our work on extreme-view 3D reconsturction and planar neural field are accepted to CVPR 2022!
  • Jul. 2021: BARF! Train your own NeRF from a collection of images without knowing camera poses!
  • Mar. 2021: Check out our latest effort on simulating pedestrians in the wild at CoRL and CVPR!
  • Jul. 2020: Four papers (two spotlight) accepted to ECCV 2020! Stay tuned!
  • Mar. 2020: Our work on LiDAR simulation and instance segmentation are accepted to CVPR 2020!
  • Oct. 2019: The source code of our real-time stereo algorithm is available now! Make sure to check out the amzaing differentiable PatchMatch module!
  • Jul. 2019: Three papers accepted to ICCV 2019! Details coming soon!
  • Jun. 2019: Our paper on light-weight localization is accepted to IROS 2019!
  • Mar. 2019: Two papers (scene flow and road boundary extraction) accepted to CVPR 2019!
  • Jul. 2018: Our paper on Unsupervised Intrinsic Decomposition is accepted to ECCV 2018!
  • Mar. 2018: 3 papers accepted to CVPR 2018!
  • Apr. 2017: Our book chapter on Activity Forecasting is published! Check it out!
  • Mar. 2017: Our work (a game-theoretic approach to multi-agent activity forecasting) is accepted to CVPR 2017!
  • Jan. 2017: Our work (self-localization for autonomous vehicles) is accepted to ICRA 2017!
  • May. 2016: I graduated from CMU --- thanks Kris for all your support in the past two years!

 

Research Projects

  (show selected / show by date)

Prompting with the Future: Open-World Model Predictive Control with Interactive Digital Twins
Chuanruo Ning, Kuan Fang*, Wei-Chiu Ma*
RSS 2025 / project page / arXiv / code


Beyond the Frame: Generating 360° Panoramic Videos from Perspective Videos
Rundong Luo, Matthew Wallingford, Ali Farhadi, Noah Snavely, Wei-Chiu Ma
ICCV 2025 / project page / arXiv / code (coming soon)


DRAWER: Digital Reconstruction and Articulation With Environment Realism
Hongchi Xia, Entong Su, Marius Memmel, Arhan Jain, Raymond Yu, Numfor Mbiziwo-Tiapo, Ali Farhadi, Abhishek Gupta, Shenlong Wang, Wei-Chiu Ma
CVPR 2025 / project page / arXiv (coming soon) / code (coming soon)


Eval3D: Interpretable and Fine-grained Evaluation for 3D Generation
Shivam Duggal*, Yushi Hu*, Oscar Michel, Aniruddha Kembhavi, William T. Freeman, Noah A. Smith, Ranjay Krishna, Antonio Torralba, Ali Farhadi, Wei-Chiu Ma
CVPR 2025 / project page / arXiv / code


From an Image to a Scene: Learning to Imagine the World from a Million 360° Videos
Matthew Wallingford, Anand Bhattad, Aditya Kusupati, Vivek Ramanujan, Matt Deitke, Sham Kakade, Aniruddha Kembhavi, Roozbeh Mottaghi, Wei-Chiu Ma, Ali Farhadi
NeurIPS 2024 / project page / arXiv / huggingface


BLINK: Multimodal Large Language Models Can See but Not Perceive
Xingyu Fu*, Yushi Hu*, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma†, Ranjay Krishna†
ECCV 2024 / project page / paper / dataset / Eval AI / code


Video2Game: Real-time, Interactive, Realistic and Browser-Compatible Environment from a Single Video
Hongchi Xia, Zhi-Hao Lin, Wei-Chiu Ma, Shenlong Wang
CVPR 2024 / project page / paper / shooting demo / code


UniSim: A Neural Closed-Loop Sensor Simulator
Ze Yang*, Yun Chen*, Jingkang Wang*, Sivabalan Manivasagam*, Wei-Chiu Ma, Joyce Anqi Yang, Raquel Urtasun
CVPR 2023 / project page / paper / 4K demo / video (8 mins)
Highlight presentation


UltraLiDAR: Learning Compact Representations for LiDAR Completion and Generation
Yuwen Xiong, Wei-Chiu Ma, Jingkang Wang, Raquel Urtasun
CVPR 2023 / project page / paper / video (1 min)


SGAM: Building a Virtual 3D World through Simultaneous Generation and Mapping
Yuan Shen, Wei-Chiu Ma, Shenlong Wang
NeurIPS 2022 / project page / paper / code


CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Realistic and Controllable Sensor Simulation
Jingkang Wang, Sivabalan Manivasagam, Yun Chen, Ze Yang, Ioan Andrei Bârsan, Joyce Anqi Yang, Wei-Chiu Ma, Raquel Urtasun
CoRL 2022 / project page / paper / video


Virtual Correspondence: Humans as a Cue for Extreme-View Geometry
Wei-Chiu Ma, Anqi Joyce Yang, Shenlong Wang, Raquel Urtasun, Antonio Torralba
CVPR 2022 / project page / paper / video (1.5 mins) / video (5 mins) / MIT News / TechXplore


NeurMiPs: Neural Mixture of Planar Experts for View Synthesis
Zhi-Hao Lin, Wei-Chiu Ma, Hao-Yu Max Hsu, Yu-Chiang Frank Wang, Shenlong Wang
CVPR 2022 / project page / paper / code / video


BARF: Bundle-Adjusting Neural Radiance Fields
Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, Simon Lucey
ICCV 2021 / project page / arXiv / code / video / news coverage (The Batch: DeepLearning.AI)
Oral presentation


Deep Feedback Inverse Problem Solver
Wei-Chiu Ma, Shenlong Wang, Jiayuan Gu, Sivabalan Manivasagam, Antonio Torralba, Raquel Urtasun
ECCV 2020 / project page / arXiv / short video (1.5 mins) / long video (10 mins)
Spotlight presentation


LidarSIM: Realistic LiDAR Simulation by Leveraging the Real World
Sivabalan Manivasagam, Shenlong Wang, Kelvin Wong, Wenyuan Zeng, Bin Yang, Shuhan Tan, Mikita Sazanovich, Wei-Chiu Ma, Raquel Urtasun
CVPR 2020 / paper
Oral presentation

Exploiting Sparse Semantic HD Maps for Self-Driving Vehicle Localization
Wei-Chiu Ma*, Ignacio Tartavull*, Ioan Andrei Bârsan*, Shenlong Wang*, Min Bai, Gellert Mattyus, Namdar Homayounfar, Shrinidhi K. Lakshmikanth, Andrei Pokrovsky, Raquel Urtasun
IROS 2019 / arXiv / video
Oral presentation

Deep Rigid Instance Scene Flow
Wei-Chiu Ma, Shenlong Wang, Rui Hu, Yuwen Xiong, Raquel Urtasun
CVPR 2019 / project page / arXiv / paper + supp (uncompressed) / GN solver gif

Deep structured scene flow model that rank 1st on KITTI Scene Flow Benchmark.
Faster than prior art by 800 times.

DeepPruner: Learning Efficient Stereo Matching via Differentiable PatchMatch
Shivam Duggal, Shenlong Wang, Wei-Chiu Ma, Rui Hu, Raquel Urtasun
ICCV 2019 / arXiv / code / differentiable PatchMatch module

Real-time stereo estimation (62 ms) via Differentiable PatchMatch!

Deep Parametric Continuous Convolutional Neural Networks
Shenlong Wang*, Simon Suo*, Wei-Chiu Ma, Andrei Pokrovsky, and Raquel Urtasun
CVPR 2018 / paper
Spotlight presentation

Find Your Way by Observing the Sun and Other Semantic Cues
Wei-Chiu Ma, Shenlong Wang, Marcus A. Brubaker, Sanja Fidler, Raquel Urtasun
ICRA 2017 / arXiv / demo video
Oral presentation

 

Thesis


Learning 3D Modeling and Simulation From and For the Real World
Wei-Chiu Ma
Ph.D. Thesis

 

Pro bono office hour

Inspired by Prof. Kyunghyun Cho and Krishna Murthy, starting January 2021, I have decided to commit 1~2 hours every week to provide guidance, suggestions, and/or mentorships for students from underrepresented groups or whoever is in need. Please fill in this form if you are interested.

Need more (diverse) opinions? Consider talking to people with different expertise or from different background: Tongzhou Wang (ML/RL), Alexander Haojan Liu (Speech/NLP), Zhijian Liu (ML), Vlas Zyrianov (CV), Yuan Shen (CV), Jun Gao (CV/Graphics).

Misc.


Accessibility