Romy Mi Luo

(Romy) Mi LUO

Hi there 👋! I am a final-year PhD student at UT Austin, co-advised by Prof. Kristen Grauman and Prof. Alex Dimakis.

I am currently a Student Researcher at Google DeepMind, working closely with Xuezhi Wang, Lijun Yu, Canoee Liu, and Adams Yu. Previously, I interned at Google, where I was fortunate to work with Du Tran, Wen-Sheng Chu, and Yujia Chen. Before joining UT Austin, I worked with Jiashi Feng at the National University of Singapore.

My research focuses on building multimodal generalist AI systems that understand, generate, and reason about the world they act in, from the digital to the physical, with broader interests in vision language models, multimodal agents and world models. [read more]

News

  • [Nov. 2026]Check out our recent work on personalized visual context learning 📄.
  • [Sep. 2026]Two papers accepted to NeurIPS 2026: AnchorSim and VIOS 🎉. See you in Sydney 🇦🇺!
  • [Jul. 2026]LDO is accepted to ECCV 2026 🎉. See you in Malmö 🇸🇪!
  • [May 2026]Started as a Student Researcher at Google DeepMind in Mountain View .
  • [May 2026]Received the ICML 2026 Silver Reviewer Award 🥈.
  • [Sep. 2025]Two papers accepted to NeurIPS 2025: Video-VER and AoT 🎉. See you in San Diego 🌴!
  • [May 2025]Started as a Student Researcher at Google in Mountain View .
  • [Feb. 2025]ViewpointRosetta is accepted to CVPR 2025 as an Oral 🎉. See you in Nashville 🎸!
  • [Sep. 2024]Two papers accepted to NeurIPS 2024: HOI-Swap and EnsV 🎉. See you in Vancouver 🍁!
  • [Jul. 2024]Two papers accepted to ECCV 2024: 4DIFF and Exo2Ego 🎉. See you in Milan 🇮🇹!
  • [May 2024]Started a research internship at FAIR, Meta AI in Menlo Park .
  • [Feb. 2024]Ego-Exo4D is accepted to CVPR 2024 as an Oral 🎉. See you in Seattle 🏔️!

Research Interests

My research builds multimodal generalist AI systems that cross from the digital world into the physical world. I work on the learning signals and data that let multimodal LLMs ground their reasoning in verifiable evidence, acquire skill from existing human demonstrations, and represent the state of the world they act in. Specifically:

  • Sketch: an agent holds a magnifying glass up to a photo and a written note, with a film strip of video evidence and a green check mark.
    Digital Agents → Verifiable Reasoning. I am interested in agents that reason over image, video, language, and interface without drifting from what is actually there. My work develops verifiable rewards, self-auditing, and long-term memory so that reasoning stays anchored in visual evidence and personalized to the user.
    Video-VERAnchorSimPersonal Visual Context LearningVIOS
  • Sketch: an agent between film strips of a first-person hand and a third-person figure holding a mug, with dashed arrows running to a robot arm holding the same mug.
    Human Video → Transferable Skill. I am interested in turning human video, especially egocentric data, into scalable training data for generalist agents. My work explores how paired and unpaired first-person and third-person web video, multimodal human demonstrations, and 3D geometry enable knowledge transfer across viewpoints, from watching someone else act to the first-person view an agent would have itself.
    Viewpoint Rosetta StoneEgo-Exo4D4DiffExo2Ego
  • Sketch: an agent beside a dashed cloud labelled RL environment, where a robot hand tries and retries dropping a ball into a bowl.
    World Models → Physical Intelligence. I am interested in world models in two roles: as interactive environments an agent can act in, learn from, and co-evolve with, and as a representation of world state, especially causality and physics. My work tests whether today's frontier video models actually hold that state. Largely they do not. I think the missing piece is latent state: a model has to recover what is actually in a scene before it can predict what follows, which is why structured representations of that state matter.
    FlipTestSeeing the Arrow of TimeLatent Dynamics OptimizationHOI-Swap

Publications & Manuscripts

My work falls into three threads: 🧠 multimodal reasoning & agents, 🎬 video world models & multimodal generation, and ⚙️ model architecture, optimization & reliability. In total, these projects cover multimodal data, post-training / RL rewards, and evaluation / benchmarks.

🧠 Multimodal Reasoning & Agents

Data, RL / Verifiable Rewards, Evaluation, Personalization

  • When Thinking Drifts project logo
    When Thinking Drifts: Evidential Grounding for Robust Video Reasoning Mi Luo, Zihui Xue, Alex Dimakis, Kristen Grauman
    In Advances in Neural Information Processing Systems, NeurIPS 2025.
    [PDF][Project]
    Vision Language ModelsEvidence-Grounded Multi-modal ReasoningRL Reward
  • Viewpoint Rosetta Stone teaser figure
    Viewpoint Rosetta Stone: Unlocking Unpaired Ego-Exo Videos for View-invariant Representation Learning Mi Luo, Zihui Xue, Alex Dimakis, Kristen Grauman
    In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025. (Oral)
    [PDF][Project]
    World ModelsEmbodied AICross-View TransferEgocentric Data
  • Ego-Exo4D logo
    Ego-Exo4D: Understanding Skilled Human Activity from First-and Third-Person Perspectives Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, …, Mi Luo, …, Pablo Arbelaez, Gedas Bertasius, David Crandall, Dima Damen, Jakob Engel, Giovanni Maria Farinella, Antonino Furnari, Bernard Ghanem, Judy Hoffman, C. V. Jawahar, Richard Newcombe, Hyun Soo Park, James M. Rehg, Yoichi Sato, Manolis Savva, Jianbo Shi, Mike Zheng Shou, Michael Wray
    In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024. (Oral, EgoVis Distinguished Paper Award)
    [PDF][Project][Blog][Video]
    AR/VREgocentric DataHuman Skill LearningEmbodied AI3D & Multimodal Understanding
  • Seeing the Arrow of Time benchmark examples
    Seeing the Arrow of Time in Large Multimodal Models Zihui Xue, Mi Luo, Kristen Grauman
    In Advances in Neural Information Processing Systems, NeurIPS 2025.
    [PDF][Project]
    Vision Language ModelsCausalityTemporal Reasoning
  • Mental Imagery That Matters teaser figure
    Mental Imagery That Matters: Curing Latent Collapse in Multimodal Reasoning Mi Luo, Rohan Dugad, Zihui Xue, Alex Dimakis, Kristen Grauman
    In Advances in Neural Information Processing Systems, NeurIPS 2026.
    [PDF][Project]
    Multimodal Latent ReasoningSelf-distillationVision Language Models
  • Personal Visual Context Learning teaser figure
    Personal Visual Context Learning in Large Multimodal Models Zihui Xue, Ami Baid, Sangho Kim, Mi Luo, Kristen Grauman
    Arxiv Preprint, 2026.
    [PDF][Project]
    Personalized Multimodal AgentsLong-Term MemoryVisual Context Learning
  • VIOS self-auditing visual reliance teaser figure
    Watching Itself Watch: Self-Auditing Visual Reliance for Video Reasoning Rohan Dugad, Mi Luo, Kristen Grauman
    In Advances in Neural Information Processing Systems, NeurIPS 2026.
    [PDF][Project]
    Video ReasoningSelf-AuditingOn-policy Self-distillationVision Language Models

🎬 Video World Models & Multimodal Generation

Generative Modeling, Training Signals, Evaluation

  • Predictive Structure Improves Video Diffusion Dynamics teaser figure
    Predictive Structure Improves Video Diffusion Dynamics Mi Luo, Yujia Chen, Alex Dimakis, Kristen Grauman, Wen-Sheng Chu, Du Tran
    In 2026 European Conference on Computer Vision, ECCV 2026.
    [PDF][Project]
    World ModelsRL RewardDiffusion Models
  • 4DIFF teaser figure
    4DIFF: 3D-Aware Diffusion Model for Third-to-First Viewpoint Translation Feng Cheng*, Mi Luo*, Huiyu Wang, Alex Dimakis, Lorenzo Torresani, Gedas Bertasius, Kristen Grauman
    In 2024 European Conference on Computer Vision, ECCV 2024.
    [PDF][Project]

    * denotes equal contribution (co-first author).

    3D-Aware Video Generation3D & Multimodal UnderstandingEmbodied AI
  • Put Myself in Your Shoes teaser figure
    Put Myself in Your Shoes: Lifting the Egocentric Perspective from Exocentric Videos Mi Luo, Zihui Xue, Alex Dimakis, Kristen Grauman
    In 2024 European Conference on Computer Vision, ECCV 2024.
    [PDF][Project]
    Cross-View TransferEmbodied AIGenerative ModelsImage Generation
  • HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awareness Zihui Xue, Mi Luo, Changan Chen, Kristen Grauman
    In Advances in Neural Information Processing Systems, NeurIPS 2024.
    [PDF][Project]
    Controllable Video GenerationHuman-Object InteractionDiffusion Models

⚙️ Model Architecture, Optimization & Reliability

Architecture, Optimization, Test-time Calibration, Model Selection

  • MetaFormer / PoolFormer architecture figure
    MetaFormer is Actually What You Need for Vision Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, Shuicheng Yan
    In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022. (Oral)
    [PDF][Project][Marktechpost News][QbitAI News (in Chinese)]
    Vision TransformersArchitecture DesignToken Mixing
  • Classifier Calibration (CCVR) teaser figure
    No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, Jiashi Feng
    In Advances in Neural Information Processing Systems, NeurIPS 2021.
    [PDF][Code]
    Federated LearningNon-IID DataClassifier Calibration
  • Ensemble-based validation (EnsV) teaser figure
    Towards Reliable Model Selection for Unsupervised Domain Adaptation: An Empirical Study and A Certified Baseline Dapeng Hu, Mi Luo, Jian Liang, Chuan Sheng Foo
    In Advances in Neural Information Processing Systems, NeurIPS 2024.
    [PDF][Project]
    Model SelectionDomain Adaptation
  • MetaFormer Baselines results figure
    MetaFormer Baselines for Vision Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, Xinchao Wang
    IEEE Transactions on Pattern Analysis and Machine Intelligence, T-PAMI 2023.
    [PDF][Project][Synced News (in Chinese)]
    Vision TransformersArchitecture DesignToken Mixing
  • Architecture Personalization in Federated Learning teaser figure
    Architecture Personalization in Resource-constrained Federated Learning Mi Luo, Fei Chen, Zhenguo Li, Jiashi Feng
    In NFFL Workshop, NeurIPS 2021. (Selected as outstanding paper, acceptance rate: 9%)
    [PDF]
    Federated LearningNeural Architecture SearchPersonalization
  • MetaSelector teaser figure
    MetaSelector: Meta-Learning for Recommendation with User-Level Adaptive Model Selection Mi Luo, Fei Chen, Pengxiang Cheng, Zhenhua Dong, Xiuqiang He, Jiashi Feng, Zhenguo Li
    In Proceedings of The Web Conference, WWW 2020.
    [PDF]
    Recommender SystemsMeta-LearningModel Selection
  • FedMeta federated meta-learning framework figure
    Federated Meta-Learning with Fast Convergence and Efficient Communication Fei Chen*, Mi Luo*, Zhenhua Dong, Zhenguo Li, Xiuqiang He
    Arxiv Preprint, 2019.
    [PDF][Code]

    * denotes equal contribution (co-first author).

    Federated LearningMeta-LearningCommunication Efficiency

Research Experiences

  • Mountain View, May 2026 - Present

    Google DeepMind
    Student Researcher, Topics: Gemini-Omni, reasoning

  • Mountain View, May 2025 - Dec 2025

    Google
    Student Researcher, Topics: video world models

  • Menlo Park, May 2024 - Dec 2024

    FAIR, Meta AI
    Research Intern, Topics: video encoder and representation learning

  • Singapore, Sep 2021 - Apr 2022

    SEA AI Lab (SAIL)
    Research Intern, Topic: vision transformer architecture

Professional Service

  • Conference Reviewer
    • IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), since 2023
    • European Conference on Computer Vision (ECCV), since 2024
    • IEEE/CVF International Conference on Computer Vision (ICCV), since 2023
    • Asian Conference on Computer Vision (ACCV), since 2024
    • Conference on Neural Information Processing Systems (NeurIPS), since 2023
    • International Conference on Learning Representations (ICLR), since 2024
    • International Conference on Machine Learning (ICML), since 2024
    • International Conference on Artificial Intelligence and Statistics (AISTATS), since 2022
    • AAAI Conference on Artificial Intelligence (AAAI), since 2025

  • Journal Reviewer
    • International Journal of Computer Vision (IJCV)
    • IEEE Transactions on Knowledge and Data Engineering (TKDE)
    • Transactions on Machine Learning Research (TMLR)

  • Teaching Assistant
    • EE2211 Introduction to Machine Learning
    • CG3207 Computer Architecture

Blog

  • November 2026 · 10 min read

    The Flip Test: Video World Models Fail to Learn the World State

    Video models are increasingly used as world models for robotics and planning. We test the assumption underneath: if a small change to a scene flips the real outcome, the prediction should flip too. Across 11 frontier models it mostly does not, while a coding agent given tools to measure the same images solves 93.3% of the pairs.

    World ModelsEvaluationPhysical Reasoning

Personal

  • Outside of research, I'm usually out with a camera 📷 or being supervised by my cats, Kiwi and Wiki 🐈.
Kiwi the cat
Kiwi
Wiki the cat
Wiki

Built with 🧑‍💻 + AI 🤖 with ❤️
Typography adapted from the clarity template.