Jinwei Gu

Principal Research Scientist & Senior Manager · NVIDIA Cosmos Lab
Adjunct Associate Professor · CUHK CSE

Currently a Principal Research Scientist & Senior Manager at NVIDIA Cosmos Lab and Adjunct Associate Professor at CUHK CSE. He co-leads research in world models for Physical AI, including NVIDIA Cosmos and Cosmos 3, and computational imaging/camera systems. Previously R&D Executive Director at SenseBrain, co-developing AI imaging sensors with SONY for flagship smartphones; Senior Research Scientist at NVIDIA (2015–2018); and Assistant Professor at RIT (2010–2013). Associate Editor, IEEE TCI (2019–2022) & IEEE TPAMI (2022–). Ph.D. Columbia University; B.S. & M.S. Tsinghua University.

Google Scholar Cosmos Lab CUHK LinkedIn ORCID

Research Focus

World Models for Physical AI

Building the NVIDIA Cosmos platform end-to-end: neural tokenizers, omnimodal world model pre-training, video diffusion & autoregressive generation, Sim2Real transfer, embodied reasoning, robot policy learning, scalable policy evaluation, and synthetic data generation.

Computational Imaging & Photography

AI-driven camera systems and image/video processing: novel image sensors (RGBW), mobile ISP/SDK pipelines, image restoration & enhancement (denoising, HDR, super-resolution), computational optics, lensless imaging, and polarization.

World Models & Physical AI

NVIDIA Cosmos and Omnimodal World Models

Research and platform work across neural tokenizers, video/world foundation models, Sim2Real transfer, embodied reasoning, robot policies, evaluation, and synthetic data generation. Recent Cosmos releases connect understanding, simulation, generation, and action for robotics, autonomous driving, and embodied AI.

Cosmos 3 Paper Models/Data Code Tokenizer Cosmos-Predict1 Reason1 Predict2 Predict2.5 Transfer2.5

Jun 2026 Cosmos 3

Feb 2026 DreamDojo

Jan 2026 Cosmos-Policy

Jan 2026 Plenoptic Video

Nov 2025 Policy Evaluation

Nov 2025 Cosmos 2.5

Mar 2025 Cosmos-Reason1

Mar 2025 Cosmos-Transfer1

Jan 2025 Cosmos1 + Tokenizer

Selected Projects, Latest First

Cosmos 3: Omnimodal World Models for Physical AI

NVIDIA Cosmos Team

arXiv 2606.02800 · Jun 2026 (top open-model results across many Physical AI benchmarks)

Paper Project Code Models/Data Product

DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos

Shenyuan Gao, William Liang, Kaiyuan Zheng, ..., Jinwei Gu, Jitendra Malik, Pieter Abbeel, Ming-Yu Liu, Yuke Zhu, Linxi Fan (30 authors)

arXiv 2602.06949 · Feb 2026

Paper Project Code

Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning

Moo Jin Kim, Yihuai Gao, Tsung-Yi Lin, Yen-Chen Lin, Yunhao Ge, Percy Liang, Shuran Song, Ming-Yu Liu, Chelsea Finn, Jinwei Gu

ICLR 2026 · arXiv 2601.16163

Paper Project Code

Plenoptic Video Generation

Xiao Fu, Shitao Tang, Min Shi, Xian Liu, Jinwei Gu, Ming-Yu Liu, Dahua Lin, Chen-Hsuan Lin

CVPR 2026 · arXiv 2601.05239

Paper Project

Scalable Policy Evaluation with Video World Models

Wei-Cheng Tseng, Jinwei Gu, Qinsheng Zhang, Hanzi Mao, Ming-Yu Liu, Florian Shkurti, Lin Yen-Chen

arXiv 2511.11520 · Nov 2025

Paper

World Simulation with Video Foundation Models for Physical AI (Cosmos-Predict2.5 / Cosmos-Transfer2.5)

NVIDIA Cosmos Team

arXiv 2511.00062 · Nov 2025

Paper Project (Predict2.5) Project (Transfer2.5) Code (Predict2.5) Code (Transfer2.5)

Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning

NVIDIA Cosmos Team

arXiv 2503.15558 · Mar 2025

Paper Project Code

Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control

NVIDIA Cosmos Team

arXiv 2503.14492 · Mar 2025

Paper Project Code

Cosmos World Foundation Model Platform for Physical AI (Cosmos1)

NVIDIA Cosmos Team

arXiv 2501.03575 · Jan 2025

Paper Project Code

Cosmos Tokenizer: A Suite of Image and Video Neural Tokenizers

NVIDIA Cosmos Team

arXiv 2501.03575 · Jan 2025

Paper Project Code TokenBench

Computational Imaging & Photography

Camera Systems, Computational Photography, and Image Formation

Work spanning AI image sensors, mobile camera pipelines, differentiable ISPs, restoration and enhancement, computational optics, lensless imaging, polarization, event cameras, and high-speed capture.

RGBW Challenge DualDn PolarFree PhoCoLens

Systems & Products

📱

AI Imaging Sensors & Mobile Camera Systems

SenseBrain / SONY · 2018–2023

Built & led a 30-person R&D team to co-develop with SONY the world's first AI sensor (RGBW IMX866) and 200MP camera (IMX777), shipping in flagship smartphones. Delivered full AI ISP, Super Resolution, Super Night, Portrait Restoration, RGBCMY sensor, Under-Display Camera, and HDR video pipelines.

MIPI RGBW Challenge

🚗

NVIDIA DRIVE IX — AI Co-Pilot SDK

NVIDIA Research · 2017–2018

Core contributor to the AI Co-Pilot SDK: head pose, gaze estimation, facial keypoint tracking, drowsiness/distraction detection, gesture recognition. Demoed in NVIDIA CEO keynote at CES 2017; shipped in NVIDIA DRIVE IX SDK.

NVIDIA Blog TechCrunch

🎥

VirtualEye — Multi-Camera Free-View System

NVIDIA + DARPA · 2015–2017

Key member for this NVIDIA–DARPA project: real-time multi-camera Co-SLAM, 3D reconstruction, novel view synthesis, and free-view video streaming for telepresence. Led Co-SLAM tracking; ported full pipeline on NVIDIA Jetson TX2.

Engadget

Selected Projects, Latest First

2025

4DSloMo: 4D Reconstruction for High-Speed Scenes with Asynchronous Capture

Yutian Chen, Shi Guo, Tianshuo Yang, Lihe Ding, Xiuyuan Yu, Jinwei Gu, Tianfan Xue

SIGGRAPH Asia 2025 · arXiv 2507.05163

Paper Project Code

PolarFree: Polarization-based Reflection-free Imaging

Mingde Yao, Menglu Wang, King Man Tam, Lingen Li, Tianfan Xue, Jinwei Gu

CVPR 2025 · arXiv 2503.18055

Paper Project Code

A Physics-Informed Blur Learning Framework for Imaging Systems

Liqun Chen, Yuxuan Li, Jun Dai, Jinwei Gu, Tianfan Xue

CVPR 2025 · arXiv 2502.11382

Paper Code

Tolerance-Aware Deep Optics

Jun Dai, Liqun Chen, Xinge Yang, Yuyao Hu, Jinwei Gu, Tianfan Xue

arXiv 2502.04719 · Feb 2025

Paper Project

Uni-ISP: Toward Unifying the Learning of ISPs from Multiple Mobile Cameras

Lingen Li, Mingde Yao, Xingyu Meng, Muquan Yu, Tianfan Xue, Jinwei Gu

IEEE Transactions on Image Processing, 2025

Paper Project

2024

AdaptiveISP: Learning an Adaptive Image Signal Processor for Object Detection

Yujin Wang, Tianyi Xu, Zhang Fan, Tianfan Xue, Jinwei Gu

NeurIPS 2024 · arXiv 2410.22939

Paper Project Code

DualDn: Dual-domain Denoising via Differentiable ISP

Ruikang Li, Yujin Wang, Shiqi Chen, Fan Zhang, Jinwei Gu, Tianfan Xue

ECCV 2024 · arXiv 2409.18783

Paper Project Code

PhoCoLens: Photorealistic and Consistent Reconstruction in Lensless Imaging

Xin Cai, Zhiyuan You, Hailong Zhang, Jinwei Gu, Wentao Liu, Tianfan Xue

NeurIPS 2024 (Spotlight) · arXiv 2409.17996

Paper Project Code

Matting by Generation

Zhixiang Wang, Baiang Li, Jian Wang, Yu-Lun Liu, Jinwei Gu, Yung-Yu Chuang, Shinichi Satoh

SIGGRAPH 2024 · arXiv 2407.21017

Paper Project Code

HDRFlow: Real-time HDR Video Reconstruction with Large Motion

Gangwei Xu, Yujin Wang, Jinwei Gu, Tianfan Xue, Xin Yang

CVPR 2024 · arXiv 2403.03447

Paper Project Code

AutoDIR: Automatic All-in-one Image Restoration with Latent Diffusion

Yitong Jiang, Zhaoyang Zhang, Tianfan Xue, Jinwei Gu

ECCV 2024 · arXiv 2310.10123

Paper Project Code

2023

Learning Image-Adaptive Codebooks for Class-Agnostic Image Restoration

Kechun Liu, Yitong Jiang, Inchang Choi, Jinwei Gu

ICCV 2023 · arXiv 2306.06513

Paper Project Code

Real-time Controllable Denoising for Image and Video

Zhaoyang Zhang, Yitong Jiang, Wenqi Shao, Xiaogang Wang, Ping Luo, Kaimo Lin, Jinwei Gu

CVPR 2023 · arXiv 2303.16425

Paper Code

Generating Aligned Pseudo-Supervision from Non-Aligned Data for Image Restoration in Under-Display Camera

Ruicheng Feng, Chongyi Li, Huaijin Chen, Shuai Li, Jinwei Gu, Chen Change Loy

CVPR 2023

Project Code

2022

Deep Fourier Up-Sampling

Man Zhou, Yu Hu, Jie Huang, Feng Zhao, Jinwei Gu, Chen Change Loy, Deyu Meng, Chongyi Li

NeurIPS 2022

Project Code

Deep Camera Obscura: An Image Restoration Pipeline for Lensless Pinhole Photography

Joshua Rego, Huaijin Chen, Shuai Li, Jinwei Gu, Suren Jayasuriya

Optics Express, 2022

Paper

Earlier Highlights (Imaging)

GLEAN: Generative Latent Bank for Image Super-Resolution and Beyond

Kelvin C.K. Chan, Xintao Wang, Xiangyu Xu, Jinwei Gu, Chen Change Loy

CVPR 2021 (Oral) · IEEE TPAMI 2023

Project Code

Low-light Image and Video Enhancement Using Deep Learning: A Survey

Chongyi Li, Chunle Guo, Linghao Han, Jun Jiang, Ming-Ming Cheng, Jinwei Gu, Chen Change Loy

IEEE TPAMI 2022

Paper Code

Discriminative Illumination: Per-Pixel Classification of Raw Materials based on Optimal Projections of Spectral BRDFs

Jinwei Gu, Chao Liu

CVPR 2012 (Oral) · IEEE TPAMI 2014

Paper

Other Selected Work

2026

CubeComposer: Spatio-Temporal Autoregressive 4K 360° Video Generation from Perspective Video

Lingen Li, Guangzhi Wang, Xiaoyu Li, Tianfan Xue, Ying Shan, Jinwei Gu

CVPR 2026 · arXiv 2603.04291

Paper Project Code

ToonComposer: Streamlining Cartoon Production with Generative Post-Keyframing

Lingen Li, Guangzhi Wang, Zhaoyang Zhang, Yaowei Li, Xiaoyu Li, Qi Dou, Jinwei Gu, Tianfan Xue, Ying Shan

ICLR 2026 · arXiv 2508.10881

Paper Project Code

2025

GSPN-2: Efficient Parallel Sequence Modeling

Hongjun Wang, Yitong Jiang, Collin McCarthy, David Wehr, Hanrong Ye, Xinhao Li, Ka Chun Cheung, Wonmin Byeon, Jinwei Gu, Ke Chen, Kai Han, Hongxu Yin, Pavlo Molchanov, Jan Kautz, Sifei Liu

NeurIPS 2025 · arXiv 2512.07884

Paper Project Code