Jinwei Gu

Principal Research Scientist & Senior Manager · NVIDIA Research
Adjunct Associate Professor · CUHK CSE

Currently a Principal Research Scientist & Senior Manager at NVIDIA Research and Adjunct Associate Professor at CUHK CSE. He is one of the core contributors and tech leads of NVIDIA Cosmos (Best of CES 2025). Previously R&D Executive Director at SenseBrain, co-developing AI imaging sensors with SONY for flagship smartphones; Senior Research Scientist at NVIDIA (2015–2018); and Assistant Professor at RIT (2010–2013). Associate Editor, IEEE TCI (2019–2022) & IEEE TPAMI (2022–). Ph.D. Columbia University; B.S. & M.S. Tsinghua University.

Google Scholar NVIDIA Research CUHK LinkedIn ORCID

Research Focus

🌍 World Models for Physical AI

Building the NVIDIA Cosmos platform end-to-end: neural tokenizers, world model pre-training (video diffusion & autoregressive), Sim2Real transfer, embodied reasoning (LLM-based), robot policy learning, and scalable policy evaluation.

📷 Computational Imaging & Photography

AI-driven camera systems and image/video processing: novel image sensors (RGBW), mobile ISP/SDK pipelines, image restoration & enhancement (denoising, HDR, super-resolution), computational optics, lensless imaging, and polarization.

Cosmos World Model Journey

🏅 Best of CES 2025

NVIDIA Cosmos — World Foundation Model Platform for Physical AI

One of the core contributors and tech leads of the Cosmos platform — working across tokenizers, world models, Sim2Real transfer, embodied reasoning, robot policies, and Physical AI evaluation. Open-source. Deployed for robotics, autonomous driving, and embodied AI research.

Project GitHub Cosmos1 Paper Cosmos2.5 Paper Transfer1 Reason1 Policy

Jan 2025 Cosmos Tokenizer + TokenBench

Jan 2025 Cosmos1 (CES 2025 🏆)

Mar 2025 Cosmos-Transfer1

Mar 2025 Cosmos-Reason1

Oct 2025 Cosmos2.5

Oct 2025 Cosmos-Transfer2.5

Nov 2025 Policy Evaluation

Jan 2026 Cosmos-Policy (ICLR)

Feb 2026 DreamDojo

ongoing Cosmos3 ...

Cosmos World Foundation Model Platform for Physical AI (Cosmos1)

NVIDIA Cosmos Team

arXiv 2501.03575 · Jan 2025 (Best of CES 2025)

Paper Project GitHub

World Simulation with Video Foundation Models for Physical AI (Cosmos-Predict2.5 / Cosmos-Transfer2.5)

NVIDIA Cosmos Team

arXiv 2511.00062 · Oct 2025

Paper Project (Predict2.5) Project (Transfer2.5) Code (Predict2.5) Code (Transfer2.5)

Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning

Moo Jin Kim, Yihuai Gao, Tsung-Yi Lin, Yen-Chen Lin, Yunhao Ge, Percy Liang, Shuran Song, Ming-Yu Liu, Chelsea Finn, Jinwei Gu

ICLR 2026

Paper Project Code

Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control

NVIDIA Cosmos Team

arXiv 2503.14492 · Mar 2025

Paper Project Code

Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning

NVIDIA Cosmos Team

arXiv 2503.15558 · Mar 2025

Paper Project Code

Cosmos Tokenizer: A Suite of Image and Video Neural Tokenizers

NVIDIA Cosmos Team

arXiv 2501.03575 · Jan 2025

Paper Project GitHub TokenBench

DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos

Shenyuan Gao, William Liang, Kaiyuan Zheng, ..., Jinwei Gu, Jitendra Malik, Pieter Abbeel, Ming-Yu Liu, Yuke Zhu, Linxi Fan (30 authors)

arXiv 2602.06949 · Feb 2026

Paper Project Code

Scalable Policy Evaluation with Video World Models

Wei-Cheng Tseng, Jinwei Gu, Qinsheng Zhang, Hanzi Mao, Ming-Yu Liu, Florian Shkurti, Lin Yen-Chen

arXiv 2511.11520 · Nov 2025

Paper

Plenoptic Video Generation

Xiao Fu, Shitao Tang, Min Shi, Xian Liu, Jinwei Gu, Ming-Yu Liu, Dahua Lin, Chen-Hsuan Lin

CVPR 2026

Paper Project

Computational Imaging & Photography

📱

AI Imaging Sensors & Mobile Camera Systems

SenseBrain / SONY · 2018–2023

Built & led a 30-person R&D team to co-develop with SONY the world's first AI sensor (RGBW IMX866) and 200MP camera (IMX777), shipping in flagship smartphones. Delivered full AI ISP, Super Resolution, Super Night, Portrait Restoration, RGBCMY sensor, Under-Display Camera, and HDR video pipelines.

MIPI RGBW Challenge

🚗

NVIDIA DRIVE IX — AI Co-Pilot SDK

NVIDIA Research · 2017–2018

Core contributor to the AI Co-Pilot SDK: head pose, gaze estimation, facial keypoint tracking, drowsiness/distraction detection, gesture recognition. Demoed in NVIDIA CEO keynote at CES 2017; shipped in NVIDIA DRIVE IX SDK.

NVIDIA Blog TechCrunch

🎥

VirtualEye — Multi-Camera Free-View System

NVIDIA + DARPA · 2015–2017

Key member for this NVIDIA–DARPA project: real-time multi-camera Co-SLAM, 3D reconstruction, novel view synthesis, and free-view video streaming for telepresence. Led Co-SLAM tracking; ported full pipeline on NVIDIA Jetson TX2.

Engadget

2025

PolarFree: Polarization-based Reflection-free Imaging

Mingde Yao, Menglu Wang, King Man Tam, Lingen Li, Tianfan Xue, Jinwei Gu

CVPR 2025

Paper Project Code

A Physics-Informed Blur Learning Framework for Imaging Systems

Liqun Chen, Yuxuan Li, Jun Dai, Jinwei Gu, Tianfan Xue

CVPR 2025

Paper Code

4DSloMo: 4D Reconstruction for High-Speed Scenes with Asynchronous Capture

Yutian Chen, Shi Guo, Tianshuo Yang, Lihe Ding, Xiuyuan Yu, Jinwei Gu, Tianfan Xue

SIGGRAPH Asia 2025

Paper Project Code

Uni-ISP: Toward Unifying the Learning of ISPs from Multiple Mobile Cameras

Lingen Li, Mingde Yao, Xingyu Meng, Muquan Yu, Tianfan Xue, Jinwei Gu

IEEE Transactions on Image Processing, 2025

Paper Project

Tolerance-Aware Deep Optics

Jun Dai, Liqun Chen, Xinge Yang, Yuyao Hu, Jinwei Gu, Tianfan Xue

arXiv, 2025

Paper Project

2024

DualDn: Dual-domain Denoising via Differentiable ISP

Ruikang Li, Yujin Wang, Shiqi Chen, Fan Zhang, Jinwei Gu, Tianfan Xue

ECCV 2024

Paper Project Code

PhoCoLens: Photorealistic and Consistent Reconstruction in Lensless Imaging

Xin Cai, Zhiyuan You, Hailong Zhang, Jinwei Gu, Wentao Liu, Tianfan Xue

NeurIPS 2024 (Spotlight)

Paper Project Code

AdaptiveISP: Learning an Adaptive Image Signal Processor for Object Detection

Yujin Wang, Tianyi Xu, Zhang Fan, Tianfan Xue, Jinwei Gu

NeurIPS 2024

Paper Project Code

HDRFlow: Real-time HDR Video Reconstruction with Large Motion

Gangwei Xu, Yujin Wang, Jinwei Gu, Tianfan Xue, Xin Yang

CVPR 2024

Paper Project Code

Matting by Generation

Zhixiang Wang, Baiang Li, Jian Wang, Yu-Lun Liu, Jinwei Gu, Yung-Yu Chuang, Shinichi Satoh

SIGGRAPH 2024

Paper Project Code

AutoDIR: Automatic All-in-one Image Restoration with Latent Diffusion

Yitong Jiang, Zhaoyang Zhang, Tianfan Xue, Jinwei Gu

ECCV 2024

Paper Project Code

2023

Learning Image-Adaptive Codebooks for Class-Agnostic Image Restoration

Kechun Liu, Yitong Jiang, Inchang Choi, Jinwei Gu

ICCV 2023

Paper Project Code

Real-time Controllable Denoising for Image and Video

Zhaoyang Zhang, Yitong Jiang, Wenqi Shao, Xiaogang Wang, Ping Luo, Kaimo Lin, Jinwei Gu

CVPR 2023

Paper Code

Generating Aligned Pseudo-Supervision from Non-Aligned Data for Image Restoration in Under-Display Camera

Ruicheng Feng, Chongyi Li, Huaijin Chen, Shuai Li, Jinwei Gu, Chen Change Loy

CVPR 2023

Project Code

2022

Deep Fourier Up-Sampling

Man Zhou, Yu Hu, Jie Huang, Feng Zhao, Jinwei Gu, Chen Change Loy, Deyu Meng, Chongyi Li

NeurIPS 2022

Project Code

Deep Camera Obscura: An Image Restoration Pipeline for Lensless Pinhole Photography

Joshua Rego, Huaijin Chen, Shuai Li, Jinwei Gu, Suren Jayasuriya

Optics Express, 2022

Paper

Earlier Highlights (Imaging)

GLEAN: Generative Latent Bank for Image Super-Resolution and Beyond

Kelvin C.K. Chan, Xintao Wang, Xiangyu Xu, Jinwei Gu, Chen Change Loy

CVPR 2021 (Oral) · IEEE TPAMI 2023

Project Code

Low-light Image and Video Enhancement Using Deep Learning: A Survey

Chongyi Li, Chunle Guo, Linghao Han, Jun Jiang, Ming-Ming Cheng, Jinwei Gu, Chen Change Loy

IEEE TPAMI 2022

Paper Code

Discriminative Illumination: Per-Pixel Classification of Raw Materials based on Optimal Projections of Spectral BRDFs

Jinwei Gu, Chao Liu

CVPR 2012 (Oral) · IEEE TPAMI 2014

Paper

Other Selected Publications

2026

CubeComposer: Spatio-Temporal Autoregressive 4K 360° Video Generation from Perspective Video

Lingen Li, Guangzhi Wang, Xiaoyu Li, Tianfan Xue, Ying Shan, Jinwei Gu

CVPR 2026

Paper Project Code

ToonComposer: Streamlining Cartoon Production with Generative Post-Keyframing

Lingen Li, Guangzhi Wang, Zhaoyang Zhang, Yaowei Li, Xiaoyu Li, Qi Dou, Jinwei Gu, Tianfan Xue, Ying Shan

ICLR 2026

Paper Project Code

2025

ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediary

Zeqi Gu, Yin Cui, Max Li, Fangyin Wei, Yunhao Ge, Jinwei Gu, Ming-Yu Liu, Abe Davis, Yifan Ding

CVPR 2025

Paper Project Code

NVComposer: Boosting Generative Novel View Synthesis with Multiple Sparse and Unposed Images

Lingen Li, Zhaoyang Zhang, Yaowei Li, ..., Jinwei Gu, Tianfan Xue, Ying Shan

CVPR 2025

Paper Project Code

Parallel Sequence Modeling via Generalized Spatial Propagation Network

Hongjun Wang, Wonmin Byeon, Jiarui Xu, Jinwei Gu, Ka Chun Cheung, Xiaolong Wang, Kai Han, Jan Kautz, Sifei Liu

CVPR 2025

Paper Project Code

GSPN-2: Efficient Parallel Sequence Modeling

Hongjun Wang, Yitong Jiang, Collin McCarthy, David Wehr, Hanrong Ye, Xinhao Li, Ka Chun Cheung, Wonmin Byeon, Jinwei Gu, Ke Chen, Kai Han, Hongxu Yin, Pavlo Molchanov, Jan Kautz, Sifei Liu

NeurIPS 2025

Paper Project Code

Earlier Highlights (3D Vision & Rendering)

Neural RGB→D Sensing: Depth and Uncertainty from a Video Camera

Chao Liu, Jinwei Gu, Kihwan Kim, Srinivas Narasimhan, Jan Kautz

CVPR 2019 (Oral · Best Paper Finalist)

Paper Project Code

PlaneRCNN: 3D Plane Detection and Reconstruction from a Single Image

Chen Liu, Kihwan Kim, Jinwei Gu, Yasutaka Furukawa, Jan Kautz

CVPR 2019 (Oral)

Paper Project Code

MapNet: Geometry-Aware Learning of Maps for Camera Localization

Samarth Brahmbhatt, Jinwei Gu, Kihwan Kim, James Hayes, Jan Kautz

CVPR 2018 (Spotlight)

Paper Project Code

Honors & Awards

Best of CES Award (2025) — NVIDIA Cosmos World Foundation Models
CVPR 2019 Best Paper Finalist — Neural RGB→D Sensing
CVPR 2018 Outstanding Reviewer
Ph.D. Degree with Distinction, Columbia University, 2010
Ph.D. Service Award, Columbia University, 2009
Principal Scholarship Award, Tsinghua University, 2005
Outstanding M.S. Thesis Award, Tsinghua University, 2005

Professional Service

Editorial

Associate Editor, IEEE TPAMI (2022–)
Associate Editor, IEEE Trans. Computational Imaging (2019–2022)
IEEE Senior Member (2018–)

Area Chair / Organizer

Area Chair: CVPR, ICCV, ECCV, NeurIPS
Industry Chair: ICCP 2020, 2023
Organizing Chair: MIPI Workshop, RichMediaGAI Workshop