Jinwei Gu

	Jinwei Gu Principal Research Scientist NVIDIA [Google Scholar]
I am currently a principal research scientist at NVIDIA, working on deep generative models, vision foundation models, world models, and the general fields of computer vision, computer graphics, and machine learning. I am also an adjunct associate professor at the Chinese University of Hong Kong. I received my Ph.D. from Columbia University, and B.S. and M.S. from Tsinghua University.
Recent Research (to be updated ...)
	Learning Image-Adaptive Codebooks for Class-Agnostic Image Restoration [Paper] Kechun, Yitong, Inchang, Jinwei ICCV 2023
	Real-time Controllable Denoising for Image and Video [Paper] Zhaoyang, Yitong, Wenqi, Xiaogang, Ping, Kaimo, Jinwei. CVPR 2023 (to appear)
	Generating Aligned Pseudo-Supervision from Non-Aligned Data for Image Restoration in Under-Display Camera[Paper] Ruicheng, Chongyi, Huaijin, Shuai, Jinwei, and Cavan. CVPR 2023 (to appear)
	Deep Fouier Up-Sampling[Paper] Man, Yu, Jie, Feng, Jinwei, Cavan, Deyu, Chongyi. NeurIPS 2022
	Deep Camera Obscura: An Image Restoration Pipeline for Lenless Pinhole Photography[Paper] Joshua, Huaijin, Shuai, Jinwei, and Suren. Optics Express, 2022
	Low-light Image and Video Enhancement Using Deep Learning: A Survey[Paper][Data+Code] Chongyi, Chunle, Linghao, Jun, Ming-Ming, Jinwei and Chen Change. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021. (to appear)
	iToF2dToF: A Robust and Flexible Representation for Data-driven Time-of-Flight Imaging[Project(Paper+Data+Code)] Felipe, Huaijin, Mohit, Andreas, and Jinwei. IEEE Transactions on Computational Imaging, 2021. (to appear)
	ReconfigISP: Reconfigurable Camera Image Processing Pipeline[Project (Paper+Code)] Ke, Zexian, Yue, Cavan, and Jinwei. ICCV, 2021.
	STAR: A Structure-Aware Lightweight Transformer for Real-time Image Enhancement [Paper] Zhaoyang, Yitong, Jun, Xiaogang, Ping, and Jinwei ICCV, 2021.
	Dynamic Quantization with Mixed Precision and Adaptive Resolution [Paper] Zhaoyang, Wenqi, Jinwei, Xiaogang, and Ping ICML, 2021. (spotlight)
	Removing Diffraction Image Artifacts in Under-Display Camera via Dynamic Skip Connection Network [Project/Code] Ruicheng, Chongyi, Huaijin, Shuai, Cavan, and Jinwei CVPR, 2021.
	GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution [Project/Code] Kelvin, Xintao, Xiangyu, Jinwei, and Cavan CVPR, 2021. (Oral)
	Video Stitching for Linear Camera Arrays [Project] Wei-Sheng, Orazio, Jinwei, Deqing, Ming-hsuan, and Jan. BMVC, 2019. (to appear)
	NRMVS: Non-Rigid Multi-View Stereo [Arxiv][PDF][Supplementary Video][Project (coming soon)] Matthias, Kihwan, Jinwei, Matthias, Charles, Marc, and Jan. Arxiv, 2019.
	Neural RGB->D Sensing: Depth and Uncertainty from a Video Camera [Arxiv][Code] Chao, Jinwei, Kihwan, Srinivas, and Jan. CVPR 2019 (Oral, Best Paper Finalist).
	PlaneRCNN: 3D Plane Detection and Reconstruction from a Single Image[Arxiv] [Code] Chen, Kihwan, Jinwei, Yasutaka, and Jan. CVPR 2019 (Oral).
	Neural Inverse Rendering of an Indoor Scene from a Single Image[Arxiv][Project and Dataset] Soumyadip, Jinwei, Kihwan, Guilin, David, and Jan. Arxiv 2019. ICCV 2019 (to appear)
	Very Power Efficient Neural Time-of-Flight[Arxiv][PDF][Supplementary Video][Project (coming soon)] Yan, Jimmy, Xuanye, Keyuan, and Jinwei. Arxiv 2019.
	Context-Aware Synthesis and Placement of Object Instances[Arxiv][PDF][Supplementary Video][Code] Donghoon, Sifei, Jinwei, Ming-Yu, Ming-Hsuan, and Jan. NIPS 2018.
	Switchable Temporal Propagation Network [Arxiv][PDF][Supplementary Video][Applications] Sifei, Jinwei, Shalini, Guangyu, Varun, Ming-Hsuan, and Jan. ECCV 2018.
	Separating Reflection and Transmission Images in the Wild [Arxiv][PDF][Supplementary Video][Code] Patrick, Orazio, Jinwei, and Jan. ECCV 2018.
	MapNet: Geometry-Aware Learning of Maps for Camera Localization[PDF][Code][Project] Samarth, Jinwei, Kihwan, James, and Jan. CVPR 2018 (spotlight).
	Space-Time-Brightness Sampling Using an Adaptive Pixel-Wise Coded Exposure [PDF] Hajime, Toshiki, Dengyu, and Jinwei. CVPR 2018, Computational Camera and Display Workshop.
	Reblur2Deblur: Deblurring Videos via Self-Supervised Learning[PDF][Project] Huaijin, Jinwei, Orazio, Ming-Yu, Ashok, and Jan. ICCP 2018.
	Learning Affinity via Spatial Propagation Networks [PDF][Supplementary Video] Sifei, Shalini, Jinwei, Guangyu, Ming-Hsuan,and Jan. NIPS 2017 (Accepted).
	A Lightweight Approach for On-the-Fly Reflectance Estimation[PDF][Supplementary Video] Kihwan, Jinwei, Stephen, Pavlo, Matthias, and Jan. ICCV 2017 Oral. arXiv:1705.07162
	Dynamic Facial Analysis: From Bayesian Filtering to Recurrent Neural Networks [Project][PDF][Supplementary Video] Jinwei, Xiaodong, Shalini and Jan. CVPR 2017
	Polarimetric Multi-view Stereo [PDF][Supplementary Document][Video] Zhaopeng, Jinwei, Boxin, Ping and Jan. CVPR 2017
	Reconstructing Intensity Images from Binary Spatial Gradient Cameras [PDF][Supplementary Document][Video] Suren, Orazio, Jinwei, Timo and Jan. CVPR EVW 2017 (Oral)
	Deep Learning with Energy-efficient Binary Gradient Cameras [PDF] Suren, Orazio, and Jinwei. arXiv:1612.00986, 2016
	Video Stitching with Spatial-Temporal Content-Preserving Warping[PDF][Video Results] Wei and Jinwei. CVPR 2015 Workshop on Computational Cameras and Displays top

Previous Research
	Exemplar-Based Method for Automatic Visual Editing and Retouching of Fine Art Reproduction [project] Jun and Jinwei. CIC 2013
	Learning Discriminative Illumination and Filters for BTF Classification[project] [database](with code) Chao, Jinwei, and Gefei. CVPR 2013
	What is the Space of Camera Spectral Sensitivities?[project] [database](with code) Jun, Dengyu, Jinwei, and Sabine. IEEE WACV 2013
	Classifying Raw Materials with Discriminative Illumination [project] [database] Chao and Jinwei. CVPR 2012 (Oral)
	Spectral Imaging Under Commonly Available Lighting [project] Jun and Jinwei. CVPR 2012 Workshop on Computational Camera and Displays
	Multiplexed Illumination for Scene Recovery in the Presence of Global Illumination[project] Toshi, Jinwei, Mohit, and Shree. ICCV 2011 (Oral)
	Video from a Single Exposure Coded Photograph using a Learned Over-Complete Dictionary[project] Yas, Jinwei, Mohit, Tomoo, and Shree. ICCV 2011
	Coded Rolling Shutter Photography: Flexible Space-Time Sampling [project] Yas, Jinwei, Tomoo, and Shree. ICCP 2010
	Measurement, Modeling, and Synthesis of Time-Varying Appearance of Natural Phenomena [PDF] Jinwei Gu. PhD Thesis. Columbia University. March 2010
	Removing Image Artifacts Due to Dirty Camera Lenses and Thin Occluders[project] Jinwei, Ravi, Peter, and Shree. SIGGRAPH Asia 2009
	Printing Spatially-Varying Reflectance [project] Jinwei, Wojciech Matusik, Boris Ajdin, Jason Lawrence, Hendrik P. A. Lensch, Fabio Pellacini, and Szymon Rusinkiewicz. SIGGRAPH Asia 2009
	Compressive Structured Light for Recovering Inhomogenous Participating Media [project](with code) Jinwei, Ravi, Peter, Shree, and Eitan. ECCV 2008 (Oral)
	Dirty Glass: Modeling and Rendering Contamination on Transparent Surfaces [project] [database] (with code) Jinwei, Ravi, Peter, and Shree. EGSR 2007
	Time-Varying Surface Appearance [project][database] (with code) Jinwei, Chien-i Tu, Ravi, Peter, Wojciech, and Shree. SIGGRAPH 2006
	Radiometric Calibration from a Single Image [project] Jinwei, Stephen Lin, Shuntaro Yamazaki, and Harry Shum, CVPR 2004 (Oral)
	Fingerprint Orientation Field Modeling and Its Applications [project] with Jie Zhou, Chunyu Yang. CVPR 2003 top
Selected Publications (to be updated ...) Computer Vision and Machine Learning Kihwan Kim, Jinwei Gu, Stephen Tyree, Pavlo Molchanov, Matthias Niebner, and Jan Kautz. "A Lightweight Approach for On-the-Fly Reflectance Estimation", arXiv 1705.07162, 2017 Jinwei Gu, Xiaodong Yang, Shalini Gupta and Jan Kautz. "Dynamic Facial Analysis: From Bayesian Filterings to Recurrent Neural Networks". CVPR, 2017. Zhaopeng Cui, Jinwei Gu, Boxin Shi, Ping Tan and Jan Kautz. "Polarimetric Multi-view Stereo". CVPR, 2017. Suren Jayasuriya, Orazio Gallo, Jinwei Gu, Timo Aila, and Jan Kautz. "Reconstructing Intensity Images from Binary Spatial Gradient Cameras", CVPR Embedded Vision Workshop (Oral), 2017 Suren Jayasuriya, Orazio Gallo, Jinwei Gu and Jan Kautz. "Deep Learning with Energy-efficient Binary Gradient Cameras", arXiv:1612.00986, 2016 Computational Photography Wei Jiang and Jinwei Gu. "Video Stitching with Spatial-Temporal Content-Preserving Warping. IEEE Conferences on Computer Vision and Pattern Recognition (CVPR) Workshop on Computational Cameras and Displays, 2015 Jun Jiang, Dengyu Liu, Jinwei Gu and Sabine Susstrunk. "What is the Space of Spectral Sensitivity Functions for Digital Color Cameras?. IEEE Workshop on the Applications of Computer Vision (WACV), 2013. Jinwei Gu and Chao Liu. "Discriminative Illumination: Per-Pixel Classification of Raw Materials based on Optimal Projections of BRDFs", IEEE Conferences on Computer Vision and Pattern Recognition (CVPR), 2012. Jun Jiang and Jinwei Gu. "Recovering Spectral Reflectance under Commonly Available Lighting Conditions", IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop on Computational Camera and Displays, 2012. Jinwei Gu, Toshihiro Kobayashi, Mohit Gupta, and Shree Nayar. "Multiplexed Illumination for Scene Recovery in the Presence of Global Illumination" IEEE International Conference on Computer Vision (ICCV), 2011. Yasunobu Hitomi, Jinwei Gu, Mohit Gupta, Tomoo Mitsunaga, and Shree Nayar. "Video From a Single Exposure Coded Photograph using a Learned Over-Complete Dictionary". IEEE International Conference on Computer Vision (ICCV), 2011. Jinwei Gu, Yasunobu Hitomi, Tomoo Mitsunaga, and Shree Nayar. "Coded Rolling Shutter Photography: Flexible Space-Time Sampling". IEEE International Conference on Computational Photography (ICCP), Mar. 2010. Jinwei Gu, Ravi Ramamoorthi, Peter Belhumeur and Shree Nayar. "Removing Image Artifacts Due to Dirty Camera Lenses and Thin Occluders". ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia), Dec. 2009. Computer Graphics and Physics-based Computer Vision Adria Fores, James A. Ferwerda, and Jinwei Gu. "Toward a Perceptually Based Metric for BRDF Modeling", Proceedings IS\&T/SID 20th Color Imaging Conference (CIC), November 2012. Wojciech Matusik, Boris Ajdin, Jinwei Gu, Jason Lawrence, Hendrik P.A. Lensch, Fabio Pellacini, and Szymon Rusinkiewicz. "Printing Spatially-Varying Reflectance". ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia), Dec. 2009. Jinwei Gu, Shree Nayar, Eitan Grinspun, Peter Belhumeur and Ravi Ramamoorthi. "Compressive Structured Light for Recovering Inhomogenous Participating Media". Proceedings of European Conference on Computer Vision (ECCV) , Oct. 2008, pp 845-858. Jinwei Gu, Ravi Ramamoorthi, Peter Belhumeur, and Shree Nayar. " Dirty Glass: Rendering Contamination on Transparent Surfaces". Proceedings of Eurographics Symposium on Rendering (EGSR), June 2007. Jinwei Gu, Chien-I Tu, Ravi Ramamoorthi, Peter Belhumeur, Wojciech Matusik, and Shree Nayar. "Time-Varying Surface Appearance: Acquisition, Modeling, and Rendering". ACM Transactions on Graphics (Proceedings of SIGGRAPH), vol. 25, no. 3, June 2006, pp 762-771. Stephen Lin, Jinwei Gu, Shuntaro Yamazaki, and Heung-Yeung Shum. "Radiometric Calibration from a Single Image". Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), June 2004, vol. 2, pp 938-945. Image Processing and Biometrics Jie Zhou, Fanglin Chen, and Jinwei Gu. "A Novel Algorithm for Detecting Singular Points from Fingerprint Images". IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), vol 31, no.7, July 2009, pp 1239-1250. Jie Zhou, Jinwei Gu and David Zhang, "Singular Points Analysis in Fingerprints Based on Topological Structure and Orientation Field". International Conference of Biometrics (ICB), August 2007, pp 261-270. Jinwei Gu, Jie Zhou, and Chunyu Yang. "Fingerprint Recognition by Combining Global Structure and Local Cues". IEEE Transactions on Image Processing, vol. 15, no. 7, July 2006, pp 1952-1964. Jie Zhou and Jinwei Gu, "A Model-based Method for the Computation of Fingerprints' Orientation Field". IEEE Transactions on Image Processing, vol. 13, no. 6, June 2004, pp 821-835. Jie Zhou, David Zhang, Jinwei Gu, and Nannan Wu. "Chapter 14. Graphical Representation of Fingerprint Images". Integrated Image and Graphics Technologies, Kluwer academic publisher, Boston, 2004. Jie Zhou and Jinwei Gu. "Modeling Orientation Fields of Fingerprints with Rational Complex Functions". Pattern Recognition, vol. 37, no. 2, 2004, pp 389-391. Jinwei Gu and Jie Zhou, "A Novel Model for Orientation Field of Fingeprints". Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), June 2003, vol 2, pp 493-498. Other Areas Wei Jiang, Guihua Er, Qionghai Dai, and Jinwei Gu. "Similarity-based Online Feature Selection in Content-based Image Retrieval". IEEE Transactions on Image Processing, vol. 15, no. 3, March 2006, pp 702-712. Wei Jiang, Mingjing Li, Hongjiang Zhang, and Jinwei Gu. "Online Feature Selection based on Generalized Feature Contrast Model". Proceedings of IEEE International Conference on Multimedia and Expo (ICME), June 2004, vol. 3, pp 27-30. top

Jinwei Gu

Principal Research Scientist

Recent Research (to be updated ...)

Learning Image-Adaptive Codebooks for Class-Agnostic Image Restoration [Paper]

Real-time Controllable Denoising for Image and Video [Paper]

Generating Aligned Pseudo-Supervision from Non-Aligned Data for Image Restoration in Under-Display Camera[Paper]

Deep Fouier Up-Sampling[Paper]

Deep Camera Obscura: An Image Restoration Pipeline for Lenless Pinhole Photography[Paper]

Low-light Image and Video Enhancement Using Deep Learning: A Survey[Paper][Data+Code]

iToF2dToF: A Robust and Flexible Representation for Data-driven Time-of-Flight Imaging[Project(Paper+Data+Code)]

ReconfigISP: Reconfigurable Camera Image Processing Pipeline[Project (Paper+Code)]

STAR: A Structure-Aware Lightweight Transformer for Real-time Image Enhancement [Paper]

Dynamic Quantization with Mixed Precision and Adaptive Resolution [Paper]

Removing Diffraction Image Artifacts in Under-Display Camera via Dynamic Skip Connection Network [Project/Code]

GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution [Project/Code]

Video Stitching for Linear Camera Arrays [Project]

NRMVS: Non-Rigid Multi-View Stereo [Arxiv][PDF][Supplementary Video][Project (coming soon)]

Neural RGB->D Sensing: Depth and Uncertainty from a Video Camera [Arxiv][Code]

PlaneRCNN: 3D Plane Detection and Reconstruction from a Single Image[Arxiv] [Code]

Neural Inverse Rendering of an Indoor Scene from a Single Image[Arxiv][Project and Dataset]

Very Power Efficient Neural Time-of-Flight[Arxiv][PDF][Supplementary Video][Project (coming soon)]

Context-Aware Synthesis and Placement of Object Instances[Arxiv][PDF][Supplementary Video][Code]

Switchable Temporal Propagation Network [Arxiv][PDF][Supplementary Video][Applications]

Separating Reflection and Transmission Images in the Wild [Arxiv][PDF][Supplementary Video][Code]

MapNet: Geometry-Aware Learning of Maps for Camera Localization[PDF][Code][Project]

Space-Time-Brightness Sampling Using an Adaptive Pixel-Wise Coded Exposure [PDF]

Reblur2Deblur: Deblurring Videos via Self-Supervised Learning[PDF][Project]

Learning Affinity via Spatial Propagation Networks [PDF][Supplementary Video]

A Lightweight Approach for On-the-Fly Reflectance Estimation[PDF][Supplementary Video]

Dynamic Facial Analysis: From Bayesian Filtering to Recurrent Neural Networks [Project][PDF][Supplementary Video]

Polarimetric Multi-view Stereo [PDF][Supplementary Document][Video]

Reconstructing Intensity Images from Binary Spatial Gradient Cameras [PDF][Supplementary Document][Video]

Deep Learning with Energy-efficient Binary Gradient Cameras [PDF]

Video Stitching with Spatial-Temporal Content-Preserving Warping[PDF][Video Results]

Previous Research

Exemplar-Based Method for Automatic Visual Editing and Retouching of Fine Art Reproduction [project]

Learning Discriminative Illumination and Filters for BTF Classification[project] [database](with code)

What is the Space of Camera Spectral Sensitivities?[project] [database](with code)

Classifying Raw Materials with Discriminative Illumination [project] [database]

Spectral Imaging Under Commonly Available Lighting [project]

Multiplexed Illumination for Scene Recovery in the Presence of Global Illumination[project]

Video from a Single Exposure Coded Photograph using a Learned Over-Complete Dictionary[project]

Coded Rolling Shutter Photography: Flexible Space-Time Sampling [project]

Measurement, Modeling, and Synthesis of Time-Varying Appearance of Natural Phenomena [PDF]

Removing Image Artifacts Due to Dirty Camera Lenses and Thin Occluders[project]

Printing Spatially-Varying Reflectance [project]

Compressive Structured Light for Recovering Inhomogenous Participating Media [project](with code)

Dirty Glass: Modeling and Rendering Contamination on Transparent Surfaces [project] [database] (with code)

Time-Varying Surface Appearance [project][database] (with code)

Radiometric Calibration from a Single Image [project]

Fingerprint Orientation Field Modeling and Its Applications [project]

Selected Publications (to be updated ...)

Computer Vision and Machine Learning

Computational Photography

Computer Graphics and Physics-based Computer Vision

Image Processing and Biometrics

Other Areas