Recovering Spectral Reflectance under Commonly Available Lighting Conditions
Recovering the spectral reflectance of a scene is important for scene
understanding. Previous approaches use either specialized filters or
controlled illumination where the extra hardware prevents many practical
applications. In this paper, we propose a method that accurately
recovers spectral reflectance from two images taken with conventional
consumer cameras under commonly available lighting conditions, such as
daylight at different times over a day, camera flash and ambient light,
and fluorescent and tungsten light. Our approach does not require camera
spectral sensitivities or the spectra of the illumination, which makes
it easy to implement for a variety of practical applications. Based on
noise analysis, we also derive theoretical predictors that answer: (1)
which two lighting conditions lead to the most accurate spectral
recovery overall, and (2) for two given lighting conditions, which
spectral reflectance is more likely to be estimated accurately. We
implement the method on a variety of cameras from high-end DSLRs to
cellphone cameras, and apply the recovered spectral reflectance for
several applications such as fine art reproduction, fruit
identification, and material classification. Both simulation and
experimental results demonstrate the effectiveness of the proposed
Jun Jiang and Jinwei Gu. Recovering Spectral Reflectance under
Commonly Available Lighting Conditions. CVPR 2012
Jun Jiang and Jinwei Gu.Supplementary
Document (with other experimental details).
Spectral Recovery of Fine Art Painting:
We recover the spectral reflectance of the oil painting
‘Daisy’. (a) and (b) The captured images under the fluorescent light
and the tungsten light. (c), (d) and (e) The reflectance estimated
and measured at selected areas (P, Q and R) on the painting. (f) and
(g) The rendered image under CIE D65 and IllA.
Identification of Fruits:
The spectral recovery and identification of fruits. Before the experiment, a
database was created by including measured reflectance of ten different
kinds of fruits. Next, we captured fruit under
the fluorescent light (a) and the tungsten light (b) by Canon 60D.
(c) The measured and estimated reflectance of each fruit. (d) The rendering of fruits under CIE IllA. The artifacts in green
were due to the shadows in the captured images. (e) All fruits were identified correctly except the pumpkin (P2) and
mango (P3), due to the very similarity of their
reflectance. On the other hand, while being similar in color
in (a) and (b), the squash (P5) was able to be
distinguished from the pumpkin (P2) and mango (P3).
The identification of fruits by the spectral information is robust to
the scene illuminant and to the distortion of the food shape during the
processing of food. In addition, if the spectral images of the fruits
at different times (g when they were ripe or rotten) are available, the
identification results can be useful as a quality control measure.
Classification of Outdoor Materials:
We recover the spectral reflectance of outdoor materials under daylight. Based on the recovered reflectance, classifications
of materials are made. (a) and (b) The captured pictures under daylight
at 7:41 a.m. and 10:05 a.m. (c) The daylight spectra (normalized to be 1 at 560 nm). (d) The measured and
estimated reflectance of the aloe, ceramic (the specular side),
copper, ceramic (the diffuse side), gray plastic, soil, green
plastic, and aluminum. (e) The rendering of the scene under CIE
IllA. (f) All materials were classified (indicated by different colors). The ColorChecker Passport (CCP) was included to
characterize the lighting.
Extension to Smartphone Camera: Nokia N900:
Smartphones and other mobile platforms are gradually changing how we operate our daily life by integrating
both the eyes (the camera) and brain (the computing unit) on a cell phone.
While a handheld spectrophotometer can be used to measure the spectral reflectance of materials, the use of cell phone cameras has the benefit of lower cost
and acquisition of the scene reflectance rather than of a small uniform area on a surface.
We did an experiment to evaluate the spectral recovery performance of smartphones, Nokia N900, by validating CCDC. Moreover, the picture under one (rather than two)
lighting condition was used to recover the scene reflectance.
The validation of CCDC using the camera phone, Nokia N900, under the tungsten light only. (a) The estimated and measured reflectance of certain patches in CCDC.
The numbers on the top of each plot are the spectral RMS error, and color difference under CIE D65 and IllA. The patch index (\#) is shown as well.
A close spectral and colorimetric match could be achieved generally between the ground truth and our results except for patches of reddish colors.
(b) Noise analysis was performed by calculating $
ho$ to tell which reflectance is likely to be predicted better under the tungsten light.
Two patches were selected with small and large $
ho$. A smaller $
ho$ means a better spectral match.
CVPR 2012 CCD Poster (PDF)
Code (including an
example)The zip file includes a set
of exemplar images and code to recover the spectral reflectance under commonly available lighting conditions.
Multispectral Imaging Using Multiplexed Illumination
Multispectral Imaging on Fine Art Reproductions