Image Guided Depth Map Upsampling

Abstract

In this project we present a novel method for the challenging problem of depth image upsampling. Modern depth cameras such as Kinect or ToF cameras deliver dense, high quality depth measurements but are limited in their lateral resolution. To overcome this limitation we formulate a convex optimization problem using higher order regularization for depth image upsampling. In this optimization an anisotropic diffusion tensor, calculated from a high resolution intensity image, is used to guide the upsampling. We derive a numerical algorithm based on a primal-dual formulation that is efficiently parallelized and runs at multiple frames per second. We show that this novel upsampling clearly outperforms state of the art approaches in terms of speed and accuracy on the widely used Middlebury 2007 datasets. Furthermore, we introduce novel datasets with highly accurate groundtruth, which, for the first time, enable to benchmark depth upsampling methods using real sensor data.
tof elefant intensity elefant Multi Reconstruction Multi Reconstruction
Upsampling of a low resolution depth image using an additional high resolution intensity image through image guided anisotropic Total Generalized Variation. Depth maps are color coded for better visualization.

People/Contact

David Ferstl

Chrisitan Reinbacher

Matthias Rüther

Evaluation on Synthetic Datasets

Multi Reconstruction     Multi Reconstruction     Multi Reconstruction     Multi Reconstruction

Multi Reconstruction     Multi Reconstruction     Multi Reconstruction     Multi Reconstruction

Multi Reconstruction     Multi Reconstruction     Multi Reconstruction     Multi Reconstruction

Visual evaluation of x8 upsampling on the Middlebury 2007 dataset. Images form left to right: low resolution depth input of size 172x136 with added noise, high resolution intensity image of size 1390x1110, upsampling result after Image Guided Depth Upsampling of size 1390x1110, groundtruth depth image.
Algorithm Art

x2


x4


x8


x16
Books

x2


x4


x8


x16
Moebius

x2


x4


x8


x16
Nearest Neighbor 4.65 5.01 5.71 7.10 4.30 4.68 4.85 5.23 5.08 5.20 5.31 5.65
Bilinear 3.09 3.59 4.39 5.91 2.91 3.12 3.34 3.71 3.21 3.45 3.62 4.00
Yang [1] 1.36 1.93 2.45 4.52 1.12 1.47 1.81 2.92 1.25 1.63 2.06 3.21
He [2] 1.92 2.40 3.32 5.08 1.60 1.82 2.31 3.06 1.77 2.03 2.60 3.34
Diebel [3] 1.62 2.24 3.85 5.70 1.34 2.08 2.85 3.54 1.47 2.29 3.09 3.81
Chan [4] 1.83 2.90 4.75 7.70 1.04 1.36 1.94 3.07 1.17 1.55 2.28 3.55
Park [5] 1.24 1.82 2.78 4.17 0.99 1.43 1.98 3.04 1.03 1.49 2.13 3.09
OURS [6] 0.84 1.29 2.06 3.56 0.51 0.75 1.16 1.89 0.57 0.90 1.38 2.15
IMPORTANT: All errors in the table are given as Mean Absolute Error (MAE). Not as RMSE as reported in the paper!!
The RMSE results can be found in the table below.

Algorithm Art

x2


x4


x8


x16
Books

x2


x4


x8


x16
Moebius

x2


x4


x8


x16
Nearest Neighbor 6.55 7.48 9.02 11.45 6.16 6.31 6.62 7.33 6.59 6.78 7.00 7.52
Bilinear 4.58 5.62 7.14 9.72 3.95 4.31 4.71 5.38 4.20 4.56 4.87 5.43
Yang [1] 3.01 4.02 4.99 7.85 1.87 2.38 2.88 4.27 1.92 2.42 2.98 4.40
He [2] 3.55 4.41 5.72 8.49 2.37 2.74 3.42 4.53 2.48 2.83 3.57 4.58
Diebel [3] 3.49 4.51 6.39 9.39 2.06 3.00 4.05 5.13 2.13 3.11 4.18 5.17
Chan [4] 3.44 4.46 6.12 8.68 2.09 2.77 3.78 5.45 2.08 2.76 3.87 5.57
Park [5] 3.76 4.56 5.93 9.32 1.95 2.61 3.31 4.85 1.96 2.51 3.22 4.48
OURS [6] 3.19 4.06 5.08 7.61 1.52 2.21 2.47 3.54 1.47 2.03 2.58 3.50
Qualitative Middlebury evaluation measured as Root Mean Squared Error (RMSE).

[1] Q. Yang, R. Yang, J. Davis, and D. Nister. Spatial-depth super resolution for range images. In Proc. CVPR, 2007.

[2] K. He, J. Sun, and X. Tang. Guided image filtering. In Proc. ECCV, 2010.

[3] J. Diebel and S. Thrun. An application of markov random fields to range sensing. In Proc. NIPS, 2006.

[4] D. Chan, H. Buisman, C. Theobalt, and S. Thrun. A Noise-Aware Filter for Real-Time Depth Upsampling. In Proc. ECCV Workshops, 2008.

[5] J. Park, H. Kim, Y.-W. Tai, M. Brown, and I. Kweon. High quality depth map upsampling for 3d-tof cameras. In Proc. ICCV, 2011.

[6] D. Ferstl, C. Reinbacher, R. Ranftl, M. Ruether, and H. Bischof . Image Guided Depth Upsampling using Anisotropic Total Generalized Variation. In Proc. ICCV, 2013.

Evaluation on Novel Real-World Dataset

Multi Reconstruction     Multi Reconstruction     Multi Reconstruction     Multi Reconstruction

Multi Reconstruction     Multi Reconstruction     Multi Reconstruction     Multi Reconstruction

Multi Reconstruction     Multi Reconstruction     Multi Reconstruction     Multi Reconstruction

Visual evaluation of approximately x6.25 upsampling on our novel real-world dataset available for download at the ToFMark depth upsampling evaluation page. Images form left to right: low resolution depth input image from a PMD Nano Time-of-Flight camera of size 160x120 with added noise, high resolution intensity image of size 810x610, upsampling result after Image Guided Depth Upsampling of size 810x610, groundtruth depth image acquired by a high accuracy structured light scanner.
Algorithm Books

MAE[mm]
Shark

MAE[mm]
Devil

MAE[mm]
Nearest Neighbor 18.21 21.83 19.36
Bilinear 17.10 20.17 18.66
Kopf [1] 16.03 18.79 27.57
He [2] 15.74 18.21 27.04
OURS [3] 12.36 15.29 14.68

[1] J. Kopf, M. F. Cohen, D. Lischinski, and M. Uyttendaele. Joint bilateral upsampling. ACM Transactions on Graphics, 26(3), 2007.

[2] K. He, J. Sun, and X. Tang. Guided image filtering. In Proc. ECCV, 2010.

[3] D. Ferstl, C. Reinbacher, R. Ranftl, M. Ruether, and H. Bischof . Image Guided Depth Upsampling using Anisotropic Total Generalized Variation. In Proc. ICCV, 2013.

Downloads

MATLAB Source Code

Paper

Supplemental Material

Middlebury Evaluation Results

Publications

Conference Papers

2013

  1. Image Guided Depth Upsampling using Anisotropic Total Generalized Variation  [bib] [supp] David Ferstl, Christian Reinbacher, Rene Ranftl, Matthias Ruether, and Horst BischofIn Proceedings International Conference on Computer Vision (ICCV), IEEE, 2013
    [abstract]