HPM-TDP: An Efficient Hierarchical PatchMatch Depth Estimation Approach using Tree Dynamic Programming

石莹 2021-08-08 浏览

摘要

利用立体影像精准、高效的重建真实场景稠密三维几何信息是数字表面模型生产、三维重建与可视化、自动驾驶以及机器人导航的关键步骤。近年来，尽管立体匹配技术在匹配精度和效率方面显著提升，但弱纹理以及重复纹理区域匹配仍然面临严峻挑战。针对这些困难和挑战，本文提出一种高效层次PatchMatch深度重建方法HPM-TDP，该方法集成粗到精的图像金字塔策略到连续全局能量函数最优化框架，通过有效的集成层次PatchMatch策略（HPM）和树动态规划的局部扩展运动（LEM-TDP），快速实现全局能量函数最小化。首先，粗到精的图像金字塔策略被有效的集成到PMF框架以快速的获得原始分辨率影像的层次视差平面先验信息，从而用于能量函数最优化框架中各像元的视差平面初始化。其次，通过多层次代价聚合策略提高弱纹理以及重复纹理区域像元匹配代价函数鲁棒性。最后， 采用HPM框架以及LEM-TDP策略来求解非模块化的全局能量函数最优化问题，生成全局最优化的视差平面图。通过Middlebury 3.0、KITTI 2015以及Vaihingen数据集来测试HPM-TDP的性能。实验结果表明，HPM-TDP在三个数据集中都能获得好的性能，HPM-TDP在Middlebury 3.0以及KITTI 2015训练数据集上的（“Out-Noc”，“Avg-Noc”，“Out-All”，“Avg-All”）分别为（15.45%，4.16px，24.26%，12.14px）和（5.46%，1.20px，6.55%，1.54px），在Vaihingen数据集上的（“Out-All”，“Avg-All”）为（26.32%，4.04px）。本文所提出HPM-TDP方法框架如下：

 

图1 HPM-TDP流程图

图2及图3显示了不同方法生成的视差图结果。从图2和图3可以看出，HPM-TDP方法能够得到比PMF以及SPM-BP方法更优的视差图结果，特别是对于蓝色及红色框所对应弱纹理及重复纹理区域，原因有三：1）利用层次视差平面先验信息初始化每个像元的视差平面, 为能量函数最优化提供稳定可靠的初值，避免图像中噪声、弱纹理及重复纹理区域视差平面图陷入局部最优；2）通过多层次代价聚合策略提高弱纹理及重复纹理区域匹配代价函数鲁棒性；3）通过LEM-TDP能量函数最优化策略迭代更新超像素中各像元视差平面，从而得到各像元在连续视差平面空间近似最优解。相比于BP求解器，TDP具有全局特性，它同时更新超像素中所有像元视差平面，有效避免生成的视差平面图陷入局部最优。

 

图2 KITTI 2015数据集中影像“000011_10”（左）和“000006_10”（右）视差图结果

 

图3 Vaihingen数据集中影像“10040083”视差图结果

Abstract

Accurate and efficient estimation of the dense depth information from a pair of stereo images is a key step for many applications such as digital surface model production, 3D reconstruction and visualization, autonomous driving, and robotic navigation. Although great progress has been achieved in stereo matching over the past decade, the matching difficulties in poor and repetitive texture regions remain an issue. Aiming at solving the shortcomings of the current methods, this paper proposes HPM-TDP, which is an efficient hierarchical PatchMatch depth estimation approach that integrates a coarse-to-fine image pyramid strategy with a continuous Markov random field (MRF)-based global energy optimization framework, and minimizes the energy function by combining a hierarchical PatchMatch (HPM) framework and local α-expansion based tree dynamic programming (TDP). Firstly, the coarse-to-fine image pyramid strategy is integrated with the PatchMatch filter algorithm to quickly generate the hierarchical disparity plane prior for initializing each pixel’s disparity plane of the energy function optimization. Secondly, a multi-resolution cost aggregation strategy is adopted to boost the robustness of the matching cost function in the poor and repetitive texture areas. Finally, the HPM framework and local α-expansion based TDP are adopted to solve the non-submodular energy optimization problem, resulting in a globally optimized disparity plane map. Three benchmark datasets—the Middlebury 3.0, KITTI 2015, and Vaihingen datasets—were used to test the performance of HPM-TDP. The comprehensive experimental results demonstrate that HPM-TDP obtains a good performance on all datasets in terms of the (“Out-Noc”, “Avg-Noc”, “Out-All”, “Avg-All”) of (15.45%, 4.16px, 24.26%, 12.14px) and (5.46%, 1.20px, 6.55%, 1.54px) for Middlebury 3.0 and KITTI 2015 training datasets, and the (“Out-All”, “Avg-All”) of (26.32%, 4.04px) for Vaihingen dataset, respectively.