A probabilistic attention model with occlusion-aware texture regression for 3D hand reconstruction from a single RGB image

Jiang, Zheheng and Rahmani, Hossein and Black, S. and Williams, Bryan (2023) A probabilistic attention model with occlusion-aware texture regression for 3D hand reconstruction from a single RGB image. In: Proceedings - 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023 :. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition . IEEE, pp. 758-767.

Text (A Probabilistic Attention Model with Occlusion-aware Texture Regression for 3D Hand Reconstruction from a Single RGB Image)
3D-HandReconstruction-Accepted.pdf - Accepted Version
Available under License Creative Commons Attribution-NonCommercial.
Download (17MB)

Abstract

Recently, deep learning based approaches have shown promising results in 3D hand reconstruction from a single RGB image. These approaches can be roughly divided into model-based approaches, which are heavily dependent on the model's parameter space, and model-free approaches, which require large numbers of 3D ground truths to reduce depth ambiguity and struggle in weakly-supervised scenarios. To overcome these issues, we propose a novel probabilistic model to achieve the robustness of model-based approaches and reduced dependence on the model's parameter space of model-free approaches. The proposed probabilistic model incorporates a model-based network as a prior-net to estimate the prior probability distribution of joints and vertices. An Attention-based Mesh Vertices Uncertainty Regression (AMVUR) model is proposed to capture dependencies among vertices and the correlation between joints and mesh vertices to improve their feature representation. We further propose a learning based occlusion-aware Hand Texture Regression model to achieve high-fidelity texture reconstruction. We demonstrate the flexibility of the proposed probabilistic model to be trained in both supervised and weakly-supervised scenarios. The experimental results demonstrate our probabilistic model's state-of-the-art accuracy in 3D hand and texture reconstruction from a single image in both training schemes, including in the presence of severe occlusions.