top of page

Deep Stereo Matching for Camera Pose Estimation

The Challenge

Accurate camera pose estimation is fundamental to applications such as robotics, 3D reconstruction, autonomous systems, and augmented reality. Traditional feature-based approaches estimate camera motion primarily from image correspondences, but their performance can degrade when visual features are sparse, ambiguous, or affected by challenging viewpoints. The goal of this project was to investigate whether stereo depth information from deep learning-based models could improve the robustness and accuracy of feature-based camera pose estimation.


The Vision

This project explored a depth-guided approach to camera pose estimation, combining learned stereo depth with conventional feature-based vision techniques. Instead of relying solely on 2D image correspondences, the system leveraged estimated depth to recover 3D scene information, enabling more geometrically informed estimation of camera motion.


System Objective

  • Develop a feature-based camera pose estimation pipeline using stereo depth information

  • Investigate whether depth can improve relative and global camera pose estimation

  • Compare depth-guided estimation against a classical image-based baseline

  • Evaluate the influence of different deep stereo matching models on pose accuracy

  • Analyze robustness under varying levels of depth noise

  • Build an interactive evaluation application to visualize and compare the results


Key Technologies & Components

  • Deep Stereo Models: WAFT-Stereo, FoundationStereo, MonSter++

  • Computer Vision: Feature detection, feature matching, stereo depth, 3D reconstruction

  • Pose Estimation: Depth-guided rigid registration and global pose estimation

  • Optimization: Ceres Solver

  • Programming: C++

  • Libraries & Tools: OpenCV, Ceres Solver, ONNX Runtime / TensorRT

  • Evaluation: Position error and rotation error

  • Application: Custom IMGUI-based Pose Test App


Pose Estimation Flow

  • Generate a set of synthetic stereo images from a 3D scene

  • Detect visual features and establish correspondences between image pairs

  • Generate stereo depth maps using deep stereo matching models

  • Associate matched image features with their corresponding 3D points using depth

  • Estimate relative camera transformations through depth-guided geometric alignment

  • Optimize and combine pairwise transformations to obtain a globally consistent camera trajectory

  • Compare the resulting pose against the known ground-truth camera configuration


Why Deep Stereo Matching ?

Advantages:

  • Provides 3D information in addition to conventional 2D feature correspondences

  • Enables geometric reasoning using estimated scene depth

  • Can improve robustness when image-only correspondences are ambiguous

  • Allows camera motion estimation without relying exclusively on the Essential Matrix

  • Provides a framework for investigating how the quality of learned depth affects pose estimation

Trade-Offs:

  • Deep stereo models introduce additional computational complexity

  • Pose accuracy depends on the quality and consistency of estimated depth

  • Depth errors and noise can propagate into the final camera pose

  • Different stereo models can produce significantly different depth characteristics


Deep Stereo & Pose Estimation Implementation

The system was evaluated using three deep stereo matching approaches:

  • WAFT-Stereo — evaluated at multiple resolutions to study its effect on pose estimation

  • FoundationStereo — a foundation-model-based approach for stereo depth estimation

  • MonSter++ — a unified stereo matching approach incorporating monocular depth priors

For each model, the generated depth maps were integrated into the pose estimation pipeline. A ground-truth depth map was also used as an ideal reference, with controlled Gaussian noise introduced to evaluate the sensitivity of pose estimation to depth inaccuracies.


Experimental Results

  • Successfully developed a complete feature-based camera pose estimation pipeline

  • Compared classical image-based and depth-guided pose estimation approaches

  • Evaluated pose accuracy using average position error and average rotation error

  • Investigated the effect of depth noise on camera pose estimation

  • Compared the performance of WAFT-Stereo, FoundationStereo, and MonSter++

  • Analyzed pairwise pose connections and their impact on global camera pose consistency

  • Identified failure cases involving disconnected views and insufficient feature correspondences


Evaluation Application

A custom Pose Test App was developed specifically for experimentation and evaluation. The application provides an end-to-end workflow for:

  • Creating camera projects from 3D models

  • Planning camera poses

  • Rendering synthetic stereo images

  • Detecting and matching image features

  • Generating stereo depth

  • Selecting between classical and depth-guided pose estimation

  • Applying controlled depth noise

  • Performing global pose estimation

  • Visualizing position and rotation errors


Conclusion

This project demonstrates how deep stereo matching can be integrated with feature-based computer vision to provide depth-aware camera pose estimation. By incorporating 3D information into the estimation process, the work investigates an alternative to purely image-based pose estimation and highlights the importance of depth quality, model selection, and geometric consistency for robust camera localization.

As a Master’s thesis project, the work combines deep learning, stereo vision, geometric computer vision, and optimization to study how learned depth can contribute to more robust camera pose estimation.

bottom of page