Bundle Adjustment with Ceres Solver: A Deep Dive into Non-linear Optimization

Bundle adjustment is a crucial optimization problem in computer vision and photogrammetry. In this article, we'll explore how to implement bundle adjustment using Google's Ceres Solver in C++, breaking down the components of non-linear optimization and their practical applications.

Understanding the Problem

Bundle adjustment is the process of refining a visual reconstruction to produce jointly optimal structure and viewing parameter estimates. It minimizes the reprojection error between the image locations of observed and predicted image points.

Key Components of Ceres Solver

class ReprojectionError {
public:
    ReprojectionError(double observed_x, double observed_y)
        : observed_x_(observed_x), observed_y_(observed_y) {}

    template <typename T>
    bool operator()(const T* const camera,
                   const T* const point,
                   T* residuals) const {
        // Project 3D point to 2D
        T p[3];
        // Apply camera rotation
        ceres::AngleAxisRotatePoint(camera, point, p);
        // Apply camera translation
        p[0] += camera[3];
        p[1] += camera[4];
        p[2] += camera[5];

        // Perspective projection
        T xp = p[0] / p[2];
        T yp = p[1] / p[2];

        // Apply camera intrinsics
        T predicted_x = camera[6] * xp + camera[8];
        T predicted_y = camera[7] * yp + camera[9];

        // Compute residuals
        residuals[0] = predicted_x - T(observed_x_);
        residuals[1] = predicted_y - T(observed_y_);

        return true;
    }

private:
    double observed_x_;
    double observed_y_;
};

void BundleAdjustment(const std::vector<Point3D>& points,
                      const std::vector<Camera>& cameras,
                      const std::vector<Observation>& observations) {
    ceres::Problem problem;
    
    for (const auto& observation : observations) {
        ceres::CostFunction* cost_function =
            new ceres::AutoDiffCostFunction<ReprojectionError, 2, 9, 3>(
                new ReprojectionError(observation.x, observation.y));
                
        problem.AddResidualBlock(cost_function,
                               new ceres::HuberLoss(1.0),
                               cameras[observation.camera_idx].data(),
                               points[observation.point_idx].data());
    }
    
    // Set camera parameters constant
    for (int i = 0; i < cameras.size(); ++i) {
        problem.SetParameterBlockConstant(cameras[i].data());
    }
}

ceres::Solver::Options options;
options.linear_solver_type = ceres::SPARSE_SCHUR;
options.minimizer_progress_to_stdout = true;
options.max_num_iterations = 100;
options.trust_region_strategy_type = ceres::LEVENBERG_MARQUARDT;

ceres::Solver::Summary summary;
ceres::Solve(options, &problem, &summary);
std::cout << summary.FullReport() << std::endl;

Visualizing Reconstruction Accuracy

The accuracy of 3D reconstruction using bundle adjustment depends on several factors: - Quality of feature matching - Camera calibration accuracy - Number and distribution of observations - Presence of outliers - Initial estimate quality

In the visualization above, you can see: - Green points: Original 3D points - Red points: Reconstructed points after bundle adjustment - Use the slider to adjust the simulated reconstruction error - Rotate the view to see the point cloud from different angles

Understanding the Components

1. Cost Function

The cost function defines the error metric we want to minimize. In bundle adjustment, this is typically the reprojection error - the difference between observed image points and projected 3D points. The ReprojectionError class implements this using Ceres's automatic differentiation.

2. Problem Setup

The problem setup involves: - Creating residual blocks for each observation - Adding robust loss functions (Huber loss in this case) - Setting parameter blocks as constant when needed - Managing the optimization variables (camera parameters and 3D points)

3. Solver Configuration

Ceres offers various solver options: - Linear solver type (SPARSE_SCHUR for bundle adjustment) - Trust region strategy (Levenberg-Marquardt) - Maximum iterations - Convergence criteria

Practical Considerations

• Use robust loss functions to handle outliers
• Consider parameterization of rotations (angle-axis vs. quaternions)
• Implement proper initialization of camera parameters and 3D points
• Monitor convergence and adjust solver parameters accordingly

Conclusion

Bundle adjustment with Ceres Solver provides a powerful framework for solving non-linear optimization problems in computer vision. Understanding the components and their interactions is crucial for implementing efficient and accurate solutions.