Image Distortion in Surgical Robots

When cameras take pictures, they are effectively mapping the light reflected by a 3D object in the world to a 2D image at the back of the camera. In this process, lenses in cameras produce distortions that are caused by random errors in manufacturing. We use calibration patterns to estimate and account for these distortions in the 'eyes' of a surgical robot.

Project Dates: 5/1/20 - 12/1/20

Date Published: 6/23/24

Overview

This project focuses on the calibration of cameras for robotic surgical systems, particularly those developed by Vicarious Surgical. My main contributions for project include the development of a precise camera calibration method tailored for robotic surgical applications and implementation of algorithms to enhance the accuracy and reliability of the calibration process.

Specifically, we created a GUI software that allows users to input images of certain known patterns for both stereo and mono camera systems. The software uses these patterns to estimate the camera intrinsics and extrinsics by detecting the corners on each pattern. Additionally, knowing these values, it is able to generate a heat-map for images taken by the same lens to visualize distortions across the image.

I would like to thank Vicarious Surgical for my time their as an intern and Fabrizio Santini for his guidance.

Theoretical Background

When cameras take pictures, they are effectively mapping the light reflected by a 3D object in the world to a 2D image at the back of the camera. Pinhole cameras, starting in the late 20th century, followed this basic principle. Modern cameras use lenses instead of pinholes. However, these lenses still produce distortions that are caused by random errors in manufacturing. There are two major types of distortions: radial and tangential. Radial distortions cause images to appear curved while tangential distortions cause the image to appear longer or tilted. Luckily, distortion can be captured using five numbers called Distortion Coefficients. We can estimate these values by taking pictures of known patterns such as chessboards and checkerboards.

Intrinsic Parameters

These parameters include the focal length, principal point, and skew coefficient, and are represented by the matrix ( K ):

\[\begin{align*} K = \begin{bmatrix} f_x & s & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix} \end{align*}\]

where:

  • \(f_x\) and \(f_y\) are the focal lengths in the x and y directions.
  • \(c_x\) and \(c_y\) are the coordinates of the principal point.
  • \(s\) is the skew coefficient.

Extrinsic Parameters

These parameters define the rotation and translation of the camera with respect to the world coordinate system: \([R|t]\), where \(R\) is the rotation matrix and \(t\) is the translation vector.

Camera Calibration Process

The calibration process involves capturing multiple images of a known calibration pattern (e.g., a checkerboard) and solving the following system of equations to estimate the camera parameters:

\[\begin{align*} s \begin{bmatrix} u \\ v \\ 1 \end{bmatrix} = K [R|t] \begin{bmatrix} X \\ Y \\ Z \\ 1 \end{bmatrix} \end{align*}\]

where:

  • \((u, v)\) are the pixel coordinates of the observed point.
  • \((X, Y, Z)\) are the coordinates of the point in the world coordinate system.
  • \(s\) is a scaling factor.

Approach and Technical Details

Methodology

  1. Pattern Detection:
    • Used a checkerboard pattern for calibration.
    • Employed computer vision techniques to detect the corners of the checkerboard in the captured images.
  2. Parameter Estimation:
    • Implemented algorithms to estimate the intrinsic and extrinsic parameters of the camera.
    • Utilized optimization techniques to minimize the re-projection error, which is the difference between the observed and projected points.
  3. Validation:
    • Conducted extensive testing using synthetic and real-world data.
    • Validated the accuracy of the calibration method by comparing the re-projection errors and ensuring they are within acceptable limits.

Tools and Technologies

  • Programming Languages: Python, C++
  • Libraries: OpenCV, NumPy
  • Hardware: Robotic surgical system from Vicarious Surgical