Reference frames
It is important to properly distinguish between reference frames. For this document, there are five relevant reference frames:
- Static frame: The external frame that the camera will be calibrated to. Commonly the static frame is identical to the robot base frame, however it can be any static frame defined in the world.
- Robot base frame: The coordinate system defined by the robot. The exact origin and directions of the axes are hardware specific. It is generally used to command the robot to move, and for poses reported by the robot.
- Robot end-effector frame: A moving reference frame attached to the end-effector of the robot. Again, the exact origin and directions of the axes are hardware specific. A common convention is to have the Z axis perpendicular to the robot flange, pointing away from the robot.
- Camera frame: The reference frame of data given by the camera when no calibration has been executed. For an Ensenso the origin is at the left stereo lens.
- Calibration plate frame: The frame of the calibration plate. For the Ensenso calibration plates, the origin is at the center of the calibration plate. If you look at the calibration plate from above with the black triangle in the top left corner, positive X is pointing right and positive Y pointing down, meaning Z is pointing down into the calibration plate. Note that this can also be seen in nxView.
All of these reference frames are hardware specific. However, their exact origin and direction (with the exception of the calibration plate) is irrelevant, as long as they are used consistently. The purpose of this calibration routine is to find the transformation from the camera frame to the static frame. In most cases, the static frame and the robot base frame are the same.
After calibrating, the camera will output coordinates in the static frame.
Calibrations
This protocol supports two types of calibration: the hand-eye calibration sequence and the workspace calibration.
The main advantage of hand-eye calibration sequence is that the position of the calibration plate w.r.t. the robot does not need to be known accurately, only the position of the robot end-effector needs to be known. Its main disadvantage is that it requires multiple views of the calibration plate, with sufficient variation in translation and rotation in all available axes (XYZ). The hand-eye calibration routine works by recording images of the calibration plate in different positions. Recording more images can help to improve calibration accuracy. A minimum of 5 images is required, recommended is 30 images.
In cases where recording this variation is not possible, it is advised to use the workspace calibration. For the workspace calibration, it is required to know the exact position of the calibration plate in the world. Using this information along with the position calibration plate w.r.t. the camera (which is determined by the camera using the calibration plate pattern), we can determine the position of the camera in the world. The main advantage of the workspace calibration is that you do not need to capture a variety of calibration plate poses. The main disadvantage is the requirement that the pose of the calibration plate w.r.t. the static frame needs to be known with high accuracy. Note that this type of calibration can also be performed in nxView for Ensenso cameras; the API provided here is intended for automation purposes.
This document describes both calibration routines for the Fizyr vision system.
Hand-eye calibration
The steps required to perform the hand-eye calibration are shown in the pseudo-code below.
It assumes a list of pre-determined robot poses that is used to record the calibration plate at different poses. There are two version to acquire different calibration plate poses, namely Eye-to-hand and Eye-in-hand. The Eye-to-hand achieves this variety by moving the calibration pattern attached to the end-effector. The eye-in-hand, on the other hand, achieves that by moving the camera attached to the end-effector. It is recommended that these poses are around the area of interest where the robot will operate. The camera should have the highest accuracy in this region.
Eye-to-hand
The moving frame depends on the robot link on which the calibration pattern is mounted. In most cases, it is the robot end-effector. The fixed frame, in most cases, is the robot base.
# The poses of the moving frame to which the calibration pattern is attached, with respect to a fixed frame.
calibration_poses = {pose_0, pose_1, pose_2, ..., pose_N}
# The source frame of the transformation.
image_type = "left_stereo"
# The target frame of the transformation.
target_frame = "base"
# Set the algorithm to eye-to-hand.
request = sendRequest(START_EYE_TO_HAND)
for pose in calibration_poses:
# Move the robot to the desired position.
# These positions are defined as poses for the robot end-effector
# with respect to the robot base frame.
robot.move(pose)
# Wait 1000 ms to ensure no vibrations.
wait(1000)
# Send record request update.
pose = serialize(pose)
request.sendUpdate(RECORD_SAMPLE, pose)
# Compute the final calibration.
request.sendUpdate(COMPUTE)
result = request.getResponse()
# Set calibration in camera.
calibration = serialize(result.calibration_id)
sendRequest(SET_ACTIVE_CALIBRATION, calibration)Eye-in-hand
The moving frame depends on the robot link on which the camera is mounted. In most cases, it is the robot end-effector. The fixed frame, in most cases, is the robot base.
# The poses of the moving frame to which the camera is attached, with respect to a fixed frame.
calibration_poses = {pose_0, pose_1, pose_2, ..., pose_N}
# The source frame of the transformation.
image_type = "left_stereo"
# The target frame of the transformation.
target_frame = "base"
# Set the algorithm to eye-in-hand.
request = sendRequest(START_EYE_IN_HAND)
for pose in calibration_poses:
# Move the robot to the desired position.
# These positions are defined as poses fo
# with respect to the robot base frame.
robot.move(pose)
# Wait 1000 ms to ensure no vibrations.
wait(1000)
# Send record request update.
pose = serialize(pose)
request.sendUpdate(RECORD_SAMPLE, pose)
# Compute the final calibration.
request.sendUpdate(COMPUTE)
result = request.getResponse()
# Set calibration in camera.
set_calibration_request = serialize(result.calibration_id)
sendRequest(SET_ACTIVE_CALIBRATION, set_calibration_request)Robot poses
To obtain the best results, it is advised to define the robot poses such that there is variation in each axes, both in translation and rotation. The recommended method is to define a 3D grid of points where the robot moves to. At each location on this 3D grid, the end-effector can be rotated clockwise and counter-clockwise around each axis. For each rotation a calibration sample can be recorded.
One-Shot calibration
The steps required to perform the one-shot calibration are shown in the pseudo-code below.
It is recommended that the pose is around the area of interest where the robot will operate.
# The source frame of the transformation.
image_type = "left_stereo"
# The target frame of the transformation.
target_frame = "base"
# Move the robot to the desired position.
# This position is defined as a pose for the robot end-effector
# with respect to the robot base frame.
robot.move(calibration_pose)
# Wait 1000 ms to ensure no vibrations.
wait(1000)
# Now that we know the location of the end-effector of
# the robot in the static frame we need to know the location
# of the calibration plate in the static frame.
# EE_TO_CALIBRATION is the transformation from end-effector
# to calibration plate. Note that any error in this transformation
# will propagate directly to the calibration.
pattern_pose = pose * EE_TO_CALIBRATION
# Send the pattern pose for calibration computation.
pattern_pose = serialize(pattern_pose)
request = sendRequest(RECORD_COMPUTE_ONE_SHOT_CALIBRATION, pattern_pose)
# Get the computed calibration.
result = request.getResponse()
# Set calibration in camera.
set_calibration_request = serialize(result.calibration_id)
sendRequest(SET_ACTIVE_CALIBRATION, set_calibration_request)Final remark
Note that both cases return the pose of the camera in the static frame. Any data coming from the camera after calibration will be w.r.t. the static frame. The returned camera pose is mostly for debugging purposes.