Protocol Documentation

Feedback about a pick cycle.

This is only necessary if providing feedback about a particular pick plan, rather than the entire pick cycle. | | object_id | uint32uint32 | optional | Optional ID of the detection.

If specified together with pick_plan_id, then the feedback is about a particular object in a pick plan. For example: one object from a pick plan was not picked successfully.

If specified without a pick_plan_id, the feedback is about the object detection itself. For example: the classification of a detected object must be wrong based on application knowledge. | | code | int32int32 | optional | An optional code associated with the provided feedback.

The code can be chosen freely by the application. It can be used to generate more detailed performance statistics. | | message | stringstring | optional | An optional message associated with the provided feedback.

The message can be chosen freely by the application. | | close_cycle | boolbool | | If true, close the pick cycle. |

The response to a FeedbackRequest.

A request to add tags to a cycle.

The response to the add tags service.

A rotation of a specific angle around a specific axis.

The request to check whether the container is completely empty.

The response that indicates whether or not the container is completely empty.

The request for container layout check.

The response of container layout check.

True if the detected container matches the expected container. The expected container can be specified in the container_id field of the [start_pick_cycle] request. |

The scores of all classes of a single descriptor.

The first element corresponds to class ID 0, then 1, etc.. |

The scores of all classes of a single descriptor, as named scores.

Close a pick cycle.

Response body for the close_pick_cycle service.

An empty message

Euler rotation representation

This determines the order in which the rotations are applied. | | angles | floatfloat | repeated | An array containing the angles of rotation for each axis.

The order of the angles is the order as per the variant of the format field.

This field needs exactly 3 values |

Request body for the get_api_version request.

Response body for the get_api_version request.

Request for the get_cycle_status service.

Also returns when the cycle progresses past this status. |

Get the current status of the pick cycle.

Get 2D information about the detected objects of a pick cycle.

The response will only contain information about the detections with the corresponding IDs. This can be useful for querying information about objects from a particular pick plan.

If this list is empty or not specified, then information about all available detections will be included in the response. | | report_class_ids | uint32uint32 | repeated | Report the ID of the predicted class for the requested descriptors.

This is an easy way to receive information about classification descriptors, like shape or material. Only the ID of the highest-scoring class in the descriptor is reported.

The provided u32 values indicate which descriptors should be included in the report.

For example: - request.report_class_ids: [0, 1] where 0 indicates "shape" and 1 indicates "material" - response.class_ids: [0, 3] where 0 indicates "box" and 3 indicates "plastic"

Note that the available descriptors and the meaning of their values depend on the installation. These are not covered by this document.

The information reported is identical to report_class_names, except that this field uses numerical IDs rather than string names. | | report_class_names | stringstring | repeated | Report the name of the predicted class for the requested descriptors.

This is an easy way to receive information about classification descriptors, like shape or material. Only the name of the highest-scoring class in the descriptor is reported.

The provided String values indicate which descriptors should be included in the report.

For example: - request.report_class_names: ["shapes", "materials"] - response.class_names: ["box", "plastic"]

Note that the available descriptors and the meaning of their values depend on the installation.

The information reported is identical to report_class_ids, except that this field uses string names rather than numerical IDs. | | report_class_scores_by_id | uint32uint32 | repeated | Report the scores of the requested descriptors.

This will report the scores of all possible classes for a descriptor.

The provided u32 values indicate which descriptors should be included in the report.

For example: - request.report_class_scores_by_id: [0, 1] - response.class_scores_by_id: [[0.97, 0.01, 0.0, 0.0], [0.07, 0.02, 0.98]]

Note that the available descriptors and the meaning of their values depend on the installation. These are not covered by this document. | | report_class_scores_by_name | stringstring | repeated | Report the scores of the requested descriptors.

This will report the scores of all possible classes for a descriptor.

The provided String values indicate which descriptors should be included in the report.

For example: - request.report_class_scores_by_name: ["shapes", "materials"] - response.class_scores_by_name: [[0.97, 0.01, 0.0, 0.0], [0.07, 0.02, 0.98]]

Note that the available descriptors and the meaning of their values depend on the installation. These are not covered by this document. |

2D information about the detected objects of a pick cycle.

The detections included depend on the object_ids field of the request. |

Get 3D information about the detected objects of a pick cycle.

The response will only contain information about the detections with the corresponding IDs. This can be useful for querying information about objects from a particular pick plan.

If this list is empty or not specified, then information about all available detections will be included in the response. | | rotation_format | RotationFormatRotationFormat | optional | Request that all 3D rotations in the response use the specified format. |

3D information about the detected objects of a pick cycle.

The detections included depend on the object_ids field of the request. |

Get the next pick plans(s) of a pick cycle.

The service call will delay it's response until either this number of pick plans are available, or until the computation has finished and no more pick plans are available.

If not specified, no limit is applied and this call will return only when the whole scene has been fully analyzed. | | rotation_format | RotationFormatRotationFormat | optional | Request that all 3D rotations in the response use the specified format. |

A number of ordered pick plans for a pick cycle.

Request body for the get_server_info request.

Response body for the get_server_info request.

Inverse kinematics for the attached gripper.

This will usually correspond with the flange of a robot to which the gripper is attached. | | joint_values | floatfloat | repeated | The joint values to set on the gripper. |

List open pick cycles.

Response body for the list_pick_cycles service.

The score of a class.

2D information about the detected objects of a pick cycle.

An object detected in the scene, after filtering and prioritization.

Note: the Z information is included, but in most cases it is not meaningful, since the camera can usually only see the top surface of the object. |

A single object in a pick plan.

Multiple pick plans may try to pick the same objects. In such cases, the detection_id will be the same for the object in both pick plans.

However, suction_groups, object_pose, and bounding_box can still be different in each pick plan.

More information about the detected object can be queried using the [get_object_info2] service. | | suction_groups | uint32uint32 | repeated | The suction groups that must be used to pick this object. | | object_pose | Pose3Pose3 | | The estimated pose of the object in 3D space.

This pose represents the estimated position and orientation of the face of the object being picked. It should not be used as a position for picking, but only as an estimation on how the object is held in the gripper. | | dimensions | Vector2Vector2 | | The dimensions of the object on the XY plane of the object pose.

The x component of the dimensions is aligned along the x axis of the object pose. |

A pick plan describing how to pick one or more objects.

Normally, the robot should move the gripper to this pose to perform the pick.

If inverse kinematics are computed, you should use the information from that instead. | | objects | PickObjectPickObject | repeated | The objects that will be picked. | | inverse_kinematics | InverseKinematicsInverseKinematics | optional | Inverse kinematics for gripper with movable joints.

This will automatically be added to the pick plan if configured on the selected gripper. |

A three dimensional point in space.

A pose in three dimensional space.

A pose combines a position and an orientation.

This struct represents a quaternion.

The quaternion w+xi+yj+zk is a convenient representation of orientations and rotations of objects in three dimensions. Operations interpreting the quaternion as rotation have undefined behavior if the quaternion is not normalized.

A rotation in one of several possible formats.

Information about a server, including software and API versions.

This is important for interfaces that are implemented by multiple applications, (eg: cameras). | | application_version | VersionVersion | | The version of the application.

This can be useful for logging purposes. However, for checking compatibility with an interface you should always look at the interface name and version, not at the application name or version. | | service_name | stringstring | | The name of this particular instance of the application.

This can be used to distinguish between different instances of a service if there is more than one instance of the same application running on the system. |

Request for the start_pick_cycle service.

Only objects within the 2D (pixel based) region of interest are considered for picking.

The available region of interests are specified in the server configuration. | | region_of_interest_3d_id | uint32uint32 | optional | Select a 3D region of interest by ID.

Only objects within the 3D (point cloud based) region of interest are considered for picking.

The available region of interests are specified in the server configuration. | | container_id | uint32uint32 | optional | Select a container by ID to be used for this cycle.

The selected container influences collision avoidance and the container layout check.

The available containers are specified in the server configuration. | | gripper_id | uint32uint32 | optional | Select a gripper by ID to be used for this cycle.

The gripper influences pick plan computation and collision avoidance.

The available grippers are specified in the server configuration. | | virtual_wall_ids | uint32uint32 | repeated | Select dynamic virtual walls by ID to be added to the collision model.

The available virtual walls are specified in the server configuration. |

Response for the start_pick_cycle service.

A tag that can be used for logging purposes.

This will be used for advanced analysis of the logs. Examples of keys: pallet_number, sku_number and shift.

Tag keys may only use letters (upper and lowercase), digits and underscores. They may not start with a digit. | | value | stringstring | | The value of the tag.

Examples of values: 23, SN05942BL and resorting_shift.

Tag values can be any string value. |

A two dimensional vector.

A three dimensional vector.

A version with major, minor and patch number.

A change to this value indicates a backwards-incompatible change to the public interface. | | minor | uint32uint32 | | The minor version.

A change to this value indicates a backwards-compatible change to the public interface. | | patch | uint32uint32 | | The patch version.

A change to this value indicates an internal change with no impact on the public interface. |

The status of a pick cycle.

This update is sent whenever the server status changes.

If used, the server will act as if no value has been specified. | | CYCLE_STATUS_WAITING | 1 | The server is waiting for recording or processing of a previous cycle to complete.

The robot should not move into view of the camera as the recording phase may start at any moment. | | CYCLE_STATUS_RECORDING | 2 | The server is still recording data.

The robot should not move into view of the camera while the server is recording data. | | CYCLE_STATUS_PROCESSING | 3 | The server has finished recording data and is computing pick plans.

The robot may now move into view of the camera. | | CYCLE_STATUS_DONE | 4 | The server is done processing data.

You can still use the pick cycle with any of the available services. |

The type of Euler (Tait-Bryan) angles and the sequence of the rotation.

If used, the server will act as if no value has been specified. | | EULER_ROTATION_FORMAT_INTRINSIC_XYZ | 1 | The angles represent intrinsic rotations in XYZ order.

See [RotationFormat::EulerIntrinsicXyz] for more information. | | EULER_ROTATION_FORMAT_INTRINSIC_XZY | 2 | The angles represent intrinsic rotations in XZY order.

See [RotationFormat::EulerIntrinsicXzy] for more information. | | EULER_ROTATION_FORMAT_INTRINSIC_YXZ | 3 | The angles represent intrinsic rotations in YXZ order.

See [RotationFormat::EulerIntrinsicYxz] for more information. | | EULER_ROTATION_FORMAT_INTRINSIC_YZX | 4 | The angles represent intrinsic rotations in YZX order.

See [RotationFormat::EulerIntrinsicYzx] for more information. | | EULER_ROTATION_FORMAT_INTRINSIC_ZXY | 5 | The angles represent intrinsic rotations in ZXY order.

See [RotationFormat::EulerIntrinsicZxy] for more information. | | EULER_ROTATION_FORMAT_INTRINSIC_ZYX | 6 | The angles represent intrinsic rotations in ZYX order.

See [RotationFormat::EulerIntrinsicZyx] for more information. | | EULER_ROTATION_FORMAT_EXTRINSIC_XYZ | 7 | The angles represent extrinsic rotations in XYZ order.

See [RotationFormat::EulerExtrinsicXyz] for more information. | | EULER_ROTATION_FORMAT_EXTRINSIC_XZY | 8 | The angles represent extrinsic rotations in XZY order.

See [RotationFormat::EulerExtrinsicXzy] for more information. | | EULER_ROTATION_FORMAT_EXTRINSIC_YXZ | 9 | The angles represent extrinsic rotations in YXZ order.

See [RotationFormat::EulerExtrinsicYxz] for more information. | | EULER_ROTATION_FORMAT_EXTRINSIC_YZX | 10 | The angles represent extrinsic rotations in YZX order.

See [RotationFormat::EulerExtrinsicYzx] for more information. | | EULER_ROTATION_FORMAT_EXTRINSIC_ZXY | 11 | The angles represent extrinsic rotations in ZXY order.

See [RotationFormat::EulerExtrinsicZxy] for more information. | | EULER_ROTATION_FORMAT_EXTRINSIC_ZYX | 12 | The angles represent extrinsic rotations in ZYX order.

See [RotationFormat::EulerExtrinsicZyx] for more information. |

High level classification of feedback: success, failed or ignored.

This struct does not include specific error codes or messages. Such information should be added to the extensible FeedbackRequest struct.

This can be used when you decide not to attempt a pick plan or a specific object from a pick plan.

You are encouraged to add code and message to your feedback to provide more information. | | FEEDBACK_FAILED | -1 | An action failed.

This could refer to an entire pick cycle, a pick plan, or a detection.

You are encouraged to add code and message to your feedback to provide more information on why the action failed. | | FEEDBACK_SUCCEEDED | 1 | An action succeeded.

This could refer to an entire pick cycle, a pick plan, or a detection. |

The format of a 3D rotation.

If used, the server will act as if no value has been specified. | | ROTATION_FORMAT_ANGLE_AXIS | 1 | The rotation is represented as an axis and an angle of rotation. | | ROTATION_FORMAT_QUATERNION | 2 | The rotation is represented as a quaternion. | | ROTATION_FORMAT_EULER_INTRINSIC_XYZ | 3 | The rotation is represented as three intrinsic rotations: around the X axis, then the Y axis and then the Z axis.

The rotations are intrinsic: this means that the second rotation is applied in the rotated reference frame of the first rotation, and the third rotation is applied in the rotated reference frame of the second rotation.

This representation is also known as (improper) Euler angles or Tait-Bryan angles.

Known robots manufacturers that use this representation: - Staubli (NOTE: uses degrees rather than radians) | | ROTATION_FORMAT_EULER_INTRINSIC_XZY | 4 | The rotation is represented as three intrinsic rotations: around the X axis, then the Z axis and then the Y axis.

The rotations are intrinsic: this means that the second rotation is applied in the rotated reference frame of the first rotation, and the third rotation is applied in the rotated reference frame of the second rotation.

This representation is also known as (improper) Euler angles or Tait-Bryan angles. | | ROTATION_FORMAT_EULER_INTRINSIC_YXZ | 5 | The rotation is represented as three intrinsic rotations: around the Y axis, then the X axis and then the Z axis.

The rotations are intrinsic: this means that the second rotation is applied in the rotated reference frame of the first rotation, and the third rotation is applied in the rotated reference frame of the second rotation.

This representation is also known as (improper) Euler angles or Tait-Bryan angles. | | ROTATION_FORMAT_EULER_INTRINSIC_YZX | 6 | The rotation is represented as three intrinsic rotations: around the Y axis, then the Z axis and then the X axis.

The rotations are intrinsic: this means that the second rotation is applied in the rotated reference frame of the first rotation, and the third rotation is applied in the rotated reference frame of the second rotation.

This representation is also known as (improper) Euler angles or Tait-Bryan angles. | | ROTATION_FORMAT_EULER_INTRINSIC_ZXY | 7 | The rotation is represented as three intrinsic rotations: around the Z axis, then the X axis and then the Y axis.

The rotations are intrinsic: this means that the second rotation is applied in the rotated reference frame of the first rotation, and the third rotation is applied in the rotated reference frame of the second rotation.

This representation is also known as (improper) Euler angles or Tait-Bryan angles. | | ROTATION_FORMAT_EULER_INTRINSIC_ZYX | 8 | The rotation is represented as three intrinsic rotations: around the Z axis, then the Y axis and then the X axis.

The rotations are intrinsic: this means that the second rotation is applied in the rotated reference frame of the first rotation, and the third rotation is applied in the rotated reference frame of the second rotation.

This representation is also known as (improper) Euler angles or Tait-Bryan angles.

Known robots models that use intrinsic Euler ZYX: - Mitsubishi Industrial Robot CR750/CR751 Series - Kawasaki (NOTE: uses degrees rather than radians) - Kuka (NOTE: uses degrees rather than radians) - Yaskawa (NOTE: uses degrees rather than radians) | | ROTATION_FORMAT_EULER_EXTRINSIC_XYZ | 9 | The rotation is represented as three extrinsic rotations: around the X axis, then the Y axis and then the Z axis.

The rotations are extrinsic: this means that the second rotation is applied in the unrotated base or world frame, and the third rotation is again applied in the unrotated base or world frame.

This representation is also known as (improper) Euler angles or Tait-Bryan angles.

Known robots models that use intrinsic Euler ZYX: - Fanuc (NOTE: Fanuc calls them W, P, and R instead of X, Y and Z, and they use degrees rather than radians). | | ROTATION_FORMAT_EULER_EXTRINSIC_XZY | 10 | The rotation is represented as three extrinsic rotations: around the X axis, then the Z axis and then the Y axis.

The rotations are extrinsic: this means that the second rotation is applied in the unrotated base or world frame, and the third rotation is again applied in the unrotated base or world frame.

This representation is also known as (improper) Euler angles or Tait-Bryan angles. | | ROTATION_FORMAT_EULER_EXTRINSIC_YXZ | 11 | The rotation is represented as three extrinsic rotations: around the Y axis, then the X axis and then the Z axis.

The rotations are extrinsic: this means that the second rotation is applied in the unrotated base or world frame, and the third rotation is again applied in the unrotated base or world frame.

This representation is also known as (improper) Euler angles or Tait-Bryan angles. | | ROTATION_FORMAT_EULER_EXTRINSIC_YZX | 12 | The rotation is represented as three extrinsic rotations: around the Y axis, then the Z axis and then the X axis.

The rotations are extrinsic: this means that the second rotation is applied in the unrotated base or world frame, and the third rotation is again applied in the unrotated base or world frame.

This representation is also known as (improper) Euler angles or Tait-Bryan angles. | | ROTATION_FORMAT_EULER_EXTRINSIC_ZXY | 13 | The rotation is represented as three extrinsic rotations: around the Z axis, then the X axis and then the Y axis.

The rotations are extrinsic: this means that the second rotation is applied in the unrotated base or world frame, and the third rotation is again applied in the unrotated base or world frame.

This representation is also known as (improper) Euler angles or Tait-Bryan angles. | | ROTATION_FORMAT_EULER_EXTRINSIC_ZYX | 14 | The rotation is represented as three extrinsic rotations: around the Z axis, then the Y axis and then the X axis.

The rotations are extrinsic: this means that the second rotation is applied in the unrotated base or world frame, and the third rotation is again applied in the unrotated base or world frame.

This representation is also known as (improper) Euler angles or Tait-Bryan angles. |

The server will trigger the camera system attached to the software and process the content of the image to detect pickable objects, filter them, and compute pick poses for each detection.

Various results and properties of the pick cycle are accessible via the other services of this interface.

This service will return immediately once the pick cycle has been created. It does not wait for camera acquisition to start or for processing to finish.

You should ensure that the robot is not in view of the camera when you call this service, and that it remains out of view until the pick cycle reaches the [CycleStatus::Processing] state. You can use the [get_cycle_status] service to wait for this event. | | ClosePickCycle | ClosePickCycleRequestClosePickCycleRequest | ClosePickCycleResponseClosePickCycleResponse | Close a pick cycle.

This call will stop processing and communication for this cycle. Future requests for this pick cycle are invalid and will return an error.

You must close a pick cycle when done to free up resources on the server. | | ListPickCycles | ListPickCyclesRequestListPickCyclesRequest | ListPickCyclesResponseListPickCyclesResponse | List all open pick cycles. | | AddTags | AddTagsRequestAddTagsRequest | AddTagsResponseAddTagsResponse | Add tags to the pick cycle.

Tags are arbitrary key-value pairs that are placed into the cycle logs. They can be used to find the pick cycle in the review tool later.

Providing useful tags can also help to gather training data.

Tag keys may only use letters (upper and lowercase), digits and underscores. They may not start with a digit.

Tag values can be anything. | | AddFeedback | AddFeedbackRequestAddFeedbackRequest | AddFeedbackResponseAddFeedbackResponse | Add feedback to a pick cycle.

You should always add feedback for attempted picks. This will allow the server to collect performance statistics. If you reject pick plans on your end, it is also useful to add feedback with the [Feedback::Ignored] value.

If you set the close_cycle parameter to true, the pick cycle will also be closed. | | GetCycleStatus | GetCycleStatusRequestGetCycleStatusRequest | GetCycleStatusResponseGetCycleStatusResponse | Get or wait for the status of a pick cycle.

This service can be used to determine when acquisition is complete so that the robot can move in front of the camera. It can also be used to indicate when a pick cycle has completely finished processing all objects in the scene.

You do not need to wait for a specific status before using other services. Other services will simply delay their response if needed. | | GetPickPlans | GetPickPlansRequestGetPickPlansRequest | GetPickPlansResponseGetPickPlansResponse | Get the next pick plans(s) of a pick cycle.

These plans indicate robot poses and gripper information to grasp a set of objects detected in the scene.

Plans are provided in order of quality, so the first plans are most likely to succeed. The first plans are also more likely to pick multiple objects when that functionality is supported by the gripper.

If you do not wish to execute the received pick plan(s), you can request the next plan(s) by calling the service again. Subsequent requests will consistently yield new pick plans until all plans have been exhausted.

You can retrieve additional information about objects referenced in the pick plan with the [get_object_info2] service. | | GetObjectInfo2 | GetObjectInfo2RequestGetObjectInfo2Request | GetObjectInfo2ResponseGetObjectInfo2Response | Get 2D information about detected objects of a pick cycle.

This service can be used to get additional information about detected objects. Because it only provides 2D information, this information is usually available before any pick plans are computed.

The detected objects returned by this service are also referenced by the same ID in the pick plans returned by [get_pick_plans]. | | GetObjectInfo3 | GetObjectInfo3RequestGetObjectInfo3Request | GetObjectInfo3ResponseGetObjectInfo3Response | Get 3D information about detected objects of a pick cycle.

This service can be used to get 3D information about detected objects, even when they do not appear in a pick plan.

Note that the reported object pose and dimensions may be different from the pose and dimensions for the same object in a pick plan. This is because the pick plan reports information relative to the (sub)surface of the object being used to pick it.

The detected objects returned by this service are also referenced by the same ID in the pick plans returned by [get_pick_plans]. | | CheckContainerLayout | CheckContainerLayoutRequestCheckContainerLayoutRequest | CheckContainerLayoutResponseCheckContainerLayoutResponse | Check the container layout for a pick cycle.

This service checks whether or not the expected container was found in the scene. | | CheckContainerEmpty | CheckContainerEmptyRequestCheckContainerEmptyRequest | CheckContainerEmptyResponseCheckContainerEmptyResponse | Check to see if the container is completely empty.

This service checks whether or not the container in the scene is completely empty using point cloud data. It is possible that the object detection algorithm misses some objects. If it is important that the container is truly empty, this service allows you to verify this more rigorously.

Note that the server needs a proper container configuration in order to run this check. |