Old Interface

11 min

Grasp Proposals

3 proposal methods:

plan-update: Each object is provided as an individual update, along with all the proposals for that object.

all-plans-update: A single update with all the objects is provided.

all-plans-response: All objects are provided at once in the response.

Request Extensions

These extensions are included in the initial request. Essential extensions to the processing pipeline, and we need to have them at the start of the cycle. Non-essential extensions simply add extra information to the logs, or the request additional information in the form of updates.

Essential Extensions

These extensions are necessary up front, because they influence the processing pipeline.

[x]RegionOfInterest2d

[x]RegionOfInterest3d

[x]SelectGripper

~~[ ]~~SelectMultiPickProfile

[x]AddSimulatedWalls

[x]SelectContainer

Non-Essential Extensions

These extensions provide or request extra information, but they do not meaningfully influence the pipeline.

[x]EnableContainerLayoutCheck

[x]AddTags

[ ]RequestObjectCount

[ ]SendContainerWalls

[ ]RequestObjectBoundingBox (2d bounding boxes for client visualization)

[ ]EnablePalletFillLevel

[ ]RequestRgbImage (color image for client visualization)

[ ]RequestPointCloud (point cloud for client collision check)

Additional Updates

These updates are toggled by the Non-essential extensions

PalletFillLevel

Sanity Check for empty pallets

Confirm that the pallet is indeed empty when the network fails to detect objects

This is done using 3d data and the container definition.

ObjectCount

It's all in the name.

ContainerLayoutCheck

Check to make sure our detected container segments match the selected layout configuration.

ImageUpdate

Contains the rgb image or point cloud.

Config Extensions

These extensions are put in the configuration, and they augment the grasp proposals that we send. This is a paradigm that we should avoid, because it means that small configuration mistakes can result in breaking API changes.

AddGripperName

AddSegmentationSize

AddSuctionGroups

Add3dBoundingBox

AddSubmaskCoverage

New Interface

Config Extensions are moved to the pick-plan and detection-info requests. The config will no longer influence the contents of API calls.

Mission Critical Extensions Extensions will remain in place, there is no other sensible way to handle them.

Non-essential will be moved to separate requests. The PickCycleHandle exposes read access to the full Scene struct, so it is very easy to add things like ObjectCount without making any changes to our pipeline. We just add the API call, and then add a single function to query that particular value from the Scene.

Open Questions

Do we need PalletFillLevel?

We can add it relatively easily by adding it as a field in Scene.

However, it's really just a band-aid for detection false-negatives.

Should we include descriptors and segmentation-size directly in our PickPlans?

The current interface would require a separate request/response to query the descriptors for a PickObject.

This may be inconvenient, in which case we should consider adding them to PickObject as well.

Should Tags be a separate request?

There is no practical reason to include them in the first request, because they can appear anywhere in the logs.

Should the client be able to select their own CycleId when starting the request

It would allow users to simply hard-code 0 if they only need one request open at a time.

Minimalist Example

StartPickCycle: [region_of_interest_id]
PickCycleStarted: []

RequestPickPlans: []
PickPlan: [pose, objects]

Feedback: [success!]
FeedbackReceived: []

sequenceDiagram
    client -->> server: StartPickCycle
    server ->> client: PickCycleStarted

    client -->> server: RequestPickPlans
    server ->> client: PickPlan

    client -->> server: Feedback
    server ->> client: FeedbackReceived

Maximalist Example

StartPickCycle: [region_of_interest_2d, region_of_interest_3d, container_id, gripper_id, simulated_wall_ids]
PickCycleStarted: []

RequestContainerLayoutCheck: []
ContainerLayoutCheckResult: [success!]

RequestPickPlans: [n-plans]
PickPlan:
    - plan_1:
        - pose: ...
        - objects: [object_0, object_1]
    - ...
    - plan_n:
        - pose: ...
        - objects: [object_3, object_4]


RequestDetectionInfo:
    detection_ids: [object_0, object_1]
    requested_info: [descriptors, segmentation_size]
DetectionInfoResponse:
    - object_0: [descriptors, segmentation_size]
    - object_1: [descriptors, segmentation_size]

Feedback: [plan_id, object_id, failure_reason]
FeedbackReceived: []

sequenceDiagram
    client -->> server: StartPickCycle
    server ->> client: PickCycleStarted

    client -->> server: RequestContainerLayoutCheck
    server ->> client: ContainerLayoutCheckResult

    client -->> server: RequestPickPlans
    server ->> client: PickPlan

    client -->> server: RequestDetectionInfo
    server ->> client: DetectionInfoResponse

    client -->> server: Feedback
    server ->> client: FeedbackReceived