Add lab desk objects, AprilTag bottle picking, and clean up lab_sim by davetcoleman · Pull Request #512 · PickNikRobotics/moveit_pro_example_ws

davetcoleman · 2026-02-14T08:52:11Z

Summary

Add realistic lab environment objects to MuJoCo scene: desk lamp, laptop with screen/keyboard textures, potted plant, alligator trophy statue on shelf, test tube rack with 6 colored tubes, and 4 manipulable medicine bottles
Add new lab_sim_behaviors package with ComputeTrayPlacePositionsUsingAprilTags behavior plugin for computing tray grid placement positions from AprilTag detections
Add "Pick All Bottles with AprilTags" objective: detects AprilTag-labeled bottles, picks them, and places them in a tray using vision-guided grasping with force-controlled lowering
Rework constrained pick-and-place subtree
Clean up lab_sim objectives: rename files for consistency, remove deprecated objectives (_scan_scene, load_mesh_as_*_pointcloud, pick_and_place_example, scan_multiple_views)
Simplify ur_waypoints.yaml and add new waypoints
Add CLAUDE.md with workspace instructions (MuJoCo keyframes, objective MetadataFields CI requirements)

Dependencies

Requires picknik_accessories#feature/lab-sim-desk-objects for test tube stand repositioning
Related: PR #518 — lab_sim_behaviors package (included here as shared commit)

Test plan

Launch lab_sim in MuJoCo and verify all objects render correctly on the desk
Verify test tubes and medicine bottles are manipulable (have freejoints)
Run "Pick All Bottles with AprilTags" objective end-to-end in simulation
Verify constrained pick-and-place still works after rework
Confirm no objects are floating or falling through the table

🤖 Generated with Claude Code

src/lab_sim/description/scene.xml

davetcoleman · 2026-02-17T19:32:16Z

src/picknik_accessories

@MikeWrock i dont understand why some things live in this picknik_accessories, and others in the example_ws. Can we get rid of picknik_accessories? What's the point? We have GitLFS already.

I'm not sure what things are living in example_ws, but the idea is to have one repo that can hold all our reusable assets, and if you have config specific assets that cannot justifiably be put in the accessories package then it can live in the config package. Due to https://github.com/PickNikRobotics/moveit_pro/issues/13842 we have to do a little CMake hacking so that we can re-use these assets, otherwise they would have to be duplicated within each package that used them and copied out for re-use

But who is reusing these assets? It appears these assets are just used in the example_ws

I propose I have Claude merge these repos together. Maybe @nbbrooks has thoughts.

I've always liked monorepos but we sacrifice all portability for our digital assets. Any mujoco scenes or objects we want to use in more than one package requires duplicating code and files, which then results in bit rot and missed fixes/updates or divergent configurations. Whats the advantage we gain by having the files spread out across multiple packages in the example_ws?

picknik_accessories is where some of our tuned mujoco assets like the robotiq_gripper live

I'd be open to keeping those in accessories for reuse and moving the "example_ws world objects" into example_ws, but we will still have a submodule

davetcoleman · 2026-02-18T05:23:11Z

src/lab_sim/description/scene.xml

+        contype="0"
+        conaffinity="0"
+      />
+      <!-- Bezel removed - it occludes the screen texture -->


remove this commented out section

MikeWrock · 2026-02-18T17:01:28Z

This is failing CI for both integration testing and pre-commit. Please confirm the objectives in lab_sim still run successfully and that the new assets don't introduce any unexpected behaviors (ungraspable objects due to collisions, collisions with new objects due to waypoints, confusing input to ML prompts, poor friction coefficient for new objects). As our flagship config, I wouldn't want to introduce a whole bunch of unpickable objects or add complexity that would make the product look poorly functional.

Please also monitor the change in CPU usage, there are a lot of additional free joints that would significantly increase the CPU load and I want to make sure it's not a huge jump. I noticed some of the free joints (like the alligator) are not even reachable, and would be better as fixed joints.

When this PR is tested and ready for review, please add a story to track your work on the project board

…lTags - New lab_sim_behaviors package providing custom Behavior plugins for lab_sim - ComputeTrayPlacePositionsUsingAprilTags: computes grid-based place positions relative to a tray AprilTag for bottle placement workflows - Fix tray center direction and reverse row fill order - Register the behavior loader plugin in lab_sim config.yaml - Add lab_sim_behaviors exec_depend to lab_sim package.xml Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

… up lab_sim Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

rlpratt12 · 2026-02-27T20:43:19Z

This is for https://github.com/PickNikRobotics/moveit_pro/issues/17169 (priority for sprint 133)

nbbrooks

I still need to test this locally.

nbbrooks · 2026-03-02T20:22:42Z

...lab_sim_behaviors/include/lab_sim_behaviors/compute_tray_place_positions_using_apriltags.hpp

+ * | place_positions         | output    | std::vector<geometry_msgs::msg::PoseStamped>        |
+ */
+class ComputeTrayPlacePositionsUsingAprilTags final
+  : public moveit_pro::behaviors::SharedResourcesNode<BT::SyncActionNode>


I won't block this, but I would have recommended using AddURDF to localize a URDF model of the tray+AprilTag instead of a custom behavior.

This would be a good example of using that for a grasping workflow. Could you make a followup story for Claude to do this?

davetcoleman mentioned this pull request Feb 14, 2026

Update lab desk objects, textures, and positions PickNikRobotics/picknik_accessories#42

Open

5 tasks

davetcoleman force-pushed the feature/lab-sim-desk-objects branch 3 times, most recently from ff401c8 to 4cee822 Compare February 14, 2026 19:32

davetcoleman requested a review from MikeWrock February 14, 2026 19:32

davetcoleman commented Feb 14, 2026

View reviewed changes

src/lab_sim/description/scene.xml Show resolved Hide resolved

davetcoleman force-pushed the feature/lab-sim-desk-objects branch 2 times, most recently from 17e3dc9 to f06197a Compare February 17, 2026 19:28

davetcoleman added this to the 9.1.0 milestone Feb 17, 2026

davetcoleman commented Feb 17, 2026

View reviewed changes

davetcoleman force-pushed the feature/lab-sim-desk-objects branch 7 times, most recently from b621d9c to 6c53a30 Compare February 18, 2026 05:22

davetcoleman removed the request for review from MikeWrock February 18, 2026 05:22

davetcoleman commented Feb 18, 2026

View reviewed changes

MikeWrock assigned davetcoleman Feb 18, 2026

davetcoleman force-pushed the feature/lab-sim-desk-objects branch 7 times, most recently from 4ec0c55 to b58e16f Compare February 23, 2026 07:09

davetcoleman force-pushed the feature/lab-sim-desk-objects branch from 2e76ab2 to a3d0b64 Compare February 25, 2026 01:02

davetcoleman force-pushed the feature/lab-sim-desk-objects branch 7 times, most recently from e670555 to c86f75c Compare February 25, 2026 18:51

davetcoleman changed the title ~~Add lab desk objects to MuJoCo scene~~ Add lab desk objects, AprilTag bottle picking, and clean up lab_sim Feb 25, 2026

davetcoleman force-pushed the feature/lab-sim-desk-objects branch from c86f75c to 95cabe9 Compare February 26, 2026 17:03

Add lab desk objects, adjust collision wall/tray positions, and clean…

e35c146

… up lab_sim Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

davetcoleman force-pushed the feature/lab-sim-desk-objects branch from 95cabe9 to e35c146 Compare February 27, 2026 05:31

nbbrooks reviewed Mar 2, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add lab desk objects, AprilTag bottle picking, and clean up lab_sim#512

Add lab desk objects, AprilTag bottle picking, and clean up lab_sim#512
davetcoleman wants to merge 2 commits intomainfrom
feature/lab-sim-desk-objects

davetcoleman commented Feb 14, 2026 •

edited

Loading

Uh oh!

Uh oh!

davetcoleman Feb 17, 2026

Uh oh!

MikeWrock Feb 17, 2026

Uh oh!

davetcoleman Feb 17, 2026

Uh oh!

MikeWrock Feb 18, 2026

Uh oh!

nbbrooks Feb 19, 2026

Uh oh!

nbbrooks Feb 19, 2026

Uh oh!

davetcoleman Feb 18, 2026

Uh oh!

MikeWrock commented Feb 18, 2026

Uh oh!

rlpratt12 commented Feb 27, 2026

Uh oh!

nbbrooks left a comment

Uh oh!

nbbrooks Mar 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

davetcoleman commented Feb 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Dependencies

Test plan

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MikeWrock commented Feb 18, 2026

Uh oh!

rlpratt12 commented Feb 27, 2026

Uh oh!

nbbrooks left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

davetcoleman commented Feb 14, 2026 •

edited

Loading