Webcam to Humanoid: Four Things the Research Got Wrong

Before writing the webcam-to-humanoid mirror in RoboLLM I did the research pass: which robot, which landmark model, which retargeting formulas. Most of it held. Four things did not survive contact with a desk, a webcam and a ROS 2 subscriber — and each one is the kind of fact you only learn by measuring.

1. MoveIt ships no humanoid

moveit_resources is Panda, Fanuc and a PR2 that is description-only: a 75-line stub SRDF, no head group, <test_depend>. If you want an apt-installable Jazzy humanoid whose MoveIt config already has arm_l, arm_r and head groups, there is exactly one: the ROBOTIS FFW "AI Worker" (ffw_bg2_rev4_follower, Apache-2.0). Semi-humanoid, no legs. That single fact decided the robot. (For the later full-body Talos work I had to hand-build a MoveIt config — no legged humanoid has one for Jazzy at all.)

2. Your hips are invisible

Every retargeting paper assumes a full-body view. Measured visibility of MediaPipe Pose landmarks with me seated at a desk, webcam on the monitor:

The torso "up" vector in the research is shoulder-mid → hip-mid. At a desk it is simply not there. The design changed: torso lean is dropped, the upper-body frame is built from the shoulder line alone, and everything is gated on `visibility ≥ 0.5` — TF is never published for a landmark below the gate, and the robot holds its last pose rather than chasing a hallucinated wrist. This is also why the deadman is a /mirror_enable service instead of a hand gesture: a gesture needs a hand landmark model that has no visibility field.

3. cv2.flip swaps the left/right labels

Hand-teleop demos flip the camera image so it behaves like a mirror, and it is correct there. Do it before Pose inference and the landmark labels swap sides. Measured separation on the same frames:

|flip.LEFT − (1 − raw.RIGHT)| = 0.018 … 0.022   ← content-following
|flip.LEFT − (1 − raw.LEFT)|  = 0.445 … 0.670   ← side-following

20–38× apart — unambiguous. So inference runs on the raw frame, and the only flip in the pipeline is the one that draws the finished preview picture. "Mirror mode" (my left arm drives the robot's right, as if it is facing me) is then a deliberate mapping in the retargeter, not an accident of image processing.

4. Never compute a speed limit from subscriber arrival times

The rate-limit test originally measured joint speed as delta / arrival_dt between consecutive messages in a ROS 2 subscriber. Arrival times jitter — two commands published 20 ms apart can arrive 7 ms apart — and the test reported 3× the true speed, failing a limiter that was actually working. Two fixes, both kept:

• publish the command period with the command, and
• assert the timing-free invariant: consecutive commands never differ by more than max_joint_speed / rate_hz.

The slew_hits counter settling at a constant (141 on this box) and never growing is now the fingerprint of a healthy limiter.

What verified means here

• Retargeted joint angles match MoveIt /compute_fk to 0.0000° on the acceptance sweep.
• 50 Hz control tick never blocks on the ~31 ms PoseLandmarker inference (own thread).
• Bring-up, tracking, mirror and the rate limiter each have an acceptance script; the live session is a gate, not a vibe.

The research was not wrong so much as unconditioned — it described a lab with a full-body camera and a robot that exists in a paper. The desk, the webcam and the subscriber are the conditions, and they are where the engineering actually lives.

Code and the full TECHNICAL notes: github.com/santapong/RoboLLM → examples/humanoid_mirror.