Language-guided manipulation on a low-cost robot arm

I tested GPT-6 Astra with xhigh reasoning to control my LeRobot SO101 for a real-world pickup using a third-person RGB camera.

The result looks good. Note that I deliberately placed the pen where a straightforward pick-and-place approach wouldn't work: the arm had to bend back to reach it.

My prompts

1. “We have connected a camera (for a third-person view) and a LeRobot SO101 robotic arm for control. Please control the robotic arm to pick up the red pen and hold it. You need to figure out for yourself how to adjust the robotic arm's pose.”
2. “Please record a video while operating.”
3. “Did you record a video of our entire process? Give me a final MP4.” (after I saw the robot complete the task)

What it did

It automatically reused some of my existing code for motor control, joint coordinate conversion, configuration loading, camera capture, and robot kinematics. Similar code and robot models are available online. It wrote new motion and recording scripts, adjusted its approach from camera feedback, and successfully picked up and held the pen.

The obvious limitation

Speed: 22 minutes for one pickup. Slow, approximately quasi-static motion reduces inertial effects, but contact, friction, and grasp stability still matter.

Each attempt returns camera feedback, allowing the agent to inspect the outcome and revise its next command. The successful pickup does not, by itself, reveal how accurately the model estimated the scene geometry or physical properties.

Depth estimation might make this loop more efficient, but I have not tested that comparison.

Where the time goes

Twenty-two minutes gives the robot plenty of chances to stop, look, and correct itself. I would log how much of that time goes into model calls, motion, and waiting for feedback, then repeat the task with the pen in different positions. The interesting result would be a shorter loop that keeps the same ability to recover from mistakes.

What I would compare

The reused control and kinematics code is part of the experimental setup, not an incidental detail. I would keep those tools fixed when comparing agents, and document which geometric quantities were available before the first action. Otherwise, differences in the supplied software could be mistaken for differences in the models' manipulation ability.

A useful baseline would execute a fixed pickup sequence using the same controller. I would compare it with feedback-guided revisions across repeated pen placements, recording success, recovery attempts, and wall-clock time. That comparison would test whether the additional interaction improves reliability enough to justify its latency.

My original experiment on X.

@misc{dou2026so101pickup,
  author = {Dou, Zhiyang},
  title = {Language-guided manipulation on a low-cost robot arm},
  year = {2026},
  month = sep,
  howpublished = {Research Notes},
  url = {https://frank-zy-dou.github.io/blog/so101-pen-pickup/}
}