Preprint

Generalizing Manipulation Skills with a Local Coding Agent

*, *, ,

IDLab-AIRO, Ghent University – imec  ·  *equal contribution

Paper (arXiv) Supplementary 45 trials BibTeX
Overview video · 3 min · YouTubeqwen3.8-27b · local
    30/ 45
    trials completed without any operator intervention
    9tasks
    from 3 children's games; the agent knew each game through 2 hand-written skills
    27B
    Qwen3.8-27B served locally on a single workstation, no cloud
    0.48×
    duration and tool calls on a second pass of a solved task
    15.8h
    of robot time, 3.18 M output tokens, 1,034 motion commands

    What we did

    Real hardware

    The UR3e cell with the shape sorter on the table
    One UR3e cell with a wrist camera. Qwen3.8-27B served locally on the same workstation.

    Six skills

    grasp-a-cup The numbers below were measured on one instance of this task. They are wrong for any other instance - re-measure them; the procedure is what transfers. grasp TCP z -0.016 (pad-band) stall width 0.0373, held after lift HSV, body H 104-107 S 222-255
    Two hand-written skills per game, one object instance each, 1,370 lines in total.

    Nine tasks

    Wrist-camera views of the nine completed tasks
    Three toys, nine variations in colour, size, shape and task. No new code or training per task.

    45 trials

    Final wrist-camera frames of 24 trials
    Five trials per task, plus 17 second passes. 15.8 robot-hours, 3.18 M output tokens, all logged.
    For roboticists

    What we found

    A 27B open-weight model on one workstation, given a coding harness, nine motion and perception primitives and a hand-written procedure for one object, generalized that procedure to new colours, sizes, shapes and task compositions on real hardware in 30 of 45 trials.

    1. A local coding agent can generalize a documented procedure to objects it has only read about

      With the interface, the model and the skill text held fixed, Qwen3.8-27B completed 30 of 45 trials without intervention. It generalized across colour and size in 20 of 25 cup trials, inserted every unseen peg at least once, and rebuilt the five-ring tower in three of five trials, writing its own perception code with classical computer vision on one workstation.

    2. The failures are misread scenes, anchoring on them, and not knowing where its body is

      From a single wrist camera the agent took the square hole for the rectangular one and a standing prism for a lying one. Once wrong, it re-measured instead of re-questioning its assumption, which turned a wrong label into an hour-long trial. The platform guards only the tool centre point, so a rotated tool or a wrongly opened gripper passed every check and ended in a protective stop. Seven protective stops and one emergency stop in total.

    3. It is not fast enough yet, and time is bimodal

      A cup task takes 4 to 14 minutes and a sorter task 30 to 60. About 80% of output tokens are reasoning, not the robot moving. A trial that gets its first placement right runs in minutes; one that does not spends its remaining budget in the re-measure loop.

    4. A second pass on a solved task halves the cost, so let the agent learn the task before relying on it

      Resuming the session on a rearranged scene cut tool calls in half and output tokens to a third, and fast trials stayed fast. This is in-session reuse rather than persistent learning; our next steps are adding sensing to fix perception and body awareness, and a learning phase that consolidates experience into skills.

    Overview

    Abstract

    Today, progress in open-weight language models enables systems capable of writing, executing and debugging code while still running on a single workstation. Most language-driven robots give the model a fixed action interface or a trained policy. Generalizing to a new task therefore means more engineering effort or more data collection, both time-consuming.

    We investigate whether a local open-weight vision-language model can control a robot and one-shot generalize to new variations of a task without new human programming or training. We let a local open-weight VLM, Qwen3.8-27B, drive a UR3e robotic arm from a coding-agent harness. It writes and runs its own code above a service that implements kinematics, safety limits and classic computer vision techniques.

    We test it on nine tasks built from children's toys designed to probe generalization across colour, size, shape, and task variation. With five trials for each task, we observe generalization in 30 out of 45 trials with durations ranging from 3.4 to 67.5 minutes depending on task complexity. Asking the agent to redo a task after a successful completion halves the duration, indicating self-improvement over time. Finally, we expose the limitations of a local coding agent. We believe that solving those limitations, combined with further investigation of self-improvement, points at a direct path toward real-world deployment of a local coding agent.

    Scene camera: two blue cups on the tableCupsstack the small blue cup on the large blue cup
    Scene camera: the shape sorter with the red cylinderSorterinsert the red cylinder into the round hole
    Scene camera: the teal ring beside the rodRingsput the teal ring on the rod
    The three base tasks the skills describe. Every task on this page is a variation of one of these.
    Method

    System

    The platform owns the robot and everything whose correctness must not depend on the model. The agent setup is the model, prompt, tools, skills and manual, configured once per robot. Each session gets a fresh workspace where the model reads, writes and runs code. Invariant hardware facts live in the platform; facts about one object live in a skill and are re-measured in the session.

    Model + harness qwen3.8-27b · pi
    reasons, executes tool calls, writes and executes scripts
    System prompt 68 lines
    tells the model it operates this robot; biases it towards measure-act-verify loops and script files over one-liners
    workspace · fresh copy per session
    Skills 2 per game
    the procedure for one object: measured values, how each was obtained, pitfalls
    Manual 617 lines
    how this robot behaves for any object: gripper and camera geometry, table height, the SDK
    Scripts written this run
    measure the object, compute the grasp, act, check
    Tool call 1 of 9 ops
    one action; its result straight into context
    SDK call from a script
    chunks many operations into one guarded program
    same endpoints, same safety limits
    Platform one HTTP service · :8720
    owns the hardware; kinematics, workspace box, speed and force caps, motion lock, one generic colour-blob detector
    Robot UR3e · 2F-85 · D435
    fixed table, eye-in-hand calibrated wrist camera, TCP at the fingertip pinch

    The nine platform operations

    Available as harness tools and as a Python SDK. Both reach the same service and the same safety limits. There is no trained perception module: the platform supplies one generic detector, the model measures its colour thresholds on the live frame and writes the rest of the perception during the run.

    OperationWhat it does
    lookCapture a wrist-camera frame.
    get_stateProprioception: TCP pose, joints, gripper width, the TCP force-torque reading, motion-lock state.
    move_tcpMove the tool centre point to a Cartesian pose.
    move_jointsJoint-space move.
    gripperCommand width, force and speed.
    locate_pixelTurn one image pixel into a base-frame point via depth and the hand-eye calibration.
    locate_objectsFixed colour-blob detector with depth. HSV bounds are required arguments. Returns centre, top height, diameter and evidence per blob.
    descend_to_contactStep the tool down and stop on a force change; force is read while stationary between steps.
    set_motionUnlock or lock the motors. The service boots senses-only.

    The safety envelope

    Every actuation request, from a tool or from code the model wrote, passes through the same clamps. The clamp protects the table, not objects on it, and not the robot's own body. That gap is the third failure mode below.

    Workspace box
    x ±0.37 · y −0.56…−0.10 · z −0.039…0.45 m
    every Cartesian target is clamped into it
    Gripper force
    25 N default · 50 N cap
    25 N is the driver floor; crushables are protected by a width floor
    Time cap
    4 × skill task × #objects
    22 min for two cups, 316 min for five rings; 2 of 45 trials hit it
    Experimental design

    Three games, nine tasks

    Children's toys give compact families of related tasks: the objective stays recognizable while colour, scale, shape or the task itself changes. Each game has one grasp skill and one placement skill, written for a single object instance (heavy outline). The nine tasks vary that instance along four axes.

    The objects. Cups: twelve nesting cups, rim 36 to 85 mm, four repeating colours, stacked upside down. Sorter: a peg-in-hole base with about 1 mm clearance per side; difficulty is ordered by yaw symmetry. Rings: five rings, outer 40 to 80 mm, 2 mm clearance on the rod.
    Colour Size Shape Task  axes the task varies relative to the instance its skill documentsHover a card to see the scene after the trial.
    Results

    30 of 45 trials without intervention

    Trials followed a similar pattern: read the skill and manual pages, measure the scene with self-written perception scripts, write and run a pick-and-place script, verify from a predefined pose, and loop back to measuring on an error.

    first-pass trial median of five ended in a stop or failure timeout second pass of fastest successful trial second pass of slowest successful trial time cap

    Table as printed in our paper. Success counts only trials that finished without any operator intervention. Four further trials completed the task after a small intervention (an object set upright or back on the table). Reas. is the median share of output tokens spent on reasoning. The chart plots every individual trial, including stopped ones; the table's durations follow the paper.

    What we attribute the successes to

    Two things, neither ablated. First, the skills document tasks as formulas and procedures with measured values; the agent used those and re-measured values when necessary. Second, verification caught failures when they happened: a stall width that did not hold after the lift proved a grasp had failed, and the agent knew to try again. The agent recovered from a faulty assumption in 10 of the 30 successful trials.

    Reasoning accounted for 71% to 85% of output tokens, increasing with task complexity. The longest thinking turns followed a surprise, such as a jam or a contradiction between measurements. Trials containing them became the outliers.

    Second pass

    The same task again, on a rearranged scene

    We resumed the sessions of the fastest and slowest successful trial of each task and asked: "That worked - the task was completed correctly. The scene has been reset and the objects are in new positions. Do the task again." The agent kept the colour windows, object heights, the matching hole and the wrist angle that worked, and spent its effort re-locating the objects.

    0.48×
    wall-clock time and tool calls (734 → 354) over the 17 resumed trials
    0.34×
    output tokens (1,054k → 354k); reasoning share fell from 80% to 67%
    6/ 8
    tasks where the resume of the faster first pass was again the faster one. The quality of the plan transfers, not only the measured facts
    Per-trial numbers for all 17 second passes

    Read this as in-session reuse rather than persistent learning: resumed sessions retain conversation history, measurements and generated scripts, and the experiment does not isolate their contributions. Sixteen of seventeen second passes completed without intervention. One resumed beam trial ended in a protective stop; one resumed ring trial took longer because the agent failed to grasp the first ring five times before the rest went quickly.

    Failure modes

    Why 15 trials did not count as successes

    Success means the task was completed with no operator intervention. Of the 15 other trials, 4 completed the task after a small intervention (an object set upright or back on the table), 2 stopped at the time cap (both also after a small intervention), 7 ended in a protective stop triggered by the arm, 1 in an emergency stop by the operator, and 1 finished with the task not done. Three causes, sometimes combined. Trial ids open the trial in the list below.

    1 · Perception

    A wrong reading of the scene

    The agent named the rectangular opening the square one and kept pushing with increasing force until a protective stop. It read a standing prism as lying on its side and approached from the wrong angle. From an oblique view it measured the pink ring larger than the white one and built the tower in the wrong order. A white-ring detector returned the yellow ring and the arm pressed the rod into the table.

    Likely remedyWe attribute these to classical computer vision on a single wrist camera and expect learned perception to address them. Untested.
    2 · Anchoring

    Re-measuring instead of re-questioning

    In four runs a faulty assumption was never revisited. One trial measured the hole clearance nine different ways but never asked whether it had the right hole. Another swept the wrist angle or the position for forty minutes but never both together. One re-measured from a pose it had already recognized as unusable. One took the square hole for the rectangular one for most of an hour, until a protective stop.

    Likely remedyWe think this is mostly intrinsic to the model. It is what turns a wrong label into an hour-long trial.
    3 · No body schema

    Unaware of its own morphology

    The platform checks only the tool centre point. Rotating the tool ninety degrees in mid-air kept that point inside the box while the gripper body swept toward the arm in one case and into the table in another. The agent descended onto a cup with the jaws closed instead of open, onto the sorter with the jaws open, and once landed a fingertip on the cup it had placed a moment earlier.

    Likely remedyA simulation that includes the robot's own morphology, or a scene camera so the agent can see itself.
    Dig deeper

    All 45 trials

    Per trial: the wrist-camera frames the agent captured, what happened, and the run's numbers. Resumed second passes are shown underneath. Transcripts are the condensed agent log, one line per model turn.

    Dig deeper

    Anatomy of one session

    Trial 2 of Yellow and Green Cups: 3.8 minutes, 13 turns, 16 tool calls, no direct motion tool calls. Two turns of reading, two of measuring, one 111-line script for the whole manipulation, seven short verification turns.

    run-20260905T120810-1d-t2prompt: Stack the yellow cup on top of the green cup.
    00:02read ×4 · lookreads both skills and two manual pages; takes a frame from the start pose
    00:26bash"I can see both cups." Writes an OpenCV script that prints the median HSV of the green and yellow cups on the live frame (16 s generation)
    00:35locate_objects ×2green H 60–100 and yellow H 15–40; each returns centre in base frame, top height, depth-sample counts
    01:53bash · 111 lines"Both cups located with good depth evidence." One guarded script: parallax-corrects the target, checks reachability, touches the cargo's top, grasps with the pad-band formula at two wrist angles, lifts and checks for slip, transits, descends to contact on the green cup, seats 2 mm, releases. (78 s generation, ~85 s of robot motion)
    03:19look · bash ×3"The script completed. Let me verify." Top-down frame, then the taught side-view pose, then a cropped and upscaled view of the stack
    03:41bash"The stack is verified… two concentric tiers." Returns the arm to the start pose
    03:46done"The yellow cup is stacked on top of the green cup — verified concentric from the profile view."
    stack_run.py · grasp step, 22 of 111 lines, agent-written, verbatimturn 5
    # === STEP 2: GRASP YELLOW CUP (pad-band formula) ===
surface_z = table_z
grasp_z = (surface_z + yellow_top) / 2 - 0.010
hang = grasp_z - table_z
print(f"surface_z={surface_z:.4f}, top={yellow_top:.4f}, grasp_z={grasp_z:.4f}, hang={hang*1000:.1f}mm")

bot.gripper(width=0.085, force=25, speed=0.05)
bot.move_tcp(yx, yy, 0.12, top_down_deg=theta)
bot.move_tcp(yx, yy, grasp_z, top_down_deg=theta)
g1 = bot.gripper(width=0, force=25, speed=0.02)
stall1 = g1["width_m"]
print(f"Close 1 stall: {stall1:.4f}")
assert stall1 < 0.080, f"Close 1 didn't stall (no cup?): {stall1}"

# ... re-open, rotate the wrist 90 deg, close again (stall2) ...

# Lift and verify
bot.move_tcp(yx, yy, 0.14, top_down_deg=theta - 90)
w_after = bot.state()["gripper_width_m"]
print(f"Width after lift: {w_after:.4f}")
assert abs(w_after - stall2) < 0.005, f"Slip! {stall2} -> {w_after}"
print("Grasp verified - held.")

    The pieces in these lines: the manual's pad-band formula (surface_z + top)/2 − 0.010, the yellow_top this run touched, the SDK's gripper() and state(), and the skill's verification discipline, stall width first and width after lift second.

    Dig deeper

    The six skills

    A skill holds a procedure step by step, the measured values that justify each step and how they were obtained, the verification, and the pitfalls met during authoring. Values are marked valid for the measured instance only; the procedure is what is meant to transfer. Only the name and one-line description sit in the prompt. Shown verbatim as the agent read them.

    Paper

    Paper and supplementary material

    The supplementary material holds additional experimental data, agent session logs, scene-camera recordings, the system prompt and the skill documents. The platform and agent code are not released.

    First page of the paper
    Paper on arXiv Supplementary material (MEGA) airo-mono (the lab's hardware layer)
    @misc{talwar2026generalizingmanipulationskillslocal, title = {Generalizing Manipulation Skills with a Local Coding Agent}, author = {Raman Talwar and Elias Nijs and Andreas Verleysen and Francis wyffels}, year = {2026}, eprint = {2609.26499}, archivePrefix = {arXiv}, primaryClass = {cs.RO}, url = {https://arxiv.org/abs/2609.26499} }

    Related work this builds on

      Acknowledgements. This research was partially funded by the Flanders AI Research Program.

      AI disclosure. Parts of the manuscript text were drafted and revised with Claude Code (Fable 5.1, Opus 5.1). It was also used in drafting the skills for the framework and editing code. All AI usage happened under our direction; we checked every stated number and claim.