Can I Ask a Robot to Pack My Picnic in My Own Words?

Description and use

The author evaluated SmolVLA’s ability to pack picnic items from paraphrased natural language instructions. Only the template containing "put ... in the basket" achieved moderate success; natural phrasing failed almost entirely.

Problem

Can a robot understand and execute picnic packing instructions given in natural language variations instead of a fixed template?

Outcome

Out of 160 paraphrase trials, only 52 (33%) succeeded. The original template succeeded 100% of the time. Paraphrases A and D (containing "put ... in the basket") succeeded about half the time, while B and C failed almost entirely.

Workflow

• 1. Load SmolVLA model (lerobot/smolvla_libero) on CPU.
• 2. Run baseline evaluation with original instructions for all ten items (10 episodes each).
• 3. Record success rate (89/100 episodes).
• 4. Select four items that scored 10/10 and generate four paraphrases (A–D).
• 5. Run 10 episodes per paraphrase, swapping instruction while keeping initial states and seeds.
• 6. Aggregate success counts for each paraphrase.
• 7. Analyze results and hypothesize causes of failure.

Evidence

The training sentence worked every time. The paraphrases split in two: A and D, which keep "put ... in the basket", worked about half the time (22/40 and 26/40), while B and C almost never worked (2/40 each).

Source

89 of 100 episodes succeeded.

Source

The robot is fast in simulation and slow in real life. Each SmolVLA call plans 0.5 s of motion but takes 1.15 s on this CPU.

Source