Physical Prompting: LLM-Style In-Context Learning For Robots
GEN-1.5 is built as a single large multimodal model that ingests video, sensor, language, and proprioceptive input and outputs action trajectories at 100 Hz, holding roughly 30 seconds of that stream in working memory at once [1]. A physical prompt - a 3-12 second clip of a human using handheld grippers, or a rollout recorded by the robot itself - gets dropped into that context window the way a few-shot example gets pasted into an LLM prompt, and the robot attempts the task immediately, with no gradient update at all [1]. When Generalist does apply light fine-tuning instead, the adaptation is tiny: 10 gradient steps on about 1-5 minutes of data (roughly 10-50 demonstrations) changed the model's weights by less than 0.15%, a fraction of what prior robot-adaptation pipelines needed, which could run to tens of thousands of gradient steps [3]. That light touch is enough to move average success across ten held-out tasks from 59% (+-10%) one-shot to 83% (+-9%) [1][2]. Online reaction has repeatedly reached for the same comparison - commentary on X and Reddit independently calls it a 'GPT-2 moment for robotics,' the idea that scale plus the right data structure produces in-context learning nobody explicitly programmed.



