Code as Controller: Why These Agents Never Saw a Single Pixel
The most counterintuitive part of this experiment is the interface. These agents never looked at the screen. They received structured text and code describing the game state, then acted by writing and running short JavaScript programs through Mineflayer - instructions like walk to these coordinates and mine every iron ore within 16 blocks - and read back text results to decide their next move[1]. There was no vision model, no pixel parsing, no screenshot loop. Researchers could watch a live dashboard and had the ability to pause or stop the run at any point, but across 27 hours and 14 minutes they never needed to[2]. That distinction matters: the agents were essentially programming their way through Minecraft rather than playing it in any sense a human would recognize, which reframes this less as a gaming stunt and more as a stress test of autonomous code generation sustained over a full day.




