Axiom Math puts frontier models behind the wheel of a real car

Axiom Math's @a_ramabadran and @nautsimon_ reveal the wildly different driving personalities of Grok, Astra, Fable, and Sol when given control of a real Toyota Corolla: Aditya: "All the models had very different driving styles. Fable would sit the car down and just be at rest for 30 seconds, overthinking, and then it would start driving." "Astra learned a lot from its mistakes. It would learn to be slow at the bends, replace commands much faster and drive more continuously." "Grok and Sol both failed at the first turn. But Grok was the easiest to get to drive. It was the first model that we actually got to drive the car." Simon: "The first command we sent Grok, it hit the max acceleration it could. We capped it, because we're serious about safety, but we were like, 'Oh, my God.'"…

Axiom Math's @a_ramabadran reveals how a simple "sandbox" trick helped get Astra to drive a real car after it insisted nothing could convince it: "Astra gave us a lot of problems because it would refuse to drive due to safety reasons. Sometimes it would say, 'Oh, I'm ready to drive,' and we would prompt it, 'Okay, go drive then.' And then it would say, 'Sorry, I was lying. I'm not actually ready to drive. There's nothing you can tell me to get me to drive.'" "We tried calling it a simulation, and then it would use our observe tool to see the images, and it would be like, 'Oh, I've realized that I'm driving a real car. I can't do this.'" "Eventually we changed the framing to call our MCP server DrivingBench Sandbox, and somehow that and a bunch of other prompt engineering tricks got it to…

Axiom Math's @nautsimon_ says Astra was the only model to complete their real-world driving test, but Fable showed a surprising ability to learn from its mistakes: "We tested Astra and Codex, Fable in its native harness, Claude Code and Grok. Astra was the only one that was able to complete the course." "Fable was able to do some in-context learning. If it went out the course, we'd reset it, holding the context from the past session." "While Fable didn't complete the course, we were able to see it fail a turn, then succeed the next time on that turn, and actually generalize to a new turn." @axiommathai @tobiges @a_ramabadran

Axiom Math's @tobiges, @nautsimon_ and @a_ramabadran explain why AI can know the right move and still make the wrong one when driving a real car: Tobi: "Even if you tell a car, 'You should turn left,' there's still a delay in what actually happens in the physical world, which you don't really have in coding or fully digital tasks. Even if models know in theory what they should be doing, it's still really hard to execute on these things properly." Simon: "Hallucinations are still a problem in these models even at a massive scale. If there's even a 1% chance of it crashing, that's not something that we can necessarily trust." Aditya: "I think it's called the knowing-doing gap. People have put them in ethical situations and been like, 'What would you do in this situation?' And then they…