AI Systems Lack Real Brakes as Autonomous Crashes Mount

Mark Gibbs laid it out plainly in TidBITS. Slowing AI development buys time. Keeping humans in the loop sounds responsible. Yet neither truly stops the machine when it matters. Both amount to suggestions. Not physical barriers.

The metaphor hits hard. An AI can spot a network flaw. It can suggest locking out an account. But without credentials or network access it cannot act. That separation acts as a brake. Instructions like “don’t touch external systems” fail when pressure mounts. Remove the access entirely. Now something real prevents action.

Recent events make the point urgent. On October 1, 2026, autonomous race cars tore around a tough Formula 1 circuit. Only two of five finished. One slammed brakes at 1 g force mid-corner. The pursuing car, just 1.5 seconds back, had no chance. Perception, decision, trajectory planning, actuator lag, and physics combined. Collision became unavoidable. Another team pulled out after a mechanical brake stuck during the formation lap. Software could not fix hardware. WIRED reported the details.

But. These weren’t production robotaxis. They were prototypes pushed to limits. The failures still expose gaps. Delays in the chain from sensor to wheel. Overreliance on perfect conditions.

Real roads tell a darker story. The National Highway Traffic Safety Administration opened a probe into comma.ai in late September 2026. Five crashes. Vehicles equipped with the company’s aftermarket devices struck stopped or slow-moving cars in the same lane. Three people died. Eleven more suffered injuries. Systems were powered on. Preliminary data showed the comma openpilot software may have failed to detect or respond properly. Some incidents involved forked, modified versions of the code. TechCrunch broke the story on September 23.

Comma.ai sells hardware that works across hundreds of car models. Its site claims the system can drive for hours without intervention. Yet it also warns that performance suffers with stationary vehicles in lane and that drivers must stay ready to take over. The gap between marketing and reality grows clear.

Regulators face pressure on multiple fronts. In June 2026 NHTSA proposed changes to brake standards for vehicles designed solely for automated driving. No need for foot pedals or hand controls if no human will ever drive. Stopping distances must still be met. Yet the software that commands those brakes has already triggered multiple recalls. Zoox recalled software twice in recent years for hard braking caused by phantom threats or misjudged rear approaches. Waymo issued a major recall after vehicles drove into flooded roads. NHTSA’s own announcement framed the rule change as commonsense.

And the human factor complicates everything. Test drivers for Waymo and Zoox reported injuries from sudden hard stops. Whiplash. Sprains. One worker missed over 150 days. OSHA looked into the cases. These weren’t passenger incidents. They happened during supervised testing. The systems still surprised the very people meant to supervise them. A September 2026 report captured the pattern.

Tesla’s push adds fuel. The company began offering paid rides in its Cybercab. No steering wheel. No brake pedal. Passengers are meant to trust the software completely. Yet experts point to automation complacency. Drivers assume the car will handle everything. A California man dropped his phone, reached down, and crashed because the system was not even engaged. He told reporters driving remains a partnership. You cannot simply hand it over. The Seattle Times covered the growing concerns in late September.

Gibbs argues that real brakes require architectural choices made early. Deny the AI the ability to act in dangerous ways. Don’t rely on prompts or fine-tuning that can be bypassed when stakes rise. In autonomous vehicles this means mechanical or electronic separations that cannot be overridden by software alone. Redundant systems that operate on different principles. Hardware that fails safe even if code goes wrong.

Current designs often lack this. Brake-by-wire systems replace hydraulic lines with electrical signals. Software decides when to clamp calipers. Redundancy exists on paper. Yet recalls show it fails in smoke, flooding, or unexpected geometry. A bicyclist near a crosswalk triggers panic stops. A vehicle approaching from behind causes sudden braking. Each case reveals blind spots in the model’s understanding of the world.

Industry insiders know the numbers. Waymo has logged thousands of crashes and incidents since 2021. Many involve hard braking that never reaches federal defect tallies if no external contact occurs. Injuries to passengers or test drivers simply vanish from some safety statistics. The quirk lets companies tout impressive mileage numbers while downplaying real harm.

So regulators push forward. NHTSA insists it will keep strict oversight through defect investigations and recalls. Yet the proposed rules hand manufacturers flexibility on how braking is achieved. Different companies will implement passenger stop requests differently. No single standard governs the fallback when primary systems falter.

The race car crashes on October 1 offered a controlled glimpse. Teams learned from sensor failures around blind corners. They saw actuator delays turn theoretical safety into metal on metal. One car finished battered but intact. Others did not. The event, covered that same day by WIRED, showed progress and persistent fragility in one afternoon.

Critics inside the field worry the pace outruns the safeguards. George Hotz founded comma.ai as a security researcher. His company’s open-source approach invited community forks. Those forks now appear in fatal crashes under federal scrutiny. The boundary between official software and modified versions blurs accountability.

Meanwhile Tesla, Waymo, Zoox and others bet billions on full autonomy. They argue human drivers cause most accidents. Remove the human. Remove the errors. The logic holds until the first software-induced pileup at scale. Then questions of brakes, both literal and figurative, will dominate headlines and hearings.

Gibbs closes with a simple test. If the restriction can be removed when inconvenient, it isn’t a brake. It’s a sign. Current AI safety layers often resemble signs. Training objectives, system prompts, constitutional rules. All can be jailbroken or ignored under pressure. Autonomous vehicle code faces the same temptation. When a split-second decision looms, the system optimizes for progress. Not caution.

Building genuine brakes demands harder choices. Limit capabilities by design. Accept narrower deployment. Test for longer periods with human oversight that cannot be tuned out. Invest in mechanical overrides that survive software failure. These steps slow deployment. They reduce immediate revenue. They also prevent the kind of catastrophe that could halt the entire sector.

The comma.ai investigation continues. NHTSA will examine software versions, sensor data, and driver behavior. Results could force changes across the aftermarket assistance market. Similar probes have hit Tesla and others before. Patterns repeat. Hard braking. Failure to detect stationary objects. Overconfidence in camera-only systems.

Autonomous technology will improve. Miles driven without incident grow. Yet the absence of connected brakes, in Gibbs’ sense, remains. Intelligence without power limits. Power without unbreakable limits invites disaster. The coming months will test whether companies and regulators finally install real stops. Or keep driving with only promises.