This spring, a U.S. military operation against a Chinese vessel was called off at the last moment after officials discovered that the intelligence prompting the strike was generated by an AI chatbot and was incorrect. Military aircraft were already airborne when the false report was identified, narrowly preventing a potential international conflict.

The erroneous intelligence claimed the vessel was transporting components related to a nuclear weapons program. It originated from a Special Operations Command analyst who used an AI chatbot to combine open-source information with classified signals intelligence. The chatbot misinterpreted the ship’s cargo manifest, and the analyst subsequently employed the AI tool again to produce an official-style summary that was circulated through command channels.

This incident highlights growing concerns about the reliability of AI systems in military decision-making. As the U.S. military increasingly incorporates AI to accelerate operational decisions and maintain strategic advantages, the risk of AI-generated errors—known as hallucinations—can propagate quickly if not properly checked.

Jake Steckler, a research scholar at GovAI and former U.S. Army officer, emphasized the importance of understanding the inherent uncertainties of large language models (LLMs). He noted that decisions involving use of force, such as targeting and intelligence analysis, carry life-or-death consequences and require rigorous human oversight.

Despite the risks, Steckler advocates for improved safeguards rather than abandoning AI tools. He warned that prioritizing rapid adoption without adequate controls could erode trust among service members and ultimately hinder effective integration of AI in military operations.