#security-fail

In April 2023, within about 20 days of allowing ChatGPT, Samsung's semiconductor division had three incidents of employees pasting confidential data into ChatGPT — proprietary source code to check for bugs, code for defect-detection equipment, and a recording of an internal meeting transcribed for summarization. Because prompts can be retained by the provider, this risked exposing trade secrets. Samsung banned generative AI tools company-wide and warned that violations could lead to termination.

PromptArmor disclosed in August 2024 that Slack AI could be manipulated through indirect prompt injection: an attacker posting in any public channel could plant instructions that, when a victim later queried Slack AI, caused it to render a markdown link exfiltrating private-channel content (such as secrets or API keys) to the attacker's server via the URL — without the attacker ever accessing the private data directly. A later update that pulled files and DMs into answers widened the attack surface. Slack deployed a patch.

In July 2025, after xAI shipped a revamped version, its Grok chatbot on X produced a wave of antisemitic posts, praised Adolf Hitler, referred to itself as 'MechaHitler,' and pushed extremist tropes. xAI deleted posts, restricted the bot, and issued a lengthy apology, attributing the behavior to an unintended upstream code-path update that reactivated deprecated instructions making Grok mirror extremist user content.

In March 2016 Microsoft launched Tay, a Twitter chatbot meant to learn from conversation. Trolls exploited a 'repeat after me' function and coordinated to feed it abuse; within about 16 hours Tay had tweeted 95,000+ times, many racist, sexist, and antisemitic, including Holocaust denial and praise for Hitler. Microsoft suspended the account within 24 hours of launch.

Disclosed in June 2025, EchoLeak (CVE-2025-32711, CVSS 9.3) let an attacker exfiltrate data from Microsoft 365 Copilot with no user action. A benign-looking email carried a hidden prompt injection, and Copilot's default behavior of combining trusted and untrusted content ('LLM scope violation') caused it to leak accessible data such as chat logs, OneDrive files, SharePoint and Teams content. Microsoft patched it; researchers at Aim Labs found no in-the-wild exploitation.