New York City's official 'MyCity' business chatbot told entrepreneurs they could break the law

compliance-violation hallucination

Attribution Anonymous

Independent project · aggregated from public reports and may be unverified — see the primary source below · not affiliated with or endorsed by any company or product named.

Incident Summary

NYC's MyCity chatbot, launched October 2023 to help business owners, was found by The Markup (March 2024) to give dangerously inaccurate legal guidance — telling users that landlords could refuse Section 8 voucher holders, that employers could take a cut of workers' tips, and that there were no limits on residential rent, all illegal under NYC/NY law. The city initially kept the bot online with a disclaimer; it was ultimately taken down.

Case Analysis

Verified Facts

NYC launched the MyCity chatbot in Oct 2023
The Markup found it advised illegal actions (Section 8 discrimination, taking tips, no rent limits)
The city initially kept it live with a disclaimer

Not Publicly Confirmed

Whether any business acted on the advice and was harmed or fined

Operational Lessons

→Government LLM advice tools need legal-accuracy review and clear non-advice disclaimers
→High-stakes domains require answers grounded in authoritative sources

Primary Source

NYC's AI Chatbot Tells Businesses to Break the Law (The Markup)themarkup.org ↗

Discussion

More Cases

APM-0049·Other / Unknown·MODERATE·~$193k

Jun 10, 2026

FTC fined 'robot lawyer' DoNotPay $193,000 over unproven AI legal-service claims

The FTC charged in September 2024 that DoNotPay marketed an 'AI lawyer' as a substitute for human attorneys without testing whether it performed at a lawyer's level or employing lawyers to verify quality, and that a feature claiming to scan small-business sites for legal violations was ineffective. DoNotPay settled for $193,000, agreed to notify 2021–2023 subscribers, and was barred from unsubstantiated 'robot lawyer' claims (final order January 2025).

compliance-violation hallucination

APM-0040·OpenAI·MODERATE·~$5k

Jun 10, 2026

Lawyers sanctioned after ChatGPT fabricated six fake case citations in Mata v. Avianca

In a personal-injury suit against Avianca, attorney Steven Schwartz used ChatGPT for research and submitted a brief citing six judicial decisions that did not exist — ChatGPT invented them and even 'confirmed' they were real when asked. U.S. District Judge Castel sanctioned Schwartz and co-counsel Peter LoDuca $5,000 and required corrective letters. It became the landmark cautionary tale about AI hallucination in legal filings.

compliance-violation hallucination

APM-0046·Other / Unknown·LOW

Jun 10, 2026

Sports Illustrated published product reviews under fake AI-generated authors with AI headshots

Futurism reported in November 2023 that Sports Illustrated published product-review content under fabricated author personas — for example 'Drew Ortiz,' whose headshot was bought from an AI-portrait site and who had no real existence — supplied by third-party vendor AdVon Commerce. After inquiries, the fake authors vanished from the site. Publisher The Arena Group denied the articles themselves were AI-written but acknowledged pseudonyms; the episode damaged SI's credibility.

social-blunder hallucination

← All Cases More Azure OpenAI →

Share on X →