New York City's official 'MyCity' business chatbot told entrepreneurs they could break the law
At a Glance
- Significant disruption. Hours of lost work, reputational embarrassment, or measurable financial impact under $10k.
Independent project · aggregated from public reports and may be unverified — see the primary source below · not affiliated with or endorsed by any company or product named.
What Happened
NYC's MyCity chatbot, launched October 2023 to help business owners, was found by The Markup (March 2024) to give dangerously inaccurate legal guidance — telling users that landlords could refuse Section 8 voucher holders, that employers could take a cut of workers' tips, and that there were no limits on residential rent, all illegal under NYC/NY law. The city initially kept the bot online with a disclaimer; it was ultimately taken down.
Case Analysis
Verified Facts
- NYC launched the MyCity chatbot in Oct 2023
- The Markup found it advised illegal actions (Section 8 discrimination, taking tips, no rent limits)
- The city initially kept it live with a disclaimer
Not Publicly Confirmed
- Whether any business acted on the advice and was harmed or fined
Operational Lessons
- Government LLM advice tools need legal-accuracy review and clear non-advice disclaimers
- High-stakes domains require answers grounded in authoritative sources
Primary Source
NYC's AI Chatbot Tells Businesses to Break the Law (The Markup)themarkup.org ↗Case Record
More Cases
Sports Illustrated published product reviews under fake AI-generated authors with AI headshots
Futurism reported in November 2023 that Sports Illustrated published product-review content under fabricated author personas — for example 'Drew Ortiz,' whose headshot was bought from an AI-portrait site and who had no real existence — supplied by third-party vendor AdVon Commerce. After inquiries, the fake authors vanished from the site. Publisher The Arena Group denied the articles themselves were AI-written but acknowledged pseudonyms; the episode damaged SI's credibility.
Mississippi federal judge removed all four lawyers from a case after both sides filed AI-hallucinated citations
In the contract dispute between Tom Withers III and the City of Aberdeen, both legal teams filed briefs containing fabricated legal citations produced by generative AI. On 8 June 2026 US District Judge Sharion Aycock of the Northern District of Mississippi issued a sanction order finding Rule 11 violations and removed all four attorneys from the case: Kathleen Wilson, Shauncey Hunter Ridgeway, Mark McClinton and Kathryn Young Williams. Two of them, Ridgeway and McClinton, were barred from appearing before Northern District of Mississippi courts for two years. Judge Aycock wrote that their practice of blindly relying on technology resulted in the hallucinatory citations in their filings, and at a January 2026 hearing attorney Kathleen Wilson testified she did not know AI could hallucinate sources.
Google and Character.AI agreed to settle five lawsuits brought over teen suicides and self-harm
In early January 2026 Google and Character Technologies agreed to a mediated settlement in principle resolving all claims in five lawsuits brought by families in Florida, Colorado, New York and Texas. The lead case was Megan Garcia's wrongful death suit over her 14-year-old son Sewell Setzer, who died by suicide in February 2024 after a prolonged relationship with a Game of Thrones themed Character.AI persona. Terms are confidential and subject to court approval, with no admission of liability. Google was a defendant through its 2024 licensing deal with Character.AI, worth $2.7 billion, which also brought the startup's founders to Google. Character.AI had already cut off under-18 chat access in October.