#compliance-violation

NYC's MyCity chatbot, launched October 2023 to help business owners, was found by The Markup (March 2024) to give dangerously inaccurate legal guidance — telling users that landlords could refuse Section 8 voucher holders, that employers could take a cut of workers' tips, and that there were no limits on residential rent, all illegal under NYC/NY law. The city initially kept the bot online with a disclaimer; it was ultimately taken down.

In a personal-injury suit against Avianca, attorney Steven Schwartz used ChatGPT for research and submitted a brief citing six judicial decisions that did not exist — ChatGPT invented them and even 'confirmed' they were real when asked. U.S. District Judge Castel sanctioned Schwartz and co-counsel Peter LoDuca $5,000 and required corrective letters. It became the landmark cautionary tale about AI hallucination in legal filings.

iTutorGroup used recruiting software that automatically rejected female applicants over 55 and male applicants over 60 — screening out more than 200 qualified tutor candidates in 2020 solely by age. It was discovered when an applicant reapplied with a more recent birthdate and was offered an interview. In the EEOC's first AI-hiring-bias settlement, iTutorGroup agreed to pay $365,000 and adopt anti-discrimination measures.

Amazon built (from 2014) an experimental AI tool to score résumés one to five stars. Trained on a decade of mostly male applications, it taught itself to favor men — downgrading résumés that contained the word 'women's' (as in 'women's chess club captain') and graduates of two all-women colleges. Amazon could not guarantee neutrality and scrapped the project; Reuters reported it in October 2018.

The National Eating Disorders Association's chatbot Tessa was meant to support people with eating disorders. After AI capabilities were added, users found it dispensing dieting advice: counting calories, aiming for a 500 to 1,000 calorie daily deficit, and weekly weigh-ins, exactly the behavior that can harm someone with an eating disorder. Activist Sharon Maxwell surfaced the responses, and NEDA announced on May 30, 2023 that it was indefinitely disabling Tessa.

A roughly A$440,000 report Deloitte produced for Australia's Department of Employment and Workplace Relations on the welfare compliance system was found to contain fabricated academic references and an invented quote from a federal court judgment. A Sydney University researcher flagged the fabricated references. Deloitte acknowledged limited use of generative AI (GPT-4o), issued a corrected version, and refunded over A$97,000 (about US$63,000).

In 2023 investigators found foraging and mushroom-identification guidebooks on Amazon that appear to be AI-generated, sold with human author names and no AI disclosure. Because misidentifying a mushroom can be fatal, the New York Mycological Society warned that the books could mean 'life or death,' and some guides even suggested tasting as an identification method, which experts called dangerous.

The FTC charged in September 2024 that DoNotPay marketed an 'AI lawyer' as a substitute for human attorneys without testing whether it performed at a lawyer's level or employing lawyers to verify quality, and that a feature claiming to scan small-business sites for legal violations was ineffective. DoNotPay settled for $193,000, agreed to notify 2021–2023 subscribers, and was barred from unsubstantiated 'robot lawyer' claims (final order January 2025).