MyCity Business Chatbot — New York City's AI Assistant Told Business Owners to Break the Law

New York City launched the MyCity Business chatbot in October 2023, built on Microsoft Azure AI services, to give small-business owners plain-language answers about municipal rules on hiring, housing and operations. By March 2024, testing by the nonprofit newsroom The Markup — co-reported with THE CITY — found the bot routinely gave confident, incorrect and sometimes illegal advice: it told users a landlord 'is not required' to accept tenants with Section 8 vouchers, when NYC law explicitly bans source-of-income discrimination; it said employers could 'take a cut of your workers' tips'; and it claimed there were 'no regulations' requiring businesses to accept cash, despite a 2020 city law mandating cash acceptance.
The OECD's AI Incidents and Hazards Monitor logged the case as a hallucination-driven incident, also noting the bot gave unsafe food-safety guidance in some tests. Despite the reporting, the city's Office of Technology and Innovation kept the tool live, calling it a 'beta' program that had 'already provided thousands of people with timely, accurate answers,' and committed only to future upgrades rather than an immediate fix or suspension.
The chatbot remained online for well over a year after the flaws were first reported publicly; incoming mayor Zohran Mamdani later announced plans to shut it down, describing it as 'functionally unusable' and noting it had cost the city roughly $600,000.

Read the full analysis: https://themarkup.org/artificial-intelligence/2024/03/29/nycs-ai-chatbot-tells-businesses-to-break-the-law