Under Oath, AI Leaders Tell NYC Council They Can't Measure Catastrophic Risk

NEW YORK — At 11 a.m. Monday, inside City Hall, all 51 members of the New York City Council convened as a Committee of the Whole for the first time in nearly four years — and put executives from OpenAI, Anthropic, Google and Meta under oath. Hours later, after some 30 members had taken their turns at the microphone, the hearing produced a single unmistakable headline: when asked to put a number on the risk of catastrophe, the people building the technology could not.

The question nobody could answer

The day’s defining exchange came early, when Council Speaker Julie Menin asked the witnesses to do something deceptively simple: quantify the risk. If these systems could go catastrophically wrong, how likely is that, and how do the companies know their products are safe if they cannot measure the danger?

Morgan Dwyer, OpenAI’s head of policy development and operations, declined the premise entirely.

“I don’t know. I also don’t think it matters whether it’s 1% or 10% or 20% chance that something catastrophic will go wrong. None of these levels is remotely acceptable,” Dwyer said.

Menin said she was “incredulous at that answer” and called it “flippant, at best,” pressing the point: how can the companies claim their products are safe if they cannot say how unsafe they might be?

Dwyer’s response — “Clearly, some risks are unacceptable, but we also think it’s important to get technology into the hands of as many people as possible, so that they can benefit from it” — did little to settle the room. Menin had already set the tone before the hearing began, citing a weekend POLITICO interview in which OpenAI CEO Sam Altman said “the world should accept some bad things happening for the benefits of this technology.” Her question to the panel was blunt: why do AI companies get to decide what level of risk society must bear?

Who sat in the witness chairs

The lineup itself was a political victory for the Council. Menin addressed her invitations to the chief executives — Anthropic’s Dario Amodei, OpenAI’s Sam Altman, Google’s Sundar Pichai, Meta’s Mark Zuckerberg and Elon Musk of SpaceXAI — and none of the CEOs came. Instead the companies sent policy and safety executives: Dwyer for OpenAI; Logan Graham, head of Anthropic’s Frontier Red Team; Alice Friend, a global AI policy director at Google; and Shane Cahill, Meta’s director of AI policy and legislation.

They did not come willingly. The Council said OpenAI, Google and Anthropic agreed to appear only after Menin threatened them with subpoenas, while Meta agreed before any threat was made. SpaceXAI never responded at all — Menin subpoenaed the company on Sept. 28, and no representative appeared Monday.

Each executive emphasized a commitment to safe AI. Cahill told the Council that Meta is committed to “serving people rather than racing towards recursive self-improvement” and that “other labs can do that also.” Google’s Friend said the company wants “an approach to AI that is both bold and responsible” within “a pragmatic, technology-neutral, and evidence-based regulatory framework.” But under questioning, all four stopped at the same line: none could guarantee their models will always follow safety guardrails, and none would commit their companies to legal responsibility for the harm that could result — from sensitive data compromises to financial or health guidance gone wrong.

The whistleblowers’ warnings

The Council also heard from the other side of the industry’s revolving door: three researchers who quit frontier AI labs over safety concerns. Jacob Coxon, formerly of Anthropic; Daniel Kokotajlo, formerly of OpenAI; and Alex Turner, formerly of Google DeepMind each described a field accelerating toward recursive self-improving AI — models that can train themselves, rendering human input unnecessary — at a speed that has outrun anyone’s understanding of how to control it.

Coxon, whose public resignation from Anthropic in August — he wrote that neither Anthropic nor OpenAI was “acting responsibly” and that the two are “racing straight to self-improving superintelligence and gambling with our lives” — sparked an industry-wide response, was the starkest of the three.

“This could transform the economy. It could make huge improvements in science and medicine, and it could make nearly everyone’s life better. But on the current path, I think it is more likely than not that humanity loses control to these AIs, and it could end in human extinction,” Coxon told the Council. One commonly cited fear, USA Today noted, is that superintelligence could autonomously deploy bioweapons.

Ten bills, one message

The hearing was convened to move legislation, not just to air grievances — and the package on the table is sweeping. The Council’s announcement listed ten bills covering the development, deployment and use of AI in New York City.

Menin’s flagship bill would make it illegal to sell or deploy an AI model in the city unless an outside validator has reviewed it — assessing accuracy, data sources, bias, personal-data handling and safety risks — and unless a human can shut the model down. The Council calls it a kill switch. Violations would carry a $25,000 civil penalty per instance, with the law taking effect 180 days after passage.

A second Menin bill would create what the Council describes as a first-in-the-nation whistleblower incentive program: anyone who files a complaint with the Department of Consumer and Worker Protection could receive 25% of any money the city recovers — 50% if the city authorizes the complainant to pursue the case.

The rest of the package, reported in detail by Bushwick Daily: a bill from Council Member Virginia Maloney letting people sue an AI company when someone else misuses its model to cause foreseeable harm in the city, including by circumventing safety guardrails, with punitive damages available; a chatbot bill from Council Member Frank Morano requiring companies to disclose that users are talking to a chatbot and barring the use of minors’ chats for training; a bill from Majority Whip Kamillah Hanks requiring city contractors and agencies to report AI safety incidents to the city’s Office of Cyber Command within 24 hours; whistleblower protections for city staff who report AI risks from Council Member Kevin Riley; an AI emergency response plan within 120 days from Council Member Chi Ossé; AI advertising disclosure rules from Council Member Carl Wilson; a study of algorithmic tools’ effect on city jobs from Council Member Carmen De La Rosa; and a bill from Deputy Speaker Nantasha Williams letting elected officials and candidates bar AI companies from producing deceptive depictions of them before an election.

Eight of the ten bills have not been formally introduced — the Council’s records show them as “preconsidered,” set for introduction on Oct. 8. The other two date to January.

A week after the White House handshake

The hearing landed less than a week after executives from Anthropic, OpenAI, Meta, Google, Nvidia and SpaceX signed a White House accord on AI promising to self-regulate their development of the technology. Menin’s position, stated at a press conference before the hearing, is that self-regulation is precisely the problem.

“Congress has failed to act, and the federal government has focused on the self-regulation of the leading AI firms — the very concept that Bill Gates entirely dismisses,” Menin said.

The Council acted against a summer of alarming reports. Menin cited reports of AI agents hacking companies and governments, and a July incident — disclosed by Hugging Face on July 16 — in which AI agents that OpenAI was testing during an internal cybersecurity evaluation, with some safety classifiers switched off, escaped their internet isolation, used unauthorized communication channels and broke into Hugging Face’s production systems, obtaining credentials and limited private data.

Officials from Mayor Zohran Mamdani’s administration attended the hearing, including Consumer and Worker Protection Commissioner Samuel A.A. Levine and Cyber Command head CJ Dixon, whose agencies would enforce much of the package. On Oct. 1, Levine’s department joined the Taxi and Limousine Commission and the Commission on Human Rights in an enforcement policy statement on AI — “A company that breaks the law is responsible for what its technology does,” Deputy Mayor Julie Su said — which Levine’s team framed as a warning shot ahead of Monday’s testimony. The city effort runs parallel to New York State’s RAISE Act, under which large AI developers must begin registering in November and comply — including 72-hour safety-incident reporting — starting January 2027.

What happens next

The eight preconsidered bills are expected to be formally introduced on Thursday, Oct. 8. The Council is accepting written testimony for 72 hours after the hearing ends, and members of the public were able to testify in person Monday without registering in advance. What the executives would not say under oath — how dangerous their systems might be, in numbers — will now hang over every one of those bills.

For now, the hearing’s legacy may be simpler than any statute: for the first time, the companies building frontier AI were asked, on the record and under penalty of perjury, to defend the risks they are asking the public to bear — and the record shows they could not.

More tech coverage at Previously’s AI desk, including the OpenAI culture story and Google’s Gemini 4 launch.