California lawmakers took a key vote today to pass a controversial bill that requires companies that make or modify powerful forms of artificial intelligence to test for their ability to enable critical harm to society.

Following a 32-1 vote in the Senate in May, the Assembly voted 41-9 to pass the bill late Wednesday afternoon.  The Senate must take one more vote for the bill to make it to Gov. Gavin Newsom, concurring with any amendments in the Assembly.

Under Senate Bill 1047, companies that spend $100 million to train an AI model or $10 million to modify one must test the model for its ability to enable cybersecurity or infrastructure attacks or the development of chemical, biological, radioactive, or nuclear weaponry.

Eight members of Congress who represent California districts earlier this month took the unusual step of urging Newsom to veto the bill. It’s not clear if he will do so. In May, at a generative AI symposium required by an executive order he signed, Newsom said California must respond to calls for regulation but avoid overregulation. California is home to many of the dominant AI companies in the world.

Powerful interests lined up in favor and in opposition to the bill. Opposition has come from companies including Google and Meta, ChatGPT maker OpenAI, startup incubator Y Combinator, and Fei-Fei Li, an advisor to President Joe Biden and co-organizer of the generative AI symposium ordered by Newsom. They argue that costs to comply with the bill will hurt the industry, particularly startups, and discourage the release of open source AI tools, since companies will fear legal liability under the bill.

Whistleblowers who used to work at OpenAI, and Anthropic, a company co-founded by former OpenAI employees, support the bill. Also in favor of the bill are Twitter CEO Elon Musk, who helped start OpenAI and frequently-cited AI researcher Yoshua Bengio. They argue, in part, that AI tools pose significant harms and that the federal government has not done enough to address those harms, including through regulation.

Sen. Scott Wiener, a Democrat from San Francisco and author of the bill, has said the purpose of SB 1047 is to codify safety testing that companies already agreed to with President Biden and leaders of other countries.

“With this vote, the Assembly has taken the truly historic step of working proactively to ensure an exciting new technology protects the public interest as it advances,” Wiener said in a press release Wednesday.

In the past year, major AI companies reached voluntary agreements with the White House and government leaders in Germany, South Korea, and the United Kingdom to test their AI models for dangerous capabilities. In remarks responding to OpenAI’s opposition to the bill, Wiener rejected the assertion that passage of SB 1047 will lead businesses to leave the state, calling it a “tired” argument.

Wiener said similar claims were made when California adopted net neutrality and data privacy laws in 2018, but those predictions did not come true.

The bill’s critics, including OpenAI, have said they would rather Congress regulate AI than have it regulated at the state level. Wiener said Monday that he agrees and probably wouldn’t have proposed SB 1047 had Congress done so. He added that Congress is paralyzed and that “other than banning TikTok hasn’t passed major tech regulation since the 1990s.”

SB 1047 went through several rounds of amendments on its way to passage. Cosponsor Ari Kagan, cofounder of AI toolmaker Momentum, in July told lawmakers that passage of SB 1047 is necessary to prevent an AI disaster akin to Chernobyl or Three Mile Island. Earlier this month, he told CalMatters that amendments getting rid of a Frontier Model Division that was initially part of the bill weakens the legislation but said he still supported passage.

In addition to SB 1047, earlier this week the California Legislature moved to pass laws that require large online platforms like Facebook to take down deep fakes related to elections and create a working group to issue guidance to schools on how to safely use AI. Other AI policy bills up for a vote this week include a bill to empower the Civil Rights Department to combat automated discrimination, require creatives get permission before using the likeness of a dead person’s voice, body, or face in any capacity, and additional measures to keep voters safe from deceptive deep fakes.

In accordance with a generative AI executive order, the California Government Operations Agency is expected to release a report on how AI can harm vulnerable communities in the coming weeks.

Khari Johnson is a reporter for CalMatters.