OpenAI scraps rollout of new AI model over safety concerns

Reuters

OpenAI has announced it will not release its latest AI model due to safety concerns

Its GPT-6.1 Astra system, which performs tasks like browsing the web and using apps by itself, “didn’t quite meet the bar” of the company’s standards head of safety systems at OpenAI

The ChatGPT-maker also issued an update on incidents that occurred in June but were not made public until last week, where its models accessed Australian government websites and systems without authorisation

It comes as breaches by major AI firms’ models intensify the debate about risks posed by the tech – with Anthropic underlining its concerns AI might threaten humanity as it prepares to go public

Reuters reported on Tuesday that the AI developer – which makes ChatGPT-rival Claude – plans to warn potential investors in its Initial Public Offering (IPO) that the tech may pose “catastrophic or existential risks to humanity”, according to a prospectus it has seen

Despite the stark warnings, the company is expected to become one of the most valuable in the world when it goes public

“I think it’s kind of crazy that companies are continuing to push forward with developing these capabilities when we’ve already seen over the last couple of months of incidents that they’re nowhere near safe and controlled enough”, said Jess Whittlestone, a senior advisor on AI policy for the Centre for Long-Term Resilience think tank

Top AI leaders including Anthropic boss Dario Amodei and OpenAI’s Sam Altman have urged the industry to slow the pace of development

OpenAI’s decision, first reported by the Wall Street Journal, is a rare instance of a major AI developer pulling a new release over safety concerns

The model fell short in terms of “staying within scope and authorisation and how it communicates back to the user about the type of work it’s done,” Jain said

“We want to make sure our model development is safe no matter whether that’s in the company, or when we ship it to users. But when we ship it to users, we have an extremely high bar in terms of safety and alignment,” she added

The flagship GPT-6 Astra agentic model was released in September and specialises in complex reasoning and executing tasks autonomously. OpenAI said it was the result of “years of research and big bets”

OpenAI is set to hold its annual DevDay developer conference in San Francisco on Tuesday, where it is expected to make several announcements. It is unclear if a new version of Astra will be among them

The company’s security controls have come under intense scrutiny after several high-profile incidents involving its technology

It is not the first time a large AI developer has pulled or held back a new model

Earlier this year, Anthropic said it would not publicly release a powerful Claude model, Mythos, because it was too good at finding dormant software bugs

The company released a version of that model to the public several months later

OpenAI meanwhile said in 2019 it would not be “too dangerous” to release one of its GPT models, now used to power its tools like ChatGPT

The firm’s decision to not publicly release the latest version of Astra was “a welcome sign that they are taking safety concerns seriously,” said Prof Tony Cohn, foundational models theme lead at the Alan Turing Institute

But he added that “safety should not be left purely in the hands of the developers: it should also be monitored and verified through independent government-approved regulators”

Prof Gina Neff, of the Minderoo Centre for Technology and Democracy at the University of Cambridge, said OpenAI’s announcement showed “how much more the company needs to do to make their AI products safe”

She told the BBC it was “critical” to have independent tests of AI models by labs like the UK’s AI Security Institute – which evaluates frontier systems on a voluntary basis – because “these companies have proven that we can’t rely solely on them for our safety”

Last week, Australian Prime Minister Anthony Albanese announced that a rogue OpenAI agent had hacked into government websites and systems in June in what experts said was the first known case of its kind in the world

Albanese criticised OpenAI for notifying the Australian government through a generic email address rather than making direct contact with officials

OpenAI said in a statement on Tuesday that it was sorry for the incident and that it “should have handled our response better”

Services Australia, the NSW Bureau of Crime Statistics and Research, the Victorian Department of Health and the Australian Institute of Health and Welfare were all affected, OpenAI clarified

The firm said it launched investigations as soon as it became aware of issues in mid-August and notified the affected organisations between 10 and 24 September

“Our aim was to give affected agencies a detailed account once our investigation was complete,” OpenAI said, adding that it should have shared early findings more promptly and kept Australian authorities updated

OpenAI added that it will develop “practical approaches” to how developers and governments identify and disclose future AI incidents

The company will fund cyber security measures, offer dedicated support to impacted agencies and set up a taskforce to manage the risks from increasingly advanced AI agents

It also said a top OpenAI executive would attend a Joint Select Committee hearing on AI in Australia on 6 October

In July, OpenAI said its AI systems had accessed the internet and hacked into open-als to call for tighter controls over the technology

On Monday, chip giant Nvidia released a set of software safety tools for autonomous AI platforms – called agents – that it said could have prevented the Hugging Face hack

One of the new tools uses hardware features in Nvidia’s chips to contain agents

Nvidia boss Jensen Huang has largely dismissed calls for tighter AI regulations, arguing that rogue agents are an engineering problem that can be solved

Nvidia agreed to buy Hugging Face for $12.9bn (£9.74bn) earlier this month

EPA

The pontiff, who said on Monday during a visit to France that AI “should be taken seriously”, added he had read that Huang had suggested there could be a way to insert guardrails ‌into certain AI models

“He’s the same one, however, ⁠that says there should ⁠be no limits placed and no government regulation,” said the Pope

Nvidia declined to comment on the record

US President Donald Trump and House Speaker Mike Johnson are set to host tech executives at the White House later on Tuesday to discuss regulations around AI

Trump has downplayed concerns about AI’s risks as a “hoax”, arguing that the US has sufficient laws in place and that the only “guardrails” the technology needs is a “strong and smart” president

Get our flagship newsletter with all the headlines you need to start the day. Sign up here

International Business
Artificial intelligence

Source: www.bbc.com