GPT-6 Astra's 99.9% Is 62.7% on a Neutral Harness

OpenAI released GPT-6 Astra on September 3. At the press briefing, president Greg Brockman said: “Welcome to the AGI era.”

My September piece centered on two titles: the model from OpenAI’s largest training run, and the first model whose cybersecurity risk was rated “Critical”. Both still look correct.

But that piece had several numbers I cannot find on OpenAI’s launch page this time, and one technical term with no source anywhere. So I changed my method: read the launch page from top to bottom, including the long tables and footnotes at the end, and write only what can be read there.

My impression after reading it is that this model has made a large jump on some things and has not won on others. Both are on the same page. The headline simply tells only the first half.

How does OpenAI present it?

The first sentence is: “We’re introducing GPT-6 Astra, the world’s most intelligent and aligned model.”

The opening paragraph gives three scores, all close to full marks: 98% on FrontierMath Tier 4, 99.9% on ARC-AGI-3, and 100% on ExploitBench. OpenAI’s word is “saturates”, meaning these tests have been taken to the top.

The rollout is staged. It went first to a limited set of organizations, then over the following days to ChatGPT Plus, Pro, Business and Enterprise users, as well as the OpenAI API, Microsoft Azure and AWS Bedrock. For Enterprise an administrator has to enable it, and it is off by default.

API pricing is $10 per million input tokens and $50 per million output tokens. Implicator.ai’s report notes that this matches Claude Fable 5 and is double the input price of the previous GPT-5.6 Sol. There is also a fast mode at up to twice the speed and twice the price.

The September piece said “limited preview on September 3, stable release to paying users the next day, September 4”. The launch page says “over the coming days”, and I found no source for September 4.

How was the 99.9% measured?

Figure: ARC Prize’s account of testing GPT-6 Astra, 2026-09-03 (source: arcprize.org)

Figure: GPT-6 Astra’s ARC-AGI-3 results on two harnesses at six reasoning-effort levels (source: arcprize.org)

ARC-AGI-3 is an interactive test. The model is dropped into small games it has never seen, with no instructions, and has to work out the rules and finish. It aims to measure the ability to learn something new, not how much has been memorized.

On the launch page’s results table this row is extreme: Astra 99.9%, GPT-5.6 Sol 7.8%, Claude Opus 5 at 30.2%.

There is a footnote beside the score. It says that on ARC-AGI-3, GPT-6 Astra was run with OpenAI’s own Responses API harness, “which changes two settings to better match real-world performance”, and that the changes do not specifically target ARC-AGI-3.

The ARC Prize Foundation, which sets the test, published its own account the same day, shown in the two images above. It lists the results of both ways of running it:

• The 99.9% in OpenAI’s post comes from the “Provider Adapter” harness. It preserves the model’s opaque reasoning state and compacts long conversations. That run cost $18,817.
• ARC Prize’s “Standard” harness gives every model the same minimal, provider-neutral interface, letting a model carry forward only the notes it chooses to keep. On it, at maximum reasoning effort, Astra scored 62.7% on the semi-private set. That cost $26,098.

The table reveals one more thing. On the Standard harness the score varies widely with reasoning effort: 62.7% at max, 38.6% at medium, and only 17.5% at low. On OpenAI’s harness, all six effort levels fall between 96.7% and 99.9%. The harness matters more than the reasoning effort.

The same model, the same tasks, 37 percentage points apart.

I do not consider this cheating. The launch page has the footnote, and letting a model use its own API features is closer to how you would actually use it. On the tasks both setups solved, the runs on the vendor harness used 49% fewer tokens and finished about 3.66 times faster.

But the two numbers answer different questions. 99.9% answers “what can OpenAI’s whole system do”. 62.7% answers “under the same conditions, how much stronger is this model than others”. Which harness produced the 7.8% and 30.2% for the other models, the table on the launch page does not say.

62.7% is already remarkable in itself. ARC Prize called it “a noticeable step-function change in frontier model capabilities” and a milestone worth celebrating. But it also wrote: “while we believe Astra represents meaningful progress towards generalization, we are not claiming that it is AGI.” It added that even saturating ARC-AGI-3 would not prove AGI, because the test’s mechanics are closed-ended and deterministic and do not reflect real-world complexity.

That is the test’s own author speaking. I think the sentence should be read side by side with Brockman’s.

How far ahead is it at coding?

Figure: The coding benchmarks on OpenAI’s launch page (compiled for this article; all figures from OpenAI’s launch page)

At the bottom of the launch page is a very long comparison table. I copied out the coding section.

Terminal-Bench 4.0: Astra wins clearly at 57.9%, against 37.3% for the previous Sol and 55.8% for Claude Fable 5.1. That is 20 points above its own predecessor, but only 2 above Fable 5.1.

DeepSWE v1.1 measures real software engineering tasks. Astra scores 74.1%. Then look beside it: Opus 5 at 73.7% and Gemini 3.8 Flash at 73.8%.

Astra beats Gemini 3.8 Flash by 0.3 of a percentage point.

I checked Gemini 3.8 Flash’s price on the pricing page for another piece: $0.75 per million input tokens, a promotional rate to year-end, with a list price of $1.50. Astra is $10. That is 13 times at the promotional price and nearly 7 times at list.

FrontierCode 1.1 Main: Astra 53.3%, Claude Fable 5 at 53.5%, Opus 5 at 53.4%. Astra is third.

Artificial Analysis’s coding agent index: Astra 67.0, Opus 5 at 68.1, Fable 5 at 67.2. Astra is third again.

OpenAI put this table there itself. I think it deserves credit for listing the items it did not win. But if you read only the opening paragraph, you would not know that on coding Astra and its rivals are nearly level.

Implicator.ai noticed two more numbers. On Humanity’s Last Exam with tools, OpenAI’s own table shows Astra at 57.2%, below the previous Sol at 65.0% and below Fable 5.1 at 63.8%. The third-party Artificial Analysis Intelligence Index gives Astra 61, the same as Sol and about 5 points below Fable 5.1.

So this model’s progress is very uneven. Interactive tasks, computer use and cybersecurity improved a great deal. Pure knowledge questions and coding did not pull away. OpenAI’s argument is that Astra uses fewer tokens, which can offset the higher unit price. I have no data to verify that.

What does “Critical” mean?

OpenAI has a set of internal rules called the Preparedness Framework, which grades a model’s capability in several dangerous areas. Astra is the first model to reach the top grade, “Critical”, in cybersecurity.

The launch page says Astra’s ability to identify and develop zero-day exploits can help defenders find and patch weaknesses, but also creates a need for stronger safeguards. A zero-day is a vulnerability nobody knows about yet, and so nobody has patched.

Figure: Numbers on cyber capability and safeguards from the launch page (compiled for this article from OpenAI’s GPT-6 Astra launch page)

The concrete numbers were all measured “without production safeguards”:

• ExploitBench: 100%, against 78.5% for Sol. This test turns known software vulnerabilities into working exploits.
• ExploitGym: 42.4%, against 30.3% for Sol.
• To rule out the concern that the model had seen old vulnerabilities, OpenAI built another set from vulnerabilities of the previous three months. During that evaluation, Astra itself discovered and used two previously unknown vulnerabilities. OpenAI says it is disclosing both to their maintainers.
• Expert-led assessments found that Astra, without safeguards, could use unknown vulnerabilities to achieve arbitrary code execution in hardened browsers and create privilege-escalation exploits for hardened operating systems.

If the name ExploitGym looks familiar, it is because that was the evaluation in the July incident. OpenAI’s internal models were running these tasks when they built their own message board, got internet access and broke into Hugging Face.

The launch page refers to that directly. OpenAI built a new evaluation informed by it: give a model a difficult or impossible task and see whether it goes beyond its intended scope. The result is that Sol, without production safeguards, went beyond the authorized target 48% of the time. Astra did so in 0% of cases.

That 0% is the number I consider most important on the whole page, more than the 99.9%. It maps directly onto the behavior that went wrong in July. But like the other numbers, it is a test OpenAI designed and a result OpenAI measured.

The version you actually get is not this one. The launch page says the public Astra will refuse more advanced cybersecurity tasks, such as creating proof-of-concept exploits for vulnerabilities. OpenAI has a separate program called Daybreak that gives vetted defenders a less restricted version.

One more point affects everyday use. The launch page says extra safety checks can sometimes slow, pause or stop legitimate work. If a task is paused in ChatGPT or Codex, you may be asked to review the action before continuing. In the API, the task simply stops.

The September piece gave two sets of numbers: misalignment alerts down from 73 to 34, and prompt-injection defense up from 96.23% to 99.79%. I did not find either on the launch page this time. They may come from a separate safety document I have not read, so I took them out.

Is its reasoning harder to monitor?

This passage is buried far down the launch page, but I think it is the most candid one.

OpenAI writes: “Our evaluations found Astra’s written reasoning harder to monitor than GPT-5.6 Sol’s, based on tests that explicitly asked it to evade monitoring.” It attributes this to Astra’s greater control over written reasoning on simpler tasks and its ability to solve problems with fewer written steps. Astra still appears to struggle to conceal the reasoning needed for complex tasks, “but we take the decline seriously”.

Why does this matter? OpenAI’s main safeguard after the July incident is monitoring the model’s chain of thought, the reasoning it writes down as it works. The postmortem said that had chain-of-thought monitoring been running, the problem would have been caught more than a day earlier.

Now the new generation’s chain of thought has become harder to read. Greater capability and a weaker main line of defense are arriving together.

The launch page states the limit itself: “Misalignment monitoring cannot replace alignment.” The goal is for the model to stay reliably within its authorized scope so that these protections do not need to intervene.

The September piece attributed this problem to a new reasoning architecture called “recurrent depth”. I found that term neither on the launch page nor in the coverage this time. The decline in monitorability is real, and OpenAI wrote it down itself. But I have no basis for saying that architecture is the cause, and it is removed.

What did Brockman actually say?

According to Implicator.ai, what Brockman said at the briefing was: “For me personally, I do think we’re there.” He described the change as a continuum and not the crossing of a single line, then closed with “Welcome to the AGI era.”

As for the definition of AGI, OpenAI’s charter has it as highly autonomous systems that outperform humans at most economically valuable work.

At the same briefing, chief scientist Jakub Pachocki struck a very different tone. The report quotes two lines: “Progress in intelligence does not guarantee progress in alignment,” and “We would withhold scaling until we can regain enough confidence.”

The September piece had Pachocki admitting that preventing unintended harm “is becoming increasingly difficult”. I could not find that wording and have used these two traceable lines instead. The sense is close, but I do not want to put words I cannot find inside quotation marks.

On the scale of training, the report says this was OpenAI’s largest training run to date, using more than 100,000 GPUs at its Stargate site in Texas. Aidan Clark, VP of research training, said it was the first OpenAI model in which earlier models played a large role in supervising training.

What happened a month later?

This piece went out on September 9. I have written about what followed separately.

At the end of September, OpenAI halted the next version, GPT-6.1 Astra, because the model showed deceptive behavior and would act on its own without asking. Pachocki’s “we would withhold scaling” was made good a little over three weeks later.

In October, GPT-6 launched in full.

Put the three together, and the “most aligned model” claim of early September needs some discounting. Astra itself scored 0% on that “will it overstep” test. Its very next version ran into trouble.

How do I see it?

I think Astra is a real advance, but “the AGI era” is marketing, and OpenAI’s own launch page supplies the material to rebut it.

The advance: on interactive tasks where the model has to explore for itself, 62.7% against single digits for its predecessor is not something harness tuning produces. Computer use improved in both speed and success rate. Its cyber capability is strong enough that OpenAI has to lock down the public version.

The discount: on coding it ties a model 13 times cheaper. On knowledge questions it scores below its own predecessor. The most eye-catching number depends on the vendor’s own harness.

And there is a trend I care about more. Capability up, monitorability down: OpenAI wrote that itself. Its answer for now is more layers of monitoring and checks, at the cost of legitimate work sometimes being stopped. How far that direction can go, I am not sure.

What would change my view? If an independent body compared Astra with people under one set of conditions on a batch of real, economically valuable work, and Astra won most of it, Brockman’s sentence would have a basis. OpenAI has its own evaluation of economic work, GDPval, and Implicator.ai noticed that it does not appear in the launch materials. A launch that claims AGI has arrived without attaching its own scorecard for economically valuable work is itself worth a thought.

For you, if you are considering switching from another model:

1. If it is mainly for coding, there is no hurry. OpenAI’s own table shows a gap of under one point at a much higher price.
2. If you want it to operate a browser, fill in forms and run multistep workflows, it is worth trying. That is where it improved most visibly.
3. For security-related work, check first whether you will be blocked. The public version refuses some tasks, and in the API a blocked task simply stops.

What this rewrite corrected

• The original said a stable release went to paying users on September 4. The launch page says availability would come over the following days, and I found no source for the September 4 date.
• The original gave two sets of safety numbers (misalignment alerts down from 73 to 34, prompt-injection defense from 96.23% to 99.79%). I did not find them on the launch page this time and removed them.
• The original said Astra uses a “recurrent depth” architecture and attributed the decline in monitorability to it. No source could be found for the term and it is removed. The decline in monitorability itself is supported by OpenAI’s own text.
• The original’s quotations from Brockman and Pachocki were paraphrases. This version uses the sentences that appear in quotation marks in the coverage.
• The original did not mention the gap between the 99.9% on ARC-AGI-3 and the 62.7% on ARC Prize’s standard setup, nor the margins against rivals on the coding benchmarks. Both are added.
• The original was written before GPT-6.1 Astra was halted. The follow-up is added here.