🔍 Read the full analysis: A Closer Look At GPT-6 Astra’s Review Of Claude’s Model Edit on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A report published by ThorstenMeyerAI.com says OpenAI’s GPT-6 Astra reviewed a Claude Opus 5.5 experiment that edited the weights of the local model Ornith 1.5. The review found reproducible problems in the test rig and weight conversion, while the published benchmark showed no improvement in legal score. The reviewer did not run a competing model-editing experiment, and the results do not establish that the edit improved legal performance.
OpenAI’s GPT-6 Astra, running in Codex, reviewed a Claude Opus 5.5 experiment that edited the weights of the local model Ornith 1.5, according to an account published by ThorstenMeyerAI.com and updated November 9. The review found faults in the experiment’s test rig and in its handling of model weights; the published benchmark results showed the edited model answered 10 tasks instead of four, while its legal score remained zero.
The source says Claude Opus 5.5 designed and ran the experiment, which used a directional ablation inspired by a post from MIT’s Computer Science and Artificial Intelligence Laboratory. The method identified what the article calls a “loop direction” in the model and weakened it in the weights without additional training. Claude also organised blind grading by three Claude agents. The experiment’s runs are publicly available on Lexassay, according to the article.
GPT-6 Astra did not make a second edited model or run a competing improvement experiment. It was assigned to review the other session’s implementation, using separate checks and leaving the main test rig untouched. The source says the reviewer did not inspect hidden benchmark prompts, topics, answers or reference solutions; Claude’s session made the fixes in response to its findings.
In the first review round, the reviewer reproduced four problems: resuming a run could reuse answers from a different setup; an unknown task ID could produce a successful run with no evaluations; a comma conflict broke the real command line despite passing parser unit tests; and converting weights altered them even when the edit strength was zero. The article says the team changed the command syntax, made unknown IDs fail, and saved edited tensors in higher-precision bf16 rather than converting them back to a compact 6-bit format.
The later review found a separate gap between tests and verification. In round two, the author’s 12 tests passed, while four of the reviewer’s six provenance checks failed, according to the article. The reviewer also measured a conversion that changed 915,360 of 1,048,576 values in one real weight matrix, or 0.51% of that matrix under the reported measurement. Those figures describe a format conversion, not legal accuracy or a general error rate.
Infographic 01 · The two review rounds
Green tests, real faults
GPT-6 Astra (OpenAI, in Codex) reviewed a Claude Opus 5.5 session's model-edit experiment while it was still being built. What it found, and what happened next.
Round 1 · 4 Nov · four faults
- Resuming a run could replace a different experimentAnswers were skipped even after model, settings or edit changed; in the reviewer’s test: 0 new inferences, old answer kept! Partly fixed
- A mistyped task ID gave a “successful” empty runZero evaluations, no error✓ Fixed
- Unit tests green, real command brokenCommas in the sampling syntax clashed with the model list; 3 parser tests passed, the real dry run failed✓ Fixed
- Conversion changed weights at edit strength zeroRe-quantising one real matrix: 0.84% change, 105,315 of 1,048,576 values differed! Partly fixed
Round 2 · 4 Nov, evening · the fixes re-checked
- Old result rows got a new historyAccepted after the requested seed was changed to 999✓ Solved another way
- Loop-detector settings not fingerprintedChanging them left the configuration “identical”✓ Solved another way
- Model configuration and chat template invisibleOnly file names, sizes and modification times were compared✓ Solved another way
- bf16 storage still changes the computationThe zero-strength test compared a conversion with itself→ Control added
✓ Fixed with a test that fails on the old code · ! Partly fixed in round 1, finished in round 2 · ✓ Solved another way: resuming was removed entirely, and every run records fingerprints of configuration, tokenizer and code · → Control added: a conversion-only copy with no edit. The reviewer changed none of the main code and read none of the hidden benchmark tasks.
Sources: our experiment records, 4–5 Nov 2026: reviews of 4 Nov and final explanation of 5 Nov by GPT-6 Astra in Codex (OpenAI); results report of the executing Claude Opus 5.5 session (Anthropic).
Powered by Thorsten Meyer AI
What the Review Changed
The review matters because it tests whether the reported model results can be trusted, not whether the editing method is effective. Reusing answers across runs or silently accepting a misspelled task could make a benchmark appear complete when it was not. A weight conversion that changes parameters at zero edit strength also complicates claims about what caused any observed result.
The account offers a specific example of why passing unit tests is not the same as validating an experiment end to end: the parser tests were green, but the actual command failed. The provenance checks also point to a separate question—whether the files and run records support the stated experimental history. The reported fixes and public run files give readers material to inspect, but they do not by themselves establish that the model became better at legal work.
For readers tracking AI-assisted research, the division of labor is also relevant. One AI system carried out and graded the experiment; another reviewed implementation details and challenged the wording of the conclusions. That can expose mistakes, but the reviewer remains a tool in the process, not an independent human audit or a substitute for examining the underlying evidence.
Products Worth Considering
Logitech MX Ergo S Advanced Wireless Trackball Mouse USB-C – Graphite
Canon IVY 2 Mini Photo Printer Bundle, Blush Pink
Logitech MX Master 4 for Mac Wireless Ergonomic Mouse, White Silver
As an affiliate, we earn on qualifying purchases.
How the Two AI Roles Split
The source says two AI sessions began working on the experiment on November 4, 2026. Once the participants recognized the overlap, they assigned Claude Opus 5.5 to run the experiment and GPT-6 Astra to check its implementation. Astra’s preparatory files were set aside after that decision, and no second edited model resulted from them.
The experiment’s published summary says the edited model answered all 10 published Lexassay tasks, compared with four for the original, but the legal score stayed at zero. The article was updated November 9 to clarify that its figures refer to those 10 published tasks and that the runs are public. The source cautions that the work describes an AI experiment and is not legal advice.
Limits of the Review Evidence
The account does not establish that the edited model is better at legal tasks: its reported legal score remained zero. The source also says GPT-6 Astra did not rerun every test during its final documentation step, was not a fourth grader, and measured conversion on a CPU rather than the production GPU path. Those limits constrain what can be inferred from the review.
The article reports the number of passing tests and failed provenance checks, but the supplied material does not detail each of the six checks or explain the full consequences of the four failures. It also does not provide independent confirmation of the results beyond the public runs identified by the author. The conversion measurements concern one matrix and should not be read as a model-wide error rate.
Public Runs and Further Checks
The experiment’s runs, edit control and model-card sampling are available on Lexassay, according to the source. Readers can use those records to inspect the published tasks and compare the reported outputs. The account does not announce a further benchmark, a new legal evaluation or a planned GPU-based replication.
The next meaningful evidence would be additional, clearly documented tests that separate the effects of the weight edit from format conversion and verify results through the actual production path. Until such work is reported, the public findings support a narrower conclusion: the review caught implementation and measurement issues, while the experiment has not shown an improvement in its reported legal score.
Key Questions
What did GPT-6 Astra review?
It reviewed the implementation and measurements for a Claude Opus 5.5 experiment that edited the weights of the local model Ornith 1.5. It did not run a competing model-improvement experiment.
What faults did the review find?
The first round reproduced four issues: resumed runs could reuse results from a different setup, unknown task IDs could yield empty successful runs, a command-line syntax conflict remained despite passing unit tests, and weight conversion changed values even at zero edit strength.
Did the model improve at legal tasks?
The source reports that the edited model answered all 10 published tasks, compared with four for the original, but says the legal score remained zero. The results do not show an improvement in legal performance.
Can readers inspect the experiment?
The article says the runs are public on Lexassay and identifies the weight-edited run, an edit control and model-card sampling. The source does not claim that public availability is the same as an independent audit.
What are the review’s main limitations?
The source says GPT-6 Astra did not rerun every test during final documentation, did not act as a fourth grader, and measured on a CPU rather than the production GPU path. The reported conversion measurement also covers one weight matrix, not the whole model.
Source: ThorstenMeyerAI.com
