🔍 Read the full analysis: AI Weight Editing At Home: A Zero Legal Score Outcome on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A home experiment edited the weights of the local AI model Ornith 1.5 35B-A3B to suppress a looping behavior. On the 10 published Lexassay tasks, the edited model completed all 10 answers, up from 4 in the same setup, but its legal score remained zero. The results are limited and do not show that the edit improved legal knowledge or answer accuracy.
A home experiment reported by ThorstenMeyerAI.com found that editing a local AI model’s weights helped it finish more legal benchmark answers, but did not improve its score: the edited Ornith 1.5 35B-A3B completed 10 of 10 published tasks in one comparison, while scoring zero on the legal benchmark. The result suggests the edit reduced looping rather than adding legal knowledge, and comes from a limited experiment rather than a broad evaluation.
The experiment used directional ablation, a technique that measures a direction in a model’s internal signals and removes some of its influence from the weights. The researchers’ account says the edit was applied at half strength to output weights in 27 of the model’s 40 layers. No additional training or legal data was used, and an AI did not inspect the weights as text.
On the same experimental setup, the edited model’s completed answers rose from 4 of 10 to 10 of 10 on Lexassay’s published tasks. Its legal score stayed at zero. The report says the edited answers contained 55 wrong or invented citations and met 7 of 50 checked criteria, compared with 3 before editing. Completion, in other words, did not mean the answers were legally sound.
The account also describes a separate run using the model card’s recommended sampling settings, without a weight edit. In that setup the unchanged model answered 9 of 10 tasks. The author cautions that the two unchanged-model runs used different setups and cannot be directly compared. The figures in the updated account refer to 10 published tasks; five additional benchmark tasks remain hidden.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, graded blind by three Claude agents; figures cover the 10 published tasks
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
More Completed Answers, Same Legal Score
The experiment separates two problems that can look similar in a benchmark: a model may fail because it cannot produce a finished response, or because it lacks the knowledge and accuracy needed to answer correctly. Here, the weight edit appears to have addressed the completion problem, not the legal one. A model that stops repeating itself can still produce an answer filled with errors.
That distinction matters for people testing local AI systems for consequential work. A higher number of completed responses may look like progress, but without checks on factual content, citations and task criteria, it can give a misleading impression of capability. In this test, the legal score remained zero despite the increase in completed tasks. The findings do not establish that the method is useful for legal work or that it will have the same effect on other models.
The account says an OpenAI GPT-6 Astra session in Codex reviewed the code, figures and claims and identified errors. That independent review adds a check from another AI provider, but it does not replace replication or broader evaluation. The experiment’s result is best understood as a limited technical observation: an edit can change whether a model finishes, without making its answers reliable.
Products Worth Considering
Dell UltraSharp U2724DE 27-Inch WQHD IPS Black Monitor
Apple Mac mini (2026), M6 12-core CPU and GPU, 24GB Memory, 1TB SSD
Logitech MX Ergo S Advanced Wireless Trackball Mouse USB-C – Graphite
As an affiliate, we earn on qualifying purchases.
The MIT Idea and the Local Test
The experiment followed a report by four MIT CSAIL researchers: Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab. Their work describes “weight-space programming,” in which one general-purpose model reasons about another network’s weights and edits them directly, without a conventional training loop. In a Qwen example, the researchers reported that a model’s looping behavior was linked to a direction in its internal signals; after an edit, its mathematics score rose from 7.9% to 48.1%, according to the source account.
The home test was inspired by that work but did not reproduce its method. Instead of having an AI reason directly over the weights, a Claude Opus 5.5 session generated code to run Ornith, inspect its internal signals and remove a direction from the weights. The account identifies this as a more established technique, directional ablation. It also says Claude performed the edit and Claude agents conducted blind grading, a potential source of bias the author disclosed.
Lexassay tests models on questions involving German, US and EU law, without internet access or tools, and checks cited statutes and court decisions against primary sources. The benchmark has 15 questions, but only 10 tasks are public. The report says the five hidden tasks are kept out of public view to limit tuning against them. Its published figures cover only the public tasks, not the full benchmark.
“The edit repairs finishing, not legal knowledge.”
— Thorsten Meyer, reporting the experiment
Limits of the Published Results
The results come from one run per setting and a small development set. The account says the loop detector changed during the experiment, and that three graders came from the same model family. Although GPT-6 Astra reviewed the work and found real faults, that review was one independent check, not a second experiment. These limits make it difficult to estimate how repeatable the result is.
The comparison also depends on setup: the unchanged model answered 8 of 10 tasks in its published run but only 4 of 10 in the setup used for the direct comparison with the edited model. The report says those setups are not directly comparable. It does not establish how the edit would perform across repeated runs, the five hidden Lexassay tasks, other benchmarks or other model families.
The account gives no evidence that the edited model’s legal answers became reliable. Its reported zero legal score and citation errors point the other way, but the small task count limits broader conclusions. The article describes an AI experiment, not legal advice or a validation of AI for legal use.
Public Runs and Further Testing
The experiment’s answers, grades and citation checks are published on Lexassay.com, alongside the original run, the model-card sampling run, the edited model run and an edit control. Readers can inspect the task-level records and the method page rather than relying only on the summary figures.
The next useful test would be to repeat the comparison under the same sampling and hardware settings, with a fixed loop detector and clearly separated grading procedures. Testing the method on the five hidden tasks could also help show whether any completion change generalizes beyond the public development set. The source account does not report a planned follow-up or a date for one.
Key Questions
What changed after the model’s weights were edited?
In the same reported setup, Ornith completed 10 of 10 published tasks, compared with 4 of 10 before editing. The report says the edit targeted a looping behavior, not legal knowledge.
Did the edit improve the model’s legal score?
No. The legal score remained at zero. The account reports 55 wrong or invented citations in the 10 edited answers and says the answers met 7 of 50 checked criteria.
Was this the same technique used by the MIT researchers?
No. The experiment was inspired by the MIT work but used directional ablation: code measured the model’s internal signals and removed a direction from some weights. The MIT report described an AI reasoning directly about another model’s weights.
How strong is the evidence?
The report describes one run per setting, a small development set and a loop detector that changed during the experiment. An independent AI review found faults, but the result has not been established by repeated trials in the account.
Where can readers inspect the experiment?
The source says the runs, answers, grades and citation checks are public on Lexassay.com, along with a method page. The reported figures cover the 10 published tasks; five benchmark tasks are hidden.
Source: ThorstenMeyerAI.com
