🔍 Read the full analysis: Testing Claude And GPT-6 For Home AI Weight Editing: Zero Legal Score on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
An experiment reported by ThorstenMeyerAI.com found that a targeted weight edit helped the local model Ornith 1.5 35B-A3B finish all 10 published Lexassay tasks, compared with 4 of 10 in the same experimental setup before editing. Its legal score remained zero: the edited answers included 55 wrong or invented citations, and the model met only 7 of 50 checked criteria. The result is limited by one run per setting, a small test set and inconsistent comparison setups.
A home AI experiment found that a targeted edit to the weights of local model Ornith 1.5 35B-A3B helped it complete 10 of 10 published legal benchmark tasks, up from 4 of 10 in the same experimental setup before editing. But the model’s legal score remained zero: its answers contained 55 wrong or invented citations, according to ThorstenMeyerAI.com’s account of the test.
The experiment followed a report from four MIT CSAIL researchers on “weight-space programming,” in which one AI system changes another model’s weights. The home test was related but did not reproduce that method. Instead, a Claude Opus 5.5 session wrote code to run Ornith, identify an internal direction associated with looping, and remove that direction at half strength from the output weights of 27 of the model’s 40 layers. The account says there was no additional training or legal data, and no AI read the weights as text.
The reported comparison is specific to the same experimental setup: the edited model completed all 10 published tasks, while the unedited model completed 4. The edited answers met 7 of 50 checked criteria, compared with 3 before the edit. The account also reports that using the model card’s recommended sampling settings without any weight edit led the unchanged model to answer 9 of 10 on its original setup. Those figures should not be treated as a direct comparison across identical settings: the article says an unchanged-model published run completed 8 of 10, while the model completed only 4 of 10 on the separate setup used for the edit comparison.
OpenAI’s GPT-6 Astra, running in Codex, reviewed the code, results and claims, and identified faults, according to the report. It was an independent reviewer, not a second experiment. The article also discloses that Claude carried out the edit and Claude agents performed blind grading. Its figures refer to the 10 published Lexassay tasks; the benchmark has 15 questions overall, with five kept hidden, and the experiment’s runs are public on lexassay.com.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, graded blind by three Claude agents; figures cover the 10 published tasks
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
Finishing Answers Is Not Legal Accuracy
The result highlights a practical distinction for people evaluating local AI models: an intervention can improve whether a model completes a response without making the response reliable. Here, the edit appears to have reduced a looping problem, but the reported legal score did not move above zero. Completing a task is not the same as providing correct legal analysis or valid citations.
That matters because legal answers can mislead readers when citations are fabricated or point to the wrong authority. Lexassay says it checks cited statutes and court decisions against primary sources, rather than treating a fluent answer as evidence of correctness. On this test, the 55 citation errors and low number of criteria met are more relevant to legal usefulness than the increase in completed answers.
The experiment also offers a narrow illustration of a broader claim: removing a bottleneck may expose an ability a model already has, but should not be expected to supply knowledge it lacks. The report attributes that distinction to the MIT researchers. Its own results are consistent with the possibility that looping obstructed completion, while providing no evidence that the edit added legal expertise.
Products Worth Considering
Logitech MX Master 4 Ergonomic Wireless Mouse with Haptic Feedback, Graphite
Bose Smart Soundbar 700 with Alexa, Black
LG UltraGear 27-inch QHD OLED Gaming Monitor, 240Hz
As an affiliate, we earn on qualifying purchases.
From MIT Weight Editing to Lexassay
The MIT CSAIL work described by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab used GPT-6 Astra to reason directly about another network’s weights. In a highlighted Qwen example, the researchers reported that a model’s looping stopped and its mathematics score rose from 7.9% to 48.1%. Their method and the home experiment differ: the latter used a conventional approach called directional ablation, based on recording model activations and removing a calculated direction.
The home test selected Ornith after the experimenters checked four local models for the problem the edit was meant to address. The report describes Ornith as a 35-billion-parameter mixture-of-experts model, with about 3 billion parameters active per token, and says it was designed for coding and agent work rather than law. On the published Lexassay tasks, two of its answers did not arrive before a limit of 62,000 output tokens; the remaining transcripts showed repeated reasoning, according to the report. Other tested models either did not loop or could not be tested under comparable conditions.
Lexassay’s full benchmark consists of 15 legal questions covering German, US and EU law, with no internet access or tools. Ten tasks are public and five are hidden to reduce the possibility of tuning directly to the test. The article says its reported figures use only the published tasks. Its results therefore describe a small development set, not a broad evaluation of legal capability.
“The model finishes is not the answer is right.”
— ThorstenMeyerAI.com’s experiment report
Small Sample, Uneven Comparisons
The results are preliminary. The account reports one run per setting, only 10 published tasks, and a loop detector that changed during the experiment. It also says three graders came from the same model family, which limits how independent their assessments were. GPT-6 Astra’s review identified faults, but it does not turn the test into a replicated evaluation.
Comparison conditions also differ. The unchanged model completed 4 of 10 tasks in the setup used for the edited-versus-unedited comparison, but 8 of 10 in a published run and 9 of 10 with recommended sampling settings on its original setup. The report explicitly says those two local setups are not directly comparable. The available information does not establish how much of the completion change came from the weight edit rather than the experimental setup, sampling or other run-to-run variation.
It is also unclear whether the edit would generalize to other legal questions, other models or future runs. The five hidden tasks were not part of the reported figures, and the account does not report a successful legal score on them. No result here establishes that the method is suitable for legal use.
The immediate next test would be to evaluate the edited model on Lexassay’s five hidden tasks and repeat the experiment across multiple runs with the same model settings and a fixed loop detector. That would help distinguish a repeatable effect from variation between local setups and individual generations.
Readers can inspect the published answers, grades and citation checks on lexassay.com, according to the report. Further testing would need to show whether the completion improvement persists while citation accuracy and substantive legal criteria improve. Until then, the finding supports only a limited claim: this particular edit was reported to help Ornith finish more of the public tasks, not to make its legal answers dependable.
Key Questions
What did the weight edit change?
Claude Opus 5.5 wrote code that measured an internal direction associated with looping and removed it at half strength from the output weights of 27 of Ornith’s 40 layers. The report says the process did not train the model or add legal data.
Did the model become better at law?
The reported legal score stayed at zero. The edited answers met 7 of 50 checked criteria and contained 55 wrong or invented citations, according to the experiment report.
How did task completion change?
In the same setup used for the edit comparison, the model completed 10 of 10 published tasks after editing, compared with 4 of 10 before. Other reported completion figures came from different setups and are not directly comparable.
Was this the same method used by MIT researchers?
No. The home experiment used directional ablation: code ran the model and altered its weights based on measured internal activity. The MIT example described in the source involved an AI reasoning directly over another model’s weights.
Can the result be treated as legal advice or proof of reliability?
No. The source says the experiment is not legal advice. It involved one run per setting and a small set of published tasks, and the reported legal score remained zero.
Source: ThorstenMeyerAI.com
