🔍 Read the full analysis: Claude Edits, GPT-6 Checks: Safe Home AI Weight Programming Test on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A home experiment edited the weights of the local model Ornith 1.5 35B-A3B to reduce repetitive output. On the same setup, it answered all 10 published Lexassay tasks rather than 4, but its legal score remained zero and its answers contained 55 wrong or invented citations. The experiment was limited and does not show that weight editing improves legal knowledge.
A home experiment reported by ThorstenMeyerAI.com found that editing the weights of a local AI model helped it finish more legal benchmark tasks, but did not make its answers more legally accurate. On the same test setup, Ornith 1.5 35B-A3B completed 10 of 10 published tasks after an edit, compared with 4 of 10 before it; its legal score stayed at zero.
The experiment ran from Wednesday evening to Thursday morning and was inspired by an MIT CSAIL study of what its authors call weight-space programming. The home test used a related but different method: a Claude Opus 5.5 session wrote code to run Ornith, identify a direction associated with repetitive output, and remove that direction at half strength from the output weights of 27 of the model’s 40 layers. The work involved no training and added no legal data.
That distinction matters. The MIT work described an AI reasoning directly over another model’s weights. In this home experiment, the AI helped write an analysis and editing procedure; it did not inspect the weights as text and independently devise the change in the same way. OpenAI’s GPT-6 Astra, running in Codex, reviewed the code, results and claims. The source says that review identified real faults. It was an independent review of one experiment, not a second experiment.
The edited model’s legal score remained 0. Across the 10 published tasks, graders marked just 7 of 50 checked criteria as met, up from 3 before the edit, and the edited answers included 55 wrong or invented citations. A separate comparison also showed the importance of settings: the unedited model answered 9 of 10 tasks with sampling settings recommended by its model card. The article says that result used a different setup, so it should not be treated as a directly comparable measure of the edit’s effect.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, graded blind by three Claude agents; figures cover the 10 published tasks
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
Finishing Answers Is Not Legal Accuracy
The result illustrates a practical limit of model editing: reducing a failure mode can improve task completion without supplying missing knowledge. A model that stops repeating itself may produce a complete answer, but completeness does not establish that its law, reasoning or citations are correct. In this test, the unchanged legal score and the citation errors make that distinction concrete.
That matters for people evaluating local AI systems for legal or other high-stakes work. A count of completed tasks can make an intervention look successful if readers do not also examine correctness. Lexassay checks cited statutes and court decisions against primary sources, and the reported results show why those checks matter. The experiment is a small test, not evidence that weight editing is safe or effective for legal advice. The article explicitly says it is not legal advice.
The comparison also points to a simpler source of apparent improvement: sampling configuration. The model-card settings produced more completed responses without any weight edit. Since the runs used different setups, the reported figures do not establish how much of the change came from the edit itself, how much from other settings, or how reliably either outcome would repeat.
Products Worth Considering
Jackery Explorer 1000 v2 Portable Power Station
Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Bluetooth 5.3, USB-C Charging, IPX4, Black
Bose QuietComfort Wireless Noise-Cancelling Headphones, White Smoke
As an affiliate, we earn on qualifying purchases.
From MIT Weight Editing to Lexassay
The home test followed an MIT CSAIL report by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab. In one example described in the source, GPT-6 Astra identified a direction associated with looping in a Qwen model; after an edit, its reported mathematics score rose from 7.9% to 48.1%. The researchers cautioned that editing might remove an obstacle to an existing ability, but would not add an ability a model lacked. Those figures describe the MIT example, not the home legal test.
Lexassay is the experimenter’s benchmark for questions on German, U.S. and EU law. It runs models without internet access or tools and checks citations against primary sources. The benchmark contains 15 questions, with 10 tasks published and five kept hidden; the figures in this report refer only to the published tasks. The hidden tasks are intended to prevent direct tuning to every test question.
Ornith was chosen because the source says it showed the relevant failure pattern among the local models considered. It is a mixture-of-experts model described by its publisher as having about 35 billion parameters in total, with about 3 billion active per token, and is aimed at coding and agent work rather than law. On the published legal benchmark, its earlier score was 1 out of 100. Two tasks in that run did not produce completed answers before a 62,000-token limit; the source reports that the cut-off output was highly repetitive.
“The edit repairs finishing, not legal knowledge.”
— Thorsten Meyer, describing the home experiment
Limits of the One-Run Comparison
The results come from one run per setting, a small development set and a loop detector that changed during the experiment. The source also notes that three graders came from the same model family, creating a potential evaluation limitation. GPT-6 Astra’s review added a separate check, but the account does not provide enough detail here to establish how every fault was resolved or how the corrected results changed.
The unedited model’s completion rate differed across setups: it answered 8 of 10 in a published run and 4 of 10 in the setup used for the direct comparison, while model-card sampling yielded 9 of 10 without an edit. The article says these setups are not directly comparable. The evidence therefore does not establish a stable completion-rate advantage from editing, and the unchanged legal score offers no indication of improved legal performance.
It is also unclear whether the edit would generalize to other questions, models, hardware or sampling settings. The source describes public answers, grades and citation checks, but the reported test covers only the 10 published tasks. The five hidden tasks are not included in the figures, and no repeated-run results are provided.
The experiment’s answers, grades and citation checks are available through Lexassay, according to the article, along with a method page and runs covering the original setup, model-card sampling, the weight-edited model and an edit control. Those records give readers a way to inspect the reported results rather than relying only on the summary figures.
The next useful evidence would be repeated comparisons under matched settings, with the editing procedure and evaluation criteria held constant. Results on the five unpublished benchmark tasks could also test whether any change extends beyond the questions used during development. Until such evidence is reported, the finding should be read narrowly: in one setup, a particular edit helped one local model finish more published tasks, while its measured legal score remained zero.
Key Questions
What did the weight edit change?
It removed half the strength of a measured direction associated with looping from the output weights of 27 of Ornith 1.5 35B-A3B’s 40 layers. The experiment did not train the model or add legal data.
Did the edited model become better at law?
Not according to the reported legal score. It remained at zero, despite completing more of the 10 published tasks. The article also reports 55 wrong or invented citations in the edited answers.
Was this the same method used in the MIT study?
No. The home experiment used code to measure and remove a direction in the model’s weights. The MIT example described an AI reasoning directly over another model’s weights. The home author characterizes the test as related, but simpler.
How reliable are the results?
The report gives one run per setting and notes a small development set, a changing loop detector, grader-family overlap and non-comparable model setups. The figures are limited to the 10 published tasks, so they do not establish a repeatable effect or performance on the five hidden tasks.
Can people use the results as legal advice?
No. The experiment article says it is not legal advice. Its findings concern a limited test of model behavior and report that legal accuracy did not improve.
Source: ThorstenMeyerAI.com
