🔍 Read the full analysis: What One Matrix Edit Did For Qwen’s JSON Tab Escaping: 49 To 70 Of 120 on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
An OpenAI GPT-6 Astra agent edited one matrix in Qwen3-4B-Instruct-2507, raising its exact score on a frozen test of JSON escaping from 49 of 120 cases to 70. The result is narrow: 50 cases remained wrong, and the edit did not improve a separate table-row extraction task, which stayed at zero of 30.
An OpenAI GPT-6 Astra agent raised Qwen3-4B-Instruct-2507’s exact score on a frozen test of copying special characters into JSON from 49 of 120 cases to 70, after editing one matrix in the model. The reported gain was 21 additional correct answers, with no cases that were previously correct becoming wrong; the result applies only to this narrow, synthetic test.
The task tested whether the model could copy short text containing tabs and other special characters into a JSON object without changing the source. A real tab must be represented in JSON as the escape sequence \\t and decode back to the same character. Answers counted as correct only when a standard JSON parser recovered the exact original text; valid JSON that altered or lost a character still failed.
According to the experiment report, the agent made a calculated rank-one update to the output projection of the model’s last multilayer perceptron layer. It did not use gradient training, and the report says the model’s other 397 stored tensors remained bit-identical to the original. On the frozen final test, the edited model scored 70 of 120, compared with 49 of 120 before the edit: a gain of 17.5 percentage points.
The test followed an earlier attempt to improve extraction of fields from synthetic business messages. That effort had not improved the frozen test score: both model versions scored 97 of 200. The report says both versions also produced 47 invalid answers, mostly by putting raw tab characters in JSON strings. That recurring error informed the new experiment’s focus. The updated report says the figures refer to 10 published Lexassay tasks and that runs are public on lexassay.com.
Infographic 01 · The measured result
Where the edit helped, and where it did not
Qwen3-4B-Instruct-2507, original versus one edited matrix, on synthetic texts that must survive the trip into JSON without losing a character.
Final test · 120 cases by group
- Texts with tabsthe target failureOriginal19/30Edit30/30
- Line breaksalready perfectOriginal30/30Edit30/30
- Mixed special characterstabs, quotes, backslashesOriginal0/30Edit10/30
- Second table rowstill unsolvedOriginal0/30Edit0/30
Development, validation and controls
- Developmentused to fit the edit · not independentOriginal24/40Edit40/40
- Validationnew cases · 9 wins, 0 lossesOriginal10/40Edit19/40
- Random directioncontrol on validation · same strengthOriginal10/40Edit10/40
- Plain-text tabscontrol · real tabs must surviveOriginal10/10Edit10/10
Exact answer = after normal JSON parsing, the text field matches the source text character for character. Same prompt, temperature 0, seed and output limit for both models; saved and reloaded edit, no hooks, no grammar constraints, no repair. Final test, validation and controls frozen before any independent output of the edited model. Fifty of 120 final answers are still wrong.
Sources: our experiment records, 5 Nov 2026 (Lexassay research, internal): experiment 3 result report and final result by GPT-6 Astra working as a Codex agent (OpenAI). Synthetic test cases, not published.
Powered by Thorsten Meyer AI
A Narrow Gain in JSON Reliability
The result suggests a small, targeted weight edit can improve a specific behavior without retraining the entire model. Correctly escaping control characters matters in data exchange between software systems: an answer that looks close but changes a tab can fail exact comparisons or corrupt information passed into another process. The test’s exact-match standard makes that kind of error visible.
But the score does not establish that Qwen is generally more reliable at producing JSON, or that the edited model is suitable for production. Fifty of 120 cases still failed, and the reported edit was designed around a previously observed failure rather than a broad search for model weaknesses. The finding is best read as evidence of a measurable improvement on one defined behavior, not a general performance upgrade.
The reported absence of regressions within these 120 test cases is useful, but it does not show that the edit has no side effects elsewhere. The experiment’s claims have not yet received the independent second-AI review that was performed on an earlier project in the series.
Products Worth Considering
Logitech MX Master 4 Ergonomic Wireless Mouse with Haptic Feedback, Graphite
Bose TV Speaker (2nd Gen) – Compact Soundbar with Bluetooth and HDMI‑ARC Connectivity, Clear Dialogue, Dolby Atmos, Deep Bass, Remote Contro
Apple Watch Series 12 GPS 42mm
As an affiliate, we earn on qualifying purchases.
The Earlier Test Pointed to Tabs
The experiment is the third in a series described by ThorstenMeyerAI.com as tests of AI systems editing or reviewing neural-network weights. In an earlier project, Claude edited a local model named Ornith; a later review by GPT-6 Astra identified problems in that work, including a resume bug and command-line and weight-conversion issues. Those findings concern the earlier experiment, not this Qwen result.
Before the JSON escaping test, GPT-6 Astra had also tried editing Qwen for a different task: extracting fields such as email addresses and due dates from synthetic business messages. That edit did not change the final score, which was 97 of 200 for both versions. The shared pattern of invalid answers involving raw tabs provided a concrete target for this experiment.
The report says a more explicit prompt did not improve the original model’s score on the initial screening set: both the original prompt and a stronger escaping prompt scored 16 of 32. The new experiment therefore tested a weight change against the known error. The frozen final test is presented as distinct from that screening set, so its 49-to-70 result should not be confused with the earlier 16-of-32 screening result.
“The starting hint for experiment 3 “came from the previous attempt.””
— The experiment report
The Test Does Not Settle General Reliability
The report describes the test cases as synthetic. It does not establish how often the same errors occur in ordinary use, or whether the score gain carries over to other prompts, datasets, model versions or software environments. The available material also does not give a broad evaluation of the edited model’s behavior beyond the reported tasks.
It remains unclear whether the edit causes regressions on tasks that were not tested. The statement that none of the 120 cases broke applies to the frozen final test, not every possible input. The separate table-row extraction result remained 0 of 30, indicating that this edit did not fix that reported task.
The experiment has not yet been checked by a second AI, according to the source. The article’s author says a Claude agent helped research and draft the account and check figures against stored results, but that is not the same as an independent technical review of the experiment’s methods or conclusions.
Independent Checks Are Still Pending
The next useful step is an independent examination of the experiment records, including how the matrix update was calculated, how the final test was frozen and whether the published runs reproduce the reported scores. The source says the runs are public on lexassay.com, giving other reviewers material to inspect.
Further testing would need to establish whether the improvement holds on new, non-synthetic examples and whether the edit changes performance on unrelated tasks. Until those checks are reported, the confirmed result remains a 21-case gain on one frozen 120-case test, alongside unresolved questions about generalization and possible side effects.
Key Questions
What changed in the Qwen model?
The GPT-6 Astra agent made a rank-one update to one matrix, the output projection in the model’s last MLP layer. The report says it did not use gradient training and that the other 397 stored tensors were unchanged.
What does “49 to 70 of 120” measure?
It is the number of cases answered exactly correctly on a frozen test of copying text with tabs and other special characters into JSON. The score rose from 49 correct answers to 70, a gain of 21 cases, or 17.5 percentage points.
Did the edit fix JSON escaping across the board?
No. The edited model still got 50 of the 120 cases wrong. The result covers the reported test only and does not establish general JSON reliability.
Was the result independently reviewed?
The source says the experiment has not yet had a second-AI review. It reports that runs are public on lexassay.com, but public availability is not itself an independent verification.
Did the edit help with other tasks?
The source reports that table-row extraction stayed at 0 of 30. It also describes an earlier business-message field-extraction attempt where both versions scored 97 of 200, so the JSON escaping gain should not be treated as a general improvement.
Source: ThorstenMeyerAI.com
