I Tested 71 AI Configurations On Real News Workflows

🔍 Read the full analysis: I Tested 71 AI Configurations On Real News Workflows on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says a test of 71 AI configurations on real news-writing tasks found that GPT-6 Astra produced the highest-scoring articles, while he selected the cheaper GPT-6 Luna for production. In results published October 13, Meyer reported that Astra scored 4.00 out of 5 in both English and German, but cost about $0.12 per article; Luna scored lower in the longer test, at 3.67, and cost about $0.0012 per article.

The evaluation ran from September 8 to October 13, 2026, using the same production prompt Meyer says his publishing pipeline uses. It covered API and subscription models, free-tier options available through OpenRouter, and open-weight models run on his own hardware. Each source was used to produce both an English and a German article.

The test included two sources: a 6,340-character product filing and a 619-character political news item. Meyer said the shorter source better reflected much of his day-to-day work. Two AI judges, GPT-6.1 Sol and Claude Opus 5.5, rated articles on a five-point scale for source fidelity, fabrication, clarity about uncertainty, news value, structure and polish, with German-language quality also assessed.

In a follow-up robustness run, the strongest configurations wrote 10 additional articles, mostly from short sources. Meyer said GPT-6 Astra averaged 3.93 across 12 articles, while Luna scored 3.67. He reported that every Luna article was valid under the pipeline’s production repair step. The findings are the author’s evaluation of a particular workflow, not an independent comparison of all AI writing uses.

Why Luna Won the Production Slot

The results illustrate a practical choice for publishers using AI: the top score may not be the best operational option if its cost is much higher. Meyer reported Astra at about $0.12 per article and roughly 50 seconds, compared with Luna at about $0.0012 per article and 17 seconds. He said the price difference led him to rule out Astra for routine news production, despite its stronger scores.

Meyer’s account also points to reliability risks beyond the primary model. He said two fallback systems generated invented quotations, and one failed to produce any valid articles in its test sample. For newsrooms, those findings underline the need to test the full publishing route, including backup models and output validation, rather than relying only on a model’s headline score.

The results do not establish that Luna is the best choice for other publishers. Performance can depend on the prompt, source material, languages, model versions, editorial standards and cost assumptions. Meyer’s test is most useful as an example of how task-specific checks can reveal trade-offs that general-purpose rankings may miss.

How the Newsroom Test Worked

Meyer framed the project around a specific editorial task: turning source material into a finished article using the prompt in his own production pipeline. That differs from broad model evaluations, which may test general knowledge or coding rather than source-grounded reporting. The test’s two source lengths were intended to capture both a detailed document and a much thinner news item.

For the initial comparison, Meyer set a rule that score gaps under about 0.3 points across two test articles should be treated as noise. He said the follow-up run offered a more reliable ranking. He also reported 95% bootstrap intervals for the robustness results: Astra’s interval was 3.83–4.02 and Luna’s was 3.59–3.76.

The test also compared six reasoning-effort settings for Luna. Meyer reported no consistent quality improvement as effort increased. The maximum setting scored 3.58 in English and 3.57 in German, compared with 3.67 and 3.68 at the production setting, while costing about 12 times as much and taking about 12 times as long.

“The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, author of the test

Limits of the Model Comparison

The report describes a single publisher’s workflow and a limited set of source material. Although the robustness run added 10 articles, the published account does not provide enough detail to establish how results would generalize across other news topics, prompts, editorial review processes or publishing systems. Meyer’s scoring was conducted by two AI judges; the source does not report a human-editor scoring comparison.

It is also unclear from the account whether other publishers would see the same per-article prices, since costs can depend on usage, access terms and configuration. The article reports model names and scores as of the test period, but does not establish how performance may change with later versions. The findings should be read as Meyer’s measured results, not as a universal ranking.

Meyer said the two weak fallback models were removed from his route on October 7. The source text ends with an incomplete description of the replacement routing setup, so it does not fully specify the current failover chain. It also does not report whether a human editor reviews every article before publication.

What Meyer Should Test Next

Meyer’s published account presents the test period as complete but does not announce a date for another evaluation. The immediate operational change he describes is the removal of GPT-4.1 nano and gpt-oss-120b from the news route, with GPT-6 Luna used for production. The full replacement and fallback arrangement is not clear in the source.

Further reporting would benefit from testing more sources and topics, publishing fuller score breakdowns, and comparing the AI judges’ ratings with those of human editors. Readers will also need updated results if models, pricing or the production prompt change. Until then, the reported conclusions apply to Meyer’s tested configurations and workflow.

Key Questions

Which model scored highest in the test?

GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test. Meyer reported a 3.93 average across 12 articles in its robustness run.

Why did Meyer choose GPT-6 Luna for production?

Meyer said Luna produced consistently usable articles at about $0.0012 per article, compared with about $0.12 for Astra. Luna’s reported robustness score was 3.67.

Did more reasoning effort improve Luna’s articles?

Not consistently in Meyer’s results. He said score differences across six effort settings were within noise, while the maximum setting took about 12 times as long and cost about 12 times as much as the production setting.

What problems did the test find in the fallback models?

Meyer reported that GPT-4.1 nano produced invented quotations and scored 1.00 for source fidelity. He said local gpt-oss-120b produced no valid articles in four attempts and also invented named quotations.

Can these results be applied to other newsrooms?

Not automatically. The findings reflect Meyer’s prompt, sources, languages, models and pipeline. Other publishers would need to test their own workflows before drawing the same conclusions.

Source: ThorstenMeyerAI.com