The AI Model That Tested Best Isn’t What I Run For News

🔍 Read the full analysis: The AI Model That Tested Best Isn’t What I Run For News on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says he tested 71 AI model configurations on the news-writing task used by his websites and chose GPT-6 Luna for production, even though GPT-6 Astra scored higher. In his October 13 report, Meyer said Astra received the top writing scores but cost about 100 times as much per article, while Luna delivered consistent results at a lower price.

Meyer’s test used his production prompt and two real news sources: a longer product filing and a short political news item. Each model configuration produced articles in English and German. Two AI judges assessed source fidelity, fabrication, clarity about uncertainty, news value, structure and polish, with an additional language-quality measure for German. Meyer also ran a follow-up test of 10 more articles for the strongest configurations, mostly using short sources.

GPT-6 Astra scored 4.00 out of 5 in both languages in the standard test and received a perfect source-fidelity score there. In the follow-up, it averaged 3.93 across 12 articles, with a reported 95% bootstrap interval of 3.83–4.02. Meyer put its cost at about $0.12 per article and its writing time at roughly 50 seconds. GPT-6 Luna averaged 3.67 in English and 3.68 in German in the standard test; in the follow-up it scored 3.67, with source fidelity of 4.71. Meyer reported a production cost of about $0.0013 per article and a median writing time of 17 seconds.

The report says the test covered API, subscription, free-tier and locally run open-weight models. Meyer judged differences below about 0.3 points in the two-article standard test as potentially noise and treated the larger follow-up as a more reliable ranking. He also required output to be usable by the same repair step as his production pipeline before counting it as valid.

Cost and Reliability Shaped the Choice

The results illustrate a practical distinction between the highest-scoring model and the model a publisher can justify using at scale. On Meyer’s reported prices, Astra’s roughly $0.12 per article is far above Luna’s approximate $0.0012–$0.0013. For a high-volume publisher, even a quality advantage may not warrant that expense if a less costly model consistently meets its editorial standard.

The findings also point to risks beyond the main model’s average score. Meyer says two fallback models generated invented quotations or failed to produce valid articles. A system’s backup route can therefore affect published output, even when its primary model performs well. These results are specific to Meyer’s prompts, sources, evaluation setup and workflow; they do not establish a universal ranking of models for journalism.

A Benchmark Built Around News Work

Meyer’s central question was not which model had the strongest general benchmark result, but which one could turn his actual source material into articles that his sites could use. The evaluation ran from September 8 through October 13, 2026 and compared 71 configurations under a shared writing task. The short political source was included because Meyer said much of his news production begins with brief material.

For the standard scoring, the two judges were GPT-6.1 Sol and Claude Opus 5.5, and both were shown the original source. Meyer says the strongest configurations then faced a 10-article robustness run, with results reported alongside bootstrap intervals. The test’s scope matters: its scores describe performance in this particular news-production setup, not general intelligence or writing ability across every publication and subject.

Meyer also varied GPT-6 Luna’s reasoning effort across six settings. He reported that the scores remained within the test’s noise range, while the maximum setting cost about 12 times more and took about 12 times longer than the no-effort production setting. He said his system would stay on the no-effort setting.

“For a high-volume task, the right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, in his October 13, 2026 report

Limits of the Reported Comparison

The results come from one publisher’s workflow, two initial sources and a limited follow-up sample, rather than a broad, independently replicated study. Meyer reports that two AI judges scored the articles, but the report does not establish how closely those scores match assessments by human editors or readers. The models’ performance could differ with other prompts, topics, languages, source lengths or editorial standards.

It is also unclear from the supplied report whether model prices will remain stable or whether the findings have been independently checked. Meyer’s cost figures and scores should be understood as reported test results, not guarantees of future performance. The article identifies GPT-6 Luna as the production choice, but does not provide a longer-term comparison of publication errors, reader response or editorial corrections under that setup.

Luna Remains the Production Model

Meyer says GPT-6 Luna remains in production, while GPT-6 Astra was ruled too expensive for his news use on October 7. He also says the two underperforming fallbacks were removed from the route that day. The report does not describe a separate independent audit of the revised routing or give a date for another evaluation.

The next useful evidence would be performance over a longer production period, particularly on a wider range of sources and across routine editorial checks. Until then, the reported conclusion is limited: in Meyer’s test, Luna provided the cost and consistency he wanted, while Astra led the measured writing scores.

Key Questions

Which model scored highest in Meyer’s news-writing test?

GPT-6 Astra scored 4.00 out of 5 in English and German in the standard test. It averaged 3.93 across 12 articles in the follow-up run, according to Meyer.

Why did Meyer choose GPT-6 Luna for production?

Meyer said Luna’s follow-up score was 3.67 and that it produced valid articles consistently at about $0.0013 per article in production. He estimated Astra cost about $0.12 per article, roughly 100 times as much.

Did higher reasoning effort improve Luna’s articles?

Meyer reported that results across six reasoning settings were within the test’s noise range. The maximum setting cost and took about 12 times more than the no-effort setting without a reported reliable quality gain.

What problems did the fallback models have?

Meyer said GPT-4.1 nano received low source-fidelity scores and produced invented quotations. He said locally run gpt-oss-120b produced no valid articles in four attempts and also generated fabricated named quotes. Both were removed from his news route on October 7.

Do these scores show which model is best for every newsroom?

No. The test used Meyer’s prompts, two initial sources, English and German editions, and AI judges. Its results describe that particular workflow and do not establish a general ranking for other publishers or tasks.

Source: ThorstenMeyerAI.com