📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva LLM, built from scratch with extensive Italian data, demonstrates impressive technical performance but underwhelms on real-world academic benchmarks. This challenges assumptions about scale and native-language investment in European sovereign AI development.
Italy’s Minerva-3B, a large language model trained entirely from scratch on over 2.5 trillion tokens with approximately 50% Italian content, scored only 4.9% on the INVALSI Italian school-exam benchmark, despite its impressive technical architecture. This stark result raises questions about the effectiveness of large-scale native-language training at current parameter levels and the strategic assumptions underpinning European sovereign AI initiatives.
Minerva was developed by Sapienza University of Rome’s NLP group, led by Roberto Navigli, using Italy’s national supercomputing infrastructure and funding from Italy’s PNRR initiative. The project trained models ranging from 350 million to 7 billion parameters, with the 3B model being publicly released along with training data and code, exemplifying transparency in European AI efforts.
Despite these efforts, Minerva-3B’s performance on the INVALSI benchmark was only 4.9%, a near-chance level for a model trained on 660 billion tokens with half Italian data. Researchers concluded that while dataset composition matters, overall dataset size and model parameters are more critical for complex language tasks, suggesting current investments may be insufficient for true language understanding at this scale.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Large Language Models (LLMs)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
Engineering a Small AI Language Model: Training, Evaluation, and Deployment Without Myth
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
Introduction to Machine Learning with Python: A Guide for Data Scientists
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
AI Engineering: Building Applications with Foundation Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications of Scale and Investment in European Sovereign LLMs
The results from Minerva challenge the assumption that simply increasing native-language data and parameters guarantees high performance on real-world tasks. It highlights that European AI projects may need to consider even larger investments and more sophisticated architectures to achieve meaningful language comprehension. This has broad implications for national AI strategies across Europe, emphasizing the importance of realistic scaling and resource commitments to justify sovereign AI initiatives.
European Sovereign LLM Strategies and the Minerva Benchmark
Italy’s Minerva project represents a significant effort in building a European sovereign language model from scratch, utilizing extensive Italian data and national infrastructure. It contrasts with approaches like Portugal’s AMÁLIA, which relies on continuation pre-training of multilingual models with smaller native-language datasets. Minerva’s publicly released weights and data mark a departure from proprietary models, aiming for transparency and local control. However, its poor performance on academic benchmarks reveals the complexity of translating technical scale into practical language understanding.
“Despite training on 660 billion tokens with 50% Italian content, the model’s performance on INVALSI tests was near chance.”
— Research team evaluating Minerva
Unresolved Questions About Scaling and Practical Effectiveness
It remains unclear whether increasing model size further or adopting different training methodologies will significantly improve Minerva’s performance on complex language tasks. The evaluation was limited to the INVALSI benchmark, and broader assessments are needed to determine if the findings generalize across other real-world applications. Additionally, the ongoing iterative development of Minerva suggests future improvements are possible, but the current results highlight a fundamental challenge in sovereign-language modeling at existing scales.
Next Steps for European Sovereign Language Models
Researchers and policymakers will likely reevaluate the scale and resource commitments necessary for effective native-language models. Further training, larger datasets, and architectural innovations may be pursued to bridge the gap between technical performance and practical language understanding. The Minerva team is expected to continue refining their models, with upcoming evaluations on broader benchmarks and real-world tasks providing more clarity on the future of European AI sovereignty.
Key Questions
Why did Minerva perform poorly on the Italian academic benchmark?
Despite extensive training on large amounts of Italian data, the results suggest that current parameter scales and dataset sizes are insufficient for complex language understanding tasks like academic assessments.
Does this mean large native-language models are ineffective?
Not necessarily. The results highlight that scale and investment levels are critical. Larger models or different training strategies may improve performance, but current efforts show significant challenges at existing scales.
How does Minerva compare to other European sovereign models?
Minerva is notable for training from scratch with publicly available data and weights, contrasting with models like Portugal’s AMÁLIA, which rely on continuation pre-training. Its performance challenges assumptions about the effectiveness of native-language training at current scales.
What are the implications for European AI policy?
The findings suggest that European countries may need to commit more resources and adopt larger-scale architectures to develop truly effective sovereign-language models, impacting future AI strategies and funding priorities.
Source: ThorstenMeyerAI.com
