📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva, a European sovereign language model trained from scratch, demonstrates that larger native-language datasets do not guarantee high performance on complex tasks. Its results challenge assumptions about scale and investment in national AI projects.
Italy’s Minerva, a large-scale European sovereign language model trained from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored only 4.9% on the INVALSI Italian school-exam benchmark, highlighting significant challenges in achieving country-specific knowledge depth despite substantial investment.
The Minerva project, led by Sapienza University of Rome and supported by Italy’s national research infrastructure, trained models ranging from 350 million to 7 billion parameters on a dataset of 2.5 trillion tokens, with half of the data in Italian. Despite this extensive effort, the 3B parameter version scored near chance on the INVALSI exam, a key indicator of academic language understanding in Italy.
Researchers from the project concluded that, while dataset composition matters, the overall size of the dataset and the number of parameters are more crucial for complex language tasks. This empirical result suggests that even large native-language training may not be sufficient at current scales, raising questions about the investment’s effectiveness and the necessary scale for meaningful country-knowledge modeling.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Large Language Models (LLMs)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
Engineering a Small AI Language Model: Training, Evaluation, and Deployment Without Myth
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
Natural Language Processing with Transformers, Revised Edition
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
AI Engineering: Building Applications with Foundation Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications of Scale and Investment in Sovereign-Language Models
The results from Minerva challenge the assumption that simply increasing native-language data and model size guarantees high performance on complex, country-specific tasks. For policymakers and researchers, this highlights the need to reconsider the scale of investment and the architecture choices in developing national AI models. The findings suggest that more extensive or different training strategies may be required to produce models with genuine country-knowledge depth, influencing future European AI strategies and funding priorities.
Background on European Sovereign-Language Model Strategies
Italy’s Minerva project represents a significant effort within Europe’s broader push for sovereign AI infrastructure, contrasting with approaches like Portugal’s AMÁLIA, which layered specialization onto multilingual foundations. Minerva trained from scratch on a massive dataset, leveraging Italy’s national research infrastructure, and publicly released its weights and code. Despite these efforts, its performance on academic benchmarks reveals a complex challenge: scaling data and parameters alone may not suffice for high-level language understanding within a national context.
Unresolved Questions About Model Scaling and Performance
It remains unclear what specific architectural or training adjustments could improve Minerva’s performance on complex, country-specific tasks. The ongoing research aims to explore whether larger models, different data strategies, or specialized training can bridge the gap between dataset size and task complexity. Additionally, the long-term impact of these findings on European AI policy and funding remains to be seen.
Next Steps for European Sovereign-Language AI Development
The Minerva team plans to continue iterating on training methodologies, including larger models and refined data strategies, with upcoming case studies in 2025. Policymakers and research institutions are expected to reassess investment scales and architectural choices based on these empirical findings, potentially shifting European AI strategies toward more targeted or scaled approaches. Further benchmarking and evaluation on complex tasks will determine whether these adjustments succeed.
Key Questions
Why did Minerva score so poorly on Italian academic tests despite large-scale training?
Researchers concluded that dataset size and model parameters are more influential than native-language data volume alone, and current scales may still be insufficient for complex country-specific tasks.
Does this mean training from scratch is ineffective for national models?
Not necessarily; it suggests that scale and training strategies need to be carefully calibrated, and that simply training from scratch at current sizes may not produce the desired depth of country-specific knowledge.
What are the implications for European AI policy?
The findings imply that European investments should consider larger models and different data strategies, as current efforts may not yet achieve the intended level of country-specific expertise.
Will future models overcome these limitations?
Ongoing research aims to explore whether larger models, different training approaches, or additional data can improve performance, but definitive solutions are still under development.
Source: ThorstenMeyerAI.com
