📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.

Italy’s Minerva-3B, a large-scale language model trained entirely from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored only 4.9% on the INVALSI Italian school-exam benchmark, revealing significant challenges in developing country-specific language models despite substantial investment.

The Minerva project, led by Sapienza University of Rome and supported by Italy’s national research and supercomputing infrastructure, aimed to produce a high-performance Italian language model with open weights, data, and code. It trained models ranging from 350 million to 7 billion parameters, using a dataset of 2.5 trillion tokens, half of which was Italian.

While Minerva-3B outperformed comparable multilingual models on Italian benchmarks, its performance on the actual Italian academic content tests was near chance, with a score of just 4.9%. Researchers noted that, despite the large dataset and native-language focus, the model lacked sufficient depth in country-specific knowledge, suggesting that size alone is insufficient for complex language tasks.

Implications for European Sovereign-Language Models

This development highlights that even large-scale, native-language training may not guarantee deep country-specific knowledge in language models. It questions the assumption that more native-language data and larger models automatically lead to better performance on complex, real-world tasks. The findings suggest that future European AI strategies may need to consider even greater resource investments or different architectural approaches to achieve desired levels of language understanding, impacting national AI sovereignty efforts and policy debates.

European Sovereign-Language Model Strategies and Challenges

Italy’s Minerva project represents a significant effort in the European sovereign-LLM movement, training a large Italian language model from scratch with substantial institutional support, including Italy’s national supercomputing resources and research funding. The project aimed to demonstrate that a country could develop its own high-performance language models without relying on multilingual or proprietary datasets. However, the low performance on academic benchmarks underscores ongoing debates about the scale and scope needed for effective language-specific models, especially in non-English contexts. Prior efforts, like Portugal’s AMÁLIA, took different approaches, layering specialization onto multilingual foundations, but faced similar questions about depth and resource requirements.

“While our dataset and model size are substantial, the performance gap on complex language tasks suggests that more targeted or larger-scale investments are necessary.”

— Research team member, Orlando et al.

Unresolved Questions About Model Scaling and Performance

It remains unclear how much larger or more specialized models would need to be to achieve meaningful results on complex country-specific tasks. The current findings do not specify the threshold at which native-language models become effective, nor do they clarify whether architectural changes could improve outcomes. Additionally, the long-term implications for European AI sovereignty strategies are still under discussion, with ongoing debates about resource allocation and model design.

Next Steps in European Sovereign-Language AI Development

The research team plans to continue iterating on Minerva, including exploring larger models and different training methodologies. Future evaluations will focus on whether increased scale or architectural adjustments can improve performance on complex language tasks. Policymakers and AI strategists are likely to reassess resource commitments in light of these findings, potentially emphasizing larger investments or alternative approaches to achieve deeper country-specific language understanding. Further transparency and collaborative research are expected to shape the next phase of European sovereign-LLM development.

Key Questions

Why did Minerva-3B perform poorly on the Italian academic benchmark?

Despite being trained on a large, native-language dataset, the model lacked sufficient depth in country-specific knowledge, indicating that dataset size and parameters alone are insufficient for complex language understanding.

What does this mean for European AI sovereignty efforts?

The results suggest that current investments may need to be increased or rethought, as larger models and native-language data alone might not be enough to achieve desired performance levels in country-specific tasks.

Are there architectural changes that could improve results?

It is still under investigation whether different training architectures or methodologies could yield better performance, but current results highlight the importance of scale and data quality.

Will future models surpass Minerva’s performance?

Researchers plan to experiment with larger models and new techniques, so it is possible that future iterations could improve upon Minerva’s results, but the exact scale needed remains uncertain.

Source: ThorstenMeyerAI.com