Polish AI Models Bielik and PLLuM Rank Last, Even Lose on Polish Culture Knowledge

Polish research firm Oxido has developed the first large language model test focused specifically on Polish tasks and culture. The results caught the study's authors off guard: Poland's own Bielik and PLLuM models landed at the bottom of the pack, losing not only to Google but also to China's Qwen, in tasks that were supposed to play to their strengths.

The test set out to measure something beyond the standard international benchmarks, which typically evaluate models in English on math or coding tasks. Oxido designed tasks grounded in Polish reality: writing business correspondence, legal and tax advice for entrepreneurs, language correctness, and knowledge of Polish history and traditions, including Christmas Eve customs.

A Surprise With Pan Tadeusz

The most painful result for the creators of Poland's own models came in the national culture category. Experts have long argued that models trained with Polish in mind should grasp historical and literary nuance better than systems built mainly for the global market. The test showed the opposite.

On the task of reciting the first twelve lines of Pan Tadeusz, one of the most iconic passages in Polish literature, Bielik came in only 8th out of the twelve models tested. PLLuM fared even worse, placing third from the bottom. The models meant to be Poland's answer to foreign giants lost on home turf.

Who Won the Ranking

A Google model topped the ranking, which is no surprise given the scale of computing resources and training data available to the company. The real surprise was China's Qwen, which landed on the podium and beat out Meta's Llama, despite not being known for Polish-language optimization.

The result undercuts the long-held assumption that local training automatically translates into a better grasp of local cultural context. Large global models, trained on massive datasets that include Polish-language text, can now compete with specialized domestic models even on tasks rooted in native culture.

The Tech Sovereignty Question

The test results feed into a broader debate about Poland's technological sovereignty in AI, a debate that has picked up pace as the EU AI Act's August compliance deadline approaches. If domestic models can't hold their own even in tasks where they should theoretically have the edge, it's hard to build a case for their strategic value based on origin alone.

Bielik's result, given such limited resources, is still a decent outcome - Marek Jeleśniański, the study's author

Jeleśniański noted, however, that the comparison isn't entirely fair in terms of scale. Bielik and PLLuM are built on far smaller computing budgets and research teams than models from companies like Google, Alibaba, or Meta, which have thousands of GPUs and global engineering teams numbering in the hundreds.

What's Next for Polish Models

The study's author called for increased funding for domestic language model development, arguing that without greater investment, Polish teams won't be able to close the gap with global leaders. Whether that call translates into actual budget decisions remains an open question, especially amid the ongoing debate over how much building domestic AI infrastructure should cost.

For businesses and public institutions in Poland, the test results are a signal not to pick a language model for a given use case based on domestic origin alone, but to check real-world performance on tasks similar to the intended application. A global model may today handle Polish context better than is commonly assumed.

Sources: Szokujący test AI: polskie modele na końcu stawki (rp.pl), Test sztucznej inteligencji. Wiadomo, jak wypadły polskie modele AI (ai.infor.pl), Polskie modele AI radzą sobie słabo. Oto wyniki testu (polskiprzemysl.com.pl).