Seminal
The magazine and atlas of EU-funded researchA new publication from the web and AI team at Scott Charnley

Atlas / Spain

Marie Skłodowska-Curie Action · Running

GenEval GenEval: Linking Generation and Evaluation for Reliable NLG Assessment

Coordinator
Universidad Pompeu Fabra, Spain
Runs
1 October 2026 to 30 September 2028
Status
Running
EU funding
€194k
Total cost
—
Partners
1 organisations in 1 countries
Type
Marie Skłodowska-Curie Action
Grant agreement
101282681

What the project is doing

In the project's own words, from its summary on CORDIS.

Large Language Models (LLMs) are increasingly leveraged as evaluators of machine-generated text, a paradigm known as ""LLMs-as-judges."" While this approach offers flexibility and typically strong performance, its reliability remains inconsistent and poorly understood. Strong generative performance does not guarantee reliable evaluation, and the mechanisms linking these two capabilities remain opaque. Without systematic validation, current evaluation practices risk being blind to misleading and factually incorrect content and misrepresenting system capabilities.

GenEval addresses this challenge by investigating the fundamental relationship between generation and evaluation in LLMs, developing novel representation-based metrics, and predicting LLMs' evaluation reliability across tasks and models. We will analyze LLMs at a mechanistic level, identifying circuits and representations that underlie generative and evaluative behaviors. This understanding will enable the design of new evaluation metrics that directly exploit LLMs' internal representations, providing interpretable, efficient, and robust alternatives to existing approaches.

Finally, GenEval will develop predictive tools to estimate when an LLM is likely to be a reliable evaluator even in the absence of human judgment data, supporting informed model selection and human-in-the-loop evaluation. By integrating mechanistic insights with practical evaluation methods, GenEval will deliver both theoretical advances and applied tools. This action will be possible thanks to the integration of the scientific expertise of Prof.

Horacio Saggion, an internationally recognized expert in Natural Language Generation, and that of the researcher, who has a strong background in evaluation, Natural Language Processing, and Machine Learning. The action will develop impacting technology, and provide the researcher with the necessary training to become independent and strengthen her academic profile.""

Topics

Who is involved

OrganisationCityCountryRole
Universidad Pompeu FabraBarcelonaSpainCoordinator

Source: CORDIS, European Commission. Data of 5 Nov 2026, reused under the Commission's reuse policy. View on CORDIS · Grant DOI