Seminal
The magazine and atlas of EU-funded researchA new publication from the web and AI team at Scott Charnley

Atlas / France

Marie Skłodowska-Curie Action · Running

Bernar Understanding and Fixing Bottlenecks in Optimization for Modern Machine Learning

Coordinator
Institut National De Recherche En Informatique Et Automatique, France
Runs
1 May 2025 to 30 April 2027
Status
Running
EU funding
€226k
Total cost
—
Partners
1 organisations in 1 countries
Type
Marie Skłodowska-Curie Action
Grant agreement
101210427

What the project is doing

In the project's own words, from its summary on CORDIS.

Modern machine learning models have been successfully deployed across fields, from scientific studies to tech-nological developments in industry, but their development remains poorly understood. The training of a largelanguage model such as GPT-3 is estimated to cost $4.6M, and public attempts to replicate the training processalone required teams of engineers to rotating on-call for months, monitoring various statistics and constantlytweaking the training procedure when it broke.

Existing theoretical frameworks offer limited insights into thisprocess, as they do not capture the main difficulties that arise in practice when training neural networks, leavingpractitioners to rely on error-prone heuristics and expensive trial-and-error. This leads not only to a large devel-opment cost dominated by wasted resources, but also limits the possible impacts of machine learning to areasconsidered profitable by industries that have the resources to carry this development.

The objective of this project is to build a better understanding of how recently identified bottlenecks in neuralnetwork training slow down optimization and how to adress them. The specific aims are to: (a) Understandthe impact of class imbalance on the dynamics of neural networks to identify where to allocate algorithmicresources. (b) Develop a theory to capture optimization difficulties early in training to guide the developmentof algorithms that improve performance during this crucial phase. (c) Identify new bottlenecks that arise fromapplications to new data types.

The project combines experimental expertise of the postdoctoral and the theoretical expertise of the host insti-tution to identify and describe the real impact of data characteristics on neural network training. Understandingthese bottlenecks will help develop more efficient and reliable algorithms and guidelines on best practices thatdepend on properties of the data.

Topics

Who is involved

OrganisationCityCountryRole
Institut National De Recherche En Informatique Et AutomatiqueLe Chesnay CedexFranceCoordinator

Source: CORDIS, European Commission. Data of 5 Nov 2026, reused under the Commission's reuse policy. View on CORDIS · Grant DOI