Seminal
The magazine and atlas of EU-funded researchA new publication from the web and AI team at Scott Charnley

Atlas / Germany

European Research Council grant · Running

GraViLa Graphs without Labels: Multimodal Structure Learning without Human Supervision

Coordinator
Eberhard Karls Universitaet Tuebingen, Germany
Runs
1 April 2024 to 31 March 2029
Status
Running
EU funding
€1.5m
Total cost
€1.5m
Partners
2 organisations in 1 countries
Type
European Research Council grant
Grant agreement
101117556

What the project is doing

In the project's own words, from its summary on CORDIS.

Multimodal learning focuses on training models with data in more than one modality, such as videos capturing visual and audio information or documents containing image and text. Current approaches use such data to train large-scale deep learning models without human supervision by sampling pair-wise data e.g., an image-text pair from a website and train the network e.g. to identify matching vs. not matching pairs to learn better representations.

We argue that multimodal learning can do more: by combining information from different sources, multimodal models capture cross-modal semantic entities, and as most multimodal documents are a collection of connected modalities and topics, multimodal models should allow us to capture the inherent high-level topology of such data. The goal of the following project is to learn semantic structures from multimodal data to capture long-range concepts and relations in multimodal data via multimodal and self-supervision learning without human annotation.

We will represent this information in form of a graph, considering latent semantic concepts as nodes and their connectivity as edges. Based on this structure, we will extend current unimodal approaches to capture and process data from different modalities in a single structure. Finally, we will explore the challenges and opportunities of the proposed idea with respect to their impact on two main challenges in machine learning: data-efficient learning and fairness in label-free learning.

By bridging the gap between those two parallel trends, multimodal supervision and graph-based representations, we combine their strengths of generating and processing topological data, which will not only allow to build new applications and tools but also opens new ways of processing and understanding large-scale data that are out-of-reach at the moment.

Topics

Who is involved

OrganisationCityCountryRole
Eberhard Karls Universitaet TuebingenTuebingenGermanyCoordinator
Rheinische Friedrich-Wilhelms-Universitat BonnBonnGermanyParticipant

Source: CORDIS, European Commission. Data of 5 Nov 2026, reused under the Commission's reuse policy. View on CORDIS · Grant DOI