Seminal
The magazine and atlas of EU-funded researchA new publication from the web and AI team at Scott Charnley

Atlas / Germany

Research and Innovation Action · Running

LUMINOUS Language Augmentation for Humanverse

Coordinator
Deutsches Forschungszentrum Fur Kunstliche Intelligenz GmbH, Germany
Runs
1 January 2024 to 31 March 2027
Status
Running
EU funding
€5.6m
Total cost
€5.6m
Partners
13 organisations in 8 countries
Type
Research and Innovation Action
Grant agreement
101135724

What the project is doing

In the project's own words, from its summary on CORDIS.

LUMINOUS aims at the creation of the next generation of Language Augmented XR systems, where natural language-based communication and Multimodal Large Language Models (MLLM) enable adaptation to individual, not predefined user needs and unseen environments. This will enable future XR users to interact fluently with their environment, while having instant access to constantly updated global as well as domain- specific knowledge sources to accomplish novel tasks.

We aim to exploit MLLMs injected with domain specific knowledge for describing novel tasks on user demand. These are then communicated through a speech interface and/or a task adaptable avatar (e.g., coach/teacher) in terms of different visual aids and procedural steps for the accomplishment of the task. Language driven specification of the style, facial expressions, and specific attitudes of virtual avatars will facilitate generalisable and situation-aware communication in multiple use cases and different sectors.

LLMs will benefit in parallel in identifying new objects that were not part of their training data and then describing them in a way that they become visually recognizable. Our results will be prototyped and tested in three pilots, focussing on neurorehabilitation (support of stroke patients with language impairments), immersive industrial safety training, and 3D architectural design review. A consortium of six leading R&D institutes experts in six different disciplines (AI, Augmented Vision, NLP, Computer Graphics, Neurorehabilitation, Ethics) will follow a challenging workplan, aiming to bring about a new era at the crossroads of two of the most promising current technological developments (LLM/AI and XR), made in Europe.

Topics

Who is involved

OrganisationCityCountryRole
Deutsches Forschungszentrum Fur Kunstliche Intelligenz GmbHKaiserslauternGermanyCoordinator
Centre Hospitalier Universitaire VaudoisLausanneSwitzerlandAssociatedpartner
MindMazeLausanneSwitzerlandAssociatedpartner
Neurox Group SALausanneSwitzerlandAssociatedpartner
Fraunhofer Gesellschaft Zur Forderung Der Angewandten Forschung eVMunchenGermanyParticipant
Hypercliq IkeAthinaGreeceParticipant
Fundacion Centro De Tecnologias De Interaccion Visual Y Comunicaciones VicomtechDonostia San SebastianSpainParticipant
Ludus Tech SLBilbaoSpainParticipant
Universidad Del Pais Vasco/ Euskal Herriko UnibertsitateaLeioaSpainParticipant
University College Dublin, National University of Ireland, DublinDublinIrelandParticipant
Mindesk Societa a Responsabilita LimitataBariItalyParticipant
Ricoh International BVAmsterdamNetherlandsParticipant
University College LondonLondonUnited KingdomAssociatedpartner

Source: CORDIS, European Commission. Data of 5 Nov 2026, reused under the Commission's reuse policy. View on CORDIS · Grant DOI