Seminal
The magazine and atlas of EU-funded researchA new publication from the web and AI team at Scott Charnley

Atlas / Netherlands

European Research Council grant · Running

STRIPES SpatioTemporal Reconstruction of Interacting People for pErceiving Systems

Coordinator
Universiteit Van Amsterdam, Netherlands
Runs
1 February 2025 to 31 January 2030
Status
Running
EU funding
€1.5m
Total cost
€1.5m
Partners
1 organisations in 1 countries
Type
European Research Council grant
Grant agreement
101165317

What the project is doing

In the project's own words, from its summary on CORDIS.

People constantly interact with objects to perform tasks. To help people accomplish these, computers need to perceive Human-Object Interactions (HOI), and for this, they need to reconstruct HOI from whole-body color images of people interacting with objects or scenes. This is challenging, due to the occlusions between bodies and objects, motion blur, depth ambiguities, and the low image resolution of hands and graspable object parts.

There has been significant prior work on estimating 3D humans without considering objects, and estimating 3D objects without considering humans. Little prior work estimates these jointly, but, for tractability, focuses either on interacting hands, ignoring the body, or on interacting bodies, ignoring hands. Only recent work addresses dexterous interaction of whole bodies, but instruments bodies with intrusive markers or sensors, and uses non-standard cameras to capture video of interactions.

Moreover, reconstruction lacks hand detail that is crucial for grasping, and videos are captured in constrained settings, consequently, methods trained on these struggle generalizing. My goal is to infer HOI from natural whole-body images/videos. To this end, I present an ambitious 5-year research agenda with novelties in four directions: (1) developing strong generative 3D shape models for objects and humans for a novel HOI representation; (2) developing methods that estimate 3D HOI from a color image with rich contact and proximal awareness; (3) instilling spatiotemporal reasoning into the heart of these for estimating 4D HOI from color video; and (4) extending these methods to also infer their own confidence that will be correlated with the reconstruction quality.

The outcome will be novel and robust methods for HOI reconstruction from natural images/videos. This will fill an important gap, enabling future intelligent systems to amplify people’s skills and help them accomplish tasks, e.g. for assistive robots or virtual 3D assistants or trainers.

Topics

Who is involved

OrganisationCityCountryRole
Universiteit Van AmsterdamAmsterdamNetherlandsCoordinator

Source: CORDIS, European Commission. Data of 5 Nov 2026, reused under the Commission's reuse policy. View on CORDIS · Grant DOI