Environmental awareness is a crucial skill for robotic systems intended to autonomously navigate and interact with their surroundings.
Robots access knowledge about their environment through maps. However, currently we see a big “complexity gap” in robotic mapping: while in recent years advances in computer vision have given us the ability to perceive our surroundings like never before through object detection and people tracking, robots still rely on maps containing only enough information for them to be able to navigate, but insufficient for many other tasks required by advanced autonomy. For example, most maps do not host semantic or dynamic information about the environment, needed for any application where interaction with people or specific objects is required. Until this gap is bridged, mobile robots will not be able to operate autonomously in dynamic environments.
Hypermaps lays the groundwork for the next level of interaction between robots and their environment by closing the complexity gap. In this project, we propose to go beyond today’s multi-layer maps by a new formalism, called hypermaps, where spatio-temporal knowledge (e.g., occupancy, semantics through deep object recognition, people movement in the environment) is stored and processed through advanced artificial intelligence to offer the robot task-specific maps to complete its missions. The core hypothesis of the project is that such a formalism will leverage the interplay between different maps to extract even more information and allow deeper reasoning. Anomalies in one map will be detected and corrected by looking at its correlation with the other maps, and information not visible in any single map will be made visible when the information of the layers is combined.
Closing the complexity gap constitutes a fundamental step towards the development of general, fully autonomous robots, able to execute high-level tasks and interact with us and their environment.
related publications
journal articles
RA-L
Event-Grounding Graph: Unified Spatio-Temporal Scene Graph from Robotic Observations
A fundamental aspect for building intelligent autonomous robots that can assist humans in their daily lives is the construction of rich environmental representations. While advances in semantic scene representations have enriched robotic scene understanding, current approaches lack a connection between spatial features and dynamic events; e.g., connecting the blue mug to the event washing a mug. In this work, we introduce the event-grounding graph (EGG), a framework grounding event interactions to spatial features of a scene. This representation allows robots to perceive, reason, and respond to complex spatio-temporal queries. Experiments using real robotic data demonstrate EGG’s capability to retrieve relevant information and respond accurately to human inquiries concerning the environment and events within.
@article{202605_nguyen_event-grounding,title={Event-{Grounding} {Graph}: {Unified} {Spatio}-{Temporal}
{Scene} {Graph} from {Robotic} {Observations}},shorttitle={Event-{Grounding} {Graph}},volume={11},doi={10.1109/LRA.2026.3669042},number={5},journal={IEEE Robotics and Automation Letters},publisher={IEEE},author={Nguyen, Phuoc and Verdoja, Francesco and Kyrki, Ville},month=may,year={2026},pages={5286--5293},}
conference articles
ICRA
QuASH: Using Natural-Language Heuristics to Query Visual-Language Robotic Maps
Matti Pekkanen, Francesco Verdoja, and Ville Kyrki
In 2026 IEEE Int. Conf. on Robotics and Automation (ICRA), Jun 2026
Embeddings from Visual-Language Models are increasingly utilized to represent semantics in robotic maps, offering an open-vocabulary scene understanding that surpasses traditional, limited labels. Embeddings enable on-demand querying by comparing embedded user text prompts to map embeddings via a similarity metric. The key challenge in performing the task indicated in a query is that the robot must determine the parts of the environment relevant to the query. This paper proposes a solution to this challenge. We leverage natural-language synonyms and antonyms associated with the query within the embedding space, applying heuristics to estimate the language space relevant to the query, and use that to train a classifier to partition the environment into matches and non-matches. We evaluate our method through extensive experiments, querying both maps and standard image benchmarks. The results demonstrate increased queryability of maps and images. Our querying technique is agnostic to the representation and encoder used, and requires limited training.
@inproceedings{202606_pekkanen_quash,address={Vienna, Austria},title={{QuASH}: {Using} {Natural}-{Language} {Heuristics} to {Query}
{Visual}-{Language} {Robotic} {Maps}},shorttitle={{QuASH}},booktitle={2026 {IEEE} {Int.} {Conf.} on {Robotics} and {Automation}
({ICRA})},publisher={IEEE},author={Pekkanen, Matti and Verdoja, Francesco and Kyrki, Ville},url={http://arxiv.org/abs/2510.14546},month=jun,year={2026},}
IROS
REACT: Real-time Efficient Attribute Clustering and Transfer for Updatable 3D Scene Graph
Phuoc Nguyen, Francesco Verdoja, and Ville Kyrki
In 2025 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Oct 2025
Modern-day autonomous robots need high-level map representations to perform sophisticated tasks. Recently, 3D scene graphs (3DSGs) have emerged as a promising alternative to traditional grid maps, blending efficient memory use and rich feature representation. However, most efforts to apply them have been limited to static worlds. This work introduces REACT, a framework that efficiently performs real-time attribute clustering and transfer to relocalize object nodes in a 3DSG. REACT employs a novel method for comparing object instances using an embedding model trained on triplet loss, facilitating instance clustering and matching. Experimental results demonstrate that REACT is able to relocalize objects while maintaining computational efficiency. The REACT framework’s source code will be available as an open-source project, promoting further advancements in reusable and updatable 3DSGs.
@inproceedings{202510_nguyen_react,address={Hangzhou, China},title={{REACT}: {Real}-time {Efficient} {Attribute} {Clustering} and
{Transfer} for {Updatable} {3D} {Scene} {Graph}},shorttitle={{REACT}},doi={10.1109/IROS60139.2025.11247273},booktitle={2025 {IEEE}/{RSJ} {Int.}\ {Conf.}\ on {Intelligent} {Robots}
and {Systems} ({IROS})},publisher={IEEE},author={Nguyen, Phuoc and Verdoja, Francesco and Kyrki, Ville},month=oct,year={2025},pages={2209--2216},}
IROS
Do Visual-Language Grid Maps Capture Latent Semantics?
Matti Pekkanen, Tsvetomila Mihaylova, Francesco Verdoja, and Ville Kyrki
In 2025 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Oct 2025
Visual-language models (VLMs) have recently been introduced in robotic mapping using the latent representations, i.e., embeddings, of the VLMs to represent semantics in the map. They allow moving from a limited set of human-created labels toward open-vocabulary scene understanding, which is very useful for robots when operating in complex real-world environments and interacting with humans. While there is anecdotal evidence that maps built this way support downstream tasks, such as navigation, rigorous analysis of the quality of the maps using these embeddings is missing. In this paper, we propose a way to analyze the quality of maps created using VLMs. We investigate two critical properties of map quality: queryability and distinctness. The evaluation of queryability addresses the ability to retrieve information from the embeddings. We investigate intra-map distinctness to study the ability of the embeddings to represent abstract semantic classes and inter-map distinctness to evaluate the generalization properties of the representation. We propose metrics to evaluate these properties and evaluate two state-of-the-art mapping methods, VLMaps and OpenScene, using two encoders, LSeg and OpenSeg, using real-world data from the Matterport3D data set. Our findings show that while 3D features improve queryability, they are not scale invariant, whereas image-based embeddings generalize to multiple map resolutions. This allows the image-based methods to maintain smaller map sizes, which can be crucial for using these methods in real-world deployments. Furthermore, we show that the choice of the encoder has an effect on the results. The results imply that properly thresholding open-vocabulary queries is an open problem.
@inproceedings{202510_pekkanen_visual-language,address={Hangzhou, China},title={Do {Visual}-{Language} {Grid} {Maps} {Capture} {Latent}
{Semantics}?},doi={10.1109/IROS60139.2025.11247447},booktitle={2025 {IEEE}/{RSJ} {Int.}\ {Conf.}\ on {Intelligent} {Robots}
and {Systems} ({IROS})},publisher={IEEE},author={Pekkanen, Matti and Mihaylova, Tsvetomila and Verdoja, Francesco and Kyrki, Ville},month=oct,year={2025},pages={4059--4066},}
IROS
Bayesian Floor Field: Transferring people flow predictions across environments
Francesco Verdoja, Tomasz Piotr Kucner, and Ville Kyrki
In 2024 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Oct 2024
Mapping people dynamics is a crucial skill for robots, because it enables them to coexist in human-inhabited environments. However, learning a model of people dynamics is a time consuming process which requires observation of large amount of people moving in an environment. Moreover, approaches for mapping dynamics are unable to transfer the learned models across environments: each model is only able to describe the dynamics of the environment it has been built in. However, the impact of architectural geometry on people’s movement can be used to anticipate their patterns of dynamics, and recent work has looked into learning maps of dynamics from occupancy. So far however, approaches based on trajectories and those based on geometry have not been combined. In this work we propose a novel Bayesian approach to learn people dynamics able to combine knowledge about the environment geometry with observations from human trajectories. An occupancy-based deep prior is used to build an initial transition model without requiring any observations of pedestrian; the model is then updated when observations become available using Bayesian inference. We demonstrate the ability of our model to increase data efficiency and to generalize across real large-scale environments, which is unprecedented for maps of dynamics.
@inproceedings{202410_verdoja_bayesian,address={Abu Dhabi, UAE},title={Bayesian Floor Field: Transferring people flow predictions
across environments},doi={10.1109/IROS58592.2024.10802300},booktitle={2024 {IEEE}/{RSJ} {Int.}\ {Conf.}\ on {Intelligent} {Robots}
and {Systems} ({IROS})},publisher={IEEE},author={Verdoja, Francesco and Kucner, Tomasz Piotr and Kyrki, Ville},month=oct,year={2024},pages={12801--12807},}
workshop articles
ICRA
Evaluating the quality of robotic visual-language maps
Matti Pekkanen, Tsvetomila Mihaylova, Francesco Verdoja, and Ville Kyrki
May 2024
Presented at the “Vision-Language Models for Navigation and Manipulation (VLMNM)” workshop at the IEEE Int. Conf. on Robotics and Automation (ICRA)
Visual-language models (VLMs) have recently been introduced in robotic mapping by using the latent representations, i.e., embeddings, of the VLMs to represent the natural language semantics in the map. The main benefit is moving beyond a small set of human-created labels toward open-vocabulary scene understanding. While there is anecdotal evidence that maps built this way support downstream tasks, such as navigation, rigorous analysis of the quality of the maps using these embeddings is lacking. In this paper, we propose a way to analyze the quality of maps created using VLMs by evaluating two critical properties: queryability and consistency. We demonstrate the proposed method by evaluating the maps created by two state-of-the-art methods, VLMaps and OpenScene, using two encoders, LSeg and OpenSeg, using real-world data from the Matterport3D data set. We find that OpenScene outperforms VLMaps with both encoders, and LSeg outperforms OpenSeg with both methods.
@online{202405_pekkanen_evaluating,title={Evaluating the quality of robotic visual-language maps},url={https://vlmnm-workshop.github.io},author={Pekkanen, Matti and Mihaylova, Tsvetomila and Verdoja, Francesco and Kyrki, Ville},month=may,year={2024},note={Presented at the ``Vision-Language Models for Navigation and
Manipulation (VLMNM)'' workshop at the IEEE Int.\ Conf.\ on
Robotics and Automation (ICRA)},}
ICRA
Using occupancy priors to generalize people flow predictions
Francesco Verdoja, Tomasz Piotr Kucner, and Ville Kyrki
May 2024
Presented at the “Long-term Human Motion Prediction” workshop at the IEEE Int. Conf. on Robotics and Automation (ICRA)
Mapping people dynamics is a crucial skill for robots, because it enables them to coexist in human-inhabited environments. However, learning a model of people dynamics is a time consuming process which requires observation of large amount of people moving in an environment. Moreover, approaches for mapping dynamics are unable to transfer the learned models across environments: each model is only able to describe the dynamics of the environment it has been built in. However, the impact of architectural geometry on people’s movement can be used to anticipate their patterns of dynamics, and recent work has looked into learning maps of dynamics from occupancy. So far however, approaches based on trajectories and those based on geometry have not been combined. In this work we propose a novel Bayesian approach to learn people dynamics able to combine knowledge about the environment geometry with observations from human trajectories. An occupancy-based deep prior is used to build an initial transition model without requiring any observations of pedestrian; the model is then updated when observations become available using Bayesian inference. We demonstrate the ability of our model to increase data efficiency and to generalize across real large-scale environments, which is unprecedented for maps of dynamics.
@online{202405_verdoja_using,title={Using occupancy priors to generalize people flow predictions},url={https://motionpredictionicra2024.github.io},author={Verdoja, Francesco and Kucner, Tomasz Piotr and Kyrki, Ville},month=may,year={2024},note={Presented at the ``Long-term Human Motion Prediction'' workshop
at the IEEE Int.\ Conf.\ on Robotics and Automation (ICRA)},}
preprints
IROS
Rheos: Modelling Continuous Motion Dynamics in Hierarchical 3D Scene Graphs
Iacopo Catalano, Francesco Verdoja, Javier Civera, Jorge Peña-Queralta, and Julio A. Placed
Mar 2026
2026 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) (submitted)
3D Scene Graphs (3DSGs) provide hierarchical, multi-resolution abstractions that encode the geometric and semantic structure of an environment, yet their treatment of dynamics remains limited to tracking individual agents. Maps of Dynamics (MoDs) complement this by modeling aggregate motion patterns, but rely on uniform grid discretizations that lack semantic grounding and scale poorly. We present Rheos, a framework that explicitly embeds continuous directional motion models into an additional dynamics layer of a hierarchical 3DSG that enhances the navigational properties of the graph. Each dynamics node maintains a semi-wrapped Gaussian mixture model that captures multimodal directional flow as a principled probability distribution with explicit uncertainty, replacing the discrete histograms used in prior work. To enable online operation, Rheos employs reservoir sampling for bounded-memory observation buffers, parallel per-cell model updates and a principled Bayesian Information Criterion (BIC) sweep that selects the optimal number of mixture components, reducing per-update initialization cost from quadratic to linear in the number of samples. Evaluated across four spatial resolutions in a simulated pedestrian environment, Rheos consistently outperforms the discrete baseline under continuous as well as unfavorable discrete metrics. We release our implementation as open source.
@misc{202603_catalano_rheos,title={Rheos: {Modelling} {Continuous} {Motion} {Dynamics} in
{Hierarchical} {3D} {Scene} {Graphs}},shorttitle={Rheos},author={Catalano, Iacopo and Verdoja, Francesco and Civera, Javier and Peña-Queralta, Jorge and Placed, Julio A.},url={http://arxiv.org/abs/2603.20239},month=mar,year={2026},publisher={arXiv},note={2026 {IEEE}/{RSJ} {Int.}\ {Conf.}\ on {Intelligent} {Robots}
and {Systems} ({IROS}) (submitted)}}
RO-MAN
Relational Scene Graphs for Object Grounding of Natural Language Commands
Julia Kuhn, Francesco Verdoja, Tsvetomila Mihaylova, and Ville Kyrki
Mar 2026
2026 IEEE IEEE Int. Conf. on Robot and Human Interactive Communication (RO-MAN) (submitted)
Robots are finding wider adoption in human environments, increasing the need for natural human-robot interaction. However, understanding a natural language command requires the robot to infer the intended task and how to decompose it into executable actions, and to ground those actions in the robot’s knowledge of the environment, including relevant objects, agents, and locations. This challenge can be addressed by combining the capabilities of Large language models (LLMs) to understand natural language with 3D scene graphs (3DSGs) for grounding inferred actions in a semantic representation of the environment. However, many 3DSGs lack explicit spatial relations between objects, even though humans often rely on these relations to describe an environment. This paper investigates whether incorporating open- or closed-vocabulary spatial relations into 3DSGs can improve the ability of LLMs to interpret natural language commands. To address this, we propose an LLM-based pipeline for target object grounding from open-vocabulary language commands and a vision language model (VLM)-based pipeline to add open-vocabulary spatial edges to 3DSGs from images captured while mapping. Finally, two LLMs are evaluated in a study assessing their performance on the downstream task of target object grounding. Our study demonstrates that explicit spatial relations improve the ability of LLMs to ground objects. Moreover, open-vocabulary relation generation with VLMs proves feasible from robot-captured images, but their advantage over closed-vocabulary relations is found to be limited.
@misc{202603_kuhn_relational,title={Relational {Scene} {Graphs} for {Object} {Grounding} of
{Natural} {Language} {Commands}},author={Kuhn, Julia and Verdoja, Francesco and Mihaylova, Tsvetomila and Kyrki, Ville},url={http://arxiv.org/abs/2602.04635},month=mar,year={2026},publisher={arXiv},note={2026 IEEE {IEEE} {Int.}\ {Conf.}\ on {Robot} and {Human}
{Interactive} {Communication} ({RO}-{MAN}) (submitted)}}
