Our research is in the area of computational interaction. We use methods of optimization, machine learning, and mathematical modeling to understand and improve our interaction with computers. In particular, our work centers around the following topics:
Optimal adaptation of interfaces
We investigate how to optimally adapt user interfaces at run-time to changes in the usage context, such as the user’s environment, their cognitive load, current task, or personal interaction strategy. On the one hand, this includes formulating optimization models for adapting interfaces. On the other hand, we develop methods to identify changes in the user’s context and models to predict how such changes impact the user’s behavior. Example projects include:
- Towards Flexible and Robust User Interface Adaptations With Multiple Objectives. In: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, Association for Computing Machinery, San Francisco, CA, USA, 2023, ISBN: 9798400701320.
- AUIT – the Adaptive User Interfaces Toolkit for Designing XR Applications. In: Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, UIST'22, 2022.
- XRgonomics: Facilitating the Creation of Ergonomic 3D Interfaces. In: Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI'21), ACM, 2021.
- Detecting Relevance during Decision-Making from Eye Movements for UI Adaptation. In: Symposium on Eye Tracking Research and Applications, Association for Computing Machinery, 2020, ISBN: 9781450371339.
- Context-Aware Online Adaptation of Mixed Reality Interfaces. In: Symposium on User Interface Software and Technology, ACM, 2019, ISBN: 9781450368162.
- AdaM: Adapting Multi-User Interfaces for Collaborative Environments in Real-Time. In: SIGCHI Conference on Human Factors in Computing Systems, ACM, 2018.
Human-AI interaction and oversight
Our recent work focuses on understanding, modeling, and supporting people in their interaction with AI systems. On the one hand, we consider contexts that involve AI systems supporting users in their tasks (e.g. intelligent text input systems, AI-supported decision making). On the other hand, we are also interested in supporting humans in overseeing highly automated systems. We study how users rely on and adapt to AI support (e.g. from their gaze behavior), how AI support affects people’s perceived responsibility and autonomy, and how interface design can shape human motivation, perceived autonomy, and trust.
- RelEYEance: Gaze-Based Assessment of Users’ AI-reliance at Run-Time. In: Proc. ACM Hum.-Comput. Interact., vol. 9, no. ETRA16, 2025.
- Typing Behavior is About More than Speed: Users' Strategies for Choosing Word Suggestions Despite Slower Typing Rates. In: Proc. ACM Hum.-Comput. Interact., vol. 7, no. MHCI, 2023.
Computational design of user interfaces
We develop optimization methods that can be used to automate (parts of) the design process of user interfaces. In addition to developing optimization models that capture the problem of UI design, we also focus on methods and tools that facilitate participatory optimizaton to enable stakeholders at all levels to take part in the design process. Interactive optimization tools can help them to learn about the problem and broadly explore possible solutions. They also make assumptions and trade-offs more explicit and help to objectively assess individual’s expectations and ideas.
See for example the following works:
- AZERTY Amélioré: Computational Design on a National Scale. In: Communications of the ACM, vol. 64, no. 2, 2021, ISSN: 0001-0782.
- Computational Support for Functionality Selection in Interaction Design. In: ACM Transaction on Computer-Human Interaction, vol. 24, no. 5, 2017, ISSN: 1073-0516.
- Assignment Problems for Optimizing Text Input. Aalto University, 2018.
- Towards Multi-Objective Optimization for UI Design. Workshop on Principles, Techniques and Perspectives on Optimization and HCI, CHI'15, Seoul, Korea, 2015.
Understanding and predicting user behavior
Our work builds on empirical studies to capture users’ behavior in interaction with different interfaces. This includes planning and conducting studies, analyzing data, and transforming empirical observations into predictive models using machine learning, statistical modeling, and other techniques. The results are actionable insights that can be used for optimization and adaptation or implemented as part of design tools. See for example the following projects:
- Shifting Focus with HCEye: Exploring the Dynamics of Visual Highlighting and Cognitive Load on User Attention and Saliency Prediction. In: Proceedings of the ACM on Human-Computer Interaction, vol. 8, No. ETRA , no. 236 , 2024.
- Typing Behavior is About More than Speed: Users' Strategies for Choosing Word Suggestions Despite Slower Typing Rates. In: Proc. ACM Hum.-Comput. Interact., vol. 7, no. MHCI, 2023.
- XRgonomics: Facilitating the Creation of Ergonomic 3D Interfaces. In: Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI'21), ACM, 2021.
- Complex Interaction as Emergent Behaviour: Simulating Mid-Air Virtual Keyboard Typing using Reinforcement Learning. IEEE Transactions on Visualization and Computer Graphics, IEEE, 2021.
- How We Type: Movement Strategies and Performance in Everyday Typing. In: SIGCHI Conference on Human Factors in Computing Systems, ACM, New York, NY, US, 2016, ISBN: 978-1-4503-3362-7.
- Investigating the Dexterity of Multi-Finger Input for Mid-Air Text Entry. In: SIGCHI Conference on Human Factors in Computing Systems, ACM, New York, New York, USA, 2015, ISBN: 9781450331456.
Context-Aware Adaptations in Mixed Reality
User interfaces in mixed reality cannot always behave the same way. They should be adapted based on mobility conditions, interface type, environment, real and virtual objects, and personal preferences. We aim to develop context-aware adaptations based on these different factors. The goal is to use machine learning techniques to learn appropriate adaptations across different contexts and situations. The system should learn user preferences through a few trials and then use a context vector (mobility, interface type, environment, real and virtual objects, and personal preferences) to predict the best adaptation for each user in a specific situation. Thus, the system should maintain a memory of previous interactions and learned user preferences, allowing it to recognize similar situations and automatically apply appropriate adaptations.
ConverseMR V2: Improving the Conversational Interface for Adaptations in MR
We built a conversational interface for adaptations in mixed reality. This interface is activated by a user gesture, and then the user can say their command. The command is then processed to generate an intermediate representation that includes the optimization objectives and parameters. These objectives are then applied in real time. There are a few possible improvements to this process, including: (1) improving object selection by allowing users to point at objects using ray-based pointing, (2) improving speech recognition and intermediate representation extraction by moving client-side processing to the server side and (3) optimizing the prompts for intermediate representation extraction, (4) improving feedback to provide users with clearer feedback and apply commands more smoothly by highlighting the objects and changes, and (4) supporting additional gestures for adaptations. These improvements could form the next version of ConverseMR and provide users with a smoother and more natural conversational interaction in mixed reality.