Publications

2025

1. Open, Reproducible and Trustworthy Robot-Based Experiments with Virtual Labs and Digital-Twin-Based Execution Tracing

   Benjamin Alt, Mareike Picklum, Sorin Arion, and 2 more authors

   In 1st IEEE IROS 2025 Workshop on Embodied AI and Robotics for Future Scientific Discovery (AIR4S), Oct 2025

   Abs DOI arXiv PDF

   We envision a future in which autonomous robots conduct scientific experiments in ways that are not only precise and repeatable, but also open, trustworthy, and transparent. To realize this vision, we present two key contributions: a semantic execution tracing framework that logs sensor data together with semantically annotated robot belief states, ensuring that automated experimentation is transparent and replicable; and the AICOR Virtual Research Building (VRB), a cloud-based platform for sharing, replicating, and validating robot task executions at scale. Together, these tools enable reproducible, robot-driven science by integrating deterministic execution, semantic memory, and open knowledge representation, laying the foundation for autonomous systems to participate in scientific discovery.

2. Semi-Autonomous Robotic Assistance for Gallbladder Retraction in Surgery

   Alexander Schüßler, Christian Kunz, Rayan Younis, and 4 more authors

   IEEE Robotics and Automation Letters, Oct 2025

   Abs DOI PDF

   (Semi-)autonomous robotic assistance in minimally invasive surgery has the potential to alleviate surgical staff shortage and decrease the workload of medical professionals. These robots must execute complex tasks within unpredictable and unstructured environments encountered during surgery. Although imitation learning approaches have the potential to learn complex surgical skills, the interpretation of robot behavior during safety-critical scenarios, such as surgery, remains a challenge. Through combining interpretable 3D point cloud feature vectors based on domain knowledge with feedforward neural networks and probabilistic movement primitives, domain knowledge-informed movement primitives effectively learn surgical skills while improving the interpretation of robot behavior. The evaluation on test data proves that the proposed method can effectively learn surgical skills based on a small number of demonstrations. Using the proposed imitation learning method, a semi-autonomous robotic assistance for directed gallbladder retraction is introduced and evaluated during gallbladder removal on a silicone liver phantom and ex vivo porcine livers. Achieving over 91 % and 92 % successful gallbladder retractions, the robotic assistance enables effective support for surgeons during these surgical interventions.

3. Digital Twin Generation from Visual Data: A Survey

   Andrew Melnik, Benjamin Alt, Giang Nguyen, and 7 more authors

   Apr 2025

   Abs DOI arXiv

   This survey explores recent developments in generating digital twins from videos. Such digital twins can be used for robotics application, media content creation, or design and construction works. We analyze various approaches, including 3D Gaussian Splatting, generative in-painting, semantic segmentation, and foundation models highlighting their advantages and limitations. Additionally, we discuss challenges such as occlusions, lighting variations, and scalability, as well as potential future research directions. This survey aims to provide a comprehensive overview of state-of-the-art methodologies and their implications for real-world applications. Awesome list: https://github.com/ndrwmlnk/awesome-digital-twins

4. AI-based Framework for Robust Model-Based Connector Mating in Robotic Wire Harness Installation

   Claudius Kienle, Benjamin Alt, Finn Schneider, and 3 more authors

   Mar 2025

   Abs DOI arXiv PDF

   Despite the widespread adoption of industrial robots in automotive assembly, wire harness installation remains a largely manual process, as it requires precise and flexible manipulation. To address this challenge, we design a novel AI-based framework that automates cable connector mating by integrating force control with deep visuotactile learning. Our system optimizes search-and-insertion strategies using first-order optimization over a multimodal transformer architecture trained on visual, tactile, and proprioceptive data. Additionally, we design a novel automated data collection and optimization pipeline that minimizes the need for machine learning expertise. The framework optimizes robot programs that run natively on standard industrial controllers, permitting human experts to audit and certify them. Experimental validations on a center console assembly task demonstrate significant improvements in cycle times and robustness compared to conventional robot programming approaches. Videos are available under https://claudius-kienle.github.io/AppMuTT.

5. LODGE: Joint Hierarchical Task Planning and Learning of Domain Models with Grounded Execution

   Claudius Kienle, Benjamin Alt, Oleg Arenz, and 1 more author

   May 2025

   Abs DOI arXiv

   Large Language Models (LLMs) enable planning from natural language instructions using implicit world knowledge, but often produce flawed plans that require refinement. Instead of directly predicting plans, recent methods aim to learn a problem domain that can be solved for different goal states using classical planners. However, these approaches require significant human feedback to obtain useful models. We address this shortcoming by learning hierarchical domains, where low-level predicates and actions are composed into higher-level counterparts, and by leveraging simulation to validate their preconditions and effects. This hierarchical approach is particularly powerful for long-horizon planning, where LLM-based planning approaches typically struggle. Furthermore, we introduce a central error reasoner to ensure consistency among the different planning levels. Evaluation on two challenging International Planning Competition (IPC) domains and a long-horizon robot manipulation task demonstrates higher planning success rates than state-of-the-art domain synthesis and LLM-modulo planning methods, while constructing high-quality models of the domain.

6. QueryCAD: Grounded Question Answering for CAD Models

   Claudius Kienle, Benjamin Alt, Darko Katic, and 1 more author

   In 2025 IEEE International Conference on Robotics and Automation (ICRA), May 2025

   Abs DOI arXiv PDF

   CAD models are widely used in industry and are essential for robotic automation processes. However, these models are rarely considered in novel AI-based approaches, such as the automatic synthesis of robot programs, as there are no readily available methods that would allow CAD models to be incorporated for the analysis, interpretation, or extraction of information. To address these limitations, we propose QueryCAD, the first system designed for CAD question answering, enabling the extraction of precise information from CAD models using natural language queries. QueryCAD incorporates SegCAD, an open-vocabulary instance segmentation model we developed to identify and select specific parts of the CAD model based on part descriptions. We further propose a CAD question answering benchmark to evaluate QueryCAD and establish a foundation for future research. Lastly, we integrate QueryCAD within an automatic robot program synthesis framework, validating its ability to enhance deep-learning solutions for robotics by enabling them to process CAD models (https://claudius-kienle.github.com/querycad).

7. Shadow Program Inversion with Differentiable Planning: A Framework for Unified Robot Program Parameter and Trajectory Optimization

   Benjamin Alt, Claudius Kienle, Darko Katic, and 2 more authors

   In 2025 IEEE International Conference on Robotics and Automation (ICRA), May 2025

   Abs DOI arXiv PDF

   This paper presents SPI-DP, a novel first-order optimizer capable of optimizing robot programs with respect to both high-level task objectives and motion-level constraints. To that end, we introduce DGPMP2-ND, a differentiable collision-free motion planner for serial N-DoF kinematics, and integrate it into an iterative, gradient-based optimization approach for generic, parameterized robot program representations. SPI-DP allows first-order optimization of planned trajectories and program parameters with respect to objectives such as cycle time or smoothness subject to e.g. collision constraints, while enabling humans to understand, modify or even certify the optimized programs. We provide a comprehensive evaluation on two practical household and industrial applications.

2024

1. BANSAI: Towards Bridging the AI Adoption Gap in Industrial Robotics with Neurosymbolic Programming

   Benjamin Alt, Julia Dvorak, Darko Katic, and 3 more authors

   In Procedia CIRP, Jan 2024

   Abs DOI arXiv PDF

   Over the past decade, deep learning helped solve manipulation problems across all domains of robotics. At the same time, industrial robots continue to be programmed overwhelmingly using traditional program representations and interfaces. This paper undertakes an analysis of this "AI adoption gap" from an industry practitioner’s perspective. In response, we propose the BANSAI approach (Bridging the AI Adoption Gap via Neurosymbolic AI). It systematically leverages principles of neurosymbolic AI to establish data-driven, subsymbolic program synthesis and optimization in modern industrial robot programming workflow. BANSAI conceptually unites several lines of prior research and proposes a path toward practical, real-world validation.

2. Domain-Specific Fine-Tuning of Large Language Models for Interactive Robot Programming

   Benjamin Alt, Urs Keßner, Aleksandar Taranovic, and 4 more authors

   In European Robotics Forum 2024, Mar 2024

   Abs DOI arXiv PDF

   Industrial robots are applied in a widening range of industries, but robot programming mostly remains a task limited to programming experts. We propose a natural language-based assistant for programming of advanced, industrial robotic applications and investigate strategies for domain-specific fine-tuning of foundation models with limited data and compute.

3. Human-AI Interaction in Industrial Robotics: Design and Empirical Evaluation of a User Interface for Explainable AI-Based Robot Program Optimization

   Benjamin Alt, Johannes Zahn, Claudius Kienle, and 7 more authors

   In 57th CIRP Conference on Manufacturing Systems 2024, Apr 2024

   Abs DOI arXiv PDF

   While recent advances in deep learning have demonstrated its transformative potential, its adoption for real-world manufacturing applications remains limited. We present an Explanation User Interface (XUI) for a state-of-the-art deep learning-based robot program optimizer which provides both naive and expert users with different user experiences depending on their skill level, as well as Explainable AI (XAI) features to facilitate the application of deep learning methods in real-world applications. To evaluate the impact of the XUI on task performance, user satisfaction and cognitive load, we present the results of a preliminary user survey and propose a study design for a large-scale follow-up study.

4. RoboGrind: Intuitive and Interactive Surface Treatment with Industrial Robots

   Benjamin Alt, Florian Stöckl, Silvan Müller, and 10 more authors

   In 2024 IEEE International Conference on Robotics and Automation (ICRA), May 2024

   Abs DOI arXiv PDF

   Surface treatment tasks such as grinding, sanding or polishing are a vital step of the value chain in many industries, but are notoriously challenging to automate. We present RoboGrind, an integrated system for the intuitive, interactive automation of surface treatment tasks with industrial robots. It combines a sophisticated 3D perception pipeline for surface scanning and automatic defect identification, an interactive voice-controlled wizard system for the AI-assisted bootstrapping and parameterization of robot programs, and an automatic planning and execution pipeline for force-controlled robotic surface treatment. RoboGrind is evaluated both under laboratory and real-world conditions in the context of refabricating fiberglass wind turbine blades.

5. MuTT: A Multimodal Trajectory Transformer for Robot Skills

   Claudius Kienle, Benjamin Alt, Onur Celik, and 4 more authors

   In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Aug 2024

   Abs DOI arXiv PDF

   High-level robot skills represent an increasingly popular paradigm in robot programming. However, configuring the skills’ parameters for a specific task remains a manual and time-consuming endeavor. Existing approaches for learning or optimizing these parameters often require numerous real-world executions or do not work in dynamic environments. To address these challenges, we propose MuTT, a novel encoder-decoder transformer architecture designed to predict environment-aware executions of robot skills by integrating vision, trajectory, and robot skill parameters. Notably, we pioneer the fusion of vision and trajectory, introducing a novel trajectory projection. Furthermore, we illustrate MuTT’s efficacy as a predictor when combined with a model-based robot skill optimizer. This approach facilitates the optimization of robot skill parameters for the current environment, without the need for real-world executions during optimization. Designed for compatibility with any representation of robot skills, MuTT demonstrates its versatility across three comprehensive experiments, showcasing superior performance across two different skill representations.

6. EASY: Energy-Efficient Analysis and Control Processes in the Dynamic Edge-Cloud Continuum for Industrial Manufacturing

   Alexander Schultheis, Benjamin Alt, Sebastian Bast, and 14 more authors

   KI - Künstliche Intelligenz, Sep 2024

   Abs DOI PDF

   According to the guiding principles of Industry 4.0, edge computing enables the data-sovereign and near-real-time processing of data directly at the point of origin. Using these edge devices in manufacturing organization will drive the use of industrial analysis, control, and Artificial Intelligence (AI) applications close to production. The goal of the EASY project is to make the added value of edge computing available by providing an easily usable Edge-Cloud Continuum with a runtime environment and services for the execution of AI-based Analysis and Control processes. Within this continuum, a dynamic, distributed, and optimized execution of services is automated across the entire spectrum from centralized cloud to decentralized edge instances to increase productivity and resource efficiency.

2023

1. EfficientPPS: Part-aware Panoptic Segmentation of Transparent Objects for Robotic Manipulation

   Benjamin Alt, Minh Dang Nguyen, Andreas Hermann, and 4 more authors

   In ISR Europe 2023, Sep 2023

   Abs arXiv PDF

   The use of autonomous robots for assistance tasks in hospitals has the potential to free up qualified staff and improve patient care. However, the ubiquity of deformable and transparent objects in hospital settings poses significant challenges to vision-based perception systems. We present EfficientPPS, a neural architecture for part-aware panoptic segmentation that provides robots with semantically rich visual information for grasping and manipulation tasks. We also present an unsupervised data collection and labelling method to reduce the need for human involvement in the training process. EfficientPPS is evaluated on a dataset containing real-world hospital objects and demonstrated to be robust and efficient in grasping transparent transfusion bags with a collaborative robot arm.

2. Knowledge-Driven Robot Program Synthesis from Human VR Demonstrations

   Benjamin Alt, Franklin Kenghagho Kenfack, Andrei Haidu, and 3 more authors

   In Proceedings of the 20th International Conference on Principles of Knowledge Representation and Reasoning, Sep 2023

   Abs DOI arXiv PDF

   Aging societies, labor shortages and increasing wage costs call for assistance robots capable of autonomously performing a wide array of real-world tasks. Such open-ended robotic manipulation requires not only powerful knowledge representations and reasoning (KR&R) algorithms, but also methods for humans to instruct robots what tasks to perform and how to perform them. In this paper, we present a system for automatically generating executable robot control programs from human task demonstrations in virtual reality (VR). We leverage common-sense knowledge and game engine-based physics to semantically interpret human VR demonstrations, as well as an expressive and general task representation and automatic path planning and code generation, embedded into a state-of-the-art cognitive architecture. We demonstrate our approach in the context of force-sensitive fetch-and-place for a robotic shopping assistant. The source code is available at https://github.com/ease-crc/vr-program-synthesis.

3. Artificial Neural Network Guided Compensation of Nonlinear Payload and Wear Effects for Industrial Robots

   Julian Raible, Oliver Rettig, Benjamin Alt, and 7 more authors

   In 2023 IEEE 19th International Conference on Automation Science and Engineering (CASE), Aug 2023

   Abs DOI

   The absolute accuracy of industrial robots is influ-enced by numerous geometric and non-geometric errors. Most state-of-the-art calibration and compensation methods consider only the geometric errors and neglect the non-geometric ones. In this paper, a hybrid compensation approach is proposed that combines well-known kinematic models with a model-free data-driven component using a neural network. This allows the use of established calibration methods for the geometric influences captured in the kinematic model to improve the non-geometric error compensation by the neural network. The proposed approach is applied in two use cases: payload and wear compensation. Simulations and real experiments show the improved absolute accuracy of the hybrid compensation approach compared to classical calibration methods.

4. Autonomous Surface Grinding of Wind Turbine Blades

   Florian Stöckl, Marcus Strand, Silvan Müller, and 6 more authors

   In Intelligent Autonomoous Systems 18, Jul 2023

   Abs DOI

   Discarded wind turbine blades generate a considerable amount of waste that could be reduced by remanufacturing. Manual remanufacturing is too costly, which is why research is being conducted into automation techniques. The main problem is the individuality of work pieces due to damages. This work presents a workflow that includes damage analysis based on scans of the blade, subsequent path planning, control engineering with an AI controller for grinding and automatic review of the grinding process. Current problems are the inaccuracy of the robot used for scanning and the colour sensitivity of the used laser scanner. Our next steps besides solving the mentioned problems are to train a supervised machine learning algorithm with damage examples and to implement a specific and multi-step path planning algorithm.

2022

1. Heuristic-Free Optimization of Force-Controlled Robot Search Strategies in Stochastic Environments

   Benjamin Alt, Darko Katic, Rainer Jäkel, and 1 more author

   In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Oct 2022

   Abs DOI arXiv PDF

   In both industrial and service domains, a central benefit of the use of robots is their ability to quickly and reliably execute repetitive tasks. However, even relatively simple peg-in-hole tasks are typically subject to stochastic variations, requiring search motions to find relevant features such as holes. While search improves robustness, it comes at the cost of increased runtime: More exhaustive search will maximize the probability of successfully executing a given task, but will significantly delay any downstream tasks. This trade-off is typically resolved by human experts according to simple heuristics, which are rarely optimal. This paper introduces an automatic, data-driven and heuristic-free approach to optimize robot search strategies. By training a neural model of the search strategy on a large set of simulated stochastic environments, conditioning it on few real-world examples and inverting the model, we can infer search strategies which adapt to the time-variant characteristics of the underlying probability distributions, while requiring very few real-world measurements. We evaluate our approach on two different industrial robots in the context of spiral and probe search for THT electronics assembly.**See github.com/benjaminalt/dpse for code and data.

2. LapSeg3D: Weakly Supervised Semantic Segmentation of Point Clouds Representing Laparoscopic Scenes

   Benjamin Alt, Christian Kunz, Darko Katic, and 5 more authors

   In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Oct 2022

   Abs DOI arXiv PDF

   The semantic segmentation of surgical scenes is a prerequisite for task automation in robot assisted interventions. We propose LapSeg3D, a novel DNN-based approach for the voxel-wise annotation of point clouds representing surgical scenes. As the manual annotation of training data is highly time consuming, we introduce a semi-autonomous clustering-based pipeline for the annotation of the gallbladder, which is used to generate segmented labels for the DNN. When evaluated against manually annotated data, LapSeg3D achieves an F1 score of 0.94 for gallbladder segmentation on various datasets of ex-vivo porcine livers. We show LapSeg3D to generalize accurately across different gallbladders and datasets recorded with different RGB-D camera systems.

3. Method and System for Determining Optimized Program Parameters for a Robot Program

   Benjamin Alt, Rainer Jäkel, and Darko Katic

   Feb 2022
4. Mensch-Roboter-Kollaboration in KMU – Potenziale identifizieren, analysieren und realisieren

   Lina Kluy, Lena Kölmel, Benjamin Alt, and 11 more authors

   In Digitalisierung der Arbeitswelt im Mittelstand 1: Ergebnisse und Best Practice des BMBF-Forschungsschwerpunkts "Zukunft der Arbeit: Mittelstand – innovativ und sozial", Feb 2022

   Abs DOI

   Die Einführung von kollaborativen Robotern (Cobots) verspricht eine aus arbeitswissenschaftlicher Sichtweise sinnvolle Zusammenarbeit zwischen Mensch und Roboter. Gerade kleinen und mittleren Unternehmen (KMU) gelingt es allerdings häufig nicht oder nur teilweise, die damit verbundenen Potenziale zu erschließen. Das vorliegende Kapitel beschreibt zum einen die Identifizierung, Analyse und Realisierung von Cobot-Potenzialen im Projekt ProBot (,,Proaktive Diagnose und Gestaltung des CoBot-Einsatzes in kleinen und mittleren Unternehmen“). Zum anderen wird die Entwicklung und der Aufbau einer interaktiven Einführungsunterstützung zur niederschwelligen Planung und Umsetzung von Mensch-Cobot-ArbeitssystemenMensch-Cobot-Arbeitssystem (MCA) (MCA) skizziert, wobei einige der enthaltenen Methoden vertieft vorgestellt werden. Darüber hinaus werden anhand realer Fallbeispiele von KMU die Möglichkeiten und Grenzen von Mensch-Roboter-Kollaborationen (MRK) anschaulich dargestellt. Die interaktive Einführungsunterstützung (KMU Cobot Coach) wird interessierten KMU unter folgender Adresse zur freien Verfügung online bereitgestellt: https://www.kmu-cobot-coach.de/.

2021

1. Robot Program Parameter Inference via Differentiable Shadow Program Inversion

   Benjamin Alt, Darko Katic, Rainer Jäkel, and 2 more authors

   In 2021 IEEE International Conference on Robotics and Automation (ICRA), May 2021

   Abs DOI arXiv PDF

   Challenging manipulation tasks can be solved effectively by combining individual robot skills, which must be parameterized for the concrete physical environment and task at hand. This is time-consuming and difficult for human programmers, particularly for force-controlled skills. To this end, we present Shadow Program Inversion (SPI), a novel approach to infer optimal skill parameters directly from data. SPI leverages unsupervised learning to train an auxiliary differentiable program representation ("shadow program") and realizes parameter inference via gradient-based model inversion. Our method enables the use of efficient first-order optimizers to infer optimal parameters for originally non-differentiable skills, including many skill variants currently used in production. SPI zero-shot generalizes across task objectives, meaning that shadow programs do not need to be retrained to infer parameters for different task variants. We evaluate our methods on three different robots and skill frameworks in industrial and household scenarios. Code and examples are available at https://innolab.artiminds.com/icra2021.

2. Localization and Tracking of User-Defined Points on Deformable Objects for Robotic Manipulation

   Sven Dittus, Benjamin Alt, Andreas Hermann, and 3 more authors

   In IEEE ICRA Workshop on Representing and Manipulating Deformable Objects, May 2021

   Abs arXiv PDF

   This paper introduces an efficient procedure to localize user-defined points on the surface of deformable objects and track their positions in 3D space over time. To cope with a deformable object’s infinite number of DOF, we propose a discretized deformation field, which is estimated during runtime using a multi-step non-linear solver pipeline. The resulting high-dimensional energy minimization problem describes the deviation between an offline-defined reference model and a pre-processed camera image. An additional regularization term allows for assumptions about the object’s hidden areas and increases the solver’s numerical stability. Our approach is capable of solving the localization problem online in a data-parallel manner, making it ideally suitable for the perception of non-rigid objects in industrial manufacturing processes.

2020

1. Modulare, Datengetriebene Roboterprogrammierung Für Die Lösung Komplexer Handhabungsaufgaben in Alltagsumgebungen

   Benjamin Alt, Florian Aumann, Lennart Gienger, and 4 more authors

   In AAL-Kongress 2020, May 2020

   Abs PDF

   Increasing demand for high-standard elderly care and stagnant availability of qualified labor makes the need for partially autonomous robotic ambient assisted living (AAL) solutions increasingly urgent. The preparation and serving of food and beverages is an important subdomain of care robotics, which is rendered challenging by unstructured environments, a large variety of manipulated objects and the need to interact with human users. This work presents a modular, hardware agnostic approach for programming robots to perform complex assistance functions in unstructured environments. We propose a robot-agnostic library of parameterized robot skills which encapsulate primitive tasks central to food preparation and serving, from which hierarchical manipulation sequences can be constructed. We further propose an approach for the automatic adaption of generated motions based on 3D point clouds of the environment to manipulate objects of different dimensions and shapes. We evaluate our solution on two 6-DOF robotic manipulators as well as a bimanual humanoid assistance robot.

2. RoPHa - Robuste Wahrnehmungsfähigkeiten Für Roboter Zur Unterstützung Älterer Nutzer Im Häuslichen Umfeld

   Birgit Graf, Florian Jordan, Gabriele Blume, and 8 more authors

   In Autonome Roboter Für Assistenzfunktionen, May 2020

   Abs HTML

   Ziel des RoPHa-Projekts war es, Grundfunktionen für die Unterstützung älterer und pflegedürftiger Menschen bei der Handhabung typischer Alltagsobjekte zu entwickeln. Als Use Case für die beispielhafte Umsetzung und Integration der entwickelten Grundfunktionen wurde die mundgerechte Bereitstellung von Nahrung betrachtet. Dafür wurden zunächst eine systematische Anforderungsanalyse inklusive der Betrachtung von ELS-Aspekten und eine Risikobetrachtung durchgeführt. Danach wurde ein Interaktionskonzept basierend auf einer tabletgestützten GUI und Sprachausgaben entwickelt. Im Bereich der Perzeption wurden Funktionen für die Objekterkennung, die Lokalisierung und Klassifizierung von Speisen, die Wahrnehmung des Menschen und für die aufgabenspezifische Auswahl und Parametrierung von Perzeptionsfunktionen entwickelt und in ein Gesamtsystem integriert. Für die sichere Manipulation wurden unterschiedliche elementare Handhabungsfertigkeiten, bspw. für das Schneiden, Bestreuen, Aufnehmen und Anreichen von Speisen entwickelt. Im Rahmen von zwei Testreihen mit Mitarbeitern aus der Pflege wurden die entwickelten Szenarien evaluiert.