Publications

Preprint

1. Alt, B., Dvorak, J., Katic, D., Jäkel, R., Beetz, M., & Lanza, G. (2024). BANSAI: Towards Bridging the AI Adoption Gap in Industrial Robotics with Neurosymbolic Programming. arXiv.

   Abstract ArXiv PDF

   Over the past decade, deep learning helped solve manipulation problems across all domains of robotics. At the same time, industrial robots continue to be programmed overwhelmingly using traditional program representations and interfaces. This paper undertakes an analysis of this "AI adoption gap" from an industry practitioner’s perspective. In response, we propose the BANSAI approach (Bridging the AI Adoption Gap via Neurosymbolic AI). It systematically leverages principles of neurosymbolic AI to establish data-driven, subsymbolic program synthesis and optimization in modern industrial robot programming workflow. BANSAI conceptually unites several lines of prior research and proposes a path toward practical, real-world validation.

2. Alt, B., Zahn, J., Kienle, C., Dvorak, J., May, M., Katic, D., Jäkel, R., Kopp, T., Beetz, M., & Lanza, G. (2024). Human-AI Interaction in Industrial Robotics: Design and Empirical Evaluation of a User Interface for Explainable AI-Based Robot Program Optimization. arXiv.

   Abstract ArXiv PDF

   While recent advances in deep learning have demonstrated its transformative potential, its adoption for real-world manufacturing applications remains limited. We present an Explanation User Interface (XUI) for a state-of-the-art deep learning-based robot program optimizer which provides both naive and expert users with different user experiences depending on their skill level, as well as Explainable AI (XAI) features to facilitate the application of deep learning methods in real-world applications. To evaluate the impact of the XUI on task performance, user satisfaction and cognitive load, we present the results of a preliminary user survey and propose a study design for a large-scale follow-up study.

3. Alt, B., Stöckl, F., Müller, S., Braun, C., Raible, J., Alhasan, S., Rettig, O., Ringle, L., Katic, D., Jäkel, R., Beetz, M., Strand, M., & Huber, M. F. (2024). RoboGrind: Intuitive and Interactive Surface Treatment with Industrial Robots. arXiv.

   Abstract ArXiv PDF

   Surface treatment tasks such as grinding, sanding or polishing are a vital step of the value chain in many industries, but are notoriously challenging to automate. We present RoboGrind, an integrated system for the intuitive, interactive automation of surface treatment tasks with industrial robots. It combines a sophisticated 3D perception pipeline for surface scanning and automatic defect identification, an interactive voice-controlled wizard system for the AI-assisted bootstrapping and parameterization of robot programs, and an automatic planning and execution pipeline for force-controlled robotic surface treatment. RoboGrind is evaluated both under laboratory and real-world conditions in the context of refabricating fiberglass wind turbine blades.

Conference papers

2024

1. Alt, B., Keßner, U., Taranovic, A., Katic, D., Hermann, A., Jäkel, R., & Neumann, G. (2024). Domain-Specific Fine-Tuning of Large Language Models for Interactive Robot Programming. European Robotics Forum 2024.

   Abstract ArXiv PDF

   Industrial robots are applied in a widening range of industries, but robot programming mostly remains a task limited to programming experts. We propose a natural language-based assistant for programming of advanced, industrial robotic applications and investigate strategies for domain-specific fine-tuning of foundation models with limited data and compute.

2. Stöckl, F., Strand, M., Müller, S., Huber, M., Raible, J., Braun, C., Katic, D., Alt, B., & Merkt, H. (2024). Autonomous Surface Grinding of Wind Turbine Blades. In S.-G. Lee, J. An, N. Y. Chong, M. Strand, & J. H. Kim (Eds.), Intelligent Autonomous Systems 18 (pp. 451–457). Springer Nature Switzerland.

   Abstract Link

   Discarded wind turbine blades generate a considerable amount of waste that could be reduced by remanufacturing. Manual remanufacturing is too costly, which is why research is being conducted into automation techniques. The main problem is the individuality of work pieces due to damages. This work presents a workflow that includes damage analysis based on scans of the blade, subsequent path planning, control engineering with an AI controller for grinding and automatic review of the grinding process. Current problems are the inaccuracy of the robot used for scanning and the colour sensitivity of the used laser scanner. Our next steps besides solving the mentioned problems are to train a supervised machine learning algorithm with damage examples and to implement a specific and multi-step path planning algorithm.

2023

1. Alt, B., Nguyen, M. D., Hermann, A., Katic, D., Jäkel, R., Dillmann, R., & Sax, E. (2023, September). EfficientPPS: Part-aware Panoptic Segmentation of Transparent Objects for Robotic Manipulation. ISR Europe 2023.

   Abstract ArXiv PDF Link

   The use of autonomous robots for assistance tasks in hospitals has the potential to free up qualified staff and improve patient care. However, the ubiquity of deformable and transparent objects in hospital settings poses significant challenges to vision-based perception systems. We present EfficientPPS, a neural architecture for part-aware panoptic segmentation that provides robots with semantically rich visual information for grasping and manipulation tasks. We also present an unsupervised data collection and labelling method to reduce the need for human involvement in the training process. EfficientPPS is evaluated on a dataset containing real-world hospital objects and demonstrated to be robust and efficient in grasping transparent transfusion bags with a collaborative robot arm.

2. Alt, B., Kenfack, F. K., Haidu, A., Katic, D., Jäkel, R., & Beetz, M. (2023). Knowledge-Driven Robot Program Synthesis from Human VR Demonstrations. Proceedings of the 20th International Conference on Principles of Knowledge Representation and Reasoning, 34–43.

   Abstract ArXiv PDF Link

   Aging societies, labor shortages and increasing wage costs call for assistance robots capable of autonomously performing a wide array of real-world tasks. Such open-ended robotic manipulation requires not only powerful knowledge representations and reasoning (KR&R) algorithms, but also methods for humans to instruct robots what tasks to perform and how to perform them. In this paper, we present a system for automatically generating executable robot control programs from human task demonstrations in virtual reality (VR). We leverage common-sense knowledge and game engine-based physics to semantically interpret human VR demonstrations, as well as an expressive and general task representation and automatic path planning and code generation, embedded into a state-of-the-art cognitive architecture. We demonstrate our approach in the context of force-sensitive fetch-and-place for a robotic shopping assistant. The source code is available at https://github.com/ease-crc/vr-program-synthesis.

3. Raible, J., Rettig, O., Alt, B., Yaman, A., Gauger, I., Biasi, L., Müller, S., Katic, D., Strand, M., & Huber, M. F. (2023). Artificial Neural Network Guided Compensation of Nonlinear Payload and Wear Effects for Industrial Robots. 2023 IEEE 19th International Conference on Automation Science and Engineering (CASE), 1–8. https://doi.org/10.1109/CASE56687.2023.10260559

   Abstract IEEE Xplore

   The absolute accuracy of industrial robots is influ-enced by numerous geometric and non-geometric errors. Most state-of-the-art calibration and compensation methods consider only the geometric errors and neglect the non-geometric ones. In this paper, a hybrid compensation approach is proposed that combines well-known kinematic models with a model-free data-driven component using a neural network. This allows the use of established calibration methods for the geometric influences captured in the kinematic model to improve the non-geometric error compensation by the neural network. The proposed approach is applied in two use cases: payload and wear compensation. Simulations and real experiments show the improved absolute accuracy of the hybrid compensation approach compared to classical calibration methods.

2022

1. Alt, B., Katic, D., Jäkel, R., & Beetz, M. (2022). Heuristic-free Optimization of Force-Controlled Robot Search Strategies in Stochastic Environments. 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 8887–8893. https://doi.org/10.1109/IROS47612.2022.9982093

   Abstract IEEE Xplore ArXiv PDF

   In both industrial and service domains, a central benefit of the use of robots is their ability to quickly and reliably execute repetitive tasks. However, even relatively simple peg-in-hole tasks are typically subject to stochastic variations, requiring search motions to find relevant features such as holes. While search improves robustness, it comes at the cost of increased runtime: More exhaustive search will maximize the probability of successfully executing a given task, but will significantly delay any downstream tasks. This trade-off is typically resolved by human experts according to simple heuristics, which are rarely optimal. This paper introduces an automatic, data-driven and heuristic-free approach to optimize robot search strategies. By training a neural model of the search strategy on a large set of simulated stochastic environments, conditioning it on few real-world examples and inverting the model, we can infer search strategies which adapt to the time-variant characteristics of the underlying probability distributions, while requiring very few real-world measurements. We evaluate our approach on two different industrial robots in the context of spiral and probe search for THT electronics assembly.**See github.com/benjaminalt/dpse for code and data.

2. Alt, B., Kunz, C., Katic, D., Younis, R., Jäkel, R., Müller-Stich, B. P., Wagner, M., & Mathis-Ullrich, F. (2022). LapSeg3D: Weakly Supervised Semantic Segmentation of Point Clouds Representing Laparoscopic Scenes. 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 5265–5270. https://doi.org/10.1109/IROS47612.2022.9981178

   Abstract IEEE Xplore ArXiv PDF

   The semantic segmentation of surgical scenes is a prerequisite for task automation in robot assisted interventions. We propose LapSeg3D, a novel DNN-based approach for the voxel-wise annotation of point clouds representing surgical scenes. As the manual annotation of training data is highly time consuming, we introduce a semi-autonomous clustering-based pipeline for the annotation of the gallbladder, which is used to generate segmented labels for the DNN. When evaluated against manually annotated data, LapSeg3D achieves an F1 score of 0.94 for gallbladder segmentation on various datasets of ex-vivo porcine livers. We show LapSeg3D to generalize accurately across different gallbladders and datasets recorded with different RGB-D camera systems.

2021

1. Alt, B., Katic, D., Jäkel, R., Bozcuoglu, A. K., & Beetz, M. (2021). Robot Program Parameter Inference via Differentiable Shadow Program Inversion. 2021 IEEE International Conference on Robotics and Automation (ICRA).

   Abstract IEEE Xplore ArXiv PDF Blog Code Website

   Challenging manipulation tasks can be solved effectively by combining individual robot skills, which must be parameterized for the concrete physical environment and task at hand. This is time-consuming and difficult for human programmers, particularly for force-controlled skills. To this end, we present Shadow Program Inversion (SPI), a novel approach to infer optimal skill parameters directly from data. SPI leverages unsupervised learning to train an auxiliary differentiable program representation (“shadow program”) and realizes parameter inference via gradient-based model inversion. Our method enables the use of efficient first-order optimizers to infer optimal parameters for originally non-differentiable skills, including many skill variants currently used in production. SPI zero-shot generalizes across task objectives, meaning that shadow programs do not need to be retrained to infer parameters for different task variants. We evaluate our methods on three different robots and skill frameworks in industrial and household scenarios. Code and examples are available at https://innolab.artiminds.com/icra2021.

2. Dittus, S., Alt, B., Hermann, A., Katic, D., Jäkel, R., & Fleischer, J. (2021, May). Localization and Tracking of User-Defined Points on Deformable Objects for Robotic Manipulation. IEEE ICRA Workshop on Representing and Manipulating Deformable Objects.

   Abstract ArXiv PDF

   This paper introduces an efficient procedure to localize user-defined points on the surface of deformable objects and track their positions in 3D space over time. To cope with a deformable object’s infinite number of DOF, we propose a discretized deformation field, which is estimated during runtime using a multi-step non-linear solver pipeline. The resulting high-dimensional energy minimization problem describes the deviation between an offline-defined reference model and a pre-processed camera image. An additional regularization term allows for assumptions about the object’s hidden areas and increases the solver’s numerical stability. Our approach is capable of solving the localization problem online in a data-parallel manner, making it ideally suitable for the perception of non-rigid objects in industrial manufacturing processes.

2020

1. Alt, B., Aumann, F., Gienger, L., Jordan, F., Katic, D., Jäkel, R., & Graf, B. (2020). Modulare, datengetriebene Roboterprogrammierung für die Lösung komplexer Handhabungsaufgaben in Alltagsumgebungen. "Active Assisted Living" Wissenschaftskongress (AAL)/Deutsche Gesellschaft Für Biomedizinische Technik (DGBMT Jahrestagung), 17–22.

   Abstract Fraunhofer Publica PDF

   Increasing demand for high-standard elderly care and stagnant availability of qualified labor makes the need for partially autonomous robotic ambient assisted living (AAL) solutions increasingly urgent. The preparation and serving of food and beverages is an important subdomain of care robotics, which is rendered challenging by unstructured environments, a large variety of manipulated objects and the need to interact with human users. This work presents a modular, hardware agnostic approach for programming robots to perform complex assistance functions in unstructured environments. We propose a robot-agnostic library of parameterized robot skills which encapsulate primitive tasks central to food preparation and serving, from which hierarchical manipulation sequences can be constructed. We further propose an approach for the automatic adaption of generated motions based on 3D point clouds of the environment to manipulate objects of different dimensions and shapes. We evaluate our solution on two 6-DOF robotic manipulators as well as a bimanual humanoid assistance robot.

Book chapters

2022

1. Kluy, L., Kölmel, L., Alt, B., Baumgartner, M., Deml, B., Hornung, L., Katic, D., Kinkel, S., Kopp, T., Lorenz, M., Nicolai, P., Riedel, N., Schäfer, A., & Wurll, C. (2022). Mensch-Roboter-Kollaboration in KMU – Potenziale identifizieren, analysieren und realisieren. In V. Nitsch, C. Brandl, R. Häußling, J. Lemm, T. Gries, & B. Schmenk (Eds.), Digitalisierung der Arbeitswelt im Mittelstand 1: Ergebnisse und Best Practice des BMBF-Forschungsschwerpunkts "Zukunft der Arbeit: Mittelstand – innovativ und sozial" (pp. 55–97). Springer. https://doi.org/10.1007/978-3-662-64803-2_3

   Abstract Link

   Die Einführung von kollaborativen Robotern (Cobots) verspricht eine aus arbeitswissenschaftlicher Sichtweise sinnvolle Zusammenarbeit zwischen Mensch und Roboter. Gerade kleinen und mittleren Unternehmen (KMU) gelingt es allerdings häufig nicht oder nur teilweise, die damit verbundenen Potenziale zu erschließen. Das vorliegende Kapitel beschreibt zum einen die Identifizierung, Analyse und Realisierung von Cobot-Potenzialen im Projekt ProBot („Proaktive Diagnose und Gestaltung des CoBot-Einsatzes in kleinen und mittleren Unternehmen“). Zum anderen wird die Entwicklung und der Aufbau einer interaktiven Einführungsunterstützung zur niederschwelligen Planung und Umsetzung von Mensch-Cobot-ArbeitssystemenMensch-Cobot-Arbeitssystem (MCA) (MCA) skizziert, wobei einige der enthaltenen Methoden vertieft vorgestellt werden. Darüber hinaus werden anhand realer Fallbeispiele von KMU die Möglichkeiten und Grenzen von Mensch-Roboter-Kollaborationen (MRK) anschaulich dargestellt. Die interaktive Einführungsunterstützung (KMU Cobot Coach) wird interessierten KMU unter folgender Adresse zur freien Verfügung online bereitgestellt: https://www.kmu-cobot-coach.de/.

2020

1. Graf, B., Jordan, F., Blume, G., Martin, R., Abdel-Keream, M., Mania, P., Gronbach, F., Emmerich, C., Schaller, R., Katic, D., & Alt, B. (2020). RoPHa - Robuste Wahrnehmungsfähigkeiten für Roboter zur Unterstützung älterer Nutzer im häuslichen Umfeld. In Autonome Roboter für Assistenz­funktionen (pp. 118–133). Bundesanstalt für Arbeitsschutz und Arbeitsmedizin.

   Abstract Fraunhofer Publica

   Ziel des RoPHa-Projekts war es, Grundfunktionen für die Unterstützung älterer und pflegedürftiger Menschen bei der Handhabung typischer Alltagsobjekte zu entwickeln. Als Use Case für die beispielhafte Umsetzung und Integration der entwickelten Grundfunktionen wurde die mundgerechte Bereitstellung von Nahrung betrachtet. Dafür wurden zunächst eine systematische Anforderungsanalyse inklusive der Betrachtung von ELS-Aspekten und eine Risikobetrachtung durchgeführt. Danach wurde ein Interaktionskonzept basierend auf einer tabletgestützten GUI und Sprachausgaben entwickelt. Im Bereich der Perzeption wurden Funktionen für die Objekterkennung, die Lokalisierung und Klassifizierung von Speisen, die Wahrnehmung des Menschen und für die aufgabenspezifische Auswahl und Parametrierung von Perzeptionsfunktionen entwickelt und in ein Gesamtsystem integriert. Für die sichere Manipulation wurden unterschiedliche elementare Handhabungsfertigkeiten, bspw. für das Schneiden, Bestreuen, Aufnehmen und Anreichen von Speisen entwickelt. Im Rahmen von zwei Testreihen mit Mitarbeitern aus der Pflege wurden die entwickelten Szenarien evaluiert.

Patents

2022

1. Alt, B., Jäkel, R., & Katic, D. (2022). Method and System for Determining Optimized Program Parameters for a Robot Program.

   DEPATISnet