AR Situational Task Accept & Repair
HRI 2024 LBR Publication
Presentation
About
Robotic solutions are being deployed in industry to perform increasingly challenging maintenance and inspection tasks. In many industrial applications, robots can encounter uncertainties that prevent task completion, potentially resulting in unexpected costs and unsafe working conditions. Supervised autonomy allows personnel to intervene and assist in negotiating these challenging situations. Augmented Reality (AR) allows for the visualization of complex robotic sensor data and provides an opportunity for non-expert users to interact with it in a natural and intuitive manner. We present a comprehensive AR application module—Augmented Reality-Situational Task Accept and Repair (AR-STAR)—that allows users to visualize LiDAR point cloud data along with sensor images and navigational goals. AR-STAR enables users to interact with LiDAR point clouds and modify them in real time. We developed and evaluated three interaction modalities that enable users to manipulate point cloud data in situ while observing the data superimposed on the physical environment. Demonstrated in a human-robot teaming scenario where tasked to identify and repair surface corrosion in a simulated industrial setting, we evaluate AR-STAR with pilot studies to determine the preferred interaction modality based on user workload, system usability, and task completion time to improve improve interaction efficiency and maximize cost savings, respectively.
System Overview
Citation
@inproceedings{regalswanbeckARSTARAugmentedReality2024,
title = {AR-STAR: An Augmented Reality Tool for Online Modification of Robot Point Cloud Data},
author = {Regal, Frank and Swanbeck, Steven and Parra, Fabian and Pryor, Mitch},
booktitle = {HRI '24 Companion},
eventtitle = {ACM/IEEE International Conference on Human-Robot Interaction (HRI)},
publisher = {ACM},
isbn = {979-8-4007-0323-2/24/03},
doi = {10.1145/3610978.3640571},
year = {2024},
location = {Boulder, CO}
}
IROS 2023 XR-ROB Workshop
About
This work was presented at 2023 IEEE/RSJ IROS Second International Horizons of an Extended Robotics Reality (XR-ROB) Workshop where it won 2nd place!
The work is focused on human-robot interactions in industry. Industrial robotics are revolutionizing inspection and maintenance in various sectors by improving safety, efficiency, and sustainability. Specifically, in outdoor industrial settings, mobile manipulators are being used to detect and repair corrosion on complex surfaces. They identify corroded areas and apply protective coatings. However, since this technology is relatively new and errors in coating essential equipment can have serious consequences, human supervision is necessary. To facilitate this, an Augmented Reality (AR) system has been developed to allow non-experts to easily review and modify robot-generated corrosion repair plans in real-time. This system, built on the Augmented Robot Environment (AugRE) framework, includes a module called Situational Task Accept and Repair (STAR). STAR enables users to view corrosion images, point cloud data, and robot objectives in the physical environment, and adjust the repair plan using interactive holographic tools to avoid damaging nearby equipment. The system is demonstrated using Microsoft HoloLens 2 and a dual-arm mobile manipulator. Future research will focus on assessing user experience, system reliability, and real-world application.
Citation
@inproceedings{regalUsingAugmentedReality2023,
title = {Using Augmented Reality to Assess and Modify Mobile Manipulator Surface Repair Plans},
author = {Regal, Frank and Swanbeck, Steven and Parra, Fabian and Rosenbaum, Jared and Pryor, Mitch},
booktitle = {2023 IEEE/RSJ IROS Second International Horizons of an Extended Robotics Reality (XR-ROB) Workshop},
eventtitle = {2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
publisher = {arXiv:2311.00988},
url = {https://arxiv.org/abs/2311.00988},
year = {2023},
location = {Detroit, USA}
}