Sungboo Yoon

In the context of the unstructured and fast-changing construction environment, the adaptability of robots to human improvisation is crucial. However, existing construction robots are limited in understanding spatial goals communicated spontaneously in the workplace and require substantial human-user training. Incorporating deictic gestures into human–robot interfaces holds promise for enhancing the intuitive operation of construction robots and for on-site collaboration. However, applying deictic gestures in precision-demanding construction tasks presents challenges because in a large-scale work environment, humans have limited accuracy in pointing at objects at a distance. This study introduces LaserDex, a novel interface for distant spatial tasking based on a global-to-local identification approach, in which the human first guides the robot toward the general area of the task using a deictic gesture and then indicates the precise area by dynamically using a laser pointer. Our user study with 11 participants demonstrated that LaserDex can achieve an intersection over union (IoU) of 0.830 when outlining a rectangular drywall opening (compared with an IoU of 0.514 for the baseline, a handheld controller), and an 11.4-mm distance error of the center of the estimated rectangle compared with the center of the targeted rectangle. The findings of this study underscore the potential of LaserDex to seamlessly combine intuitive human–robot interaction with precise robot operations.

Construction robots are expected to frequently communicate in situ improvisations with human workers to adapt and change their workflow and methods. One way to achieve this is through deictic gestures that are one of the most effective forms of human–robot interaction (HRI) in delivering spatial information. Nevertheless, the limited coverage of deictic gestures in large-scale environments poses some challenges for both humans and robots in leveraging such techniques for HRI in construction. To identify the feasibility of deictic gestures in the construction domain and find applicable solutions for improving performance, this study aims to extend current knowledge on the performance in communicating positional information using deictic gestures by investigating the effects of spatial characteristics on spatial referencing, focusing on the target configuration, target distance, and relative position of human and robot. We observed that the recognition and estimation of deictic gestures were affected by the target plane, target position, and the target layout and that the robot performance was significantly reduced as the distance between the human and robot increased. The findings of this study demonstrate the challenges in spatial referencing within a large-scale environment and highlight the need for bidirectional communication in HRI.

Dynamic viewpoints offer effective visual feedback in teleoperation for construction, where tasks often require precise manipulation during frequent viewpoint adjustments. However, the comparative performance of various dynamic viewpoint control techniques remains unclear. This paper investigates the impact of dynamic viewpoint control techniques on task performance and user experience during teleoperation in construction. A user study was conducted in a remote welding-at-height scenario with 20 participants, including experienced welders and university students, to compare five techniques: (1) coupled vision-motion, (2) decoupled vision-motion with hand or head motion-based control, and (3) hybrid vision-motion with manual or automatic switching. Results showed that decoupled vision-motion with head motion-based control outperformed other techniques in task efficiency and user preference. Hybrid vision-motion with manual switching was more effective than decoupled vision-motion in contexts involving occlusions, reducing physical demand and enhancing welding quality. Based on these findings, guidelines are proposed for viewpoint control in teleoperated construction robots.

Visual perception is critical for teleoperation in construction, where optimal visibility directly impacts task performance. Hybrid viewpoint control systems enhance the flexibility of visual perception by adaptively coupling or decoupling the viewpoint from robot movements according to situational demands. However, determining the optimal timing for transitions between these perspectives remains a major challenge, as existing autonomous methods are not directly applicable to hybrid control for construction tasks. In this work, we propose a viewpoint control mode prediction model that autonomously manages transitions during teleoperation with hybrid control. Our learning scheme with a transition-guided weighting method leverages sporadic transition commands from human interactions with the teleoperation system as demonstration data for imitation learning. User evaluation in a virtual reality (VR) environment simulating construction welding tasks shows that our model outperforms the baselines, achieving an 11% improvement over the state-of-the-art behavioral cloning (BC) algorithm and a 19% improvement over the state-of-the-art weighted BC algorithm in replicating human transition behaviors. This work contributes novel insights into the design of visual perception systems for teleoperation in construction, enabling reliable, user-aligned viewpoint transitions.