Abstract
Soft-bodied robots with multimodal sensing capabilities hold promise for versatile and user-friendly robotics. However, seamlessly integrating multiple sensing functionalities into soft artificial skins remains a challenge due to compatibility issues between soft materials and conventional electronics. While vision-based tactile sensing has enabled simple and effective sensor designs for robotic touch, there has been limited exploration of this technique for intrinsic multimodal sensing in large-sized soft robot bodies. To address this gap, this paper introduces a novel vision-based soft sensing technique, named ProTac, capable of operating either in tactile or proximity sensing modes. This vision-based sensing technology relies on a soft functional skin that can actively switch its optical properties between opaque and transparent states. Furthermore, the paper develops efficient learning pipelines for proximity and tactile perceptions, as well as sensing strategies enabled through the timing activation of the two sensing modes. The effectiveness of the soft sensing technology is demonstrated through a soft ProTac link, which can be integrated into newly constructed or existing commercial robot arms. Results suggest that robots integrated with the ProTac link, along with rigorous control formulation, can perform safe and purposeful control actions, enhancing human-robot interaction scenarios and enabling motion control tasks that are difficult with conventional rigid links.
ProTac Design
We introduce ProTac, a vision-based soft sensing link with controllable optical properties. Its design integrates tactile and proximity sensing into a soft robotic link, enabling whole-body, contact-rich, and proximity-aware manipulation.
Inspiration from Electrochromic Windows
ProTac draws inspiration from electrochromic windows, which regulate transparency using voltage. This concept enables a switchable, vision-based sensing mechanism for soft robot bodies.
Internal Design and Construction
ProTac features a modular cylindrical structure composed of a soft PDLC skin, internal markers, embedded cameras, and a supporting brace. This design enables effective multimodal sensing over a large surface area.
Working Principle: Transparency Switching
ProTac operates in two modes: in the opaque state, it detects contact across the large-area skin using a deep neural network applied to marker-featured images; in the transparent state, it performs proximity sensing via monocular depth estimation applied to see-through images. This switchable mechanism enables dual-mode perception in a single soft robot body.
ProTac Perception
Proximity Sensing
In its transparent state, ProTac estimates the proximity of nearby objects, providing pre-contact awareness for safe interaction.
Tactile Sensing
In the opaque state, ProTac functions as a tactile modality by leveraging a sim-to-real learning pipeline to estimate contact location and intensity.
Flickering Sensing
The flickering mode enables near-simultaneous proximity and tactile perception by rapidly switching between the two sensing states at a high frequency. This enables ProTac’s unique operational mode, providing simultaneous proximity and contact awareness.
ProTac Control
Multimodal ProTac Control
Adaptive Motion Control with Obstacle and Contact Awareness
By combining tactile and proximity sensing, ProTac delivers resilient, contact-reactive motion in environments with unavoidable physical obstacles and contacts.
Multi-phase Human-Robot Interaction
ProTac supports multiphase human-robot interaction by distinguishing between scenarios with and without physical interaction intent. This facilitates shared autonomy and close human collaboration.
Human Passerby without Physical Interaction Intention
Human with Physical Interaction Intention
Other ProTac Control Integration
Admittance-Based Reflex Control
ProTac enables reflexive motion through admittance control, responding flexibly to approaching obstacles and unexpected contacts.
Proximity-Aware Speed Regulation
By continuously estimating the distance to nearby objects, ProTac dynamically adjusts the motion speed, which is expected to ensure safe operation in shared environments.
Related Work
This work builds upon our previous proof-of-concept soft robotic link with controllable skin transparency, ProTac. The tactile sensing pipeline is adapted from our earlier work, SimTacLS, a physics-informed simulation and learning platform for large-area vision-based tactile sensors.
Soft Robotic Link with Controllable Transparency for Vision-based Tactile and Proximity Sensing
Quan Khanh Luu, Dinh Quang Nguyen, Nhan Huu Nguyen, Van Anh Ho
IEEE RoboSoft 2023
[Paper]
[Code]
[Video]
[BibTeX]
This study presents ProTac, a novel soft robotic link with integrated tactile and proximity sensing.
Simulation, Learning, and Application of Vision-Based Tactile Sensing at Large Scale
Quan Khanh Luu, Nhan Huu Nguyen, Van Anh Ho
IEEE Transaction on Robotics (T-RO), 2023
[Paper]
[Code]
[Video]
[BibTeX]
This study introduces SimTacLS, a physics-informed simulation and learning pipeline designed for large-area soft vision-based tactile sensors.
TacLink-Integrated Robot Arm toward Safe Human-Robot Interaction
Quan Khanh Luu, Alessandro Albini, Perla Maiolino, Van Anh Ho
IROS 2024
[Paper]
[Code]
[Video]
[BibTeX]
This study explores the use of TacLink, a soft vision-based tactile link, as a safety control mechanism that can potentially replace conventional rigid robot links and impact observers.
Vi2TaP: A Cross-Polarization Based Mechanism for Perception Transition in Tactile-Proximity Sensing with Applications to Soft Grippers
Nhan Huu Nguyen, Nhat Minh Dinh Le, Quan Khanh Luu, Tuan Tai Nguyen, Van Anh Ho
IEEE Robotics and Automation Letters (RA-L), 2025
[Paper]
[Video]
[BibTeX]
This study introduces Vi2TaP, a cross-polarization-based multimodal soft gripper that seamlessly switches between tactile and proximity sensing.