Vision-based Force Estimation and Haptic Feedback using Neural Networks

Research

Vision-based Force Estimation and Haptic Feedback using Neural Networks

Forcep tip force sensing in robot-assisted minimally invasive surgery is challenging due to strict requirements for miniaturization, biocompatibility, and sterilizability. Indirect force estimation is a promising method to measure forces while circumventing these constraints. Much like how humans can estimate forces visually, neural networks can attempt to do something similar. However, there are concerns as to the generalizability of these methods as well as the relative importance of visual information compared to robot kinematic state information as inputs. We characterize the performance of vision-based neural networks with these considerations in mind and quantify the quality of the closed-loop haptic feedback they can provide to the operator.

Primary Researcher: Shuyuan Yang



Offline NN predictions
Our neural networks are capable of estimating interaction forces offline. In the figure below, the black line is the ground truth force while the green, red, and blue lines are our vision-only, state-only, and vision+state neural networks. We included two benchmarks, an RNN (purple) as well as a physics-based dynamic model (gold).



We also performed a gradient class activation to understand what visual features were being used by the neural networs to estimate force. Here we see from left to right that in the x-, y-, and z-directions, the neural networks are looking at the shadows and deformation of the silicone material. NN gradcam



We have posted part of the dataset from [1] online. This can be access via our github repo page here.

Published Works

S. Yang, M. H. Le, K. R. Golobish, J. C. Beaver, and Z. Chua, “Vision-Based Force Estimation for Minimally Invasive Telesurgery Through Contact Detection and Local Stiffness Models,” Journal of Medical Robotics Research, p. 2440008, 2024.

Z. Chua and A. M. Okamura, “Characterization of Real-time Haptic Feedback from Multimodal Neural Network-based Force Estimates during Teleoperation,” in IEEE/RSJ International Conference on Intelligent Robots and Systems, 2022.

Z. Chua, A. M. Jarc, and A. M. Okamura, “Toward Force Estimation in Robot-Assisted Surgery using Deep Learning with Vision and Robot State,” in IEEE International Conference on Robotics and Automation, 2021, pp. 12335–12341.

Other Research