Research
Learning-Enhanced Force and Tactile Sensing
Model-mediated Intrinsic Visual Force Sensing for Compliant Robotic Grippers
Primary Researcher: Kaiwen Zuo
Robots often rely on force sensors to determine how strongly they are grasping an object, but integrating such sensors can increase cost, complexity, and mechanical fragility. We developed a vision-based approach that estimates grasp forces using images captured by a wrist-mounted camera, eliminating the need for dedicated force sensors. Our method combines geometric contact localization with a physics-based model of gripper deformation, allowing the robot to determine both where contact occurs and how much force is being applied. Because the approach explicitly models the mechanics of the gripper rather than relying solely on data-driven learning, it generalizes more effectively to previously unseen objects and grasping scenarios.
Model-mediated Extrinsic Visual Force Sensing in Telesurgery
Primary Researcher: Shuyuan Yang
Integrating force sensors into surgical tools can be challenging because designers have to account for miniaturization, manufacturability, robustness, and sterilizability. However, deformation is a crucial perceptual cue that humans use to infer forces, relating the movement of tissue to force through our own estimate of the tissue's mechanical stiffness. What if we imbued this capability in surgical robots? We developed a vision-based telesurgical force sensing approach that uses data-driven computer vision techniques on endoscope video to observe and track tool-tissue contact and tissue deformation. We integrate machine visual perception with an physics-based stiffness tissue stiffness model to infer tool-tissue interaction forces.
Shaft-integrated Force Sensing with Neural Network-based Dynamics Compensation for Telesurgery
Primary Researcher: Shuyuan Yang, Grant Boone
In collaboration with Resense GmbH, a force sensor manufacturer, we developed a simple and reproducible method for adding a force sensor to a surgical instrument shaft without changing how the instrument operates. One challenge with measuring tool-tissue interaction forces through a shaft-mounted force sensor is that the measurements include both tissue interaction forces and internal forces from the robot's cable-driven transmission system, such as friction, backlash, cable stretch, and cable tension. By combining force measurements with a machine learning model that compensates for the transmission dynamics, our integrated sensing approach can distinguish forces caused by tissue interaction from forces generated internally by the robot itself. This enables accurate force estimation that can inform future force-aware surgical robotic systems.
Open Source Force Sensors for Minimally Invasive Surgical Robotics Research
Primary Researcher: Xiangru Li
We developed a customizable force sensor design that balance ease of manufacture, cost, size, and accuracy for surgical robotics research. This force sensor can be added onto the jaw of a telesurgical forceps, directly measuring tool-tissue interaction forces at their point of application. Unlike shaft-based approaches sensing is more straightforward because it is not affected by internal tool dynamics. We open-sourced the design files and code for this sensor to enable other researchers to easily integrate force sensing into their surgical robotics research.
Assistive and Instructional Feedback for Improved Human-Machine Performance
Pneumatic Suction Sensory Substitution Feedback for Telemanipulation
Primary Researcher: Jiaji Su
Traditional force-feedback devices are often bulky, heavy, and difficult to integrate into surgical robotic interfaces. We developed a lightweight fingertip haptic display that uses gentle pneumatic suction to convey grasping forces during telemanipulation. The device can be integrated into existing robotic manipulanda with minimal modification and works alongside passive kinesthetic feedback already present in many commercial surgical robots. User studies showed that suction-based feedback improved stiffness discrimination and encouraged operators to apply lower grasp forces during manipulation, demonstrating a compact and practical approach for enhancing human performance in teleoperation.
Soft Continuum Kinesthetic Feedback for Telemanipulation
Primary Researcher: Jiaji Su
We developed a soft handheld haptic device that fits on the finger and recreates three-dimensional force sensations during teleoperation. Unlike conventional haptic devices that rely on rigid mechanisms that are affixed to a table or workspace, our design uses soft continuum structures and pneumatic actuation to improve comfort and reduce weight, to make them more suitable for handheld wireless interfaces while still providing meaningful kinesthetic feedback.
Haptic Assistance and Resistance for Improving Dexterous Telemanipulation
Primary Researcher: Zonghe Chua
Much like a coach can physically guide a learner through a motion, haptic systems can be programmed to provide instructional feedback. One mechanism by which humans learn, is by observing their mistakes and correcting them. We investigate how humans learn from haptic feedback that either assists and reduces their mistakes, or resist their movements and amplifies their mistakes. Would amplifying their mistakes increase their learning rate? Would assisting them make them more likely to be too reliant on our assistance? We found that assistive feedback helped less skilled users learn, althought their performance quickly plateaued. In contrast, resistive feedback helped more skilled users learn, and was particularly beneficial if the user preferred to move slowly and patiently to complete the task. This work suggests that haptic feedback can be used to improve telemanipulation performance, but the design of this feedback should consider the skill level and preferences of the user.
Learning Visual Force Estimation during Teleoperation of a Surgical Robot
Primary Researcher: Zonghe Chua
Leading robot-assisted minimally invasive surgery platforms do not provide force feedback to the surgeon, so surgeons rely on visual force estimation to judge tissue interaction forces. We investigated how training with force feedback influences visual force estimation during teleoperated soft environment manipulation. We found that teleoperator are able to learn to make more accurate visual force estimates with training, and that this learning is enhanced when force feedback is provided during training. This work suggests that providing force feedback during training can improve surgeons' ability to visually estimate forces during teleoperation, which has implications for the design of future surgical robotic systems.