![](https://robohub.org/wp-content/uploads/2023/10/MIT-Svelte-Finger-01-PRESS_0.jpg)
MIT researchers have developed a long, curved camera-based touch sensor that resembles the shape of a human finger. Their device, which provides high-resolution tactile sensing over a large area, allows robotic hands to perform different types of grasps. Image: Provided by researchers
Posted by Adam Zewe | MIT News
Imagine holding a heavy object, such as a pipe wrench, with one hand. You are likely to use your entire finger to hold the wrench, not just your fingertips. Sensory receptors in the skin that run along the entire length of each finger send information to the brain about the tool being held.
Tactile sensors, which use a camera in a robot hand to obtain information about the object being grasped, are small and flat and are often located at the fingertips. These robots use only their fingertips to grasp objects, typically in a pinching motion. This limits the manipulation operations that can be performed.
MIT researchers have developed a long, curved camera-based touch sensor that resembles the shape of a human finger. The device provides high-resolution tactile sensing over a large area. Called GelSight Svelte, the sensor uses two mirrors to reflect and refract light, allowing a single camera on the bottom of the sensor to see along the entire length of the finger.
The researchers also created a finger-shaped sensor with a flexible backbone. By measuring how the spine bends when a finger touches an object, the force applied to the sensor can be estimated.
They used the GelSight Svelte sensor to create a robotic hand that can grasp heavy objects like a human, using the full sensing area of three fingers. The hand can also perform the same pinch grip as a traditional robotic gripper.
![](https://news.mit.edu/sites/default/files/images/inline/Svelte-finger_0.gif)
This gif shows a robotic hand incorporating three finger-shaped GelSight Svelte sensors. Providing high-resolution tactile detection over a large area, the sensor allows the hand to perform multiple grips, including a pinch grip using only the fingertips and a power grip using the entire sensing area of three fingers. Credit: Provided by Researcher
“Because our new sensor is shaped like a human finger, it can be used to perform different types of grips for different tasks, rather than using a pinch grip for every task. There are limits to what a parallel jaw gripper can do. “Our sensors open up new possibilities for a variety of manipulation tasks that can be done with robots,” says Alan (Jialiang) Zhao, a graduate student in mechanical engineering and lead author of the paper on GelSight Svelte.
Zhao co-wrote the paper with lead author Edward Adelson, professor of vision science in the Department of Brain and Cognitive Sciences, and the John and Dorothy Wilson, a member of the Computer Science and Artificial Intelligence Laboratory (CSAIL). The research will be presented at the IEEE Intelligent Robots and Systems Conference.
mirror mirror
Cameras used in tactile sensors are limited by their size, focal length of the lens, and viewing angle. Therefore, these tactile sensors tend to be small and flat, confined to the robot's fingertips.
A longer sensing area, more similar to a human finger, requires the camera to be farther away from the sensing surface to see the entire area. This is particularly challenging due to the size and shape limitations of the robotic gripper.
Zhao and Adelson solved this problem by using two mirrors that reflect and refract light toward a single camera located at the base of the finger.
The GelSight Svelte integrates one flat, angled mirror opposite the camera and one long, curved mirror on the back of the sensor. These mirrors redistribute the light rays coming from the camera so that the camera can see along the entire length of the finger.
To optimize the mirror's shape, angle, and curvature, the researchers designed software to simulate the reflection and refraction of light.
“The software allows us to easily manipulate the position of the mirror and the shape of the curve to get an idea of how good the image will look after we actually build the sensor,” Zhao explains.
The mirror, camera and two sets of LEDs for lighting are attached to a plastic backbone and encased in a flexible skin made of silicone gel. The camera looks backwards, inside the skin. Based on the deformation, you can determine where contact occurs and measure the shape of the object's contact surface.
![](https://robohub.org/wp-content/uploads/2023/10/MIT-Svelte-Finger-02-PRESS.jpg)
An analysis of the components that make up a finger-shaped touch sensor. Image: Provided by researchers
Additionally, an array of red and green LEDs indicate how deeply the gel is pressed when grasping an object through color saturation at various positions on the sensor.
Researchers can use this color saturation information to reconstruct a 3D depth image of the object being viewed.
The plastic backbone of the sensor allows proprioceptive information, such as the twisting torque applied to the finger, to be determined. The spine bends and bends when grasping an object. Researchers use machine learning to estimate how much force is being applied to the sensor based on these backbone deformations.
But combining these elements into a working sensor is no easy task, Zhao says.
“It is quite difficult to get the right curvature of the mirror to match what we get from the simulation. Additionally, I realized that there is some kind of super glue that prevents the silicone from curing. “It took a lot of experimentation to create a sensor that actually worked,” he adds.
different grips
After finalizing the design, the researchers tested GelSight Svelte by pressing objects, such as screws, into different locations on the sensor to check image clarity and see how well it could determine the shape of an object.
We also created the GelSight Svelte hand, which can perform a variety of grasps using three sensors, including a pinch grasp, a lateral pinch grasp, and a power grasp using the entire sensing area of three fingers. Most robotic hands in the parallel jaw dripper shape can only perform pinch grasps.
The three-finger power grip allows the robot hand to grip heavier objects more reliably. However, pinch grabbing is still useful even when the object is very small. Being able to perform both types of grips with one hand would make the robot more versatile, he says.
In the future, researchers plan to improve GelSight Svelte so that the sensor can bend at the joints like a human finger.
“Phototactile finger sensors allow robots to collect high-resolution images of surface contact using inexpensive cameras, and by observing the deformation of the flexible surface, the robot estimates the shape of the contact and the force applied. “This work represents an advancement in GelSight finger design with improved overall finger coverage and the ability to approximate bending deflection torque using image differences and machine learning,” said Monroe Kennedy, assistant professor of mechanical engineering at Stanford University. III, who was not involved in this study, says, “Improving the sense of touch in robots to approach human capabilities is essential and perhaps a catalyst issue for developing robots that can perform complex and skilled tasks.”
This research is supported in part by the Toyota Research Institute.
MIT News