Edited by: Anjella Kang Lee | Graphic by: Jeslyn Chen
Why teaching machines to feel may be harder than teaching them to see.
Suppose a robot is given an egg. Its cameras identify the shell, gauge its dimensions, and guide its fingers towards it. Yet, we tend to overlook the most intricate part of the problem: picking it up!
If the grip is too loose, the egg simply slips from the robot’s fingers. It’ll be tempting to give the egg a tighter squeeze, but if you apply just a little too much force, you’ll end up with shattered shells and messy yolk all over your kitchen table. Humans can perform this balancing act almost subconsciously because our fingertips are constantly signaling our brain on whether something is sliding, deforming, or pushing back. Yet this perpetual exchange of pressure, motion, and resistance between hand and object remains far beyond what a camera can perceive.
This is precisely AI’s body problem. Machines have made great strides in their ability to interpret images through computer vision, but physical intelligence is much more difficult since it still hinges on what they can sense at the point of contact. The solution is to give robots touch sensations, which is something researchers commonly refer to as “tactile sensing”.
One ingenious solution is an innovation invented by the Massachusetts Institute of Technology called GelSight. They have found that by using a tactile sensor at the end of the robot’s finger, an embedded camera underneath the soft material can record just how the gel surface deforms when it comes into contact with an object. Through observing these minute deformations, the system can infer surface texture, calculate the contact force, and, amazingly, even how much it is currently slipping – information which would be almost impossible for an external camera to capture.
However, detecting touch is only a fragment of the challenge. A dexterous robotic hand must possess the ability to detect the precise moment an object begins to slip and respond to it immediately by modulating its grip.
In a Nature publication, Gautham and collaborators used a skin-like electronic textile wrapped around robotic fingers to demonstrate the closed feedback loop. They steadily added metal beads to a container held by the robot until the overburdening weight caused the container to start slipping out of the robot’s hand. The artificial skin detected the slip within an average of 34 milliseconds, triggering the gripper to tighten its grip and eventually stop the object from its impending fall. The key insight here is not simply the fact that the skin can detect a slip quickly (which it evidently can), but also the discovery that robots can now physically act on information given to them.
NeuralFeels, a multimodal robotics framework, successfully fused both touch and vision to identify the position of the object and reconstruct its geometry all while moving in a multi-fingered artificial hand. The fusion of these two technologies is especially valuable as a robot’s own fingers will often obscure its view of the object, thus hindering the camera from capturing precise measurements of the object. For instance, in experiments, one could see that when visual occlusion is high, incorporating touch remarkably improved tracking by up to 94% compared with vision alone. This type of fusion of multiple sensory inputs is scientifically referred to as multimodal perception. Instead of replacing vision, touch acts as a corroborating witness to an otherwise incomplete or ambiguous visual scene.
Researchers have taken this concept out of perception and into dexterous action. In 3D-ViTac, visual and tactile data are typically combined into a single 3D model, which is then used to inform the control policy. In hardware experiments, it has been demonstrated that robots with both senses could achieve better performance than those with vision alone. This is particularly conspicuous in the safe manipulation of fragile and delicate objects and in long-term, precise manipulation tasks.
Vision can tell a robot when it is currently holding a strawberry. Touch tells it the difference between a slipping, deformable strawberry and one that is one squeeze away from becoming jam — subtleties that vision alone cannot perceive.
Intelligent fingertips could one day become as paramount as the processors that power household robots, surgical systems, and responsive prosthetic hands. As machines move from simply perceiving the world to physically interacting with it, touch becomes increasingly indispensable. Computers have long relied on vision to make sense of their surroundings; touch could help them to learn to exist within them. For AI to become truly physically intelligent, it will have to move past the passive process of learning by seeing and begin to sense the resistance, the fragility, and the effects of its own actions. The future of embodied intelligence may well be determined not by what machines see, but what they feel on their fingertips.