OmniTact: a multi-directional high-resolution contact sensor


Human thumb subsequent to our OmniTact sensor, and a US penny for scale.
By Akhil Padmanabha and Frederik Ebert

Contact has been proven to be vital for dexterous manipulation in robotics. Just lately, the GelSight sensor has caught important curiosity for learning-based robotics resulting from its low price and wealthy sign. For instance, GelSight sensors have been used for studying inserting USB cables (Li et al, 2014), rolling a die (Tian et al. 2019) or greedy objects (Calandra et al. 2017).

The explanation why learning-based strategies work nicely with GelSight sensors is that they output high-resolution tactile photographs from which quite a lot of options corresponding to object geometry, floor texture, regular and shear forces might be estimated that usually show important to robotic management. The tactile photographs might be fed into commonplace CNN-based laptop imaginative and prescient pipelines permitting using quite a lot of totally different learning-based methods: In Calandra et al. 2017 a grasp-success classifier is skilled on GelSight information collected in self-supervised method, in Tian et al. 2019 Visible Foresight, a video-prediction-based management algorithm is used to make a robotic roll a die purely primarily based on tactile photographs, and in Lambeta et al. 2020 a model-based RL algorithm is utilized to in-hand manipulation utilizing GelSight photographs.

Sadly making use of GelSight sensors in sensible real-world situations continues to be difficult resulting from its massive dimension and the truth that it’s only delicate on one aspect. Right here we introduce a brand new, extra compact tactile sensor design primarily based on GelSight that enables for omnidirectional sensing, i.e. making the sensor delicate on all sides like a human finger, and present how this opens up new potentialities for sensorimotor studying. We display this by educating a robotic to select up electrical plugs and insert them purely primarily based on tactile suggestions.

An ordinary GelSight sensor, proven within the determine under on the left, makes use of an off-the-shelf webcam to seize high-resolution photographs of deformations on the silicone gel pores and skin. The within floor of the gel pores and skin is illuminated with coloured LEDs, offering ample lighting for the tactile picture.


Comparability of GelSight-style sensor (left aspect) to our OmniTact sensor (proper aspect).

Present GelSight designs are both flat, have small delicate fields or solely present low-resolution indicators. For instance, prior variations of the GelSight sensor, present excessive decision (400×400 pixel) photographs however are massive and flat, offering sensitivity on just one aspect, whereas the business OptoForce sensor (lately discontinued by OnRobot) is curved, however solely offers power readings as a single third-dimensional power vector.

Our OmniTact sensor design goals to handle these limitations. It offers each multi-directional and high-resolution sensing on its curved floor in a compact type issue. Much like GelSight, OmniTact makes use of cameras embedded right into a silicone gel pores and skin to seize deformation of the pores and skin, offering a wealthy sign from which a variety of options corresponding to shear and regular forces, object pose, geometry and materials properties might be inferred. OmniTact makes use of a number of cameras giving it each high-resolution and multi-directional capabilities. The sensor itself can be utilized as a “finger” and might be built-in right into a gripper or robotic hand. It’s extra compact than earlier GelSight sensors, which is completed by using micro-cameras sometimes utilized in endoscopes, and by casting the silicone gel instantly onto the cameras. Tactile photographs from OmniTact are proven within the figures under.


Tactile readings from OmniTact with numerous objects. From left to proper: M3 Screw Head, M3 Screw Threads, Mixture Lock with numbers four three 9, Printed Circuit Board (PCB), Wi-fi Mouse USB. All photographs are taken from the upward-facing digital camera.


Tactile readings from the OmniTact being rolled over a gear rack. The multi-directional capabilities of OmniTact preserve the gear rack in view because the sensor is rotated.

One among our main targets all through the design course of was to make OmniTact as compact as attainable. To perform this objective, we used micro-cameras with massive viewing angles and a small focus distance. Particularly we picked cameras which might be generally utilized in medical endoscopes measuring simply (1.35 x 1.35 x 5 mm) in dimension with a spotlight distance of 5 mm. These cameras had been organized in a 3D printed digital camera mount as proven within the determine under which allowed us to attenuate blind spots on the floor of the sensor and scale back the diameter (D) of the sensor to 30 mm.


This picture exhibits the fields of view and association of the 5 micro-cameras contained in the sensor. Utilizing this association, a lot of the fingertip might be made delicate successfully. Within the vertical aircraft, proven in A, we receive $alpha=270$ levels of sensitivity. Within the horizontal aircraft, proven in B, we receive 360 levels of sensitivity, apart from small blind spots between the fields of view.

We present that OmniTact’s multi-directional tactile sensing capabilities might be leveraged to unravel a difficult robotic management drawback: Inserting an electrical connector blindly right into a wall outlet purely primarily based on data from the multi-directional contact sensor (proven within the determine under). This process is difficult because it requires localizing the electrical connector relative to the gripper and localizing the gripper relative to the wall outlet.

To be taught the insertion process, we used a easy imitation studying algorithm that estimates the end-effector displacement required for inserting the plug into the outlet primarily based on the tactile photographs from the OmniTact sensor. Our mannequin was skilled with simply 100 demonstrations of insertion by controlling the robotic utilizing keyboard management. Profitable insertions obtained by working the skilled coverage are proven within the gifs under.



As proven within the desk under, utilizing the multi-directional capabilities (each the highest and aspect digital camera) of our sensor allowed for the best success charge (80%) compared to utilizing only one digital camera from the sensor, indicating that multi-directional contact sensing is certainly essential for fixing this process. We moreover in contrast efficiency with one other multi-directional tactile sensor, the OptoForce sensor, which solely had successful charge of 17%.

We consider that compact, excessive decision and multi-directional contact sensing has the potential to remodel the capabilities of present robotic manipulation techniques. We suspect that multi-directional tactile sensing may very well be a necessary component in general-purpose robotic manipulation along with purposes corresponding to robotic teleoperation in surgical procedure, in addition to in sea and area missions. Sooner or later, we plan to make OmniTact cheaper and extra compact, permitting it for use in a wider vary of duties. Our workforce moreover plans to conduct extra robotic manipulation analysis that can inform future generations of tactile sensors.

This weblog publish is predicated on the next paper which can be introduced on the Worldwide Convention on Robotics and Automation 2020:

We wish to thank Professor Sergey Levine, Professor Chelsea Finn, and Stephen Tian for his or her precious suggestions when making ready this weblog publish.

This text was initially printed on the BAIR weblog, and seems right here with the authors’ permission.

BAIR Weblog

visitor creator

BAIR Weblog is the official weblog of the Berkeley Synthetic Intelligence Analysis (BAIR) Lab.



Supply hyperlink

Leave a Comment