HX5-D20: ACT with Tactile Sensors

1. When the Hand Hides the Object

Baseline ACT: placement fails, followed by another pick attempt.

As the fingers close around the jelly package, the hand can hide it from the head camera. The image and joint positions alone may not reveal whether the robot is holding the package. In the Baseline ACT clip above, the robot fails to complete placement and attempts another pick.

This ambiguity motivated us to add HX5-D20 fingertip pressure readings to ACT alongside camera images and robot state. The resulting policy, Tactile ACT, receives contact information even when the object is hidden.

What Tactile Information Adds

Task stage	Ambiguity in the camera view	Additional evidence from touch
Closing the hand	The fingers hide the package as they close.	Fingertip pressure indicates where contact is occurring.
Lifting and carrying	The held package may remain hidden behind the hand.	The pressure distribution provides evidence of continued contact or changes in contact.
Placing and releasing	Finger pose alone does not show whether contact with the package remains.	Pressure changes during opening provide evidence that contact is changing or ending.

The policy learns to interpret these pressure readings together with vision and joint state when choosing the next action.

2. System and Tactile Dataset

The system combines an AI Worker, two 7-DoF arms, two HX5-D20 hands, and a ZED head camera. Demonstrations are collected through Meta Quest 3 teleoperation and converted into trajectories containing camera images, robot state, tactile pressure, and the operator's actions.

The demonstrations use the right arm and hand. Jelly pickup focuses on grasping an occluded object; cup stacking adds placement and release.

Each HX5-D20 hand has five fingertip sensors, and each sensor produces a 3×3 pressure array. This gives 45 pressure-sensing elements, or taxels, per hand. Before training and inference, we subtract each taxel's unloaded baseline so that the input reflects contact pressure.

Dataset layout and pressure calibration

The dataset retains both arms and hands in its 54-dimensional state and action vectors, with 90 taxels across the two hands:

Feature	Shape	Description
Head RGB	[3, 720, 1280]	ZED head-camera image
Robot state	[54]	14 arm joints and 40 hand joints
Left tactile	[5, 3, 3]	Zero-filled for these right-hand tasks
Right tactile	[5, 3, 3]	Baseline-corrected fingertip pressure
Action	[54]	Demonstrated targets for both arms and hands

The collection and preprocessing path is:

HX5-D20 pressure → ROS 2 → synchronized MCAP
                  → baseline correction → LeRobot dataset

Pressure sensors can report a nonzero value even when unloaded, and the offset may differ across taxels. During dataset preparation, we estimate a per-taxel baseline from the first 20 no-contact samples of an episode. By default, the same procedure is repeated when the policy is loaded on the robot:

baseline[i] = round(median(first 20 samples[i]))
corrected[i] = max(raw[i] - baseline[i], 0)

The left tactile tensor is filled with zeros for these right-hand tasks. Training and inference use the same feature layout and baseline correction.

3. Tactile ACT Architecture

Tactile ACT adds a tactile token encoder to ACT while retaining the ResNet-18 vision backbone, Transformer decoder, and action head.

Tactile observations enter ACT as encoder tokens; decoding and action prediction follow the standard ACT path.

3.1 Token-Based Tactile Sensor Encoding

Each hand has its own small neural network (MLP), shared across that hand's 45 taxels. It converts each pressure value into a token. A learned positional embedding identifies the finger and grid location, so the policy can distinguish where contact occurs. The two hand tensors produce 90 tactile tokens in total.

Each hand uses its own tactile MLP, shared across the 45 taxels within that hand.

3.2 Multimodal Fusion and Action Prediction

The Transformer encoder combines tactile tokens with image features, robot state, and a latent token. The decoder uses this combined representation to predict a sequence of joint targets—an action chunk. Tactile input therefore influences the same action prediction process used by ACT.

Encoder and decoder configuration

For each observation.tactile.* feature, the shared per-taxel MLP is:

scalar taxel → Linear(1, 128) → ReLU
             → Linear(128, 512) → LayerNorm(512)

A four-layer Transformer encoder fuses the latent token, projected robot state, tactile tokens, and ResNet-18 image features. The one-layer decoder uses 16 zero-initialized slots with learned query-position embeddings. A shared linear action head produces a [B, 16, 54] chunk, where B is the batch size.

4. Training and Real-Robot Deployment

4.1 Training with Imitation Learning

Tactile ACT learns from teleoperated demonstrations: given the current image, joint state, and tactile readings, it predicts the demonstrated action chunk. This teaches the policy how contact relates to the operator's next movements.

Training objective

Training follows ACT's conditional-VAE objective. The posterior encoder receives the current robot state and demonstrated action chunk, then produces a latent sample for the main Transformer. The loss combines a padding-masked L1 action loss with KL regularization:

L = L1(predicted chunk, demonstrated chunk)
    + 10 · KL(q(z | state, action chunk) || N(0, I))

4.2 Deployment Method

At startup, the LeRobot runtime loads the checkpoint, validates the tactile feature shapes, and calibrates the right-hand pressure baseline. It supplies zeros for the left tactile input and fixes the latent variable to zero.

In the configuration described here, the model predicts 16 actions and the runtime executes the first four. Each command contains the 27 right-arm and right-hand targets selected from the model's 54-dimensional output. It then obtains fresh observations and predicts again.

5. Real-Robot Demonstrations

5.1 Jelly Pickup with Tactile ACT

In the Tactile ACT example below, the robot grasps the jelly package, lifts it, and moves toward the placement area. Fingertip pressure is included at each policy update.

Tactile ACT: the robot grasps and lifts the jelly package toward the placement area.

The clips illustrate behavior with and without tactile input; they are qualitative examples, not a controlled measurement of success-rate improvement.

5.2 Cup Stacking

The robot grasps one cup, lifts it, places it on a second cup, and releases it. Vision provides the target arrangement, while touch provides contact information as the hand closes, carries the cup, and opens.

A real-robot cup-stacking run using Tactile ACT.

6. Tactile ACT's Limitation: Contact Changes Between Updates

Tactile ACT uses the pressure readings available when it predicts an action chunk. Contact may then change as the queued actions execute: the object may shift or the fingers may lose contact. The sensors continue measuring, but those new readings cannot change the queued commands until the next policy update.

If a runtime executes an entire 100-step chunk at 15 Hz before re-inference, the interval is:

100 steps / 15 steps per second ≈ 6.67 seconds

A contact change during those 6.67 seconds is not reflected in the remaining commands. The interval depends on how many actions are executed before re-inference, not just how many the model predicts. The configuration above predicts 16 steps but updates after four, shortening the interval without eliminating it.

This limitation motivated us to use ViTacFormer to model how contact would change over the action sequence, extending the information available from the initial tactile observation.

7. ViTacFormer: Modeling Contact over the Action Horizon

ViTacFormer combines vision and touch through cross-attention, predicts future tactile signals, and conditions action generation on those predictions. Its tactile forecast gives the action generator information about expected contact changes beyond the current observation.

ViTacFormer architecture. Figure 2 from Heng et al., ViTacFormer (cropped from the paper).

The architecture follows three steps:

1. Fuse vision and touch. Cross-attention relates the visible scene to fingertip contact.
2. Forecast tactile signals. The model predicts future touch from the current observations, robot state, and latent variable.
3. Generate actions using the forecast. Predicted tactile signals are combined with the current inputs to produce the action sequence.

The tactile forecast helps the model anticipate contact changes during grasping, lifting, and placement. Unexpected changes still require fresh measurements and a policy update; prediction complements measured feedback.

8. Conclusion

Tactile ACT adds contact information when the hand hides an object from the camera. ViTacFormer extends this approach by predicting how contact will evolve and using those predictions to generate actions. Together, these ideas address both what the robot is touching now and how that interaction may change as it moves.