Dataset & Benchmark · arXiv preprint · Under review at CoRL 2026
RCT: A Robot-Collected Touch–Vision–Language Dataset for Tactile Generalization
1TU Dresden & ScaDS.AI Dresden/Leipzig, Dresden, Germany · 2LASR Lab, TU Dresden, Dresden, Germany
TL;DRFrames from one robot press are near-duplicates. RCT preserves every press as a contact sequence, so tactile models can finally be evaluated on what deployment demands: unseen presses, positions, sensors — and unseen materials.
At a glance
The dataset
Full robot presses on 122 industrial reference materials
A robot arm carries a rotating adapter that holds three DIGIT vision-based tactile sensors, so each material is recorded with all three sensors under one protocol. For every material, the robot presses the surface at multiple discrete contact positions; each press is sampled from initial contact to deeper indentation in fixed 0.10 mm steps, with contact force recorded at every frame.
The central design choice: tactile data is stored as contact sequences — the ordered frames from one press, on one material, with one sensor, at one contact position. Successive frames differ by a single 0.10 mm step, which makes them near-duplicate observations of the same physical interaction. Treating the press as the structural unit is what makes controlled held-out evaluation possible.
Three aligned modalities
Touch
29,279 DIGIT frames in 1,832 contact sequences, each with indentation depth and per-frame contact force |fz|.
Vision
One 2048×1536 photograph per material — material identity, deliberately separated from contact state. Retrieval uses a material-level multi-positive criterion.
Language
Material-level descriptors from a fixed 54-term tactile vocabulary: two independent human annotators who physically touched every material (union is the default target), plus a VLM source (Gemma-3-27B) for scale.
Human–human descriptor agreement is moderate (Jaccard 0.294); VLM–human agreement is far lower (0.110 / 0.108) — visual descriptors capture only part of what people report from touch.
Material categories
122 industrial reference materials from the Musterkiste sample set (Modulor GmbH). Contact positions 1–4 are densely sampled across all materials; later positions are sparser and reported as metadata.
Why it matters
The independent unit is the press, not the frame
Within one press, the sensor advances 0.10 mm at a time — so a frame-random split routinely places near-duplicate observations of the same physical interaction in both training and test. A model can then retrieve a near-duplicate contact instead of learning a representation that transfers. Try it below.
Interactive · one real robot press · material_0701420
rest frame · no contact · sensor 1
Drag the slider — or click any frame below. It starts at the no-contact rest frame, then steps through the 16 real DIGIT contact frames of one press on material_0701420; once in contact, adjacent frames barely change. That is the point: frames within one contact sequence are near-duplicates, and any split that separates them frame-by-frame leaks the interaction itself.
Now split this press for evaluation
Decomposing frame-random performance
(full overlap)
overlap
no material overlap
Tactile→text Recall@1 on RCT, one representative held-out split (seed 42). The first drop is measured with the encoder held fixed — only the test pool changes — and an encoder trained directly under the held-out-sequence protocol lands at 62.44%, confirming the gap is the protocol, not a weaker representation. Across three held-out draws, held-out-material R@1 averages 25.1 ± 6.1%.
The released TVL/HCT split shows the same structure
Auditing the public split with its own files: every HCT test contact sequence also appears in training, and the overlap is exploitable without any learning at all.
also present in training
same-sequence training frame
training-free raw-pixel nearest neighbor
This is an evaluation-protocol result, not a criticism of touch–vision–language learning — RCT uses the same training recipe. It means frame-random retrieval scores should not be read as tactile generalization unless contact-sequence and material overlap are controlled.
A held-out sensor is as hard as a held-out material
| Held-out axis (materials seen at train) | Tac→text R@1 | Tac→vis R@1 |
|---|---|---|
| Contact position (p5; seen at p1–p4) | 65.46% | 74.16% |
| DIGIT sensor instance (d3; seen at d1, d2) | 21.51% | 13.00% |
| Held-out material (K=20; not seen) | 20.35% | 18.87% |
Holding out a contact position is easy — the material is still seen elsewhere. Holding out one DIGIT instance is as hard as holding out the material entirely: sensor-instance robustness is its own bottleneck, and honest evaluation needs sensor-disjoint splits.
Uniformly sampling the press beats using every frame
| Training density | Frames | Tac→text R@1 | Tac→vis R@1 |
|---|---|---|---|
| Full press (~16 frames / sequence) | 22,576 | 62.44% | 70.95% |
| Deep5 (five deepest frames) | 7,055 | 59.52% | 65.96% |
| Uniform5 (five frames, shallow→deep) | 7,055 | 68.18% | 78.29% |
Covering the press from shallow to deep contact (+5.7 pp over full density, at one third of the data) is more useful than densely sampling near-duplicates — consistent with near-duplicates acting as false negatives in contrastive objectives.
RCT training improves material structure on unseen materials
| Encoder | Separability margin | Category probe (7-class) | Hard/soft probe |
|---|---|---|---|
| Released TVL encoder | 0.048 | 34.2% | 52.2% |
| Random init | 0.004 | 37.8% | 45.7% |
| RCT encoder | 0.297 | 49.8% | 52.2% |
Category-level material information transfers to unseen materials; binary hardness stays at or below the majority baseline (59%) for every representation — including force features — marking it as an open problem.
Benchmark protocols
Hold out physically meaningful units
RCT ships split files and a split-generation toolkit for six evaluation settings. Held-out material (K = 20) is the primary protocol, because it matches the deployment question: can a robot recognize material properties it has never touched?
| Setting | Held out at test time | Purpose |
|---|---|---|
| Held-out material | All sequences of K materials (K = 2, 5, 20) | Unseen materials |
| Held-out category | All materials from one top-level category | Unseen material family |
| Held-out sequence | Entire contact sequences; materials may be seen | Sequence overlap |
| Held-out position | One contact position across materials | Novel location |
| Held-out sensor | One DIGIT sensor instance | Sensor transfer |
| Frame-random control | Random frames; material and sequence overlap | Split inflation |
Reporting recommendations
Tactile benchmarks should expose contact-sequence identifiers and report held-out results alongside any frame-random score, stating the level at which test samples are independent of training. For vision-based tactile sensors that level is at least the contact sequence; for manipulation it is usually the material. Because a single K = 20 draw is noisy (±6.1 pp across seeds with diverse selection; 29.8 ± 2.9% with category-balanced selection), report the mean and standard deviation over multiple held-out draws rather than a single split.
Download
Get the dataset
The release contains everything needed to reproduce the paper's protocols and to build new ones:
- 29,279 tactile frames with contact-sequence identifiers
- Per-frame force signals and indentation depths
- One 2048×1536 photograph per material
- Material-level descriptor annotations (two human annotators, human-union, and VLM source) over a 54-term vocabulary
- Material / category / sensor / position metadata
- Split files for all held-out settings + split-generation and evaluation scripts
License: CC BY 4.0. One material had an ID-format mismatch in early annotator labels; the released metadata already includes the correction.
Citation
BibTeX
arXiv preprint entry — fill in the ID and author list after the announcement; switch to the CoRL/PMLR entry on acceptance.
@misc{he2026rct,
title = {{RCT}: A Robot-Collected Touch--Vision--Language
Dataset for Tactile Generalization},
author = {Jingbo He and Michael F{\"a}rber and Roberto Calandra},
year = {2026},
eprint = {2606.31694},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
doi = {10.48550/arXiv.2606.31694},
url = {https://arxiv.org/abs/2606.31694},
}