Does spatial sensor-pose conditioning help contact dynamics?
Goal: a better action-conditioning method for visuo-tactile contact. We render each GelSight sensor's projected image location as a per-frame Gaussian heatmap and inject it (zero-init) into the view tokens — CameraCtrl/EA-WM style. Result across all representations (heatmap channel, mask channel, in-target overlay): in-target > side-channel (novel ablation), but none beats contact-supervision (contactw1 0.202). Below: exactly what each model adds, the added input, and the prediction results.
| knob | p01rand | contactw1 | senshm(σ1.6) | senshm(σ0.6) | senshm+depth |
|---|---|---|---|---|---|
| contact-region supervision | ✗ | ✓ w1.0 | ✓ | ✓ | ✓ |
| sensor-pose heatmap (spatial cond) | ✗ | ✗ | ✓ | ✓ | ✓ |
| heatmap σ | — | — | 1.6 | 0.6 | 0.6 |
| depth channel | — | — | ✗ | ✗ | ✓ |
| run | contact IoU (val)↑ | view LPIPS↓ | tactile-L LPIPS↓ |
|---|---|---|---|
| p01rand | 0.173 | — | — |
| contactw1 (best) | 0.202 | 0.046 | 0.094 |
| senshm σ1.6 | 0.147 | 0.069 | 0.095 |
| senshm heatmap (σ0.6/depth) | 0.146 / 0.157 | — | — |
| B · mask channel | 0.158 | — | — |
| A · overlay-in-target | 0.169 | — | — |
Key ablation (novel): in-target overlay (0.169) > side-channel mask (0.158) > heatmap (0.15) — representation ordering matches Genima/In-Video-Instructions. But all < contactw1 (0.202): spatial pose re-representation adds no held-out contact signal (camera-frame ≠ gel-frame).
Adding the heatmap degraded both visual and tactile, train and val — while latent val_loss stayed ~equal (0.0467). The signal is redundant (pose already in the global action tokens + visible in the views) and a harmful perturbation; and the camera-frame location doesn't map to gel-frame contact.
Each GelSight sensor's 3D pose is projected (calibrated) into every camera view and rendered as a Gaussian. Left: it tracks the sensors over time across 3 views. Right: the 16×16 heatmap the model receives.
Columns: view + sensor heatmap (INPUT) · GT contact · contactw1 pred · senshm pred (red = above τ=8).
Matched val windows. Play/scrub; view is the clock.
window 0
2026-05-10__episode_005.pt · t=606 — GT ‖ prediction. rows: middle view, tactile L/R.
window 1
2026-05-10__episode_006.pt · t=1116 — GT ‖ prediction. rows: middle view, tactile L/R.
window 2
2026-05-10__episode_006.pt · t=1226 — GT ‖ prediction. rows: middle view, tactile L/R.
window 3
2026-05-10__episode_006.pt · t=1238 — GT ‖ prediction. rows: middle view, tactile L/R.
window 4
2026-05-10__episode_006.pt · t=1420 — GT ‖ prediction. rows: middle view, tactile L/R.
window 5
2026-05-10__episode_006.pt · t=2460 — GT ‖ prediction. rows: middle view, tactile L/R.