Method and data
The collection combines Kine2Go policy executions based on animal and robot motions with motion-capture recordings of teleoperated Go2 motions. Ten independently trained temporal encoders learn from reconstruction, instance contrast, and weak coarse-label guidance. Their cosine geometries are averaged before HDBSCAN identifies 16 candidate styles without a prescribed count.
A reinforcement-learning policy combines a selected reference code with forward, lateral, and yaw-rate commands. The supported command range depends on the style. All decomposition methods use the same 53 recordings and 10,661 two-second windows, with no decomposition train/test split.
Recorded examples
The same rule is used for every method: select up to three distinct two-second reference windows, prioritizing different recordings before additional non-overlapping windows from the same recording. Selection does not use controller performance. VQ-VAE code 8 has only one available window, so its remaining slots are empty.
These clips show recorded motions, not policy executions. Group numbers do not correspond between methods. All videos use the same camera and 720p / 30 fps rendering. Navigation thumbnails use the frame at 0.5 seconds.
Free-form reference comparison
Choose any two methods, groups, and examples. This browser does not imply that the selected groups correspond.
Left reference
Right reference
Playback starts together without aligning the motion phase. Examples within one VQ code supply the same code to its controller.
Coherence and decomposition ablations
Silhouette measures within-group cohesion relative to between-group separation. Cross-recording AUC measures whether same-group windows from different recordings are mechanically closer than different-group windows; chance is 0.5. Both use the same label-independent, phase-insensitive 32-dimensional kinematic PCA representation, not each method's learned space. These are kinematic diagnostics, not semantic ground truth.
CODES uses weak coarse-label guidance and ten encoders. “Without coarse labels” is the paper's “No coarse terms” ablation: it keeps the ensemble but removes label guidance. VQ-VAE is a label-free, protocol-matched offline baseline, not a reproduction of a complete published VQ-based controller. The comparison does not isolate clustering versus vector quantization.
Full-coverage CODES propagates assignments without retraining. Recording-balanced VQ-VAE is a decomposition control; Kinematic PCA and TS2Vec have decomposition results only.
Effective K discounts sparsely occupied groups through the exponential of occupancy entropy. Locomotion K counts groups predominantly associated with that diagnostic coarse label. Coverage is the fraction not rejected as density noise. Core switching excludes transitions through rejected regions and can be low when a representation collapses.
Shading identifies CODES, not a universal winner. Seven groups remain with uniform attraction; removing reconstruction leaves three. CODES' lower coverage reflects more selective density support.
Velocity explorer protocol
All 16 CODES styles are tested at 135 forward/lateral velocity pairs, with zero yaw-rate command and three trials per cell. Command fidelity is the fraction completing the trial with planar RMSE ≤ 0.30 m/s and yaw-rate RMSE ≤ 0.40 rad/s. The yellow outline marks recorded velocity support, not a training holdout.
Style fidelity is reference-consensus cosine, averaged over available scored windows and trials. Larger values indicate more similar encoded motion; this is not a perceptual rating or a style pass/fail threshold. Colors match the paper's asinh scale (width 0.12), spanning −0.1 to 1.0; lower values use the lower-end color, while the displayed numbers are unchanged. Gray cells have no scored windows. The two grid modes assess different properties.
The default reference is the fixed controller-conditioning window used throughout that style's sweep. The optional velocity-matched display instead selects a recorded two-second window from the same style whose mean forward and lateral velocity is closest to the cell, using the original grid-scaled distance and the zero-yaw slice. This display choice does not change the policy rollout or the score. Style 5 has no recorded example in that slice, so this option falls back to the actual controller reference.
The video always shows repeat 0 (labeled trial 1 on the website), not the best-performing trial. Policy clips replay seconds 5–15 of saved Isaac Sim trials; MuJoCo supplies the kinematic rendering, not a new dynamics simulation. Early terminations are labeled and the last alive state is held. The two-second reference repeats without phase alignment. Reference-consensus cosine is always computed against the controller-conditioning reference, regardless of the video displayed.
The explorer uses the 20,000-update CODES controller, before deployment fine-tuning. Grid statistics summarize all three trials; clip metrics describe only the displayed trial. Reference selection and hover/click controls do not run a live policy.
Command tracking over the full grid
All three policies use 20,000 training iterations, the same controller recipe, and the same command grid in Isaac Sim. Each trial lasts 15 seconds; the first 5 seconds are settling context. Command success requires completion, planar velocity RMSE at most 0.30 m/s, and yaw-rate RMSE at most 0.40 rad/s. The table reports median planar RMSE.
Each method uses its own reference collection, so these are not identical requested styles. Better command tracking does not establish better style fidelity. Cross-simulator and real-world demonstrations are provided for CODES only.
Controller comparisons with identical references
Watch the controller comparisons →
Each pair's 120 × 49 reference windows are bitwise identical, encoded through the respective method. All controllers are the iteration-19,999 checkpoints (20,000 updates). Forward commands are 0, 0.25, 0.5, 0.75, 1.0, and 1.5 m/s, with zero lateral and yaw-rate commands, five reset seeds, and 15-second holds.
Physical distance (called motion distance on the controller-comparison page) compares equally weighted posture, foot geometry, kinematic contact structure, and joint-motion features, excluding base linear and angular velocity. Features use source-only scaling. Distance is reported on completed trials; completion is shown alongside it. Recording retrieval asks whether the nearest source descriptor belongs to the requested recording, with failed trials counted as unsuccessful. This is not reference consensus cosine or a perceptual score. Random recording-level retrieval is 4.3% for CODES–without coarse labels and 11.2% for CODES–VQ-VAE.
The displayed examples are the first, middle, and last entries of each fixed pair list, at 0.5 m/s and reset seed 0; they are not selected by performance. The videos replay saved Isaac Sim states. After an early termination, the last alive frame is held and the failure is labeled.
The two comparisons use different overlapping reference sets, so their CODES rows are not the same population. Scores are recording-balanced means rather than whole-dictionary medians. Only exactly shared retained windows are tested. There is one training seed per policy; confidence intervals describe source-recording variability, not variation across training runs. CODES does not outperform VQ-VAE in every metric, including residual lateral and yaw-rate error.
Real-world deployment
The yellow cue in the opening video marks each logged style selection: C04 → C00 → C14 → C11. The forward command is constant at 0.5 m/s; the operator provides heading corrections. The cue marks selection, not an instantaneous physical response. The controller and state estimation run onboard.
Deployment uses an additional 2,000 simulation-training iterations with the joint-target safeguard, with no policy updates during the run.