Sim2Real

The AI Sapiens Sim2Real package provides the control layer for deploying trained policies on the real robot. It connects robot feedback, operator input, API requests, policy inference, and joint command publishing into one control loop.

The package is centered around ai_sapiens_sim2real_node. On each control tick, the node reads the robot state, decides the active mode, runs the corresponding behavior, and publishes joint commands to the hardware controller.

Robot feedbackIMU, joint states

Operator / API inputRC, heartbeat, mode request, cmd_vel

ai_sapiens_sim2real_node

1. Check input validity and failsafe conditions
2. Select authority: MANUAL or API
3. Select mode: Damping, ReadyPose, Velocity, or Mimic
4. Run the selected behavior

Hardware command/joint_group_position_controller/commands

info

• Mode: what the robot does.
• Authority: who can request the next mode and provide velocity commands, either Manual or API.
• Failsafe: safety logic that can override both and force Damping.

Launching Sim2Real

Use the package launch file with the target robot configuration. For K1, the default launch argument resolves to config/k1_config.yaml.

ros2 launch ai_sapiens_sim2real ai_sapiens_sim2real.launch.py robot:=k1

The main launch arguments are:

Argument	Default	Description
robot	k1	Selects config/{robot}_config.yaml.
imu_topic	/imu_sensor_broadcaster/imu	IMU feedback used by policy inference.
joint_states_topic	/joint_states	Robot joint state feedback.
control_rate	1000.0	Realtime control loop rate in Hz.
joint_command_topic	/joint_group_position_controller/commands	Final joint command output topic.
command_publisher_enabled	true	Enables or disables publishing commands to the hardware controller.

Before policy inference starts, the node waits for IMU feedback, joint states, and the first RC input sample. If any of them do not arrive within wait_for_ready_timeout, the node exits instead of starting without robot feedback or operator input.

Default Manual Operation

Manual operation through the RC transmitter is the default control path. Operator input is converted into mode requests and velocity commands. Even during API authority, RC input must keep arriving so the operator can take control back or force Damping immediately.

The root config maps RC inputs to mode conditions:

Condition	Meaning
DampingRequested	Force the robot into damping before normal mode selection.
ReadyPoseRequested	Move to the ready posture defined in k1_config.yaml.
VelocityRequested	Run the velocity policy mode.
MimicRequested	Select and run a mimic motion mode.

The default state machine starts in Damping. At startup, the first accepted operator input must match either DampingRequested or ReadyPoseRequested. This prevents the robot from immediately entering a motion mode because of a stale switch position.

Failsafe Rules

Failsafe checks run first on every control tick. The first active rule below wins before normal mode switching, RC selection, API service requests, mimic completion, or startup gating.

1. RC input unavailableNo accepted RC command, or the latest command is older than teleop_input.timeout (0.2 seconds for K1).Response: zero velocity, request Damping, revoke API authority, and discard pending API service requests. This also applies while API authority is active.
2. RC damping switchThe operator holds the input mapped to DampingRequested.Response: keep the robot pinned in Damping. API re-entry requires a fresh API switch toggle after leaving damping.
3. Robot feedback watchdogIMU or joint state feedback stalls after startup. Defaults: imu_timeout=0.5, joint_states_timeout=0.5.Response: latch a damping request until the node is restarted and discard pending API service requests.
4. Policy action faultA processed policy action is non-finite or exceeds 2π rad.Response: force Damping and clear the action fault flag.
5. Unsafe body orientationBody orientation exceeds the safety angle from k1_config.yaml while a policy mode is active. Default: 2.0 rad. Mimic modes ignore the check for the first 0.5 seconds, then use 3.0 rad.Response: force Damping to stop the active policy behavior.

In short, a failsafe can force Damping, zero velocity, revoke API authority, discard pending API service requests, or latch damping until restart when robot feedback stalls.

note

API heartbeat loss is not a Damping failsafe. If the heartbeat is lost during API authority while a mimic mode is active, the node returns to manual authority, zeros the velocity command for that control tick, and switches to Velocity.

Policy Location and Mode Switching Configuration

The root configuration file defines the robot joint order, policy asset search paths, teleoperation mapping, state machine, and behavior assets. For K1, the entry point is:

ai_sapiens_sim2real/config/k1_config.yaml

The file is organized in this top-level order:

robot_joint_order:     # robot joint order used by commands and policies
policy_asset_roots:    # directories searched for policy folders
mimic_defaults:        # default playback settings for mimic modes
authority:             # API handoff settings
teleop_input:          # RC input plugin and watchdog timeout
teleop_conditions:     # maps RC input codes to named mode requests
selectors:             # maps selector codes to concrete modes
state_machine:         # mode graph, transitions, and state roles
state_behaviors:       # behavior implementation for each mode

Policy Asset Layout

The K1 config uses policy_asset_roots as the base folder for Sim2Real assets:

policy_asset_roots:
  - ../assets/k1

The node looks for each behavior's asset folder under policy_asset_roots.

policy_asset_roots../assets/k1

+

assetmimic/dance1

=

loaded directory../assets/k1/mimic/dance1

Required inside the loaded directory

exported/

policy.onnx

params/

sim2real.yaml<motion_reference>.csvmimic only

Mimic behaviors also reference a motion CSV under the same params directory. For example, the current K1 mimic_dance1 behavior is declared as:

mimic_dance1:
  kind: mimic
  asset: mimic/dance1
  motion: dance1.csv

When a mimic mode starts, it loads the ONNX policy, the sim2real.yaml observation/action mapping, and the selected motion reference.

Built-in K1 Modes

The default K1 configuration includes these built-in modes:

Mode	Behavior
Damping	Sends the damping command values from k1_config.yaml. Used for startup and failsafe.
ReadyPose	Moves the robot to the ready posture defined in k1_config.yaml.
ZeroPose	Moves the robot to the zero posture defined in k1_config.yaml.
Velocity	Runs the velocity policy behavior.
MimicDance1	Runs the first Kimodo-generated dance mimic policy and motion reference.
MimicDance2	Runs the second Kimodo-generated dance mimic policy and motion reference.

Mimic modes share the Mimic parent state. That parent state gives all mimic motions the same transition and failsafe rules, while each child mode still points to its own policy asset and motion CSV.

The K1 mimic selector currently maps RC selector codes to modes as follows:

Selector code	Mode
200	MimicDance1
201	MimicDance2

How to Deploy Your Own Cyclo Lab-Trained Policy

Treat a Cyclo Lab export as a Sim2Real asset, then bind that asset to a robot mode.

info

A deployable Cyclo Lab policy is a full Sim2Real asset, not only policy.onnx. The asset also needs the matching sim2real.yaml and, for mimic policies, the motion reference CSV.

1. Export for deployment. The required files are exported/policy.onnx and params/sim2real.yaml. For mimic policies, also include the motion reference CSV required by the mimic behavior.

2. Place the policy under assets/k1.

Create one policy folder under ai_sapiens_sim2real/assets/k1/<category>/.

ai_sapiens_sim2real/assets/k1/<category>/<your_policy_name>/

• exported/
  • policy.onnx
• params/
  • sim2real.yaml
  • <motion_reference>.csvmimic only

Use locomotion/velocity for velocity policies and mimic for mimic policies. The category is only part of the asset path; the behavior type is selected in the config.

3. Update k1_config.yaml.

Use the locomotion example for a velocity policy, or the mimic example for a mimic policy.

Locomotion policy

Add one state_behaviors entry that points to your asset folder, then change only the existing Velocity state's run value.

state_behaviors:  velocity_policy:                            # already exists    kind: policy    asset: locomotion/velocity/walk_default  my_velocity_policy:                         # 1. Add this below the existing behavior.    kind: policy    asset: locomotion/velocity/<your_policy_name>state_machine:  states:    Velocity:                                 # already exists      run: my_velocity_policy                 # 2. Change only this value.      transitions:                            # keep the existing transitions        ...

Mimic policy

Add one state_behaviors entry, add one state_machine.states entry for the new mimic mode, then map an unused RC selector code to that mode.

state_behaviors:  mimic_dance1:                               # already exists    kind: mimic    asset: mimic/dance1    motion: dance1.csv  my_mimic_motion:                            # 1. Add a new behavior.    kind: mimic    asset: mimic/my_motion    motion: my_motion_reference.csv           # From the asset's params/ directory.state_machine:  states:    MimicDance1:                              # already exists      parent: Mimic      run: mimic_dance1    MyMimicMotion:                            # 2. Add a new mode.      parent: Mimic      run: my_mimic_motionselectors:  mimic_selector:    table:      200: MimicDance1                        # already exists      202: MyMimicMotion                      # 3. Add an RC selector code.

Policy Compatibility Checks

The node uses the observation and action layout from the config files as written. Check these items before running the policy:

• Joint names listed in policy_joints inside sim2real.yaml.
• Observation terms and their order in sim2real.yaml.
• Action scaling, offset, and clipping in sim2real.yaml.
• Motion reference file, FPS, and playback window for mimic policies.

info

• policy_joints: joint order used by the policy input and output.
• robot_joint_order: joint order expected by the lower-level controller.

The two lists can use different orders. Just make sure every joint name in policy_joints exists with the same name in robot_joint_order.

If the joint names, observation list, action settings, or mimic CSV settings do not match the exported policy, the node may still launch, but the robot behavior will not represent the trained policy correctly.

Recommended First-Run Flow

After you place the policy folder under assets/k1, update k1_config.yaml, and verify the compatibility checks above, use this sequence for the first dry-run validation.

1. Start the robot bringup and confirm that /joint_states, the IMU topic, the joint command controller, and RC input are active.
2. Start ai_sapiens_sim2real_node with command_publisher_enabled:=false for the first dry run.
3. Start from Damping and confirm that the launch terminal logs do not report a startup timeout or missing IMU, joint state, or RC input.
4. While still in dry run, select the mode that should run your policy and confirm that the launch terminal logs show the expected mode name.
5. If the logs show a different mode name, stop here and recheck the state_machine, state_behaviors, and selectors entries in config/k1_config.yaml.

After the launch logs show the expected mode name in dry run, follow the Quick Start Operation Guide to run the robot, then select that mode from the RC transmitter. For API-driven tests, first run the policy once through RC control, then use the optional API control flow below.

Optional API Control

Manual operation is the default path. API authority is optional. It lets an external API client request modes and send velocity commands.

Sim2Real separates the robot's behavior mode from the authority that is allowed to request it.

• Mode is the active robot behavior, such as Damping, ReadyPose, Velocity, or a mimic motion mode.
• Authority is who can request the next mode and provide velocity commands. Manual authority follows the operator's teleoperation input. API authority accepts ROS2 service requests and velocity commands from an external API client.

info

Even in API authority, RC input must keep arriving. The operator can use the RC transmitter to take control back or force Damping, and missing RC input triggers failsafe.

The authority states are:

Authority	Meaning
MANUAL	The RC transmitter owns mode switching. This is used at startup and after API control ends.
API_WARMUP	A short guarded handoff period after the operator requests API control. Service requests are still rejected.
API	ROS2 service requests can switch to modes that are currently available through the state machine.

API Handoff

API authority must be requested by the operator from the RC transmitter. The API switch must be toggled while the node is in manual authority. If the switch is already enabled during boot, the node does not enter API control until the operator lowers and raises it again.

An API handoff is accepted only when:

• The API heartbeat is arriving within api_mode_heartbeat_timeout.
• The robot is currently in one of these modes:
  • Damping / posture mode, such as ReadyPose / Velocity
• The RC velocity command is zero.
• The /cmd_vel velocity command is zero.

After acceptance, the node enters API_WARMUP for the configured warmup duration. For the K1 config, the warmup duration is 3.0 seconds, defined in the authority.api_entry section of config/k1_config.yaml.

API_WARMUP is a short confirmation window before the external API client receives API authority. During this window, the heartbeat must keep arriving and the operator must keep the API switch enabled.

During warmup, the robot keeps the current mode and rejects service requests. When warmup completes, authority becomes API without changing the active mode.

If API control is released or the heartbeat is lost, the node returns to manual authority using the current operator switch position. If the heartbeat is lost during API authority while a mimic mode is active, the node returns to manual authority, zeros the velocity command for that control tick, and switches to Velocity.

Heartbeat

The node subscribes to ai_sapiens_sim2real/msg/ApiModeHeartbeat on:

/ai_sapiens/api_mode_heartbeat

An external API client must keep publishing this heartbeat while it wants to use API authority. The message contains a Header and a monotonically updated sequence. The node treats the heartbeat as invalid when updates stop for longer than api_mode_heartbeat_timeout. The default timeout is 0.2 seconds.

Velocity Command

When API authority is active, velocity commands are read from:

/cmd_vel

This replaces the RC transmitter as the velocity command source while API authority is active.

Request a Mode

Use /ai_sapiens/request_mode_by_name to request a mode by its name, such as Velocity or MyMimicMotion.

ros2 service call /ai_sapiens/request_mode_by_name \
  ai_sapiens_sim2real/srv/RequestModeByName \
  "{mode_name: Velocity}"

The response reports whether the request was accepted, the active mode at the time of the response, and a message explaining the result. Requests are rejected when authority is MANUAL or API_WARMUP, when another request is not allowed by the state machine, or when the requested mode is not a valid concrete mode.

List Modes

Use /ai_sapiens/list_modes to inspect the configured modes.

ros2 service call /ai_sapiens/list_modes ai_sapiens_sim2real/srv/ListModes "{}"

The response includes:

• modes: all concrete modes in the state machine.
• available_modes: modes that can currently be requested through the API service.

Monitor Node Status

The node publishes ai_sapiens_sim2real/msg/ModeStatus to:

/ai_sapiens/mode_status

API clients can read this topic to track the node state. It reports the active mode, authority, RC input status, API heartbeat status, whether API service requests are currently accepted, and the last transition reason.

When debug publishing is enabled with debug_publish_enabled:=true, the node also publishes policy observation and action data:

Topic	Purpose
/ai_sapiens/observations	Structured observation, action, and policy input snapshot.
/policy_input/raw_observation	Raw policy input vector.
/policy_output/processed_action	Processed policy action before command publishing.

Use these topics to debug policy inputs and outputs; do not send robot commands through them.