Sim2Real

The AI Sapiens Sim2Real package provides the control layer for deploying trained policies on the real robot. It connects robot feedback, operator input, API requests, policy inference, and joint command publishing into one control loop.

The package is centered around ai_sapiens_sim2real_node. On each control tick, the node reads the robot state, decides the active mode, runs the corresponding behavior, and publishes joint commands to the hardware controller.

Robot feedbackIMU, joint states

Operator / API inputRC, heartbeat, mode request, cmd_vel

ai_sapiens_sim2real_node

1. Check input validity and failsafe conditions
2. Select authority: MANUAL or API
3. Select mode: Damping, ReadyPose, Velocity, or Mimic
4. Run the selected behavior

Hardware command/joint_group_impedance_controller/commands

info

• Mode: what the robot does.
• Authority: who can request the next mode and provide velocity commands, either Manual or API.
• Failsafe: safety logic that can override both and force Damping.

Launching Sim2Real

Use the package launch file with the target robot configuration. For K1, the default launch argument resolves to config/k1_config.yaml.

ros2 launch ai_sapiens_sim2real ai_sapiens_sim2real.launch.py robot:=k1

The main launch arguments are:

Argument	Default	Description
robot	k1	Selects config/{robot}_config.yaml.
imu_topic	/imu_sensor_broadcaster/imu	IMU feedback used by policy inference.
joint_states_topic	/joint_states	Robot joint state feedback.
control_rate	1000.0	Realtime control loop rate in Hz.
joint_command_topic	/joint_group_impedance_controller/commands	Final joint command output topic.
command_publisher_enabled	true	Enables or disables publishing commands to the hardware controller.

Before the control loop starts, the node waits for IMU feedback, joint states, and the first RC input sample. If any of them do not arrive within wait_for_ready_timeout, the node exits instead of starting without robot feedback or operator input.

Default Manual Operation

Manual operation through the RC transmitter is the default control path. Operator input is converted into mode requests and velocity commands. Even during API authority, RC input must keep arriving so the operator can take control back or force Damping immediately.

The root config maps RC inputs to mode conditions:

Condition	Meaning
DampingRequested	Force the robot into damping before normal mode selection.
ReadyPoseRequested	Move to the ready posture defined in k1_config.yaml.
VelocityRequested	Run the velocity policy mode.
MimicRequested	Select and run a mimic motion mode.

The default state machine starts in Damping. At startup, the first accepted operator input must match either DampingRequested or ReadyPoseRequested. This prevents the robot from immediately entering a motion mode because of a stale switch position.

Failsafe Rules

Failsafe checks run first on every control tick. The first active rule below wins before normal mode switching, RC selection, API service requests, mimic completion, or startup gating.

1. RC input unavailableNo accepted RC command, or the latest command is older than teleop_input.timeout (0.2 seconds for K1).Response: zero velocity, force Damping, revoke API authority, and discard pending API service requests. This also applies while API authority is active.
2. RC damping switchThe operator holds the input mapped to DampingRequested.Response: keep the robot pinned in Damping. API re-entry after this failsafe requires a fresh API switch toggle.
3. Robot feedback watchdogIMU or joint state feedback stalls after startup. Defaults: imu_timeout=0.5, joint_states_timeout=0.5.Response: latch a damping request until the node is restarted and discard pending API service requests.
4. Policy action faultA processed policy action is non-finite or exceeds 2π rad.Response: force Damping and clear the action fault flag.
5. Unsafe body orientationBody orientation exceeds the hardcoded safety angle while a policy mode is active. Default: 2.0 rad. Mimic modes ignore the check for the first 0.5 seconds, then use 3.0 rad.Response: force Damping to stop the active policy behavior.

In short, a failsafe can force Damping, zero velocity, revoke API authority, discard pending API service requests, or latch damping until restart when robot feedback stalls.

note

API heartbeat loss is not a Damping failsafe. If the heartbeat is lost during API authority, the node returns to manual authority and zeros the velocity command for that control tick. This handoff never starts or continues a mimic mode; an active mimic mode falls back to Velocity.

Policy Location and Mode Switching Configuration

The root configuration file defines the robot joint order, policy asset search paths, teleoperation mapping, state machine, and behavior assets. For K1, the entry point is:

ai_sapiens_sim2real/config/k1_config.yaml

The file is organized in this top-level order:

robot_joint_order:     # joint order expected by the lower-level controller
policy_asset_roots:    # directories searched for policy folders
mimic_defaults:        # default playback settings for mimic modes
authority:             # API handoff settings
teleop_input:          # RC input plugin and watchdog timeout
teleop_conditions:     # maps RC input codes to named mode requests
selectors:             # maps selector codes to concrete modes
state_machine:         # mode graph, transitions, and state roles
state_behaviors:       # behavior implementation for each mode

Policy Asset Layout

The K1 config uses policy_asset_roots as the base folder for Sim2Real assets:

policy_asset_roots:
  - ../assets/k1

The asset value is a path relative to policy_asset_roots.

policy_asset_roots../assets/k1

+

assetmimic/dance1

=

loaded directory../assets/k1/mimic/dance1

Required inside the loaded directory

exported/

policy.onnx

params/

sim2real.yaml<motion_reference>.csvmimic only

Mimic behaviors also reference a motion CSV under the same params directory. For example, the current K1 mimic_dance1 behavior is declared as:

mimic_dance1:
  kind: mimic
  asset: mimic/dance1
  motion: dance1.csv

Built-in K1 Modes

The default K1 configuration includes these built-in modes:

Mode	Behavior
Damping	Sets joint stiffness to zero. Damping is applied using the values in k1_config.yaml. Used for startup and failsafe.
ReadyPose	Moves the robot to the ready posture defined in k1_config.yaml.
ZeroPose	Moves the robot to the zero posture defined in k1_config.yaml.
Velocity	Runs the velocity policy behavior.
MimicSquat	Runs the squat mimic policy and motion reference.
MimicDance1	Runs the first dance mimic policy and motion reference.
MimicDance2	Runs the second dance mimic policy and motion reference.

The K1 mimic selector currently maps RC selector codes to modes as follows:

Selector code	Mode
200	MimicSquat
201	MimicDance1
202	MimicDance2

How to Deploy Your Own Cyclo Lab-Trained Policy

Treat a Cyclo Lab export as a Sim2Real asset, then bind that asset to a robot mode.

info

A deployable Cyclo Lab policy is a full Sim2Real asset, not only policy.onnx. The asset also needs the matching sim2real.yaml and, for mimic policies, the motion reference CSV.

1. Export for deployment. The required files are exported/policy.onnx and params/sim2real.yaml. For mimic policies, also include the motion reference CSV required by the mimic behavior.

2. Place the policy under assets/k1.

Create one policy folder under ai_sapiens_sim2real/assets/k1/<category>/.

<category>: locomotion/velocity for a velocity policy or mimic for a mimic policy.

ai_sapiens_sim2real/assets/k1/<category>/

• <your_policy_name>/
  • exported/
    • policy.onnx
  • params/
    • sim2real.yaml
    • <motion_reference>.csvmimic only

The category is only part of the asset path; the behavior type is selected in the config.

3. Update k1_config.yaml.

Use the locomotion example for a velocity policy, or the mimic example for a mimic policy.

Locomotion policy

Add one state_behaviors entry that points to your asset folder, then change only the existing Velocity state's run value.

state_behaviors:  velocity_policy:                            # already exists    kind: policy    asset: locomotion/velocity/walk_default  my_velocity_policy:                         # 1. Add this below the existing behavior.    kind: policy    asset: locomotion/velocity/<your_policy_name>state_machine:  states:    Velocity:                                 # already exists      run: my_velocity_policy                 # 2. Change only this value.      transitions:                            # keep the existing transitions        ...

Mimic policy

Add one state_behaviors entry, add one state_machine.states entry for the new mimic mode, then map an unused RC selector code to that mode.

Mimic is an abstract parent state and cannot be selected directly. New Mimic modes use parent: Mimic to share the existing RC transitions.

state_behaviors:  mimic_dance1:                               # already exists    kind: mimic    asset: mimic/dance1    motion: dance1.csv  my_mimic_motion:                            # 1. Add a new behavior.    kind: mimic    asset: mimic/my_motion    motion: my_motion_reference.csv           # From the asset's params/ directory.state_machine:  states:    MimicDance1:                              # already exists      parent: Mimic      run: mimic_dance1    MyMimicMotion:                            # 2. Add a new mode.      parent: Mimic      run: my_mimic_motionselectors:  mimic_selector:    table:      200: MimicSquat                         # already exists      201: MimicDance1                        # already exists      202: MimicDance2                        # already exists      203: MyMimicMotion                      # 3. Add an unused RC selector code.

Policy Compatibility Checks

The node uses the observation and action layout from the config files as written. Check these items before running the policy:

• Joint names listed in policy_joints inside sim2real.yaml.
• Observation terms and their order in sim2real.yaml.
• Action scaling, offset, and clipping in sim2real.yaml.
• Motion reference file, FPS, and playback window for mimic policies.

info

• policy_joints: joint order used by the policy input and output.
• robot_joint_order: joint order expected by the lower-level controller.

The two lists can use different orders. Just make sure every joint name in policy_joints exists with the same name in robot_joint_order.

The node stops during startup if policy joint names or ONNX input/output sizes do not match the configuration. Incorrect ordering or values may pass these checks but still produce unexpected robot behavior.

Recommended First-Run Flow

After you place the policy folder under assets/k1, update k1_config.yaml, and verify the compatibility checks above, validate the configuration with a dry run first. In a dry run, the node runs the full control loop but publishes no joint commands, so the robot does not move.

1. Follow the "Without cycloctl" steps in Bringup with cycloctl → Start to start zenoh_daemon and ai_sapiens_bringup, but do not start ai_sapiens_sim2real yet. Confirm that /joint_states, the IMU topic, the joint command controller, and RC input are active.

2. Start ai_sapiens_sim2real in the same way, but add command_publisher_enabled:=false for the dry run:

   ros2 launch ai_sapiens_sim2real ai_sapiens_sim2real.launch.py command_publisher_enabled:=false

3. Confirm that the launch terminal logs do not report a startup timeout or missing IMU, joint state, or RC input. The node starts in Damping.

4. Select the mode that should run your policy from the RC transmitter and confirm that the logs show the expected mode name. If a different mode name appears, recheck the state_machine, state_behaviors, and selectors entries in config/k1_config.yaml.

After the dry run passes, stop the dry-run node with Ctrl-C, launch it again without command_publisher_enabled:=false, and select that mode from the RC transmitter. For API-driven tests, first run the policy once through RC control, then use the optional API control flow below.

Optional API Control

Manual operation is the default path. API authority is optional. It lets an external API client request modes and send velocity commands.

Sim2Real separates the robot's behavior mode from the authority that is allowed to request it.

• Mode is the active robot behavior, such as Damping, ReadyPose, Velocity, or a mimic motion mode.
• Authority is who can request the next mode and provide velocity commands. Manual authority follows the operator's teleoperation input. API authority accepts ROS2 service requests and velocity commands from an external API client.

info

Even in API authority, RC input must keep arriving. The operator can use the RC transmitter to take control back or force Damping, and missing RC input triggers failsafe.

The authority states are:

Authority	Meaning
MANUAL	The RC transmitter owns mode switching. This is used at startup and after API control ends.
API_WARMUP	A short guarded handoff period after the operator requests API control. Service requests are still rejected.
API	ROS 2 service requests can switch to modes that are currently available through the API.

API Handoff

API authority must be requested by the operator from the RC transmitter. The API switch must be toggled while the node is in manual authority. If the switch is already enabled during boot, the node does not enter API control until the operator lowers and raises it again.

An API handoff is accepted only when:

• The API heartbeat is arriving within api_heartbeat_timeout.
• The robot is currently in an allowed entry mode: Damping, ReadyPose, or Velocity.
• The RC velocity command is zero.
• The /cmd_vel velocity command is zero.

note

Allowed entry modes are configured under authority.api_entry.allowed_from_states.

For the RC and /cmd_vel commands above, "zero" means that each of linear.x, linear.y, and angular.z is within authority.api_entry.velocity_neutral_threshold (±0.05 for K1).

If the handoff is rejected, fix the failing condition, then toggle the API switch off and on again to retry. Keeping the switch on does not retry the handoff; the node waits for a fresh switch toggle.

After acceptance, the node enters API_WARMUP for the configured warmup duration. For the K1 config, the warmup duration is 3.0 seconds, defined in the authority.api_entry section of config/k1_config.yaml.

API_WARMUP is a short confirmation window before the external API client receives API authority. During this window, the heartbeat must keep arriving and the operator must keep the API switch enabled.

During warmup, the robot keeps the current mode and rejects service requests. When warmup completes, authority becomes API without changing the active mode.

To end API control, select Damping, ReadyPose, or Velocity on the RC transmitter, then turn off the API switch. The node returns to manual authority and applies the RC selection through the normal RC transition rules, so the selected mode is not always entered immediately. For example, Damping cannot transition directly to Velocity; select ReadyPose first.

If the heartbeat is lost, API control ends and the node returns to manual authority. The current RC selection is applied through the normal RC transition rules, but this handoff never starts or continues a mimic mode: when the result would be a mimic mode, the node enters Velocity instead.

Heartbeat

The node subscribes to ai_sapiens_interfaces/msg/ApiHeartbeat on:

/ai_sapiens/api_heartbeat

An external API client must keep publishing this heartbeat while it wants to use API authority. The message contains a Header and a sequence value that must keep changing. The node treats the heartbeat as invalid when sequence does not change within api_heartbeat_timeout. The default timeout is 0.2 seconds.

Velocity Command

When API authority is active, velocity commands are read from:

/cmd_vel

This replaces the RC transmitter as the velocity command source while API authority is active.

Request a Mode

Use /ai_sapiens/set_mode_by_name to request a mode by its name, such as Velocity or MyMimicMotion.

ros2 service call /ai_sapiens/set_mode_by_name \
  ai_sapiens_interfaces/srv/SetModeByName \
  "{mode_name: Velocity}"

Request field	Description
mode_name	Name of the mode to request.

Response field	Description
success	Whether the request was accepted.
active_mode	Mode active when the request was handled.
message	Message explaining the result.

A request can be accepted only when api_request_available is true in /ai_sapiens/mode_status and the requested mode is currently listed in available_modes returned by /ai_sapiens/list_modes.

From Damping, request ReadyPose before requesting Velocity or a mimic mode.

List Modes

Use /ai_sapiens/list_modes to inspect the configured modes.

ros2 service call /ai_sapiens/list_modes ai_sapiens_interfaces/srv/ListModes "{}"

Response field	Description
success	Whether the mode list was returned successfully.
modes	All concrete modes in the state machine.
available_modes	Modes allowed as API targets from the current active mode. Empty unless authority is API.
message	Message explaining the result.

Monitor Node Status

The node publishes ai_sapiens_interfaces/msg/ModeStatus on /ai_sapiens/mode_status:

ros2 topic echo /ai_sapiens/mode_status

Field	Description
header.stamp	Time when the status was published.
active_mode	Mode currently running.
authority	Current control authority.
teleop_input_valid	Whether RC input is available.
api_heartbeat_valid	Whether the API heartbeat is valid.
api_request_available	Whether the node is ready for a new API mode request. It is true when authority is API, the API switch remains on, the heartbeat is valid, and no other API mode request is pending.
last_transition_reason	Reason for the most recent mode or authority transition.

Debug Policy Data

When debug publishing is enabled with debug_publish_enabled:=true, the node also publishes policy observation and action data:

Topic	Purpose
/policy_input/raw_observation	Observation vector supplied to the policy, after scale, clip, and history processing.
/policy_output/processed_action	Processed policy action before command publishing.

Use these topics to debug policy inputs and outputs; do not send robot commands through them.