Skip to main content

VR Teleoperation

This guide explains how to set up and operate the AI Worker using a VR device. Currently, only the Meta Quest 3 is supported.

VR Teleoperation is compatible with all AI Worker models.

  • Gripper Models: BG2, SG2, W1, W2
  • Hand Models: FFW-BH5, FFW-SH5

For more information on the models, please refer to Product Models.

The VR setup process is identical for all models unless otherwise specified.

warning

Modify initial pose for safety.

The default initial pose for the AI Worker (arms straight down) is not ideal for VR, as VR teleoperation requires the hands to be within the camera's field of view. It is highly recommended to change the initial position to a "ready" pose (e.g., elbows bent 90° with hands forward). Refer to this Discord post for instructions on modifying the initial pose (a Discord account may be required).

1. VR Device Setup​

1. Enable Developer Mode​

Enable Developer Mode via the Meta Horizon mobile app:

Menu → Device management → Devices → Headset settings → Developer Mode → On

menu_metaheadset_settings_metadeveloper_mode_metadeveloper_mode_toggle_meta

2. Configure Headset Settings​

  1. Power / Sleep : Settings → General (use Power or Display depending on your firmware): set Display off and Sleep mode to 4 hours (or the longest option available) so the headset does not sleep during use.

Quest power and sleep settings

  1. Play area : Settings → Environment setup → Boundary → Stationary boundary size → Large.

Stationary boundary set to Large

Alternatively, use Create new boundary to define the boundary manually.

VR Startup​

1. Clone the repository​

Clone the repository on either the Robot PC or a USER PC. The default setup uses the Robot PC.

Replace SNPR48A0000 with the serial number printed on the back of the robot body.

ssh robotis@ffw-SNPR48A0000.local
cd ~/
git clone -b jazzy https://github.com/ROBOTIS-GIT/robotis_applications.git
cd robotis_applications

2. Start and Enter the Docker Container​

cd docker

# Start container
./container.sh start

# Enter running container
./container.sh enter

3. Start VR Publisher Node​

In the Docker environment, launch the VR publisher node corresponding to your model.

Note that sg2 model is used as an example.

ros2 launch robotis_vuer vr.launch.py model:=sg2

Or use the shortcut:

vr model:=sg2

BG2 and SG2 share the model:=sg2 VR publisher. For W1/W2, replace it with model:=f2. The Cyclo Motion Controller uses a separate model setting that must match the actual Follower, as shown under Running VR Teleoperation.

4. Open the Vuer Page (Quest Browser)​

Open the Meta Quest browser (or a browser on the same network, if applicable) and go to:

https://{pc_ip}:8012?ws=wss://{pc_ip}:8012

Replace {pc_ip} with the actual IP of the machine hosting the Vuer server.

Example: The AI Worker PC (SG2’s Orin) has the IP address 192.168.6.2.

https://192.168.6.2:8012?ws=wss://192.168.6.2:8012

The PC on BG2 should be connected to a router and use a corresponding IP address within that network.

Self-signed HTTPS warning​

If the browser shows a certificate warning, use Advanced → proceed to the site (wording may vary by browser).

Browser advanced / certificate bypass

Proceed to site (unsafe)

Enter VR​

warning

Positioning and Calibration

Ensure you are at your intended operating location before proceeding. The following step initializes the VR coordinate system based on your current physical position.

Click the Enter VR button.

Enter VR button

The session is ready when passthrough becomes active and axis markers appear on your hands.

Passthrough with hand axes

Once the VR setup is complete, hang the headset around your neck and tighten the strap until the device is stable and secure.

warning

If the Vuer server is restarted, you must refresh the browser page and click Enter VR again.

Cyclo Control Setup​

VR teleoperation relies on Cyclo Control as the robot motion-control layer. It receives VR references and generates the arm trajectories that the robot follows. Ensure it is installed and configured before proceeding.

You can find the installation steps in the cyclo_control repository.

For more information, see Cyclo Control.

Running VR Teleoperation​

Execute these commands on the Robot PC. You can either connect a keyboard and mouse directly to the Nvidia Orin, or access it via SSH (see the Setup Guide for instructions). Enter the ai_worker Docker container first:

cd ~/ai_worker
./docker/container.sh enter

Run Zenoh Daemon​

RMW_ZENOH NOTIFICATION

AI WORKER uses zenoh as RMW since version 2.0.0, so you need to bring up the Zenoh daemon before bringup the robot.
Please refer to the Zenoh Communication guide.

warning

If Cyclo Manager is installed, the Zenoh daemon may already be running. Check whether the zenoh_daemon container is running. If it is, you do not need to start the Zenoh daemon separately.

docker ps | grep zenoh_daemon

In the AI Worker container, run the following command to bring up the Zenoh daemon:

ros2 run rmw_zenoh_cpp rmw_zenohd

or alias:

zenohd

Note that sg2 model is used here as an example. The VR publisher can be shared between models, but Cyclo Motion Controller must use the actual Follower model. BG2 and W1 do not have mobile-base movement.

FollowerVR Publisher (vr.launch.py)Cyclo Motion Controller (ai_worker_controller.launch.py)
BG2model:=sg2model:=bg2
SG2model:=sg2model:=sg2
W1model:=f2model:=f1
W2model:=f2model:=f2

1. Bring up the robot.​

For BG2, W1, or W2, replace sg2 in the launch filename or shortcut below with bg2, f1, or f2, respectively.

ros2 launch ffw_bringup ffw_sg2_follower_ai.launch.py

Or use the shortcut:

ffw_sg2_follower_ai

2. Start Cyclo Motion Controller​

Once the robot reaches its initial position, start Cyclo Motion Controller with controller_type parameter set as vr:

ros2 launch cyclo_motion_controller_ros ai_worker_controller.launch.py model:=sg2 controller_type:=vr

Or use shortcut:

motion_controller model:=sg2 controller_type:=vr

For BG2, W1, or W2, replace model:=sg2 with model:=bg2, model:=f1, or model:=f2, respectively. This selects the robot model used for motion calculation and collision checking.

3. Activate VR Publisher​

The VR node only publishes reference data while both squeeze (grip) buttons on the controllers are held down. This acts as a safety deadman switch.

Activate VR publisher

4. Activate the Controller​

warning

Pre-Activation Alignment: Align your physical arms with the robot's current pose before activating to prevent sudden movements. Maintain a safe distance from the robot.

You can activate the controller by pressing the X button on the left controller and the A button on the right controller at the same time.

Activate VR controller

Right after the controller is activated, the system checks the difference between the detected controller poses and the robot wrist poses. If the difference is small enough, the arm controller starts after 3 seconds. After startup, the slow start function remains active for 5 seconds. Because of this, it is recommended to keep your arm posture as close as possible to the robot posture before activating the controller.

SG2 Slow Start

5. Pause and resume teleoperation​

  • Release the squeeze buttons.
  • To resume, repeat Step 3 and 4.
warning

Avoid resuming VR publishing when your hands are far from the previous pose, because the robot may move quickly. Before resuming, it is recommended to make your arm posture as close as possible to the robot posture.

Troubleshooting​

For common issues, see VR Teleoperation Troubleshooting and ROS 2 Topics Are Not Visible.