Skip to content

Sim2Real

This guide covers the middleware and deployment-side software stack used to move LAV2 from simulator-only validation toward software-in-the-loop and hardware-adjacent workflows. In the current repository, the current entrypoint for that work lives under sim2real/.

Scope

This page is about how to assemble and run the deployment-side software stack:

  • simulator
  • flight controller
  • ROS workspace
  • ground station
  • LAV2 middleware and policy node

The ROS workspace under sim2real/ is independent from the main lav2/ package. That separation is intentional: the lav2/ package contains the core simulation, control, and training-side logic, while the ROS workspace hosts the middleware and deployment-facing integration layer.

flowchart LR
  Sim[Simulator]
  PX4[PX4 SITL]
  MAVROS[MAVROS and ROS workspace]
  Node[LAV2 deployment node]
  Policy[Policy or controller]
  Mixer[Mixer and mapping]
  GCS[Ground station]

  Sim <--> PX4
  PX4 <--> MAVROS
  MAVROS --> Node
  Node --> Policy
  Policy --> Mixer
  Mixer --> Node
  Node --> PX4
  GCS <--> PX4

SITL Stack

Overview

By the time sim-to-real validation reaches the SITL stage, the software stack usually expands into several independently evolving layers. In practice, the full stack includes:

  1. simulator: MuJoCo, Isaac Sim, Gazebo, or another physics backend
  2. flight controller: PX4 plus the controller or policy side used by LAV2
  3. ROS: middleware, MAVROS, and any additional robotics packages
  4. ground station: typically QGroundControl

ROS often pulls in even more supporting packages, such as localization, vision, or lidar integrations, so keeping the orchestration explicit matters.

The current recommended path is a Linux environment with ROS 2, PX4, and a Pegasus-Simulator-based Isaac Sim backend. The practical target matrix from the existing SITL notes is:

  • Ubuntu 22.04 with ROS 2 Humble, or Ubuntu 24.04 with ROS 2 Jazzy
  • PX4 installed and runnable for SITL
  • Isaac Sim available locally
  • Pegasus Simulator installed on top of that base

At the moment, the repository leans toward Pegasus Simulator for PX4 SITL work. The reason is not that it is the only possible backend, but that it is the one the current LAV2-side setup has been organized around.

Component Installation

The installation work can be thought of as four related component groups:

  • simulator installation and configuration
  • PX4 installation and SITL readiness
  • ROS 2 installation and Python environment setup
  • ground-station availability

Keeping these grouped explicitly is useful because SITL issues often come from version drift between these layers rather than from one package in isolation.

The simulator-to-flight-controller bridge is one of the biggest moving parts in this stack. In the current SITL notes, the most relevant references are Pegasus Simulator and PX4-Isaac-Sim.

For LAV2, the practical recommendation is still to assemble the environment manually instead of prioritizing a containerized stack. Containers can reduce environment drift, but the current simulator-side customization level is high enough that manual setup remains easier to iterate on. For container-oriented setups, the current notes point to nvidia_isaac-sim_ros2_docker and the Pegasus Simulator Docker PR for Isaac Sim 4.5.0 as useful references.

Pegasus Simulator Setup Notes

Pegasus Simulator should be installed according to its own installation guide, but there are a few LAV2-specific caveats from the current workflow. The base procedure should still follow the Pegasus Simulator installation guide.

If Isaac Sim was installed through a Python or uv/pip style workflow rather than through the prebuilt layout assumed by the upstream Pegasus instructions, you need to adjust the Pegasus configuration scripts so that ISAACSIM_PATH, ISAACSIM_PYTHON, and related paths actually point to the local Isaac Sim installation.

To stay aligned with the current LAV2 platform setup, the working notes also assume using the LAV2-maintained Pegasus fork rather than upstream defaults.

Then adjust items such as the PX4 path in extensions/pegasus.simulator/config/configs.yaml, and the simulator step size plus LAV2 model path in extensions/pegasus.simulator/pegasus/simulator/params.py.

Those settings are the boundary where the simulator-side backend and the LAV2 platform description meet.

How To Run The Simulator Side

For Pegasus Simulator, there are two main ways to run the stack.

Pegasus can run through the Isaac Sim graphical interface. This is useful for manual inspection. Pegasus documents this path in its extension mode tutorial.

Pegasus can also run through standalone Python. This is the recommended path for LAV2 SITL work because it is easier to script, reproduce, and modify during development. Pegasus documents that path in its standalone application tutorial, and the current LAV2-oriented example lives in examples/100_sitl_lav2.py inside the maintained fork.

The important point for this guide is that the simulator should be runnable in a form where PX4, ROS 2, and the LAV2 middleware can all attach to it in a stable way. The standalone Python path is the one that currently best fits that need.

ROS 2 Workspace

Build Environment

Once the simulator side is available, the next step is to build and run the ROS 2 workspace under sim2real/ros2_ws.

There is one important environment constraint here: the MAVROS integration inside the ROS 2 workspace depends on the system ROS 2 Python environment. That means the virtual environment used for this workspace should be created from the system Python rather than from an isolated Python that cannot see the ROS 2 packages.

The existing workflow notes recommend using a system-Python-based virtual environment, for example:

uv venv --no-managed-python --system-site-packages

Then build the ROS 2 workspace from inside sim2real/ros2_ws using the Python interpreter associated with that environment, for example via python -m colcon build.

If the generated install tree still points at the wrong interpreter, the other practical workaround is to patch the Python shebang in the generated entrypoint scripts under install/ so they point to the intended virtual environment's Python executable.

This constraint exists because ROS 2 still does not cleanly support arbitrary Python virtual-environment layouts in all workflows, so the workspace build and runtime environment need to be treated carefully.

ROS Workspace Environment Detail

If the ROS workspace builds but the installed entrypoints still resolve to the wrong interpreter, patch the shebang in the generated scripts under sim2real/ros2_ws/install/ so they point to the intended virtual environment Python.

LAV2 MAVROS Entry Point

After the ROS 2 workspace is built, the main LAV2-side launch entrypoint is:

  • ros2 launch lav2_mavros base_play.launch.py --show-args

That launch file is the practical control hub for the current deployment path. It is where the policy model, middleware topics, PX4 connection, and runtime mode selection are assembled into one ROS launch entrypoint.

The associated node implementation in base_play_node.py then ties together:

  • MAVROS odometry and state input
  • target generation or external target subscription
  • policy inference
  • controller-side command mapping
  • PX4-facing setpoint publication

The most important structural point is that the node runs a small finite-state machine around the vehicle mode and arming state. In practice, that logic is what lets the deployment node distinguish between waiting, hold, and active Offboard control phases instead of blindly publishing commands all the time.

stateDiagram-v2
  [*] --> WaitingForState
  WaitingForState --> Hold: odometry available
  Hold --> ActiveOffboard: armed and Offboard ready
  ActiveOffboard --> Hold: mode switch or disarm
  Hold --> WaitingForState: state lost
  WaitingForState --> [*]

Runtime Flow

sequenceDiagram
  autonumber
  participant Sim as Simulator
  participant PX4 as PX4 SITL
  participant MAVROS as MAVROS
  participant Node as LAV2 deployment node
  participant Policy as Policy or controller
  participant Mixer as Mixer and mapping
  participant GCS as Ground station

  Sim->>PX4: Advance SITL physics and sensor bridge
  PX4-->>MAVROS: Publish vehicle state
  MAVROS-->>Node: Forward odometry and mode state
  Node->>Policy: Build observation and run inference
  Policy-->>Node: Return command in policy space
  Node->>Mixer: Convert and normalize command
  Mixer-->>Node: Return PX4-facing setpoint
  Node-->>PX4: Publish Offboard setpoint
  GCS->>PX4: Arm or mode switch
  PX4-->>GCS: Report state and log stream

Launch Sequence

At a high level, the SITL runtime sequence is:

  1. bring up the simulator backend
  2. start PX4 SITL and verify the simulator-to-flight-controller connection
  3. launch MAVROS and the LAV2 ROS 2 node through base_play.launch.py
  4. connect the ground station
  5. switch into Offboard when the rest of the stack is healthy
  6. run the policy-controlled flight
  7. switch back out of Offboard after the test and inspect logs

This sequence matters because the deployment-side stack is only useful if the simulator, PX4, ROS 2, and the ground station all agree on the current vehicle state and command flow.

Ground Station And Validation

Because sim-to-real validation should stay close to real deployment conditions, the workflow also assumes a ground station such as QGroundControl is part of the test loop.

In the current workflow, after MAVROS and the simulator are running:

  • connect the ground station to PX4
  • arm and switch into Offboard only when the rest of the software stack is healthy
  • run the LAV2 policy through the ROS 2 node
  • switch back to a non-Offboard mode after the flight
  • export the flight-controller log for later analysis

Those logs can then be checked locally or uploaded to PX4 analysis tools for post-flight validation, such as PX4 Flight Review.

PX4 Control Mode Alignment

Why Alignment Matters

SITL can validate transport and orchestration, but usable deployment also depends on command semantics lining up across the full control path. The policy, controller, Mixer, ROS middleware, and PX4 must all agree on at least three things:

  • which control quantity is being commanded
  • which frame that quantity lives in
  • how the numeric range is normalized before it reaches PX4

If any one of those assumptions drifts, a policy that appears stable in local replay can become unusable once it is driven through Offboard control.

PX4-Oriented Command Families

For the current LAV2 stack, the most useful PX4-facing command families are:

  • cmd_motor_thrusts
  • cmd_ctbm
  • cmd_ctbr
  • cmd_ctatt
  • cmd_acc
  • cmd_vel
  • cmd_pos

The lower-level modes map more directly to the LAV2 control stack, while the higher-level modes let PX4 close more of the loop internally.

cmd_motor_thrusts is the closest to actuator-side control. It requires the command to be normalized before publication, with the physically meaningful thrust range living inside the normalized interval expected by PX4.

cmd_ctbm, cmd_ctbr, and cmd_ctatt are the most relevant modes when the LAV2 policy or controller already reasons in terms of collective thrust and rotational control. In these modes, collective thrust still needs to be mapped into the normalized range expected by PX4, and attitude-aware commands must respect PX4's frame convention.

cmd_acc, cmd_vel, and cmd_pos operate at a higher abstraction level. They are useful when the outer loop remains outside PX4 but should still align with PX4's expected NED-style semantics for acceleration, velocity, and position targets.

Normalization and Mapping

The most important normalization issue is collective thrust. Inside LAV2, the Mixer does more than allocate roll, pitch, yaw, and collective demand to rotors. It also computes the normalized thrust quantity needed to stay aligned with PX4-style command interfaces.

That means the deployment path should preserve a clean separation:

  • the controller or policy computes the desired command in LAV2's internal representation
  • the mixer and mapping layer convert that command into a normalized PX4-facing representation
  • the ROS node publishes the final PX4-compatible setpoint

This keeps the command semantics explicit instead of scattering normalization logic across multiple layers.

For attitude-aware interfaces such as cmd_ctatt, attitude should be expressed in the representation expected by PX4, and the bridge layer needs to be careful about frame and ordering conventions before publication.

Where To Implement The Alignment

There are two practical places to perform PX4 control-mode alignment.

The first option is to align directly in the policy action space. In that design, the learned policy already emits commands in the same normalized and frame-consistent format that PX4 expects. The advantage is that there is less translation work in the ROS node, and the control path is easier to inspect end-to-end. The tradeoff is tighter coupling between the learned policy and the PX4 interface.

The second option is to keep the policy output in the representation most useful for training and then implement the alignment in the ROS bridge node. That usually gives a cleaner separation between training-side design and deployment side adaptation, but it places more responsibility on the middleware layer to translate commands correctly and consistently.

In the current LAV2 structure, the practical split is usually:

  • keep the policy output compatible with the training environment
  • reuse lav2.controller.mapping and Mixer for command interpretation and normalization
  • perform the final PX4-mode-specific publication inside the ROS node

This keeps the deployment node thin while still allowing PX4-specific alignment to happen at the edge of the system.

Offboard FSM and Publication Strategy

The implementation detail that matters most in base_play_node.py is not the exact launch argument list, but the finite-state machine around Offboard activation.

The node should not behave like a stateless command forwarder. Instead, it tracks whether odometry is available, whether the vehicle has entered a state where command publication is meaningful, and when it is safe to begin active Offboard control. In practice this separates the runtime into phases such as:

  1. waiting for valid state feedback
  2. hold or standby before Offboard engagement
  3. active command publication once the vehicle is armed and ready

That state-machine boundary is where command alignment becomes operational. Before the vehicle reaches the active phase, the node should avoid treating the policy output as a valid flight command. Once Offboard control is active, the same node becomes responsible for continuously publishing setpoints in the control mode selected for PX4.

Practical Validation Focus

When this section is exercised in SITL or on a real platform, the most useful checks are:

  • whether the selected PX4 control mode matches the command representation produced by the policy or controller
  • whether collective thrust and rotational terms are normalized exactly once
  • whether frame conventions stay consistent from state estimation to final setpoint publication
  • whether the Offboard FSM enters and exits its active phase cleanly

If these four items are consistent, the remaining work is usually controller tuning or parameter alignment rather than interface mismatch.

API Cross-References