Skip to content

Sim2Real

This guide covers the middleware and deployment-side software stack used to move LAV2 from simulator-only validation toward software-in-the-loop and hardware-adjacent workflows. In this repository, the entrypoint for that work lives under sim2real/.

Scope

This page is about how to assemble and run the deployment-side software stack:

  • simulator
  • flight controller
  • ROS workspace
  • ground station
  • LAV2 middleware and policy node

The ROS workspace under sim2real/ is independent from the main lav2/ package. That separation is intentional: the lav2/ package contains the core simulation, control, and training-side logic, while the ROS workspace hosts the middleware and deployment-facing integration layer.

flowchart LR
  Sim[Simulator]
  PX4[PX4 SITL]
  MAVROS[MAVROS and ROS workspace]
  Node[LAV2 deployment node]
  Policy[Policy or controller]
  Mixer[Mixer and mapping]
  GCS[Ground station]

  Sim <--> PX4
  PX4 <--> MAVROS
  MAVROS --> Node
  Node --> Policy
  Policy --> Mixer
  Mixer --> Node
  Node --> PX4
  GCS <--> PX4

SITL Stack

Overview

By the time sim-to-real validation reaches the SITL stage, the software stack usually expands into several independently evolving layers. In practice, the full stack includes:

  1. simulator: MuJoCo, Isaac Sim, Gazebo, or another physics backend
  2. flight controller: PX4 plus the controller or policy side used by LAV2
  3. ROS: middleware, MAVROS, and any additional robotics packages
  4. ground station: typically QGroundControl

ROS often pulls in even more supporting packages, such as localization, vision, or lidar integrations, so keeping the orchestration explicit matters.

The recommended path is a Linux environment with ROS 2, PX4, and a Pegasus-Simulator-based Isaac Sim backend. The practical target matrix is:

  • Ubuntu 22.04 with ROS 2 Humble, or Ubuntu 24.04 with ROS 2 Jazzy
  • PX4 installed and runnable for SITL
  • Isaac Sim available locally
  • Pegasus Simulator installed on top of that base

The repository leans toward Pegasus Simulator for PX4 SITL work because the current LAV2-side setup is organized around it.

Component Installation

The installation work can be thought of as four related component groups:

  • simulator installation and configuration
  • PX4 installation and SITL readiness
  • ROS 2 installation and Python environment setup
  • ground-station availability

Keeping these grouped explicitly is useful because SITL issues often come from version drift between these layers rather than from one package in isolation.

The simulator-to-flight-controller bridge is one of the biggest moving parts in this stack. The most relevant references are Pegasus Simulator and PX4-Isaac-Sim.

For LAV2, the practical recommendation is to assemble the environment manually instead of prioritizing a containerized stack. Containers can reduce environment drift, but the simulator-side customization level is high enough that manual setup remains easier to iterate on. For container-oriented setups, useful references include nvidia_isaac-sim_ros2_docker and the Pegasus Simulator Docker PR for Isaac Sim 4.5.0.

Pegasus Simulator Setup Notes

Pegasus Simulator should be installed according to its own installation guide, but there are a few LAV2-specific caveats. The base procedure should still follow the Pegasus Simulator installation guide.

If Isaac Sim was installed through a Python or uv/pip style workflow instead of the prebuilt layout assumed by the upstream Pegasus instructions, the Pegasus configuration scripts need to be adjusted so that ISAACSIM_PATH, ISAACSIM_PYTHON, and related paths point to the local Isaac Sim installation.

To stay aligned with the LAV2 platform setup, the working notes also assume use of the LAV2-maintained Pegasus fork instead of upstream defaults.

Then adjust items such as the PX4 path in extensions/pegasus.simulator/config/configs.yaml, and the simulator step size plus LAV2 model path in extensions/pegasus.simulator/pegasus/simulator/params.py.

Those settings are the boundary where the simulator-side backend and the LAV2 platform description meet.

How To Run The Simulator Side

For Pegasus Simulator, there are two main ways to run the stack.

Pegasus can run through the Isaac Sim graphical interface. This is useful for manual inspection. Pegasus documents this path in its extension mode tutorial.

Pegasus can also run through standalone Python. This is the recommended path for LAV2 SITL work because it is easier to script, reproduce, and modify during development. Pegasus documents that path in its standalone application tutorial, and the LAV2-oriented example lives in examples/100_sitl_lav2.py inside the maintained fork.

The important point for this guide is that the simulator should be runnable in a form where PX4, ROS 2, and the LAV2 middleware can all attach to it in a stable way. The standalone Python path best fits that need.

ROS Workspaces

Hardware Build and Deployment on Jetson Orin NX

If the deployment target is an NVIDIA Jetson Orin NX, decide the platform release before getting into ROS workspace setup. In the current LAV2 workflow, this is not only an Ubuntu version choice. It also determines the ROS generation, the practical availability of inference runtimes, and how smoothly the stack can stay aligned with the officially supported Isaac Lab, Isaac ROS, and PX4 toolchains.

At the moment, the two most relevant Orin NX release lines are:

  • JetPack 5, typically paired with Ubuntu 20.04 and ROS 1
  • JetPack 6, typically paired with Ubuntu 22.04 and ROS 2

The difference is not limited to workspace layout. The deployment-time runtime story is different as well, and that has a direct effect on how much engineering friction the hardware path will carry.

Unless there is a hard legacy constraint, this guide recommends treating JetPack 6 + ROS 2 as the default deployment baseline for Orin NX. For new projects, long-term maintenance, and workflows that should align with Isaac Lab, Isaac ROS, and PX4 upstream support, prefer that combination. Keep JetPack 5 + ROS 1 only when older ROS 1 assets, MAVROS nodes, or existing deployment images must be preserved.

For new sim2real and hardware deployment work, JetPack 6 with ROS 2 is the recommended combination. Upstream support is stronger, and the path lines up more cleanly with the current LAV2 simulator-validation workflow.

ROS 2 is the primary officially supported version for Isaac Lab, Isaac ROS, and PX4. For LAV2, that means once the ROS 2 Python packages have been validated inside Pegasus Simulator, the move to Orin NX hardware can usually reuse nearly the same middleware and node structure instead of maintaining a separate ROS 1 deployment path.

At runtime, JetPack 6 is also a better fit if the deployment path should remain Python-friendly. The required Python runtime dependencies on Orin NX can usually be provided through jetson containers, which makes packages such as onnxruntime, torch, and related inference dependencies easier to obtain and reproduce. On top of that, the ROS 2 path can directly use px4_msgs and the PX4-native DDS interface for control message delivery, reducing dependence on a MAVROS adaptation layer and shortening the deployment loop.

From a practical engineering perspective, the intended flow is:

  • validate the ROS 2 Python package path in Pegasus Simulator
  • keep the same ROS 2 node-level interface when moving to Orin NX hardware
  • choose MAVROS or direct px4_msgs publication depending on the integration boundary that is needed

That keeps simulator validation and hardware deployment closer to the same interface contract and reduces the amount of extra bridge code that needs to be carried.

JetPack 5 can still be used for deployment, but it should be treated as a legacy-compatibility path, not the default for new work. LAV2 keeps it mainly to preserve existing ROS 1 workspaces, MAVROS nodes, and older deployment assets.

The practical issue is runtime availability on the Python side. Because jetson containers no longer host JetPack 5 pip wheels, common compute packages such as onnxruntime are usually not available through a straightforward Python package installation flow. torch can still be obtained through NVIDIA's official JetPack 5 distribution channel, but the available builds are typically older, for example: https://developer.download.nvidia.com/compute/redist/jp/v512/pytorch/.

In practice, continuing on JetPack 5 usually means choosing between two less convenient options:

  • rebuild the required prebuilt Python packages yourself, or recover them from an older build cache
  • move more of the Python inference path into C++ deployment code and rely primarily on the TensorRT runtime that is expected to be available on-device

If the deployment path also depends on the system Python, there is another constraint: JetPack 5 commonly stays on Python 3.8. That can conflict with the dependency versions expected by LAV2. The issue is usually solvable, but it often requires extra dependency pinning, environment isolation, or package-source workarounds.

Both options can work, but neither is especially flexible. The first increases environment maintenance and version pinning cost. The second shifts model integration and iteration cost into the C++ side of the deployment stack, and the older system Python version adds more deployment-side compatibility work. For that reason, this guide treats JetPack 5 as a retained compatibility path, not the recommended baseline.

Build Environment

Once the simulator side is available, the next step is to build and run the ROS workspace under sim2real/.

There is one important environment constraint here: the MAVROS integration inside the ROS workspace depends on the system ROS Python environment. The virtual environment used for a workspace should therefore be created from the system Python, not from an isolated Python that cannot see the ROS packages.

Use a system-Python-based virtual environment, for example:

uv venv --no-managed-python --system-site-packages

Then build the ROS 2 workspace from inside sim2real/ros2_ws, for example via colcon build.

For PX4 uXRCE-DDS agent setup inside a ROS 2 workspace, follow the official PX4 guide: https://docs.px4.io/main/en/middleware/uxrce_dds#build-run-within-ros-2-workspace

For the ROS 1 MAVROS path, the corresponding catkin package now lives under sim2real/ros1_ws/src/lav2_mavros. Build that workspace with the ROS 1 environment sourced, then use catkin tooling such as catkin_make or catkin build.

As with the ROS 2 deployment path, the workspace expects the main lav2 Python package to be available from the virtual environment rather than being vendored into the ROS package itself.

This constraint exists because ROS 2 still does not cleanly support arbitrary Python virtual-environment layouts in all workflows, so the workspace build and runtime environment need to be treated carefully.

Shell helpers are provided for the two main workspace paths:

  • scripts/sim2real/px4_ros1_setup.sh prepares the ROS 1 workspace environment for lav2_mavros, including PX4 Gazebo Classic setup and the required package/model search paths
  • scripts/sim2real/px4_ros2_setup.sh prepares the ROS 2 workspace environment together with PX4 Gazebo Classic paths needed by the ROS 2 MAVROS workflow
  • scripts/sim2real/open_ros2_tabs.sh opens one GNOME Terminal window with tabs for mavros, livox_ros_driver2, fast_lio, and lav2_mavros, with staggered startup delays between tabs
ROS Workspace Environment Detail

If the ROS workspace builds but the installed entrypoints still resolve to the wrong interpreter, patch the shebang in the generated scripts under sim2real/ros2_ws/install/ so they point to the intended virtual environment Python.

LAV2 ROS Entry Points

After the workspace is built, the main launch entrypoints are:

  • ros2 launch lav2_mavros base_play.launch.py
  • ros2 launch lav2_px4_msgs base_play.launch.py

For ROS 1, the equivalent launch entrypoint is:

  • roslaunch lav2_mavros base_play.launch
  • roslaunch lav2_mavros base_play_sitl.launch

These launch files are the practical control hub for the deployment path. They assemble the policy model, middleware topics, PX4 connection, and runtime mode selection into one ROS launch entrypoint.

The package-level division remains narrow in scope:

  • lav2_mavros for ROS 1 deploys the base_play policy through MAVROS and also provides the bundled Gazebo SITL launch path
  • lav2_mavros for ROS 2 deploys the same base_play policy through MAVROS
  • lav2_px4_msgs for ROS 2 deploys the same base_play policy through PX4 native px4_msgs topics rather than MAVROS

This keeps the package responsibilities close while changing only the ROS/PX4 interface boundary.

The associated node implementation in base_play_node.py then ties together:

  • vehicle odometry and state input from the chosen ROS bridge
  • target generation or external target subscription
  • policy inference
  • controller-side command mapping
  • PX4-facing setpoint publication

The most important structural point is that the node runs a small finite-state machine around the vehicle mode and arming state. In practice, that logic is what lets the deployment node distinguish between waiting, hold, and active Offboard control phases instead of blindly publishing commands all the time.

stateDiagram-v2
  [*] --> WaitingForState
  WaitingForState --> Hold: odometry available
  Hold --> ActiveOffboard: armed and Offboard ready
  ActiveOffboard --> Hold: mode switch or disarm
  Hold --> WaitingForState: state lost
  WaitingForState --> [*]

Runtime Flow

sequenceDiagram
  autonumber
  participant Sim as Simulator
  participant PX4 as PX4 SITL
  participant MAVROS as MAVROS
  participant Node as LAV2 deployment node
  participant Policy as Policy or controller
  participant Mixer as Mixer and mapping
  participant GCS as Ground station

  Sim->>PX4: Advance SITL physics and sensor bridge
  PX4-->>MAVROS: Publish vehicle state
  MAVROS-->>Node: Forward odometry and mode state
  Node->>Policy: Build observation and run inference
  Policy-->>Node: Return command in policy space
  Node->>Mixer: Convert and normalize command
  Mixer-->>Node: Return PX4-facing setpoint
  Node-->>PX4: Publish Offboard setpoint
  GCS->>PX4: Arm or mode switch
  PX4-->>GCS: Report state and log stream

Launch Sequence

At a high level, the SITL runtime sequence is:

  1. bring up the simulator backend
  2. start PX4 SITL and verify the simulator-to-flight-controller connection
  3. launch the matching ROS package and the LAV2 ROS node through base_play.launch or base_play_sitl.launch
  4. connect the ground station
  5. switch into Offboard when the rest of the stack is healthy
  6. run the policy-controlled flight
  7. switch back out of Offboard after the test and inspect logs

This sequence matters because the deployment-side stack is only useful if the simulator, PX4, the selected ROS interface, and the ground station all agree on the vehicle state and command flow.

Timing Alignment With The Training Environment

Getting SITL to run is only the first step. For policy deployment, the more important requirement is that the timing assumptions across SITL, state feedback, and action publication stay aligned with the training environment. In LAV2, that should map directly back to the training-side sim_dt, step_dt, and decimation assumptions rather than being treated as independent runtime knobs.

A practical alignment pass should check four items:

  1. the SITL simulator's own physics stepping rate matches the training-side simulation rate
  2. the state publication rate is high enough that observations are built from the current vehicle state rather than stale data
  3. the action publication rate matches the policy's effective inference rate during training
  4. actuator and dynamics behavior stays close enough to the training environment that the closed-loop system is still meaningfully the same

Simulation-Rate Alignment

On the training side, LAV2 usually expresses timing through sim_dt and step_dt in VehicleParams. In frameworks such as Isaac Lab, step_dt is often decimation * sim.dt. Once the policy moves into SITL, the first timing check should be the simulator backend itself: its physics stepping rate should still correspond to the rate assumed during training.

For example, if training used a 100 Hz physics update and a 50 Hz control update, the SITL backend should preserve the same or an equivalent relationship. If that relationship changes, then even identical observation and action dimensions no longer imply the same closed-loop system.

MAVROS State-Rate Alignment

When observations are built from MAVROS topics, low state-publication rates directly introduce observation latency. The policy no longer reasons over the current state, but over a state from several control periods earlier. In practice that often shows up as tracking lag, additional phase delay, or even oscillation in SITL.

For that reason, the MAVROS state rate should be at least as high as the policy action rate, and usually somewhat higher. The ROS 1 and ROS 2 SITL launch files in this repository already expose set_message_interval-related parameters, and the service can also be called manually at runtime, for example:

ros2 service call /mavros/set_message_interval mavros_msgs/srv/MessageInterval \
  "{message_id: 32, message_rate: 100}"

rosservice call /mavros/set_message_interval 32 100

The important point is not the literal value 100 Hz. The requirement is that the observation update rate be high enough to support the trained policy rate after accounting for bridge latency and the training-side step_dt.

Action-Rate Alignment

On the deployment side, the action publication rate is controlled by timer_hz. This should not be tuned as a free empirical parameter. It should correspond directly to the policy's effective action rate during training, which is the inverse of step_dt.

For example, if the trained policy updates every 0.02 s, then deployment should set timer_hz:=50.0. Increasing the rate does not usually make the controller "finer"; instead, it tends to make the same policy react on a faster cadence than it was trained for. Lowering it makes action hold periods longer and the outer loop noticeably slower.

ros2 launch lav2_mavros base_play.launch.py \
  model_path:=/abs/path/to/checkpoint.pt \
  timer_hz:=50.0

The same interpretation applies on the lav2_px4_msgs path: timer_hz is still the deployment-side inference and action-publication rate.

Dynamics Differences Across SITL Backends

Even with timing aligned, different SITL engines can still differ in ways that matter. A common example is motor or actuator time constants. LAV2's own dynamics use parameters such as tau_m, tau_up, and tau_down to describe rotor response, while Gazebo, Pegasus, MuJoCo, or other backends may use different motor models, filters, or defaults.

That means "PX4 SITL" is not automatically equivalent to "the same actuator loop." For example, Gazebo's motor time constant may differ from the one assumed in the LAV2-side implementation, which can change:

  • throttle build-up speed
  • collective thrust and angular-rate lag
  • the phase margin seen by the same policy

So alignment in Sim2Real should not be interpreted as only matching topic names and interface wiring. It also includes checking actuator and dynamics equivalence at the backend level.

Ground Station And Validation

Because sim-to-real validation should stay close to real deployment conditions, the workflow assumes a ground station such as QGroundControl is part of the test loop.

After MAVROS and the simulator are running:

  • connect the ground station to PX4
  • arm and switch into Offboard only when the rest of the software stack is healthy
  • run the LAV2 policy through the ROS node
  • switch back to a non-Offboard mode after the flight
  • export the flight-controller log for later analysis

Those logs can then be checked locally or uploaded to PX4 analysis tools for post-flight validation, such as PX4 Flight Review.

PX4 Control Mode Alignment

Why Alignment Matters

SITL can validate transport and orchestration, but usable deployment also depends on command semantics lining up across the full control path. The policy, controller, Mixer, ROS middleware, and PX4 must all agree on at least three things:

  • which control quantity is being commanded
  • which frame that quantity lives in
  • how the numeric range is normalized before it reaches PX4

If any one of those assumptions drifts, a policy that appears stable in local replay can become unusable once it is driven through Offboard control.

PX4-Oriented Command Families

For the LAV2 stack, the most useful PX4-facing command families are:

  • cmd_motor_thrusts
  • cmd_ctbm
  • cmd_ctbr
  • cmd_ctatt
  • cmd_acc
  • cmd_vel
  • cmd_pos

The lower-level modes map more directly to the LAV2 control stack, while the higher-level modes let PX4 close more of the loop internally.

cmd_motor_thrusts is the closest to actuator-side control. It requires the command to be normalized before publication, with the physically meaningful thrust range living inside the normalized interval expected by PX4.

cmd_ctbm, cmd_ctbr, and cmd_ctatt are the most relevant modes when the LAV2 policy or controller already reasons in terms of collective thrust and rotational control. In these modes, collective thrust still needs to be mapped into the normalized range expected by PX4, and attitude-aware commands must respect PX4's frame convention.

cmd_acc, cmd_vel, and cmd_pos operate at a higher abstraction level. They are useful when the outer loop remains outside PX4 but should still align with PX4's expected NED-style semantics for acceleration, velocity, and position targets.

Normalization and Mapping

The most important normalization issue is collective thrust. Inside LAV2, the Mixer does more than allocate roll, pitch, yaw, and collective demand to rotors. It also computes the normalized thrust quantity needed to stay aligned with PX4-style command interfaces.

That means the deployment path should preserve a clean separation:

  • the controller or policy computes the desired command in LAV2's internal representation
  • the mixer and mapping layer convert that command into a normalized PX4-facing representation
  • the ROS node publishes the final PX4-compatible setpoint

This keeps the command semantics explicit instead of scattering normalization logic across multiple layers.

For attitude-aware interfaces such as cmd_ctatt, attitude should be expressed in the representation expected by PX4, and the bridge layer needs to be careful about frame and ordering conventions before publication.

Where To Implement The Alignment

There are two practical places to perform PX4 control-mode alignment.

The first option is to align directly in the policy action space. In that design, the learned policy already emits commands in the same normalized and frame-consistent format that PX4 expects. The advantage is that there is less translation work in the ROS node, and the control path is easier to inspect end-to-end. The tradeoff is tighter coupling between the learned policy and the PX4 interface.

The second option is to keep the policy output in the representation most useful for training and then implement the alignment in the ROS bridge node. That usually gives a cleaner separation between training-side design and deployment side adaptation, but it places more responsibility on the middleware layer to translate commands correctly and consistently.

In the LAV2 structure, the practical split is usually:

  • keep the policy output compatible with the training environment
  • reuse lav2.controller.mapping and Mixer for command interpretation and normalization
  • perform the final PX4-mode-specific publication inside the ROS node

This keeps the deployment node thin while still allowing PX4-specific alignment to happen at the edge of the system.

Offboard FSM and Publication Strategy

The implementation detail that matters most in base_play_node.py is not the exact launch argument list, but the finite-state machine around Offboard activation.

The node should not behave like a stateless command forwarder. Instead, it tracks whether odometry is available, whether the vehicle has entered a state where command publication is meaningful, and when it is safe to begin active Offboard control. In practice this separates the runtime into phases such as:

  1. waiting for valid state feedback
  2. hold or standby before Offboard engagement
  3. active command publication once the vehicle is armed and ready

That state-machine boundary is where command alignment becomes operational. Before the vehicle reaches the active phase, the node should avoid treating the policy output as a valid flight command. Once Offboard control is active, the same node becomes responsible for continuously publishing setpoints in the control mode selected for PX4.

Practical Validation Focus

When this section is exercised in SITL or on a real platform, the most useful checks are:

  • whether the selected PX4 control mode matches the command representation produced by the policy or controller
  • whether collective thrust and rotational terms are normalized exactly once
  • whether frame conventions stay consistent from state estimation to final setpoint publication
  • whether the Offboard FSM enters and exits its active phase cleanly

If these four items are consistent, the remaining work is usually controller tuning or parameter alignment rather than interface mismatch.

API Cross-References