use tmux, with 3+ vehiles I see sporadic 100ms IMU dropouts

Tools/simulation/README_gz_multi_vehicle.md

@@ -0,0 +1,174 @@
+# Multi-Vehicle Gazebo Simulation Script
+
+## Overview
+
+`gz_multi_vehicle.sh` is a script to easily launch multiple PX4 gz_x500 drones in Gazebo simulation with configurable positioning. All drone instances run in a tmux grid layout within your current terminal, making it easy to monitor all vehicles simultaneously.
+
+**Platform**: Ubuntu Linux only
+
+## Prerequisites
+
+1. PX4 SITL must be built:
+   ```bash
+   make px4_sitl
+   ```
+
+2. Gazebo (gz) must be installed (Harmonic, Ionic, or Jetty)
+
+3. **Required**: terminator and tmux:
+   ```bash
+   sudo apt install terminator tmux
+   ```
+
+## Usage
+
+```bash
+./Tools/simulation/gz_multi_vehicle.sh [num_vehicles] [spacing]
+```
+
+### Parameters
+
+- `num_vehicles` (optional, default: 2, max: 10): Number of drones to spawn
+- `spacing` (optional, default: 0.5): Distance between drones in meters along the Y-axis
+
+### Examples
+
+Launch 3 drones with default 0.5m spacing:
+```bash
+./Tools/simulation/gz_multi_vehicle.sh 3
+```
+
+Launch 5 drones with 1.0m spacing:
+```bash
+./Tools/simulation/gz_multi_vehicle.sh 5 1.0
+```
+
+Launch 2 drones (default configuration):
+```bash
+./Tools/simulation/gz_multi_vehicle.sh
+```
+
+## What the Script Does
+
+1. **Validates** that PX4 SITL is built, terminator and tmux are installed
+2. **Prepares** working directories for all vehicle instances
+3. **Creates a tmux session** with a grid layout in your current terminal
+4. **Launches all vehicles** in separate tmux panes:
+   - **Instance 0** (first pane):
+     - Starts Gazebo server and GUI
+     - Spawns x500_0 at position (0, 0, 0)
+     - MAV_SYS_ID = 1
+   - **Instances 1+** (additional panes):
+     - Connect to running Gazebo instance
+     - Spawn x500_N at position (0, N×spacing, 0)
+     - MAV_SYS_ID = N+1
+     - Wait appropriately for Gazebo to be ready
+
+All vehicles are displayed in a tiled grid layout within the current terminal window, allowing you to see all outputs simultaneously.
+
+## Vehicle Configuration
+
+Each vehicle gets:
+- **Unique instance number**: 0, 1, 2, ...
+- **Unique MAV_SYS_ID**: instance + 1 (1, 2, 3, ...)
+- **Unique model name**: x500_0, x500_1, x500_2, ...
+- **Unique position**: Spaced along Y-axis
+- **Unique ROS 2 namespace** (if using ROS 2): px4_1, px4_2, px4_3, ...
+- **Dedicated tmux pane**: Each instance displays its output in a separate pane in the grid
+
+## Using with QGroundControl
+
+QGroundControl should automatically detect and connect to all vehicles. You can switch between them in the vehicle dropdown menu.
+
+## Using with ROS 2
+
+To use the vehicles with ROS 2:
+
+1. Start the MicroXRCE-DDS Agent (all vehicles connect to the same agent):
+   ```bash
+   MicroXRCEAgent udp4 -p 8888
+   ```
+
+2. List topics to see all vehicles:
+   ```bash
+   ros2 topic list
+   ```
+
+You should see topics namespaced by vehicle:
+- `/px4_1/fmu/in/...`
+- `/px4_1/fmu/out/...`
+- `/px4_2/fmu/in/...`
+- `/px4_2/fmu/out/...`
+- etc.
+
+## Stopping the Simulation
+
+To stop all vehicles:
+
+1. **Exit tmux session**: Press `Ctrl+B` then `D` to detach (vehicles keep running) or `Ctrl+C` in each pane to stop
+2. **Kill the tmux session**:
+   ```bash
+   tmux kill-session -t px4_multi_<timestamp>
+   ```
+   Or kill all PX4 multi-vehicle sessions:
+   ```bash
+   tmux list-sessions | grep px4_multi | cut -d: -f1 | xargs -I {} tmux kill-session -t {}
+   ```
+3. **Kill all PX4 processes**:
+   ```bash
+   pkill -x px4
+   ```
+
+## Tmux Navigation
+
+Useful tmux commands while in the session:
+- **Navigate panes**: `Ctrl+B` then arrow keys
+- **Zoom pane**: `Ctrl+B` then `Z` (toggle fullscreen for current pane)
+- **Scroll in pane**: `Ctrl+B` then `[`, then use arrow keys or Page Up/Down (press `q` to exit scroll mode)
+- **Detach from session**: `Ctrl+B` then `D`
+- **Reattach to session**: `tmux attach-session -t px4_multi_<timestamp>`
+
+## Troubleshooting
+
+### Vehicles not spawning
+- Check the tmux pane for the specific vehicle - all output is displayed there
+- Ensure Gazebo is installed: `gz sim --versions`
+- Make sure the first vehicle (instance 0) has fully started Gazebo before others spawn
+- Zoom into the first pane (`Ctrl+B` then `Z`) to see full output
+
+### Gazebo GUI not appearing
+- Check the first pane (instance 0) for Gazebo startup messages
+- Try setting display: `export DISPLAY=:0`
+
+### Missing terminator or tmux
+- Install required tools:
+  ```bash
+  sudo apt install terminator tmux
+  ```
+
+### Port conflicts
+- The script automatically handles MAVLink ports (14540 + instance)
+- UXRCE-DDS uses unique keys for namespacing
+
+## Monitoring Output
+
+All vehicles are displayed in a grid layout using tmux panes. Each pane shows the output for one vehicle instance. You can:
+- View all vehicles simultaneously in the grid
+- Zoom into any pane for detailed viewing (`Ctrl+B` then `Z`)
+- Scroll through output history in any pane (`Ctrl+B` then `[`)
+- Navigate between panes using arrow keys (`Ctrl+B` then arrow key)
+
+## Architecture Details
+
+The script uses tmux for terminal multiplexing and implements the multi-vehicle pattern described in the PX4 docs:
+- **Tmux session**: Creates a single tmux session with multiple panes in a tiled grid layout
+- **Instance 0**: Launches Gazebo server (no PX4_GZ_STANDALONE)
+- **Instances 1+**: Connect to running Gazebo server (PX4_GZ_STANDALONE=1)
+- **Positioning**: Each instance uses PX4_GZ_MODEL_POSE for spawn location
+- **Identifiers**: Automatic MAV_SYS_ID and UXRCE_DDS_KEY assignment based on instance number
+- **Synchronization**: Sequential startup with delays to ensure Gazebo is ready
+
+## References
+
+- [PX4 Multi-Vehicle Gazebo Docs](https://docs.px4.io/main/en/sim_gazebo_gz/multi_vehicle_simulation.html)
+- [PX4 ROS 2 Multi-Vehicle](https://docs.px4.io/main/en/ros2/multi_vehicle.html)

Tools/simulation/gz_multi_vehicle.sh

@@ -0,0 +1,205 @@
+#!/bin/bash
+# Multi-vehicle Gazebo (gz) simulation launcher for PX4
+# Spawns multiple gz_x500 drones with configurable spacing
+# Each instance runs in its own terminal window
+# Usage: ./gz_multi_vehicle.sh [num_vehicles] [spacing]
+#   num_vehicles: Number of drones to spawn (default: 2, max: 10)
+#   spacing: Distance between drones in meters (default: 0.5)
+
+set -e
+
+# Configuration
+num_vehicles=${1:-2}
+spacing=${2:-1}
+MAX_VEHICLES=10
+
+# Get script directory and PX4 root
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+PX4_DIR="$SCRIPT_DIR/../.."
+
+# Build directory
+BUILD_DIR="${PX4_DIR}/build/px4_sitl_default"
+
+# Check for required tools
+if ! command -v terminator &> /dev/null; then
+    echo "ERROR: terminator is required"
+    echo "Please install terminator:"
+    echo "  sudo apt install terminator"
+    exit 1
+fi
+
+if ! command -v tmux &> /dev/null; then
+    echo "ERROR: tmux is required for grid layout"
+    echo "Please install tmux:"
+    echo "  sudo apt install tmux"
+    exit 1
+fi
+
+echo "Using terminator with tmux grid layout"
+
+# Validate inputs
+if ! [[ "$num_vehicles" =~ ^[0-9]+$ ]] || [ "$num_vehicles" -lt 1 ]; then
+    echo "ERROR: Number of vehicles must be a positive integer"
+    echo "Usage: $0 [num_vehicles] [spacing]"
+    exit 1
+fi
+
+if [ "$num_vehicles" -gt "$MAX_VEHICLES" ]; then
+    echo "ERROR: Number of vehicles cannot exceed $MAX_VEHICLES"
+    echo "Usage: $0 [num_vehicles] [spacing]"
+    exit 1
+fi
+
+if ! [[ "$spacing" =~ ^[0-9]+\.?[0-9]*$ ]]; then
+    echo "ERROR: Spacing must be a positive number"
+    echo "Usage: $0 [num_vehicles] [spacing]"
+    exit 1
+fi
+
+# Check if PX4 is built
+if [ ! -f "${BUILD_DIR}/bin/px4" ]; then
+    echo "ERROR: PX4 SITL not built. Please run 'make px4_sitl' first"
+    exit 1
+fi
+
+echo "=================================================="
+echo "PX4 Multi-Vehicle Gazebo Launcher"
+echo "=================================================="
+echo "Number of vehicles: $num_vehicles"
+echo "Spacing: ${spacing}m"
+echo "Model: gz_x500"
+echo "PX4 Directory: $PX4_DIR"
+echo "=================================================="
+echo ""
+
+# Kill any existing PX4 instances (exact match only, won't kill this script)
+echo "Cleaning up any existing PX4 instances..."
+pkill -x px4 || true
+
+# Kill any existing Gazebo instances (use exact match to avoid killing this script)
+echo "Cleaning up any existing Gazebo instances..."
+pkill -x gz || true
+
+echo ""
+echo "Launching vehicles..."
+echo ""
+
+# Prepare all vehicle instances
+for i in $(seq 0 $((num_vehicles - 1))); do
+    # Create working directory for this instance
+    working_dir="${BUILD_DIR}/instance_${i}"
+    mkdir -p "$working_dir"
+
+    # Create symlink to gz_env.sh if it doesn't exist
+    if [ ! -f "$working_dir/gz_env.sh" ]; then
+        ln -sf "${BUILD_DIR}/rootfs/gz_env.sh" "$working_dir/gz_env.sh"
+    fi
+
+    # Calculate Y position (spacing along Y-axis)
+    y_pos=$(echo "$i * $spacing" | bc -l)
+    mav_sys_id=$((i + 1))
+
+    echo "----------------------------------------"
+    echo "Preparing Vehicle $i"
+    echo "  Working directory: $working_dir"
+    echo "  Position: (0, $y_pos, 0)"
+    echo "  MAV_SYS_ID: $mav_sys_id"
+    echo "  Model name: x500_${i}"
+    echo "----------------------------------------"
+done
+
+echo ""
+
+# Create tmux session with grid layout
+SESSION_NAME="px4_multi_$(date +%s)"
+
+echo "Creating tmux session with $num_vehicles panes in grid layout..."
+
+# Start tmux session with first pane
+working_dir_0="${BUILD_DIR}/instance_0"
+tmux new-session -d -s "$SESSION_NAME" -c "$working_dir_0"
+
+# Create additional panes (one for each vehicle after the first)
+for i in $(seq 1 $((num_vehicles - 1))); do
+    tmux split-window -t "$SESSION_NAME:0" -c "${BUILD_DIR}/instance_${i}"
+    tmux select-layout -t "$SESSION_NAME:0" tiled
+done
+
+# Send commands to each pane
+for i in $(seq 0 $((num_vehicles - 1))); do
+    working_dir="${BUILD_DIR}/instance_${i}"
+    y_pos=$(echo "$i * $spacing" | bc -l)
+    mav_sys_id=$((i + 1))
+
+    # Send commands to pane
+    tmux send-keys -t "$SESSION_NAME:0.$i" "cd '$working_dir'" C-m
+    tmux send-keys -t "$SESSION_NAME:0.$i" "echo '========================================='" C-m
+    tmux send-keys -t "$SESSION_NAME:0.$i" "echo 'PX4 Vehicle $i (MAV_SYS_ID=$mav_sys_id)'" C-m
+    tmux send-keys -t "$SESSION_NAME:0.$i" "echo '========================================='" C-m
+
+    if [ $i -eq 0 ]; then
+        tmux send-keys -t "$SESSION_NAME:0.$i" "echo 'Mode: Primary (launching Gazebo)'" C-m
+        tmux send-keys -t "$SESSION_NAME:0.$i" "export PX4_SYS_AUTOSTART=4001" C-m
+        tmux send-keys -t "$SESSION_NAME:0.$i" "export PX4_SIM_MODEL=gz_x500" C-m
+        tmux send-keys -t "$SESSION_NAME:0.$i" "'$BUILD_DIR/bin/px4' -i $i -d '$BUILD_DIR/etc'" C-m
+    else
+        tmux send-keys -t "$SESSION_NAME:0.$i" "echo 'Mode: Secondary (connecting to Gazebo)'" C-m
+        tmux send-keys -t "$SESSION_NAME:0.$i" "echo 'Position: (0, $y_pos, 0)'" C-m
+        # For secondary instances, wait for Gazebo to start
+        if [ $i -eq 1 ]; then
+            tmux send-keys -t "$SESSION_NAME:0.$i" "echo 'Waiting for Gazebo to start...'" C-m
+            tmux send-keys -t "$SESSION_NAME:0.$i" "sleep 15" C-m
+        else
+            tmux send-keys -t "$SESSION_NAME:0.$i" "sleep $((15 + (i-1) * 2))" C-m
+        fi
+        tmux send-keys -t "$SESSION_NAME:0.$i" "export PX4_SYS_AUTOSTART=4001" C-m
+        tmux send-keys -t "$SESSION_NAME:0.$i" "export PX4_SIM_MODEL=gz_x500" C-m
+        tmux send-keys -t "$SESSION_NAME:0.$i" "export PX4_GZ_MODEL_POSE='0,$y_pos'" C-m
+        tmux send-keys -t "$SESSION_NAME:0.$i" "export PX4_GZ_STANDALONE=1" C-m
+        tmux send-keys -t "$SESSION_NAME:0.$i" "'$BUILD_DIR/bin/px4' -i $i -d '$BUILD_DIR/etc'" C-m
+    fi
+done
+
+echo ""
+echo "All vehicles configured in tmux session: $SESSION_NAME"
+echo "Attaching to tmux session..."
+echo ""
+echo "Tmux Commands:"
+echo "  - Detach from tmux: Ctrl+B then D"
+echo "  - Kill session: tmux kill-session -t $SESSION_NAME"
+echo "  - Navigate panes: Ctrl+B then arrow keys"
+echo "  - Zoom pane: Ctrl+B then Z (toggle)"
+echo ""
+
+# Attach to the session
+exec tmux attach-session -t "$SESSION_NAME"
+
+echo ""
+echo "=================================================="
+echo "All vehicles launched successfully!"
+echo "=================================================="
+echo ""
+echo "Vehicle Summary:"
+echo "----------------"
+for i in $(seq 0 $((num_vehicles - 1))); do
+    y_pos=$(echo "$i * $spacing" | bc -l)
+    mav_sys_id=$((i + 1))
+    echo "Vehicle $i:"
+    echo "  - MAV_SYS_ID: $mav_sys_id"
+    echo "  - Model name: x500_${i}"
+    echo "  - Position: (0, $y_pos, 0)"
+    echo "  - ROS 2 namespace: px4_${mav_sys_id}"
+    echo "  - Terminal: PX4 Vehicle $i (MAV_SYS_ID=$mav_sys_id)"
+    echo ""
+done
+
+echo "Tips:"
+echo "-----"
+echo "- Each vehicle runs in its own tmux pane in the grid"
+echo "- QGroundControl should auto-connect to all vehicles"
+echo "- For ROS 2, start MicroXRCEAgent: MicroXRCEAgent udp4 -p 8888"
+echo "- Navigate panes: Ctrl+B then arrow keys"
+echo "- Zoom pane: Ctrl+B then Z"
+echo "- To stop all: Ctrl+C in each pane or pkill -x px4"
+echo ""
+echo "=================================================="