use tmux, with 3+ vehiles I see sporadic 100ms IMU dropouts

Tools/simulation/README_gz_multi_vehicle.md

@@ -2,7 +2,7 @@
 
 ## Overview
 
-`gz_multi_vehicle.sh` is a script to easily launch multiple PX4 gz_x500 drones in Gazebo simulation with configurable positioning. Each drone instance runs in its own terminal window for easy monitoring.
+`gz_multi_vehicle.sh` is a script to easily launch multiple PX4 gz_x500 drones in Gazebo simulation with configurable positioning. All drone instances run in a tmux grid layout within your current terminal, making it easy to monitor all vehicles simultaneously.
 
 **Platform**: Ubuntu Linux only
 
@@ -15,13 +15,9 @@
 
 2. Gazebo (gz) must be installed (Harmonic, Ionic, or Jetty)
 
-3. A terminal emulator (terminator or gnome-terminal):
+3. **Required**: terminator and tmux:
    ```bash
-   # Recommended:
-   sudo apt install terminator
-
-   # Or use gnome-terminal:
-   sudo apt install gnome-terminal
+   sudo apt install terminator tmux
    ```
 
 ## Usage
@@ -54,20 +50,21 @@ Launch 2 drones (default configuration):
 
 ## What the Script Does
 
-1. **Validates** that PX4 SITL is built and terminal emulator is available
-2. **Cleans up** any existing PX4 and Gazebo instances
-3. **Launches the first vehicle in a new terminal** (instance 0):
-   - Starts Gazebo server and GUI
-   - Spawns x500_0 at position (0, 0, 0)
-   - MAV_SYS_ID = 1
-   - Terminal title: "PX4 Vehicle 0 (MAV_SYS_ID=1)"
-4. **Launches subsequent vehicles in separate terminals** (instances 1, 2, ...):
-   - Connects to running Gazebo instance
-   - Spawns x500_N at position (0, N×spacing, 0)
-   - MAV_SYS_ID = N+1
-   - Terminal title: "PX4 Vehicle N (MAV_SYS_ID=N+1)"
+1. **Validates** that PX4 SITL is built, terminator and tmux are installed
+2. **Prepares** working directories for all vehicle instances
+3. **Creates a tmux session** with a grid layout in your current terminal
+4. **Launches all vehicles** in separate tmux panes:
+   - **Instance 0** (first pane):
+     - Starts Gazebo server and GUI
+     - Spawns x500_0 at position (0, 0, 0)
+     - MAV_SYS_ID = 1
+   - **Instances 1+** (additional panes):
+     - Connect to running Gazebo instance
+     - Spawn x500_N at position (0, N×spacing, 0)
+     - MAV_SYS_ID = N+1
+     - Wait appropriately for Gazebo to be ready
 
-Each vehicle runs in its own terminal window, making it easy to monitor the output and debug individual drones.
+All vehicles are displayed in a tiled grid layout within the current terminal window, allowing you to see all outputs simultaneously.
 
 ## Vehicle Configuration
 
@@ -77,7 +74,7 @@ Each vehicle gets:
 - **Unique model name**: x500_0, x500_1, x500_2, ...
 - **Unique position**: Spaced along Y-axis
 - **Unique ROS 2 namespace** (if using ROS 2): px4_1, px4_2, px4_3, ...
-- **Dedicated terminal window**: Each instance displays its output in a separate terminal
+- **Dedicated tmux pane**: Each instance displays its output in a separate pane in the grid
 
 ## Using with QGroundControl
 
@@ -106,33 +103,48 @@ You should see topics namespaced by vehicle:
 
 ## Stopping the Simulation
 
-To stop all vehicles, you can either:
+To stop all vehicles:
 
-1. **Close the terminal windows** - Each terminal will prompt you to press Enter before closing
-2. **Kill all PX4 processes**:
+1. **Exit tmux session**: Press `Ctrl+B` then `D` to detach (vehicles keep running) or `Ctrl+C` in each pane to stop
+2. **Kill the tmux session**:
+   ```bash
+   tmux kill-session -t px4_multi_<timestamp>
+   ```
+   Or kill all PX4 multi-vehicle sessions:
+   ```bash
+   tmux list-sessions | grep px4_multi | cut -d: -f1 | xargs -I {} tmux kill-session -t {}
+   ```
+3. **Kill all PX4 processes**:
    ```bash
    pkill -x px4
    ```
 
-To stop Gazebo:
-```bash
-gz sim -k
-```
+## Tmux Navigation
+
+Useful tmux commands while in the session:
+- **Navigate panes**: `Ctrl+B` then arrow keys
+- **Zoom pane**: `Ctrl+B` then `Z` (toggle fullscreen for current pane)
+- **Scroll in pane**: `Ctrl+B` then `[`, then use arrow keys or Page Up/Down (press `q` to exit scroll mode)
+- **Detach from session**: `Ctrl+B` then `D`
+- **Reattach to session**: `tmux attach-session -t px4_multi_<timestamp>`
 
 ## Troubleshooting
 
 ### Vehicles not spawning
-- Check the terminal window for the specific vehicle - all output is displayed there
+- Check the tmux pane for the specific vehicle - all output is displayed there
 - Ensure Gazebo is installed: `gz sim --versions`
 - Make sure the first vehicle (instance 0) has fully started Gazebo before others spawn
+- Zoom into the first pane (`Ctrl+B` then `Z`) to see full output
 
 ### Gazebo GUI not appearing
-- Check if running in headless mode
+- Check the first pane (instance 0) for Gazebo startup messages
 - Try setting display: `export DISPLAY=:0`
 
-### Terminal emulator not found
-- Install terminator (recommended): `sudo apt install terminator`
-- Or install gnome-terminal: `sudo apt install gnome-terminal`
+### Missing terminator or tmux
+- Install required tools:
+  ```bash
+  sudo apt install terminator tmux
+  ```
 
 ### Port conflicts
 - The script automatically handles MAVLink ports (14540 + instance)
@@ -140,15 +152,21 @@ gz sim -k
 
 ## Monitoring Output
 
-Each vehicle displays its output in its own terminal window. The terminal title shows the vehicle instance and MAV_SYS_ID for easy identification. If a vehicle crashes or exits, the terminal will remain open and prompt you to press Enter before closing, allowing you to review any error messages.
+All vehicles are displayed in a grid layout using tmux panes. Each pane shows the output for one vehicle instance. You can:
+- View all vehicles simultaneously in the grid
+- Zoom into any pane for detailed viewing (`Ctrl+B` then `Z`)
+- Scroll through output history in any pane (`Ctrl+B` then `[`)
+- Navigate between panes using arrow keys (`Ctrl+B` then arrow key)
 
 ## Architecture Details
 
-The script implements the multi-vehicle pattern described in the PX4 docs:
-- First instance launches Gazebo server (no PX4_GZ_STANDALONE)
-- Subsequent instances connect to running server (PX4_GZ_STANDALONE=1)
-- Each instance uses PX4_GZ_MODEL_POSE for positioning
-- Automatic MAV_SYS_ID and UXRCE_DDS_KEY assignment based on instance number
+The script uses tmux for terminal multiplexing and implements the multi-vehicle pattern described in the PX4 docs:
+- **Tmux session**: Creates a single tmux session with multiple panes in a tiled grid layout
+- **Instance 0**: Launches Gazebo server (no PX4_GZ_STANDALONE)
+- **Instances 1+**: Connect to running Gazebo server (PX4_GZ_STANDALONE=1)
+- **Positioning**: Each instance uses PX4_GZ_MODEL_POSE for spawn location
+- **Identifiers**: Automatic MAV_SYS_ID and UXRCE_DDS_KEY assignment based on instance number
+- **Synchronization**: Sequential startup with delays to ensure Gazebo is ready
 
 ## References
 

Tools/simulation/gz_multi_vehicle.sh

@@ -20,22 +20,22 @@ PX4_DIR="$SCRIPT_DIR/../.."
 # Build directory
 BUILD_DIR="${PX4_DIR}/build/px4_sitl_default"
 
-# Detect available terminal emulator (Ubuntu only)
-TERMINAL=""
-if command -v terminator &> /dev/null; then
-    TERMINAL="terminator"
-elif command -v gnome-terminal &> /dev/null; then
-    TERMINAL="gnome-terminal"
-else
-    echo "ERROR: No compatible terminal emulator found"
-    echo "Please install either terminator or gnome-terminal"
+# Check for required tools
+if ! command -v terminator &> /dev/null; then
+    echo "ERROR: terminator is required"
+    echo "Please install terminator:"
     echo "  sudo apt install terminator"
-    echo "  or"
-    echo "  sudo apt install gnome-terminal"
     exit 1
 fi
 
-echo "Using terminal: $TERMINAL"
+if ! command -v tmux &> /dev/null; then
+    echo "ERROR: tmux is required for grid layout"
+    echo "Please install tmux:"
+    echo "  sudo apt install tmux"
+    exit 1
+fi
+
+echo "Using terminator with tmux grid layout"
 
 # Validate inputs
 if ! [[ "$num_vehicles" =~ ^[0-9]+$ ]] || [ "$num_vehicles" -lt 1 ]; then
@@ -72,87 +72,19 @@ echo "PX4 Directory: $PX4_DIR"
 echo "=================================================="
 echo ""
 
-# # Kill any existing PX4 instances
-# echo "Cleaning up any existing PX4 instances..."
-# pkill -x px4 || true
-# sleep 1
+# Kill any existing PX4 instances (exact match only, won't kill this script)
+echo "Cleaning up any existing PX4 instances..."
+pkill -x px4 || true
 
-# # Kill any existing Gazebo instances
-# echo "Cleaning up any existing Gazebo instances..."
-# pkill -x ruby || true  # Gazebo uses ruby
-# pkill gz || true
-# sleep 2
+# Kill any existing Gazebo instances (use exact match to avoid killing this script)
+echo "Cleaning up any existing Gazebo instances..."
+pkill -x gz || true
 
 echo ""
 echo "Launching vehicles..."
 echo ""
 
-# Function to launch terminal based on type
-launch_terminal() {
-    local title="$1"
-    local instance="$2"
-    local working_dir="$3"
-    local y_pos="$4"
-
-    case "$TERMINAL" in
-        terminator)
-            terminator --title="$title" -x bash -c "
-                cd '$working_dir'
-                echo '=========================================='
-                echo '$title'
-                echo '=========================================='
-                echo 'Working directory: $working_dir'
-                echo 'Instance: $instance'
-                if [ $instance -eq 0 ]; then
-                    echo 'Mode: Primary (launching Gazebo)'
-                    export PX4_SYS_AUTOSTART=4001
-                    export PX4_SIM_MODEL=gz_x500
-                    '$BUILD_DIR/bin/px4' -i $instance -d '$BUILD_DIR/etc'
-                else
-                    echo 'Mode: Secondary (connecting to Gazebo)'
-                    echo 'Position: (0, $y_pos, 0)'
-                    export PX4_SYS_AUTOSTART=4001
-                    export PX4_SIM_MODEL=gz_x500
-                    export PX4_GZ_MODEL_POSE='0,$y_pos'
-                    export PX4_GZ_STANDALONE=1
-                    '$BUILD_DIR/bin/px4' -i $instance -d '$BUILD_DIR/etc'
-                fi
-                echo ''
-                echo 'PX4 exited. Press Enter to close this terminal...'
-                read
-            " &
-            ;;
-        gnome-terminal)
-            gnome-terminal --title="$title" -- bash -c "
-                cd '$working_dir'
-                echo '=========================================='
-                echo '$title'
-                echo '=========================================='
-                echo 'Working directory: $working_dir'
-                echo 'Instance: $instance'
-                if [ $instance -eq 0 ]; then
-                    echo 'Mode: Primary (launching Gazebo)'
-                    export PX4_SYS_AUTOSTART=4001
-                    export PX4_SIM_MODEL=gz_x500
-                    '$BUILD_DIR/bin/px4' -i $instance -d '$BUILD_DIR/etc'
-                else
-                    echo 'Mode: Secondary (connecting to Gazebo)'
-                    echo 'Position: (0, $y_pos, 0)'
-                    export PX4_SYS_AUTOSTART=4001
-                    export PX4_SIM_MODEL=gz_x500
-                    export PX4_GZ_MODEL_POSE='0,$y_pos'
-                    export PX4_GZ_STANDALONE=1
-                    '$BUILD_DIR/bin/px4' -i $instance -d '$BUILD_DIR/etc'
-                fi
-                echo ''
-                echo 'PX4 exited. Press Enter to close this terminal...'
-                read
-            " &
-            ;;
-    esac
-}
-
-# Launch each vehicle instance
+# Prepare all vehicle instances
 for i in $(seq 0 $((num_vehicles - 1))); do
     # Create working directory for this instance
     working_dir="${BUILD_DIR}/instance_${i}"
@@ -165,38 +97,83 @@ for i in $(seq 0 $((num_vehicles - 1))); do
 
     # Calculate Y position (spacing along Y-axis)
     y_pos=$(echo "$i * $spacing" | bc -l)
-
-    # MAV_SYS_ID will be instance + 1
     mav_sys_id=$((i + 1))
 
     echo "----------------------------------------"
-    echo "Launching Vehicle $i"
+    echo "Preparing Vehicle $i"
     echo "  Working directory: $working_dir"
     echo "  Position: (0, $y_pos, 0)"
     echo "  MAV_SYS_ID: $mav_sys_id"
     echo "  Model name: x500_${i}"
-
-    if [ $i -eq 0 ]; then
-        echo "  Mode: Primary (launching Gazebo)"
-    else
-        echo "  Mode: Secondary (connecting to Gazebo)"
-    fi
-
-    # Launch in new terminal
-    title="PX4 Vehicle $i (MAV_SYS_ID=$mav_sys_id)"
-    launch_terminal "$title" "$i" "$working_dir" "$y_pos"
-
-    echo "  Status: Terminal launched"
     echo "----------------------------------------"
-    echo ""
+done
+
+echo ""
+
+# Create tmux session with grid layout
+SESSION_NAME="px4_multi_$(date +%s)"
+
+echo "Creating tmux session with $num_vehicles panes in grid layout..."
+
+# Start tmux session with first pane
+working_dir_0="${BUILD_DIR}/instance_0"
+tmux new-session -d -s "$SESSION_NAME" -c "$working_dir_0"
+
+# Create additional panes (one for each vehicle after the first)
+for i in $(seq 1 $((num_vehicles - 1))); do
+    tmux split-window -t "$SESSION_NAME:0" -c "${BUILD_DIR}/instance_${i}"
+    tmux select-layout -t "$SESSION_NAME:0" tiled
+done
+
+# Send commands to each pane
+for i in $(seq 0 $((num_vehicles - 1))); do
+    working_dir="${BUILD_DIR}/instance_${i}"
+    y_pos=$(echo "$i * $spacing" | bc -l)
+    mav_sys_id=$((i + 1))
+
+    # Send commands to pane
+    tmux send-keys -t "$SESSION_NAME:0.$i" "cd '$working_dir'" C-m
+    tmux send-keys -t "$SESSION_NAME:0.$i" "echo '========================================='" C-m
+    tmux send-keys -t "$SESSION_NAME:0.$i" "echo 'PX4 Vehicle $i (MAV_SYS_ID=$mav_sys_id)'" C-m
+    tmux send-keys -t "$SESSION_NAME:0.$i" "echo '========================================='" C-m
 
-    # Wait for first instance to start Gazebo before launching others
     if [ $i -eq 0 ]; then
-        echo "Waiting for Gazebo to start (15 seconds)..."
-        sleep 15
+        tmux send-keys -t "$SESSION_NAME:0.$i" "echo 'Mode: Primary (launching Gazebo)'" C-m
+        tmux send-keys -t "$SESSION_NAME:0.$i" "export PX4_SYS_AUTOSTART=4001" C-m
+        tmux send-keys -t "$SESSION_NAME:0.$i" "export PX4_SIM_MODEL=gz_x500" C-m
+        tmux send-keys -t "$SESSION_NAME:0.$i" "'$BUILD_DIR/bin/px4' -i $i -d '$BUILD_DIR/etc'" C-m
+    else
+        tmux send-keys -t "$SESSION_NAME:0.$i" "echo 'Mode: Secondary (connecting to Gazebo)'" C-m
+        tmux send-keys -t "$SESSION_NAME:0.$i" "echo 'Position: (0, $y_pos, 0)'" C-m
+        # For secondary instances, wait for Gazebo to start
+        if [ $i -eq 1 ]; then
+            tmux send-keys -t "$SESSION_NAME:0.$i" "echo 'Waiting for Gazebo to start...'" C-m
+            tmux send-keys -t "$SESSION_NAME:0.$i" "sleep 15" C-m
+        else
+            tmux send-keys -t "$SESSION_NAME:0.$i" "sleep $((15 + (i-1) * 2))" C-m
+        fi
+        tmux send-keys -t "$SESSION_NAME:0.$i" "export PX4_SYS_AUTOSTART=4001" C-m
+        tmux send-keys -t "$SESSION_NAME:0.$i" "export PX4_SIM_MODEL=gz_x500" C-m
+        tmux send-keys -t "$SESSION_NAME:0.$i" "export PX4_GZ_MODEL_POSE='0,$y_pos'" C-m
+        tmux send-keys -t "$SESSION_NAME:0.$i" "export PX4_GZ_STANDALONE=1" C-m
+        tmux send-keys -t "$SESSION_NAME:0.$i" "'$BUILD_DIR/bin/px4' -i $i -d '$BUILD_DIR/etc'" C-m
     fi
 done
 
+echo ""
+echo "All vehicles configured in tmux session: $SESSION_NAME"
+echo "Attaching to tmux session..."
+echo ""
+echo "Tmux Commands:"
+echo "  - Detach from tmux: Ctrl+B then D"
+echo "  - Kill session: tmux kill-session -t $SESSION_NAME"
+echo "  - Navigate panes: Ctrl+B then arrow keys"
+echo "  - Zoom pane: Ctrl+B then Z (toggle)"
+echo ""
+
+# Attach to the session
+exec tmux attach-session -t "$SESSION_NAME"
+
 echo ""
 echo "=================================================="
 echo "All vehicles launched successfully!"
@@ -218,9 +195,11 @@ done
 
 echo "Tips:"
 echo "-----"
-echo "- Each vehicle is running in its own terminal window"
+echo "- Each vehicle runs in its own tmux pane in the grid"
 echo "- QGroundControl should auto-connect to all vehicles"
 echo "- For ROS 2, start MicroXRCEAgent: MicroXRCEAgent udp4 -p 8888"
-echo "- To stop all vehicles, close the terminal windows or run: pkill -x px4"
+echo "- Navigate panes: Ctrl+B then arrow keys"
+echo "- Zoom pane: Ctrl+B then Z"
+echo "- To stop all: Ctrl+C in each pane or pkill -x px4"
 echo ""
 echo "=================================================="