fly316/PX4-Autopilot
7
0
Fork 0
mirror of https://gitee.com/mirrors_PX4/PX4-Autopilot.git synced 2026-10-03 12:38:54 +08:00
Code Issues Packages Projects Releases Wiki Activity

Move PX4 Guide source into /docs (#24490)

Browse Source 
* Add vitepress tree

* Update existing workflows so they dont trigger on changes in the docs path

* Add nojekyll, package.json, LICENCE etc

* Add crowdin docs upload/download scripts

* Add docs flaw checker workflows

* Used docs prefix for docs workflows

* Crowdin obvious fixes

* ci: docs move to self hosted runner

runs on a beefy server for faster builds

Signed-off-by: Ramon Roche <mrpollo@gmail.com>

* ci: don't run build action for docs or ci changes

Signed-off-by: Ramon Roche <mrpollo@gmail.com>

* ci: update runners

Signed-off-by: Ramon Roche <mrpollo@gmail.com>

* Add docs/en

* Add docs assets and scripts

* Fix up editlinks to point to PX4 sources

* Download just the translations that are supported

* Add translation sources for zh, uk, ko

* Update latest tranlsation and uorb graphs

* update vitepress to latest

---------

Signed-off-by: Ramon Roche <mrpollo@gmail.com>
Co-authored-by: Ramon Roche <mrpollo@gmail.com>
This commit is contained in:
Hamish Willee
2025-03-13 16:08:27 +11:00
committed by GitHub
co-authored by Ramon Roche
parent 8e6d2ebe4a
commit 88d623bedb
5176 changed files with 558771 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/en/dev_log/flight_log_analysis.md
+8
View File
@@ -0,0 +1,8 @@
# Flight Log Analysis
Information about collecting and analysing flight logs is covered in:
- [Flight Reporting](../getting_started/flight_reporting.md) - How to download a log and report/discuss issues about a flight.
- [Log Analysis using Flight Review](../log/flight_review.md) - How to analyse many common vehicle problems using the [Flight Review](https://logs.px4.io/) online tool.
- [Flight Log Analysis](../log/flight_log_analysis.md) - Introduction to flight analysis and links to a number of analysis tools.
- [Statistical Analysis](flight_log_analysis_statistical.md) - Information & resources for statistical analysis (including Flight Review logs).
docs/en/dev_log/flight_log_analysis_statistical.md
+23
View File
@@ -0,0 +1,23 @@
# Statistical Flight Log Analysis
This topic contains information and resources related to statistical flight log analysis.
## Flight Review Public Logs
[Flight Review](../log/flight_log_analysis.md#flight-review-online-tool) hosts a large set of publicly available log files that can be used for statistical analysis, machine learning, or other purposes.
The dataset contains a set of different:
- vehicle types
- PX4 versions (including development versions)
- boards
- flight modes
The logs are accessible on [logs.px4.io/browse](https://logs.px4.io/browse) and are licensed under [CC-BY PX4](https://creativecommons.org/licenses/by/4.0/).
Log files can also be downloaded in bulk with the [download_logs.py](https://github.com/PX4/flight_review/blob/main/app/download_logs.py) script.
The script allows to filter by different attributes (like flight modes, airframe name or type).
Use the `--help` flag for a full list.
The newest logs will be downloaded first, and downloads can be interrupted and resumed later on.
There are different parsing libraries, for example [pyulog](../log/flight_log_analysis.md#pyulog) can be used to read logs with Python.
docs/en/dev_log/log_encryption.md
+255
View File
@@ -0,0 +1,255 @@
# Log Encryption
<Badge type="tip" text="PX4 v1.13" />
The [System Logger](../modules/modules_system.md#logger) can be used to create encrypted logs, which may then be decrypted manually before analysis.
The default encryption algorithm is XChaCha20, and the default wrapping algorithm is RSA2048-OAEP.
::: warning
Log encryption is not enabled by default in PX4 firmware builds.
To use it you will need to build firmware with this feature enabled and then upload it to the flight controller (see instructions below).
:::
::: tip
Log encryption was has been improved in PX4 main (v1.16+) to generate a single encrypted log file that contains both encrypted log data, and an encrypted symmetric key that you can use to decrypt it (provided you can decrypt the symmetric key).
In earlier versions the encrypted symmetric key was stored in a separate file.
For more information see the [Log Encryption (PX4 v1.15)](https://docs.px4.io/v1.15/en/dev_log/log_encryption.html).
:::
## How ULog Encryption Works
::: info
The encryption algorithm used is set in [SDLOG_ALGORITHM](../advanced_config/parameter_reference.md#SDLOG_ALGORITHM).
At time of writing, only `XChaCha20` is supported (AES can be selected, but there is no implementation).
If another algorithm is supported in future, the process is _likely_ to remain the same as documented here.
:::
The encryption process for each new ULog is:
1. A XChaCha20 symmetric key is generated and encrypted using an RSA2048 public key.
This wrapped (encrypted) key is stored on the SD card in the beginning of a file that has the suffix `.ulge` ("ulog encrypted").
2. When a log is captured, the ULog data is encrypted with the unwrapped symmetric key and the resulting data is appended into the end of the `.ulge` file immediately after the wrapped key data.
After the flight, the `.ulge` file containing both the wrapped symmetric key and the encrypted log data can be found on the SD card.
In order to extract the log file, a user must first decrypt the wrapped symmetric key, which can then be used to decrypt the log.
Decrypting the wrapped symmetric key file is only possible if the user has the corresponding RSA private key for the public key that was used to wrap it.
This process is covered in more detail in [Download & Decrypt Log Files](#download-decrypt-log-files) below.
## File Structure
Encrypted `.ulge` file contains following sections:
```plain
-------------------------
| Header |
-------------------------
| Wrapped symmetric key |
-------------------------
| Encrypted ulog data |
-------------------------
```
Header section (22 bytes) contains following fields:
| Bytes | Field |
| ------ | --------------------- |
| 0..6 | File magic identifier |
| 7 | Header version |
| 8..15 | Timestamp |
| 16 | exchange algorithm |
| 17 | exchange key index |
| 18..19 | key size |
| 20..21 | nonce size |
The header part begins with magic string: `"ULogEnc"`, which identifies this is encrypted ulog file.
The file offset of the symmetric key section is `22` and the file offset of the log data section is `22 + key_size + nonce_size` (`key_size` and `nonce_size` are taken from the header section).
## Custom PX4 Firmware with Log Encryption
You will need to build custom firmware that contains your own public RSA key and the required Crypto API modules to support log encryption.
This section shows how to do this using the `px4-fmu-v5` board as an example.
::: tip
We show you how to generate your own keys in the [Generate RSA Public & Private Keys](#generate-rsa-public-private-keys) section below.
:::
::: info
The modules in a PX4 build are defined in configuration files, which may be modified either manually or using the `menuconfig` tool.
For more information see: [PX4 Board Configuration (Kconfig)](../hardware/porting_guide_config.md).
:::
### Cryptotest Make Target
Crypto uses large amounts of flash memory, and is therefore not included in the default PX4 make targets for each board (such as `make px4-fmu-v5`).
The easiest way to add support for encrypted logs is to define a custom `make` target that includes the required modules and your public RSA keys.
::: warning
Many builds are close to their maximum capacity.
If you run into a build error telling you that you have gone above the maximum flash memory, you will need to disable other features in the `.px4board` file you are working on, or in the `default.px4board` file.
Be careful not to disable something you need.
For example, if you found you were running out of memory on FMUv4 boards you could disable SIH mode by setting `CONFIG_MODULES_SIMULATION_SIMULATOR_SIH=n` in [boards/px4/fmu-v4/default.px4board](https://github.com/PX4/PX4-Autopilot/blob/main/boards/px4/fmu-v4/default.px4board#L76), which may free up enough flash memory to allow crypto to be added.
:::
#### Pixhawk FMUv5 boards
The FMUv5 board already has a custom make target `px4-fmu-v5_cryptotest` that you can use to build custom firmware with the required modules and "test" RSA keys.
The configuration file that enables the above make target is [`cryptotest.px4board`](https://github.com/PX4/PX4-Autopilot/blob/main/boards/px4/fmu-v5/cryptotest.px4board) file in `boards/px4/fmu-v5`.
The relevant keys in that file are reproduced below:
```plain
CONFIG_BOARD_CRYPTO=y
CONFIG_DRIVERS_STUB_KEYSTORE=y
CONFIG_DRIVERS_SW_CRYPTO=y
CONFIG_PUBLIC_KEY1="../../../Tools/test_keys/rsa2048.pub"
```
::: info
The file also sets `CONFIG_PUBLIC_KEY0` to a key named `key0.pub`.
This is not used in the current PX4 implementation and can be ignored.
:::
::: details Overview of crypto-relevant keys
| Argument | Description |
| ---------------------------- | ----------------------------------------------------------------------------------------------------- |
| CONFIG_BOARD_CRYPTO | Include crypto module in firmware.<br>= `y`: Enable log encryption.<br>= `n`: Disable log encryption. |
| CONFIG_DRIVERS_SW_CRYPTO | Include the PX4 crypto backend library (used by above library).<br>= `y`: Enable<br>= `n`: Disable |
| CONFIG_DRIVERS_STUB_KEYSTORE | Includes the PX4 stub keystore driver.<br>= `y`: Enable<br>= `n`: Disable |
| CONFIG_PUBLIC_KEY0 | Location of public key for keystore index 0. |
| CONFIG_PUBLIC_KEY1 | Location of public key for keystore index 1.<br>= `{path to key1}` |
| CONFIG_PUBLIC_KEY2 | Location of public key for keystore index 2.<br>= `{path to key2}` |
| CONFIG_PUBLIC_KEY3 | Location of public key for keystore index 3.<br>= `{path to key3}` |
The stub keystore is a keystore implementation that can store up to four keys.
The initial values of these keys are set in the locations defined by `CONFIG_PUBLIC_KEY0` to `CONFIG_PUBLIC_KEY3`.
The keys can be used for different cryptographic purposes, which are determined by parameters.
The _exchange key_, which is the public key used for encrypting the symmetric key stored in the beginning of the `.ulge` file, is specified using [SDLOG_EXCH_KEY](../advanced_config/parameter_reference.md#SDLOG_EXCH_KEY) as an index value into the key store.
The value is `1` by default, which maps to the key defined in `CONFIG_PUBLIC_KEY1`.
The _logging key_ is the unencrypted symmetric key.
This is specified using [SDLOG_KEY](../advanced_config/parameter_reference.md#SDLOG_KEY) as an index value into the key store, and default to `2`.
Note that the value is generated fresh for each log, and any value specified in `CONFIG_PUBLIC_KEY2` would be overwritten.
You can use choose different locations for your keys as long as they aren't used by anything else.
:::
The key in `CONFIG_PUBLIC_KEY1` is the public key used to wrap the symmetric key in the the beginning of `.ulge` file (by default: see [SDLOG_EXCH_KEY](../advanced_config/parameter_reference.md#SDLOG_EXCH_KEY)).
You can use the `rsa2048.pub` key for testing, or replace it with the path to your own public key in the file (see [Generate RSA Public & Private Keys](#generate-rsa-public-private-keys)).
Build the firmware like this:
```sh
make px4-fmu-v5_cryptotest
```
#### Other Boards
For other boards you will need to first copy `cryptotest.px4board` into the root of the target board directory.
For example, for FMUv6 you would copy the board to [/boards/px4/fmu-v6x](https://github.com/PX4/PX4-Autopilot/tree/main/boards/px4/fmu-v6x).
Then you will need to add a few more configuration settings that are present in FMUv5 default configuration but not in the other boards.
We do add these using the `menuconfig` tool.
To use `menuconfig` you will need to add these dependencies:
```sh
sudo apt-get install libncurses-dev flex bison openssl libssl-dev dkms libelf-dev libudev-dev libpci-dev libiberty-dev autoconf
```
Now, in PX4, run the normal `make` command you would use to build the board you are targeting, but add "menuconfig" at the end of it.
Here we use `px4_fmu-v5_cryptotest` as an example, because that already has the settings that we want to copy:
```sh
make px4_fmu-v5_cryptotest menuconfig
```
Navigate to `Crypto API` and use the **Y** key to select it.
![Menuconfig Crypto API Main Menu Option](../../assets/hardware/kconfig-crypto-1.png)
This will open the menu below.
Enable the settings: `Blake2s hash algorithm`, `Entropy pool and strong random number generator`, and `Use interrupts to feed timing randomness to entropy pool`.
![Menuconfig Crypto Options Set](../../assets/hardware/kconfig-crypto-2.png)
::: tip
Some of these options can be tweaked if desired.
:::
After enabling encryption settings, exit `menuconfig`.
You can now build and test.
## Download & Decrypt Log Files
Encrypted log files are downloaded using the QGroundControl [Log Download](https://docs.qgroundcontrol.com/master/en/qgc-user-guide/analyze_view/log_download.html) view (**Analyze Tools > Log Download**) just like ordinary log files.
Note that the encrypted files will be downloaded with the `.ulg` suffix, instead of `.ulge`.
### Decrypt ULogs
Before you can analyze your encrypted logs, you will need to decrypt them.
There is a Python script that can be used to decrypt logs in `Tools/decrypt_ulog.py`.
When decrypting a `.ulge` file the script takes 3 arguments:
1. The encrypted log file.
2. An empty string `''`.
3. The decryption key (the RSA2048 `.pem` private key which is used to unwrap the symmetric key).
For example:
```sh
python3 decrypt_ulog.py \
/home/john/Downloads/log_24_2024-10-6-23-39-50.ulg '' \
new_keys/private_key.pem
```
On success the decrypted log file is created with the `.ulog` suffix.
::: info
The script can be used with both `.ulge` logs and the `.ulgc`/`.ulgk` files used in [PX4 v1.15 Log Encryption](https://docs.px4.io/v1.15/en/dev_log/log_encryption.html).
The full command line syntax is given below:
```sh
usage: decrypt_ulog.py [-h] [ulog_file] [ulog_key] [rsa_key]
CLI tool to decrypt an ulog file
positional arguments:
ulog_file .ulge/.ulgc, encrypted log file
ulog_key .ulgk, legacy encrypted key (give empty string '' to ignore for .ulge)
rsa_key .pem format key for decrypting the ulog key
optional arguments:
-h, --help show this help message and exit
```
:::
## Generate RSA Public & Private Keys
To generate a RSA2048 private and public key, you can use OpenSSL:
```sh
openssl genpkey -algorithm RSA -out private_key.pem -pkeyopt rsa_keygen_bits:2048
```
Then you can create a public key from this private key:
```sh
# Convert private_key.pem to a DER file
openssl rsa -pubout -in private_key.pem -outform DER -out public_key.der
# From the DER file, generate a public key in hex format, separated by commas
xxd -p public_key.der | tr -d '\n' | sed 's/\(..\)/0x\1, /g' > public_key.pub
```
To use this key you would modify your `.px4board` file to point `CONFIG_PUBLIC_KEY1` to the file location of `public_key.pub`.
The private key generated should be stored safely and used when you need to decrypt log files.
docs/en/dev_log/logging.md
+191
View File
@@ -0,0 +1,191 @@
# Logging
The [system logger](../modules/modules_system.md#logger) is able to log any ORB topic with all included fields.
Everything necessary is generated from the `.msg` file, so that only the topic name needs to be specified.
An optional interval parameter specifies the maximum logging rate of a certain topic.
All existing instances of a topic are logged.
The output log format is [ULog](../dev_log/ulog_file_format.md).
[Encrypted logging](../dev_log/log_encryption.md) is also supported.
## Usage
By default, logging is automatically started when arming, and stopped when disarming.
A new log file is created for each arming session on the SD card.
To display the current state, use `logger status` on the console.
If you want to start logging immediately, use `logger on`.
This overrides the arming state, as if the system was armed.
`logger off` undoes this.
If logging stops due to a write error, or reaching the [maximum file size](#file-size-limitations), PX4 will automatically restart logging in a new file.
For a list of all supported logger commands and parameters, use:
```
logger help
```
## Configuration
The logging system is configured by default to collect sensible logs for [flight reporting](../getting_started/flight_reporting.md) with [Flight Review](http://logs.px4.io).
Logging may further be configured using the [SD Logging](../advanced_config/parameter_reference.md#sd-logging) parameters.
The parameters you are most likely to change are listed below.
| Parameter | Description |
| ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [SDLOG_MODE](../advanced_config/parameter_reference.md#SDLOG_MODE) | Logging Mode. Defines when logging starts and stops.<br />- `-1`: Logging disabled.<br />- `0`: Log when armed until disarm (default).<br />- `1`: Log from boot until disarm.<br />- `2`: Log from boot until shutdown.<br />- `3`: Log based on the [AUX1 RC channel](../advanced_config/parameter_reference.md#RC_MAP_AUX1).<br />- `4`: Log from first armed until shutdown. |
| [SDLOG_PROFILE](../advanced_config/parameter_reference.md#SDLOG_PROFILE) | Logging profile. Use this to enable less common logging/analysis (e.g. for EKF2 replay, high rate logging for PID & filter tuning, thermal temperature calibration). |
| [SDLOG_MISSION](../advanced_config/parameter_reference.md#SDLOG_MISSION) | Create very small additional "Mission Log".<br>This log can _not_ be used with [Flight Review](../log/flight_log_analysis.md#flight-review-online-tool), but is useful when you need a small log for geotagging or regulatory compliance. |
Useful settings for specific cases:
- Raw sensor data for comparison: [SDLOG_MODE=1](../advanced_config/parameter_reference.md#SDLOG_MODE) and [SDLOG_PROFILE=64](../advanced_config/parameter_reference.md#SDLOG_PROFILE).
- Disabling logging altogether: [SDLOG_MODE=`-1`](../advanced_config/parameter_reference.md#SDLOG_MODE)
### Logger module
_Developers_ can further configure what information is logged via the [logger](../modules/modules_system.md#logger) module.
This allows, for example, logging of your own uORB topics.
### SD Card Configuration
Separately, the list of logged topics can also be customized with a file on the SD card.
Create a file `etc/logging/logger_topics.txt` on the card with a list of topics (For SITL, it's `build/px4_sitl_default/rootfs/fs/microsd/etc/logging/logger_topics.txt`):
```plain
<topic_name> <interval> <instance>
```
The `<interval>` is optional, and if specified, defines the minimum interval in ms between two logged messages of this topic.
If not specified, the topic is logged at full rate.
The `<instance>` is optional, and if specified, defines the instance to log.
If not specified, all instances of the topic are logged.
To specify `<instance>`, `<interval>` must be specified. It can be set to 0 to log at full rate
The topics in this file replace all of the default logged topics.
Example :
```plain
sensor_accel 0 0
sensor_accel 100 1
sensor_gyro 200
sensor_mag 200 1
```
This configuration will log sensor_accel 0 at full rate, sensor_accel 1 at 10Hz, all sensor_gyro instances at 5Hz and sensor_mag 1 at 5Hz.
## Scripts
There are several scripts to analyze and convert logging files in the [pyulog](https://github.com/PX4/pyulog) repository.
## File size limitations
The maximum file size depends on the file system and OS.
The size limit on NuttX is currently around 2GB.
## Dropouts
Logging dropouts are undesired and there are a few factors that influence the
amount of dropouts:
- Most SD cards we tested exhibit multiple pauses per minute.
This shows itself as a several 100 ms delay during a write command.
It causes a dropout if the write buffer fills up during this time.
This effect depends on the SD card (see below).
- Formatting an SD card can help to prevent dropouts.
- Increasing the log buffer helps.
- Decrease the logging rate of selected topics or remove unneeded topics from being logged (`info.py <file>` is useful for this).
## SD Cards
The maximum supported SD card size for NuttX is 32GB (SD Memory Card Specifications Version 2.0).
The **SanDisk Extreme U3 32GB** and **Samsung EVO Plus 32** are known to be reliable cards (do not exhibit write-time spikes, and thus virtually no dropouts).
The table below shows the **mean sequential write speed [KB/s]** / **maximum write time per block (average) [ms]** for F4- (Pixracer), F7-, and H7-based flight controllers.
| SD Card | F4 | F7 | H7 |
| ------------------------------------------------------------- | ------------- | ---------- | --------- |
| SanDisk Extreme U3 32GB | 1500 / **15** | 1800/10 | 2900/8 |
| Samsung EVO Plus 32GB | 1700/10-60 | 1800/10-60 | 1900/9-60 |
| Sandisk Ultra Class 10 8GB | 348 / 40 | ?/? | ?/? |
| Sandisk Class 4 8GB | 212 / 60 | ?/? | ?/? |
| SanDisk Class 10 32 GB (High Endurance Video Monitoring Card) | 331 / 220 | ?/? | ?/? |
| Lexar U1 (Class 10), 16GB High-Performance | 209 / 150 | ?/? | ?/? |
| Sandisk Ultra PLUS Class 10 16GB | 196 /500 | ?/? | ?/? |
| Sandisk Pixtor Class 10 16GB | 334 / 250 | ?/? | ?/? |
| Sandisk Extreme PLUS Class 10 32GB | 332 / 150 | ?/? | ?/? |
Logging bandwidth with the default topics is around 50 KB/s, which almost all SD cards satisfy in terms of their mean sequential write speed.
More important than the mean write speed is spikes (or generally high values) in the maximum write time per block (of 4 KB) or `fsync` times, as a long write time means a larger log buffer is needed to avoid dropouts.
PX4 uses bigger buffers on F7/H7 and read caching, which is enough to compensate for spikes in many poor cards.
That said, if your card has an `fsync` or write duration of several 100ms it is should not be preferred for use with PX4.
You can check the value by running [sd_bench](../modules/modules_command.md#sd-bench) should be run with more iterations (around 100 should do).
```sh
sd_bench -r 100
```
This defines the minimum buffer size: the larger this maximum, the larger the log buffer needs to be to avoid dropouts.
PX4 uses bigger buffers on F7/H7 and read caching to make up for some of these issues.
::: info
If you have concerns about a particular card you can run the above test and report the results to https://github.com/PX4/PX4-Autopilot/issues/4634.
:::
## Log Streaming
The traditional and still fully supported way to do logging is using an SD card on the FMU.
However there is an alternative, log streaming, which sends the same logging data via MAVLink.
This method can be used for example in cases where the FMU does not have an SD card slot (e.g. Intel® Aero Ready to Fly Drone) or simply to avoid having to deal with SD cards.
Both methods can be used independently and at the same time.
The requirement is that the link provides at least ~50KB/s, so for example a WiFi link.
And only one client can request log streaming at the same time.
The connection does not need to be reliable, the protocol is designed to handle drops.
There are different clients that support ulog streaming:
- `mavlink_ulog_streaming.py` script in PX4-Autopilot/Tools.
- QGroundControl:
![QGC Log Streaming](../../assets/gcs/qgc-log-streaming.png)
- [MAVGCL](https://github.com/ecmnet/MAVGCL)
### Diagnostics
- If log streaming does not start, make sure the `logger` is running (see above), and inspect the console output while starting.
- If it still does not work, make sure that MAVLink 2 is used.
Enforce it by setting `MAV_PROTO_VER` to 2.
- Log streaming uses a maximum of 70% of the configured MAVLink rate (`-r` parameter).
If more is needed, messages are dropped.
The currently used percentage can be inspected with `mavlink status` (1.8% is used in this example):
```sh
instance #0:
GCS heartbeat: 160955 us ago
mavlink chan: #0
type: GENERIC LINK OR RADIO
flow control: OFF
rates:
tx: 95.781 kB/s
txerr: 0.000 kB/s
rx: 0.021 kB/s
rate mult: 1.000
ULog rate: 1.8% of max 70.0%
accepting commands: YES
MAVLink version: 2
transport protocol: UDP (14556)
```
Also make sure `txerr` stays at 0.
If this goes up, either the NuttX sending buffer is too small, the physical link is saturated or the hardware is too slow to handle the data.
## See Also
- [Encrypted logging](../dev_log/log_encryption.md)
docs/en/dev_log/ulog_file_format.md
+511
View File
@@ -0,0 +1,511 @@
# ULog File Format
ULog is the file format used for logging messages. The format is self-describing, i.e. it contains the format and [uORB](../middleware/uorb.md) message types that are logged.
This document is meant to be the ULog File Format Spec Documentation.
It is intended especially for anyone who is interested in writing a ULog parser / serializer and needs to decode / encode files.
PX4 uses ULog to log uORB topics as messages related to (but not limited to) the following sources:
- **Device inputs:** Sensors, RC input, etc.
- **Internal states:** CPU load, attitude, EKF state, etc.
- **String messages:** `printf` statements, including `PX4_INFO()` and `PX4_ERR()`.
The format uses [little endian](https://en.wikipedia.org/wiki/Endianness) memory layout for all binary types (the least significant byte (LSB) of data type is placed at the lowest memory address).
## Data types
The following binary types are used for logging. They all correspond to the types in C.
| Type | Size in Bytes |
| ----------------- | ------------- |
| int8_t, uint8_t | 1 |
| int16_t, uint16_t | 2 |
| int32_t, uint32_t | 4 |
| int64_t, uint64_t | 8 |
| float | 4 |
| double | 8 |
| bool, char | 1 |
Additionally the types can be used as a fixed-size array: e.g. `float[5]`.
Strings (`char[length]`) do not contain the termination NULL character `'\0'` at the end.
::: info
String comparisons are case sensitive, which should be taken into account when comparing message names when [adding subscriptions](#a-subscription-message).
:::
## ULog File Structure
ULog files have the following three sections:
```
----------------------
| Header |
----------------------
| Definitions |
----------------------
| Data |
----------------------
```
A description of each section is provided below.
### Header Section
The header is a fixed-size section and has the following format (16 bytes):
```plain
----------------------------------------------------------------------
| 0x55 0x4c 0x6f 0x67 0x01 0x12 0x35 | 0x01 | uint64_t |
| File magic (7B) | Version (1B) | Timestamp (8B) |
----------------------------------------------------------------------
```
- **File Magic (7 Bytes):** File type indicator that reads "ULogXYZ where XYZ is the magic bytes sequence `0x01 0x12 0x35`"
- **Version (1 Byte):** File format version (currently 1)
- **Timestamp (8 Bytes):** `uint64_t` integer that denotes when the logging started in microseconds.
### Definition & Data Section Message Header
The _Definitions and Data_ sections contain a number of **messages**. Each message is preceded by this header:
```c
struct message_header_s {
uint16_t msg_size;
uint8_t msg_type;
};
```
- `msg_size` is the size of the message in bytes without the header.
- `msg_type` defines the content, and is a single byte.
::: info
Message sections below are prefixed with the character that corresponds to it's `msg_type`.
:::
### Definitions Section
The definitions section contains basic information such as software version, message format, initial parameter values, and so on.
The message types in this section are:
1. [Flag Bits](#b-flag-bits-message)
2. [Format Definition](#f-format-message)
3. [Information](#i-information-message)
4. [Multi Information](#m-multi-information-message)
5. [Parameter](#p-parameter-message)
6. [Default Parameter](#q-default-parameter-message)
#### 'B': Flag Bits Message
::: info
This message must be the **first message** right after the header section, so that it has a fixed constant offset from the start of the file!
:::
This message provides information to the log parser whether the log is parsable or not.
```c
struct ulog_message_flag_bits_s {
struct message_header_s header; // msg_type = 'B'
uint8_t compat_flags[8];
uint8_t incompat_flags[8];
uint64_t appended_offsets[3]; // file offset(s) for appended data if appending bit is set
};
```
- `compat_flags`: compatible flag bits
- These flags indicate the presence of features in the log file that are compatible with any ULog parser.
- `compat_flags[0]`: _DEFAULT_PARAMETERS_ (Bit 0): if set, the log contains [default parameters message](#q-default-parameter-message)
The rest of the bits are currently not defined and must be set to 0.
These bits can be used for future ULog changes that are compatible with existing parsers.
For example, adding a new message type can be indicated by defining a new bit in the standard, and existing parsers will ignore the new message type.
It means parsers can just ignore the bits if one of the unknown bits is set.
- `incompat_flags`: incompatible flag bits.
- `incompat_flags[0]`: _DATA_APPENDED_ (Bit 0): if set, the log contains appended data and at least one of the `appended_offsets` is non-zero.
The rest of the bits are currently not defined and must be set to 0.
This can be used to introduce breaking changes that existing parsers cannot handle. For example, when an old ULog parser that didn't have the concept of _DATA_APPENDED_ reads the newer ULog, it would stop parsing the log as the log will contain out-of-spec messages / concepts.
If a parser finds any of these bits set that isn't specified, it must refuse to parse the log.
- `appended_offsets`: File offset (0-based) for appended data.
If no data is appended, all offsets must be zero.
This can be used to reliably append data for logs that may stop in the middle of a message.
For example, crash dumps.
A process appending data should do:
- set the relevant `incompat_flags` bit
- set the first `appended_offsets` that is currently 0 to the length of the log file without the appended data, as that is where the new data will start
- append any type of messages that are valid for the Data section.
It is possible that there are more fields appended at the end of this message in future ULog specifications.
This means a parser must not assume a fixed length of this message.
If the `msg_size` is bigger than expected (currently 40), any additional bytes must be ignored/discarded.
#### 'F': Format Message
Format message defines a single message name and its inner fields in a single string.
```c
struct message_format_s {
struct message_header_s header; // msg_type = 'F'
char format[header.msg_size];
};
```
- `format` is a plain-text string with the following format: `message_name:field0;field1;`
- There can be an arbitrary amount of fields (minimum 1), separated by `;`.
- `message_name`: an arbitrary non-empty string with these allowed characters: `a-zA-Z0-9_-/` (and different from any of the [basic types](#data-types)).
A `field` has the format: `type field_name`, or for an array: `type[array_length] field_name` is used (only fixed size arrays are supported).
`field_name` must consist of the characters in the set `a-zA-Z0-9_`.
A `type` is one of the [basic binary types](#data-types) or a `message_name` of another format definition (nested usage).
- A type can be used before it's defined.
- e.g. The message `MessageA:MessageB[2] msg_b` can come before the `MessageB:uint_8[3] data`
- There can be arbitrary nesting but **no circular dependencies**
- e.g. `MessageA:MessageB[2] msg_b` & `MessageB:MessageA[4] msg_a`
Some field names are special:
- `timestamp`: every message format with a [Subscription Message](#a-subscription-message) must include a timestamp field (for example a message format only used as part of a nested definition by another format may not include a timestamp field)
- Its type must be `uint64_t`.
- The unit is microseconds.
- The timestamp must always be monotonic increasing for a message series with the same `msg_id` (same subscription).
- `_padding{}`: field names that start with `_padding` (e.g. `_padding[3]`) should not be displayed and their data must be ignored by a reader.
- These fields can be inserted by a writer to ensure correct alignment.
- If the padding field is the last field, then this field may not be logged, to avoid writing unnecessary data.
- This means the `message_data_s.data` will be shorter by the size of the padding.
- However the padding is still needed when the message is used in a nested definition.
- In general, message fields are not necessarily aligned (i.e. the field offset within the message is not necessarily a multiple of its data size), so a reader must always use appropriate memory copy methods to access individual fields.
#### 'I': Information Message
The Information message defines a dictionary type definition `key` : `value` pair for any information, including but not limited to Hardware version, Software version, Build toolchain for the software, etc.
```c
struct ulog_message_info_header_s {
struct message_header_s header; // msg_type = 'I'
uint8_t key_len;
char key[key_len];
char value[header.msg_size-1-key_len]
};
```
- `key_len`: Length of the key value
- `key`: Contains the key string in the form`type name`, e.g. `char[value_len] sys_toolchain_ver`. Valid characters for the name: `a-zA-Z0-9_-/`. The type may be one of the [basic types including arrays](#data-types).
- `value`: Contains the data (with the length `value_len`) corresponding to the `key` e.g. `9.4.0`.
::: info
A key defined in the Information message must be unique. Meaning there must not be more than one definition with the same key value.
:::
Parsers can store information messages as a dictionary.
Predefined information messages are:
| key | Description | Example for value |
| ----------------------------------- | ------------------------------------------- | ------------------ |
| `char[value_len] sys_name` | Name of the system | "PX4" |
| `char[value_len] ver_hw` | Hardware version (board) | "PX4FMU_V4" |
| `char[value_len] ver_hw_subtype` | Board subversion (variation) | "V2" |
| `char[value_len] ver_sw` | Software version (git tag) | "7f65e01" |
| `char[value_len] ver_sw_branch` | git branch | "master" |
| `uint32_t ver_sw_release` | Software version (see below) | 0x010401ff |
| `char[value_len] sys_os_name` | Operating System Name | "Linux" |
| `char[value_len] sys_os_ve`r | OS version (git tag) | "9f82919" |
| `uint32_t ver_os_release` | OS version (see below) | 0x010401ff |
| `char[value_len] sys_toolchain` | Toolchain Name | "GNU GCC" |
| `char[value_len] sys_toolchain_ver` | Toolchain Version | "6.2.1" |
| `char[value_len] sys_mcu` | Chip name and revision | "STM32F42x, rev A" |
| `char[value_len] sys_uuid` | Unique identifier for vehicle (eg. MCU ID) | "392a93e32fa3"... |
| `char[value_len] log_type` | Type of the log (full log if not specified) | "mission" |
| `char[value_len] replay` | File name of replayed log if in replay mode | "log001.ulg" |
| `int32_t time_ref_utc` | UTC Time offset in seconds | -3600 |
::: info
`value_len` represents the data size of the `value`. This is described in the `key`.
:::
- The format of `ver_sw_release` and `ver_os_release` is: 0xAABBCCTT, where AA is **major**, BB is **minor**, CC is patch and TT is the **type**.
- **Type** is defined as following: `>= 0`: development, `>= 64`: alpha version, `>= 128`: beta version, `>= 192`: RC version, `== 255`: release version.
- For example, `0x010402FF` translates into the release version v1.4.2.
This message can also be used in the Data section (this is however the preferred section).
#### 'M': Multi Information Message
Multi information message serves the same purpose as the information message, but for long messages or multiple messages with the same key.
```c
struct ulog_message_info_multiple_header_s {
struct message_header_s header; // msg_type = 'M'
uint8_t is_continued; // can be used for arrays
uint8_t key_len;
char key[key_len];
char value[header.msg_size-2-key_len]
};
```
- `is_continued` can be used for split-up messages: if set to 1, it is part of the previous message with the same key.
Parsers can store all information multi messages as a 2D list, using the same order as the messages occur in the log.
Valid names and types are the same as for the Information message.
#### 'P': Parameter Message
Parameter message in the _Definitions_ section defines the parameter values of the vehicle when logging is started. It uses the same format as the [Information Message](#i-information-message).
```c
struct message_info_s {
struct message_header_s header; // msg_type = 'P'
uint8_t key_len;
char key[key_len];
char value[header.msg_size-1-key_len]
};
```
If a parameter dynamically changes during runtime, this message can also be [used in the Data section](#messages-shared-with-the-definitions-section) as well.
The data type is restricted to `int32_t` and `float`. Valid characters for the name: `a-zA-Z0-9_-/`.
#### 'Q': Default Parameter Message
The default parameter message defines the default value of a parameter for a given vehicle and setup.
```c
struct ulog_message_parameter_default_header_s {
struct message_header_s header; // msg_type = 'Q'
uint8_t default_types;
uint8_t key_len;
char key[key_len];
char value[header.msg_size-2-key_len]
};
```
- `default_types` is a bitfield and defines to which group(s) the value belongs to.
- At least one bit must be set:
- `1<<0`: system wide default
- `1<<1`: default for the current configuration (e.g. an airframe)
A log may not contain default values for all parameters.
In those cases the default is equal to the parameter value, and different default types are treated independently.
This message can also be used in the Data section, and the same data type and naming applies as for the Parameter message.
This section ends before the start of the first [Subscription Message](#a-subscription-message) or [Logging](#l-logged-string-message) message, whichever comes first.
### Data Section
The message types in the _Data_ section are:
1. [Subscription](#a-subscription-message)
2. [Unsubscription](#r-unsubscription-message)
3. [Logged Data](#d-logged-data-message)
4. [Logged String](#l-logged-string-message)
5. [Tagged Logged String](#c-tagged-logged-string-message)
6. [Synchronization](#s-synchronization-message)
7. [Dropout Mark](#o-dropout-message)
8. [Information](#i-information-message)
9. [Multi Information](#m-multi-information-message)
10. [Parameter](#p-parameter-message)
11. [Default Parameter](#q-default-parameter-message)
#### `A`: Subscription Message
Subscribe a message by name and give it an id that is used in [Logged data Message](#d-logged-data-message).
This must come before the first corresponding [Logged data Message](#d-logged-data-message).
```c
struct message_add_logged_s {
struct message_header_s header; // msg_type = 'A'
uint8_t multi_id;
uint16_t msg_id;
char message_name[header.msg_size-3];
};
```
- `multi_id`: the same message format can have multiple instances, for example if the system has two sensors of the same type. The default and first instance must be 0.
- `msg_id`: unique id to match [Logged data Message](#d-logged-data-message) data. The first use must set this to 0, then increase it.
- The same `msg_id` must not be used twice for different subscriptions.
- `message_name`: message name to subscribe to.
Must match one of the [Format Message](#f-format-message) definitions.
#### `R`: Unsubscription Message
Unsubscribe a message, to mark that it will not be logged anymore (not used currently).
```c
struct message_remove_logged_s {
struct message_header_s header; // msg_type = 'R'
uint16_t msg_id;
};
```
#### 'D': Logged Data Message
```c
struct message_data_s {
struct message_header_s header; // msg_type = 'D'
uint16_t msg_id;
uint8_t data[header.msg_size-2];
};
```
- `msg_id`: as defined by a [Subscription Message](#a-subscription-message)
- `data` contains the logged binary message as defined by [Format Message](#f-format-message)
See above for special treatment of padding fields.
#### 'L': Logged String Message
Logged string message, i.e. `printf()` output.
```c
struct message_logging_s {
struct message_header_s header; // msg_type = 'L'
uint8_t log_level;
uint64_t timestamp;
char message[header.msg_size-9]
};
```
- `timestamp`: in microseconds
- `log_level`: same as in the Linux kernel:
| Name | Level value | Meaning |
| ------- | ----------- | -------------------------------- |
| EMERG | '0' | System is unusable |
| ALERT | '1' | Action must be taken immediately |
| CRIT | '2' | Critical conditions |
| ERR | '3' | Error conditions |
| WARNING | '4' | Warning conditions |
| NOTICE | '5' | Normal but significant condition |
| INFO | '6' | Informational |
| DEBUG | '7' | Debug-level messages |
#### 'C': Tagged Logged String Message
```c
struct message_logging_tagged_s {
struct message_header_s header; // msg_type = 'C'
uint8_t log_level;
uint16_t tag;
uint64_t timestamp;
char message[header.msg_size-11]
};
```
- `tag`: id representing source of logged message string. It could represent a process, thread or a class depending upon the system architecture.
- For example, a reference implementation for an onboard computer running multiple processes to control different payloads, external disks, serial devices etc can encode these process identifiers using a `uint16_t enum` into the `tag` attribute of struct as follows:
```c
enum class ulog_tag : uint16_t {
unassigned,
mavlink_handler,
ppk_handler,
camera_handler,
ptp_handler,
serial_handler,
watchdog,
io_service,
cbuf,
ulg
};
```
- `timestamp`: in microseconds
- `log_level`: same as in the Linux kernel:
| Name | Level value | Meaning |
| ------- | ----------- | -------------------------------- |
| EMERG | '0' | System is unusable |
| ALERT | '1' | Action must be taken immediately |
| CRIT | '2' | Critical conditions |
| ERR | '3' | Error conditions |
| WARNING | '4' | Warning conditions |
| NOTICE | '5' | Normal but significant condition |
| INFO | '6' | Informational |
| DEBUG | '7' | Debug-level messages |
#### 'S': Synchronization message
Synchronization message so that a reader can recover from a corrupt message by searching for the next sync message.
```c
struct message_sync_s {
struct message_header_s header; // msg_type = 'S'
uint8_t sync_magic[8];
};
```
- `sync_magic`: [0x2F, 0x73, 0x13, 0x20, 0x25, 0x0C, 0xBB, 0x12]
#### 'O': Dropout message
Mark a dropout (lost logging messages) of a given duration in ms.
Dropouts can occur e.g. if the device is not fast enough.
```c
struct message_dropout_s {
struct message_header_s header; // msg_type = 'O'
uint16_t duration;
};
```
#### Messages shared with the Definitions Section
Since the Definitions and Data Sections use the same message header format, they also share the same messages listed below:
- [Information Message](#i-information-message).
- [Multi Information Message](#m-multi-information-message)
- [Parameter Message](#p-parameter-message)
- For the _Data_ section, the Parameter Message is used when the parameter value changes
- [Default Parameter Message](#q-default-parameter-message)
## Requirements for Parsers
A valid ULog parser must fulfill the following requirements:
- Must ignore unknown messages (but it can print a warning)
- Parse future/unknown file format versions as well (but it can print a warning).
- Must refuse to parse a log which contains unknown incompatibility bits set (`incompat_flags` of [Flag Bits Message](#b-flag-bits-message)), meaning the log contains breaking changes that the parser cannot handle.
- A parser must be able to correctly handle logs that end abruptly, in the middle of a message.
The unfinished message should just be discarded.
- For appended data: a parser can assume the Data section exists, i.e. the offset points to a place after the Definitions section.
- Appended data must be treated as if it was part of the regular Data section.
## Known Parser Implementations
- PX4-Autopilot: C++
- [logger module](https://github.com/PX4/PX4-Autopilot/tree/main/src/modules/logger)
- [replay module](https://github.com/PX4/PX4-Autopilot/tree/main/src/modules/replay)
- [hardfault_log module](https://github.com/PX4/PX4-Autopilot/tree/main/src/systemcmds/hardfault_log): append hardfault crash data.
- [pyulog](https://github.com/PX4/pyulog): python, ULog reader and writer library with CLI scripts.
- [ulog_cpp](https://github.com/PX4/ulog_cpp): C++, ULog reader and writer library.
- [FlightPlot](https://github.com/PX4/FlightPlot): Java, log plotter.
- [MAVLink](https://github.com/mavlink/mavlink): Messages for ULog streaming via MAVLink (note that appending data is not supported, at least not for cut off messages).
- [QGroundControl](https://github.com/mavlink/qgroundcontrol): C++, ULog streaming via MAVLink and minimal parsing for GeoTagging.
- [mavlink-router](https://github.com/01org/mavlink-router): C++, ULog streaming via MAVLink.
- [MAVGAnalysis](https://github.com/ecmnet/MAVGCL): Java, ULog streaming via MAVLink and parser for plotting and analysis.
- [PlotJuggler](https://github.com/facontidavide/PlotJuggler): C++/Qt application to plot logs and time series. Supports ULog since version 2.1.3.
- [ulogreader](https://github.com/maxsun/ulogreader): Javascript, ULog reader and parser outputs log in JSON object format.
- [Foxglove Studio](https://github.com/foxglove/studio): an integrated visualization and diagnosis tool for robotics
(Typescript ULog parser: https://github.com/foxglove/ulog).
## File Format Version History
### Changes in version 2
- Addition of [Multi Information Message](#m-multi-information-message) and [Flag Bits Message](#b-flag-bits-message) and the ability to append data to a log.
- This is used to add crash data to an existing log.
- If data is appended to a log that is cut in the middle of a message, it cannot be parsed with version 1 parsers.
- Other than that forward and backward compatibility is given if parsers ignore unknown messages.