docs: update fuzz tests (split into separate page)

This commit is contained in:
Hamish Willee 2025-06-19 09:24:37 +10:00 committed by Beat Küng
parent 3cfaaf73b4
commit 572a06b2f3
3 changed files with 75 additions and 47 deletions

View File

@ -813,6 +813,7 @@
- [Test MC_04 - Failsafe Testing](test_cards/mc_04_failsafe_testing.md)
- [Test MC_05 - Indoor Flight (Manual Modes)](test_cards/mc_05_indoor_flight_manual_modes.md)
- [Unit Tests](test_and_ci/unit_tests.md)
- [Fuzz Tests](test_and_ci/fuzz_tests.md)
- [Continuous Integration](test_and_ci/continous_integration.md)
- [Integration Testing](test_and_ci/integration_testing.md)
- [MAVSDK Integration Testing](test_and_ci/integration_testing_mavsdk.md)

View File

@ -0,0 +1,70 @@
# Fuzz Tests
Fuzz tests are a generalised form of [unit test](../test_and_ci/unit_tests.md) that run code against a large number of random inputs.
This helps to ensure that the code is robust against any input, not just those expected by the developer.
They can be written like unit tests with possible assertions (`EXPECT_EQ`, etc), and have a set of input parameters.
The fuzzer then tries to find inputs that cause the code to crash (with [Address Sanitizer](https://clang.llvm.org/docs/AddressSanitizer.html) enabled automatically) or trigger an assertion.
The tests are run as part of normal unit tests, and in a more comprehensive fuzzing mode test.
For more information see [Running Fuzz Tests](#running-fuzz-tests) below.
## Writing a Fuzz Test
To write a fuzz test:
1. Start by writing a "normal" [functional test](../test_and_ci/unit_tests.md#functional-test).
1. Make sure the file name contains `fuzz` (lower case).
For example `my_driver_fuzz_tests.cpp`.
1. Add one or more fuzz tests to the file.
For example:
```cpp
#include <gtest/gtest.h>
#include <fuzztest/fuzztest.h>
void myDriverNeverCrashes(const std::string& s) {
MyDriver driver;
driver.handleInput(s);
}
FUZZ_TEST(MyDriverFuzzTests, myDriverNeverCrashes);
```
The file can also contain normal tests.
For more information, see https://github.com/google/fuzztest.
A complete example in the PX4 repository can be found in the [septentrio driver](https://github.com/PX4/PX4-Autopilot/blob/main/src/drivers/gnss/septentrio/septentrio_fuzz_tests.cpp).
## Running Fuzz Tests
Fuzz tests can be run in two modes:
- As part of normal unit tests with `make tests`.
This will only create a small number of inputs and not use coverage information.
- In fuzzing mode: this runs a single fuzz test with coverage information over a longer period of time (either fixed or indefinitely).
The fuzzer will try to find inputs that cover all reachable code paths.
It requires compilation with Clang and can be run with the following commands:
```sh
rm -rf build/px4_sitl_tests
export CC=clang
export CXX=clang++
make tests TESTFILTER=__no_tests__
cd build/px4_sitl_tests
./functional-<my-test> --fuzz=<test-name>
```
## Seeds
Depending on the code complexity, it might be hard for the fuzzer to find inputs that pass certain conditions.
For this it is possible to provide one or more seeds, which the fuzzer will use as first inputs.
[The google documentation](https://github.com/google/fuzztest/blob/main/doc/fuzz-test-macro.md#initial-seeds-initial-seeds) contains more details.
You can also use the `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` macro for conditional code compilation, for example to exclude CRC checks.
More information about efficient fuzzing can be found on [this page](https://chromium.googlesource.com/chromium/src/+/main/testing/libfuzzer/efficient_fuzzing.md).
## CI
Fuzz tests are run as part of the normal unit tests in CI for each pull request.
In addition, the fuzz tests are run daily for 15 minutes on the main branch.

View File

@ -23,7 +23,7 @@ The steps to create new unit tests are as follows:
Tests can be run via `make tests`, after which you will find the binary in `build/px4_sitl_test/unit-MyNewUnit`.
It can be run directly in a debugger.
## Writing a GTest Functional Test{#functional-test}
## Writing a GTest Functional Test {#functional-test}
GTest functional tests should be used when the test or the components being tested depend on parameters, uORB messages and/or advanced GTest functionality.
Additionally, functional tests can contain local usage of STL data structures (although be careful of platform differences between e.g. macOS and Linux).
@ -172,51 +172,8 @@ For example:
- `make tests TESTFILTER=Attitude` only run the `AttitudeControl` test
## Fuzz Testing
Fuzz tests are a generalized form of unit tests that are run on a large number of inputs to ensure the code is robust against any input.
They can be written like unit tests with possible assertions (`EXPECT_EQ`, etc), and have a set of input parameters.
The fuzzer then tries to find inputs that cause the code to crash (with enabled Address Sanitizer) or trigger an assertion.
### Writing a Fuzz Test
1. Start by writing a [functional test](#functional-test).
1. Make sure the file name contains `fuzz` (lower case). For example `my_driver_fuzz_tests.cpp`.
1. Add one or more fuzz tests to the file. The file can also contain normal tests.
For example:
```cpp
#include <gtest/gtest.h>
#include <fuzztest/fuzztest.h>
Fuzz tests are a generalised form of unit test that ensures code is robust against any input.
They are run as part of the unit tests, and also more extensively in their own testing mode.
void myDriverNeverCrashes(const std::string& s) {
MyDriver driver;
driver.handleInput(s);
}
FUZZ_TEST(MyDriverFuzzTests, myDriverNeverCrashes);
```
For further details, see https://github.com/google/fuzztest.
### Running a Fuzz Test
Fuzz tests can be run in two modes:
- As part of normal unit tests with `make tests`. This will only create a small number of inputs and not use coverage information.
- In fuzzing mode: this runs a single fuzz test with coverage information over a longer period of time (either fixed or indefinitely).
The fuzzer will try to find inputs that cover all reachable code paths.
It requires compilation with Clang and can be run with the following commands:
```sh
rm -rf build/px4_sitl_tests
export CC=clang
export CXX=clang++
make tests TESTFILTER=__no_tests__
cd build/px4_sitl_tests
./functional-<my-test> --fuzz=<test-name>
```
### Seeds
Depending on the code complexity, it might be hard for the fuzzer to find inputs that pass certain conditions.
For this it is possible to provide one or more seeds, which the fuzzer will use as first inputs.
[The documentation](https://github.com/google/fuzztest/blob/main/doc/fuzz-test-macro.md#initial-seeds-initial-seeds) contains more details.
You can also use the `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` macro for conditional code compilation, for example to exclude CRC checks.
More information about efficient fuzzing can be found on [this page](https://chromium.googlesource.com/chromium/src/+/main/testing/libfuzzer/efficient_fuzzing.md).
### CI
Fuzz tests are run as part of the normal unit tests in CI for each pull request.
In addition, the fuzz tests are run daily for 15 minutes on the main branch.
For more information see [Fuzz Tests](../test_and_ci/fuzz_tests.md).