diff --git a/docs/en/SUMMARY.md b/docs/en/SUMMARY.md index 38c23bda8d..e5d59c6ff7 100644 --- a/docs/en/SUMMARY.md +++ b/docs/en/SUMMARY.md @@ -813,6 +813,7 @@ - [Test MC_04 - Failsafe Testing](test_cards/mc_04_failsafe_testing.md) - [Test MC_05 - Indoor Flight (Manual Modes)](test_cards/mc_05_indoor_flight_manual_modes.md) - [Unit Tests](test_and_ci/unit_tests.md) + - [Fuzz Tests](test_and_ci/fuzz_tests.md) - [Continuous Integration](test_and_ci/continous_integration.md) - [Integration Testing](test_and_ci/integration_testing.md) - [MAVSDK Integration Testing](test_and_ci/integration_testing_mavsdk.md) diff --git a/docs/en/test_and_ci/fuzz_tests.md b/docs/en/test_and_ci/fuzz_tests.md new file mode 100644 index 0000000000..6503409f62 --- /dev/null +++ b/docs/en/test_and_ci/fuzz_tests.md @@ -0,0 +1,70 @@ +# Fuzz Tests + +Fuzz tests are a generalised form of [unit test](../test_and_ci/unit_tests.md) that run code against a large number of random inputs. +This helps to ensure that the code is robust against any input, not just those expected by the developer. + +They can be written like unit tests with possible assertions (`EXPECT_EQ`, etc), and have a set of input parameters. +The fuzzer then tries to find inputs that cause the code to crash (with [Address Sanitizer](https://clang.llvm.org/docs/AddressSanitizer.html) enabled automatically) or trigger an assertion. + +The tests are run as part of normal unit tests, and in a more comprehensive fuzzing mode test. +For more information see [Running Fuzz Tests](#running-fuzz-tests) below. + +## Writing a Fuzz Test + +To write a fuzz test: + +1. Start by writing a "normal" [functional test](../test_and_ci/unit_tests.md#functional-test). +1. Make sure the file name contains `fuzz` (lower case). + For example `my_driver_fuzz_tests.cpp`. +1. Add one or more fuzz tests to the file. + For example: + + ```cpp + #include + #include + + void myDriverNeverCrashes(const std::string& s) { + MyDriver driver; + driver.handleInput(s); + } + FUZZ_TEST(MyDriverFuzzTests, myDriverNeverCrashes); + ``` + +The file can also contain normal tests. +For more information, see https://github.com/google/fuzztest. + +A complete example in the PX4 repository can be found in the [septentrio driver](https://github.com/PX4/PX4-Autopilot/blob/main/src/drivers/gnss/septentrio/septentrio_fuzz_tests.cpp). + +## Running Fuzz Tests + +Fuzz tests can be run in two modes: + +- As part of normal unit tests with `make tests`. + This will only create a small number of inputs and not use coverage information. +- In fuzzing mode: this runs a single fuzz test with coverage information over a longer period of time (either fixed or indefinitely). + The fuzzer will try to find inputs that cover all reachable code paths. + It requires compilation with Clang and can be run with the following commands: + + ```sh + rm -rf build/px4_sitl_tests + export CC=clang + export CXX=clang++ + make tests TESTFILTER=__no_tests__ + cd build/px4_sitl_tests + ./functional- --fuzz= + ``` + +## Seeds + +Depending on the code complexity, it might be hard for the fuzzer to find inputs that pass certain conditions. +For this it is possible to provide one or more seeds, which the fuzzer will use as first inputs. +[The google documentation](https://github.com/google/fuzztest/blob/main/doc/fuzz-test-macro.md#initial-seeds-initial-seeds) contains more details. + +You can also use the `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` macro for conditional code compilation, for example to exclude CRC checks. + +More information about efficient fuzzing can be found on [this page](https://chromium.googlesource.com/chromium/src/+/main/testing/libfuzzer/efficient_fuzzing.md). + +## CI + +Fuzz tests are run as part of the normal unit tests in CI for each pull request. +In addition, the fuzz tests are run daily for 15 minutes on the main branch. diff --git a/docs/en/test_and_ci/unit_tests.md b/docs/en/test_and_ci/unit_tests.md index 0b6262354f..250a381c11 100644 --- a/docs/en/test_and_ci/unit_tests.md +++ b/docs/en/test_and_ci/unit_tests.md @@ -23,7 +23,7 @@ The steps to create new unit tests are as follows: Tests can be run via `make tests`, after which you will find the binary in `build/px4_sitl_test/unit-MyNewUnit`. It can be run directly in a debugger. -## Writing a GTest Functional Test{#functional-test} +## Writing a GTest Functional Test {#functional-test} GTest functional tests should be used when the test or the components being tested depend on parameters, uORB messages and/or advanced GTest functionality. Additionally, functional tests can contain local usage of STL data structures (although be careful of platform differences between e.g. macOS and Linux). @@ -172,51 +172,8 @@ For example: - `make tests TESTFILTER=Attitude` only run the `AttitudeControl` test ## Fuzz Testing -Fuzz tests are a generalized form of unit tests that are run on a large number of inputs to ensure the code is robust against any input. -They can be written like unit tests with possible assertions (`EXPECT_EQ`, etc), and have a set of input parameters. -The fuzzer then tries to find inputs that cause the code to crash (with enabled Address Sanitizer) or trigger an assertion. -### Writing a Fuzz Test -1. Start by writing a [functional test](#functional-test). -1. Make sure the file name contains `fuzz` (lower case). For example `my_driver_fuzz_tests.cpp`. -1. Add one or more fuzz tests to the file. The file can also contain normal tests. - For example: - ```cpp - #include - #include +Fuzz tests are a generalised form of unit test that ensures code is robust against any input. +They are run as part of the unit tests, and also more extensively in their own testing mode. - void myDriverNeverCrashes(const std::string& s) { - MyDriver driver; - driver.handleInput(s); - } - FUZZ_TEST(MyDriverFuzzTests, myDriverNeverCrashes); - ``` - For further details, see https://github.com/google/fuzztest. - -### Running a Fuzz Test -Fuzz tests can be run in two modes: -- As part of normal unit tests with `make tests`. This will only create a small number of inputs and not use coverage information. -- In fuzzing mode: this runs a single fuzz test with coverage information over a longer period of time (either fixed or indefinitely). - The fuzzer will try to find inputs that cover all reachable code paths. - It requires compilation with Clang and can be run with the following commands: - ```sh - rm -rf build/px4_sitl_tests - export CC=clang - export CXX=clang++ - make tests TESTFILTER=__no_tests__ - cd build/px4_sitl_tests - ./functional- --fuzz= - ``` - -### Seeds -Depending on the code complexity, it might be hard for the fuzzer to find inputs that pass certain conditions. -For this it is possible to provide one or more seeds, which the fuzzer will use as first inputs. -[The documentation](https://github.com/google/fuzztest/blob/main/doc/fuzz-test-macro.md#initial-seeds-initial-seeds) contains more details. - -You can also use the `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` macro for conditional code compilation, for example to exclude CRC checks. - -More information about efficient fuzzing can be found on [this page](https://chromium.googlesource.com/chromium/src/+/main/testing/libfuzzer/efficient_fuzzing.md). - -### CI -Fuzz tests are run as part of the normal unit tests in CI for each pull request. -In addition, the fuzz tests are run daily for 15 minutes on the main branch. +For more information see [Fuzz Tests](../test_and_ci/fuzz_tests.md).