Compare commits

...

9 Commits

Author SHA1 Message Date
Ramon Roche 3dc58a92a8 Merge branch 'main' into pr-logger-small-flash 2026-04-11 10:04:13 -06:00
Ramon Roche 79c43df91c Merge branch 'main' into pr-logger-small-flash 2026-04-10 21:04:42 -06:00
Julian Oes 388dbc24b2 fix(boards): no SD card on this board 2026-04-11 06:39:01 +12:00
Julian Oes 3a378cd86d fix(boards): no SD card on this board 2026-04-11 06:38:04 +12:00
Julian Oes 6fbbd13554 feat(docs): update release notes for logger rework
Add Debug & Logging entries and Read Before Upgrading notes for the
SDLOG_ROTATE addition, SDLOG_DIRS_MAX removal, the SDLOG_MAX_SIZE
default change (4095 -> 1024 MB), small-flash storage support, and
the new mklittlefs systemcmd.
2026-04-11 06:36:07 +12:00
Julian Oes 1a96bd2c94 feat(logger): add SDLOG_ROTATE, disentangle cleanup thresholds
The previous cleanup logic conflated two independent concerns into
SDLOG_MAX_SIZE: the maximum size of a single log file (rotation
trigger) AND the minimum free space to maintain. That was broken:
the 4095 MB default meant "keep 4 GB free", which over-cleaned on
large SD cards and was impossible to satisfy on small flash.

Disentangle the two:

- SDLOG_ROTATE (new, int %, default 90): maximum disk usage percentage.
  Cleanup guarantees at least (100 - SDLOG_ROTATE)% free even during
  writing of a new log file. Setting 0 disables space-based cleanup;
  100 allows filling the disk completely.
- SDLOG_MAX_SIZE (default lowered from 4095 to 1024): pure max file
  size. The value is added on top of the rotate-derived threshold as
  headroom for the next file write, so the rotate guarantee holds
  even mid-write.
- SDLOG_DIRS_MAX: removed. Directory count limits were confusing and
  orthogonal to the space-management goal; free-space cleanup alone
  covers the use case. Drop the param and all remaining overrides
  (rc.board_defaults, rcS, rc.replay, 1002_standard_vtol.hil).

Cleanup threshold is now:
    ((100 - SDLOG_ROTATE)% of disk) + SDLOG_MAX_SIZE

Small-flash boards can override SDLOG_MAX_SIZE to get more retained
logs within the available space. kakuteh7v2 and airbrainh743 drop
their SDLOG_DIRS_MAX overrides accordingly.

Update docs/en/dev_log/logging.md with the new semantics and a
worked example for the typical 8 GB SD case.
2026-04-11 06:35:58 +12:00
Julian Oes 7f76b71526 feat(boards): add W25N NAND flash support with littlefs
The kakuteh7mini ships with a W25N01GV (1Gbit/128MB) SPI NAND flash on
SPI1, but the board init was treating SPI1 as an MMC/SD slot and the
W25N driver was not enabled. Enable the chip and use it for logging:

- spi.cpp: register the device as SPIDEV_FLASH(0) instead of SPIDEV_MMCSD(0)
- init.c: initialize the W25N MTD driver, register /dev/mtd0, mount
  littlefs at /fs/flash with autoformat, and print the flash geometry
  on boot for verification.
- nuttx defconfig: enable CONFIG_MTD_W25N, CONFIG_FS_LITTLEFS,
  SPI1 DMA + DMAMUX1, drop the unused RAMTRON config.
- board_dma_map.h: define DMAMAP_SPI1_RX/TX for the SPI1 DMA channels.
- default.px4board: set CONFIG_BOARD_ROOT_PATH to /fs/flash.
- rc.board_defaults: drop the SDLOG_BACKEND=0 override that was
  disabling logging entirely and the COM_ARM_SDCARD override (the
  flash logging replaces the need for an SD card). Set SDLOG_MAX_SIZE=30
  so a few recent logs fit within the 128 MB flash.
2026-04-11 06:35:49 +12:00
Julian Oes 18493222ed feat(mklittlefs): add command to format littlefs filesystems
Add an NSH command to format a device with littlefs, analogous to
mkfatfs for FAT filesystems. The command unmounts the mount point,
then remounts with forceformat to format and mount in one step.

Enable the command on boards that use littlefs as primary storage
(airbrainh743 and kakuteh7v2), and document it in the airbrainh743
flight controller page as a recovery procedure for a corrupted
flash filesystem.

Usage: mklittlefs /dev/mtd0 /fs/flash
2026-04-11 05:59:27 +12:00
Julian Oes 5762d72004 feat(logger): add support for small flash storage
Move log cleanup from boot to log start, delete individual .ulg files
(oldest first) instead of entire directories, and add a max log file
size. This makes it practical to run the logger on small flash targets
(e.g. 128 MB W25N NAND) while still allowing logs to be downloaded via
MAVLink FTP before they are removed.

- Move cleanup from boot to log start so logs can be downloaded first.
- Delete individual .ulg files (oldest first) instead of whole dirs.
- Add SDLOG_MAX_SIZE parameter (default 4095 MiB, just under FAT32 limit).
- Prioritize directories from the naming scheme not currently in use
  (e.g. delete sess dirs first when using date dirs).
- Simplify get_log_time to rely on clock_gettime (GPS driver sets it).
- Safer string handling in directory parsing.
- Log elapsed cleanup time for diagnostics.
- Split pure parsing helpers into util_parse.{h,cpp} and add unit tests
  (loggerUtilTest.cpp). Move px4_add_unit_gtest outside the BUILD_TESTING
  guard for consistency with the rest of the tree; the macro already
  checks BUILD_TESTING internally.

Sponsored by CubePilot.
2026-04-11 05:58:49 +12:00
29 changed files with 1059 additions and 192 deletions
@@ -20,7 +20,6 @@ module: replay
ignore_others: false
EOF
param set SDLOG_DIRS_MAX 7
param set SDLOG_PROFILE 3
# apply all params before ekf starts, as some params cannot be changed after startup
-1
View File
@@ -172,7 +172,6 @@ param set-default -s MC_AT_EN 1
param set-default SDLOG_MODE 1
# enable default, estimator replay and vision/avoidance logging profiles
param set-default SDLOG_PROFILE 131
param set-default SDLOG_DIRS_MAX 7
param set-default TRIG_INTERFACE 3
@@ -56,8 +56,6 @@ param set-default NAV_DLL_ACT 2
param set-default RTL_DESCEND_ALT 10
param set-default RTL_RETURN_ALT 30
param set-default SDLOG_DIRS_MAX 7
param set-default VT_F_TRANS_THR 0.75
param set-default VT_TYPE 2
@@ -71,6 +71,7 @@ CONFIG_SYSTEMCMDS_HARDFAULT_LOG=y
CONFIG_SYSTEMCMDS_I2CDETECT=y
CONFIG_SYSTEMCMDS_LED_CONTROL=y
CONFIG_SYSTEMCMDS_MFT=y
CONFIG_SYSTEMCMDS_MKLITTLEFS=y
CONFIG_SYSTEMCMDS_MTD=y
CONFIG_SYSTEMCMDS_NSHTERM=y
CONFIG_SYSTEMCMDS_PARAM=y
@@ -12,6 +12,13 @@ param set-default CBRK_SUPPLY_CHK 894281
param set-default IMU_GYRO_RATEMAX 2000
# W25N NAND flash with littlefs (128 MB): larger buffer, auto-rotate
set LOGGER_BUF 32
param set-default SDLOG_DIRS_MAX 3
# W25N NAND flash with littlefs (128 MB): small log file size so we can keep
# a few recent logs. Default SDLOG_ROTATE=90 keeps at least 10% free during
# writing (no bad block management yet, so avoid hammering the flash near full).
param set-default SDLOG_MAX_SIZE 40
# Store missions in RAM
param set-default SYS_DM_BACKEND 1
# Ignore that there is no SD card
param set-default COM_ARM_SDCARD 0
@@ -1,5 +1,6 @@
CONFIG_BOARD_TOOLCHAIN="arm-none-eabi"
CONFIG_BOARD_ARCHITECTURE="cortex-m7"
CONFIG_BOARD_ROOT_PATH="/fs/flash"
CONFIG_BOARD_SERIAL_GPS1="/dev/ttyS3"
CONFIG_BOARD_SERIAL_TEL1="/dev/ttyS0"
CONFIG_BOARD_SERIAL_TEL2="/dev/ttyS1"
@@ -32,11 +32,13 @@ param set-default CBRK_BUZZER 782090
param set-default IMU_GYRO_RATEMAX 2000
# Store missions in RAM
param set-default SYS_DM_BACKEND 1
# Ignore that there is no SD card
param set-default COM_ARM_SDCARD 0
# Disable logging
param set-default SDLOG_BACKEND 0
# W25N NAND flash with littlefs (128 MB): small log file size so we can keep
# a few recent logs. Default SDLOG_ROTATE=90 keeps at least 10% free during
# writing (no bad block management yet, so avoid hammering the flash near full).
param set-default SDLOG_MAX_SIZE 30
# Store missions in RAM
param set-default SYS_DM_BACKEND 1
@@ -33,6 +33,9 @@
#pragma once
#define DMAMAP_SPI1_RX DMAMAP_DMA12_SPI1RX_0 /* DMA1 */
#define DMAMAP_SPI1_TX DMAMAP_DMA12_SPI1TX_0 /* DMA1 */
#define DMAMAP_SPI4_RX DMAMAP_DMA12_SPI4RX_1 /* DMA2 */
#define DMAMAP_SPI4_TX DMAMAP_DMA12_SPI4TX_1 /* DMA2 */
@@ -98,6 +98,10 @@ CONFIG_FDCLONE_STDIO=y
CONFIG_FS_BINFS=y
CONFIG_FS_CROMFS=y
CONFIG_FS_FAT=y
CONFIG_FS_LITTLEFS=y
CONFIG_FS_LITTLEFS_CACHE_SIZE_FACTOR=1
CONFIG_FS_LITTLEFS_PROGRAM_SIZE_FACTOR=1
CONFIG_FS_LITTLEFS_READ_SIZE_FACTOR=1
CONFIG_FS_FATTIME=y
CONFIG_FS_PROCFS=y
CONFIG_FS_PROCFS_INCLUDE_PROGMEM=y
@@ -126,7 +130,8 @@ CONFIG_MTD=y
CONFIG_MTD_BYTE_WRITE=y
CONFIG_MTD_PARTITION=y
CONFIG_MTD_PROGMEM=y
CONFIG_MTD_RAMTRON=y
# CONFIG_MTD_RAMTRON is not set
CONFIG_MTD_W25N=y
CONFIG_NAME_MAX=40
CONFIG_NSH_ARCHINIT=y
CONFIG_NSH_ARGCAT=y
@@ -135,7 +140,7 @@ CONFIG_NSH_CMDPARMS=y
CONFIG_NSH_CROMFSETC=y
CONFIG_NSH_LINELEN=128
CONFIG_NSH_MAXARGUMENTS=15
CONFIG_NSH_MMCSDSPIPORTNO=1
# CONFIG_NSH_MMCSDSPIPORTNO is not set
CONFIG_NSH_NESTDEPTH=8
CONFIG_NSH_QUOTE=y
CONFIG_NSH_ROMFSETC=y
@@ -148,9 +153,6 @@ CONFIG_PREALLOC_TIMERS=50
CONFIG_PRIORITY_INHERITANCE=y
CONFIG_PTHREAD_MUTEX_ROBUST=y
CONFIG_PTHREAD_STACK_MIN=512
CONFIG_RAMTRON_EMULATE_PAGE_SHIFT=5
CONFIG_RAMTRON_EMULATE_SECTOR_SHIFT=5
CONFIG_RAMTRON_SETSPEED=y
CONFIG_RAM_SIZE=245760
CONFIG_RAM_START=0x20010000
CONFIG_RAW_BINARY=y
@@ -188,6 +190,7 @@ CONFIG_STM32H7_BKPSRAM=y
CONFIG_STM32H7_DMA1=y
CONFIG_STM32H7_DMA2=y
CONFIG_STM32H7_DMACAPABLE=y
CONFIG_STM32H7_DMAMUX1=y
CONFIG_STM32H7_FLOWCONTROL_BROKEN=y
CONFIG_STM32H7_I2C1=y
CONFIG_STM32H7_I2C_DYNTIMEO=y
@@ -201,6 +204,8 @@ CONFIG_STM32H7_SDMMC1=y
CONFIG_STM32H7_SERIALBRK_BSDCOMPAT=y
CONFIG_STM32H7_SERIAL_DISABLE_REORDERING=y
CONFIG_STM32H7_SPI1=y
CONFIG_STM32H7_SPI1_DMA=y
CONFIG_STM32H7_SPI1_DMA_BUFFER=4096
CONFIG_STM32H7_SPI2=y
CONFIG_STM32H7_SPI4=y
CONFIG_STM32H7_SPI4_DMA=y
@@ -244,4 +249,5 @@ CONFIG_USBDEV=y
CONFIG_USBDEV_BUSPOWERED=y
CONFIG_USBDEV_MAXPOWER=500
CONFIG_USEC_PER_TICK=1000
CONFIG_W25N_SPIFREQUENCY=104000000
CONFIG_WATCHDOG=y
+51 -6
View File
@@ -59,6 +59,8 @@
#include <nuttx/spi/spi.h>
#include <nuttx/analog/adc.h>
#include <nuttx/mm/gran.h>
#include <nuttx/mtd/mtd.h>
#include <nuttx/fs/fs.h>
#include <chip.h>
#include <stm32_uart.h>
#include <arch/board/board.h>
@@ -231,15 +233,58 @@ __EXPORT int board_app_initialize(uintptr_t arg)
led_on(LED_RED);
}
// MARK: this will *not* work as the minis have a W25N NAND flash chip
/* Get the SPI port for the microSD slot */
struct spi_dev_s *spi_dev = stm32_spibus_initialize(CONFIG_NSH_MMCSDSPIPORTNO);
#ifdef CONFIG_MTD_W25N
/* Initialize W25N01GV NAND Flash on SPI1 */
struct spi_dev_s *spi1 = stm32_spibus_initialize(1);
if (!spi_dev) {
syslog(LOG_ERR, "[boot] FAILED to initialize SPI port %d\n", CONFIG_NSH_MMCSDSPIPORTNO);
led_on(LED_BLUE);
if (!spi1) {
syslog(LOG_ERR, "[boot] FAILED to initialize SPI1 for W25N\n");
led_on(LED_RED);
} else {
struct mtd_dev_s *mtd = w25n_initialize(spi1, 0);
if (!mtd) {
syslog(LOG_ERR, "[boot] FAILED to initialize W25N MTD driver\n");
led_on(LED_RED);
} else {
int ret = register_mtddriver("/dev/mtd0", mtd, 0755, NULL);
if (ret < 0) {
syslog(LOG_ERR, "[boot] FAILED to register MTD driver: %d\n", ret);
led_on(LED_RED);
} else {
syslog(LOG_INFO, "[boot] W25N MTD registered at /dev/mtd0\n");
struct mtd_geometry_s geo;
if (mtd->ioctl(mtd, MTDIOC_GEOMETRY, (unsigned long)((uintptr_t)&geo)) == 0) {
syslog(LOG_INFO, "[boot] W25N: %lu erase blocks, %lu bytes/block, %lu total bytes\n",
(unsigned long)geo.neraseblocks,
(unsigned long)geo.erasesize,
(unsigned long)geo.neraseblocks * (unsigned long)geo.erasesize);
}
#ifdef CONFIG_FS_LITTLEFS
ret = nx_mount("/dev/mtd0", CONFIG_BOARD_ROOT_PATH, "littlefs", 0, "autoformat");
if (ret < 0) {
syslog(LOG_ERR, "[boot] FAILED to mount littlefs: %d\n", ret);
led_on(LED_RED);
} else {
syslog(LOG_INFO, "[boot] LittleFS mounted at %s\n", CONFIG_BOARD_ROOT_PATH);
}
#endif
}
}
}
#endif
up_udelay(20);
#if defined(FLASH_BASED_PARAMS)
+1 -1
View File
@@ -37,7 +37,7 @@
constexpr px4_spi_bus_t px4_spi_buses[SPI_BUS_MAX_BUS_ITEMS] = {
initSPIBus(SPI::Bus::SPI1, {
initSPIDevice(SPIDEV_MMCSD(0), SPI::CS{GPIO::PortA, GPIO::Pin4})
initSPIDevice(SPIDEV_FLASH(0), SPI::CS{GPIO::PortA, GPIO::Pin4}) // W25N01GV NAND Flash
}),
initSPIBus(SPI::Bus::SPI2, {
initSPIDevice(DRV_OSD_DEVTYPE_ATXXXX, SPI::CS{GPIO::PortB, GPIO::Pin12}),
@@ -79,6 +79,7 @@ CONFIG_SYSTEMCMDS_HARDFAULT_LOG=y
CONFIG_SYSTEMCMDS_I2CDETECT=y
CONFIG_SYSTEMCMDS_LED_CONTROL=y
CONFIG_SYSTEMCMDS_MFT=y
CONFIG_SYSTEMCMDS_MKLITTLEFS=y
CONFIG_SYSTEMCMDS_NSHTERM=y
CONFIG_SYSTEMCMDS_PARAM=y
CONFIG_SYSTEMCMDS_PERF=y
@@ -38,6 +38,7 @@ param set-default SYS_DM_BACKEND 1
# Ignore that there is no SD card
param set-default COM_ARM_SDCARD 0
# W25N NAND flash with littlefs (128 MB): larger buffer, auto-rotate
set LOGGER_BUF 32
param set-default SDLOG_DIRS_MAX 3
# W25N NAND flash with littlefs (128 MB): small log file size so we can keep
# a few recent logs. Default SDLOG_ROTATE=90 keeps at least 10% free during
# writing (no bad block management yet, so avoid hammering the flash near full).
param set-default SDLOG_MAX_SIZE 30
@@ -59,11 +59,6 @@
# define BOARD_HAS_NBAT_V 1
# define BOARD_HAS_NBAT_I 1
/* Enable small flash logging support (for W25N NAND flash) */
#ifdef CONFIG_MTD_W25N
# define BOARD_SMALL_FLASH_LOGGING 1
#endif
/* Holybro KakuteH7 GPIOs ************************************************************************/
/* LEDs are driven with push open drain to support Anode to 5V or 3.3V */
+30
View File
@@ -83,6 +83,36 @@ This configuration will log sensor_accel 0 at full rate, sensor_accel 1 at 10Hz,
There are several scripts to analyze and convert logging files in the [pyulog](https://github.com/PX4/pyulog) repository.
## Log Cleanup
PX4 automatically manages log storage by cleaning up old logs when starting to log.
Two parameters control how much space logs may use:
- [SDLOG_ROTATE](../advanced_config/parameter_reference.md#SDLOG_ROTATE) is the maximum disk usage percentage (default 90).
Cleanup ensures at least `(100 - SDLOG_ROTATE)%` of the disk stays free at all times, **even while writing a new log file**.
Setting it to `0` disables space-based cleanup entirely; setting it to `100` lets logs fill the disk completely.
- [SDLOG_MAX_SIZE](../advanced_config/parameter_reference.md#SDLOG_MAX_SIZE) is the maximum size of a single log file in MB
(default 1024). It also reserves headroom so that a full new file always fits after cleanup.
At log start, the cleanup threshold is `((100 - SDLOG_ROTATE)% of disk) + SDLOG_MAX_SIZE`.
Oldest logs are deleted until the free space meets this threshold.
For example, on an 8 GB card with defaults, cleanup keeps at least `820 + 1024 = ~1.8 GB` free at log start,
so ~6 GB is usable for logs and disk usage never exceeds 90% during writing.
Small flash targets override `SDLOG_MAX_SIZE` to a smaller value to keep more logs within the available space.
The cleanup algorithm prioritizes deleting logs from the directory naming scheme not currently in use.
PX4 uses two directory naming schemes:
- **Session directories** (`sess001`, `sess002`, etc.) - used when the system doesn't have valid time information
- **Date directories** (`2024-01-15`, `2024-01-16`, etc.) - used when the system has valid time (e.g., from GPS)
When cleanup is needed:
- If the system has valid time (using date directories): old session directories are deleted first
- If the system doesn't have valid time (using session directories): old date directories are deleted first
This ensures that stale logs from a different time mode are cleaned up before current logs.
## File size limitations
The maximum file size depends on the file system and OS.
@@ -91,6 +91,18 @@ Firmware can be installed in any of the normal ways:
- [Load the firmware](../config/firmware.md) using _QGroundControl_.
You can use either pre-built firmware or your own custom firmware.
### Flash Storage Troubleshooting
The AirBrainH743 uses a 128MB NAND flash (W25N) with a littlefs filesystem for logging.
If the flash filesystem becomes corrupted, you can reformat it using the [System Console](../debug/system_console.md):
```sh
mklittlefs /dev/mtd0 /fs/flash
```
This will erase all data on the flash and create a fresh littlefs filesystem.
The filesystem is immediately available after the command completes.
### System Console
UART1 (ttyS0) is configured for use as the [System Console](../debug/system_console.md).
+8 -1
View File
@@ -22,7 +22,8 @@ Update these notes with features that are going to be in `main` (PX4 v1.18 or la
## Read Before Upgrading
- TBD …
- The `SDLOG_DIRS_MAX` parameter has been removed. Log storage management is now controlled entirely by [SDLOG_ROTATE](../advanced_config/parameter_reference.md#SDLOG_ROTATE) (disk usage percentage) and [SDLOG_MAX_SIZE](../advanced_config/parameter_reference.md#SDLOG_MAX_SIZE). If you previously relied on `SDLOG_DIRS_MAX`, adjust these two parameters instead. See [Log Cleanup](../dev_log/logging.md#log-cleanup) for details.
- `SDLOG_MAX_SIZE` default lowered from `4095` to `1024` MB. This caps individual log files at ~1 GB, which fits a typical flight in a single file and keeps cleanup behavior reasonable on typical SD cards. Users with unusually large flight data can raise this value manually.
Please continue reading for [upgrade instructions](#upgrade-guide).
@@ -73,6 +74,12 @@ Please continue reading for [upgrade instructions](#upgrade-guide).
### Debug & Logging
- [Asset Tracking](../debug/asset_tracking.md): Automatic tracking and logging of external device information including vendor name, firmware and hardware version, serial numbers. Currently supports DroneCAN devices. ([PX4-Autopilot#25617](https://github.com/PX4/PX4-Autopilot/pull/25617))
- Logger: support for small flash storage (e.g. 128 MB W25N NAND on kakuteh7mini, kakuteh7v2, airbrainh743). Logs can now be written directly to an internal littlefs volume instead of requiring an SD card.
- Logger: reworked log rotation and cleanup:
- Added [SDLOG_ROTATE](../advanced_config/parameter_reference.md#SDLOG_ROTATE) (default `90`): maximum disk usage percentage. Cleanup guarantees `(100 - SDLOG_ROTATE)%` of the disk stays free at all times, even while writing a new log file. Set `0` to disable space-based cleanup, `100` to allow filling the disk completely.
- [SDLOG_MAX_SIZE](../advanced_config/parameter_reference.md#SDLOG_MAX_SIZE) default lowered from `4095` to `1024` MB. The old default caused over-aggressive cleanup on typical SD cards (e.g. an 8 GB card would only retain 1 log file at a time). Users with unusually large flight data can raise it explicitly.
- Removed `SDLOG_DIRS_MAX`. Directory-count limits are now handled by the space-based cleanup alone.
- New `mklittlefs` systemcmd for reformatting a littlefs volume from the NSH console, analogous to `mkfatfs` for FAT filesystems.
### Ethernet
+9
View File
@@ -55,6 +55,7 @@ px4_add_module(
log_writer_file.cpp
log_writer_mavlink.cpp
util.cpp
util_parse.cpp
watchdog.cpp
DEPENDS
version
@@ -62,3 +63,11 @@ px4_add_module(
)
px4_add_unit_gtest(SRC ULogMessagesTest.cpp)
if(BUILD_TESTING)
# Separate library for pure parsing functions (testable without PX4 dependencies)
add_library(logger_util_parse STATIC util_parse.cpp)
target_include_directories(logger_util_parse PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})
endif()
px4_add_unit_gtest(SRC loggerUtilTest.cpp LINKLIBS logger_util_parse)
+68 -16
View File
@@ -38,6 +38,7 @@
#include "messages.h"
#include <dirent.h>
#include <inttypes.h>
#include <sys/stat.h>
#include <errno.h>
#include <string.h>
@@ -597,9 +598,14 @@ void Logger::run()
}
}
if (util::check_free_space(LOG_ROOT[(int)LogType::Full], _param_sdlog_dirs_max.get(), _mavlink_log_pub,
_file_name[(int)LogType::Full].sess_dir_index) == 1) {
return;
// Get the next session directory index
util::LogDirInfo dir_info;
if (util::scan_log_directories(LOG_ROOT[(int)LogType::Full], dir_info)) {
_file_name[(int)LogType::Full].sess_dir_index = dir_info.sess_idx_max + 1;
} else {
_file_name[(int)LogType::Full].sess_dir_index = 0;
}
}
@@ -814,7 +820,7 @@ void Logger::run()
if (_log_message_sub.update(&log_message)) {
const char *message = (const char *)log_message.text;
int message_len = strlen(message);
int message_len = strnlen(message, sizeof(log_message.text));
if (message_len > 0) {
uint16_t write_msg_size = sizeof(ulog_message_logging_s) - sizeof(ulog_message_logging_s::message)
@@ -879,6 +885,14 @@ void Logger::run()
debug_print_buffer(total_bytes, timer_start);
// Rotate log file when it exceeds max size (if SDLOG_MAX_SIZE > 0)
if (_max_log_file_size > 0 &&
_writer.get_total_written_file(LogType::Full) > _max_log_file_size) {
PX4_INFO("Log file size limit reached, rotating");
stop_log_file(LogType::Full);
start_log_file(LogType::Full);
}
was_started = true;
} else { // not logging
@@ -1250,7 +1264,8 @@ int Logger::create_log_dir(LogType type, tm *tt, char *log_dir, int log_dir_len)
if (tt) {
strftime(file_name.log_dir, sizeof(LogFileName::log_dir), "%Y-%m-%d", tt);
strncpy(log_dir + n, file_name.log_dir, log_dir_len - n);
strncpy(log_dir + n, file_name.log_dir, log_dir_len - n - 1);
log_dir[log_dir_len - 1] = '\0';
int mkdir_ret = mkdir(log_dir, S_IRWXU | S_IRWXG | S_IRWXO);
if (mkdir_ret != OK && errno != EEXIST) {
@@ -1262,7 +1277,8 @@ int Logger::create_log_dir(LogType type, tm *tt, char *log_dir, int log_dir_len)
uint16_t dir_number = file_name.sess_dir_index;
if (file_name.has_log_dir) {
strncpy(log_dir + n, file_name.log_dir, log_dir_len - n);
strncpy(log_dir + n, file_name.log_dir, log_dir_len - n - 1);
log_dir[log_dir_len - 1] = '\0';
}
/* look for the next dir that does not exist */
@@ -1275,7 +1291,8 @@ int Logger::create_log_dir(LogType type, tm *tt, char *log_dir, int log_dir_len)
return -1;
}
strncpy(log_dir + n, file_name.log_dir, log_dir_len - n);
strncpy(log_dir + n, file_name.log_dir, log_dir_len - n - 1);
log_dir[log_dir_len - 1] = '\0';
int mkdir_ret = mkdir(log_dir, S_IRWXU | S_IRWXG | S_IRWXO);
if (mkdir_ret == 0) {
@@ -1359,20 +1376,27 @@ int Logger::get_log_file_name(LogType type, char *file_name, size_t file_name_si
return -1;
}
// Find the highest existing log file number and use next
uint16_t file_number = 100; // start with file log100
uint16_t max_existing = 99;
/* look for the next file that does not exist */
while (file_number <= MAX_NO_LOGFILE) {
/* format log file path: e.g. /fs/microsd/log/sess001/log001.ulg */
snprintf(log_file_name, sizeof(LogFileName::log_file_name), "log%03" PRIu16 "%s.ulg%s", file_number, replay_suffix,
crypto_suffix);
snprintf(file_name + n, file_name_size - n, "/%s", log_file_name);
DIR *dp = opendir(file_name);
if (!util::file_exist(file_name)) {
break;
if (dp != nullptr) {
struct dirent *entry;
while ((entry = readdir(dp)) != nullptr) {
uint16_t num;
if (sscanf(entry->d_name, "log%hu", &num) == 1) {
if (num > max_existing) {
max_existing = num;
}
}
}
file_number++;
closedir(dp);
file_number = max_existing + 1;
}
if (file_number > MAX_NO_LOGFILE) {
@@ -1380,6 +1404,11 @@ int Logger::get_log_file_name(LogType type, char *file_name, size_t file_name_si
return -1;
}
/* format log file path: e.g. /fs/microsd/log/sess001/log001.ulg */
snprintf(log_file_name, sizeof(LogFileName::log_file_name), "log%03" PRIu16 "%s.ulg%s", file_number, replay_suffix,
crypto_suffix);
snprintf(file_name + n, file_name_size - n, "/%s", log_file_name);
if (notify) {
mavlink_log_info(&_mavlink_log_pub, "[logger] %s\t", file_name);
uint16_t sess = 0;
@@ -1410,6 +1439,29 @@ void Logger::start_log_file(LogType type)
}
if (type == LogType::Full) {
int32_t max_size_mb = _param_sdlog_max_size.get();
if (max_size_mb > 0) {
_max_log_file_size = (size_t)max_size_mb * 1024ULL * 1024ULL;
PX4_INFO("Max log file size: %" PRId32 " MB", max_size_mb);
} else {
_max_log_file_size = 0; // unlimited
}
// Cleanup old logs if needed.
// SDLOG_ROTATE is the max disk-usage percentage; cleanup ensures at least
// (100 - rotate)% is free even during writing. SDLOG_MAX_SIZE is passed so
// there's always room for the next log file on top of the free-space target.
hrt_abstime cleanup_start = hrt_absolute_time();
if (util::cleanup_old_logs(LOG_ROOT[(int)LogType::Full], _mavlink_log_pub,
(uint32_t)_param_sdlog_rotate.get(), (uint32_t)max_size_mb) == 1) {
return; // Not enough space even after cleanup
}
PX4_INFO("Log cleanup took %" PRIu64 " ms", hrt_elapsed_time(&cleanup_start) / 1000);
// initialize cpu load as early as possible to get more data
initialize_load_output(PrintLoadReason::Preflight);
}
+4 -1
View File
@@ -387,6 +387,8 @@ private:
hrt_abstime _logger_status_last {0};
int _lockstep_component{-1};
size_t _max_log_file_size {0}; ///< max log file size in bytes (0 = unlimited)
uint32_t _message_gaps{0};
timer_callback_data_s _timer_callback_data{};
@@ -399,7 +401,8 @@ private:
DEFINE_PARAMETERS(
(ParamInt<px4::params::SDLOG_UTC_OFFSET>) _param_sdlog_utc_offset,
(ParamInt<px4::params::SDLOG_DIRS_MAX>) _param_sdlog_dirs_max,
(ParamInt<px4::params::SDLOG_MAX_SIZE>) _param_sdlog_max_size,
(ParamInt<px4::params::SDLOG_ROTATE>) _param_sdlog_rotate,
(ParamInt<px4::params::SDLOG_PROFILE>) _param_sdlog_profile,
(ParamInt<px4::params::SDLOG_MISSION>) _param_sdlog_mission,
(ParamBool<px4::params::SDLOG_BOOT_BAT>) _param_sdlog_boot_bat,
+283
View File
@@ -0,0 +1,283 @@
/****************************************************************************
*
* Copyright (c) 2025 PX4 Development Team. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* 3. Neither the name PX4 nor the names of its contributors may be
* used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
* FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
* COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
* BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
* OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
* AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
* ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
****************************************************************************/
/**
* Test code for the logger utility parsing functions
* Run this test using: build/px4_sitl_test/unit-util
*/
#include <gtest/gtest.h>
#include "util_parse.h"
using namespace px4::logger::util;
// Session directory parsing tests
TEST(LoggerUtilTest, ParseSessDirValid)
{
int idx;
EXPECT_TRUE(parse_sess_dir_name("sess000", idx));
EXPECT_EQ(idx, 0);
EXPECT_TRUE(parse_sess_dir_name("sess001", idx));
EXPECT_EQ(idx, 1);
EXPECT_TRUE(parse_sess_dir_name("sess123", idx));
EXPECT_EQ(idx, 123);
EXPECT_TRUE(parse_sess_dir_name("sess999", idx));
EXPECT_EQ(idx, 999);
}
TEST(LoggerUtilTest, ParseSessDirInvalid)
{
int idx;
EXPECT_FALSE(parse_sess_dir_name("session001", idx));
EXPECT_FALSE(parse_sess_dir_name("2024-01-15", idx));
EXPECT_FALSE(parse_sess_dir_name(".", idx));
EXPECT_FALSE(parse_sess_dir_name("..", idx));
EXPECT_FALSE(parse_sess_dir_name("log001", idx));
EXPECT_FALSE(parse_sess_dir_name("", idx));
}
// Date directory parsing tests
TEST(LoggerUtilTest, ParseDateDirValid)
{
int y, m, d;
EXPECT_TRUE(parse_date_dir_name("2024-01-15", y, m, d));
EXPECT_EQ(y, 2024);
EXPECT_EQ(m, 1);
EXPECT_EQ(d, 15);
EXPECT_TRUE(parse_date_dir_name("2023-12-31", y, m, d));
EXPECT_EQ(y, 2023);
EXPECT_EQ(m, 12);
EXPECT_EQ(d, 31);
EXPECT_TRUE(parse_date_dir_name("2025-06-01", y, m, d));
EXPECT_EQ(y, 2025);
EXPECT_EQ(m, 6);
EXPECT_EQ(d, 1);
}
TEST(LoggerUtilTest, ParseDateDirInvalid)
{
int y, m, d;
EXPECT_FALSE(parse_date_dir_name("sess001", y, m, d));
EXPECT_FALSE(parse_date_dir_name("2024-01", y, m, d));
EXPECT_FALSE(parse_date_dir_name("2024", y, m, d));
EXPECT_FALSE(parse_date_dir_name(".", y, m, d));
EXPECT_FALSE(parse_date_dir_name("..", y, m, d));
EXPECT_FALSE(parse_date_dir_name("", y, m, d));
}
// Date comparison tests
TEST(LoggerUtilTest, IsDateOlderYearDifference)
{
// Earlier year is older
EXPECT_TRUE(is_date_older(2023, 12, 31, 2024, 1, 1));
EXPECT_FALSE(is_date_older(2024, 1, 1, 2023, 12, 31));
}
TEST(LoggerUtilTest, IsDateOlderMonthDifference)
{
// Same year, earlier month is older
EXPECT_TRUE(is_date_older(2024, 1, 15, 2024, 2, 1));
EXPECT_FALSE(is_date_older(2024, 2, 1, 2024, 1, 15));
}
TEST(LoggerUtilTest, IsDateOlderDayDifference)
{
// Same year and month, earlier day is older
EXPECT_TRUE(is_date_older(2024, 1, 1, 2024, 1, 15));
EXPECT_FALSE(is_date_older(2024, 1, 15, 2024, 1, 1));
}
TEST(LoggerUtilTest, IsDateOlderSameDate)
{
// Same date is not older
EXPECT_FALSE(is_date_older(2024, 1, 15, 2024, 1, 15));
}
TEST(LoggerUtilTest, IsDateOlderEdgeCases)
{
// Year boundary
EXPECT_TRUE(is_date_older(2023, 12, 31, 2024, 1, 1));
// Month boundary
EXPECT_TRUE(is_date_older(2024, 1, 31, 2024, 2, 1));
// Large year difference
EXPECT_TRUE(is_date_older(2000, 6, 15, 2024, 6, 15));
}
// process_dir_entry tests - combined sess vs date logic
TEST(LoggerUtilTest, ProcessDirEntrySessionOnly)
{
LogDirInfo info{};
process_dir_entry("sess000", info);
EXPECT_EQ(info.num_sess, 1);
EXPECT_EQ(info.sess_idx_min, 0);
EXPECT_EQ(info.sess_idx_max, 0);
EXPECT_EQ(info.num_dates, 0);
process_dir_entry("sess005", info);
EXPECT_EQ(info.num_sess, 2);
EXPECT_EQ(info.sess_idx_min, 0);
EXPECT_EQ(info.sess_idx_max, 5);
process_dir_entry("sess003", info);
EXPECT_EQ(info.num_sess, 3);
EXPECT_EQ(info.sess_idx_min, 0);
EXPECT_EQ(info.sess_idx_max, 5);
}
TEST(LoggerUtilTest, ProcessDirEntryDateOnly)
{
LogDirInfo info{};
process_dir_entry("2024-06-15", info);
EXPECT_EQ(info.num_dates, 1);
EXPECT_EQ(info.oldest_year, 2024);
EXPECT_EQ(info.oldest_month, 6);
EXPECT_EQ(info.oldest_day, 15);
EXPECT_EQ(info.num_sess, 0);
// Add older date
process_dir_entry("2024-01-10", info);
EXPECT_EQ(info.num_dates, 2);
EXPECT_EQ(info.oldest_year, 2024);
EXPECT_EQ(info.oldest_month, 1);
EXPECT_EQ(info.oldest_day, 10);
// Add newer date (oldest should not change)
process_dir_entry("2024-12-25", info);
EXPECT_EQ(info.num_dates, 3);
EXPECT_EQ(info.oldest_year, 2024);
EXPECT_EQ(info.oldest_month, 1);
EXPECT_EQ(info.oldest_day, 10);
}
TEST(LoggerUtilTest, ProcessDirEntryMixedSessAndDate)
{
LogDirInfo info{};
process_dir_entry("sess001", info);
process_dir_entry("2024-03-15", info);
process_dir_entry("sess005", info);
process_dir_entry("2023-12-01", info);
EXPECT_EQ(info.num_sess, 2);
EXPECT_EQ(info.sess_idx_min, 1);
EXPECT_EQ(info.sess_idx_max, 5);
EXPECT_EQ(info.num_dates, 2);
EXPECT_EQ(info.oldest_year, 2023);
EXPECT_EQ(info.oldest_month, 12);
EXPECT_EQ(info.oldest_day, 1);
}
TEST(LoggerUtilTest, ProcessDirEntryIgnoresInvalid)
{
LogDirInfo info{};
// These should be ignored
process_dir_entry(".", info);
process_dir_entry("..", info);
process_dir_entry("log001", info);
process_dir_entry("session001", info);
process_dir_entry("2024-01", info);
process_dir_entry("random_dir", info);
EXPECT_EQ(info.num_sess, 0);
EXPECT_EQ(info.num_dates, 0);
EXPECT_EQ(info.sess_idx_max, -1); // unchanged from default
EXPECT_EQ(info.sess_idx_min, INT_MAX); // unchanged from default
}
TEST(LoggerUtilTest, ProcessDirEntrySessMinMaxTracking)
{
LogDirInfo info{};
// Add in non-sequential order
process_dir_entry("sess050", info);
EXPECT_EQ(info.sess_idx_min, 50);
EXPECT_EQ(info.sess_idx_max, 50);
process_dir_entry("sess010", info);
EXPECT_EQ(info.sess_idx_min, 10);
EXPECT_EQ(info.sess_idx_max, 50);
process_dir_entry("sess100", info);
EXPECT_EQ(info.sess_idx_min, 10);
EXPECT_EQ(info.sess_idx_max, 100);
process_dir_entry("sess025", info);
EXPECT_EQ(info.sess_idx_min, 10);
EXPECT_EQ(info.sess_idx_max, 100);
}
TEST(LoggerUtilTest, ProcessDirEntryDateOldestTracking)
{
LogDirInfo info{};
// Start with a date in the middle
process_dir_entry("2024-06-15", info);
EXPECT_EQ(info.oldest_year, 2024);
EXPECT_EQ(info.oldest_month, 6);
EXPECT_EQ(info.oldest_day, 15);
// Add older year
process_dir_entry("2023-12-31", info);
EXPECT_EQ(info.oldest_year, 2023);
EXPECT_EQ(info.oldest_month, 12);
EXPECT_EQ(info.oldest_day, 31);
// Add same year, older month
process_dir_entry("2023-01-15", info);
EXPECT_EQ(info.oldest_year, 2023);
EXPECT_EQ(info.oldest_month, 1);
EXPECT_EQ(info.oldest_day, 15);
// Add same year/month, older day
process_dir_entry("2023-01-01", info);
EXPECT_EQ(info.oldest_year, 2023);
EXPECT_EQ(info.oldest_month, 1);
EXPECT_EQ(info.oldest_day, 1);
// Add newer date (oldest should not change)
process_dir_entry("2025-01-01", info);
EXPECT_EQ(info.oldest_year, 2023);
EXPECT_EQ(info.oldest_month, 1);
EXPECT_EQ(info.oldest_day, 1);
}
+27 -11
View File
@@ -102,20 +102,36 @@ parameters:
min: 0
max: 4095
reboot_required: true
SDLOG_DIRS_MAX:
SDLOG_MAX_SIZE:
description:
short: Maximum number of log directories to keep
long: 'If there are more log directories than this value, the system will
delete the oldest directories during startup. In addition, the system will
delete old logs if there is not enough free space left. The minimum amount
is 300 MB. If this is set to 0, old directories will only be removed if
the free space falls below the minimum. Note: this does not apply to mission
log files.'
short: Maximum log file size
long: 'Maximum size of a single log file in megabytes. When reached,
the log file is closed and a new one is started. This value is also
added to the cleanup threshold (see SDLOG_ROTATE) to reserve headroom
for the next log file. A value of 0 disables both file rotation and
the cleanup reservation.
Must stay below the FAT32 file size limit of 4 GiB.'
type: int32
default: 0
default: 1024
min: 0
max: 1000
reboot_required: true
max: 4095
reboot_required: false
SDLOG_ROTATE:
description:
short: Maximum disk usage percentage
long: 'Maximum percentage of disk space that logs may occupy during operation,
including while writing a new log file. For example, a value of 90 means
at least 10% of disk is always kept free, even while writing. A value of
100 lets logs fill the disk completely. A value of 0 disables space-based
cleanup entirely. At log start, oldest logs are deleted as needed to
maintain this guarantee, accounting for the next file write of up to
SDLOG_MAX_SIZE. Cleanup always happens at log start (not boot) so logs
can be downloaded via FTP before deletion.'
type: int32
default: 90
min: 0
max: 100
reboot_required: false
SDLOG_UUID:
description:
short: Log UUID
+172 -117
View File
@@ -34,14 +34,12 @@
#include "util.h"
#include <dirent.h>
#include <inttypes.h>
#include <sys/stat.h>
#include <string.h>
#include <stdlib.h>
#include <unistd.h>
#include <uORB/Subscription.hpp>
#include <uORB/topics/sensor_gps.h>
#include <drivers/drv_hrt.h>
#include <px4_platform_common/events.h>
#include <px4_platform_common/log.h>
@@ -57,8 +55,6 @@
#define GPS_EPOCH_SECS ((time_t)1234567890ULL)
typedef decltype(statfs::f_bavail) px4_statfs_buf_f_bavail_t;
namespace px4
{
namespace logger
@@ -72,32 +68,33 @@ bool file_exist(const char *filename)
return stat(filename, &buffer) == 0;
}
bool get_log_time(uint64_t &utc_time_usec, int utc_offset_sec, bool boot_time)
bool get_free_space(const char *path, uint64_t *avail_bytes, uint64_t *total_bytes)
{
uORB::Subscription vehicle_gps_position_sub{ORB_ID(vehicle_gps_position)};
struct statfs statfs_buf;
bool use_clock_time = true;
/* Get the latest GPS publication */
sensor_gps_s gps_pos;
if (vehicle_gps_position_sub.copy(&gps_pos)) {
utc_time_usec = gps_pos.time_utc_usec;
if (gps_pos.fix_type >= 2 && utc_time_usec >= (uint64_t) GPS_EPOCH_SECS * 1000000ULL) {
use_clock_time = false;
}
if (statfs(path, &statfs_buf) != 0) {
return false;
}
if (use_clock_time) {
/* take clock time if there's no fix (yet) */
struct timespec ts = {};
px4_clock_gettime(CLOCK_REALTIME, &ts);
utc_time_usec = ts.tv_sec * 1000000ULL + ts.tv_nsec / 1000ULL;
if (avail_bytes != nullptr) {
*avail_bytes = (uint64_t)statfs_buf.f_bavail * statfs_buf.f_bsize;
}
if (utc_time_usec < (uint64_t) GPS_EPOCH_SECS * 1000000ULL) {
return false;
}
if (total_bytes != nullptr) {
*total_bytes = (uint64_t)statfs_buf.f_blocks * statfs_buf.f_bsize;
}
return true;
}
bool get_log_time(uint64_t &utc_time_usec, int utc_offset_sec, bool boot_time)
{
struct timespec ts = {};
px4_clock_gettime(CLOCK_REALTIME, &ts);
utc_time_usec = ts.tv_sec * 1000000ULL + ts.tv_nsec / 1000ULL;
if (utc_time_usec < (uint64_t) GPS_EPOCH_SECS * 1000000ULL) {
return false;
}
/* strip the time elapsed since boot */
@@ -119,130 +116,188 @@ bool get_log_time(struct tm *tt, int utc_offset_sec, bool boot_time)
return result && gmtime_r(&utc_time_sec, tt) != nullptr;
}
int check_free_space(const char *log_root_dir, int32_t max_log_dirs_to_keep, orb_advert_t &mavlink_log_pub,
int &sess_dir_index)
bool scan_log_directories(const char *log_root_dir, LogDirInfo &info)
{
struct statfs statfs_buf;
DIR *dp = opendir(log_root_dir);
if (max_log_dirs_to_keep == 0) {
max_log_dirs_to_keep = INT32_MAX;
if (dp == nullptr) {
return false;
}
// remove old logs if the free space falls below a threshold
do {
if (statfs(log_root_dir, &statfs_buf) != 0) {
return PX4_ERROR;
// Reset info to defaults
info = LogDirInfo{};
struct dirent *result = nullptr;
while ((result = readdir(dp))) {
process_dir_entry(result->d_name, info);
}
closedir(dp);
return true;
}
int cleanup_old_logs(const char *log_root_dir, orb_advert_t &mavlink_log_pub,
uint32_t rotate_pct, uint32_t max_file_size_mb)
{
uint64_t avail_bytes = 0;
uint64_t total_bytes = 0;
if (!get_free_space(log_root_dir, &avail_bytes, &total_bytes)) {
return PX4_ERROR;
}
// Calculate cleanup threshold. rotate_pct is the maximum allowed disk usage;
// we guarantee the disk never exceeds rotate_pct% full even during writing of
// a new log file. So at cleanup time, free space must be at least
// ((100 - rotate_pct)% of disk) + max log file size, where the latter term
// reserves headroom for the next file write.
// rotate_pct == 0 disables space-based cleanup entirely.
uint64_t cleanup_threshold = 0;
if (rotate_pct > 0 && rotate_pct <= 100) {
cleanup_threshold = (total_bytes * (100 - rotate_pct)) / 100;
cleanup_threshold += (uint64_t)max_file_size_mb * 1024ULL * 1024ULL;
}
// Early out if we have enough space
if (avail_bytes >= cleanup_threshold) {
return PX4_OK;
}
// Scan directories for cleanup
LogDirInfo info;
if (!scan_log_directories(log_root_dir, info)) {
return PX4_OK; // ignore if we cannot access the log directory
}
PX4_INFO("Log cleanup: %u MiB free, threshold %u MiB",
(unsigned)(avail_bytes / 1024U / 1024U), (unsigned)(cleanup_threshold / 1024U / 1024U));
// Determine if we currently have valid time (using date dirs) or not (using sess dirs)
// Delete from the "other" scheme first to avoid deleting current log
uint64_t utc_time_usec;
bool have_time = get_log_time(utc_time_usec, 0, false);
// Cleanup oldest .ulg files one by one until we have enough free space
int empty_dir_failures = 0;
while (avail_bytes < cleanup_threshold) {
char oldest_file[LOG_DIR_LEN] = "";
char oldest_dir[LOG_DIR_LEN];
if (!scan_log_directories(log_root_dir, info)) {
break;
}
DIR *dp = opendir(log_root_dir);
bool found_sess = info.num_sess > 0;
bool found_date = info.num_dates > 0;
if (!found_sess && !found_date) {
PX4_WARN("No log directories found to clean up");
break; // no log directories found
}
// Delete from the "other" naming scheme first (it's old/stale)
// - Have time (using date dirs): delete sess dirs first
// - No time (using sess dirs): delete date dirs first, then sess dirs
if (have_time && found_sess) {
// Using date dirs, delete old sess dirs first
snprintf(oldest_dir, sizeof(oldest_dir), "%s/sess%03u", log_root_dir, info.sess_idx_min);
} else if (!have_time && found_date) {
// Using sess dirs, delete old date dirs first
snprintf(oldest_dir, sizeof(oldest_dir), "%s/%04u-%02u-%02u", log_root_dir,
info.oldest_year, info.oldest_month, info.oldest_day);
} else if (found_sess) {
// Delete from oldest sess dir (including current - old files are ok to delete)
snprintf(oldest_dir, sizeof(oldest_dir), "%s/sess%03u", log_root_dir, info.sess_idx_min);
} else if (found_date) {
// Delete from oldest date dir
snprintf(oldest_dir, sizeof(oldest_dir), "%s/%04u-%02u-%02u", log_root_dir,
info.oldest_year, info.oldest_month, info.oldest_day);
} else {
// Nothing left to delete
break;
}
PX4_DEBUG("Checking directory %s for old logs", oldest_dir);
// Find oldest .ulg file in that directory
DIR *dp = opendir(oldest_dir);
if (dp == nullptr) {
break; // ignore if we cannot access the log directory
PX4_WARN("Cannot open directory %s", oldest_dir);
break;
}
// Max log filename: "12_09_00_replayed.ulg" (21 chars + null = 22 bytes)
// Using 64 for margin against unexpected filenames on disk.
static constexpr unsigned MAX_LOG_FILENAME_LEN = 64;
char oldest_ulg[MAX_LOG_FILENAME_LEN] = "";
struct dirent *result = nullptr;
int num_sess = 0, num_dates = 0;
// There are 2 directory naming schemes: sess<i> or <year>-<month>-<day>.
// For both we find the oldest and then remove the one which has more directories.
int year_min = 10000, month_min = 99, day_min = 99, sess_idx_min = 99999999, sess_idx_max = 99;
while ((result = readdir(dp))) {
int year, month, day, sess_idx;
size_t len = strlen(result->d_name);
if (sscanf(result->d_name, "sess%d", &sess_idx) == 1) {
++num_sess;
if (sess_idx > sess_idx_max) {
sess_idx_max = sess_idx;
}
if (sess_idx < sess_idx_min) {
sess_idx_min = sess_idx;
}
} else if (sscanf(result->d_name, "%d-%d-%d", &year, &month, &day) == 3) {
++num_dates;
if (year < year_min) {
year_min = year;
month_min = month;
day_min = day;
} else if (year == year_min) {
if (month < month_min) {
month_min = month;
day_min = day;
} else if (month == month_min) {
if (day < day_min) {
day_min = day;
}
}
if (len > 4 && strcmp(result->d_name + len - 4, ".ulg") == 0) {
if (oldest_ulg[0] == '\0' || strcmp(result->d_name, oldest_ulg) < 0) {
strncpy(oldest_ulg, result->d_name, sizeof(oldest_ulg) - 1);
oldest_ulg[sizeof(oldest_ulg) - 1] = '\0';
}
}
}
closedir(dp);
sess_dir_index = sess_idx_max + 1;
if (oldest_ulg[0] == '\0') {
// No .ulg files, try to remove directory
if (remove_directory(oldest_dir) == 0) {
PX4_INFO("removed directory %s (no .ulg files)", oldest_dir);
empty_dir_failures = 0;
} else {
// Removal failed (littlefs may report "not empty" for empty dirs)
// Toggle have_time to try the other naming scheme next iteration
empty_dir_failures++;
uint64_t min_free_bytes = 300ULL * 1024ULL * 1024ULL;
uint64_t total_bytes = (uint64_t)statfs_buf.f_blocks * statfs_buf.f_bsize;
if (empty_dir_failures >= 3) {
PX4_WARN("Cannot remove empty directories, giving up");
break;
}
if (total_bytes / 10 < min_free_bytes) { // reduce the minimum if it's larger than 10% of the disk size
min_free_bytes = total_bytes / 10;
have_time = !have_time;
PX4_DEBUG("Cannot remove %s, trying other scheme", oldest_dir);
}
continue;
}
if (num_sess + num_dates <= max_log_dirs_to_keep &&
statfs_buf.f_bavail >= (px4_statfs_buf_f_bavail_t)(min_free_bytes / statfs_buf.f_bsize)) {
break; // enough free space and limit not reached
}
// Build full path and delete the file
snprintf(oldest_file, sizeof(oldest_file), "%s/%s", oldest_dir, oldest_ulg);
PX4_INFO("removing old log %s/%s", oldest_dir, oldest_ulg);
if (num_sess == 0 && num_dates == 0) {
break; // nothing to delete
}
char directory_to_delete[LOG_DIR_LEN];
int n;
if (num_sess >= num_dates) {
n = snprintf(directory_to_delete, sizeof(directory_to_delete), "%s/sess%03u", log_root_dir, sess_idx_min);
} else {
n = snprintf(directory_to_delete, sizeof(directory_to_delete), "%s/%04u-%02u-%02u", log_root_dir, year_min, month_min,
day_min);
}
if (n >= (int)sizeof(directory_to_delete)) {
PX4_ERR("log path too long (%i)", n);
if (unlink(oldest_file) != 0) {
PX4_ERR("Failed to delete %s", oldest_file);
break;
}
PX4_INFO("removing log directory %s to get more space (left=%u MiB)", directory_to_delete,
(unsigned int)(statfs_buf.f_bavail * statfs_buf.f_bsize / 1024U / 1024U));
if (remove_directory(directory_to_delete)) {
PX4_ERR("Failed to delete directory");
// Re-check free space
if (!get_free_space(log_root_dir, &avail_bytes, nullptr)) {
break;
}
}
} while (true);
/* use a threshold of 50 MiB: if below, do not start logging */
if (statfs_buf.f_bavail < (px4_statfs_buf_f_bavail_t)(50 * 1024 * 1024 / statfs_buf.f_bsize)) {
mavlink_log_critical(&mavlink_log_pub,
"[logger] Not logging; SD almost full: %u MiB\t",
(unsigned int)(statfs_buf.f_bavail * statfs_buf.f_bsize / 1024U / 1024U));
/* EVENT
* @description Either manually free up some space, or enable automatic log rotation
* via <param>SDLOG_DIRS_MAX</param>.
*/
// Final check: if still not enough space, refuse to log
if (avail_bytes < 10ULL * 1024ULL * 1024ULL) { // Less than 10 MiB is critical
mavlink_log_critical(&mavlink_log_pub, "[logger] Storage full: %u MiB free\t",
(unsigned)(avail_bytes / 1024U / 1024U));
events::send<uint32_t>(events::ID("logger_storage_full"), events::Log::Error,
"Not logging, storage is almost full: {1} MiB", (uint32_t)(statfs_buf.f_bavail * statfs_buf.f_bsize / 1024U / 1024U));
"Storage full: {1} MiB free", (uint32_t)(avail_bytes / 1024U / 1024U));
return 1;
}
+43 -14
View File
@@ -34,6 +34,7 @@
#pragma once
#include <stdint.h>
#include <climits>
#include <time.h>
#include <uORB/uORB.h>
@@ -44,6 +45,9 @@
#define LOG_DIR_LEN 256
#endif
// Include parsing utilities (separate file for testability)
#include "util_parse.h"
namespace px4
{
namespace logger
@@ -63,25 +67,50 @@ int remove_directory(const char *dir);
bool file_exist(const char *filename);
/**
* Check if there is enough free space left on the SD Card.
* It will remove old log files if there is not enough space,
* and if that fails return 1, and send a user message
* @param log_root_dir log root directory: it's expected to contain directories in the form of sess%i or %d-%d-%d (year, month, day)
* @param max_log_dirs_to_keep maximum log directories to keep (set to 0 for unlimited)
* @param mavlink_log_pub
* @param sess_dir_index output argument: will be set to the next free directory sess%i index.
* @return 0 on success, 1 if not enough space, <0 on error
* Get available and total storage space for a path.
* @param path path to check
* @param avail_bytes available bytes (output), can be nullptr if not needed
* @param total_bytes total bytes (output), can be nullptr if not needed
* @return true on success, false on error
*/
int check_free_space(const char *log_root_dir, int32_t max_log_dirs_to_keep, orb_advert_t &mavlink_log_pub,
int &sess_dir_index);
bool get_free_space(const char *path, uint64_t *avail_bytes, uint64_t *total_bytes);
/**
* Utility for fetching UTC time in microseconds from sensor_gps or CLOCK_REALTIME
* Scan log directory and gather information about subdirectories.
* @param log_root_dir log root directory to scan
* @param info output: populated with directory information
* @return true on success, false if directory cannot be opened
*/
bool scan_log_directories(const char *log_root_dir, LogDirInfo &info);
/**
* Cleanup old logs to ensure sufficient free space. Deletes oldest files,
* preferring the opposite directory type first (sess dirs when time is known,
* date dirs when it is not), then falls back to its own type.
*
* The cleanup threshold is computed as:
* ((100 - rotate_pct) / 100) * disk_size + max_file_size_mb
*
* i.e. after cleanup there is enough free space to write one more full log
* file while still maintaining the rotate_pct usage guarantee.
*
* @param log_root_dir log root directory
* @param mavlink_log_pub mavlink log publisher
* @param rotate_pct maximum disk usage percentage (0 disables space-based
* cleanup; 90 keeps at least 10% free during writing)
* @param max_file_size_mb maximum log file size in MB; reserved as headroom
* for the next log file write
* @return 0 on success, 1 if not enough space even after cleanup
*/
int cleanup_old_logs(const char *log_root_dir, orb_advert_t &mavlink_log_pub,
uint32_t rotate_pct, uint32_t max_file_size_mb);
/**
* Get UTC time in microseconds from CLOCK_REALTIME
* @param utc_time_usec returned microseconds
* @param utc_offset_sec UTC time offset [s]
* @param boot_time use time when booted instead of current time
* @return true on success, false otherwise (eg. if no gps)
* @return true on success, false if system time is not set
*/
bool get_log_time(uint64_t &utc_time_usec, int utc_offset_sec, bool boot_time);
@@ -90,7 +119,7 @@ bool get_log_time(uint64_t &utc_time_usec, int utc_offset_sec, bool boot_time);
* @param tt returned time
* @param utc_offset_sec UTC time offset [s]
* @param boot_time use time when booted instead of current time
* @return true on success, false otherwise (eg. if no gps)
* @return true on success, false if system time is not set
*/
bool get_log_time(struct tm *tt, int utc_offset_sec = 0, bool boot_time = false);
+90
View File
@@ -0,0 +1,90 @@
/****************************************************************************
*
* Copyright (c) 2025 PX4 Development Team. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* 3. Neither the name PX4 nor the names of its contributors may be
* used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
* FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
* COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
* BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
* OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
* AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
* ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
****************************************************************************/
#include "util_parse.h"
#include <cstdio>
namespace px4
{
namespace logger
{
namespace util
{
bool parse_sess_dir_name(const char *name, int &sess_idx)
{
return sscanf(name, "sess%d", &sess_idx) == 1;
}
bool parse_date_dir_name(const char *name, int &year, int &month, int &day)
{
return sscanf(name, "%d-%d-%d", &year, &month, &day) == 3;
}
bool is_date_older(int y1, int m1, int d1, int y2, int m2, int d2)
{
return y1 < y2 ||
(y1 == y2 && m1 < m2) ||
(y1 == y2 && m1 == m2 && d1 < d2);
}
void process_dir_entry(const char *name, LogDirInfo &info)
{
int sess_idx;
int year, month, day;
if (parse_sess_dir_name(name, sess_idx)) {
info.num_sess++;
if (sess_idx > info.sess_idx_max) {
info.sess_idx_max = sess_idx;
}
if (sess_idx < info.sess_idx_min) {
info.sess_idx_min = sess_idx;
}
} else if (parse_date_dir_name(name, year, month, day)) {
info.num_dates++;
if (is_date_older(year, month, day, info.oldest_year, info.oldest_month, info.oldest_day)) {
info.oldest_year = year;
info.oldest_month = month;
info.oldest_day = day;
}
}
}
} //namespace util
} //namespace logger
} //namespace px4
+92
View File
@@ -0,0 +1,92 @@
/****************************************************************************
*
* Copyright (c) 2025 PX4 Development Team. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* 3. Neither the name PX4 nor the names of its contributors may be
* used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
* FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
* COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
* BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
* OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
* AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
* ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
****************************************************************************/
#pragma once
#include <climits>
namespace px4
{
namespace logger
{
namespace util
{
/**
* Information about log directories gathered by scan_log_directories()
*/
struct LogDirInfo {
int sess_idx_max{-1}; ///< highest sess index found (-1 if none)
int sess_idx_min{INT_MAX}; ///< lowest sess index found
int num_sess{0}; ///< count of sess directories
int oldest_year{INT_MAX}; ///< oldest date directory year
int oldest_month{INT_MAX}; ///< oldest date directory month
int oldest_day{INT_MAX}; ///< oldest date directory day
int num_dates{0}; ///< count of date directories
};
/**
* Process a single directory entry and update LogDirInfo accordingly.
* Tries to parse as session dir first, then as date dir.
* @param name directory entry name to process
* @param info LogDirInfo struct to update
*/
void process_dir_entry(const char *name, LogDirInfo &info);
/**
* Parse a session directory name (e.g., "sess001", "sess123")
* @param name directory name to parse
* @param sess_idx output: session index if parsing succeeds
* @return true if name matches "sess%d" pattern
*/
bool parse_sess_dir_name(const char *name, int &sess_idx);
/**
* Parse a date directory name (e.g., "2024-01-15")
* @param name directory name to parse
* @param year output: year if parsing succeeds
* @param month output: month if parsing succeeds
* @param day output: day if parsing succeeds
* @return true if name matches "%d-%d-%d" pattern
*/
bool parse_date_dir_name(const char *name, int &year, int &month, int &day);
/**
* Compare two dates
* @return true if (y1,m1,d1) < (y2,m2,d2)
*/
bool is_date_older(int y1, int m1, int d1, int y2, int m2, int d2);
} //namespace util
} //namespace logger
} //namespace px4
+41
View File
@@ -0,0 +1,41 @@
############################################################################
#
# Copyright (c) 2026 PX4 Development Team. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
#
# 1. Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in
# the documentation and/or other materials provided with the
# distribution.
# 3. Neither the name PX4 nor the names of its contributors may be
# used to endorse or promote products derived from this software
# without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
# FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
# COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
# BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
# OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
#
############################################################################
px4_add_module(
MODULE systemcmds__mklittlefs
MAIN mklittlefs
STACK_MAIN 2048
COMPILE_FLAGS
SRCS
mklittlefs.cpp
DEPENDS
)
+12
View File
@@ -0,0 +1,12 @@
menuconfig SYSTEMCMDS_MKLITTLEFS
bool "mklittlefs"
default n
---help---
Enable support for mklittlefs
menuconfig USER_MKLITTLEFS
bool "mklittlefs running as userspace module"
default y
depends on BOARD_PROTECTED && SYSTEMCMDS_MKLITTLEFS
---help---
Put mklittlefs in userspace memory
+78
View File
@@ -0,0 +1,78 @@
/****************************************************************************
*
* Copyright (c) 2026 PX4 Development Team. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* 3. Neither the name PX4 nor the names of its contributors may be
* used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
* FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
* COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
* BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
* OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
* AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
* ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
****************************************************************************/
/**
* @file mklittlefs.cpp
*
* Format a device with littlefs filesystem.
*/
#include <px4_platform_common/px4_config.h>
#include <px4_platform_common/log.h>
#include <px4_platform_common/module.h>
#include <sys/mount.h>
#include <errno.h>
#include <string.h>
static void print_usage()
{
PRINT_MODULE_DESCRIPTION("Format a device with the littlefs filesystem.");
PRINT_MODULE_USAGE_NAME_SIMPLE("mklittlefs", "command");
PRINT_MODULE_USAGE_ARG("<device> <mountpoint>", "Device and mount point (e.g. /dev/mtd0 /fs/flash)", false);
}
extern "C" __EXPORT int mklittlefs_main(int argc, char *argv[])
{
if (argc < 3) {
print_usage();
return 1;
}
const char *device = argv[1];
const char *mountpoint = argv[2];
// Try to unmount first (ignore error if not mounted)
umount(mountpoint);
int ret = mount(device, mountpoint, "littlefs", 0, "forceformat");
if (ret < 0) {
PX4_ERR("format failed: %s", strerror(errno));
return 1;
}
PX4_INFO("formatted %s as littlefs, mounted at %s", device, mountpoint);
return 0;
}