Compare commits

...

34 Commits

Author SHA1 Message Date
Ramon Roche 2057dd3381 github: fix workspace path 2022-02-14 11:05:24 -08:00
Ramon Roche 771049799f github: fix env var context 2022-02-14 09:59:20 -08:00
Ramon Roche a3d5466ac3 github: fix deprecated set-env 2022-02-12 10:29:00 -08:00
Ramon Roche 4243be3a4b tools: leave out kconf platform targets 2022-02-11 12:50:01 -08:00
Ramon Roche af8c6d3cef tools: generate single target platform 2022-02-11 12:48:05 -08:00
Ramon Roche cf9584c36a github: replace env with outputs 2022-02-11 12:47:06 -08:00
Ramon Roche fb9de0c32a github: debug variables take three 2022-02-09 10:34:56 -08:00
Ramon Roche e82c68da8b github: debug variables in group 2022-02-09 10:32:38 -08:00
Ramon Roche 3de4e46cdc github: debug variables in run scope 2022-02-09 10:24:34 -08:00
Ramon Roche 85ab62dd67 github: try to fix env variables 2022-02-09 09:23:50 -08:00
Ramon Roche b26ba88d40 github: try removing newline again 2022-02-08 21:56:21 -08:00
Ramon Roche 5132ffc0e0 github: remove whitespace from env variable 2022-02-08 21:49:49 -08:00
Ramon Roche 195082ccd1 github: docker run fix environemnt variables 2022-02-08 20:03:33 -08:00
Ramon Roche f0f82b6994 github: import tagged image 2022-02-08 15:45:59 -08:00
Ramon Roche e0dc8166dd github: import with tag name 2022-02-08 11:36:34 -08:00
Ramon Roche c181aefff7 github: debug docker build 2022-02-03 11:06:39 -08:00
Ramon Roche 963e5f5df9 github: install only required pip dep 2022-02-03 10:14:11 -08:00
Ramon Roche 991b288afb github: fix docker builder run path 2022-02-03 09:16:43 -08:00
Ramon Roche f3bfad4e8a github: bump runner to python 3.9 2022-02-03 08:59:12 -08:00
Ramon Roche 0d319f7a2f github: run non interactive 2022-02-03 08:17:37 -08:00
Ramon Roche c09f7af718 github: fix container name 2022-02-02 22:24:51 -08:00
Ramon Roche a2fc0f7924 container: docker import file path 2022-02-02 21:49:40 -08:00
Ramon Roche ae0ba7cd98 container: build with no sim tools
we are removing the gazebo sim tools from this image for now
as we are trying to slim down the image
2022-02-02 10:38:45 -08:00
Ramon Roche 083a043776 github: docker import from previous job 2022-02-02 08:20:13 -08:00
Ramon Roche 854d529c81 github: generate target list 2022-02-01 21:49:52 -08:00
Ramon Roche 419c849a8e github: remove replaced actions
workflows were replaced by a single nuttx build
2022-02-01 20:56:04 -08:00
Ramon Roche c19ed79237 container: add gnu c compiler for the arm64 arch 2022-02-01 20:40:49 -08:00
Ramon Roche 104ea0c53a github: build nuttx with docker 2022-02-01 20:30:58 -08:00
Ramon Roche fc8cdb0b6f tools: docker runner accepts tags 2022-02-01 20:30:58 -08:00
Ramon Roche 64276b0c8d github: docker build and push to registry
build docker images on:
* every pull request
* push to master
* stable release published

pushes to registry if:
* is not a pull request
* is master branch
  * tag as 'latest'
* is the result of release
  * tag with release name
2022-02-01 20:30:57 -08:00
Ramon Roche 939186fa7d github: build docker on pr workflow 2022-02-01 20:30:57 -08:00
Ramon Roche fc7e422821 container: general purpose multiarch docker
* new docker container using ubuntu focal (20.04)
* adds nuttx building for any supported target
* adds initial support for gazebo simulation via X11
* updates docker_run.sh script to initialize container
* NEEDS: updated ubuntu.sh script

How to run:
```
./Tools/docker_run.sh make all_variants_px4_fmu-v5x
```
2022-02-01 20:30:57 -08:00
Ramon Roche 92deb37f64 setup: robustify install script and update deps
* increased support for running within a container
* optionally install OpenJDK 14
* optionally install RTPS
* adds better output messages
2022-02-01 20:30:57 -08:00
Ramon Roche c6a019a9d3 ci: build every target on a single step
Switching to use an all_variants approach to reduce the number of jobs
we are creating per workflow since are starting to get throttled by
GitHub, and we have limited resources.

TODO: Build artifact uploadeds to S3, and GH Releases
2022-02-01 20:30:57 -08:00
10 changed files with 628 additions and 501 deletions
+154
View File
@@ -0,0 +1,154 @@
# build docker images on:
# * every pull request
# * push to master
# * stable release published
# * tag name is:
# - commit sha if pull request
# - 'latest' if push to master
# - release name if release
# pushes to registry if:
# * is not a pull request
# * is master branch
# * is the result of release
# builds all nuttx targets and deploys metadata
name: Build docker
on:
release:
types: [released]
push:
branches:
- 'master'
pull_request:
jobs:
set_docker_tag:
name: Set docker tag
runs-on: ubuntu-latest
env:
DOCKER_TAG: "px4io/px4-dev"
outputs:
docker_tag: ${{ steps.set-outputs.outputs.docker_tag }}
steps:
- name: Set pull request docker tag
if: github.event_name == 'pull_request'
run: echo "DOCKER_TAG=px4io/px4-dev:${{ github.sha }}" >> $GITHUB_ENV
- name: Set push docker tag
if: github.event_name == 'push'
run: echo "DOCKER_TAG=px4io/px4-dev:latest" >> $GITHUB_ENV
- name: Set release docker tag
if: github.event_name == 'release'
run: echo "DOCKER_TAG=px4io/px4-dev:${{ github.event.release.name }}" >> $GITHUB_ENV
- name: Set docker tag outputs
id: set-outputs
run: echo "::set-output name=docker_tag::$DOCKER_TAG"
build_docker:
name: Build Docker image
runs-on: ubuntu-latest
needs: set_docker_tag
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Login to Github Registry
uses: docker/login-action@v1
if: github.event_name == 'push'
with:
registry: ghrc.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Set up image builder
uses: docker/setup-buildx-action@v1
- name: Build image
uses: docker/build-push-action@v2
id: docker_build
with:
file: Tools/setup/Dockerfile
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ needs.set_docker_tag.outputs.docker_tag }}
outputs: type=tar,dest=/tmp/px4_docker_image.tar
- name: Save container to artifacts
uses: actions/upload-artifact@v2
with:
name: px4_docker_image
path: /tmp/px4_docker_image.tar
- name: Push to Github Registry
uses: docker/build-push-action@v2
if: github.event_name == 'push'
- name: Image Digest
run: echo ${{ steps.docker_build.outputs.digest }}
list_targets:
name: Generate target list
runs-on: ubuntu-latest
needs: build_docker
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- name: Checkout Code
uses: actions/checkout@v2
- name: Setup Python
uses: actions/setup-python@v2
with:
python-version: 3.9
#- name: Install Python packages
#run: pip install -r Tools/setup/requirements.txt
#run: pip install kconfiglib
- id: set-matrix
name: Get all nuttx targets
run: echo "::set-output name=matrix::$(./Tools/generate_board_targets_json.py -a)"
build_px4:
name: Build targets with Docker
runs-on: ubuntu-latest
needs: [list_targets, set_docker_tag]
#strategy:
#matrix: ${{ fromJson(needs.enumerate_targets.outputs.matrix) }}
steps:
- name: Get container from artifacts
uses: actions/download-artifact@v2
with:
name: px4_docker_image
path: /tmp
- name: Load Docker image
run: |
docker import /tmp/px4_docker_image.tar ${{ needs.set_docker_tag.outputs.docker_tag }}
docker image ls -a
- name: Checkout Code
uses: actions/checkout@v2
- name: Build targets
#run: |
#docker run --rm -w "${GITHUB_ACTION_PATH}" \
#--env=LOCAL_USER_ID="$(id -u)" \
#--volume=/tmp:/tmp:rw \
#--volume=${GITHUB_ACTION_PATH}:${GITHUB_ACTION_PATH}:rw \
#px4io/px4-dev:${{ needs.set_docker_tag.outputs.docker_tag }} /bin/bash -c "make ${{ matrix.target }}"
run: |
docker run --rm -w ${{ github.workspace }} \
--env=LOCAL_USER_ID="$(id -u)" \
--volume=/tmp:/tmp:rw \
--volume=${{ github.workspace }}:${{ github.workspace }}:rw \
${{ needs.set_docker_tag.outputs.docker_tag }} /bin/bash -c "make px4_fmu-v6x"
-54
View File
@@ -1,54 +0,0 @@
name: Linux Targets
on:
push:
branches:
- 'master'
pull_request:
branches:
- '*'
jobs:
build:
runs-on: ubuntu-latest
container: px4io/px4-dev-armhf:2021-09-08
strategy:
matrix:
config: [
beaglebone_blue_default,
emlid_navio2_default,
px4_raspberrypi_default,
scumaker_pilotpi_default,
]
steps:
- uses: actions/checkout@v1
with:
token: ${{secrets.ACCESS_TOKEN}}
- name: Prepare ccache timestamp
id: ccache_cache_timestamp
shell: cmake -P {0}
run: |
string(TIMESTAMP current_date "%Y-%m-%d-%H;%M;%S" UTC)
message("::set-output name=timestamp::${current_date}")
- name: ccache cache files
uses: actions/cache@v2
with:
path: ~/.ccache
key: ${{matrix.config}}-ccache-${{steps.ccache_cache_timestamp.outputs.timestamp}}
restore-keys: ${{matrix.config}}-ccache-
- name: setup ccache
run: |
mkdir -p ~/.ccache
echo "base_dir = ${GITHUB_WORKSPACE}" > ~/.ccache/ccache.conf
echo "compression = true" >> ~/.ccache/ccache.conf
echo "compression_level = 6" >> ~/.ccache/ccache.conf
echo "max_size = 100M" >> ~/.ccache/ccache.conf
echo "hash_dir = false" >> ~/.ccache/ccache.conf
ccache -s
ccache -z
- name: make ${{matrix.config}}
run: make ${{matrix.config}}
- name: ccache post-run
run: ccache -s
-51
View File
@@ -1,51 +0,0 @@
name: Linux ARM64 Targets
on:
push:
branches:
- 'master'
pull_request:
branches:
- '*'
jobs:
build:
runs-on: ubuntu-latest
container: px4io/px4-dev-aarch64:2021-09-08
strategy:
matrix:
config: [
scumaker_pilotpi_arm64,
]
steps:
- uses: actions/checkout@v1
with:
token: ${{secrets.ACCESS_TOKEN}}
- name: Prepare ccache timestamp
id: ccache_cache_timestamp
shell: cmake -P {0}
run: |
string(TIMESTAMP current_date "%Y-%m-%d-%H;%M;%S" UTC)
message("::set-output name=timestamp::${current_date}")
- name: ccache cache files
uses: actions/cache@v2
with:
path: ~/.ccache
key: ${{matrix.config}}-ccache-${{steps.ccache_cache_timestamp.outputs.timestamp}}
restore-keys: ${{matrix.config}}-ccache-
- name: setup ccache
run: |
mkdir -p ~/.ccache
echo "base_dir = ${GITHUB_WORKSPACE}" > ~/.ccache/ccache.conf
echo "compression = true" >> ~/.ccache/ccache.conf
echo "compression_level = 6" >> ~/.ccache/ccache.conf
echo "max_size = 100M" >> ~/.ccache/ccache.conf
echo "hash_dir = false" >> ~/.ccache/ccache.conf
ccache -s
ccache -z
- name: make ${{matrix.config}}
run: make ${{matrix.config}}
- name: ccache post-run
run: ccache -s
-120
View File
@@ -1,120 +0,0 @@
name: Nuttx Targets
on:
push:
branches:
- 'master'
pull_request:
branches:
- '*'
jobs:
build:
runs-on: ubuntu-latest
container: px4io/px4-dev-nuttx-focal:2021-09-08
strategy:
fail-fast: false
matrix:
config: [
airmind_mindpx-v2,
ark_can-flow,
ark_can-gps,
ark_can-rtk-gps,
atl_mantis-edu,
av_x-v1,
bitcraze_crazyflie,
bitcraze_crazyflie21,
cuav_can-gps-v1,
cuav_nora,
cuav_x7pro,
cubepilot_cubeorange,
cubepilot_cubeyellow,
freefly_can-rtk-gps,
holybro_can-gps-v1,
holybro_durandal-v1,
holybro_kakutef7,
holybro_kakuteh7,
holybro_pix32v5,
matek_h743-slim,
matek_gnss-m9n-f4,
modalai_fc-v1,
modalai_fc-v2,
mro_ctrl-zero-f7,
mro_ctrl-zero-f7-oem,
mro_ctrl-zero-h7,
mro_ctrl-zero-h7-oem,
mro_pixracerpro,
mro_x21,
mro_x21-777,
nxp_fmuk66-e,
nxp_fmuk66-v3,
nxp_fmurt1062-v1,
nxp_ucans32k146,
omnibus_f4sd,
raspberrypi_pico,
px4_fmu-v2,
px4_fmu-v3,
px4_fmu-v4,
px4_fmu-v4pro,
px4_fmu-v5,
px4_fmu-v5x,
px4_fmu-v6u,
px4_fmu-v6x,
spracing_h7extreme,
uvify_core
]
steps:
- uses: actions/checkout@v1
with:
token: ${{secrets.ACCESS_TOKEN}}
- name: Prepare ccache timestamp
id: ccache_cache_timestamp
shell: cmake -P {0}
run: |
string(TIMESTAMP current_date "%Y-%m-%d-%H;%M;%S" UTC)
message("::set-output name=timestamp::${current_date}")
- name: ccache cache files
uses: actions/cache@v2
with:
path: ~/.ccache
key: ${{matrix.config}}-ccache-${{steps.ccache_cache_timestamp.outputs.timestamp}}
restore-keys: ${{matrix.config}}-ccache-
- name: setup ccache
run: |
mkdir -p ~/.ccache
echo "base_dir = ${GITHUB_WORKSPACE}" > ~/.ccache/ccache.conf
echo "compression = true" >> ~/.ccache/ccache.conf
echo "compression_level = 6" >> ~/.ccache/ccache.conf
echo "max_size = 120M" >> ~/.ccache/ccache.conf
echo "hash_dir = false" >> ~/.ccache/ccache.conf
ccache -s
ccache -z
- name: make all_variants_${{matrix.config}}
run: make all_variants_${{matrix.config}}
timeout-minutes: 45
- name: make ${{matrix.config}} bloaty_compileunits
run: make ${{matrix.config}} bloaty_compileunits || true
- name: make ${{matrix.config}} bloaty_inlines
run: make ${{matrix.config}} bloaty_inlines || true
- name: make ${{matrix.config}} bloaty_segments
run: make ${{matrix.config}} bloaty_segments || true
- name: make ${{matrix.config}} bloaty_symbols
run: make ${{matrix.config}} bloaty_symbols || true
- name: make ${{matrix.config}} bloaty_templates
run: make ${{matrix.config}} bloaty_templates || true
- name: make ${{matrix.config}} bloaty_ram
run: make ${{matrix.config}} bloaty_ram || true
- name: make ${{matrix.config}} bloaty_compare_master
run: make ${{matrix.config}} bloaty_compare_master || true
- name: ccache post-run
run: ccache -s
- name: Upload px4 package
uses: actions/upload-artifact@v2
with:
name: px4_package_${{matrix.config}}
path: |
build/**/*.px4
build/**/*.bin
+10 -21
View File
@@ -1,11 +1,17 @@
name: Deploy metadata for all targets
name: Build and deploy all targets with metadata
on:
release:
types: [published]
create:
tags:
- 'v*'
push:
branches:
- 'master'
- 'release/*'
- 'pr-metadata-test'
- 'build_and_deploy_mix'
jobs:
enumerate_targets:
@@ -18,8 +24,8 @@ jobs:
with:
token: ${{secrets.ACCESS_TOKEN}}
- id: set-matrix
run: echo "::set-output name=matrix::$(./Tools/generate_board_targets_json.py)"
build:
run: echo "::set-output name=matrix::$(./Tools/generate_board_targets_json.py -a)"
build_all:
runs-on: ubuntu-latest
needs: enumerate_targets
strategy:
@@ -34,21 +40,4 @@ jobs:
run: make ${{matrix.target}}
- name: parameter & events metadata
run: |
make ${{matrix.target}} ver_gen events_json actuators_json
./src/lib/version/get_git_tag_or_branch_version.sh build/${{ matrix.target }} >> $GITHUB_ENV
cd build/${{ matrix.target }}
mkdir _metadata || true
cp parameters.* events/*.xz actuators.json* _metadata
- uses: jakejarvis/s3-sync-action@master
with:
args: --acl public-read
env:
AWS_S3_BUCKET: 'px4-travis'
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_REGION: 'us-west-1'
SOURCE_DIR: 'build/${{ matrix.target }}/_metadata/'
DEST_DIR: 'Firmware/${{ env.version }}/${{ matrix.target }}/'
run: make ${{matrix.target}} ver_gen events_json actuators_json
+15 -52
View File
@@ -1,35 +1,5 @@
#! /bin/bash
if [ -z ${PX4_DOCKER_REPO+x} ]; then
echo "guessing PX4_DOCKER_REPO based on input";
if [[ $@ =~ .*px4_fmu.* ]]; then
# nuttx-px4fmu-v{1,2,3,4,5}
PX4_DOCKER_REPO="px4io/px4-dev-nuttx-focal:2021-09-08"
elif [[ $@ =~ .*navio2.* ]] || [[ $@ =~ .*raspberry.* ]] || [[ $@ =~ .*beaglebone.* ]] || [[ $@ =~ .*pilotpi.default ]]; then
# beaglebone_blue_default, emlid_navio2_default, px4_raspberrypi_default, scumaker_pilotpi_default
PX4_DOCKER_REPO="px4io/px4-dev-armhf:2021-08-18"
elif [[ $@ =~ .*pilotpi.arm64 ]]; then
# scumaker_pilotpi_arm64
PX4_DOCKER_REPO="px4io/px4-dev-aarch64:latest"
elif [[ $@ =~ .*navio2.* ]] || [[ $@ =~ .*raspberry.* ]] || [[ $@ =~ .*bebop.* ]]; then
# posix_rpi_cross, posix_bebop_default
PX4_DOCKER_REPO="px4io/px4-dev-armhf:2021-08-18"
elif [[ $@ =~ .*clang.* ]] || [[ $@ =~ .*scan-build.* ]]; then
# clang tools
PX4_DOCKER_REPO="px4io/px4-dev-clang:2021-02-04"
elif [[ $@ =~ .*tests* ]]; then
# run all tests with simulation
PX4_DOCKER_REPO="px4io/px4-dev-simulation-bionic:2021-12-11"
fi
else
echo "PX4_DOCKER_REPO is set to '$PX4_DOCKER_REPO'";
fi
# otherwise default to nuttx
if [ -z ${PX4_DOCKER_REPO+x} ]; then
PX4_DOCKER_REPO="px4io/px4-dev-nuttx-focal:2021-09-08"
fi
# docker hygiene
#Delete all stopped containers (including data-only containers)
@@ -38,30 +8,23 @@ fi
#Delete all 'untagged/dangling' (<none>) images
#docker rmi $(docker images -q -f dangling=true)
echo "PX4_DOCKER_REPO: $PX4_DOCKER_REPO";
if [[ -z "${DOCKER_TAG}" ]]; then
TAG_NAME=""
else
TAG_NAME=":${DOCKER_TAG}"
fi
PX4_DOCKER_REPO="px4io/px4-dev$TAG_NAME"
echo $PX4_DOCKER_REPO
PWD=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
SRC_DIR=$PWD/../
CCACHE_DIR=${HOME}/.ccache
mkdir -p "${CCACHE_DIR}"
docker run -it --rm -w "${SRC_DIR}" \
--env=AWS_ACCESS_KEY_ID \
--env=AWS_SECRET_ACCESS_KEY \
--env=BRANCH_NAME \
--env=CCACHE_DIR="${CCACHE_DIR}" \
--env=CI \
--env=CODECOV_TOKEN \
--env=COVERALLS_REPO_TOKEN \
--env=LOCAL_USER_ID="$(id -u)" \
--env=PX4_ASAN \
--env=PX4_MSAN \
--env=PX4_TSAN \
--env=PX4_UBSAN \
--env=TRAVIS_BRANCH \
--env=TRAVIS_BUILD_ID \
--publish 14556:14556/udp \
--volume=${CCACHE_DIR}:${CCACHE_DIR}:rw \
--volume=${SRC_DIR}:${SRC_DIR}:rw \
${PX4_DOCKER_REPO} /bin/bash -c "$1 $2 $3"
--env=LOCAL_USER_ID="$(id -u)" \
--publish 14556:14556/udp \
--volume=/tmp/.X11-unix:/tmp/.X11-unix \
--volume=/tmp:/tmp:rw \
--volume=${SRC_DIR}:${SRC_DIR}:rw \
${PX4_DOCKER_REPO} /bin/bash -c "$1 $2 $3"
+89 -48
View File
@@ -6,14 +6,14 @@ import os
import sys
import json
import re
from kconfiglib import Kconfig
# from kconfiglib import Kconfig
kconf = Kconfig()
# kconf = Kconfig()
# Supress warning output
kconf.warn_assign_undef = False
kconf.warn_assign_override = False
kconf.warn_assign_redun = False
# kconf.warn_assign_undef = False
# kconf.warn_assign_override = False
# kconf.warn_assign_redun = False
source_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), '..')
@@ -23,6 +23,10 @@ parser.add_argument('-v', '--verbose', dest='verbose', action='store_true',
help='Verbose Output')
parser.add_argument('-p', '--pretty', dest='pretty', action='store_true',
help='Pretty output instead of a single line')
parser.add_argument('-a', '--all_variants', dest='all_variants', action='store_true',
help='Prints targets into a single all_variants_* target')
parser.add_argument('-b', '--bloaty', dest='bloaty', action='store_true',
help='Includes all bloaty targets')
args = parser.parse_args()
verbose = args.verbose
@@ -35,48 +39,66 @@ excluded_labels = [
'nolockstep', 'replay', 'test',
'uavcanv1' # TODO: fix and enable
]
bloaty_helpers = [
'bloaty_compileunits',
'bloaty_inlines',
'bloaty_segments',
'bloaty_symbols',
'bloaty_templates',
'bloaty_ram',
'bloaty_compare_master',
]
def process_target(px4board_file, target_name):
ret = None
platform = None
toolchain = None
# def process_target(px4board_file, target_name):
# ret = None
# platform = None
# toolchain = None
if px4board_file.endswith("default.px4board") or \
px4board_file.endswith("recovery.px4board") or \
px4board_file.endswith("bootloader.px4board"):
kconf.load_config(px4board_file, replace=True)
else: # Merge config with default.px4board
default_kconfig = re.sub(r'[a-zA-Z\d_]+\.px4board', 'default.px4board', px4board_file)
kconf.load_config(default_kconfig, replace=True)
kconf.load_config(px4board_file, replace=False)
# if px4board_file.endswith("default.px4board") or \
# px4board_file.endswith("recovery.px4board") or \
# px4board_file.endswith("bootloader.px4board"):
# kconf.load_config(px4board_file, replace=True)
# else: # Merge config with default.px4board
# default_kconfig = re.sub(r'[a-zA-Z\d_]+\.px4board', 'default.px4board', px4board_file)
# kconf.load_config(default_kconfig, replace=True)
# kconf.load_config(px4board_file, replace=False)
if "BOARD_TOOLCHAIN" in kconf.syms:
toolchain = kconf.syms["BOARD_TOOLCHAIN"].str_value
# if "BOARD_TOOLCHAIN" in kconf.syms:
# toolchain = kconf.syms["BOARD_TOOLCHAIN"].str_value
if "BOARD_PLATFORM" in kconf.syms:
platform = kconf.syms["BOARD_PLATFORM"].str_value
# if "BOARD_PLATFORM" in kconf.syms:
# platform = kconf.syms["BOARD_PLATFORM"].str_value
assert platform, f"PLATFORM not found in {px4board_file}"
# assert platform, f"PLATFORM not found in {px4board_file}"
if platform not in excluded_platforms:
# get the container based on the platform and toolchain
container = platform
if platform == 'posix':
container = 'base-focal'
if toolchain:
if toolchain.startswith('aarch64'):
container = 'aarch64'
elif toolchain == 'arm-linux-gnueabihf':
container = 'armhf'
else:
if verbose: print(f'possibly unmatched toolchain: {toolchain}')
elif platform == 'nuttx':
container = 'nuttx-focal'
# if platform not in excluded_platforms:
# # get the container based on the platform and toolchain
# container = platform
# if platform == 'posix':
# container = 'base-focal'
# if toolchain:
# if toolchain.startswith('aarch64'):
# container = 'aarch64'
# elif toolchain == 'arm-linux-gnueabihf':
# container = 'armhf'
# else:
# if verbose: print(f'possibly unmatched toolchain: {toolchain}')
# elif platform == 'nuttx':
# container = 'nuttx-focal'
ret = {'target': target_name, 'container': container}
# ret = {'target': target_name, 'container': container}
return ret
# return ret
def process_bloaty(target_path, target_name):
response = []
for bloat in bloaty_helpers:
bloaty_name = target_name + ' ' + bloat + ' || true'
processed_target = process_target(target_path, bloaty_name)
response.append(processed_target)
return response
# Look up boards from each manufacturer
for manufacturer in os.scandir(os.path.join(source_dir, 'boards')):
if not manufacturer.is_dir():
continue
@@ -85,18 +107,37 @@ for manufacturer in os.scandir(os.path.join(source_dir, 'boards')):
continue
for board in os.scandir(manufacturer.path):
# Only boards are directories don't proceed if otherwise
if not board.is_dir():
continue
for files in os.scandir(board.path):
if files.is_file() and files.name.endswith('.px4board'):
label = files.name[:-9]
target_name = manufacturer.name + '_' + board.name + '_' + label
if label in excluded_labels:
if verbose: print(f'excluding label {label} ({target_name})')
continue
target = process_target(files.path, target_name)
if target is not None:
build_configs.append(target)
if args.all_variants:
# The all_variants target makes all targets for a board
target_name = 'all_variants_' + manufacturer.name + '_' + board.name
default_target_path = f'{board.path}/default.px4board'
# target = process_target(default_target_path, target_name)
target = {'target': target_name}
if target is not None:
build_configs.append(target)
if args.bloaty and target is not None:
# bloaty targets
bloat_target_name = manufacturer.name + '_' + board.name
# bloaty_targets = process_bloaty(default_target_path, bloat_target_name)
bloaty_targets = {'target': bloat_target_name}
build_configs += bloaty_targets
else:
# Each board can have multiple variant targets
for files in os.scandir(board.path):
if files.is_file() and files.name.endswith('.px4board'):
label = files.name[:-9]
target_name = manufacturer.name + '_' + board.name + '_' + label
if label in excluded_labels:
if verbose: print(f'excluding label {label} ({target_name})')
continue
# target = process_target(files.path, target_name)
target = {'target': target_name}
if target is not None:
build_configs.append(target)
github_action_config = { 'include': build_configs }
+55
View File
@@ -0,0 +1,55 @@
#
# PX4 base development environment
#
FROM ubuntu:20.04
LABEL maintainer="Daniel Agar <daniel@agar.ca>, Ramon Roche <mrpollo@gmail.com>"
ENV DEBIAN_FRONTEND noninteractive
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
# Installing required utilities
RUN apt-get update && apt-get -y --quiet --no-install-recommends install \
ca-certificates \
gnupg \
lsb-core \
lsb-release \
sudo \
software-properties-common \
wget \
gosu \
;
# Install PX4 Requirements
COPY Tools/setup/requirements.txt /tmp/requirements.txt
COPY Tools/setup/ubuntu.sh /tmp/ubuntu.sh
# We support pre-downloading the gcc arm none eabi compiler
# to speed up build times, if the file is not present when
# building, the ubuntu.sh script will download it from source
COPY *gcc-arm-none-eabi-9-2020-q2-update-linux.tar.bz2 /tmp/gcc-arm-none-eabi-9-2020-q2-update-linux.tar.bz2
# The PATH env is set within ubuntu.sh, but given how we
# are running the image using `gosu` to avoid read-only problems
# with the filesystem the env variable does not persist
ENV PATH="/opt/gcc-arm-none-eabi-9-2020-q2-update/bin:$PATH"
ENV PATH="/opt/jdk-14.0.2+12/bin:$PATH"
RUN bash /tmp/ubuntu.sh --from-docker --with-java --with-rtps --no-sim-tools
ENV DISPLAY :99
ENV FASTRTPSGEN_DIR="/usr/local/bin/"
ENV TERM=xterm
ENV TZ=UTC
# SITL UDP PORTS
EXPOSE 14556/udp
EXPOSE 14557/udp
# create user with id 1001 (jenkins docker workflow default)
RUN useradd --shell /bin/bash -u 1001 -c "" -m user && usermod -a -G dialout user
# create and start as LOCAL_USER_ID
COPY Tools/setup/docker-entrypoint.sh /usr/local/bin/entrypoint.sh
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
CMD ["/bin/bash"]
+28
View File
@@ -0,0 +1,28 @@
#!/bin/bash
# Start virtual X server in the background
# - DISPLAY default is :99, set in dockerfile
# - Users can override with `-e DISPLAY=` in `docker run` command to avoid
# running Xvfb and attach their screen
if [[ -x "$(command -v Xvfb)" && "$DISPLAY" == ":99" ]]; then
echo "Starting Xvfb"
Xvfb :99 -screen 0 1600x1200x24+32 &
fi
# Check if the ROS_DISTRO is passed and use it
# to source the ROS environment
if [ -n "${ROS_DISTRO}" ]; then
source "/opt/ros/$ROS_DISTRO/setup.bash"
fi
# Use the LOCAL_USER_ID if passed in at runtime
if [ -n "${LOCAL_USER_ID}" ]; then
echo "Starting with UID : $LOCAL_USER_ID"
# modify existing user's id
usermod -u $LOCAL_USER_ID user
# run as user
exec gosu user "$@"
else
exec "$@"
fi
+277 -155
View File
@@ -2,20 +2,22 @@
set -e
## Bash script to setup PX4 development environment on Ubuntu LTS (20.04, 18.04, 16.04).
## Bash script to setup PX4 development environment on Ubuntu LTS (20.04, 18.04).
## Can also be used in docker.
##
## Installs:
## - Common dependencies and tools for nuttx, jMAVSim, Gazebo
## - Common dependencies and tools for NuttX, jMAVSim, Gazebo
## - NuttX toolchain (omit with arg: --no-nuttx)
## - jMAVSim and Gazebo9 simulator (omit with arg: --no-sim-tools)
##
## Not Installs:
## - FastRTPS and FastCDR
## - jMAVSim and Gazebo simulator (omit with arg: --no-sim-tools)
## Optional:
## - FastRTPS and FastCDR (with args: --with-rtps)
INSTALL_NUTTX="true"
INSTALL_SIM="true"
INSTALL_ARCH=`uname -m`
INSTALL_RTPS="false"
INSTALL_JAVA="false"
INSIDE_DOCKER="false"
# Parse arguments
for arg in "$@"
@@ -28,19 +30,19 @@ do
INSTALL_SIM="false"
fi
done
if [[ $arg == "--with-rtps" ]]; then
INSTALL_RTPS="true"
fi
# detect if running in docker
if [ -f /.dockerenv ]; then
echo "Running within docker, installing initial dependencies";
apt-get --quiet -y update && DEBIAN_FRONTEND=noninteractive apt-get --quiet -y install \
ca-certificates \
gnupg \
lsb-core \
sudo \
wget \
;
fi
if [[ $arg == "--with-java" ]]; then
INSTALL_JAVA="true"
fi
if [[ $arg == "--from-docker" ]]; then
INSIDE_DOCKER="true"
fi
done
# script directory
DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
@@ -69,88 +71,119 @@ elif [[ "${UBUNTU_RELEASE}" == "20.04" ]]; then
echo "Ubuntu 20.04"
fi
VERBOSE_BAR="####################"
echo
echo $VERBOSE_BAR
echo "#⚡️ Starting PX4 Dependency Installer for Ubuntu ${UBUNTU_RELEASE} (${INSTALL_ARCH})"
echo "# Options:
#
# - Install NuttX = ${INSTALL_NUTTX}
# - Install Java = ${INSTALL_JAVA}
# - Install Simulation = ${INSTALL_SIM}
# - Install RTPS = ${INSTALL_RTPS}"
echo $VERBOSE_BAR
echo
echo
echo "Installing PX4 general dependencies"
echo $VERBOSE_BAR
echo "🍻 Installing System Dependencies"
echo $VERBOSE_BAR
echo
sudo apt-get update -y --quiet
sudo DEBIAN_FRONTEND=noninteractive apt-get -y --quiet --no-install-recommends install \
astyle \
build-essential \
cmake \
cppcheck \
file \
g++ \
gcc \
gdb \
git \
lcov \
libxml2-dev \
libxml2-utils \
make \
ninja-build \
python3 \
python3-dev \
python3-pip \
python3-setuptools \
python3-wheel \
rsync \
shellcheck \
unzip \
zip \
;
g++ \
gcc \
gdb \
astyle \
cmake \
cppcheck \
file \
git \
lcov \
libxml2-dev \
libxml2-utils \
make \
ninja-build \
python3 \
python3-dev \
python3-pip \
python3-setuptools \
python3-wheel \
rsync \
shellcheck \
unzip \
zip \
libssl-dev \
;
# Python3 dependencies
# Python 3 dependencies
echo
echo "Installing PX4 Python3 dependencies"
echo $VERBOSE_BAR
echo "🍻 Installing Python dependencies"
echo $VERBOSE_BAR
echo
if [ -n "$VIRTUAL_ENV" ]; then
# virtual envrionments don't allow --user option
python -m pip install -r ${DIR}/requirements.txt
python -m pip install -r ${DIR}/requirements.txt
else
# older versions of Ubuntu require --user option
python3 -m pip install --user -r ${DIR}/requirements.txt
if [[ $INSIDE_DOCKER == "true" ]]; then
# when running inside a docker container we don't need to install
# under --user since the installer user is root
# its best to install packages globaly for any user to find
python3 -m pip install -r /tmp/requirements.txt
else
python3 -m pip install --user -r ${DIR}/requirements.txt
fi
fi
# NuttX toolchain (arm-none-eabi-gcc)
if [[ $INSTALL_NUTTX == "true" ]]; then
echo
echo "Installing NuttX dependencies"
echo $VERBOSE_BAR
echo "🍻 Installing NuttX dependencies"
echo $VERBOSE_BAR
echo
sudo DEBIAN_FRONTEND=noninteractive apt-get -y --quiet --no-install-recommends install \
automake \
binutils-dev \
bison \
build-essential \
flex \
g++-multilib \
gcc-multilib \
gdb-multiarch \
genromfs \
gettext \
gperf \
libelf-dev \
libexpat-dev \
libgmp-dev \
libisl-dev \
libmpc-dev \
libmpfr-dev \
libncurses5 \
libncurses5-dev \
libncursesw5-dev \
libtool \
pkg-config \
screen \
texinfo \
u-boot-tools \
util-linux \
vim-common \
;
if [[ "${UBUNTU_RELEASE}" == "20.04" ]]; then
sudo DEBIAN_FRONTEND=noninteractive apt-get -y --quiet --no-install-recommends install \
kconfig-frontends \
;
fi
automake \
binutils-dev \
bison \
flex \
genromfs \
gettext \
gperf \
libelf-dev \
libexpat-dev \
libgmp-dev \
libisl-dev \
libmpc-dev \
libmpfr-dev \
libncurses5 \
libncurses5-dev \
libncursesw5-dev \
libtool \
pkg-config \
screen \
texinfo \
u-boot-tools \
util-linux \
vim-common \
g++-arm-linux-gnueabihf \
gcc-arm-linux-gnueabihf \
g++-aarch64-linux-gnu \
gcc-aarch64-linux-gnu \
;
if [[ "${UBUNTU_RELEASE}" == "20.04" ]]; then
sudo DEBIAN_FRONTEND=noninteractive apt-get -y --quiet --no-install-recommends install \
kconfig-frontends \
;
fi
if [ -n "$USER" ]; then
@@ -158,97 +191,186 @@ if [[ $INSTALL_NUTTX == "true" ]]; then
sudo usermod -a -G dialout $USER
fi
# arm-none-eabi-gcc
NUTTX_GCC_VERSION="9-2020-q2-update"
NUTTX_GCC_VERSION_SHORT="9-2020q2"
NUTTX_GCC_VERSION="9-2020-q2-update"
NUTTX_GCC_VERSION_SHORT="9-2020q2"
echo
echo $VERBOSE_BAR
echo "🍻 Verifying proper gcc version (${NUTTX_GCC_VERSION}), and installing if not found"
echo
source $HOME/.profile # load changed path for the case the script is reran before relogin
source $HOME/.profile # load changed path for the case the script is reran before relogin
if [ $(which arm-none-eabi-gcc) ]; then
GCC_VER_STR=$(arm-none-eabi-gcc --version)
GCC_FOUND_VER=$(echo $GCC_VER_STR | grep -c "${NUTTX_GCC_VERSION}")
fi
GCC_VER_STR=$(arm-none-eabi-gcc --version)
GCC_VER_FOUND=$(echo $GCC_VER_STR | grep -c "${NUTTX_GCC_VERSION}")
fi
if [[ "$GCC_FOUND_VER" == "1" ]]; then
echo "arm-none-eabi-gcc-${NUTTX_GCC_VERSION} found, skipping installation"
if [[ $(echo $GCC_VER_STR | grep -c "${NUTTX_GCC_VERSION}") == "1" ]]; then
echo "📌 Skipping installation, the arm cross compiler was found"
echo $VERBOSE_BAR
echo
else
echo "Installing arm-none-eabi-gcc-${NUTTX_GCC_VERSION}";
wget -O /tmp/gcc-arm-none-eabi-${NUTTX_GCC_VERSION}-linux.tar.bz2 https://armkeil.blob.core.windows.net/developer/Files/downloads/gnu-rm/${NUTTX_GCC_VERSION_SHORT}/gcc-arm-none-eabi-${NUTTX_GCC_VERSION}-${INSTALL_ARCH}-linux.tar.bz2 && \
sudo tar -jxf /tmp/gcc-arm-none-eabi-${NUTTX_GCC_VERSION}-linux.tar.bz2 -C /opt/;
else
echo "📌 The arm cross compiler was not found";
echo " * Installing arm-none-eabi-gcc-${NUTTX_GCC_VERSION}";
# The arm cross compiler hosting provider is known to throttle download speeds
# for users who reach a certain limit of downloads in a given time frame
# for this reason we allow for using a previously downloaded file
# this is specially helpful when debugging this installer script
# from within a container COMPILER_PATH="/tmp/gcc-arm-none-eabi-${NUTTX_GCC_VERSION}-linux.tar.bz2"
COMPILER_NAME="gcc-arm-none-eabi-${NUTTX_GCC_VERSION}"
COMPILER_PATH="/tmp/$COMPILER_NAME-linux.tar.bz2"
if [ ! -f "$COMPILER_PATH" ]; then
wget -O $COMPILER_PATH https://armkeil.blob.core.windows.net/developer/Files/downloads/gnu-rm/${NUTTX_GCC_VERSION_SHORT}/${COMPILER_NAME}-${INSTALL_ARCH}-linux.tar.bz2
fi
sudo tar -jxf $COMPILER_PATH -C /opt/;
# add arm-none-eabi-gcc to user's PATH
exportline="export PATH=/opt/gcc-arm-none-eabi-${NUTTX_GCC_VERSION}/bin:\$PATH"
# add arm-none-eabi-gcc to user's PATH
exportline="export PATH=\"/opt/${COMPILER_NAME}/bin:\$PATH\""
if [[ $INSIDE_DOCKER == "true" ]]; then
# when running on a docker container its best to set the environment globally
# since we don't know which user is going to be running commands on the container
touch /etc/profile.d/px4env.sh
echo $exportline >> /etc/profile.d/px4env.sh
else
if grep -Fxq "$exportline" $HOME/.profile; then
echo "${NUTTX_GCC_VERSION} path already set.";
else
echo $exportline >> $HOME/.profile;
fi
fi
echo " * arm-none-eabi-gcc (${NUTTX_GCC_VERSION}) Installed Succesful to /opt/${COMPILER_NAME}/bin"
echo $VERBOSE_BAR
echo
fi
fi
# Install JAVA
if [[ $INSTALL_JAVA == "true" ]]; then
JDK_VERSION="14.0.2_12"
echo
echo $VERBOSE_BAR
echo "🍻 Installing Java JDK
* Version: $JDK_VERSION
* Path: /opt/jdk-14.0.2+12"
echo $VERBOSE_BAR
echo
JDK_DOWNLOAD="/tmp/OpenJDK14U-jdk_x64_linux_hotspot_$JDK_VERSION.tar.gz"
wget -O $JDK_DOWNLOAD https://github.com/AdoptOpenJDK/openjdk14-binaries/releases/download/jdk-14.0.2%2B12/OpenJDK14U-jdk_x64_linux_hotspot_14.0.2_12.tar.gz
sudo tar -xzf $JDK_DOWNLOAD -C /opt/
export PATH="/opt/jdk-14.0.2+12/bin:$PATH"
fi
# Fast-RTPS
if [[ $INSTALL_RTPS == "true" ]]; then
echo
echo $VERBOSE_BAR
echo "🍻 Installing Fast-RTPS"
echo $VERBOSE_BAR
echo
GRADLE_VERSION="6.4.1"
wget -O "/tmp/gradle-$GRADLE_VERSION-bin.zip" "https://services.gradle.org/distributions/gradle-$GRADLE_VERSION-bin.zip" \
&& unzip -d /opt/gradle "/tmp/gradle-$GRADLE_VERSION-bin.zip"
export PATH="$PATH:/opt/gradle/gradle-$GRADLE_VERSION/bin"
# Intall foonathan_memory from source as it is required to Fast-RTPS >= 1.9
git clone https://github.com/eProsima/foonathan_memory_vendor.git /tmp/foonathan_memory \
&& cd /tmp/foonathan_memory \
&& mkdir build && cd build \
&& cmake .. \
&& cmake --build . --target install -- -j $(nproc)
# Fast-DDS (Fast-RTPS 2.1.1)
git clone --recursive https://github.com/eProsima/Fast-DDS.git -b v2.1.1 /tmp/FastRTPS-2.1.1 \
&& cd /tmp/FastRTPS-2.1.1 \
&& mkdir build && cd build \
&& cmake -DTHIRDPARTY=ON -DSECURITY=ON .. \
&& cmake --build . --target install -- -j $(nproc)
# Fast-RTPS-Gen 1.0.4
git clone --recursive https://github.com/eProsima/Fast-DDS-Gen.git -b v1.0.4 /tmp/Fast-RTPS-Gen-1.0.4 \
&& cd /tmp/Fast-RTPS-Gen-1.0.4 \
&& gradle assemble \
&& gradle install
if grep -Fxq "$exportline" $HOME/.profile; then
echo "${NUTTX_GCC_VERSION} path already set.";
else
echo $exportline >> $HOME/.profile;
fi
fi
fi
# Simulation tools
if [[ $INSTALL_SIM == "true" ]]; then
echo
echo "Installing PX4 simulation dependencies"
echo
echo $VERBOSE_BAR
echo "🍻 Installing PX4 Simulation Tools"
echo
# General simulation dependencies
sudo DEBIAN_FRONTEND=noninteractive apt-get -y --quiet --no-install-recommends install \
bc \
;
if [[ "${UBUNTU_RELEASE}" == "18.04" ]]; then
gazebo_version=9
elif [[ "${UBUNTU_RELEASE}" == "20.04" ]]; then
gazebo_version=11
fi
if [[ "${UBUNTU_RELEASE}" == "18.04" ]]; then
java_version=11
gazebo_version=9
elif [[ "${UBUNTU_RELEASE}" == "20.04" ]]; then
java_version=13
gazebo_version=11
else
java_version=14
gazebo_version=11
fi
# Java (jmavsim or fastrtps)
sudo DEBIAN_FRONTEND=noninteractive apt-get -y --quiet --no-install-recommends install \
ant \
openjdk-$java_version-jre \
openjdk-$java_version-jdk \
libvecmath-java \
;
echo " * Gazebo Version $gazebo_version"
echo $VERBOSE_BAR
# Set Java 11 as default
sudo update-alternatives --set java $(update-alternatives --list java | grep "java-$java_version")
# General simulation dependencies
sudo DEBIAN_FRONTEND=noninteractive apt-get -y --quiet --no-install-recommends install \
bc \
ant \
libvecmath-java \
;
# Gazebo
sudo sh -c 'echo "deb http://packages.osrfoundation.org/gazebo/ubuntu-stable `lsb_release -cs` main" > /etc/apt/sources.list.d/gazebo-stable.list'
wget http://packages.osrfoundation.org/gazebo.key -O - | sudo apt-key add -
# Update list, since new gazebo-stable.list has been added
sudo apt-get update -y --quiet
sudo DEBIAN_FRONTEND=noninteractive apt-get -y --quiet --no-install-recommends install \
dmidecode \
gazebo$gazebo_version \
gstreamer1.0-plugins-bad \
gstreamer1.0-plugins-base \
gstreamer1.0-plugins-good \
gstreamer1.0-plugins-ugly \
gstreamer1.0-libav \
libeigen3-dev \
libgazebo$gazebo_version-dev \
libgstreamer-plugins-base1.0-dev \
libimage-exiftool-perl \
libopencv-dev \
libxml2-utils \
pkg-config \
protobuf-compiler \
;
# Installing Gazebo and dependencies
# Setup OSRF Gazebo repository
sudo sh -c 'echo "deb http://packages.osrfoundation.org/gazebo/ubuntu-stable `lsb_release -cs` main" > /etc/apt/sources.list.d/gazebo-stable.list'
wget http://packages.osrfoundation.org/gazebo.key -O - | sudo apt-key add -
# Update list, since new gazebo-stable.list has been added
sudo apt-get update -y --quiet
sudo DEBIAN_FRONTEND=noninteractive apt-get -y --quiet --no-install-recommends install \
dmidecode \
gazebo$gazebo_version \
gstreamer1.0-plugins-bad \
gstreamer1.0-plugins-base \
gstreamer1.0-plugins-good \
gstreamer1.0-plugins-ugly \
gstreamer1.0-libav \
libeigen3-dev \
libgazebo$gazebo_version-dev \
libgstreamer-plugins-base1.0-dev \
libimage-exiftool-perl \
libopencv-dev \
libxml2-utils \
pkg-config \
protobuf-compiler \
;
if sudo dmidecode -t system | grep -q "Manufacturer: VMware, Inc." ; then
# fix VMWare 3D graphics acceleration for gazebo
echo "export SVGA_VGPU10=0" >> ~/.profile
fi
if sudo dmidecode -t system | grep -q "Manufacturer: VMware, Inc." ; then
# fix VMWare 3D graphics acceleration for gazebo
echo "export SVGA_VGPU10=0" >> ~/.profile
fi
fi
if [[ $INSTALL_NUTTX == "true" ]]; then
echo
echo "Relogin or reboot computer before attempting to build NuttX targets"
if [[ $INSIDE_DOCKER == "true" ]]; then
# cleanup installation
rm -rf /tmp/
fi
if [[ $INSIDE_DOCKER == "false" ]] && [[ $INSTALL_NUTTX == "true" ]]; then
echo
echo $VERBOSE_BAR
echo "💡 We recommend you relogin/reboot before attempting to build NuttX targets"
echo $VERBOSE_BAR
echo
fi
echo
echo
echo $VERBOSE_BAR
echo "#⚡️ PX4 Dependency Installer Ended Succesfully
#
# For more information on PX4 Autopilot check out our docs
# at docs.px4.io, if you find a bug please file an issue
# on GitHub https://github.com/px4/px4-autopilot"
echo $VERBOSE_BAR
echo