ccws

• Introduction
  • Features
  • Build profiles
  • Execution profiles
  • Dependencies
• Usage
  • Initial setup
  • Compilation
  • Running
  • Testing
  • Documentation
  • Build performance
  • Debian package generation
  • Cross-compilation
• Advanced usage
  • CCWS docker image
  • AI features / agentic coding
  • CCWS in CI
  • Extending CCWS
• Known issues
  • Crosscompilation and binary packages
  • Sanitizers
  • Other
• Related software
• TODO
  • Bookmarks (not going to be supported)

Introduction

CI status

CCWS is a development environment for ROS, which integrates functionality of traditional catkin workspaces and CI pipelines in order to facilitate (cross-)compilation, testing, linting, documetation, binary package generation, and agentic coding. It is intended to be used both as a CI/CD backbone and a working environment for developers. Note that CCWS is not meant to be a complete solution, but rather a basis for development of a vendor-specific workflow.

CCWS is ROS version agnostic, and should work for both ROS1 and ROS2 in most cases.

Features

• Build profiles – sets of configurations for build process, e.g., cmake toolchain, colcon configuration, environment variables, etc. Profiles do not conflict with each other and can be used in parallel without using separate clones of the workspace and packages.

• Execution profiles – simple shell mixins that are intended to modify run time environment, e.g., execute nodes in valgrind, alter node crash handling, etc.

• A number of features implemented via build profiles:

  • Cross compilation to several common platforms.

  • Documentation generation for the whole workspace or selected packages using doxygen, similar to https://github.com/mikepurvis/catkin_tools_document.

  • Linting with clang-tidy and scan_build.

  • Various static checks as in https://github.com/sscpac/statick, in particular:

    • cppcheck
    • catkin_lint https://github.com/fkie/catkin_lint
    • yamllint
    • shellcheck

  • Binary debian package generation.

  • Workspace indexing for coding agents

• Package template which demonstrates how to use some of the features.

• The number of parallel jobs can be selected based on available RAM instead of CPU cores, since RAM is likely to be the limiting factor.

• Based entirely on make and shell scripts. All scripts and configurations are kept in the workspace and easy to adjust for specific needs.

• Integration with AI coding agent (qwen).

Build profiles

Profile configurations are located in ccws/profiles/build, common subdirectory contains default parameters, which can be overriden by specific profiles:

• [default] reldebug – default compiler, cmake build type is RelWithDebInfo.
• release – default compiler, cmake build type is Release, tests are disabled.
• clang – build with clang compiler.
• clang_format – formats source files using clang-format.
• scan_build – compile with clang using scan_build and clang-tidy for static checks. clang-tidy parameters are defined in cmake toolchain and must be enabled in packages as shown in package template CMakeLists.
• clang_tidy – a simplified version of scan_build without scan_build.
• thread_sanitizer – compilation with thread sanitizer.
• addr_undef_sanitizers – compilation with address and undefined behavior sanitizers.
• static_checks – static checkers and their configuration.
• doxygen – doxygen and its configuration.
• codebase_memory_mcp – indexes the source space using https://github.com/DeusData/codebase-memory-mcp.
• cross_raspberry_pi – cross-compilation for Raspberry Pi.
• cross_arm64 – cross-compilation for arm64 (using docker containers).
• clangd – collects compilation commands from another profile and generates clangd configuration file in the workspace root.
• deb – debian package generation (see below).
• notests – wraps another build profile to disable test compilation, e.g., BUILD_PROFILE=notests,reldebug.
• deprecate – wraps another build profile to enable warnings about deprecated declarations, which are disabled by default.
• coverage – wraps another build profile to enable coverage data collection.
• iwyu – include-what-you-use checks (https://github.com/include-what-you-use/include-what-you-use).
• build_profiler – compilation time analysis using ClangBuildAnalyzer https://github.com/aras-p/ClangBuildAnalyzer.

Legacy:

• cross_jetson_xavier – cross-compilation for Jetson Xavier.
• cross_jetson_nano – cross-compilation for Jetson Nano.

Execution profiles

Execution profiles set environment variables that can be used in launch scripts to alter run time behavior as demonstrated in ccws/pkg_template/catkin/launch/bringup.launch, currently available profiles are:

• common – a set of common ROS parameters, e.g., ROS_HOME, it is automatically included in binary packages.
• test – sets CCWS_NODE_CRASH_ACTION variable so that nodes that respect it become required, i.e., termination of such nodes would result in crash of test scripts and can thus be easily detected.
• valgrind – sets CCWS_NODE_LAUNCH_PREFIX to valgrind and some variables that control behavior of valgrind.
• core_pattern – sets core pattern to save core files in the artifacts directory.
• address_sanitizer – helper for addr_undef_sanitizers profile.

Execution profiles have no effect on build process and are taken into account in *test* targets or debian packages. test execution profile is always used in tests and additional profiles can be provided with EXEC_PROFILE="<profile1> <profile2>". These targets load profiles using setup.bash script located in the root folder of CCWS, which can also be used manually, e.g., source setup.bash [<build_profile> [<exec_profile1> ...]]. Note that the setup script always includes common profile, and uses test execution profile if no other execution profiles are specified.

Dependencies

Dependencies can be installed using make bp_install_build BUILD_PROFILE=<profile>[,<profile>...], which is going to install the following tools and profile specific dependencies:

• colcon
• cmake
• ccache – can be disabled in cmake toolchains
• buildcache – alternative compiler cache to ccache, selected via CCWS_BUILD_CACHE_TOOL. See Build cache tool below.
• wget

Usage

See .ccws/test_main.mk for command usage hints.

Initial setup

• Override developer and vendor specific parameters by adding them to make/config.mk, available parameters can be found in the top section of Makefile.
• Install profile dependencies using make bp_install_build BUILD_PROFILE=<profile> targets, cross compilation profiles would require some extra steps as described below. In some minimalistic environments you may need to run ./ccws/tools/bin/bootstrap.sh before using bp_install_build target in order to install make and other utils.
• Clone packages in src subdirectory, or create new using make new PKG=<pkg>.
• Install package dependencies using make dep_install. Dependencies can be installed selectively by providing package name with PKG parameter and/or dependency type with CCWS_DEP_TYPE (build|exec|test).

Compilation

• make build PKG="<pkg>" where <pkg> is one or more space separated package names.
• make <pkg> – a shortcut for make build, but <pkg> can be a substring of package name. All packages matching the given substring will be built.
• The number of jobs can be overriden with JOBS=X parameter.
• make build PKG=<pkg> BUILD_PROFILE=scan_build overrides default profile.

Running

• Source setup.bash <profile> to be able to use packages. Setup scripts generated by colcon can also be used directly, e.g., install/<profile>/local_setup.sh, but in this case some of CCWS functionality won’t be available.

Testing

• make test PKG=<pkg> test with colcon, or make wstest to test all.
• make ctest PKG=<pkg> bypass colcon and run ctest directly or make wsctest to test all.

Documentation

• make BUILD_PROFILE=doxygen, firefox artifacts/doxygen/index.html
• See example at http://www.sherikov.net/sharf/

Build performance

Caching

CCWS configures the cmake CMAKE_<LANG>_COMPILER_LAUNCHER variables to wrap every supported language with a compiler cache:

• CCWS_BUILD_CACHE_TOOL=ccache (the default) wraps C, C++, Fortran, Objective-C and Objective-C++ with ccache.
• CCWS_BUILD_CACHE_TOOL=buildcache https://gitlab.com/bits-n-bites/buildcache wraps C and C++ with buildcache – the only languages buildcache exposes to cmake. The main advantage of this tool is clang-tidy support which is absent in ccache (and sccache).

The selection is controlled by the environment variable CCWS_BUILD_CACHE_TOOL (default: ccache). The cache directory follows CCWS_CACHE for both tools ($CCWS_CACHE/ccache and $CCWS_CACHE/buildcache).

Both tools honour the standard disable switch, so an individual run can be uncached with CCACHE_DISABLE=1 or BUILDCACHE_DISABLE=1.

In addition to compiler cache CCWS implements a simple cmake caching inspired by https://github.com/cristianadam/cmake-checks-cache and https://github.com/polysquare/cmake-forward-cache: it is similar to cmake’s own caching mechanism, but works across all packages in the workspace – this allows packages to benefit from cmake checks performed earlier for other packages. The effect of this caching would be noticable on clean builds of workspaces with large number of packages. Can be disabled with CCWS_CMAKE_CHECKS_CACHE_DISABLE environment variable.

Analysis

Compilation bottlenecks can be identified using build_profiler (https://github.com/aras-p/ClangBuildAnalyzer) profile, which relies on the -ftime-trace feature of clang, see ccws/profiles/build/build_profiler/README.md.

Additionally https://github.com/include-what-you-use/include-what-you-use can be used with iwyu profile to optimize header inclusions. iwyu relies on compilation database collected by another profile, which has to be built first. It also inherits exceptions from other chained profiles, so it can be used as this BUILD_PROFILE=iwyu,static_checks,reldebug.

Debian package generation

Overview

CCWS takes a somewhat uncommon approach to binary package generation which is a middle ground between traditional ROS (1 package = 1 deb) and docker containers: all packages built in the workspace are packed together into a single debian ‘superpackage’. Unlike bloom CCWS generates binary packages directly instead of generating source packages first.

Binary package generation is implemented as a build profile mixin that can be overlayed over an arbitrary build profile: make <pkg> BUILD_PROFILE=deb,reldebug.

CCWS approach has a number of advantages:

• Binary compatibility issues are minimized compared to traditional ROS approach:

  • no need to worry about compatibilities between multiple standalone binary packages and perform ABI checks;

  • if base ROS packages are included, it is also possible to avoid binary incompatibilities between syncs of the same ROS release (those actually happen).

• Package repository management can be sloppier compared to ROS when it comes to tags, versions, git submodules, etc, e.g., there is no need to maintain release repos for all packages.

• Debian ‘superpackages’ are easier to handle than both standalone packages and docker containers, e.g., they can be generated by developers from their working branches and easily copied and installed on the target.

• Debian packages have some advantages over docker containers in general:

  • Zero overhead during execution.

  • Straightforward access to hardware.

  • Easy installation of system services, udev rules, configs, etc.

• Multiple variants of binary ‘superpackage’ can be installed simultaneously if they are built using different VERSION parameter. Note that it alters installation path, so some workspace packages that are being “smart” with cmake may require cleaning build directory if they have been built earlier.

Building packages

Generally, it is necessary to install packages to the filesystem root during compilation in order to get all paths right in catkin cmake files and properly install system files. CCWS avoids this using proot similarly to cross-compilation profiles.

By default (CCWS_DEB_SUBPACKAGES=OFF) a single binary package is generated. When CCWS_DEB_SUBPACKAGES=ON a stripped main package is generated and several additional packages:

• dev – development files: headers, static libraries, cmake package configurations, pkg-config files. Runtime files (unversioned shared libraries in particular) remain in the main package.
• dbg – debug symbols, which are extracted from binaries and libraries and installed into /usr/lib/debug/<installation path>, build ids are not used, see https://linuxvox.com/blog/where-to-place-find-debug-library-on-linux/ for reference. Binaries in the main package are stripped and linked to the symbols, gdb locates them automatically.
• doc – documentation files: documentation (doc, man) directories and markdown files.

Example: make <pkg> BUILD_PROFILE=deb,reldebug CCWS_DEB_SUBPACKAGES=ON. Subpackages follow the naming of the main package with a type-specific suffix. Dependencies are split accordingly: the main package depends on the run dependencies of the workspace packages, dev additionally depends on their build dependencies, dbg depends on the main package (= version), doc has no dependencies. The ccws build metadata directory is shipped in the dev subpackage. Subpackages that end up empty (e.g. doc for workspaces without documentation files) are skipped automatically.

Cross-compilation

Here <profile> stands for cross_raspberry_pi, cross_arm64. Cross-compilation make targets can be found in ccws/make/cross.mk and ccws/profiles/<profile>/targets.mk

Note on legacy cross_jetson_xavier and cross_jetson_nano: these profiles require Ubuntu 18.04 / ROS melodic and install nvcc, you may want to do this in a container.

The general workflow is documented below, for more technical details see ccws/doc/cross-compilation.md and CCWS CI test in .ccws/test_cross.mk (ROS1) and .ccws/test_cross_ros2.mk (ROS2):

1. Install profile dependencies with make bp_install_build BUILD_PROFILE=<profile>
2. Obtain system image:
   • cross_raspberry_pi – bp_install_build target automatically downloads standard image;
   • cross_arm64 – use docker_mountpoint make target, which extracts root filesystem from a docker image;
   • cross_jetson_xavier, cross_jetson_nano – CCWS does not obtain these images automatically, you have to manualy copy system partition image to ccws/profiles/cross_jetson_xavier/system.img.
3. Initialize source repositories:
   • make wsinit REPOS="https://github.com/asherikov/staticoma.git"
   • [when building all ROS1 packages (does not work with ROS2)] add ROS1 dependencies of all your packages to the workspace make dep_to_repolist ROS_DISTRO=melodic, or a specific package make dep_to_repolist PKG=<pkg> ROS_DISTRO=melodic;
   • fetch all packages make wsupdate.
4. Install system dependencies of packages in your workspace to the system image: make cross_install PKG=staticoma BUILD_PROFILE=<profile> ROS_DISTRO=<distro>
5. Compile packages:
   • mount sysroot with make cross_mount BUILD_PROFILE=<profile>
   • build packages, e.g. make staticoma BUILD_PROFILE=<profile> or build and generate deb package make PKG=staticoma BUILD_PROFILE=deb,<profile>
   • unmount sysroot when done with make cross_umount BUILD_PROFILE=<profile>

Advanced usage

CCWS docker image

A docker image with preinstalled CCWS and dependencies is available for testing, but it is recommended to build a tailored image using ccws/examples/Dockerfile as an example.

The image can be used in the following way:

• docker pull asherikov/ccws:noble (or asherikov/ccws:jammy)
• mkdir tmp_ws # sources, build, install, cache will go here
• docker run --rm -ti -v ./tmp_ws:/ccws/workspace asherikov/ccws:noble bash
• make wsinit REPOS="https://github.com/asherikov/qpmad.git"
• ...

The image also includes preinstalled agentic coding tools that are described in the following subsection.

AI features / agentic coding

CCWS provides the following features to facilitate agentic coding:

• repository includes a Claude-compatible plugin with a skill that describes how to use this environment;
• codebase_memory_mcp build profile provides source code indexing functionality and a corresponding MCP server;
• https://github.com/QwenLM/qwen-code cli environment is installed in the ccws docker image by default.

Agentic coding funtionality is intended to be used in a containerized environment, e.g., the default ccws docker image. The image launches qwen-code in a tmux session on startup. It is recommended to use CCWS helper target make qwen to run the container: source, build, install, and other directories are mounted as volumes automatically. qwen configuration is stored in .ccws/qwen directory of the source space. See ccws/make/ai.mk for more details. Installation of the skill is not needed when using CCWS containers.

CCWS in CI

See ccws/examples/Jenkinsfile.example, .github/workflows/reusable_* and https://github.com/asherikov/sharf/tree/main/.github/workflows for examples.

Extending CCWS

CCWS functionality can be extended in multiple ways:

• by adding new build profiles, e.g., make bp_new BUILD_PROFILE=vendor_static_checks,static_checks, all profiles starting with vendor prefix are ignored by git;
• by adding execution profiles;
• make targets can be added by creating a ccws/profiles/build/vendor/<filename>.mk file;
• common cmake toolchain suffix can be added to ccws/profiles/build/vendor/toolchain_suffix.cmake.

Known issues

Crosscompilation and binary packages

• Segmentation fault during cross-compilation or debian package generation inside docker containers (both require proot): presumably due to seccomp Linux feature, which can be disabled with --security-opt seccomp:unconfined docker parameter. Disabling seccomp for proot with PROOT_NO_SECCOMP=1 seems to be unnecessary.

• proot segfault while building on arm64 in Ubuntu 22, e.g., while building debian packages. Newer version of proot has to be used, see https://github.com/proot-me/proot/issues/312.

• Crosscompilation may require installation of workspace dependencies on the build host.

Sanitizers

• Programs compiled with sanitizers (addr_undef_sanitizers or thread_sanitizer build profiles) output 2: AddressSanitizer:DEADLYSIGNAL or FATAL: ThreadSanitizer: unexpected memory mapping when executed: the reason is tightened memory security with ASLR (address space layout randomization) in modern Linux kernels, see https://github.com/google/sanitizers/issues/1614. The issue can be resolved by setting sudo sysctl vm.mmap_rnd_bits=28.

Other

• Some of ROS2 core packages cannot be built with CCWS due to cmake misuse, e.g., see https://github.com/ament/google_benchmark_vendor/issues/17.

• Workspace prefix is intentionally cropped from paths in debug info, you have to set path substitutions in gdb to resolve them correctly, i.e., set substitute-path / <path_to_workspace>.

Related software

• https://github.com/ros-industrial/industrial_ci – ROS specific CI scripts, noninteractive, “one-shot” design, no sanitizers, emulated cross compilation.
• https://github.com/ros-tooling/cross_compile/ – emulated cross compilation for ROS and ROS2.
• https://github.com/HesselM/rpicross_notes – cross compilation for Raspberry Pi done in a different way.
• https://github.com/ros-tooling/action-ros-ci – github action that covers some of CCWS functionality.
• https://github.com/ros-infrastructure/ros_buildfarm, http://wiki.ros.org/buildfarm – the core of ROS packaging infrastructure. Complicated, specialized on handling of individual packages rather than workspaces, not suitable for quick field redeployments.
• https://github.com/colcon/colcon-bundle provides functionality similar to ‘superpackages’ allowing to pack install space into a single archive. Naturally, it does not provide all the package features like dependencies, install scripts, etc. Moreover, it does not rely on chroot-like environment to ensure correct paths.
• https://github.com/b-robotized/ros_team_workspace – python based environment that also aims at workspace handling automation. Unlike CCWS which is primarily a build/deployment environment, it has more ROS-specific features like setting up bringup packages. However, it lacks some of advanced build features CCWS has.
• https://github.com/athackst/vscode_ros2_workspace – VSCode-specific workspace environment, that provides various build and static analysis options, but not as many as CCWS. analysis features.

TODO

• Testing

  • Fuzzing https://github.com/rosin-project/ros2_fuzz, https://github.com/sslab-gatech/RoboFuzz, https://github.com/AFLplusplus/AFLplusplus, https://github.com/google/clusterfuzzlite.
  • cmake 3.21: --output-junit <file> = Output test results to JUnit XML file.

• Static checks

  • Potential replacement for scan_build https://github.com/Ericsson/codechecker with extra checks and caching.

• Build

  • Reproducible builds https://reproducible-builds.org/.

• Containers and images

  • System images as OCI artifacts https://oras.land/docs/.
  • Generate system images in layered, container-like fashion https://osinside.github.io/kiwi/overview.html.

• Other

  • Diagrams from code https://github.com/bkryza/clang-uml.
  • Advanced dependency graph generation https://github.com/maspe36/colcon_gephi.
  • Modern documentation generator https://github.com/sourcey/sourcey.
  • Better integration with GitHub, e.g., https://github.com/cpp-linter/cpp-linter-action.

Bookmarks (not going to be supported)

• Shell formatter https://github.com/mvdan/sh.
• https://github.com/Tencent/TscanCode – C++ static analysis tool.
• https://github.com/DLu/roscompile/tree/main/roscompile – linter for catkin packages.
• https://github.com/git-afsantos/haros – ROS-aware static analysis, might have issues with non catkin_make build environments.
• Control symbol visibility and verify with https://github.com/yugr/ShlibVisibilityChecker.
• Integrate https://github.com/oclint/oclint
• Source code spellcheck https://github.com/myint/scspell.
• https://github.com/sscpac/statick is not going to be used, but some of its linters can be integrated.
• Add CodeQL profile (https://github.com/github/codeql).
• Use https://libguestfs.org/ or https://github.com/alperakcan/fuse-ext2 instead of loop devices, to avoid using sudo. There are some issues in Ubuntu though, bug 759725, see https://libguestfs.org/guestfs-faq.1.html. guestfs is too slow to be practical.
• https://github.com/mrtazz/checkmake does not appear to be very useful.
• https://github.com/oxsecurity/megalinter multi-purpose linter wrapper.
• https://github.com/myint/cppclean might be useful for unnecessary header detection, but looks stale.
• https://github.com/cheshirekow/cmake_format seem to be abandoned, linter not really helpful – mostly formatting stuff. Same applies to https://github.com/cmake-lint/cmake-lint.
• https://github.com/ejfitzgerald/clang-tidy-cache A cache for clang-tidy.
• https://github.com/mgehre/xunused – detection of unused code. No precompiled binaries, may not work with new clang.
• Execution profile with https://github.com/yugr/libdebugme to automatically start debugger on a signal. Stale, requires compilation.
• https://github.com/yugr/valgrind-preload as an alternative to valgrind execution profile – an overkill in general case though. Stale, requires compilation, sanitizers are currently preferred to valgrind.
• Distributed compilation support with https://github.com/distcc/distcc can be useful, but there are no use-cases so far.
• https://github.com/jordansissel/fpm – generic binary package generator, potential replacement for dpkg-deb. Not as flexible as the current approach, e.g., in generation of subpackages.