[Documentation] [TitleIndex] [WordIndex

API review

Proposer: The catkin team

Present at review:

The following is a reconstruction of the decisional process and information gathered from different sources, which might be wrong and/or outdated.

The introduction of catkin as a replacement for rosbuild has raised a number of discussions. The change in the approach needs to be justified by a change of requirements/constraints. This section tries to summarize the justification for catkin design decisions as discused in several mailing list threads and documentation. For the discussions, see ros-users and the ROS Buildsystem SIG

Since the workflow from rosmake/rosbuild, vanilla cmake and catkin is very different it is important to understand the implications and forces.

Prerequisites

Catkin strives to be a tool that makes setting up developer environments, building, cross compiling and packaging easier. It competes with rosbuild, vanilla cmake, and other build projects like Autotools.

To understand Catkin, several concepts are helpful in discussion (also see the catkin glossary)

Lifecycle

The lifecycle of software development has several phases, in c++ those include:

Build and install locations

Building and installing created files somewhere, and apart of the challenge is to select useful places to build and install to.

The above definitions are important because vanilla cmake, rosbuild and catkin use different concepts for managing build and install spaces.

Artefact groups

Since both rosbuild and catkin follow different design goals related to building, packaging and releasing, these definitions are important.

It is common to use the concept build group == release unit == packaging unit, because it is easiest to understand and support.

Restrictions

Environment variables

A set of environment variables is used to influence how binaries, libraries, etc. are discovered. Important in this context are:

rosbuild (rb)

Design Goals

Design Decisions

Shortcomings

Catkin (ctk)

Design Goals

Same as rosbuild: rb_G1, rb_G2, rb_G3, rb_G5

Dropped:

Additional Goals

Design Decisions

Alternatives Comparison

Traditional approach with cmake

Cmake is a widely used tool that offers some flexibility in how it is used. We describe here the most common usage.

In this approach, if source Project C depends on source Project B, and source project B depends on source project A, then the user must know this and build and install A, then build and install B, and then build and install C for a complete build.

rosbuild

rosbuild was created to make a different workflow from the traditional workflow easier. rosbuild also uses cmake, but wraps the cmake command and allows not to specify cmake install targets. rosbuild uses the in-source build space for both building and source-installing. rosmake uses custom code to invoke building in dependency projects in the right order, so the user does not need to remember doing this. rosbuild requires dependencies to be defined in special manifest files for packages and stacks. System dependencies are also defined in the manifests and can be resolved with rosdep. rosbuild also features a separation of stacks and packages, where packages are atomic build units, whereas stacks are atomic release units. Releasing also commonly implies packaging for stacks that are available through package managers.

catkin

catkin allows to build and install projects as in the traditional cmake workflow. The cmake macros however also offer a different workflow, which is the intended usage of catkin. In the intended usage, building for all workspace projects uses the workspace build space, the workspace build space is also part of the environment.

On configuration & build speed

This is based on a post in the ROS buildsystem SIG.

Note this benchmark purpose was disturbed by bug https://code.ros.org/trac/ros/ticket/4036.

A quick benchmark to get an impression yielded these results. Those are for building electric and fuerte ros-base variant (ros and ros_comm stacks), which were based on catkin in fuerte.

Full compile:

Electric

configure and build from scratch

rosmake -a

105.9s

Fuerte

configure from scratch

cmake ..

14.6s

Fuerte

build from scratch parallel

make -j 8

19.2s

Fuerte

build from scratch

make

106.4s

Noop compile

Electric

configure and build again (noop)

rosmake -a

27.1s

Fuerte

build again (noop)

make

1.7s

Fuerte

build parallel (noop)

make

0.2s

One file tweaked:

Electric

configure and build after change

rosmake -a

26.8s

Fuerte

configure again after change

cmake ..

10.0s

Fuerte

build after change

make

2.1s

Analysis by Tully:

The biggest difference I will note is that the noop build using a single cmake workspace is on the order of one second (less if you enable parallelization). And if you tweak a single file, it climbs up to 2 seconds. Whereas a noop build in rosmake takes 30 seconds. Also the actual full compile step takes about the same amount of time, if the fuerte system is not threaded. If it is threaded it takes one 5th the time.

A random comparison I'd note is that a full configure and build with a single workspace takes the same amount of time as a noop build using rosbuild. Also from profiling the rosmake builds, the minimum invocation time for cmake && make is about 0.3 to 0.5 seconds per package.

Causes for the differences:

On Catkin cmake conflicts

As the catkin design uses a single cmake process to build all processes in one catkin workspace, and uses a single build folder for all projects, collisions of names are possible. The extent to which this is possible, the plausibility of that happening and the effort to debug and fix such collisions is difficult to judge, so here are more technical details.

Build space conflicts

The suggested catkin workflow uses a single build forlder in the workspace, in which all build products (c++ executables, c++ libraries, generated files) are placed, in a structure that mirrors the FHS layout as the install target would create. This allows using the build space as if it were an install space easily.

Executable binaries will either go into a common bin folder, or into another fhs compliant folder for binaries (e.g. lib/PKGNAME/libexec or libexec/PKGNAME). Name collisions can thus happen if two projects name an executable the same and put it into the common bin folder. Conversely, if two executables are to be delivered under different package names in e.g. lib, but with the same executable name, the catkin workspace will not allow building those two projects if they name the target the same, even though installation of the executables would work fine. In this case the target name must be prefixed to avoid that collision.

Libraries thus go into a common "lib" folder. Collisions for those are possible if two projects name their libraries the same. Such collisions would also happpen if the packages were installed using apt-get. If conflicting projects lie in the same workspace, cmake will fail with a useful error message for such cases (Duplicate target). If conflicting projects are developed by different teams, the conflict will appear anytime later. Generally this problem is not more constraining than for any Linux/Unix program development. .pc files go into lib/pkgconfig/PKGNAME.pc and are therefore conflict free.

Generated files go into a common share folder in the proposed FHS layout, and in there in a subfolder named after the package. This should allow avoiding namespace conflicts between projects. However if the developer makes a mistake in the definition of the target path, this can fail very late.

Cmake variable conflicts

A catkin workspace in the default suggestion acts like a single cmake project. Single Cmake projects have different variable scopes, the default scope for a variable is the directory of the CMakeLists.txt it is defined in. Such variables are safe from conflicts.

E.g.

set(myfeature 42)

Cmake also allows global variables and cached variables (which are global). Such variabls do conflict with each other, and typically in a silent way (meaning the developer does not get a warning or an error by cmake, just something bad happens and he has to find out what and how).

Caching is a useful feature for catkin users in order not to have to pass variable values on every invocation of cmake, e.g. cmake -Dmyfeature=42

E.g.

set(myfeature 42 CACHE STRING "description")
get_filename_component(VarName FileName CACHE)
option(CPACK_PACKAGES "Set to ON to build the packages. Requires cmake >2.4" OFF)

Such cached variables are also fine if users adhere to strict naming standards like using a prefix <packagename>_ for all such variables.

Any global variable is also okay to use if ALL projects ALWAYS set the variable itself before they use it. As cmake does not process projects concurrently, projects would not influence each other that way. However if a developer forgets this, side effects between projects can occur.

Several standard cmake commands use global or cached variables, such as:

find_program
find_package
find_library
find_path

This means if two different catkin stacks used these functions with different options, the first one would win for the entire workspace, without warning or errors by cmake.

E.g.

# In project B
find_package(Boost 1.40.0 EXACT COMPONENTS system)
# In project A (Does not reproduce if both lines are used in the same directory)
find_package(Boost 1.46.1 EXACT COMPONENTS system)

The first command in one catkin project will make cmake ignore the second command in another catkin project, and there will be no warning or error if the components are exactly the same.

Notice how in my version of catkin, projects later in lexical order are build earlier.

In this case cmake will check for the boost version for every new component, meaning if an earlier project demands a superset of Boost components of a second project, the second project boost version check will be ignored, despite of the EXACT flag.

Other quirks are that a project using variables like ${Boost_INCLUDE_DIRS} does not need to call find_package(Boost...) anymore, as long as some other project in the catkin workspace does (which means people will forget to call it).

[TF] This fails on my machine w/o 1.46 installed:

find_package(Boost 1.40.0 EXACT COMPONENTS system)
find_package(Boost 1.46.1 EXACT COMPONENTS thread)

This fails because different components are used.

And this passes:

find_package(Boost 1.40.0 COMPONENTS system)
find_package(Boost 1.39.0 REQUIRED COMPONENTS thread)
message(STATUS "${Boost_LIBRARIES}")

Like this:

-- Boost version: 1.40.0
-- Found the following Boost libraries:
--   system
-- Boost version: 1.40.0
-- Found the following Boost libraries:
--   thread
-- /usr/lib/libboost_system-mt.so;/usr/lib/libboost_thread-mt.so

Command names from macros obviously also share the global name space. Therefore, e.g.

macro(mymacro)
message(STATUS bla)
endmacro()

makes this macro available to all later projects in the workspace, (e.g. people may forget to define macros they used in each project)

This also affects other commands relying on the global namespace such as

if (COMMAND mymacro) ...

include(CheckFunctionExists)

includes that standard module and its commands to ALL following projects in the workspace, meaning later projects in the workspace may use the command without calling include. Also files meant for inclusion may break if they were written with the assumption that they should only be included once per project.

CMake also includes a large number of standard and non-standard modules, for which the variable scoping is difficult to list (and which obviously also changes between versions).

Examples of a standard modules using cached variables (just to show even standard cmake modules use cached vars):

include(FindCURL)
# defines cached vars:
CURL_INCLUDE_DIR
CURL_LIBRARY

include(CPack)
CPACK_...

Apart from standard cmake modules, there are several "non-standard" cmake modules which may be used by several projects.

In general, the naming collisions make it harder to have ROS Package forks of non-ROS projects, as the CMakelists.txt may have to be changed to also be catkin compliant.

E.g.:

orocos_kinematics_dynamics/orocos_kdl/CMakelists.txt
visualization_common/ogre_tools/CMakelists.txt
driver_common/dynamic_reconfigure/cmake/cfgbuild.cmake

are examples of a files where it is difficult to check whether they are catkin compliant or not.

Debugging

The problem of cmake conflicts is worsened by the fact that the really bad ones are prone only to surface in catkin workspace with many hundreds of packages, and to surface in a way that makes it difficult to diagnose whether and which package's CMakeLists.txt or which included 3rd party cmake files may be responsible for a failed build.

Feature map

(+ means better, - means worse)

Feature

Vanilla CMake

rosbuild

Catkin

installation-from-source

+

-

+

atomic distribution unit

0

stacks

stacks

atomic build unit

CMake project

packages

stacks

machine-readable meta information

0

manifest.xml + stack.xml

stack.xml

exporting build flags (cc/ld) to other packages

+ generated

- manual in manifest.xml

+ generated

importing build flags (cc/ld) from other packages

- manual in CMakelists.txt as target_link_libraries()

implicit (but bloated, not minimal), if exported, else broken

+ semi-automatic, generates find_package() infrastructure based on catkin_project() arguments

single command multi-project build

parent project cmake

rosmake

workspace-level cmake

install target

+ (if provided)

-

+ (if provided)

FHS compliant install layout

+ (if install provided)

-

+ (if install provided)

build without custom tools

+

--

-

run without custom tools

+

--

+

cross compiling

+

-

+

isolated configuration

+

+

-

isolated build

+

+

-

fool-proof configure and build

+

+

-

multiple builds (e.g. Debug vs. Release) into separate folders

+

-

+

quick adding of custom source projects to environment

- make install

+ copy, rosmake

+ copy, make

quick removal of source projects from environment

-- (stow)

+ delete folder

rm -rf build, make (ccache)

build space

in project

in-source

in-workspace

recursive make

--

+ rosmake ...

+ make ...

quick make of other stack/package

--

+ rosmake stack

make -C path/to/workspace targetname

workspace folder layout

Arbirtary

Arbitrary

flat list of stacks

packaged sources

No

Always

possible to provide separate package with sources

Use cases

Installing ROS+stacks from source

A developer checks out the source of ROS core and several other stacks into a local folder. The user then runs a make-like command. As a result, the user is able to run ROS master and ROS nodes.

[JOQ] possible command sequence:

 $ rosinstall ~/workspace http://rosinstall/yaml/file/url
 $ cd ~/workspace
 $ mkdir build
 $ cd build
 $ cmake ..
 $ make
 $ sudo make install prefix=/usr/local

Adding a stack from source

A developer has a working ROS instance. The developer checks out the source of a stack, runs a build command. From then on, the newly-built stack is used (instead of another overlayed stack)

[JOQ] possible command sequence:

 $ roslocate info stack-name | rosws merge -
 $ rosws update stack-name
 $ cd ~/workspace
 $ rm -rf build
 $ mkdir build
 $ cd build
 $ cmake ..
 $ make
 $ source ~/workspace/build/setup.bash

Viewing source of installed package for debugging

Example: A developer wants to code a node that is somewhat similar to a node that he can run, and he wants to see the code of that rather than reinvent the wheel. Other example: the developer suspects a bug in roscpp code and wants to read the code. In this example, it is also crucial that the developer sees the code that runs, not just the latest code in a repository.

[TF] Possible process flows:

Binary installation:

apt-get source ros-fuerte-mypackage
ls ros-fuerte-mypackage

Source installation:

roscd mypackage
ls

Removing a previously added-from-source stack

A developer has a working ROS instance. The developer runs one or more commands. As a result, the artefact from the stack to be removed are no more used in that environment.

[JOQ] possible command sequence:

 $ cd ~/workspace
 $ rosws remove stack-name
 $ rm -rf stack-name build
 $ source ./setup.bash
 $ mkdir build
 $ cd build
 $ cmake ..
 $ make

Cross compiling stacks

A developer creates binaries to run on several alien architectures. With Catkin, the catkin workspace needs to be configured once for the cross-compilatio builds. The build process then creates all artifact within a single folder.

If catkin used isolated configuration processes per package, then the configuration step would have to be executed and monitored individually for many packages.

If catkin used isolated build folders per package, the build step would generate many such folders which would be harder to inspect and clean up manually.

Daily work

A developer has a workspace with several packages, that are built against a ROS installation. The developer regularly does changes in several packages and rebuilds several of them.

catkin

  # make changes
  $ make

The duration of make is essential, as this is called very frequently in the daily work. Unnecessary waiting times lead to developers taking error-prone shortcuts.

Packaging stacks

A developer packages a stack or package for Debian/Ubuntu apt-get Fedora

[JOQ] what is the relevance of this use case to catkin? Isn't it a bloom packaging issue?

Example:

Dave the developer would like to make a Debian package for his ROS based software daves_inspection_system for release. He's hoping to make his release also palatable to users on limited storage targets.

Dave's workspace/repository contains several packages (Currently a stack) that are targeted at several different platforms, for instance, daves_inspection_system_OCU for the OCU, and the daves_inspection_system_robot for the robot itself.

daves_inspection_system_OCU has many more dependencies than daves_inspection_system_robot and so he would like to package them separately.

daves_inspection_system_robot depends on laser_geometry, but Dave is worried about pulling in PCL as a depencency because laser_filters shares a stack. As such he would like to depend on ROS-packages rather than ROS-stacks.

Dave also hopes that the dev and run-time dependencies are separate not only in ROS, but also in the official Deb packages so that he can keep his separate too and avoid pulling in the full gcc toolchain.

Proposed solution:

Separate the set of packages into multiple stacks.

[JOQ] Dave needs to understand dependencies better. Relatively few problems of this type can be solved by merely dividing packages into separate stacks. Consider the case of dynamic_reconfigure (already a unary stack), which has a <rosdep name="wxpython">, pulling in large GUI dependencies. That package needs to be reorganized, which is going to affect API compatibility. The only way to avoid problems like that is to think clearly about dependencies in the initial design. Packages and stacks have nothing to do with this problem. It can happen within a single source file, if the programmer is not careful.

Discovering stack / package dependencies using the dependency graph

A user wants to see how packages are related to each other, other than being dependent.

Wrap 3rd party libraries inside a package

For some developers that do not want to bother getting a 3rd party library released properly, they have been simply downloading and building libraries inside a ROS package. They would like to be able to continue doing so.

Suggestions

Reviewers can state suggestions here, discussion should happen in the ROS buildsystem SIG, with the results to be merged into this section by the reviewers.

AndrewSomerville

Additional catkin goal:

Thibault Kruse

This does not imply parsing the cmake files, but instead telling cmake to dump variables into a file. The information would only be available after cmake invocation, but that would still be better than no access to the information at all. Also see http://www.cmake.org/Wiki/CMake_FAQ#How_can_I_generate_a_source_file_during_the_build.3F

Lorenz Mösenlechner

# Install a base set of ros stacks
> sudo apt-get install ros-fuerte-desktop
> mkdir ws; cd ws
# Configure two separate workspaces, one for 3rd party, one for your code
> rosws init --catkin 3rdparty
> rosws init --catkin src
> cd 3rdparty; rosws merge my_3rd_party_stacks.rosinstall; rosws update
> cd src; rosws merge my_src_stacks.rosinstall; rosws update
# Install the 3rd party workspace, then develop in the other
> mkdir build_3rdparty; cd build_3rdparty; cmake ../3rdparty; make; make install
> mkdir build; cd build; cmake ../src; make (optionally make install)

An alternative way would be to put your 3rdparty and src stacks in the same rosinstall and workspace and make use of CATKIN_BLACKLIST_STACKS and CATKIN_WHITELIST_STACKS in two parallel builds.

To create your 3rdparty packages, you would have to use cmake. Most 3rdparty packages with rosbuild traditionally used make, which wasn't a good solution because you couldn't pass in cmake flags to the build that you used for the rest of your packages. You can do this in cmake though, which is what you'd have to do for catkin 3rdparty packages. I used to do this for an embedded opencv rosbuild - see the CMakeLists.txt in eros_opencv. It would be simple to build some cmake macros to facilitate that, or as JB mentioned, utilise the cmake external project infrastructure.

Downside is that the above is a 2-step process (currently with rosbuild it's a 1-step process).



2020-08-01 12:30