Infrastructure

SIG Coordinator: Tully Foote

Topics: ROS.org/wiki, answers.ros.org, Hudson, Non ROS packaged debian packages (ala opencv, openni, gazebo, pcl)

Members:

• Ken Conley
• Michael Carroll
• Tully Foote
• Ethan Rublee
• Melonee Wise
• Florian Weisshardt
• Ben Pitzer
• Troy Straszheim

Meetings:

• /20110929

Wish List of Use Cases

• Smaller debs/strip debug symbols: it would be great to land this capability. It would greatly reduce the size of our debs, which has many, many wins

Enable debbuild to run on storm servers

Technically we can do this, but it has issues -- due to the deb caching, the test for deb_in_repo on external servers is not 100% correct and can make it impossible to get a full build in.

Move Hudson/Jenkins job configuration to source control

Right now we edit jobs directly on Hudson (with the exception of job_generation). It would be a better practice to maintain the job configuration scripts in source control and push them to the build server:

1. Easier to share jobs configurations with external institutions
2. Easier to revert changes

Releated to this it would be nice to make this pip installable instead of a ros package, based on rospkg.

Moving server configuration to Puppet/Chef

Right now server configuration is done in numerous places, from the Hudson jobs themselves to configuration within the run_chroot script. It's hard to know what configuration a build server will actually have and it also makes the chroot scripts less re-usable -- its frequently been forked in the past in order to do different configs and it also has a bunch of extra branch logic/options.

Puppet, for example, has intro tutorials on configuring SSH keys as well as hooks for apt/pip.

Enable external institutions to run build/test/doc/release infrastructure

Ben is already somewhat doing this with doc infrastructure, but it would be nice to make this reproducible.

Main thing to work on is externalizing configuration from scripts and some documentation.

Sub use cases:

• Run documentation infrastructure
• Run continuous integration for a particular platform (e.g. Fedora)
• Produce internal releases of code

Install release scripts via easy_install/pip

Right now the install scripts have to be installed via SVN checkout. They also rely on APIs within the ROS stack, which creates bad coupling. It would nice if the release scripts could be installed/upgraded via easy_install/pip. It would also be nice to migrate them to the new rospkg/rosdep/vcstools standalone tools, which have better APIs and are easy to keep migrating together.

Enable non rosbuild build/test/doc/release

Projects which attempt to be ROS agnostic which we develop or contribute to(ala opencv, openni, gazebo, pcl, ecto) are lacking reusable infrastructure for continuous integration.

• Setup post commit hooks for these projects, that trigger pull, build, test, documentation update.
• When a new deb is produced, need to trigger a rebuild of all ROS packages that depend on these. Its a manual process of version locking currently to rev a new debian of OpenCV, as we can never be sure of OpenCV's binary compatibility. PCL will benefit from this as well.
• Clear path for side by side installation of these libraries. Typically there can be only one on your system. Either the libraries can support side by side installs by mangling names and directories (soname is not enough), or the debians can specify that they conflict, in which case apt-get will prompt you to uninstall electric when you install fuerte.

notes on possible standalone config format

User stories

I develop a library with a README. Users of my library usually build from source, install it, and then link against it somehow. I want ROS users to be able to use my library and release against it.

I develop a library, and i have unit tests and docs. I want automatic builds to occur on a regular basis, with tests and up to date docs, without having to write my own infrastructure.

I develop a ROS application that solves object recognition. However I use a few really great libraries that I have to build from source. I want to be able to release my ROS application, and these other dependencies are blocking me.

Workflow

• register my library with ROS, writing what ever configuration is necessary for ROS to be able to build my library.
• commit some changes. It builds, tests, docs update.
• Run a release script, tags a release, kicks off a debian build, generates a rosdep, invalidates all ROS packages that use it. It gets installed to some FHS that i can point client code bases at easily.

If users are not using debs of the standalone library, most likely a sourcedep is being used? Possibly sourcedep should be auto generated for anything in the standalone index.

Make package review and support status much more visible

• Right now a development package vs a core package look the same on the wiki and in rosbrowse.
• Color coding for review status and support level would be great.

Components

Hudson automation

• hudson package, superceded by python-jenkins.

• Main needs: queue management (minor work)
• Need to rework email generation to enable migration forward to jenkins from our hudson instance.

rosdoc/wiki indexer

rosdoc and rosdoc_rosorg. Codebase isn't in great shape. Possibly want to look into Readthedoc.org infrastructure (at least for sphinx).

roslocate, rosco, and rosws are recent upgrades to this toolchain.

Build/Test

• job_generation: converts rosdistro files into Hudson jobs. Needs to be ported to standalone libraries (shouldn't be hard)

• run_chroot: runs test jobs in a chroot to keep machines clean. Also

brings machines into a known state. Pretty useful script, though I'd like to explore replacing the configuration management aspect with Puppet or Chef.

• storm/cloud: brings up external cloud servers for testing (currently private)

Release infrastructure

create.py: needs to be ported to standalone script (as well as ported to rospkg APIs)

Debbuild:

• ROS debian building (needs a lot more work)

  • rosdeb: needs major cleanup, lots of duplicate code

  • would like to make progress on migrating to pure-Cmake/rosbuild2 builds
  • would like to strip debug symbols into separate package
• Non-ROS debian building (e.g. OpenCV. Needs more tooling to make

it easier to use/verify)