Gen 1 Buildfarm Overview

The ROS buildsystem as a few main components. These all can actually be on one host, but when deploying they may be separated for clarity.

the distribution

This defines what packages are going to be processed. It is defined by REP 137 and will need to be available via http to the jenkins master and slaves.

Jenkins Instance

This is an instance of Jenkins-CI. You will need to install this manually and setup credentials, and a few plugins.

Plugins

• Credentials Plugin
• Groovy Postbuild
• Jenkins Bazaar plugin ( if bzr support is needed)
• Jenkins build timeout plugin
• Jenkins description setter plugin
• Jenkins Email Extension Plugin
• Jenkins GIT client plugin
• Jenkins GIT plugin
• Jenkins Mailer Plugin
• Jenkins Mercurial Plugin ( if hg support needed)
• Jenkins Parameterized Trigger Plugin
• Jenkins promoted builds plugin
• Jenkins SSH Slaves plugin ( if launching slaves via ssh)
• Jenkins Subversion Plugin (if svn support needed)
• Priority Sorter
• SSH Credentials Plugin( needed for ssh slaves plugin )
• SSH Agent Plugin
• Token Macro Plugin

Jenkins Slave

This is the machine which will run the builds. It will need to be provisioned appropriately with the basic dependencies and credentials to push results to the apt server

Most jobs are run in a chroot, so the depedencies are mostly in support of pbuilder and the chroot.

Note: This can be run on the master itself.

apt server

This will host the resultant debian apt packages. There are usually 2 repos, one for active building and one for testing complete builds.

You will need to setup this repo such that the slave machines can push to them. (reprepro-updater will help with most of this)

Extending another distribution

Not tested yet, but the design of this system is to enable using an existing "distribution" aka the main one, and extending it with custom packages. The vision for how to do this is that the externally maintained packages will be imported into the building repo and live there. Then rosdep will just consider them to be system dependencies already satisfied. Note if you do this you will need to use a different name than the overlaied distro, such as my_hydro instead of hydro.

Notes from Paul

I have tried to replicate a simple jenkins buildfarm with the tools used for the official one. This can be done with the following steps:

Set up an apt repository

1. create a rosbuild user and log in with it

2. clone https://github.com/ros-infrastructure/reprepro-updater into /home/rosbuild/reprepro_updater (note the underscore, not dash)

3. create a gpg key for signing the release repository. DO NOT set a password.

   gpg --gen-key

4. export the public key:

   gpg --armor --export --output public.key

5. install apache2 and create /var/www/repos/building, give rosbuild user full access to that directory, storing the public key in there is a good idea. It only needs to be accessible to the people you want to share the repo with.
6. edit /var/www/repos/building/conf/distributions with the following:

   Origin: ROS
   Label: ROSpbian
   Codename: wheezy
   Architectures: armhf source
   Components: main
   Description: ROS Debian Repository for Raspbian
   SignWith: yes
   Update: python

7. edit /var/www/repos/building/conf/updates with the following:

   Name: python
   Method: http://packages.ros.org/ros/ubuntu/
   Suite: precise
   Components: main
   Architectures: armhf source
   FilterFormula: Package (% python-* )

8. populate your repository with python packages needed to build other packages
   1. install the repository update tools

      sudo apt-get install reprepro python-yaml

   2. Update the repository

      reprepro -Vb /var/www/repos/building update

   3. (optional) if your building repository is exposed to the internet, you may want to deny access to the conf/ directory with apache directives

Set up a Jenkins Master

1. install jenkins and dependencies

   sudo apt-get install jenkins python-empy python-numpy git python-rosdistro python-jenkins python-vcstools

2. enable user-based security
3. create a user (e.g. ‘rosbuild’)
4. set up the gbp repositories
5. use the ones that I made, make sure that they are visible from all the slaves you plan to use (bitbucket/github are nice candidates)
6. if using apache2 to expose the git repositories, see the manpage of git-http-backend
7. set up rosdistro files
   1. pick a location (http) reachable from all the slaves you plan to use
   2. create an index.yaml file looking like this:

      %YAML 1.1
      # index file
      # this file describes the available ROS distributions
      # see REP 137: http://ros.org/reps/rep-0137.html
      ---
      type: index
      version: 1
      distributions:
        groovy:
          release: groovy.yaml
          release_builds: [groovy-build.yaml]
          release_cache: http://your.url/rosdistro/groovy-cache.yaml

   3. use the provided groovy.yaml
   4. you might have to modify it slightly for the newer rosdistro API (specifically the platforms: section)
   5. the groovy-build.yaml looks like this:

      %YAML 1.1
      # ROS build file
      # see REP 137: http://ros.org/reps/rep-0137.html
      ---
      type: release-build
      version: 1
      targets:
        wheezy: [armhf]
      jenkins_url: http://192.168.1.35:8080
      apt_target_repository: http://repo.url/repos/building
      apt_mirrors: ['http://ros.raspbian.org/repo/ros']

   6. create the file in ~/.config/rosdistro/config.yaml

      index_url: http://your.url/rosdistro/index.yaml

8. install python-rosdistro
9. generate the cache file

   rosdistro_build_cache http://your.url/rosdistro/index.yaml groovy

Create Jobs

1. create buildfarm/server.yaml in your ~/.ros folder with something like this:

   url: http://jenkins-master.url:8080
   username: rosbuild
   password: foo

2. clone https://github.com/ros-infrastructure/buildfarm

3. install python-jenkins, python-empy, python-vcstools, python-setuptools
4. apply raspbian patches
5. source setup.sh
6. run:

   ./scripts/create_release_jobs.py groovy --fqdn hostname_of_apt_repo --commit --wet-only

Set up Slaves

1. (there is a setup_slave.sh script that does the following)
2. install ssh credentials plugin + ssh slaves plugin for jenkins (from jenkins interface)
3. create the slaves on the jenkins master with the tag ‘debbuild’ and select ‘java web start’ as the launch method
4. on the slaves: add keys

5. http://packages.ros.org/ros.key

6. http://archive.raspbian.org/raspbian.public.key

7. http://repo.url/repos/building/public.key

8. install dependencies:

   sudo apt-get install python-jenkins python-empy pbuilder qemu-user-static python-rosdistro default-jre git git-buildpackage debhelper python-paramiko

9. Set proper permissions on the ccache directory:

   sudo chmod a+w /var/cache/pbuilder/ccache
   sudo chmod +t /var/cache/pbuilder/ccache

10. make sure slaves can ssh without password to the repo, if you launch the slave.jar with sudo, make sure root's rsa key is allowed to ssh to rosbuild@your-repo
11. make sure the build user on the slaves can run sudo without a password
12. don’t forget to run kill_zombies.sh
13. If you experience problems with the qemu ARM emulator, try a newer version from the Linaro PPA:

    sudo apt-add-repository ppa:linaro-maintainers/overlay
    sudo apt-get update
    sudo apt-get install qemu-arm-static

Trigger Jobs

1. in the buildfarm directory

   . setup.sh
   ./scripts/trigger_missing.py groovy --fqdn hostname_of_apt_repo --commit

Sync

I did not use any tool to perform the sync. Rather, I have an update rule in the apt repository conf/ folder on etherpad.ros.org/repos/rospbian that I use in conjunction with a ssh tunnel to upload the packages (my buildfarm is not reachable from the internet)

On the public repo:

reprepro -Vb repos/rospbian update

status page

1. it can be generated with the buildfarm tools: ./scripts/generate_status_page.py (use --help to figure out the options, it should be straightforward)
2. contact me

3. in case something goes wrong, I am still reachable: paul.mathieu@supaero.org

Notes:

1. mongodb: the raspbian repository does not include mongodb-dev. You will need to use the one that I made and include it in the building & public repositories. You can also contact Mike Thompson (mpthompson@gmail.com) to ask him to include the .deb. The sources, .dsc files and .deb files can be found under the mongodb folder. You should only need the .deb files.

2. ecto: ecto builds, but there is a bug in the catkin/cmake rules that result in an error in the ectoConfig.cmake that other programs use when the want to link against it. As a result, they do not build. This is why the object_recognition repositories could not be built.

NOTE: When you change the apt configuration of the slave (like add keys) OR you add/remove apt-repsitories, delete /var/cache/pbuilder-*

Alternatively when you add/delete apt repos in binary_build.sh.em, you don’t have to necessarily delete /var/cache/pbuilder*...instead you can increment the basetgz_version variable.

Note: When you change the rosdistro files, caches need to be rebuilt as in step 4

export ROSDISTRO_INDEX_URL=http://wbr.willowgarage.com/repos/rosdistro/index.yaml