277 lines
15 KiB
Markdown
277 lines
15 KiB
Markdown
---
|
|
title: 'Building debian packages in a cleanroom'
|
|
date: 2011-07-14T16:19:00.000Z
|
|
draft: false
|
|
taxonomies:
|
|
categories: [ "os" ]
|
|
tags: [ "debian" ]
|
|
---
|
|
|
|
### Overview and Goals
|
|
|
|
We build our solutions mostly on Ubuntu Natty and we deploy to Debian (currently lenny). One problem we face is that Debian has a slow release cycle and the packages are dated. Before a new release is approved and deployed to our target servers it can still take many months causing us to have to use up to 3 year old technology. So we are often faced to 'backport' packages or debianize existing packages if we want to use the current releases. In the past we had different build servers for the target architectures. However this is a heavy solution and scales poorly. It also makes upgrading to the next release that much heavier. So we need a system for building debian packages that is :
|
|
|
|
1. Fully automated
|
|
2. Target multiple distributions Debian (lenny,squeeze) and Ubuntu(natty, maverick)
|
|
3. Build on development machines(a) and Jenkins/Hudson CI servers(b)
|
|
4. easily configurable
|
|
5. memorizable process
|
|
|
|
The goal is to make packages for internal consumption, and the process outlined here falls short of the community standards.
|
|
|
|
### Enter pbuilder
|
|
|
|
Of course we are not the first or only one with this issue. In fact we are laggards and there are excellent articles on the 'net to help us with these goals. e.g.
|
|
|
|
* [PBuilder User Manual](http://www.netfort.gr.jp/~dancer/software/pbuilder-doc/pbuilder-doc.html)
|
|
* [Pbuilder Tricks on the Debian Wiki](http://wiki.debian.org/PbuilderTricks)
|
|
* [PBuilder How To over at the Ubuntu Wiki](https://wiki.ubuntu.com/PbuilderHowto)
|
|
|
|
The **pbuilder** program create a clean room environment of a freshly installed empty debian or ubuntu distro, chroot into it and starts building based on the project metadata, mostly from the **debian/control** file. It does this by unpacking a preconfigured **base** image of the selectable target , installing the build dependencies, building the package in the cleanroom, moving the artifacts to the hosting machine and cleaning everything up again. And it does this actually surprisingly fast. This clearly satisfies goals 1 and 2 (and half of 3 if we assume a developer has full control over his laptop). The **pbuilder** is configured through commandline options, which are clear and friendly enough but you end up with commandlines of several lines long which are impossible to type in a shell and are a maintenance nightmare in build scripts (clearly conflicts with point 5). Also in the ideal world we would be able to retarget a build without touching the checked out files, e.g. with environment variable (see goals 3 and 4).
|
|
|
|
### Configuring pbuilder
|
|
|
|
On the Pbuilders Tricks page I found a big smart shell script to use as the **pbuilder** configuration file **~/.pbuilderrc**.
|
|
|
|
\# Codenames for Debian suites according to their alias. Update these when needed.
|
|
UNSTABLE\_CODENAME="sid"
|
|
TESTING\_CODENAME="wheezy"
|
|
STABLE\_CODENAME="squeeze"
|
|
OLDSTABLE\_CODENAME="lenny"
|
|
STABLE\_BACKPORTS\_SUITE="$STABLE\_CODENAME-backports"
|
|
|
|
\# List of Debian suites.
|
|
DEBIAN\_SUITES=($UNSTABLE\_CODENAME $TESTING\_CODENAME $STABLE\_CODENAME $OLDSTABLE\_CODENAME
|
|
"unstable" "testing" "stable" "oldstable")
|
|
|
|
\# List of Ubuntu suites. Update these when needed.
|
|
UBUNTU\_SUITES=("natty" "maverick" "jaunty" "intrepid" "hardy" "gutsy")
|
|
|
|
\# Mirrors to use. Update these to your preferred mirror.
|
|
DEBIAN\_MIRROR="ftp.be.debian.org"
|
|
UBUNTU\_MIRROR="mirrors.kernel.org"
|
|
|
|
\# Optionally use the changelog of a package to determine the suite to use if
|
|
\# none set.
|
|
if \[ -z "${DIST}" \] && \[ -r "debian/changelog" \]; then
|
|
DIST=$(dpkg-parsechangelog | awk '/^Distribution: / {print $2}')
|
|
# Use the unstable suite for Debian experimental packages.
|
|
if \[ "${DIST}" == "experimental" \]; then
|
|
DIST="unstable"
|
|
fi
|
|
fi
|
|
|
|
\# Optionally set a default distribution if none is used. Note that you can set
|
|
\# your own default (i.e. ${DIST:="unstable"}).
|
|
: ${DIST:="$(lsb\_release --short --codename)"}
|
|
|
|
\# Optionally set the architecture to the host architecture if none set. Note
|
|
\# that you can set your own default (i.e. ${ARCH:="i386"}).
|
|
: ${ARCH:="$(dpkg --print-architecture)"}
|
|
|
|
NAME="$DIST"
|
|
if \[ -n "${ARCH}" \]; then
|
|
NAME="$NAME-$ARCH"
|
|
DEBOOTSTRAPOPTS=("--arch" "$ARCH" "${DEBOOTSTRAPOPTS\[@\]}")
|
|
fi
|
|
|
|
BASETGZ="/var/cache/pbuilder/$NAME-base.tgz"
|
|
DISTRIBUTION="$DIST"
|
|
BUILDRESULT="/var/cache/pbuilder/$NAME/result/"
|
|
APTCACHE="/var/cache/pbuilder/$NAME/aptcache/"
|
|
BUILDPLACE="/var/cache/pbuilder/build/"
|
|
|
|
\# make sure folders exist
|
|
mkdir -p $BUILDRESULT
|
|
mkdir -p $APTCACHE
|
|
|
|
echo "Target : $BUILDRESULT" >>/tmp/dist
|
|
|
|
if $(echo ${DEBIAN\_SUITES\[@\]} | grep -q $DIST); then
|
|
|
|
OTHERMIRROR="deb file:///var/cache/pbuilder/$NAME/result ./"
|
|
BINDMOUNTS="/var/cache/pbuilder/$NAME/result"
|
|
HOOKDIR="/var/cache/pbuilder/$NAME/hooks"
|
|
EXTRAPACKAGES="apt-utils"
|
|
# Debian configuration
|
|
MIRRORSITE="http://$DEBIAN\_MIRROR/debian/"
|
|
COMPONENTS="main contrib non-free"
|
|
DEBOOTSTRAPOPTS=("${DEBOOTSTRAPOPTS\[@\]}" "--keyring=/usr/share/keyrings/debian-archive-keyring.gpg")
|
|
if $(echo "$STABLE\_CODENAME stable" | grep -q $DIST); then
|
|
EXTRAPACKAGES="$EXTRAPACKAGES debian-backports-keyring"
|
|
OTHERMIRROR="$OTHERMIRROR | deb http://www.backports.org/debian $STABLE\_BACKPORTS\_SUITE $COMPONENTS"
|
|
fi
|
|
elif $(echo ${UBUNTU\_SUITES\[@\]} | grep -q $DIST); then
|
|
# Ubuntu configuration
|
|
MIRRORSITE="http://$UBUNTU\_MIRROR/ubuntu/"
|
|
COMPONENTS="main restricted universe multiverse"
|
|
DEBOOTSTRAPOPTS=("${DEBOOTSTRAPOPTS\[@\]}" "--keyring=/usr/share/keyrings/ubuntu-archive-keyring.gpg")
|
|
else
|
|
echo "Unknown distribution: $DIST"
|
|
exit 1
|
|
fi
|
|
|
|
|
|
I just updated the distribution names to the current situation and added the directory where the packages are collected as a repository so subsequent builds can use these packages as dependencies. I also specified the keyrings to use for Debian and Ubuntu and made sure the expected folders are created to mount them in the clean room. I created this in my account on my development laptop and added a symbolic link in ~root/.pbuilderrc to this file so I can update it from my desktop environment and do not have to get my brain all twisted up to try to remember with which configuration I am busy in my shell, sudo, **su -**, … THe way the script works is that the configuration adapts itself to the content of the **DIST** and **ARCH** environment variables. So to configure **lenny-amd64** as target is sufficient to do
|
|
|
|
~ > export DIST=lenny
|
|
~ > export ARCH=amd64
|
|
|
|
|
|
This approach is also perfect Jenkins or Hudson to determine the target build from checked out sources, since it can be specified in the build recipe. (satisfies goals 3b, 4 and 5) Since we have to run these programs using sudo we must make sure the environment variables are passed by sudo. We can do this in the **Defaults** line of the **/etc/sudoers** file with the **envkeep** instruction.
|
|
|
|
...
|
|
Defaults env\_reset,env\_keep="DIST ARCH http\_proxy ftp\_proxy
|
|
https\_proxy no\_proxy"
|
|
... snip ...
|
|
\# Cmnd alias specification
|
|
Cmnd\_Alias PBUILDER=/usr/sbin/pbuilder, /usr/bin/pdebuild
|
|
... snip to end of file ...
|
|
\# Allow members of group sudo to execute any command
|
|
%sudo ALL=(ALL:ALL) ALL
|
|
|
|
pti ALL=(ALL) NOPASSWD: PBUILDER
|
|
jenkins ALL=(ALL) NOPASSWD: PBUILDER
|
|
\#includedir /etc/sudoers.d
|
|
|
|
You add the **DIST** and **ARCH** variables there. I also included the environment variables for proxying so I can easily switch between environment on my laptop and these changes propagate to sudo (which is also useful for plain apt-get, by the way). I also added a line to show how to make the tools available for a user without having to give their password. This is not needed for interactive work, but very much so for the user as which the CI server is running (in our case **jenkins**). Note that the definition should be after the group definitions, otherwise these take precedence and jenkins has to provide his password (read: is hanging during the build).
|
|
|
|
### Creating the target base images
|
|
|
|
The heavy lifting is now done. Let's create an **base.tgz** for lenny-amd64.
|
|
|
|
~ > export DIST=lenny
|
|
~ > export ARCH=amd64
|
|
~ > sudo pbuilder create
|
|
|
|
Now go and have a cup of coffee (or read some emails). Rinse and repeat for the other target platforms.
|
|
|
|
### Backporting existing packages
|
|
|
|
In theory backporting would be as simple as
|
|
|
|
~ ᐅ cd tmp
|
|
~/tmp ᐅ apt-get source mongodb
|
|
Reading package lists... Done
|
|
Building dependency tree
|
|
Reading state information... Done
|
|
Need to get 1,316 kB of source archives.
|
|
Get:1 http://be.archive.ubuntu.com/ubuntu/ natty/universe mongodb 1:1.6.3-1ubuntu2 (dsc) \[2,276 B\]
|
|
Get:2 http://be.archive.ubuntu.com/ubuntu/ natty/universe mongodb 1:1.6.3-1ubuntu2 (tar) \[1,285 kB\]
|
|
Get:3 http://be.archive.ubuntu.com/ubuntu/ natty/universe mongodb 1:1.6.3-1ubuntu2 (diff) \[29.0 kB\]
|
|
Fetched 1,316 kB in 1s (679 kB/s)
|
|
gpgv: Signature made Thu 17 Mar 2011 11:49:37 PM CET using RSA key ID D5946E0F
|
|
gpgv: Can't check signature: public key not found
|
|
dpkg-source: warning: failed to verify signature on ./mongodb\_1.6.3-1ubuntu2.dsc
|
|
dpkg-source: info: extracting mongodb in mongodb-1.6.3
|
|
dpkg-source: info: unpacking mongodb\_1.6.3.orig.tar.gz
|
|
dpkg-source: info: unpacking mongodb\_1.6.3-1ubuntu2.debian.tar.gz
|
|
dpkg-source: info: applying debian-changes-1:1.6.3-1
|
|
dpkg-source: info: applying build-process-remove-rpath
|
|
dpkg-source: info: applying mozjs185
|
|
~/tmp ᐅ DIST=lenny ARCH=amd64 sudo pbuilder build mongodb\_1.6.3-1ubuntu2.dsc
|
|
I: using fakeroot in build.
|
|
I: Current time: Thu Jul 14 14:28:17 CEST 2011
|
|
I: pbuilder-time-stamp: 1310646497
|
|
I: Building the build Environment
|
|
I: extracting base tarball \[/var/cache/pbuilder/lenny-amd64-base.tgz\]
|
|
...
|
|
|
|
and you should get a nice set of debian packages in \*/var/cache/pbuilder/lenny-amd64. In practice you will often end up with errors like :
|
|
|
|
... snip ...
|
|
The following packages have unmet dependencies:
|
|
pbuilder-satisfydepends-dummy: Depends: xulrunner-dev (>= 2.0~) but it is not installable
|
|
The following actions will resolve these dependencies:
|
|
|
|
Remove the following packages:
|
|
pbuilder-satisfydepends-dummy
|
|
|
|
Score is -9850
|
|
|
|
Writing extended state information... Done
|
|
... snip ...
|
|
I: cleaning the build env
|
|
I: removing directory /var/cache/pbuilder/build//6279 and its subdirectories
|
|
|
|
In these case you have to walk the dependency tree till you find the leafs, and walk back up the branches to the trunk. Note also that chances are that unless you target machines which only serve a very specific purpose, you might end up with packages which are uninstallable since you pull out the rug from other installed packages. However we have the principle to use 1 virtual host to deliver 1 service, hence there are very little packages deployed to them and nothing complicated like desktop environments. Simple leaf packages often build without a hitch:
|
|
|
|
~/tmp ᐅ DIST=lenny sudo pbuilder build libevent\_1.4.13-stable-1.dsc
|
|
I: using fakeroot in build.
|
|
I: Current time: Thu Jul 14 14:44:00 CEST 2011
|
|
I: pbuilder-time-stamp: 1310647440
|
|
I: Building the build Environment
|
|
I: extracting base tarball \[/var/cache/pbuilder/lenny-amd64-base.tgz\]
|
|
I: creating local configuration
|
|
I: copying local configuration
|
|
I: mounting /proc filesystem
|
|
I: mounting /dev/pts filesystem
|
|
I: Mounting /var/cache/pbuilder/ccache
|
|
... snip ...
|
|
dpkg-genchanges: including full source code in upload
|
|
dpkg-buildpackage: full upload (original source is included)
|
|
W: no hooks of type B found -- ignoring
|
|
I: Copying back the cached apt archive contents
|
|
I: unmounting /var/cache/pbuilder/lenny-amd64/result filesystem
|
|
I: unmounting /var/cache/pbuilder/ccache filesystem
|
|
I: unmounting dev/pts filesystem
|
|
I: unmounting proc filesystem
|
|
I: cleaning the build env
|
|
I: removing directory /var/cache/pbuilder/build//15214 and its subdirectories
|
|
I: Current time: Thu Jul 14 14:49:57 CEST 2011
|
|
I: pbuilder-time-stamp: 1310647797
|
|
~/tmp ᐅ ls -al /var/cache/pbuilder/lenny-amd64/result
|
|
total 5260
|
|
drwxr-xr-x 2 root root 4096 2011-07-13 19:47 .
|
|
drwxr-xr-x 5 root root 4096 2011-07-13 19:13 ..
|
|
-rw-r--r-- 1 pti pti 2853 2011-07-14 14:49 libevent\_1.4.13-stable-1\_amd64.changes
|
|
-rw-r--r-- 1 pti pti 9129 2011-07-14 14:49 libevent\_1.4.13-stable-1.diff.gz
|
|
-rw-r--r-- 1 pti pti 907 2011-07-14 14:49 libevent\_1.4.13-stable-1.dsc
|
|
-rw-r--r-- 1 pti pti 499603 2009-12-05 23:04 libevent\_1.4.13-stable.orig.tar.gz
|
|
-rw-r--r-- 1 pti pti 61956 2011-07-14 14:49 libevent-1.4-2\_1.4.13-stable-1\_amd64.deb
|
|
-rw-r--r-- 1 pti pti 31262 2011-07-14 14:49 libevent-core-1.4-2\_1.4.13-stable-1\_amd64.deb
|
|
-rw-r--r-- 1 pti pti 172950 2011-07-14 14:49 libevent-dev\_1.4.13-stable-1\_amd64.deb
|
|
-rw-r--r-- 1 pti pti 51588 2011-07-14 14:49 libevent-extra-1.4-2\_1.4.13-stable-1\_amd64.deb
|
|
-rw-r--r-- 1 root root 9051 2011-07-14 14:48 Packages
|
|
~/tmp ᐅ
|
|
|
|
### Using **pdebuild** for building packages
|
|
|
|
Many of our packages are debianized and can be build using **debuild**. I use here the Ubuntu sources of tokyocabinet as an example (which uses the libevent package we just built, btw):
|
|
|
|
~/tmp/tokyocabinet-1.4.37 ᐅ DIST=lenny ARCH=amd64 pdebuild
|
|
...snip...
|
|
dpkg-genchanges >../tokyocabinet\_1.4.37-6ubuntu1\_amd64.changes
|
|
dpkg-genchanges: not including original source code in upload
|
|
dpkg-buildpackage: binary and diff upload (original source NOT included)
|
|
W: no hooks of type B found -- ignoring
|
|
I: Copying back the cached apt archive contents
|
|
I: unmounting /var/cache/pbuilder/lenny-amd64/result filesystem
|
|
I: unmounting /var/cache/pbuilder/ccache filesystem
|
|
I: unmounting dev/pts filesystem
|
|
I: unmounting proc filesystem
|
|
I: cleaning the build env
|
|
I: removing directory /var/cache/pbuilder/build//4199 and its subdirectories
|
|
I: Current time: Thu Jul 14 15:05:27 CEST 2011
|
|
I: pbuilder-time-stamp: 1310648727
|
|
~/tmp/tokyocabinet-1.4.37 ᐅ ls /var/cache/pbuilder/lenny-amd64/result
|
|
...snip...
|
|
tokyocabinet\_1.4.37-6ubuntu1\_amd64.changes
|
|
tokyocabinet\_1.4.37-6ubuntu1.debian.tar.gz
|
|
tokyocabinet\_1.4.37-6ubuntu1.dsc
|
|
tokyocabinet\_1.4.37.orig.tar.gz
|
|
tokyocabinet-bin\_1.4.37-6ubuntu1\_amd64.deb
|
|
tokyocabinet-doc\_1.4.37-6ubuntu1\_all.deb
|
|
|
|
Sometimes the dependencies break on the version of debhelpers. This version is added conservatively by the dh\* scripts and often is overly conservative. Many packages build just fine with older versions of the debhelpers.
|
|
|
|
### Setting up automated build
|
|
|
|
To set this up on the build server we have to replicate the steps above
|
|
|
|
1. Create the ~/.pbuilderrc file
|
|
2. Symbolic link to this file in ~root/.pbuilderrc
|
|
3. Allow jenkins to use sudo for building packages
|
|
4. Create a jenkins job to (re)build the packages
|
|
5. Create jobs to build the packages
|
|
|