Build Daemon Infrastructure

Summary

This pages documents the proposed infrastructure for handling package uploads, builds, and archive inclusions.

Only source package uploads are accepted, and the process is simple and acyclic. SFTP is used to upload packages for ease of maintainer authorization (uploaded files need not be signed; uploaders are simply authorized by their SSH keypair).

Process

The following diagram summarizes the process:

The general process is as follows:

1. A maintainer uploads a source package (not necessarily his or her own – non-maintainer uploads are allowed) and changes file over SFTP into the review queue. (This can be done by hand or with the assistance of a script similar to Debian's dupload and dput scripts.)
2. A reviewer (whose job is similar to that of Debian's ftpmaster team members) examines the changes file and, if necessary, the source package itself and decides to either accept or reject the upload. If accepted, the package is added to the master build queue and all of the relevant build queues.
3. Each build daemon periodically checks the queues for which it is responsible. If packages are waiting to be built, it attempts to build the oldest one (first in, first out). If the build is successful, the daemon adds the produced binary packages to the archive queue. Either way, the queue run ends with the removal of the processed source package from the relevant queue.
4. When the master build daemon detects that a package no longer exists in any architecture build queue, it moves the source package and changes file into the archive queue.
5. When the archive inclusion daemon finds a source package in the archive queue, it includes in the archive the source package along with all of the built binary packages.
6. Users can finally download the source and binary packages from the archive using opkg.

Daemons and Utilities

buildq

buildq will be a utility used by upload reviewer(s) to either accept or reject package uploads.

As this is written, uploads and build daemons must all be on the same filesystem. This can be changed someday by introducing a new queueing daemon and reducing buildq to a simple uploader (removing all queueing logic).

This utility handles step 2 in the process above.

Usage:

buildq accept <source> <version>
buildq reject <source> <version>

Operation:

write remarks template to file
$EDITOR $remarks
if action == accept
    touch src:$source_$version_src.opk
    ln src:$source_$version_src.opk $masterbuildqueue/
    remove " all " from space-padded Architecture field value
    if value changed
        ln src:$source_$version_src.opk $archindepbuildqueue/
    remove " any " from space-padded Architecture field value
    if value changed
        foreach archbuildqueue
            ln src:$source_$version_src.opk $archbuildqueue/
    else
        foreach archbuildqueue
            if Architecture value matches queue arch (use libopkh for this)
                ln src:$source_$version_src.opk $archbuildqueue/
rm src:$source_$version_src.opk
mail maintainer and uploader

Example mail message:

From: ProteanOS uploads team <proteanos-uploads@lists.proteanos.com>
To: ProteanOS Foo Maintainers <proteanos-pkg-foo@lists.proteanos.com>,
 "P. J. McDermott" <pehjota@proteanos.com>
Subject: Accepted foo 1.0-1

Remarks:
 Your upload has been accepted.  The package will be serviced by the 
 build daemons shortly.  You will be notified if your package fails to
 build for any architectures or platforms.
Source: foo
Version: 1.0-1
Architecture: any all
Platform: all
Distribution: trunk
Maintainer: ProteanOS Foo Maintainers <proteanos-pkg-foo@lists.proteanos.com>
Changed-By: "P. J. McDermott" <pehjota@proteanos.com>
Date: Sat, 04 Aug 2012 10:31:40 -0400
Changes:
 foo (1.0-1) trunk
 .
   * New package.

buildd

buildd will be a daemon that builds packages in its queue(s).

One buildd instance can be responsible for multiple architectures if the queues are linked together.

This daemon handles step 3 in the process above.

Operation:

while true
    headsrcopk = peek $queue (ls -1cr $queue | head -n 1)
    if no $headsrcopk
        sleep 00:05:00
        continue
    foreach host architecture+platform
        if ! Architecture value matches host arch (use libopkh for this)
            continue
        run builder on $headsrcopk
        if success
            mv *.opk $archivequeue/$source
        else
            rm *.opk
            touch $masterbuildqueue/$source_$version.fail
            mail log to maintainer and uploader
    pop $headsrcopk (rm $headsrcopk)

builddd

A possible name expansion is "builded it" (as in "Where's my package? Have the build daemons builded it yet?") when spoken quickly.

builddd will be a daemon that watches the build daemons and acts on successful and failed builds.

This daemon handles step 4 in the process above.

Operation:

while true
    doneheadsrcopk = oldest opk in $masterbuildqueue with exactly 1 link
    if no $doneheadsrcopk
        sleep 00:05:00
        continue
    if no $masterbuildqueue/$source_$version.fail
        mv $doneheadsrcopk $archivequeue/$package_$version
    else
        rm -Rf $archivequeue/$package_$version

archived

archived will be a daemon that includes built binary packages into the package archive.

This daemon handles step 5 in the process above.

Operation:

while true
    foreach dir in $archivequeue
        if exists src opk in $dir
            include $dir/*.opk

Filesystem Layout

 +- <uploads-root>/
 |   \- incoming/
 +- <build-queue-root>/
 |   +- master/
 |   +- arch/
 |   |   +- all/ -> ../buildd/core/
 |   |   +- core-linux-eglibc/ -> ../buildd/core/
 |   |   +- i686-linux-eglibc/ -> ../buildd/i686/
 |   |   +- cortexa8-linux-eglibc/ -> ../buildd/cortexa8/
 |   |   \- cortexa8-linux-uclibc/ -> ../buildd/cortexa8/
 |   \- buildd/
 |       +- core/
 |       +- i686/
 |       \- cortexa8/
 \- <archive-queue-root>/
     \- <source>_<source-version>/