← Back to Package Development Tutorials
In this introduction to software packaging, we will package the Expat XML parser library. This is a pretty simple but complete package, consisting of a shared library and its development files plus an executable utility and some documentation.
We will build a package in Source Package Format 2.0 (SPF 2.0) with the assistance of opkhelper 3.0.x.
It is assumed that prokit is installed and a ProteanOS development system is set up.
This tutorial assumes some knowledge of the UNIX shell command language and utilities (see the "Shell and Utilities" volume of POSIX.1-2008) and at least basic familiarity with makefile syntax.
This tutorial presents one possible packaging workflow that seems to work well. There is no mandatory workflow to packaging. The only requirements are those made by the source package format and any build helper utilities that are used.
Getting Started
Source Package Directory
First, make a source package directory. This is the directory that will contain all of our source package files. SPF 2.0 makes no requirements on the name of this directory, but using the name of the source package is recommended.
$ mkdir expat
$ cd expat
We need a file called format to identify the format of our
source package. For SPF 2.0, it should simply contain the string 2.0.
$ echo '2.0' >format
Upstream Source Archive
Obviously we need the source code of the software to be packaged. Go to Expat's Web site, find the expat 2.1.0 archive, and download it into the source package directory.
$ wget 'http://downloads.sourceforge.net/project/expat/expat/2.1.0/expat-2.1.0.tar.gz'
SPF 2.0 requires that an upstream source archive be named
<pkgname>-<pkgver>.orig.tar<ext>, where <pkgname> is the name of the source
package, <pkgver> is the upstream version of the source package, and <ext>
is an optional file extension to indicate compression. So, rename the archive
accordingly.
$ mv 'expat-2.1.0.tar.gz' 'expat-2.1.0.orig.tar.gz'
Source Package Metadata
Now we need some metadata for our source package.
Control File
First we'll make a control file. The format of this file is not yet
documented in the SPF 2.0 specification, but it is documented in the Debian
Policy Manual. The source package fields are
Maintainer (required), Build-Depends (optional), and Homepage (optional).
We'll fill in the fields whose values we know right now: Maintainer and
Homepage.
Maintainer is the name and e-mail address of the person or team responsible
for the package (i.e. usually you when you are making a package). The value
must follow the syntax of the mailbox symbol in RFC 5322 section
3.4. That is, the value must be of the form name <address>.
If name contains any of the following characters, it must be in double quotes:
( ) < > [ ] : ; @ \ , .
Build-Depends is a comma-separated list of packages needed to build the
package. We're using opkhelper-3.0, so we need to list that.
Homepage is the URL of the Web site for the package, if such a site exists.
Our expat control file looks like this:
Maintainer: "J. Random Hacker" <jrandom@example.com>
Build-Depends: opkhelper-3.0
Homepage: http://expat.sourceforge.net/
Change Log
Now we'll make a changelog file. The format of this file is documented in
the SPF 2.0 specification. We're making version "2.1.0-1" of
the "expat" source package for the "trunk" distribution. We can get the current
date and time in the RFC 5322 format using the date(1) command:
$ LC_ALL='POSIX' date '+%a, %d %b %Y %H:%M:%S %z'
Our expat changelog file looks like this:
expat (2.1.0-1) trunk
* Initial release.
-- "J. Random Hacker" <jrandom@example.com> Sun, 18 Nov 2012 11:58:19 -0500
Be careful – the changelog format is quite strict and tends to confuse many new
package maintainers. There are no spaces before the package name, version, and
distribution (called the "header"); at least two spaces before each line of
change details; one space before the line starting with -- (called the
"trailer"); and two spaces between the maintainer and the date.
Building the Software
We can now write our build makefile to try to get the Expat software to build.
The build makefile "directs the process of building and
installing data files to be provided by binary packages".
Looking Through the Source
With a "no-op" target in build, we can make opkbuild prepare a
build work area with the unpacked source code and stop. This
target isn't required by SPF 2.0, but it seems to facilitate a nice workflow.
So begin writing build as follows:
#!/usr/bin/make -f
nop:
@:
Note that, due to makefile syntax, the line after nop: must begin with a tab
character. This line is called a "command line" in makefile syntax. The :
utility is a "null utility" that returns an exit status of zero.
A command prefix of @ tells make(1) to not write the command to standard
output before executing it.
The build makefile must be executable, so set its file mode:
$ chmod 755 build
We can now make opkbuild prepare our build work area.
$ sudo prokit build $root -b -c -T nop .
We're running opkbuild through
prokit-build(8). $root is the root
directory of our ProteanOS development system.
The options are explained in the help output of opkbuild, obtained by
running prokit build $root -h .. The -b option tells opkbuild to build
only binary packages (no source package). The -c option tells it to not clean
up the work area after building packages. The -T option specifies a target to
be built instead of the standard build and install targets.
Now look in tmp/src/, the location of the source code within the build work
area.
$ ls tmp/src/
Look for some documentation file that might tell us how to build Expat. This
kind of information is usually kept in a file called INSTALL or README.
Expat's README file says to run ./configure, then make and make install.
Looking at tmp/src/configure, we see that it is "[g]enerated by GNU Autoconf
2.68 for expat 2.1.0". The tmp/src/README file reports that the makefile
supports the use of either the DESTDIR or INSTALL_ROOT macro to install
Expat somewhere other than in the root of the filesystem. So, we should be able
to use opkhelper's buildsystem utilities to automatically configure, build, and
install Expat for us.
Building
So let's add a build target to our build makefile. The makefile should now
look like this:
#!/usr/bin/make -f
nop:
@:
build:
oh-autoconfigure
oh-autobuild
touch $@
Read the manual pages and/or source code of oh-autoconfigure(1) and oh-autobuild(1) to learn more about what they do.
The touch $@ command is recommended by SPF 2.0:
The build target should create a file named build in the build work area to prevent configuration and compilation from being performed multiple times.
We can now build Expat.
$ sudo prokit build $root -b -c -T build .
Installing the Software
We can now finish our build makefile to install the Expat software and make
some binary packages.
Installing
Add a basic install target to the build makefile. The makefile should now
look like this:
#!/usr/bin/make -f
nop:
@:
build:
oh-autoconfigure
oh-autobuild
touch $@
install: build
oh-autoinstall
The install target is declared as depending on the build target:
install: build
Read the manual page and/or source code of oh-autoinstall(1) to learn more about what it does.
Install Expat:
$ sudo prokit build $root -b -c -T install .
Splitting Files Into Binary Packages
Look in the installation destination directory tmp/dest/ for files installed
by Expat's build system. This can be done with the find(1) command, which
results in the following when building for the amd64-linux-glibc architecture:
$ find tmp/dest -exec ls -Fd '{}' ';' | sed 's|^tmp/dest||'
/
/usr/
/usr/bin/
/usr/bin/xmlwf*
/usr/share/
/usr/share/man/
/usr/share/man/man1/
/usr/share/man/man1/xmlwf.1
/usr/lib/
/usr/lib/amd64-linux-glibc/
/usr/lib/amd64-linux-glibc/pkgconfig/
/usr/lib/amd64-linux-glibc/pkgconfig/expat.pc
/usr/lib/amd64-linux-glibc/libexpat.so@
/usr/lib/amd64-linux-glibc/libexpat.a
/usr/lib/amd64-linux-glibc/libexpat.la*
/usr/lib/amd64-linux-glibc/libexpat.so.1@
/usr/lib/amd64-linux-glibc/libexpat.so.1.6.0*
/usr/include/
/usr/include/expat_external.h
/usr/include/expat.h
We have the libexpat.so.1.6.0 shared library and two symbolic links to it:
libexpat.so.1 and libexpat.so. We have the libexpat.a static library and
associated libexpat.la library metadata file generated by GNU libtool. We
have a pkg-config file and two header files. We have an executable utility and
an associated manual page.
We should therefore split these files into four binary packages: one for the shared library, one for the library development files, one for the utility, and one for the utility's documentation.
To find out what we should call the library package, we can use objdump(1) to get the SONAME of the library:
$ objdump -p tmp/dest/usr/lib/amd64-linux-glibc/libexpat.so.1.6.0 | grep SONAME
SONAME libexpat.so.1
We should name our library package after the SONAME of the shared library,
without .so. The binary package shall be named libexpat.1.
The versionless libexpat.so link is only needed by ld(1) when linking a
just-compiled object with the -lexpat linker flag. So this can be provided by
our library development package. Also provided by that package will be the
header files, the pkg-config file, and the static library. The development
package can be called libexpat.1-dev.
The xmlwf utility can be provided by a package called simply xmlwf.
The xmlwf.1 manual page can be provided by a package called xmlwf-doc.
Binary Packages
Binary Package Metadata
Each binary package to be built needs to have a directory for its metadata. So let's create directories for our packages.
$ mkdir libexpat.1.pkg libexpat.1-dev.pkg xmlwf.pkg xmlwf-doc.pkg
SPF 2.0 requires a control file for each binary package. The format of this
file is the same as that of the source package control file. The required
binary package fields are Architecture, Platform, and
Description.
None of these binary packages are platform-specific, so they will all have a
Platform: all field. All of the binary packages except xmlwf-doc are
architecture-specific; that is, they provide files whose contents depend on the
host architecture (files like executable and linkable objects). So xmlwf-doc
will have an Architecture: all field while the others will have Architecture:
any fields.
Let's start with the libexpat.1.pkg/control file:
Architecture: any
Platform: all
Description: XML parser library
Expat is an XML parser library written in C. It is a stream-oriented parser in
which an application registers handlers for things the parser might find in the
XML document (like start tags).
That's fairly simple.
Now let's write a control file for libexpat.1-dev. Because it provides
development files for libexpat.so.1, libexpat.1-dev should depend on the
libexpat.1 package. This should be a versioned dependency, because the
libexpat.so symbolic link points to a specific version of libexpat.so.
Architecture: any
Platform: all
Depends: libexpat.1 (= 2.1.0-1)
Description: XML parser library - development files
Expat is an XML parser library written in C. It is a stream-oriented parser in
which an application registers handlers for things the parser might find in the
XML document (like start tags).
.
This package provides development files for Expat.
Next is xmlwf, which should also depend on libexpat.1 since the xmlwf
utility is dynamically linked against the libexpat.so.1 library.
Architecture: any
Platform: all
Depends: libexpat.1
Description: XML parser library - example application
This package provides an example application of Expat that determines if an XML
document is well-formed.
Finally, we can write metadata for xmlwf-doc, which should depend on xmlwf
since it documents the xmlwf utility.
Architecture: all
Platform: all
Depends: xmlwf
Description: XML parser library - example application documentation files
This package provides the manual page for xmlwf, an example application of
Expat that determines if an XML document is well-formed.
Binary Package Data Files
The oh-installfiles(1) utility of
opkhelper, which we'll be using to install
files into binary package data directories, requires a files file for each
binary package that is to provide data files.
Recall how we decided to split files between packages. We will now write pathname patterns to do this.
Again, let's start with libexpat.1. We can write the following pattern in
libexpat.1.pkg/files:
/usr/lib/*/libexpat.so.*
This will match /usr/lib/amd64-linux-glibc/libexpat.so.1 and
/usr/lib/amd64-linux-glibc/libexpat.so.1.6.0; these two files will be provided
by libexpat.1.
The patterns for libexpat.1-dev are a little more complicated:
/usr/include
/usr/lib/*/libexpat.so
/usr/lib/*/libexpat.a
/usr/lib/*/pkgconfig
The first pattern simply matches the directory containing header files. The
second matches the versionless symbolic link; remember this is used by ld(1)
to link a just-compiled object against libexpat.so.1.6.0. The third matches
the static library, and the fourth matches the directory containing the
expat.pc pkg-config file.
xmlwf.pkg/files need only contain a pattern to match the directory containing
the xmlwf utility.
/usr/bin
xmlwf-doc.pkg/files is similarly simple:
/usr/share/man/man1
With these pathname patterns done, we can add oh-installfiles(1) to our
build makefile:
#!/usr/bin/make -f
nop:
@:
build:
oh-autoconfigure
oh-autobuild
touch $@
install: build
oh-autoinstall
oh-installfiles
Now run opkbuild again:
$ sudo prokit build $root -b -c -T install .
You can verify that all files were installed where they should be:
$ find tmp/*.data -exec ls -Fd '{}' ';'
tmp/libexpat.1.data/
tmp/libexpat.1.data/usr/
tmp/libexpat.1.data/usr/lib/
tmp/libexpat.1.data/usr/lib/amd64-linux-glibc/
tmp/libexpat.1.data/usr/lib/amd64-linux-glibc/libexpat.so.1@
tmp/libexpat.1.data/usr/lib/amd64-linux-glibc/libexpat.so.1.6.0*
tmp/libexpat.1-dev.data/
tmp/libexpat.1-dev.data/usr/
tmp/libexpat.1-dev.data/usr/lib/
tmp/libexpat.1-dev.data/usr/lib/amd64-linux-glibc/
tmp/libexpat.1-dev.data/usr/lib/amd64-linux-glibc/pkgconfig/
tmp/libexpat.1-dev.data/usr/lib/amd64-linux-glibc/pkgconfig/expat.pc
tmp/libexpat.1-dev.data/usr/lib/amd64-linux-glibc/libexpat.so@
tmp/libexpat.1-dev.data/usr/lib/amd64-linux-glibc/libexpat.a
tmp/libexpat.1-dev.data/usr/include/
tmp/libexpat.1-dev.data/usr/include/expat_external.h
tmp/libexpat.1-dev.data/usr/include/expat.h
tmp/xmlwf.data/
tmp/xmlwf.data/usr/
tmp/xmlwf.data/usr/bin/
tmp/xmlwf.data/usr/bin/xmlwf*
tmp/xmlwf-doc.data/
tmp/xmlwf-doc.data/usr/
tmp/xmlwf-doc.data/usr/share/
tmp/xmlwf-doc.data/usr/share/man/
tmp/xmlwf-doc.data/usr/share/man/man1/
tmp/xmlwf-doc.data/usr/share/man/man1/xmlwf.1
Cleaning Up Installed Files
There are few things we can do to improve our build makefile's install
target.
You may have noticed oh-installfiles(1) warn that something hasn't been installed:
oh-installfiles: Warning: Some files have not been installed into packages
With find(1), we can see that this is the libexpat.la file that GNU
libtool generated.
$ find tmp/dest -type f -exec ls -Fd '{}' ';' | sed 's|^tmp/dest||'
/usr/lib/amd64-linux-glibc/libexpat.la*
We don't need this, and we can simply delete it in the install target.
Next, note that some file permissions aren't entirely correct. For example,
libexpat.so.1.6.0 is executable, but almost all libraries need not be.
So we can call oh-fixperms(1) in our
install target to automatically set
correct permissions for us.
Finally, note that the executable and linkable objects are not stripped: they contain all of their symbols, including those only needed for debugging.
$ file tmp/libexpat.1.data/usr/lib/amd64-linux-glibc/libexpat.so.1.6.0
tmp/libexpat.1.data/usr/lib/amd64-linux-glibc/libexpat.so.1.6.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x2d88e36feeb8245bfa2f63f2f0e9a9f8232f6d2c, not stripped
$ file tmp/xmlwf.data/usr/bin/xmlwf
tmp/xmlwf.data/usr/bin/xmlwf: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.26, BuildID[sha1]=0xdb5f686930b13b8a5e7519efb446a2da14de9856, not stripped
We can call oh-strip(1) in our install target
to automatically strip
objects for us.
So our build makefile should now look like this:
#!/usr/bin/make -f
nop:
@:
build:
oh-autoconfigure
oh-autobuild
touch $@
install: build
oh-autoinstall
rm -f 'dest/usr/lib/$(OPK_HOST_ARCH)/libexpat.la'
oh-fixperms
oh-strip
oh-installfiles
Documentation and Finishing Touches
Source Package Documentation
SPF 2.0 specifies that one of the binary packages built from a source package provides documentation files about the source package and is depended upon by all of the other binary packages from the source package.
So we should pick one common binary package that should be a dependency of all
of our other binary packages. libexpat.1 is a good candidate for this, since
it is already a direct dependency of libexpat.1-dev and xmlwf and an
indirect dependency of xmlwf-doc.
Per SPF 2.0, we can mark libexpat.1 as providing source package documentation
by making a docs file in its metadata directory.
$ touch libexpat.1.pkg/docs
We should make all of our other binary packages directly depend on libexpat.1
version 2.1.0-1. For example, xmlwf-doc.pkg/control should now look like
this:
Architecture: all
Platform: all
Depends: libexpat.1 (= 2.1.0-1), xmlwf
Description: XML parser library - example application documentation files
This package provides the manual page for xmlwf, an example application of
Expat that determines if an XML document is well-formed.
Substitution Variables
We've hardcoded the libexpat.1 binary package version in many of our control
files. What will we do when we make a new version of our source package? We'll
have to change all of these values in all of these places.
Substitution variables (substvars for short) make this
unnecessary. We can just use the Binary-Version substitution variable in our
control files to refer to the version of our binary packages. For example, our
xmlwf-doc.pkg/control file should now look like this:
Architecture: all
Platform: all
Depends: libexpat.1 (= ${Binary-Version}), xmlwf
Description: XML parser library - example application documentation files
This package provides the manual page for xmlwf, an example application of
Expat that determines if an XML document is well-formed.
But that's not all! We can define our own variables as well.
Note that the descriptions of our libexpat.1 and libexpat.1-dev packages
have a common paragraph. We can put that in a file called substvars:
Common-Description:
Expat is an XML parser library written in C. It is a stream-oriented parser in
which an application registers handlers for things the parser might find in the
XML document (like start tags).
As noted by the SPF 2.0 specification, the leading newline character in the value is fine:
Values may be comprised of multiple lines, and empty lines at the beginning and end of each substitution variable value shall be removed.
We can now use this variable in our control files. Here's
libexpat.1.pkg/control:
Architecture: any
Platform: all
Description: XML parser library
${Common-Description}
And here's libexpat.1-dev.pkg/control:
Architecture: any
Platform: all
Depends: libexpat.1 (= ${Binary-Version})
Description: XML parser library - development files
${Common-Description}
.
This package provides development files for Expat.
Copyright and License Information
We're almost done; we just have one more important thing to do. We need to document the copyright information for the upstream software and our own packaging work.
This is done in the copyright file. There is currently no standard format for
this file.
We need to collect the copyright and license information from the upstream source code (usually in comments at the tops of source files).
There are some resources available to assist us with this. First, we can look
at the work already done by package maintainers in the Debian Project. Find the
copyright file for Debian's expat source package.
We see the following copyright information:
Copyright (c) 1998, 1999, 2000 Thai Open Source Software Center Ltd
and Clark Cooper
Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006 Expat maintainers.
We also see that Expat can be dealt in under the terms of, unsurprisingly, the Expat (a.k.a. "MIT") license.
Another resource we can use is the licensecheck(1) tool,
maintained in Debian's devscripts package and originally based on a script
from the KDE SDK. Recursively run licensecheck(1) to report copyright and
license information.
$ licensecheck -r --copyright tmp/src/
We see that some source files have publication dates in their copyright notices that are newer than those that Debian's copyright file lists:
tmp/src/amiga/expat_lib.c: MIT/X11 (BSD like)
[Copyright: 2001-2009 Expat maintainers / HOLDERS BE LIABLE FOR ANY]
So collect some representative copyright notices – e.g. from
tmp/src/lib/xmlparse.c, tmp/src/examples/outline.c,
tmp/src/vms/expat_config.h, and tmp/src/amiga/expat_lib.c – and add them to
the copyright file.
Then describe the license under which the software may be used. Expat is a
"common license" included under /usr/share/common-licenses/ in this
distribution, so you can refer to it there.
You should also document the location from which the source was obtained.
Finally, add your own copyright notice and license information. You should allow your work to be used under the terms of a license that is equivalent to or compatible with the terms of the upstream software's copyright license.
Your resulting copyright file might look something like this:
Upstream Source
===============
Location: <http://sourceforge.net/projects/expat/files/expat/>
Copyright (c) 1998, 1999, 2000 Thai Open Source Software Center Ltd
Copyright 1999, Clark Cooper
Copyright 2000, Clark Cooper
Copyright (c) 2001-2009 Expat maintainers.
These files may be reproduced, distributed, modified, and otherwise dealt in
under the terms of the Expat License.
On this system, a copy of the Expat License may be found at
<file:///usr/share/common-licenses/Expat>.
Distribution Packaging
======================
Copyright (C) 2012 J. Random Hacker
These files may be reproduced, distributed, modified, and otherwise dealt in
under the terms of the Expat License.
On this system, a copy of the Expat License may be found at
<file:///usr/share/common-licenses/Expat>.
Building Everything
Now we can build all of our source and binary packages and verify that everything is correct.
opkbuild maintains a cache file in the work area; because we've modified the metadata in our packaging since the first time we ran opkbuild, this cache file is out-of-date. Also, we should make sure that the entire build process still works. So let's clean up the work area before going any further.
$ sudo rm -Rf tmp/
Now let's run opkbuild again, this time completely building all of our source and binary packages and cleaning up automatically when we're done.
$ sudo prokit build $root .
After that finishes, you should see the built packages in the parent directory.
$ ls -1 ../*.opk
../libexpat.1_2.1.0-1_amd64-linux-glibc_all.opk
../libexpat.1-dev_2.1.0-1_amd64-linux-glibc_all.opk
../src-expat_2.1.0-1_src_all.opk
../xmlwf_2.1.0-1_amd64-linux-glibc_all.opk
../xmlwf-doc_2.1.0-1_all_all.opk
src-expat is a source binary package – a binary package installable with the
package manager that provides the files in our source package. This binary
package is a convenient way to distribute our source package to others.
You can use the tar(1) command to verify that the control information and data files in packages look correct.
$ tar -xzO control.tar.gz \
> <../libexpat.1_2.1.0-1_amd64-linux-glibc_all.opk | tar -xzO ./control
Package: libexpat.1
Source: expat
Version: 2.1.0-1
Architecture: amd64-linux-glibc
Platform: all
Maintainer: "J. Random Hacker" <jrandom@example.com>
Installed-Size: 164
Description: XML parser library
Expat is an XML parser library written in C. It is a stream-oriented parser in
which an application registers handlers for things the parser might find in the
XML document (like start tags).
Homepage: http://expat.sourceforge.net/
$ tar -xzO data.tar.gz \
> <../libexpat.1_2.1.0-1_amd64-linux-glibc_all.opk | tar -tz
./
./usr/
./usr/share/
./usr/share/doc/
./usr/share/doc/libexpat.1/
./usr/share/doc/libexpat.1/changelog.dist
./usr/share/doc/libexpat.1/copyright
./usr/lib/
./usr/lib/amd64-linux-glibc/
./usr/lib/amd64-linux-glibc/libexpat.so.1
./usr/lib/amd64-linux-glibc/libexpat.so.1.6.0
Congratulations! You've made a source package that successfully builds four binary packages!