Packaging a single crate

The Rust team uses a somewhat unusual workflow to package single crates.

Instead of one packaging repository per crate or upstream repository, it goes with a shallow monorepo approach. The repo is called debcargo-conf, sometimes abbreviated as dc-c or dcc.

It's called "shallow" because actual upstream sources and a full debian/ directory are not stored, only an "overlay". The overlay consists of debian/debcargo.toml, a declarative configuration for the debcargo tool; debian/copyright, the mandatory, most of time the only file needing manual editing; and other debian/ files as needed.

From there, a Debian packaging is generated out-of-tree with debcargo, fixed and patched, built, and uploaded.

Some packaging techniques found during the years are recorded in Hacks.

Setting up

It's recommended to work in Debian unstable, unless you have specific backport needs.

First, install tools and clone the repo:

$ sudo apt update && sudo apt install cargo rustc dh-cargo debcargo git devscripts quilt
$ git clone git@salsa.debian.org:rust-team/debcargo-conf.git
$ cd debcargo-conf/

Second, set up a build environment. The "official" build tool in the Rust team is sbuild, as is used by Debian buildds. Adapt accordingly if using a different tool.

Using sbuild with unshare:

$ sudo apt install mmdebstrap sbuild uidmap autopkgtest
$ mkdir -p ~/.cache/sbuild # default cache location of sbuild
$ mmdebstrap --variant=buildd --include dh-cargo,cargo,lintian unstable ~/.cache/sbuild/debcargo-unstable-amd64.tar.xz

You may want to set up apt-cacher-ng to reduce bandwidth consumption and speed up package downloads.

Note that you need to set CHROOT_MODE=unshare to use ./build.sh with unshare.

Using sbuild with schroot:

$ sudo apt install reprepro debootstrap sbuild schroot autopkgtest
$ sudo sbuild-createchroot --include=dh-cargo,cargo,lintian \
    --chroot-prefix debcargo-unstable unstable \
    /srv/chroot/debcargo-unstable-amd64-sbuild http://deb.debian.org/debian
# replace deb.debian.org with a mirror of your preference if applicable

These are basic setups of sbuild. For more detailed usage and customization, consult the sbuild page on wiki.

Packaging scripts

debcargo only prepares a Debian packaging. Some other scripts in the debcargo-conf repo automates the updating, building, and releasing (uploading) of it, conveniently named update.sh, build.sh, release.sh, respectively. Sections below introduce their basic usage, but more usage instructions are in their headers - just read them! And improve them if you have an idea ;)

Steps to work on a crate

  1. Prepare a new version/revision of a crate
  2. Fix patches if failing to apply, then go to 1
  3. Build
  4. Fix build and test failures if present, then go to 2
  5. Upload if you have upload rights; if not, send in the work and Request for Sponsorship

Prepare a new version/revision of crate

$ ./update.sh <crate-name>

and follow its instructions.

You should usually also write collapse_features = true in src/<crate>/debian/debcargo.toml. See the reason.

Co-installable older versions ("versioned" or "semver-suffixed" packages)

Even if we strive to maintain only one version of a crate, there are times we have to split out an older version of it.

Only do it when absolutely necessary, e.g. when an important crate used by many dependents introduces a breaking, incompatible major version, yet dependents differ in the pace of migration.

If possible, please consider these before going split:

  • Downgrading or upgrading the dependency
  • Ask upstream to upgrade if there is not already similar asks
  • If it's an optional dependency, consider temporarily disabling it (and if an optional feature depends on it, also such feature)
  • Wait until upstream upgraded

To split out, or update the split package, run:

$ ./update.sh <crate-name> <old-version>

and follow its instructions.

To save time, when first splitting out, you can first copy anything relevant from src/<crate-name> to src/<crate-name>-<old-version>, then run the command above and adapt as needed.

Hold packages at versions older than the newest

If you need to keep the unversioned package in Debian at an older version than is released on crates.io, e.g. to upload an important bugfix without being blocked on dependencies of the newest version, you can prefix the scripts with REALVER=<wanted-version>:

$ REALVER=<wanted-version> ./update.sh  <crate-name>
$ REALVER=<wanted-version> ./build.sh   <crate-name>
$ REALVER=<wanted-version> ./release.sh <crate-name>

Repackage/recreate an existing revision

To build the version of a package as packaged in debcargo-conf:

$ ./repackage.sh <crate-name>

If this package is already in the archive and you want to recreate that exactly, you will need to use the exact same version of debcargo that was used previously. Its version is mentioned in debian/changelog, like this:

* Package foo a.b.c from crates.io using debcargo x.y.z

Fix patches

Due to the way this workflow organizes files, we can't use other options like gbp-pq. See "Patching crates" in Hacks.

Build

Run:

$ ./build.sh <crate-name>
$ ./build.sh <crate-name> <version>                # for a versioned package
$ REALVER=<wanted-version> ./build.sh <crate-name> # for older version of unversioned package

If you need to pass additional options to sbuild, like --arch=i386, set them in the SBUILD_OPTS environment variable:

$ SBUILD_OPTS="--arch=i386" ./build.sh <crate-name>

Normally, ./build.sh will fail early if not all the build dependencies are available in your local apt cache. If you are packaging a large dependency tree however, to avoid many round-trips through NEW it is possible to bypass this check and build all the packages together. Suppose package B depends on package A, then you can run something like:

$ ( cd build && ./build.sh A )
$ ( cd build && ./build.sh B librust-A*.deb )

The librust-A*.deb arguments after ./build.sh B are extra deb files to pass to sbuild to use as dependencies. Alternatively, use the environment variable EXTRA_DEBS, like so:

$ EXTRA_DEBS=librust-A*.deb ./build.sh B
$ EXTRA_DEBS=librust-A.deb,librust-B.deb ./build.sh C

There is also a dev/chain_build.py script that lets you build multiple packages in a "chain". Run it without arguments to see its help message.

Upload

Note: this is usually only needed for uploading the package to the Debian archive. If you don't have upload rights, and weren't asked by someone who has to send a built package to them, this is likely not for you. See the "Request for Sponsorship" section below.

To build for upload, run

$ ./release.sh <crate-name>                     # or
$ ./release.sh <crate-name> <old-version>       # as appropriate
DISTRO=experimental ./release.sh <crate-name>   # to target another distro
EXTRA_DEBS=foo.deb ./release.sh <crate-name>    # use local dependencies

A source-only build is generated (e.g. source.changes, .dsc only). The script shows some information about its upload, but does not do it; you need to do the actual upload yourself, e.g. dput build/rust-foo_1.2.3-1_source.changes.

It also creates a Git branch named pending-<crate> to signal the upload. See conventions.

Send in the work

The Rust team usually uses the merge request (MR) workflow. There isn't too much about it - commit your changes and send them in with a MR to the debcargo-conf repo.

One thing though - we have some conventions on branch names and commit messages.

Request for Sponsorship (RFS)

If you are new to the Rust team, or to Debian packaging in general, after packaging, you should ask for its sponsorship with a Request for Sponsorship (RFS).

The linked page describes the RFS process for packages on mentors.debian.net. It's not one process rules all, though.

For single crate process, create src/<crate>/debian/RFS, in the debcargo-conf repo. It should include relevant information like what is needed to build it, what it's for, etc. After the packaging is merged, ask someone in the team with upload rights to sponsor it.

For workspace process, it's more like the "usual" way in Debian: you can either upload to mentors.d.n and file a RFS bug, or ask a DD you've worked with to directly upload your package.

Expert mode & packaging multiple packages

You should get used to the single-packaging workflow a bit first, including doing a few test builds of your package. Otherwise the instructions below may seem a bit opaque.

  1. Depending on the situation, optionally clean the build/ directory.
  2. ./update.sh <CRATE> all your relevant packages.
  3. Do any manual updates.
  4. (cd build && ./build.sh <CRATE> *.deb) for all your relevant packages, in dependency order.
  5. Deal with any issues that come up.
  6. Push your updates to debcargo-conf.
  7. Run dev/list-rdeps.sh <CRATE> [<CRATE> ...] on all the crates you updated. If a breaking version is introduced, any reverse dependencies affected also need to be updated, and you should repeat steps 1-7 (including this step) for them as well until this step lists no new packages that are affected.
  8. ./release.sh <CRATE> all the packages you updated, running the build again if necessary. It may be possible to do this out of dependency order, assuming you didn't have to make significant changes in step (5). If you did, then this step also has to be done in dependency order.
  9. Upload them.
  10. Push your pending-* branches to debcargo-conf.

There are also various scripts in dev/* that might help you. They should have a few lines at the top of the source code describing their functionality and some brief usage instructions.

Whew, thanks for all your work!

Collapsing features

TL;DR: Set collapse_features = true in debcargo.toml, unless something breaks. This eliminates empty feature packages and speeds up the NEW trip.

Rust and Debian have two different levels of abstraction when handling dependencies and the relationship between them. In Rust the lowest level is a feature, while in Debian it's the binary package.

This means that the following dependency chain is not a problem in rust:

  • crate A with feature AX depends on crate B with feature BY
  • crate B with feature BX depends on crate A with feature AY

This is a perfectly valid situation in the Rust ecosystem. Notice that there is no dependency cycle on the per-feature level, as this is enforced by cargo; but if collapse_features = true then package A+AX+AY would cyclicly depend on package B+BX+BY in Debian.

This is reflected in the Debian packages by producing Provides lines for all combinations of features, and this can become a quite large section.

Setting collapse_features = true in debcargo.toml removes this behaviour and is generally recommended, unless when it leads to dependency cycles of Debian packages. If that happens, those must be broken up by having some or all of the packages set this feature to false.

Changed original tarballs

Sometimes the orig.tar generated by debcargo might change e.g. if you are using a newer version of debcargo and one of the dependencies relating to generating the tarball was updated and its behaviour changed - compression settings, tarball archive ordering, etc. This will cause your upload to get REJECTED by the Debian FTP archive for having a different orig.tar. In this case, set REUSE_EXISTING_ORIG_TARBALL=1 when running ./release.sh.