Packaging a crate for Debian

NOTE: This is a work in progress to move the README.rst file in the debcargo-conf repository here, and tidy it up.

Our current packaging process is centered around debcargo and the debcargo-conf registry.

It's recommended to work in Debian unstable, unless you have specific backport needs.

To get set up:

apt update && apt install debcargo git
git clone git@salsa.debian.org:rust-team/debcargo-conf.git
cd debcargo-conf/

Prepare new (version of) crate

./update.sh <rust-crate-name>

and follow its instructions.

Package a co-installable older version

To maintain an old version of a crate alongside the latest one, first make sure the latest version is packaged by doing all of the above, then run:

./update.sh <rust-crate-name> <old-version>

and follow its instructions. To save time, you can first copy anything relevant from src/<rust-crate-name> to src/<rust-crate-name>-<old-version>, then adapt it as needed.

You will need excellent reasons to do this. It should be done only for core crates used by many other crates or programs for which the upgrade path is complex. For example, if the API significantly changed and requires a lot of work.

Instead, please consider:

  • Downgrading or upgrading the dependency
  • If it doesn't exist, open an issue on the upstream issue tracker to encourage them to upgrade
  • If possible/relevant, disable a feature if it uses it
  • Wait until upstream upgraded

Prepare a release

./release.sh <rust-crate-name>                     # or
./release.sh <rust-crate-name> <old-version>       # as appropriate
DISTRO=experimental ./release.sh <rust-crate-name> # to target another distro
EXTRA_DEBS=build/*.deb ./release.sh <rust-crate-name> # use local dependencies

This prepares the necessary Debian files in build/, and creates a git branch to manage the packaging until it is accepted in Debian itself. You need to run additional commands after this - more specific instructions are given to you about this by the script after you run it.

Hold packages at old versions

If you need to keep the latest version in Debian at an older version than is released on crates.io, e.g. to upload an important bugfix without being blocked on having to package all the dependencies of the newest version, you can:

REALVER=<old-version> ./update.sh  <rust-crate-name>  # then
REALVER=<old-version> ./release.sh <rust-crate-name>

Repackage the existing revision

To build the version of a package already in debcargo-conf/src:

$ ./repackage.sh A
$ (cd build && ./build.sh A)

If this package is already in the archive and you want to recreate that exactly, you will need to use the exact same version of debcargo that was used previously. This version is mentioned in debian/changelog.

Set up build environment

To set up a suitable build environment for ./build.sh:

With schroot:

$ sudo apt-get install devscripts reprepro debootstrap sbuild dh-cargo schroot autopkgtest quilt
$ sudo sbuild-createchroot --include=eatmydata,ccache,gnupg,dh-cargo,cargo,lintian,perl-openssl-defaults \
    --chroot-prefix debcargo-unstable unstable \
    /srv/chroot/debcargo-unstable-amd64-sbuild http://deb.debian.org/debian

With sbuild + unshare:

$ sudo apt install devscripts mmdebstrap sbuild uidmap dh-cargo autopkgtest quilt
$ mkdir -p ~/.cache/sbuild
$ mmdebstrap --variant=buildd --include ccache,gnupg,dh-cargo,cargo,lintian,perl-openssl-defaults unstable ~/.cache/sbuild/debcargo-unstable-amd64.tar.xz

Note that you need to set CHROOT_MODE=unshare to use ./build.sh with unshare.

An explanation of this, plus more recipes, can be found on the wiki page of sbuild.

If you need to pass additional options to sbuild, like --arch=i386, then set the SBUILD_OPTS environment variable.

Normally, ./build.sh will fail early if not all the build dependencies are available in your local apt cache. If you are packaging a large dependency tree however, to avoid many round-trips through NEW it is possible to bypass this check and build all the packages together. Suppose package B depends on package A, then you can run something like:

$ ./release.sh A
$ ( cd build && ./build.sh A )
# push pending and checkout master
$ ./release.sh B
$ ( cd build && ./build.sh B librust-A*.deb )

Or you can use the EXTRA_DEBS environment variable, which build.sh will inherit, like:

$ EXTRA_DEBS=build/*.deb ./release.sh A

The extra arguments after ./build.sh B <args> are extra deb files to pass to sbuild to use as dependencies. In this case, librust-A*.deb should have been built by the previous step. Alternatively, use the environment variable EXTRA_DEBS, like so:

$ EXTRA_DEBS=librust-A*.deb ./build.sh B
$ EXTRA_DEBS=librust-A.deb,librust-B.deb ./build.sh C

After everything is built successfully, you can dput all of them and then push all the pending-* branches as normal.

Repository structure

pending-* branches are managed by ./release.sh, so please don't manage them yourself as you will interfere with the working of that script. The intention is that they should only differ from the master branch by 1 commit, i.e. the dch -r commit created by ./release.sh.

If you want to create separate non-master branches, that is fine - just don't call them pending-* and don't run ./release.sh on those branches. If you want to test your crate, instead run:

cd build && [SOURCEONLY=1] ./build.sh <rust-crate-name> [<old-version>]

omitting the stuff in [...] as needed.

Like many other Debian git repositories, we don't follow "feature branch" practices here. We generally don't package just 1 or 2 rust crates at a time, but all of its dependencies and sometimes some reverse-dependencies too. So normally we'll be touching a few dozen packages at once. In this context, it's good to merge often, to avoid conflicts with someone else that might also need to touch those too in the next few days.

To match a release (i.e. a .deb or a .dsc file) to a commit, find the commit message that actually says "package: release x.y.z-d". It's commonly followed by a merge commit.

Expert mode & packaging multiple packages

You should get used to the single-packaging workflow a bit first, including doing a few test builds of your package. Otherwise the instructions below may seem a bit opaque.

  1. rm -rf build/* && sbuild-update -udr debcargo-unstable-amd64-sbuild - clears out your build directory, making the subsequent steps a bit faster.
  2. ./update.sh <CRATE> for all your relevant packages.
  3. Do any manual updates.
  4. cd build then ./build.sh <CRATE> *.deb for all your relevant packages, in dependency order.
  5. Deal with any issues that come up.
  6. Push your updates to our git.
  7. Run dev/list-rdeps <CRATE> [<CRATE> ...] on all the crates you updated. Any reverse-dependencies that are affected also need to be updated, and you should repeat steps 1-7 (including this step) for them as well until this step lists no new packages that are affected.
  8. ./release.sh <CRATE> for all the packages you updated, running the build again if necessary. It may be possible to do this out of dependency order, assuming you didn't have to make significant changes in step (5). If you did, then this step also has to be done in dependency order.
  9. Push your pending-* branches to our git.

There are also various scripts in dev/* that might help you. They should have a couple lines at the top of the source code describing their functionality and some brief usage instructions.

Whew, thanks for all your work!

General packaging tips

Benchmarks

Dependencies on clippy

Patch away dependencies on "clippy" unless it is a "real" dependency. Usually crates only use clippy to lint themselves and it is not a "real" dependency in the sense that they actually import clippy's code for what they do.

If you want to be sure, rg clippy and check that all the usages of it are inside cfg_attr declarations. If so, then just get rid of it.

OS-specific crates

See redox-syscall for examples on how to deal with these.

If this is unclear, ask on IRC.

Architecture-specific crates

This is a bit harder. Usually there are two options:

  1. The crate should build a dummy/no-op version of itself "out-of-the-box" on the architectures it doesn't work on.
  2. Dependent crates should depend on it with a platform-specific dependency.

(1) involves less burden for others, both for dependent crates and for us packagers, since we don't have to override d/rules to ignore test failures on non-working architectures. You should communicate to upstream that this is the preferred approach.

In the case of (2), the crate should document exactly what conditional should be used, and keep this documentation up-to-date. This allows us to easily determine if dependent crates are using the correct conditional. You will then have to override d/rules for this crate, see src/simd for an example.

You should file a bug upstream if the crate does neither (1) nor document the conditions for (2), e.g. https://github.com/hsivonen/simd/issues/25

(Actually the above applies even for "OS-specific crates" but then (2) is obvious so documentation is less necessary, and dependent crates all do it correctly already.)

Collapsing features in debcargo.toml

TL;DR: Set collapse_features = true in debcargo.toml, unless something breaks. This eliminates empty feature packages and speeds up the NEW trip.

Rust and Debian have two different levels of abstraction when handling dependencies and the relationship between them. In Rust the lowest level is a feature, while in Debian it's the binary package.

This means that the following dependency chain is not a problem in rust:

  • crate A with feature AX depends on crate B with feature BY
  • crate B with feature BX depends on crate A with feature AY

This is a perfectly valid situation in the Rust ecosystem. Notice that there is no dependency cycle on the per-feature level, as this is enforced by cargo; but if collapse_features = true then package A+AX+AY would cyclicly depend on package B+BX+BY in Debian.

This is reflected in the Debian packages by producing Provides lines for all combinations of features, and this can become a quite large section.

Setting collapse_features = true in debcargo.toml removes this behaviour and is generally recommended, unless when it leads to dependency cycles of Debian packages. If that happens, those must be broken up by having some or all of the packages set this feature to false.

Changed orig tarballs

Sometimes the orig.tar generated by debcargo might change e.g. if you are using a newer version of debcargo and one of the dependencies relating to generating the tarball was updated and its behaviour changed - compression settings, tarball archive ordering, etc. This will cause your upload to get REJECTED by the Debian FTP archive for having a different orig.tar. In this case, set REUSE_EXISTING_ORIG_TARBALL=1 when running ./release.sh.

ITPs

Generally, don't file ITPs for library crates, but do file them for binary crates.

We'll usually be uploading a dozen crates or so at once. Submitting ITPs for these is unnecessary since we're the only ones uploading and there is no chance of conflict. It would only be spam for the bug tracker. Please instead coordinate uploads on the #debian-rust IRC channel.

Testing

Debian has two types of tests:

  1. pre-install tests run in debian/rules
  2. post-install tests defined in debian/tests/control, run by/with autopkgtest

For Debian Rust packages, in (1) we run the crate's test suite with default features but only if there are no dev-dependencies, and in (2) we run the whole test suite with each feature enabled separately plus --no-default-features and --all-features.

Sometimes, tests require extra tweaks and settings to work. In this case, you can tweak debian/rules for (1), and for (2) you will simply have to mark the relevant tests as broken using test_is_broken = true. See the existing crate configs for examples.

Other times, the tests are simply broken or can't be run in Debian. In this case you should disable the test in (1) by running dh_auto_test -- build instead of the default dh_auto_test -- test --all, and for (2) again you should mark the relevant tests as broken. These tests are going to be marked as flaky in autopkgtest, still executed but won't fail the autopkgtest run.

Please note that

[packages.lib]
test_is_broken = true

will transitively disable tests for all combinations of features. Sometimes this is correct e.g. if the test actually breaks for all features. Sometimes this is not correct, e.g. if the test only breaks for --no-default-features. In the latter case you should instead patch the crate to ignore those tests when the relevant features are absent - e.g. src/regex-automata/debian/patches/ignore-std-tests.patch.

Binary crate ([[bin]]) has required-features

TODO: see if it actually requires manual handling.

Binary (crate) name conflicts with other package

There are two ways to handle this:

  1. Patch the name of the [[bin]] in Cargo.toml to something else, commonly "thing-rs". This changes both the binary name and the binary package name.
  2. Install it to another location with the same name, and/or install it as another name, then let the user choose to alias/link it. The fd-find package does both: /usr/lib/cargo/bin/fd and /usr/bin/fdfind.

Update dependencies

In some cases, libraries/programs are forcing an old version of a library as dependencies. In order to limit the number of duplicated libraries in the archive, please try to evaluate if a newer version of the dependencies could be used.

To achieve that, after ./update.sh, try:

$ cd build/<package>/
$ rm -rf .pc # sometimes this is necessary due to minor debcargo bug
$ quilt push -a
$ quilt new relax-dep.diff
$ quilt edit Cargo.toml
$ quilt header -e --dep3
$ quilt refresh
$ cargo build # check that it works. if it does, then
$ cp -R patches ../../src/<package>/debian

Suppose you want to change the dependency from 0.3 to 0.5. If the crate builds with no further source changes, then we would change the required version in Cargo.toml from 0.3 to >= 0.3, < 0.6 or something like that. Then the convention is to put all these changes into a single patch called relax-dep-versions.patch.

OTOH, if the cargo build fails, and you can fix it up by editing the source code in a minor way to use the new crate API, then: for each crate that needs to be updated, you should instead name the patch update-dep-<crate>.patch and add both the Cargo.toml and the source code changes to it. The change to Cargo.toml would then simply say (e.g.) 0.5 since the older versions actually don't work, and not the version range from the previous paragraph.

If you want to make a crate work with an older dependency version than listed in Cargo.toml (for example 0.3 instead of 0.5), you cannot use a flexible version requirement like >= 0.3, < 0.6. Instead you have to specify only the older version, in this example 0.3.

An explanation was given by Ximin Luo (infinity0) in this MR:

It's a quirky annoyance of sbuild/apt (I can't remember which) that for build-depends only the first in a series of alternatives is considered, and yes in these sorts of situations we are in fact forced to downgrade the dependency version.

The authors refused to fix it the last time I spoke to them about this, giving the incorrect reasoning "it would mess with build reproducibility". The reasoning is incorrect because by the same argument one should not allow different versions of the same compiler, and when doing reproducibility (which I worked, paid, on several years ago) one is supposed to record the actually-installed versions of the build-depends and use these exact same ones when attempting a 2nd reproduction, otherwise one cannot expect reproducibility anyways.

Patch headers

DEP 3 formalized a set of patch headers. Use them accordingly to explain why a patch exists. quilt header -e --dep3 opens $EDITOR with a template.

Help, something went wrong!

Sometimes, the error messages are not the most informative. In this case you can try re-running the command with RUST_BACKTRACE=1. If you are using the debcargo from Debian's own repositories, you should also install the debcargo-dbgsym package, otherwise the stack trace will be next to useless. Make sure you have the debug repository enabled in your APT sources.

Some ramblings

In #debian-rust came these two blog posts along with the remark of good read

Now are they, those two blog posts, parked here. Waiting for better integration.

Developing Rust code using Debian-packaged crates

While perhaps not the stated intention, the Rust ecosystem in Debian is actually quite usable for developing Rust code in general. Thanks to source replacement, Cargo can be configured to use only local, Debian-provided packages by placing something like the following in ~/.cargo/config.toml (for user-wide effect) or in a given project's .cargo/config.toml:

[net]
offline = true

[source]

[source.crates-io]
replace-with = "debian"

[source.debian]
directory = "/usr/share/cargo/registry"

In this state, Cargo will only look for crates installed as Debian packages on the local system.