[Tails-dev] Reproducible Builds sprint #2 report

intrigeri intrigeri at boum.org
Fri Mar 17 08:37:56 CET 2017


here's a report of the second reproducible sprints that just ended.
Ulrike volunteered to handle broader communication about this topic,
so this report is only meant to share the news within our community.


After many iterations we finally made our ISO image build

The build environment variations we've tested include: build system
clock (last month, next month; could not test next year yet), number
of CPU cores, CPU brand and model, building in Vagrant or not.

This implied fixing a number of things:

 * APT auto-removal file (#11986): patch submitted and accepted
   upstream, backported in Tails
 * Switched to the new squashfs-tools upstream, that builds SquashFS
   in a reproducible manner (#12032).
 * Various non-determinism issues in the content of the files included
   in our SquashFS, including fixing incorrect metadata in old blog
   posts and their translations (#11966 – who would have guessed this
   affected build determinism? :)
 * Various non-determinism issues in the mtimes of the files included
   in our SquashFS, that made not only the SquashFS non-reproducible,
   but also made the initrd non-reproducible despite the patches we
   sent upstream for initramfs-tools (#12330).
 * Drop the "Posted on" timestamp ikiwiki added to some pages on
   our website (#11987).


 * Made diffoscope *way* faster when comparing SquashFS'es:
   changes made directly upstream
 * Improved performance of generating CA certificates databases on
   boot (#11971)

In progress

 * Review'n'merge the feature/5630-deterministic-builds branch into
   feature/stretch: one review happened, now blocked by a couple of
   the other WIP items and waiting for a second review, so it's
   unlikely these changes make it into 3.0~beta3, but I'm confident
   they'll be in 3.0~rc1 (mid-May)!

 * Ensure the reproducibly built ISOs pass our test suite (#11983):
   done for the subset of tests we run on Jenkins, left to be done for
   the other tests. Plus some new failures left to be investigated.

 * Build our IUKs reproducibly: branch ready for QA (#11974).

 * Avoid boot performance problems while generating the fontconfig
   cache: we've optimized this a bit with fancy systemd ordering,
   but since then one of us came up with a solution that's probably
   better (#11971).

 * Lots of progress was made to have static build environments:

   - Move the apt-cacher-ng data to a dedicated disk that can be shared
     among many Vagrant build VMs (#11979).
   - Create and provision a new Vagrant VM for every ISO build
   - Switch our Jenkins ISO build system to vagrant-libvirt (#11972).

   Next steps are to make the whole thing robust enough both for
   developers and for our Jenkins CI environment. We expect this will
   be merged and deployed either very soon, or between April 19 and
   May 12.

To be done

Not that much as far as we know! See remaining open tickets on


More information about the Tails-dev mailing list