How to get a reproducible build

[ben-grande]

Software version

I have built with docker on Debian using the docker repository

• Docker version 24.0.7, build afdd53b
• Debian 12
• 700e03d (HEAD commit)

Logs

Build log:

sha256sum mirage-firewall.tar.bz2 > mirage-firewall.sha256
SHA2 of build: e5aeaee4f6b20b539565892da06361348a9bdca9b2fe3b29f5d2047e85d17ffc ./dist/qubes-firewall.xen
SHA2 last known: 8ae5314edf5b863b788c4b873e27bc4b206a2ff7ef1051c4c62ae41584ed3e14
(hashes should match for released versions)

Manual check of build:

% sha256sum _build/mirage-firewall/vmlinuz

e5aeaee4f6b20b539565892da06361348a9bdca9b2fe3b29f5d2047e85d17ffc _build/mirage-firewall/vmlinuz

Manual check of release v0.8.5:

% sha256sum vmlinuz

8ae5314edf5b863b788c4b873e27bc4b206a2ff7ef1051c4c62ae41584ed3e14 vmlinuz

Questions

I have read that docker should be run on Fedora, but I used the Docker repository on Debian, maybe it caused the difference?

What can I do to get a reproducible build?

[palainp]

Hi @ben-grande, thanks for your detailed report.

The build host should not change the reproducibility result. However the Dockerfile assumes that http://snapshot.notset.fr/rchive/debian/ provides packets snapshots for debian (and is unfortunately offline so far, leading to apt update to use distributed debian packets). Therefore as the hashsum relies on the debian compilation toolchain stability and there was some update from time to time. I think this is the root cause for the hash difference.

I tested and got the exact same hashsum both with fresh debian-12 and fedora-38 AppVMs.

Now, to go further I changed Dockerfile snapshots from notset to debian, checked that every versions from debian packages are the same as in the last github action log, as well as every opam packages versions, and I still have the same hashsum as yours. I'll have to look deeper into packages versions and unfortunately I don't have anymore temporary compilation artifacts from a matching hash :(

[palainp]

Update: thanks to an old save, I saw that the duniverse cmdliner version is different (1.1.1 -> 1.2.0+dune). I'm unsure why this version is selected since 2 days and not since its release (april 2023). There is probably something to find behind the make depend call which is not "version fixed".

So far I think it's not a supply chain attack and as your shasum has been reproducted (at least on my computer, maybe also by @Szewcson?) I'll update the scripts to match the new shasum with updated cmdliner.

[dinosaure]

Update: thanks to an old save, I saw that the duniverse cmdliner version is different (1.1.1 -> 1.2.0+dune). I'm unsure why this version is selected since 2 days and not since its release (april 2023).

A delay exists between the upstream version in opam-repository and what is available into dune-universe/opam-overlays which is taken by opam monorepo to fetch dependencies. This specific package does not use dune. So we must maintain an extra OPAM repository which dunify such packages and this one was update via this pull-request: dune-universe/opam-overlays#210 (as you said, 2 days ago).

[palainp]

Thanks @dinosaure for your reply, definitely not a supply chain attack :) Do you know if it is possible to fix a opam-overlay hash (EDIT: with opam-monorepo) to avoid future issues?

I guess I'll have to cut out a new release with the correct hashsum (and while at it, updated opam-repository).

[hannesm]

That is indeed concerning. The underlying issue is that we attempt to pin some pieces, but we're not having a full build-info from docker and the build script (also, environment variables are not cleaned up -- there's quite some story why I worked on orb, maybe we should just use that, and publish the build-info together with a release?).

We pin the opam-repository to a specific commit. What we as well need to do is to pin the opam-overlays and mirage-overlays to a specific commit:

mirage configure --extra-repos=opam-overlays:https://github.com/dune-universe/opam-overlays.git#91a371754a2c9f4febbb6c7bb039649ad49a3c13,mirage-overlays:https://github.com/dune-universe/mirage-opam-overlays.git#05f1c1823d891ce4d8adab91f5db3ac51d86dc0b -t xen --allocation-policy=best-fit

Please note I have not tested the above, and am not sure whether the syntax for mirage and opam-monorepo is as I read it from the manual page.

[palainp]

That is indeed concerning. The underlying issue is that we attempt to pin some pieces, but we're not having a full build-info from docker and the build script (also, environment variables are not cleaned up -- there's quite some story why I worked on orb, maybe we should just use that, and publish the build-info together with a release?).

This (using orb) is probably the way to go for reproductible builds :)

We pin the opam-repository to a specific commit. What we as well need to do is to pin the opam-overlays and mirage-overlays to a specific commit:

Thanks I'll try that before merging the PR.

[palainp]

Thanks, fixed in #185.

[ben-grande]

Fix confirmed:

sha256sum mirage-firewall.tar.bz2 > mirage-firewall.sha256
SHA2 of build: 2c3f68f49afdeaeedd2c03f8ef6d30d6bb4d6306bda0a1ff40f95f440a90034c ./dist/qubes-firewall.xen
SHA2 last known: 2c3f68f49afdeaeedd2c03f8ef6d30d6bb4d6306bda0a1ff40f95f440a90034c
(hashes should match for released versions)