Use only locally available GPG keys

[calestyo]

Hey.

Not sure whether I've understood the code correctly, but it seems that mkosi quite easily falls back to disabling signature checks for at lest the DNF/YUM case:

mkosi/mkosi/__init__.py

Lines 1825 to 1826 in 28e9da0

	if not gpgcheck:
	cmdline.append("--nogpgcheck")

which seems to be set here https://github.com/systemd/mkosi/blob/main/mkosi/__init__.py#L1850-L1856

My understanding is, that it first looks locally for the key, and if not there falls back to simply downloading it if there is a URL specified (in mkosi) itself, and if not it simply doesn't verify signatures at all.
It also doesn't seem to be obviously documented whether that key must be in the host OS or inside the container image already.

For security reasons, it should per default only use/trust locally installed keys. Additionally perhaps even ignore any gpgcheck=0 or gpgkey= that use non-local key locations if possible.

Simply downloading a key via HTTPS doesn't seem really much more secure either. The usual CA bundles (e.g. Mozilla) come with some ~150 root CAs... (which themselves have thousands of intermediate CAs) many of them under the control of quite questionable organisations.

The same principles would obviously apply to debootstrap, etc. as well.. but at least for debootstrap I couldn't find that similar things are done (though I didn't look were closely).

So would it be possible to default to only using local keys and doing anything further like downloading/trusting keys or simply not checking sigs only with some --i-really-know-what-i-am-doing-promised option?

Another way would be that mkosi ships those keys, which obviously has some pros and cons.
Pros, that people using mkosi would at least get all the same keys (and not having to download them - more or less securely - on their own).

Cons, that you guys would have to maintain this... and also that e.g. in the case of bootstrapping Debian/Ubuntu on a Debian/Ubuntu system their own keyring packages should be rather used than what mkosi ships.

Cheers,
Chris.

[DaanDeMeyer]

For dnf, we do explicitly warn when we're not using gpg checking. We can't really rely only on local keys by default since local gpg keys can only be relied on to be available if we're building the same distro as the host distro. I'm open to dropping the fallback that disables gpg checking though. Or we add it behind an option so that it's opt-in and not used by default

[calestyo]

A warning is in practise probably useless,... in automatic deployments no one notices them anyway,.. and even if, possible rogue packages would have already been installed.

As for keys, as said, the best would be if mkosi would ship them, or if there was some bundle that collects the keys of all major distros and that mkosi depends upon.
Just like right now it would need something like ca-certificates (which is however far far less secure).

The best would IMO be an option that enables fallbacks,... and per default it should only use locally available keys.

[DaanDeMeyer]

We got rid of the fallback to no GPG checks if the url is unavailable, but we're going to keep using the urls. Happy to switch to using local keys as soon as there's a package in all distributions that ships the keys of all the rpm based distros.

[calestyo]

Hey @DaanDeMeyer.

Do you happen to know the commit where this was changed?

Also, what do you mean by "keep using the URLs"? That https will be enough to trust it?

If so, may I then kindly ask to keep this issue open until some solution is found that gives really strong security per default?

As outlaid above, https/TLS has the practical problem that people typically trust the browser bundle of CAs... which in turn poses a rather big attack surface.

Even if one would assume that all CAs (and all of the even more intermediate CAs) were "good" - which is at best naive - the actual verification done by most CAs is quite weak... it's some challenge response either like ACME does via http or some even just mail to addresses given in the whois.

So for an attacker it would be enough to be in the middle of some CA and e.g. the A/AAAA RR pointed to by DNS (which itself is quite often not secured).

All that seems much weaker to me, than the security provided afterwards by some OpenPGP keys/signatures,... and since an attacker would typically focus on the weakest element of the chain, that quite jeopardises the strong security provided by the repos themselves (via OpenPGP).

So please either re-open, or advise me to make another feature request for that.

[calestyo]

Oh and btw:... I think I recently read on phoronix or so, that Fedora plans on replacing dracut via something mkosi based?

If so, it would IMO warrant even more for strong (that is: already trusted OpenPGP keys) security requirements unless some --i-really-know-that-i-compromise-security option is given,... cause way more users would be affected if it could somehow accidentally happen, that untrusted packages are loaded.

And as for:

a package in all distributions that ships the keys of all the rpm based distros

I think that would definitely make sense... even more so, if such registry could contain keys for all well known distros (and not just rpm based). Not sure though, where such registry should be placed ... I mean it would need to be some organisation which a) can get the keys in some trusted manner and b) is itself credibly enough so that others would use them as source.

[DaanDeMeyer]

For Debian/Ubuntu and Arch, packages already exist and are widely packaged and we depend on those. For rpm based distros, there's https://github.com/xsuchy/distribution-gpg-keys but it's not packaged on Debian/Ubuntu or Arch so we cannot depend on it.

The change to make urls non-optional was in f360165.

I am happy to switch to using disk based locations only, but I need distribution-gpg-keys to be packaged across all distros to make that a viable option.

Will reopen until someone can make that happen.

[calestyo]

There's already an ITP in Debian (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1026087), though no progress for a while.

But anyway... I personally would think, that mkosi should simply not do anything (unless a --i-really-know-that-i-compromise-security is given) that downloads/installs/etc. packages that are not secured by the distros own crypto keys (i.e. not just TLS or even worse nothing).

Having a package like distribution-gpg-keys around everywhere, would of course make that easier... but even then there's still no need to strictly depend on it (e.g. mkosi package in Debian just recommends such key packages, even Debian's own). It would simply just need to fail if the respective key was missing...

A user, whose distro doesn't provide it, could still manually fetch it from github (and be still comparably secure[0] than with using https).

[0] A number of factors are IMO relevant here:

• Normal TLS (when trusting the typical ~150 root CAs) is IMO rather insecure (or better said non-trustworthy). It's not the TLS itself, but simply the CAs.
• If one lets users download distribution-gpg-keys they'd at first have the same security level than that of https... and, depending on how far one trusts that project, possibly less.

[calestyo]

Oh and btw:
Another point of this issue is IMO, that mkosi should ingore things like YUM’s gpgcheck=0 (not sure whether this still exists in DNF?) and that like.

[DaanDeMeyer]

Since #1949 we use local GPG keys for Fedora, CentOS, Alma and Rocky if the distribution-gpg-keys package is installed.

[DaanDeMeyer]

Since we now use local keys when available for most distributions, let's close this as completed. Please open separate RFEs and PRs for the other distributions where you'd also like to see this happen.

Note that we're unlikely to drop support for downloading the keys as a fallback. Yes CA's might be unsafe, but if you care about that then it's on you to make sure the keys are available locally at the correct location so we can pick them up from there.

[calestyo]

Hey @DaanDeMeyer

Since we now use local keys when available for most distributions,

Well... it's the same as I've said before:

What should be done, and what this request was about is: that it never uses packages/code, for which keys (and not just some TLS certs) haven't been installed (via some trusted way, not by downloading it).

If you now say "when available" and "for most distributions", it sounds like something else might be done when not available… and/or not for all distributions.

So that would still make it some gambling, whether or not untrusted code is used.

Note that we're unlikely to drop support for downloading the keys as a fallback. Yes CA's might be unsafe, but if you care about that then it's on you to make sure the keys are available locally at the correct location so we can pick them up from there.

Fine, but why not make the default safe and and add an option --fallback-to-untrusted-something for people how know what they're doing (or rather: simply don't care). Would also be easy to use in any automated system, so I don't think that use case is an excuse for not using a safe default.

Sure I can file a new bug, but it would be basically the same request... not sure if that makes sense.

[DaanDeMeyer]

@calestyo You might want to chime in on openSUSE/zypper#528 as well with regards to this issue.

[calestyo]

@DaanDeMeyer Well I don't really use zipper, so woudn't it be more appropriate if you just link to my points here?

Also, it seems convincing people to make things secure by design feels often like fighting against windmills... and that it needs something extreme like XZ to change minds (for a few weeks until everything is forgotten again).

I filed this whole issue purely for the purpose, that there should be no means (unless some --let-me-do-stupid-insecure-things is given) to get untrusted or only-via-https (which is more or less nothing) trusted stuff downloaded (neither packages nor keys).

In the end some bit of hardening was done, but still it's not really fully secure, at least as far as I understood the descriptions given above (which I understood as "local keys are used, but only if available").

And I was too lazy to open a copy of the same issue. ^^

So for that reason, we banned using mkosi at the institute.

[DaanDeMeyer]

I've reopened this and assigned the blocked tag as this is blocked on https://github.com/xsuchy/distribution-gpg-keys being packaged everywhere so there's an easy way to install GPG keys from a package. Currently it's not packaged in Debian, Ubuntu and Arch

[calestyo]

Wouldn't it be the simplest solution to add options like --download-repo-keys-via-https and --no-check-repo-keys, with both defaulting to no (i.e. every download/bootstrap/package management tool would work only with already present local keys)?

If a distribution hasn't packaged distribution-gpg-keys and/or the user hasn't retrieved the keys on his own, things will fail once an can then be easily resolved by the user adding the appropriate options to his workflow (or getting the proper keys).

It's not like not having the (local) keys would lead to catastrophic or silent corruption (and failing then per default). And most legacy installations may have anyway just downloaded them meanwhile via https?

There's IMO no need to block this on some 3rd party collection of repo keys.

[calestyo]

@DaanDeMeyer, the ITP distribution-gpg-keys is now close to be completed, waiting in Debian’s NEW queue.

Thus it will sooner or later also land in *buntu.

[DaanDeMeyer]

@calestyo It's also in Arch Linux now thanks to the help of @jelly. Once it's in debian testing I'll see about doing a PR to remove the https URLs for GPG keys.

[DaanDeMeyer]

@calestyo Please take a look at #2931. That changes our default behavior to never fetch GPG keys remotely by default unless the RepositoryKeyFetch= setting is explicitly enabled by the user.

[calestyo]

@calestyo Please take a look at #2931. That changes our default behavior to never fetch GPG keys remotely by default unless the RepositoryKeyFetch= setting is explicitly enabled by the user.

Nice. :-)

So what's the overall situation now?

• Are unverified downloads or https-only (which security-wise is not really that much better) downloads anywhere used unless explicitly enabled?
• And as you've just said, GPG keys are never downloaded automatically, unless explicitly enabled.
• Last but not least I'd assume that GPG keys are always verified...
• ... and that keys are only used from some typical local location (like /etc/ or /usr) which only root` can write?

Cheers,
Chris.

[DaanDeMeyer]

@calestyo

With one exception which I'll discuss later, all internet access done by mkosi when building images is done via the target distribution's package manager. All the downloads done by the package manager are verified with the distribution's GPG keys unless explicitly disabled with RepositoryKeyCheck=no. All GPG keys are required to be available on the local system and are not downloaded on demand unless RepositoryKeyFetch=yes is explicitly enabled. All GPG key verification is left to the package manager and mkosi does not do anything to disable key verification on its end. Finally, by default all keys are picked up from /etc/ or /usr/. Extra keys can be made available using the PackageManagerTrees= setting but this has to be explicitly done by the user.

The one exception is when building opensuse images with dnf, we use curl to download some extra GPG keys from the repository as this is done by zypper as well when building opensuse images so to ensure things behave the same with dnf which doesn't implement this behavior we do that bit ourselves. This extra downloading is also disabled by default unless RepositoryKeyFetch=yes is explicitly enabled by the user.

Finally, for full transparency, we do enable RepositoryKeyFetch=yes by default when building rpm-based distributions from Ubuntu as Ubuntu does not yet ship distribution-gpg-keys in its main repositories and won't do so for quite some time. This is only enabled by default when building rpm-based distributions, otherwise the setting is disabled by default on Ubuntu as well.