[MCR] Verifying the Engine version is insufficient

[corhere]

After installing or upgrading MCR, Launchpad checks that the installed Docker Engine version matches the MCR version in the Launchpad manifest spec.

mcc/pkg/product/mke/api/host.go

Line 146 in 58709b3

version, err := h.ExecOutput(h.Configurer.DockerCommandf(`version -f "{{.Server.Version}}"`))

mcc/pkg/product/mke/phase/upgrade_mcr.go

Lines 185 to 187 in 58709b3

	if currentVersion != p.Config.Spec.MCR.Version {
	return fmt.Errorf("%s: container runtime version not %s after upgrade", h, p.Config.Spec.MCR.Version)
	}

That only asserts the expected version of the dockerd binary (distributed in the docker-ee package) is installed. It would not catch if the containerd or runc binaries (distributed in the containerd.io package), the docker CLI (docker-ee-cli package) or rootlesskit (docker-ee-rootless-extras) were the wrong versions. If launchpad is going to check, it should check everything.

The containerd, runc and rootlesskit versions cannot be inferred from the MCR version in the manifest spec. And the Engine version might not always exactly match the MCR version, either. We'd need some kind of manifest which maps MCR version to expected component versions.

[james-nesbitt]

Launchpad does not currently install MCR, it executes the MCR installer, which is what is responsible for installing the MCR packages of dockerd, containerd, runc.

Launchpad does a version check to see if MCR needs to be installed/upgraded itself, because the MCR installer has no features - not because it is an authority on the version.

The only options we can support here are:

1. check the docker version
2. never bother with a version check

That said, we do have per platform functionality (per OS) so if you can make some suggestions, we can look at them.

[corhere]

@james-nesbitt on Slack:

if launchpad is meant to verify runc --version, where is it supposed to get the expected value from? Product discussions already lead to an agreement that Launchpad should not be a database of version compatibility.

Ahh, I was unaware of that decision.

I think this issue is related to another source of friction that came up during this patch: the only way to verify that all nodes in your MKE cluster are patched is to SSH into each box and check the runc version individually. Unfortunately launchpad describe cannot be used as it reports only the Engine version installed on each node, which does not map 1:1 to MCR releases.

The common thread is that an MCR release is defined as an "officially supported" combination of software at specific versions, but we are lacking a way to directly query which MCR release is installed or if an unsupported combination is installed. Today users can only do that indirectly by checking the installed version of each component and comparing them to the component versions listed in MCR release notes. If (e.g.) we were to publish machine-readable manifests of MCR component versions for Launchpad to use when checking if an upgrade is needed, that would only solve this particular Launchpad issue, not any of the other related issues. Everyone else would still have to do manual checks.

I think we (the Engine team) need to come up with a general solution for querying the installed MCR version directly before we decide what to do with Launchpad's MCR version checks.

[james-nesbitt]

we are going to drop all engine version detection.

The down side of this will be that we will need to always run the MCR installer, and will never be able to tell if MCR was upgraded - meaning that we will need to always restart MCR, even if it wasn't upgraded.

[corhere]

meaning that we will need to always restart MCR, even if it wasn't upgraded.

Wait... why does Launchpad restart MCR? The Linux packages and install.ps1 handle stopping and restarting the services on upgrade.