[Bug] Latest 6.1 guest kernel config results in kernel panic when booting VM

[kanpov]

Describe the bug

The x86_64 6.1 guest kernel config as it is currently on master after the 9157a0c commit, runs seemingly okay with the latest 6.1 kernel (6.1.114), but produces an unusually larger vmlinux (38MB instead of 29MB), and when booting a VM with it the following happens:

[   12.489510] /dev/root: Can't open blockdev
[   12.489784] VFS: Cannot open root device "vda" or unknown-block(0,0): error -6
[   12.490205] Please append a correct "root=" boot option; here are the available partitions:
[   12.490717] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

To Reproduce

1. Download 6.1.114 Linux kernel source via tarball from kernel.org
2. Enter the extracted kernel directory
3. Copy the full contents (much larger after that commit) of https://github.com/firecracker-microvm/firecracker/blob/main/resources/guest_configs/microvm-kernel-ci-x86_64-6.1.config into .config
4. Run make -j N vmlinux in the dir with N being the number of cores, I personally used 12 as I hit this bug but I doubt this matters
5. Copy the produced vmlinux anywhere used to start Firecracker VMs
6. Start a VM configured with: one ext4 block device that is a root device, sync io engine, and no initramfs is used (and this kernel, of course)
7. Receive the aforementioned error when booting the VM

Expected behaviour

Everything would build correctly and the VM would start.

Environment

• Firecracker version: 1.9.1, tried 1.7.0 with the same thing happening
• Host and guest kernel versions: 6.1.99 host and 6.1.114 guest
• Rootfs used: trimmed down Debian ext4 with systemd, created with buildfs as per my docs about it
• Architecture: x86_64
• Any other relevant software versions:

Additional context

Even though the output is the same as #4816, I'm quite sure this isn't the same issue as I'm not using noapic as a kernel boot arg (mine are console=ttyS0 reboot=k panic=1) and adding/removing it doesn't change anything.

HOWEVER! the way I pinned this issue down to that specific commit is that if you take the commit right before that problematic one and open the relevant guest kernel config in it: https://github.com/firecracker-microvm/firecracker/blob/86a2559b26a4b9a05405aeaa58bab0f7261d71bc/resources/guest_configs/microvm-kernel-ci-x86_64-6.1.config

And do the same steps with that config, everything works perfectly and a 29MB working vmlinux is produced.

Checks

• Have you searched the Firecracker Issues database for similar problems?
• Have you read the existing relevant Firecracker documentation?
• Are you certain the bug being reported is a Firecracker issue?

[kanpov]

Is there any progress on this?

[JackThomson2]

Hi @kanpov,

This issue appears to happen because we build from Amazon Linux which has some patches that allow ACPI to be enabled without PCI. To resolve this you can set CONFIG_PCI=y in your config file, or you can build from Amazon Linux as we do.

A link to our docs on booting with acpi on x86-64 can be found here.

Thanks

[kanpov]

Hi @kanpov,

This issue appears to happen because we build from Amazon Linux which has some patches that allow ACPI to be enabled without PCI. To resolve this you can set CONFIG_PCI=y in your config file, or you can build from Amazon Linux as we do.

A link to our docs on booting with acpi on x86-64 can be found here.

Thanks

The bare minimum to fix this would be to mention your usage of Amazon Linux as a requirement for these guest kernel configs. Even with that fixed, I doubt it's a good idea to have what are advertised as general-purpose kernel configs only work with this specific kernel.