[Bug] Could not create the network device: IoVecBuffer(Mut) in 1.10.0

[gounselor]

Describe the bug

I have a working vmconfig.json which works with firecracker-v1.9.1-aarch64 and guest kernel 6.11.* and multiple root filesystems.

If I use the same vmconfig.json with firecracker-v1.10.0-aarch64 it fails with:

2024-11-08T19:44:55.998563652 [anonymous-instance:main] Running Firecracker v1.10.0
2024-11-08T19:44:55.999048225 [anonymous-instance:main] RunWithoutApiError error: Failed to build MicroVM from Json: Configuration for VMM from one single json failed: Network device error: Could not create the network device: IoVecBuffer(Mut) error: Error with underlying `IovDeque`: Error calling mmap: Invalid argument (os error 22)
Error: RunWithoutApiError(BuildMicroVMFromJson(ParseFromJson(NetDevice(CreateNetworkDevice(IoVecError(IovDeque(Mmap(Os { code: 22, kind: InvalidInput, message: "Invalid argument" }))))))))
2024-11-08T19:44:55.999238447 [anonymous-instance:main] Firecracker exiting with error. exit_code=1

To Reproduce

Start firecracker 1.10.0 with this command line:

sudo release-v1.10.0-aarch64/firecracker-v1.10.0-aarch64 --no-api --config-file vmconfig.json

Config file vmconfig.json used:

{
  "boot-source": {
    "kernel_image_path": "kernel",
    "boot_args": "ro console=ttyS0 noapic reboot=k panic=1 random.trust_cpu=on ip=192.168.179.2::192.168.179.1:255.255.255.0::eth0:off"
  },
  "drives": [
    {
      "drive_id": "rootfs",
      "path_on_host": "/dev/nvme/alpine-minimal",
      "is_root_device": true,
      "is_read_only": false
    }
  ],
  "network-interfaces": [
      {
          "iface_id": "eth0",
          "guest_mac": "02:FC:00:00:00:01",
          "host_dev_name": "fc-tap0"
      }
  ],
  "machine-config": {
    "vcpu_count": 1,
    "mem_size_mib": 256
  }
}

Expected behaviour

I expect the vm to start like v1.9.1 does.

Environment

• Firecracker version: firecracker-v1.10.0-aarch64
• Host and guest kernel versions:
  • Host: 6.6.51+rpt-rpi-2712
  • Guest: 6.11.x, Image-6.11.7: Linux kernel ARM64 boot executable Image, little-endian, 4K pages
• Rootfs used: alpine, void, ubuntu2[24] etc. does not seem to matter as the vm creation fails
• Architecture: arm64/aarch64
• Any other relevant software versions: don't think so, works with firecracker 1.9.1

Additional context

Happy to help debugging.

Checks

• Have you searched the Firecracker Issues database for similar problems?
• Have you read the existing relevant Firecracker documentation?
• Are you certain the bug being reported is a Firecracker issue?

[roypat]

Hi @gounselor,

Thanks a lot for reaching out about this. Since you mention running on a raspberry pi, may I ask what page size your host is configured with? We suspect the issue you're seeing is due to an assumption on host page size being 4K based into the iov_deque.rs code. If your host is configured with 16K (or another non-4K page size), could you try if it still reproduces with 4K pages?

[gounselor]

Hi @roypat,

yes, good finding. Indeed the page size is 16k according to the supplied kernel config and dmesg.
kernel is 6.6.51+rpt-rpi-2712 which is configured with
CONFIG_ARM64_16K_PAGES=y

This is also reported by:
root@pi5:~# getconf PAGESIZE
16384

The kernel is supplied by Raspberry Pi OS (former raspbian?) which is based on Debian GNU/Linux 12 (bookworm).

I can try to get or compile a kernel with 4k pages, it's not possible to change via cmdline etc.

[ShadowCurse]

@gounselor You may be able to switch to 4k kernel by setting kernel=kernel8.img in the config.txt

[gounselor]

@ShadowCurse and @roypat

Not so easy on a headless pi5 but managed it. Actually moving kernel_2712.img "away" boots kernel8.img.

With this one (kernel8.img) and
`
ms@pi5:/mnt/firecracker $ getconf PAGESIZE

4096
`

it works:

2024-11-13T18:05:30.871718746 [anonymous-instance:main] Running Firecracker v1.10.0 2024-11-13T18:05:31.880249104 [anonymous-instance:main] Artificially kick devices. 2024-11-13T18:05:31.880349391 [anonymous-instance:fc_vcpu 0] Received a VcpuEvent::Resume message with immediate_exit enabled. immediate_exit was disabled before proceeding 2024-11-13T18:05:31.880387726 [anonymous-instance:main] Successfully started microvm that was configured from one single json [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x414fd0b1] [ 0.000000] Linux version 6.11.7 (ms@pi5) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP Fri Nov 8 18:35:52 CET 2024 ...