Test VM boot using an SPDK bdev_aio disk

Reproducing with QEMU (Incomplete, WIP)

0. Setup

sudo apt install qemu-system-x86 mtools

export SPDK_BIN=/opt/spdk/bin

echo '{"subsystems":[{"subsystem":"bdev","config":[{"method":"bdev_aio_create","params":{"name":"aio0","block_size":512,"filename":"disk.raw","readonly":false}}}]}' \
> spdk_conf.json

Enable hugepages:

echo 2048 | sudo tee  /proc/sys/vm/nr_hugepages

1. Download & create image

wget https://cloud-images.ubuntu.com/releases/22.04/release-20240701/ubuntu-22.04-server-cloudimg-amd64.img

wget https://raw.githubusercontent.com/cloud-hypervisor/cloud-hypervisor/main/test_data/cloud-init/ubuntu/local/user-data
wget https://raw.githubusercontent.com/cloud-hypervisor/cloud-hypervisor/main/test_data/cloud-init/ubuntu/local/network-config
wget https://raw.githubusercontent.com/cloud-hypervisor/cloud-hypervisor/main/test_data/cloud-init/ubuntu/local/meta-data

rm -f /tmp/ubuntu-cloudinit.img
mkdosfs -n CIDATA -C /tmp/ubuntu-cloudinit.img 8192
mcopy -oi /tmp/ubuntu-cloudinit.img -s ./user-data ::
mcopy -oi /tmp/ubuntu-cloudinit.img -s ./network-config ::
mcopy -oi /tmp/ubuntu-cloudinit.img -s ./meta-data ::

# convert image to raw format
qemu-img convert -p -f qcow2 -O raw ubuntu-22.04-server-cloudimg-amd64.img jammy.raw
truncate -s 5G jammy.raw

# for unencrypted, just a simple copy
rm -rf disk.raw
cp jammy.raw disk.raw

2. Start vhost

export SPDK_SCRIPTS=/opt/spdk/scripts
HUGEMEM=5120 $SPDK_SCRIPTS/setup.sh

sudo ~/projects/bdev_ubi/bin/vhost_ubi --json spdk_conf.json  -S /var/tmp/

3a. Run qemu

sudo qemu-system-x86_64 \
   --enable-kvm -cpu host -smp 2 -m 1G \
   -object memory-backend-file,id=mem0,size=1G,mem-path=/dev/hugepages,share=on -numa node,memdev=mem0 \
   -chardev socket,id=spdk_vhost_blk0,path=/var/tmp/vhost.0,reconnect=1 \
   -device vhost-user-blk-pci,chardev=spdk_vhost_blk0,num-queues=4,bootindex=0 \
   -drive file=/tmp/ubuntu-cloudinit.img,format=raw \
   -nographic \
   -nic none

With 2 disks:

sudo qemu-system-x86_64 \
   --enable-kvm -cpu host -smp 2 -m 1G \
   -object memory-backend-file,id=mem0,size=1G,mem-path=/dev/hugepages,share=on -numa node,memdev=mem0 \
   -chardev socket,id=spdk_vhost_blk0,path=/var/tmp/vhost.0,reconnect=1 \
   -device vhost-user-blk-pci,chardev=spdk_vhost_blk0,num-queues=4,bootindex=0 \
   -chardev socket,id=spdk_vhost_blk1,path=/var/tmp/vhost.1,reconnect=1 \
   -device vhost-user-blk-pci,chardev=spdk_vhost_blk1,num-queues=4 \
   -drive file=/tmp/ubuntu-cloudinit.img,format=raw \
   -nographic \
   -nic none

Login with:

• User: cloud
• Password: cloud123

3b. Run cloud-hypervisor

wget https://github.com/cloud-hypervisor/cloud-hypervisor/releases/download/v35.1/cloud-hypervisor

wget https://github.com/cloud-hypervisor/rust-hypervisor-firmware/releases/download/0.4.2/hypervisor-fw

wget https://github.com/ubicloud/build-edk2-firmware/releases/download/edk2-stable202311-x64/CLOUDHV-x64.fd

chmod a+x cloud-hypervisor

sudo ./cloud-hypervisor \
    --kernel ./CLOUDHV-x64.fd \
    --disk path=/tmp/ubuntu-cloudinit.img \
    --disk vhost_user=true,socket=/var/tmp/vhost.0,num_queues=1,queue_size=256 \
    --cpus boot=1 \
    --memory size=2G,hugepages=on,shared=true \
    --net "tap=,mac=,ip=,mask=" \
    --serial tty \
    --console off

4. Commands inside the VM

dd if=/dev/random of=1.txt bs=512 count=1000000
sync 1.txt
dd if=/dev/random of=2.txt bs=512 count=1000000
sync 2.txt
lsblk

5. Snapshots

req.0 is:

{
  "jsonrpc": "2.0",
  "method": "bdev_ubi_snapshot",
  "params": {"name": "ubi0", "path":"/tmp/ubi.snapshot"},
  "id": 1
}

req.1 is:

{
  "jsonrpc": "2.0",
  "method": "bdev_ubi_snapshot_status",
  "params": {"name": "ubi0"},
  "id": 1
}

To start snapshot

sudo socat - UNIX-CONNECT:/var/tmp/spdk.sock < ~/tmp/req.0 | jq .

To check status

sudo socat - UNIX-CONNECT:/var/tmp/spdk.sock < ~/tmp/req.1 | jq .

Test VM in Prod

Create the disk image

mkdir -p /var/storage/tmp
cd /var/storage/tmp
cp /var/scp /var/storage/images/ubuntu-jammy-20240701.raw disk.raw
truncate -s 10G disk.raw
chmod a+w disk.raw

Create the aio bdev:

export RPC_PY="/opt/spdk-v23.09-ubi-0.2/scripts/rpc.py -s /home/spdk/spdk-v23.09-ubi-0.2.sock"
$RPC_PY bdev_aio_create /var/storage/tmp/disk.raw temp_vm_aio 4096

Create the vhost controller:

$RPC_PY vhost_create_blk_controller -r temp_vm temp_vm_aio

Run the VM:

sudo /opt/cloud-hypervisor/v35.1/cloud-hypervisor \
    --kernel /opt/fw/edk2-stable202302/x64/CLOUDHV.fd \
    --disk path=/tmp/ubuntu-cloudinit.img \
    --disk vhost_user=true,socket=/var/storage/vhost/temp_vm,num_queues=1,queue_size=256 \
    --cpus boot=1 \
    --memory size=2G,hugepages=on,shared=true \
    --net "tap=,mac=,ip=,mask=" \
    --serial tty \
    --console off