Initial aarch64 support, cross arch stateless images with qemu-user-static, grub2 for x86 #7257

Obihoernchen · 2022-09-30T13:24:13Z

The following PR adds three features:

initial aarch64 support for xCAT using grub2
grub2 support for x86 arch
cross arch stateless image generation with qemu-user-static allowing you to build ppc64 or aarch64 stateless images on x86 hosts

It's not 100% ready yet and at the moment I've only tested EL8, but it should be good enough for some PoC tests :)

Status

stateful aarch64 images
stateless aarch64 images
Fix aarch64 install dependencies
activate xCAT-genesis-scripts-aarch64 build + build xCAT-genesis-base-aarch64 + add dep to xCAT.spec (optional)
grub2 support for x86 arch
cross arch stateless image generation with qemu-user-static
Man pages
Documentation

Emulate aarch64 on x86_64 with QEMU for testing

Install qemu-system-aarch64
Create VM with:
- arch: aarch64
- machine type: virt
- cpu model: max
- emulator: /usr/bin/qemu-system-aarch64
- firmware: UEFI aarch64: /usr/share/edk2/aarch64/QEMU_EFI-silent-pflash.raw

Note: tested with Fedora 36

Refs:

xCAT RPMs for aarch64

xcat-core packages are builded for aarch64 with: bf57a2b13ef5978a3e05076b195f01dc3f2ddb35
Missing dependencies:
- ipmitool-xcat
- goconserver [dep, SOL] --> built
- perl-IO-Tty.x86_64 [dep] --> perl-IO-Tty.aarch64 in powertools repo

Build goconserver for aarch64

dnf install golang
git clone https://github.com/xcat2/goconserver.git
cd goconserver
go env -w GO111MODULE=auto
make deps
make rpm
ls -l build/*.rpm

grub2 boot preparation

Download grub2 efi for x86 and aarch64 from any EL8 mirror. For instance:
- https://ftp.fau.de/almalinux/8/BaseOS/aarch64/os/EFI/BOOT/grubaa64.efi
- https://ftp.fau.de/almalinux/8/BaseOS/x86_64/os/EFI/BOOT/grubx64.efi
Copy to /tftpboot/boot/grub2/grub2.aarch64 and /tftpboot/boot/grub2/grub2.x86_64

qemu-user-static for cross arch stateless images

Get qemu-user-static binary for aarch64 and copy to: /usr/bin/qemu-aarch64-static with 755 permissions
Configure /etc/binfmt.d/qemu-aarch64-static.conf accordingly:

:qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-static:F

Restart binfmt service: systemctl restart systemd-binfmt

Misc

[root@xcat-aarch64 grub2]# lsdef -t osimage rocky8-aarch64-netboot-compute
Object name: rocky8-aarch64-netboot-compute
    exlist=/opt/xcat/share/xcat/netboot/rocky/compute.rocky8.x86_64.exlist
    imagetype=linux
    osarch=aarch64
    osdistroname=rocky8-aarch64
    osname=Linux
    osvers=rocky8
    otherpkgdir=/install/post/otherpkgs/rocky8/aarch64
    permission=755
    pkgdir=/install/rocky8/aarch64
    pkglist=/opt/xcat/share/xcat/netboot/rocky/compute.rocky8.x86_64.pkglist
    postinstall=/opt/xcat/share/xcat/netboot/rocky/compute.rocky8.x86_64.postinstall
    profile=compute
    provmethod=netboot
    rootimgdir=/install/netboot/rocky8/aarch64/compute

Add hooks to do a draft implementation of aarch64 using grub

Arm and ppc64 do not have linuxefi and initrdefi.

Obihoernchen · 2024-02-23T16:55:55Z

Works fine with el9, too.
Feel free to test it. You don't need an arm management node for testing.
Just install grub2 and qemu-user-static on your management node as described above.

Obihoernchen · 2024-07-03T19:16:20Z

Note: you might have to manually edit generated grub configs because ARM servers might use serial port ttyAMA0 and xCAT will always use the string "ttyS" + nodehm.serialport

Obihoernchen · 2024-07-12T17:39:08Z

@ocfmatt please take a look at it. Especially the documentation I have written.

ocfmatt

The changes look good to me. I like the use of qemu for cross architecture compiling.

WZT666-dev · 2024-12-06T05:34:48Z

Hello：
I am trying to distribute aarch64 on a physical server. My management node is x86_64 and the system version is rocky8.5; The distributed system is the ARM version of Rockey8.10, and the CPU of my client machine is ARM architecture. When I prepared grub2.aarch64 under/tftpboot/boot/grub2, I encountered the following error when restarting

Start PXE over IPv4
Mac Addr:F4-DE-AF-54-5D-9D.
Station IP address is 172.22.2.26
Server IP address is 172.22.2.223
NBP filename is grubaa64.efi
NBP filesize is 0 Bytes
PXE-E23:Client received TFTP error from server

How can this need be resolved？

Obihoernchen · 2024-12-06T10:45:04Z

Did you check firewall etc.?
Is grub2 boot working for x86 nodes?

WZT666-dev · 2024-12-06T11:01:10Z

Did you check firewall etc.? Is grub2 boot working for x86 nodes?

Thanks，
All firewalls are turned off.
What do you mean by grub2? My client node is ARM and I have prepared grub2.aarch64

Obihoernchen · 2024-12-06T11:42:03Z

Do you have a x86 client node?
For testing purposes you could also add the grub2 file for x86 (see doc) and boot the x86 client via netboot=grub2.
I just want to see whether this is general issue with your setup or an aarch64 specific issue.

WZT666-dev · 2024-12-06T11:44:34Z

Do you have a x86 client node? For testing purposes you could also add the grub2 file for x86 (see doc) and boot the x86 client via netboot=grub2. I just want to see whether this is general issue with your setup or an aarch64 specific issue.

Thank you for your response. I am currently testing the x86 client to see if it is a configuration issue.

WZT666-dev · 2024-12-06T12:08:36Z

NBP filename is grubaa64.efi
NBP filesize is 0 Bytes
PXE-E23:Client received TFTP error from server

I tested using the x86_64 client and successfully deployed the system, but ARM still encountered errors
NBP filename is grubaa64.efi
NBP filesize is 0 Bytes
PXE-E23:Client received TFTP error from server

Obihoernchen · 2024-12-06T13:12:09Z

Can you please post the output of:

grep -A 1 -B 10 grub2 /etc/dhcp/dhcpd.conf /var/lib/dhcpd/dhcpd.leases

WZT666-dev · 2024-12-06T13:16:03Z

Can you please post the output of:
grep -A 1 -B 10 grub2 /etc/dhcp/dhcpd.conf /var/lib/dhcpd/dhcpd.leases

`/etc/dhcp/dhcpd.conf- filename "xcat/xnba.kpxe";
/etc/dhcp/dhcpd.conf- } else if option vendor-class-identifier = "Etherboot-5.4" { #x86
/etc/dhcp/dhcpd.conf- filename "xcat/xnba.kpxe";
/etc/dhcp/dhcpd.conf- } else if option client-architecture = 00:07 { #x86_64 uefi
/etc/dhcp/dhcpd.conf- filename "xcat/xnba.efi";
/etc/dhcp/dhcpd.conf- } else if option client-architecture = 00:09 { #x86_64 uefi alternative id
/etc/dhcp/dhcpd.conf- filename "xcat/xnba.efi";
/etc/dhcp/dhcpd.conf- } else if option client-architecture = 00:02 { #ia64
/etc/dhcp/dhcpd.conf- filename "elilo.efi";
/etc/dhcp/dhcpd.conf- } else if option client-architecture = 00:0b { #aaarch64
/etc/dhcp/dhcpd.conf: filename "boot/grub2/grub2.aarch64";
/etc/dhcp/dhcpd.conf- } else if option client-architecture = 00:0e { #OPAL-v3

/var/lib/dhcpd/dhcpd.leases- supersede server.ddns-hostname = "fat09";
/var/lib/dhcpd/dhcpd.leases- supersede host-name = "fat09";
/var/lib/dhcpd/dhcpd.leases-}
/var/lib/dhcpd/dhcpd.leases-host arm01 {
/var/lib/dhcpd/dhcpd.leases- dynamic;
/var/lib/dhcpd/dhcpd.leases- hardware ethernet f4:de:af:54:5d:9d;
/var/lib/dhcpd/dhcpd.leases- uid f4:de:af:54:5d:9d;
/var/lib/dhcpd/dhcpd.leases- fixed-address 172.22.2.246;
/var/lib/dhcpd/dhcpd.leases- supersede server.ddns-hostname = "arm01";
/var/lib/dhcpd/dhcpd.leases- supersede host-name = "arm01";
/var/lib/dhcpd/dhcpd.leases: supersede server.filename = "/boot/grub2/grub2-arm01";
/var/lib/dhcpd/dhcpd.leases- supersede server.next-server = ac:16:02:f4;
/var/lib/dhcpd/dhcpd.leases-}
/var/lib/dhcpd/dhcpd.leases-host arm02 {
/var/lib/dhcpd/dhcpd.leases- dynamic;
/var/lib/dhcpd/dhcpd.leases- hardware ethernet f4:de:af:54:58:25;
/var/lib/dhcpd/dhcpd.leases- uid f4:de:af:54:58:25;
/var/lib/dhcpd/dhcpd.leases- fixed-address 172.22.2.247;
/var/lib/dhcpd/dhcpd.leases- supersede server.ddns-hostname = "arm02";
/var/lib/dhcpd/dhcpd.leases- supersede host-name = "arm02";
/var/lib/dhcpd/dhcpd.leases: supersede server.filename = "/boot/grub2/grub2-arm02";
/var/lib/dhcpd/dhcpd.leases- supersede server.next-server = ac:16:02:f4;

/var/lib/dhcpd/dhcpd.leases- dynamic;
/var/lib/dhcpd/dhcpd.leases- deleted;
/var/lib/dhcpd/dhcpd.leases-}
/var/lib/dhcpd/dhcpd.leases-host c02n04 {
/var/lib/dhcpd/dhcpd.leases- dynamic;
/var/lib/dhcpd/dhcpd.leases- hardware ethernet 9c:dc:71:68:cb:30;
/var/lib/dhcpd/dhcpd.leases- uid 9c:dc:71:68:cb:30;
/var/lib/dhcpd/dhcpd.leases- fixed-address 172.22.2.30;
/var/lib/dhcpd/dhcpd.leases- supersede server.ddns-hostname = "c02n04";
/var/lib/dhcpd/dhcpd.leases- supersede host-name = "c02n04";
/var/lib/dhcpd/dhcpd.leases: supersede server.filename = "/boot/grub2/grub2-c02n04";
/var/lib/dhcpd/dhcpd.leases- supersede server.next-server = ac:16:02:f4;

/var/lib/dhcpd/dhcpd.leases- dynamic;
/var/lib/dhcpd/dhcpd.leases- deleted;
/var/lib/dhcpd/dhcpd.leases-}
/var/lib/dhcpd/dhcpd.leases-host arm01 {
/var/lib/dhcpd/dhcpd.leases- dynamic;
/var/lib/dhcpd/dhcpd.leases- hardware ethernet f4:de:af:54:5d:9d;
/var/lib/dhcpd/dhcpd.leases- uid f4:de:af:54:5d:9d;
/var/lib/dhcpd/dhcpd.leases- fixed-address 172.22.2.246;
/var/lib/dhcpd/dhcpd.leases- supersede server.ddns-hostname = "arm01";
/var/lib/dhcpd/dhcpd.leases- supersede host-name = "arm01";
/var/lib/dhcpd/dhcpd.leases: supersede server.filename = "/boot/grub2/grub2-arm01";
/var/lib/dhcpd/dhcpd.leases- supersede server.next-server = ac:16:02:f4;

/var/lib/dhcpd/dhcpd.leases- dynamic;
/var/lib/dhcpd/dhcpd.leases- deleted;
/var/lib/dhcpd/dhcpd.leases-}
/var/lib/dhcpd/dhcpd.leases-host c02n04 {
/var/lib/dhcpd/dhcpd.leases- dynamic;
/var/lib/dhcpd/dhcpd.leases- hardware ethernet 9c:dc:71:68:cb:30;
/var/lib/dhcpd/dhcpd.leases- uid 9c:dc:71:68:cb:30;
/var/lib/dhcpd/dhcpd.leases- fixed-address 172.22.2.30;
/var/lib/dhcpd/dhcpd.leases- supersede server.ddns-hostname = "c02n04";
/var/lib/dhcpd/dhcpd.leases- supersede host-name = "c02n04";
/var/lib/dhcpd/dhcpd.leases: supersede server.filename = "/boot/grub2/grub2-c02n04";
/var/lib/dhcpd/dhcpd.leases- supersede server.next-server = ac:16:02:f4;

/var/lib/dhcpd/dhcpd.leases- dynamic;
/var/lib/dhcpd/dhcpd.leases- deleted;
/var/lib/dhcpd/dhcpd.leases-}
/var/lib/dhcpd/dhcpd.leases-host arm02 {
/var/lib/dhcpd/dhcpd.leases- dynamic;
/var/lib/dhcpd/dhcpd.leases- hardware ethernet f4:de:af:54:58:25;
/var/lib/dhcpd/dhcpd.leases- uid f4:de:af:54:58:25;
/var/lib/dhcpd/dhcpd.leases- fixed-address 172.22.2.247;
/var/lib/dhcpd/dhcpd.leases- supersede server.ddns-hostname = "arm02";
/var/lib/dhcpd/dhcpd.leases- supersede host-name = "arm02";
/var/lib/dhcpd/dhcpd.leases: supersede server.filename = "/boot/grub2/grub2-arm02";
/var/lib/dhcpd/dhcpd.leases- supersede server.next-server = ac:16:02:f4;

/var/lib/dhcpd/dhcpd.leases- dynamic;
/var/lib/dhcpd/dhcpd.leases- deleted;
/var/lib/dhcpd/dhcpd.leases-}
/var/lib/dhcpd/dhcpd.leases-host arm01 {
/var/lib/dhcpd/dhcpd.leases- dynamic;
/var/lib/dhcpd/dhcpd.leases- hardware ethernet f4:de:af:54:5d:9d;
/var/lib/dhcpd/dhcpd.leases- uid f4:de:af:54:5d:9d;
/var/lib/dhcpd/dhcpd.leases- fixed-address 172.22.2.246;
/var/lib/dhcpd/dhcpd.leases- supersede server.ddns-hostname = "arm01";
/var/lib/dhcpd/dhcpd.leases- supersede host-name = "arm01";
/var/lib/dhcpd/dhcpd.leases: supersede server.filename = "/boot/grub2/grub2-arm01";
/var/lib/dhcpd/dhcpd.leases- supersede server.next-server = ac:16:02:f4;
`

c02n04是x86节点，arm01与02是客户端节点

Obihoernchen · 2024-12-06T13:21:46Z

Hmm looks good.
Does /tftpboot/boot/grub2/grub2-arm01 link to grub2.aarch64?

WZT666-dev · 2024-12-06T13:24:44Z

Hmm looks good. Does /tftpboot/boot/grub2/grub2-arm01 link to grub2.aarch64?

Yes, I have tried the default one, which is/tftpboot/boot/grub2/grub2-arm01 link to grub2.aarch64
I also tried canceling/tftpboot/boot/grub2/grub2-arm01 link to grub2.aarch64, copying grub2.aarch64 to grub2-arm01, but still encountered an error

Obihoernchen changed the title ~~Initial aarch64 support~~ Initial aarch64 support, cross arch stateless images with qemu-user-static, grub2 for x86 Sep 30, 2022

Obihoernchen changed the title ~~Initial aarch64 support, cross arch stateless images with qemu-user-static, grub2 for x86~~ WIP: Initial aarch64 support, cross arch stateless images with qemu-user-static, grub2 for x86 Sep 30, 2022

Obihoernchen added this to the 2.17 milestone Feb 7, 2024

Obihoernchen self-assigned this Feb 7, 2024

jjohnson42 and others added 10 commits February 21, 2024 11:59

Preliminary aarch64 support

1b44680

Add hooks to do a draft implementation of aarch64 using grub

new kernel for aarch64

6403987

Support x86, ppc64 and arm in grub

826a8a8

Arm and ppc64 do not have linuxefi and initrdefi.

Fix nodeset and rinstall for arm

699a26d

Do not use hacked armgrub plugin

40ac583

Build xcat-core for aarch64

bf57a2b

Fix genesis builds for aarch64

295dff5

Do not build xCAT-genesis-scripts-aarch64 yet

02aaa24

qemu-user-static support for genimage + aarch64 support

b27f419

Update man pages with aarch64 and qemu-user-static

0d2c970

Obihoernchen force-pushed the aarch64 branch from 9947c3c to 0d2c970 Compare February 21, 2024 11:04

Obihoernchen added the component:os_provision label Jul 12, 2024

Update man pages for aarch64

857e290

Obihoernchen force-pushed the aarch64 branch 3 times, most recently from b4a6bf9 to c4edbdd Compare July 12, 2024 16:49

Update documentation for aarch64 and grub2 support

8329cf6

Obihoernchen force-pushed the aarch64 branch from c4edbdd to 8329cf6 Compare July 12, 2024 17:37

Obihoernchen changed the title ~~WIP: Initial aarch64 support, cross arch stateless images with qemu-user-static, grub2 for x86~~ Initial aarch64 support, cross arch stateless images with qemu-user-static, grub2 for x86 Jul 12, 2024

Obihoernchen requested a review from ocfmatt July 12, 2024 17:38

Enable grub2 support for x86

597fea5

ocfmatt approved these changes Jul 25, 2024

View reviewed changes

Obihoernchen merged commit c8cb1d6 into xcat2:master Jul 25, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial aarch64 support, cross arch stateless images with qemu-user-static, grub2 for x86 #7257

Initial aarch64 support, cross arch stateless images with qemu-user-static, grub2 for x86 #7257

Obihoernchen commented Sep 30, 2022 •

edited

Loading

Obihoernchen commented Feb 23, 2024 •

edited

Loading

Obihoernchen commented Jul 3, 2024

Obihoernchen commented Jul 12, 2024

ocfmatt left a comment

WZT666-dev commented Dec 6, 2024

Obihoernchen commented Dec 6, 2024

WZT666-dev commented Dec 6, 2024

Obihoernchen commented Dec 6, 2024

WZT666-dev commented Dec 6, 2024

WZT666-dev commented Dec 6, 2024

Obihoernchen commented Dec 6, 2024

WZT666-dev commented Dec 6, 2024

Obihoernchen commented Dec 6, 2024

WZT666-dev commented Dec 6, 2024

Initial aarch64 support, cross arch stateless images with qemu-user-static, grub2 for x86 #7257

Initial aarch64 support, cross arch stateless images with qemu-user-static, grub2 for x86 #7257

Conversation

Obihoernchen commented Sep 30, 2022 • edited Loading

Status

Emulate aarch64 on x86_64 with QEMU for testing

xCAT RPMs for aarch64

Build goconserver for aarch64

grub2 boot preparation

qemu-user-static for cross arch stateless images

Misc

Obihoernchen commented Feb 23, 2024 • edited Loading

Obihoernchen commented Jul 3, 2024

Obihoernchen commented Jul 12, 2024

ocfmatt left a comment

Choose a reason for hiding this comment

WZT666-dev commented Dec 6, 2024

Obihoernchen commented Dec 6, 2024

WZT666-dev commented Dec 6, 2024

Obihoernchen commented Dec 6, 2024

WZT666-dev commented Dec 6, 2024

WZT666-dev commented Dec 6, 2024

Obihoernchen commented Dec 6, 2024

WZT666-dev commented Dec 6, 2024

Obihoernchen commented Dec 6, 2024

WZT666-dev commented Dec 6, 2024

Obihoernchen commented Sep 30, 2022 •

edited

Loading

Obihoernchen commented Feb 23, 2024 •

edited

Loading