Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial aarch64 support, cross arch stateless images with qemu-user-static, grub2 for x86 #7257

Merged
merged 13 commits into from
Jul 25, 2024
Merged

Initial aarch64 support, cross arch stateless images with qemu-user-static, grub2 for x86 #7257

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
1b44680
Preliminary aarch64 support
jjohnson42 Jan 14, 2016
6403987
new kernel for aarch64
Obihoernchen May 18, 2022
826a8a8
Support x86, ppc64 and arm in grub
Obihoernchen May 18, 2022
699a26d
Fix nodeset and rinstall for arm
Obihoernchen May 19, 2022
40ac583
Do not use hacked armgrub plugin
Obihoernchen May 19, 2022
bf57a2b
Build xcat-core for aarch64
Obihoernchen May 19, 2022
295dff5
Fix genesis builds for aarch64
Obihoernchen May 19, 2022
02aaa24
Do not build xCAT-genesis-scripts-aarch64 yet
Obihoernchen May 20, 2022
b27f419
qemu-user-static support for genimage + aarch64 support
Obihoernchen May 24, 2022
0d2c970
Update man pages with aarch64 and qemu-user-static
Obihoernchen Feb 21, 2024
857e290
Update man pages for aarch64
Obihoernchen Jul 12, 2024
8329cf6
Update documentation for aarch64 and grub2 support
Obihoernchen Jul 12, 2024
597fea5
Enable grub2 support for x86
Obihoernchen Jul 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion buildcore.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -371,6 +371,9 @@ if [ "$OSNAME" != "AIX" ]; then
if [ $? -ne 0 ]; then FAILEDRPMS="$FAILEDRPMS xCAT-genesis-scripts-x86_64"; fi
./makerpm xCAT-genesis-scripts ppc64 "$EMBED"
if [ $? -ne 0 ]; then FAILEDRPMS="$FAILEDRPMS xCAT-genesis-scripts-ppc64"; fi
# Do not build xCAT-genesis-scripts-aarch64 yet
#./makerpm xCAT-genesis-scripts aarch64 "$EMBED"
#if [ $? -ne 0 ]; then FAILEDRPMS="$FAILEDRPMS xCAT-genesis-scripts-aarch64"; fi
if [ "$FAILEDRPMS" = "$ORIGFAILEDRPMS" ]; then # all succeeded
rm -f $DESTDIR/xCAT-genesis-scripts*rpm
rm -f $SRCDIR/xCAT-genesis-scripts*rpm
Expand All @@ -392,7 +395,7 @@ for rpmname in xCAT xCATsn; do
./makerpm $rpmname "$EMBED"
if [ $? -ne 0 ]; then FAILEDRPMS="$FAILEDRPMS $rpmname"; fi
else
for arch in x86_64 ppc64 ppc64le s390x; do
for arch in x86_64 ppc64 ppc64le s390x aarch64; do
if [ "$rpmname" = "xCAT-OpenStack" -a "$arch" != "x86_64" ] || [ "$rpmname" = "xCAT-OpenStack-baremetal" -a "$arch" != "x86_64" ] ; then continue; fi # only bld openstack for x86_64 for now
./makerpm $rpmname $arch "$EMBED"
if [ $? -ne 0 ]; then FAILEDRPMS="$FAILEDRPMS $rpmname-$arch"; fi
Expand Down
2 changes: 1 addition & 1 deletion buildlocal.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ echo "This is an $OSNAME system"
$CURDIR/makerpm $rpmname
if [ $? -ne 0 ]; then FAILEDRPMS="$FAILEDRPMS $rpmname"; fi
else
for arch in x86_64 ppc64 s390x; do
for arch in x86_64 ppc64 s390x aarch64; do
$CURDIR/makerpm $rpmname $arch
if [ $? -ne 0 ]; then FAILEDRPMS="$FAILEDRPMS $rpmname-$arch"; fi
done
Expand Down
22 changes: 22 additions & 0 deletions docs/source/advanced/mixed_cluster/building_stateless_images.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,28 @@ In a homogeneous cluster, the management node is the same hardware architecture

The issues arises in a heterogeneous cluster, where the management node is running a different level operating system *or* hardware architecture as the compute nodes in which to deploy the image. The ``genimage`` command that builds stateless images depends on various utilities provided by the base operating system and needs to be run on a node with the same hardware architecture and *major* Operating System release as the nodes that will be booted from the image.

When running xCAT >= 2.17 on EL >= 8 based management node with x86_64 architecture, qemu-user-static can be used to cross-build ppc64* and aarch64 osimages. Therefore, you don't need to build images on systems with the target architecture anymore.

Cross-build ppc64*/aarch64 stateless/statelite image on x86_64 management node
------------------------------------------------------------------------------

#. Download qemu-user-static binaries for ppc64le and/or aarch64: ::

wget https://github.com/multiarch/qemu-user-static/releases/latest/download/qemu-ppc64le-static -P /usr/bin
wget https://github.com/multiarch/qemu-user-static/releases/latest/download/qemu-aarch64-static -P /usr/bin
chmod 755 /usr/bin/qemu-*-static

#. Configure systemd-binfmt accordingly: ::

wget https://raw.githubusercontent.com/qemu/qemu/master/scripts/qemu-binfmt-conf.sh
bash qemu-binfmt-conf.sh --systemd 'ALL' --qemu-path '/usr/bin' --qemu-suffix '-static' --persistent 'yes'
systemctl restart systemd-binfmt.service

#. Make sure ``osimage.osarch`` of your image is set to your target architecture eg. ``ppc64le`` or ``aarch64``.

Now, ``genimage`` will use systemd-binfmt and the qemu-user-static binary to build the image for the specified architecture.


Same Operating System, Different Architecture
---------------------------------------------

Expand Down
2 changes: 1 addition & 1 deletion docs/source/advanced/mixed_cluster/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Mixed Cluster
=============

The concept of mixed cluster support in xCAT is managing cluster which contain different hardware architectures. Most commonly, this is usually a cluster consisting of both Power and x86 systems.
The concept of mixed cluster support in xCAT is managing cluster which contain different hardware architectures. Most commonly, this is usually a cluster consisting of both Power, ARM and x86 systems.

.. toctree::
:maxdepth: 2
Expand Down
77 changes: 39 additions & 38 deletions docs/source/advanced/mixed_cluster/support_matrix.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,49 +1,50 @@
Support Matrix
==============

+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| | RHEL | SLES | RHEL | SLES | Ubuntu | RHEL | SLES | Ubuntu |
| | ppc64 | ppc64 | x86_64 | x86_64 | x86_64 | ppc64le | ppc64le | ppc64el |
| | CN | CN | CN | CN | CN | CN | CN | CN |
+=========+=========+=========+=========+=========+=========+=========+=========+=========+
| RHEL | | | | | | | | |
| ppc64 | yes | yes | yes | yes | yes | yes | yes | yes |
| MN/SN | | | [1]_ | [1]_ | [1]_ | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| SLES | | | | | | | | |
| ppc64 | yes | yes | yes | yes | yes | yes | yes | yes |
| MN/SN | | | [1]_ | [1]_ | [1]_ | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| RHEL | | | | | | | | |
| x86_64 | yes | yes | yes | yes | yes | yes | yes | yes |
| MN/SN | [4]_ | [4]_ | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| SLES | | | | | | | | |
| x86_64 | yes | yes | yes | yes | yes | yes | yes | yes |
| MN/SN | [4]_ | [4]_ | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| Ubuntu | | | | | | | | |
| x86_64 | yes | yes | yes | yes | yes | yes | yes | yes |
| MN/SN | [5]_ | [5]_ | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| RHEL | | | | | | | | |
| ppc64le | yes | yes | yes | yes | yes | yes | yes | yes |
| MN/SN | [2]_ | [2]_ | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| SLES | | | | | | | | |
| ppc64le | no | no | yes | yes | yes | yes | yes | yes |
| MN/SN | | | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| Ubuntu | | | | | | | | |
| ppc64el | yes | yes | yes | yes | yes | yes | yes | yes |
| MN/SN | [3]_ | [3]_ | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+
+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| | RHEL | SLES | RHEL | SLES | Ubuntu | RHEL | SLES | Ubuntu | RHEL | SLES | Ubuntu |
| | ppc64 | ppc64 | x86_64 | x86_64 | x86_64 | ppc64le | ppc64le | ppc64el | aarch64 | aarch64 | aarch64 |
| | CN | CN | CN | CN | CN | CN | CN | CN | CN | CN | CN |
+=========+=========+=========+=========+=========+=========+=========+=========+=========+=========+=========+=========+
| RHEL | | | | | | | | | | | |
| ppc64 | yes | yes | yes | yes | yes | yes | yes | yes | no | no | no |
| MN/SN | | | [1]_ | [1]_ | [1]_ | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| SLES | | | | | | | | | | | |
| ppc64 | yes | yes | yes | yes | yes | yes | yes | yes | no | no | no |
| MN/SN | | | [1]_ | [1]_ | [1]_ | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| RHEL | | | | | | | | | | | |
| x86_64 | yes | yes | yes | yes | yes | yes | yes | yes | yes | no | no |
| MN/SN | [4]_ | [4]_ | | | | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| SLES | | | | | | | | | | | |
| x86_64 | yes | yes | yes | yes | yes | yes | yes | yes | yes | no | no |
| MN/SN | [4]_ | [4]_ | | | | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| Ubuntu | | | | | | | | | | | |
| x86_64 | yes | yes | yes | yes | yes | yes | yes | yes | yes | no | no |
| MN/SN | [5]_ | [5]_ | | | | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| RHEL | | | | | | | | | | | |
| ppc64le | yes | yes | yes | yes | yes | yes | yes | yes | no | no | no |
| MN/SN | [2]_ | [2]_ | | | | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| SLES | | | | | | | | | | | |
| ppc64le | no | no | yes | yes | yes | yes | yes | yes | no | no | no |
| MN/SN | | | | | | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| Ubuntu | | | | | | | | | | | |
| ppc64el | yes | yes | yes | yes | yes | yes | yes | yes | no | no | no |
| MN/SN | [3]_ | [3]_ | | | | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+


Notes:

* The support statements refers to hardware control, operating system (os) provisioning, and general purpose system management where we do not see any obvious problems with the indicated combination.

* For diskless mixed cluster support, the initial diskless image must be created on a node running the target operating system version and architecture. see :doc:`/advanced/mixed_cluster/building_stateless_images` for more details.
* For xCAT < 2.17 diskless mixed cluster support, the initial diskless image must be created on a node running the target operating system version and architecture. On xCAT >= 2.17 with x86 based management nodes QEMU emulation can be used for different architectures (eg. ppc64 or aarch64) if qemu-user-static is installed and configured via systemd-binfmt. See :doc:`/advanced/mixed_cluster/building_stateless_images` for more details.

.. rubric:: Footnotes

Expand Down
6 changes: 4 additions & 2 deletions docs/source/guides/admin-guides/basic_concepts/xcat_object/node.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Key Attributes
The operating system deployed on this node. Valid values: AIX, rhels*, rhelc*, rhas*, centos*, SL*, fedora*, sles* (where * is the version #)

* arch:
The hardware architecture of this node. Valid values: x86_64, ppc64, x86, ia64.
The hardware architecture of this node. Valid values: x86_64, ppc64, x86, ia64, aarch64.

* groups:
Usually, there are a set of nodes with some attributes in common, xCAT admin can define a node group containing these nodes, so that the management task can be issued against the group instead of individual nodes. A node can be a member of different groups, so the value of this attributes is a comma-delimited list of groups. At least one group is required to create a node. The new created group names should not be prefixed with "__" as this token has been preserved as the internal group name.
Expand All @@ -33,7 +33,7 @@ Key Attributes
+--------------------------+----------------------+-----------------------------------+
| Arch and Machine Type | OS | valid netboot options |
+==========================+======================+===================================+
| x86, x86_64 | ALL | pxe, xnba |
| x86, x86_64 | ALL | pxe, xnba, grub2 |
+--------------------------+----------------------+-----------------------------------+
| ppc64 | <=rhel6, <=sles11.3 | yaboot |
+--------------------------+----------------------+-----------------------------------+
Expand All @@ -42,6 +42,8 @@ Key Attributes
| ppc64le NonVirtualize | ALL | petitboot |
+--------------------------+----------------------+-----------------------------------+
| ppc64le PowerKVM Guest | ALL | grub2,grub2-http,grub2-tftp |
+--------------------------+----------------------+-----------------------------------+
| aarch64 | >=el8 | grub2 |
+-------------------------------------------------+-----------------------------------+

* postscripts:
Expand Down
6 changes: 4 additions & 2 deletions docs/source/guides/admin-guides/references/man1/genimage.1.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,9 @@ Besides prompting for some parameter values, the \ **genimage**\ command takes

If \ **-**\ **-onlyinitrd**\ is specified, genimage only regenerates the initrd for a stateless image to be used for a diskless install.

The \ **genimage**\ command must be run on a system that is the same architecture and same distro with same major release version as the nodes it will be used on. If the management node is not the same architecture or same distro level, copy the contents of
The \ **genimage**\ command must be run on a system that is the same architecture and same distro with same major release version as the nodes it will be used on.
For different architectures, \ **genimage**\ will try to use QEMU emulation if qemu-user-static is installed and configured via systemd-binfmt.
If this does not work or the management node is not the same distro level, copy the contents of
/opt/xcat/share/xcat/netboot/<os> to a system that is the proper architecture, and mount /install from
the management node to that system. Then change directory to /opt/xcat/share/xcat/netboot/<os> and run ./genimage.

Expand All @@ -70,7 +72,7 @@ OPTIONS

\ **-a**\ \ *arch*\

The hardware architecture of this node: ppc64le, x86_64, ppc64, x86, ia64, etc. If omitted, the current hardware architecture will be used.
The hardware architecture of this node: ppc64le, x86_64, ppc64, x86, ia64, aarch64 etc. If omitted, the current hardware architecture will be used.



Expand Down
3 changes: 2 additions & 1 deletion docs/source/guides/admin-guides/references/man5/noderes.5.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,11 +56,12 @@ noderes Attributes:
.. code-block:: perl

Arch OS valid netboot options
x86, x86_64 ALL pxe, xnba
x86, x86_64 ALL pxe, xnba, grub2
ppc64 <=rhel6, <=sles11.3 yaboot
ppc64 >=rhels7, >=sles11.4 grub2,grub2-http,grub2-tftp
ppc64le NonVirtualize ALL petitboot
ppc64le PowerKVM Guest ALL grub2,grub2-http,grub2-tftp
aarch64 >=el8 grub2



Expand Down
2 changes: 1 addition & 1 deletion docs/source/guides/admin-guides/references/man5/nodetype.5.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ nodetype Attributes:

\ **arch**\

The hardware architecture of this node. Valid values: x86_64, ppc64, x86, ia64.
The hardware architecture of this node. Valid values: x86_64, ppc64, x86, ia64, aarch64.



Expand Down
2 changes: 1 addition & 1 deletion docs/source/guides/admin-guides/references/man5/osimage.5.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ osimage Attributes:

\ **osarch**\

The hardware architecture of this node. Valid values: x86_64, ppc64, x86, ia64.
The hardware architecture of this node. For netboot/statelite images, QEMU emulation for non-native architectures is used if qemu-user-static is installed and configured via systemd-binfmt. Valid values: x86_64, ppc64, x86, ia64, aarch64.



Expand Down
5 changes: 3 additions & 2 deletions docs/source/guides/admin-guides/references/man7/group.7.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ group Attributes:

\ **arch**\ (nodetype.arch)

The hardware architecture of this node. Valid values: x86_64, ppc64, x86, ia64.
The hardware architecture of this node. Valid values: x86_64, ppc64, x86, ia64, aarch64.



Expand Down Expand Up @@ -591,11 +591,12 @@ group Attributes:
.. code-block:: perl

Arch OS valid netboot options
x86, x86_64 ALL pxe, xnba
x86, x86_64 ALL pxe, xnba, grub2
ppc64 <=rhel6, <=sles11.3 yaboot
ppc64 >=rhels7, >=sles11.4 grub2,grub2-http,grub2-tftp
ppc64le NonVirtualize ALL petitboot
ppc64le PowerKVM Guest ALL grub2,grub2-http,grub2-tftp
aarch64 >=el8 grub2



Expand Down
Loading