Skip to content

Commit

Permalink
Merge pull request #7257 from Obihoernchen/aarch64
Browse files Browse the repository at this point in the history 
Initial aarch64 support, cross arch stateless images with qemu-user-static, grub2 for x86
  • Loading branch information
@Obihoernchen
Obihoernchen authored Jul 25, 2024
2 parents c41a2d4 + 597fea5 commit c8cb1d6
Show file tree
Hide file tree
Showing 26 changed files with 169 additions and 85 deletions.
5 changes: 4 additions & 1 deletion buildcore.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -371,6 +371,9 @@ if [ "$OSNAME" != "AIX" ]; then
if [ $? -ne 0 ]; then FAILEDRPMS="$FAILEDRPMS xCAT-genesis-scripts-x86_64"; fi
./makerpm xCAT-genesis-scripts ppc64 "$EMBED"
if [ $? -ne 0 ]; then FAILEDRPMS="$FAILEDRPMS xCAT-genesis-scripts-ppc64"; fi
# Do not build xCAT-genesis-scripts-aarch64 yet
#./makerpm xCAT-genesis-scripts aarch64 "$EMBED"
#if [ $? -ne 0 ]; then FAILEDRPMS="$FAILEDRPMS xCAT-genesis-scripts-aarch64"; fi
if [ "$FAILEDRPMS" = "$ORIGFAILEDRPMS" ]; then # all succeeded
rm -f $DESTDIR/xCAT-genesis-scripts*rpm
rm -f $SRCDIR/xCAT-genesis-scripts*rpm
Expand All @@ -392,7 +395,7 @@ for rpmname in xCAT xCATsn; do
./makerpm $rpmname "$EMBED"
if [ $? -ne 0 ]; then FAILEDRPMS="$FAILEDRPMS $rpmname"; fi
else
for arch in x86_64 ppc64 ppc64le s390x; do
for arch in x86_64 ppc64 ppc64le s390x aarch64; do
if [ "$rpmname" = "xCAT-OpenStack" -a "$arch" != "x86_64" ] || [ "$rpmname" = "xCAT-OpenStack-baremetal" -a "$arch" != "x86_64" ] ; then continue; fi # only bld openstack for x86_64 for now
./makerpm $rpmname $arch "$EMBED"
if [ $? -ne 0 ]; then FAILEDRPMS="$FAILEDRPMS $rpmname-$arch"; fi
Expand Down
2 changes: 1 addition & 1 deletion buildlocal.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ echo "This is an $OSNAME system"
$CURDIR/makerpm $rpmname
if [ $? -ne 0 ]; then FAILEDRPMS="$FAILEDRPMS $rpmname"; fi
else
for arch in x86_64 ppc64 s390x; do
for arch in x86_64 ppc64 s390x aarch64; do
$CURDIR/makerpm $rpmname $arch
if [ $? -ne 0 ]; then FAILEDRPMS="$FAILEDRPMS $rpmname-$arch"; fi
done
Expand Down
22 changes: 22 additions & 0 deletions docs/source/advanced/mixed_cluster/building_stateless_images.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,28 @@ In a homogeneous cluster, the management node is the same hardware architecture

The issues arises in a heterogeneous cluster, where the management node is running a different level operating system *or* hardware architecture as the compute nodes in which to deploy the image. The ``genimage`` command that builds stateless images depends on various utilities provided by the base operating system and needs to be run on a node with the same hardware architecture and *major* Operating System release as the nodes that will be booted from the image.

When running xCAT >= 2.17 on EL >= 8 based management node with x86_64 architecture, qemu-user-static can be used to cross-build ppc64* and aarch64 osimages. Therefore, you don't need to build images on systems with the target architecture anymore.

Cross-build ppc64*/aarch64 stateless/statelite image on x86_64 management node
------------------------------------------------------------------------------

#. Download qemu-user-static binaries for ppc64le and/or aarch64: ::

wget https://github.com/multiarch/qemu-user-static/releases/latest/download/qemu-ppc64le-static -P /usr/bin
wget https://github.com/multiarch/qemu-user-static/releases/latest/download/qemu-aarch64-static -P /usr/bin
chmod 755 /usr/bin/qemu-*-static

#. Configure systemd-binfmt accordingly: ::

wget https://raw.githubusercontent.com/qemu/qemu/master/scripts/qemu-binfmt-conf.sh
bash qemu-binfmt-conf.sh --systemd 'ALL' --qemu-path '/usr/bin' --qemu-suffix '-static' --persistent 'yes'
systemctl restart systemd-binfmt.service

#. Make sure ``osimage.osarch`` of your image is set to your target architecture eg. ``ppc64le`` or ``aarch64``.

Now, ``genimage`` will use systemd-binfmt and the qemu-user-static binary to build the image for the specified architecture.


Same Operating System, Different Architecture
---------------------------------------------

Expand Down
2 changes: 1 addition & 1 deletion docs/source/advanced/mixed_cluster/index.rst
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Mixed Cluster
=============

The concept of mixed cluster support in xCAT is managing cluster which contain different hardware architectures. Most commonly, this is usually a cluster consisting of both Power and x86 systems.
The concept of mixed cluster support in xCAT is managing cluster which contain different hardware architectures. Most commonly, this is usually a cluster consisting of both Power, ARM and x86 systems.

.. toctree::
:maxdepth: 2
Expand Down
77 changes: 39 additions & 38 deletions docs/source/advanced/mixed_cluster/support_matrix.rst
View file
Original file line number Diff line number Diff line change
@@ -1,49 +1,50 @@
Support Matrix
==============

+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| | RHEL | SLES | RHEL | SLES | Ubuntu | RHEL | SLES | Ubuntu |
| | ppc64 | ppc64 | x86_64 | x86_64 | x86_64 | ppc64le | ppc64le | ppc64el |
| | CN | CN | CN | CN | CN | CN | CN | CN |
+=========+=========+=========+=========+=========+=========+=========+=========+=========+
| RHEL | | | | | | | | |
| ppc64 | yes | yes | yes | yes | yes | yes | yes | yes |
| MN/SN | | | [1]_ | [1]_ | [1]_ | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| SLES | | | | | | | | |
| ppc64 | yes | yes | yes | yes | yes | yes | yes | yes |
| MN/SN | | | [1]_ | [1]_ | [1]_ | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| RHEL | | | | | | | | |
| x86_64 | yes | yes | yes | yes | yes | yes | yes | yes |
| MN/SN | [4]_ | [4]_ | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| SLES | | | | | | | | |
| x86_64 | yes | yes | yes | yes | yes | yes | yes | yes |
| MN/SN | [4]_ | [4]_ | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| Ubuntu | | | | | | | | |
| x86_64 | yes | yes | yes | yes | yes | yes | yes | yes |
| MN/SN | [5]_ | [5]_ | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| RHEL | | | | | | | | |
| ppc64le | yes | yes | yes | yes | yes | yes | yes | yes |
| MN/SN | [2]_ | [2]_ | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| SLES | | | | | | | | |
| ppc64le | no | no | yes | yes | yes | yes | yes | yes |
| MN/SN | | | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| Ubuntu | | | | | | | | |
| ppc64el | yes | yes | yes | yes | yes | yes | yes | yes |
| MN/SN | [3]_ | [3]_ | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+
+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| | RHEL | SLES | RHEL | SLES | Ubuntu | RHEL | SLES | Ubuntu | RHEL | SLES | Ubuntu |
| | ppc64 | ppc64 | x86_64 | x86_64 | x86_64 | ppc64le | ppc64le | ppc64el | aarch64 | aarch64 | aarch64 |
| | CN | CN | CN | CN | CN | CN | CN | CN | CN | CN | CN |
+=========+=========+=========+=========+=========+=========+=========+=========+=========+=========+=========+=========+
| RHEL | | | | | | | | | | | |
| ppc64 | yes | yes | yes | yes | yes | yes | yes | yes | no | no | no |
| MN/SN | | | [1]_ | [1]_ | [1]_ | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| SLES | | | | | | | | | | | |
| ppc64 | yes | yes | yes | yes | yes | yes | yes | yes | no | no | no |
| MN/SN | | | [1]_ | [1]_ | [1]_ | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| RHEL | | | | | | | | | | | |
| x86_64 | yes | yes | yes | yes | yes | yes | yes | yes | yes | no | no |
| MN/SN | [4]_ | [4]_ | | | | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| SLES | | | | | | | | | | | |
| x86_64 | yes | yes | yes | yes | yes | yes | yes | yes | yes | no | no |
| MN/SN | [4]_ | [4]_ | | | | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| Ubuntu | | | | | | | | | | | |
| x86_64 | yes | yes | yes | yes | yes | yes | yes | yes | yes | no | no |
| MN/SN | [5]_ | [5]_ | | | | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| RHEL | | | | | | | | | | | |
| ppc64le | yes | yes | yes | yes | yes | yes | yes | yes | no | no | no |
| MN/SN | [2]_ | [2]_ | | | | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| SLES | | | | | | | | | | | |
| ppc64le | no | no | yes | yes | yes | yes | yes | yes | no | no | no |
| MN/SN | | | | | | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
| Ubuntu | | | | | | | | | | | |
| ppc64el | yes | yes | yes | yes | yes | yes | yes | yes | no | no | no |
| MN/SN | [3]_ | [3]_ | | | | | | | | | |
+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+


Notes:

* The support statements refers to hardware control, operating system (os) provisioning, and general purpose system management where we do not see any obvious problems with the indicated combination.

* For diskless mixed cluster support, the initial diskless image must be created on a node running the target operating system version and architecture. see :doc:`/advanced/mixed_cluster/building_stateless_images` for more details.
* For xCAT < 2.17 diskless mixed cluster support, the initial diskless image must be created on a node running the target operating system version and architecture. On xCAT >= 2.17 with x86 based management nodes QEMU emulation can be used for different architectures (eg. ppc64 or aarch64) if qemu-user-static is installed and configured via systemd-binfmt. See :doc:`/advanced/mixed_cluster/building_stateless_images` for more details.

.. rubric:: Footnotes

Expand Down
6 changes: 4 additions & 2 deletions docs/source/guides/admin-guides/basic_concepts/xcat_object/node.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Key Attributes
The operating system deployed on this node. Valid values: AIX, rhels*, rhelc*, rhas*, centos*, SL*, fedora*, sles* (where * is the version #)

* arch:
The hardware architecture of this node. Valid values: x86_64, ppc64, x86, ia64.
The hardware architecture of this node. Valid values: x86_64, ppc64, x86, ia64, aarch64.

* groups:
Usually, there are a set of nodes with some attributes in common, xCAT admin can define a node group containing these nodes, so that the management task can be issued against the group instead of individual nodes. A node can be a member of different groups, so the value of this attributes is a comma-delimited list of groups. At least one group is required to create a node. The new created group names should not be prefixed with "__" as this token has been preserved as the internal group name.
Expand All @@ -33,7 +33,7 @@ Key Attributes
+--------------------------+----------------------+-----------------------------------+
| Arch and Machine Type | OS | valid netboot options |
+==========================+======================+===================================+
| x86, x86_64 | ALL | pxe, xnba |
| x86, x86_64 | ALL | pxe, xnba, grub2 |
+--------------------------+----------------------+-----------------------------------+
| ppc64 | <=rhel6, <=sles11.3 | yaboot |
+--------------------------+----------------------+-----------------------------------+
Expand All @@ -42,6 +42,8 @@ Key Attributes
| ppc64le NonVirtualize | ALL | petitboot |
+--------------------------+----------------------+-----------------------------------+
| ppc64le PowerKVM Guest | ALL | grub2,grub2-http,grub2-tftp |
+--------------------------+----------------------+-----------------------------------+
| aarch64 | >=el8 | grub2 |
+-------------------------------------------------+-----------------------------------+

* postscripts:
Expand Down
6 changes: 4 additions & 2 deletions docs/source/guides/admin-guides/references/man1/genimage.1.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,9 @@ Besides prompting for some parameter values, the \ **genimage**\ command takes

If \ **-**\ **-onlyinitrd**\ is specified, genimage only regenerates the initrd for a stateless image to be used for a diskless install.

The \ **genimage**\ command must be run on a system that is the same architecture and same distro with same major release version as the nodes it will be used on. If the management node is not the same architecture or same distro level, copy the contents of
The \ **genimage**\ command must be run on a system that is the same architecture and same distro with same major release version as the nodes it will be used on.
For different architectures, \ **genimage**\ will try to use QEMU emulation if qemu-user-static is installed and configured via systemd-binfmt.
If this does not work or the management node is not the same distro level, copy the contents of
/opt/xcat/share/xcat/netboot/<os> to a system that is the proper architecture, and mount /install from
the management node to that system. Then change directory to /opt/xcat/share/xcat/netboot/<os> and run ./genimage.

Expand All @@ -70,7 +72,7 @@ OPTIONS

\ **-a**\ \ *arch*\

The hardware architecture of this node: ppc64le, x86_64, ppc64, x86, ia64, etc. If omitted, the current hardware architecture will be used.
The hardware architecture of this node: ppc64le, x86_64, ppc64, x86, ia64, aarch64 etc. If omitted, the current hardware architecture will be used.



Expand Down
3 changes: 2 additions & 1 deletion docs/source/guides/admin-guides/references/man5/noderes.5.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -56,11 +56,12 @@ noderes Attributes:
.. code-block:: perl
Arch OS valid netboot options
x86, x86_64 ALL pxe, xnba
x86, x86_64 ALL pxe, xnba, grub2
ppc64 <=rhel6, <=sles11.3 yaboot
ppc64 >=rhels7, >=sles11.4 grub2,grub2-http,grub2-tftp
ppc64le NonVirtualize ALL petitboot
ppc64le PowerKVM Guest ALL grub2,grub2-http,grub2-tftp
aarch64 >=el8 grub2
Expand Down
2 changes: 1 addition & 1 deletion docs/source/guides/admin-guides/references/man5/nodetype.5.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ nodetype Attributes:

\ **arch**\

The hardware architecture of this node. Valid values: x86_64, ppc64, x86, ia64.
The hardware architecture of this node. Valid values: x86_64, ppc64, x86, ia64, aarch64.



Expand Down
2 changes: 1 addition & 1 deletion docs/source/guides/admin-guides/references/man5/osimage.5.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ osimage Attributes:

\ **osarch**\

The hardware architecture of this node. Valid values: x86_64, ppc64, x86, ia64.
The hardware architecture of this node. For netboot/statelite images, QEMU emulation for non-native architectures is used if qemu-user-static is installed and configured via systemd-binfmt. Valid values: x86_64, ppc64, x86, ia64, aarch64.



Expand Down
5 changes: 3 additions & 2 deletions docs/source/guides/admin-guides/references/man7/group.7.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ group Attributes:

\ **arch**\ (nodetype.arch)

The hardware architecture of this node. Valid values: x86_64, ppc64, x86, ia64.
The hardware architecture of this node. Valid values: x86_64, ppc64, x86, ia64, aarch64.



Expand Down Expand Up @@ -591,11 +591,12 @@ group Attributes:
.. code-block:: perl
Arch OS valid netboot options
x86, x86_64 ALL pxe, xnba
x86, x86_64 ALL pxe, xnba, grub2
ppc64 <=rhel6, <=sles11.3 yaboot
ppc64 >=rhels7, >=sles11.4 grub2,grub2-http,grub2-tftp
ppc64le NonVirtualize ALL petitboot
ppc64le PowerKVM Guest ALL grub2,grub2-http,grub2-tftp
aarch64 >=el8 grub2
Expand Down
Loading

0 comments on commit c8cb1d6

Please sign in to comment.