hosted-engine --deploy fails, "this is probably a bug"

[1848]

oVirt 4.5.4
OS: Centos stream 8 and Centos stream 9

I am trying to deploy the hosted engine with
hosted-engine --deploy --4

After the step where I set the gateway address, the script crashes with:

[ INFO ] Checking available network interfaces:
[ ERROR ] b'[WARNING]: Skipping plugin (/usr/share/ovirt-hosted-engine-\n'
[ ERROR ] b'setup/he_ansible/callback_plugins/2_ovirt_logger.py), cannot load: cannot\n'
[ ERROR ] b"import name 'Callable' from 'collections'\n"
[ ERROR ] b'(/usr/lib64/python3.11/collections/init.py)\n'
[ ERROR ] b"ERROR! Unexpected Exception, this is probably a bug: cannot import name 'Callable' from 'collections' (/usr/lib64/python3.11/collections/init.py)\n"
[ ERROR ] Failed to execute stage 'Environment customization': Failed executing ansible-playbook
[ INFO ] Stage: Clean up
[ INFO ] Cleaning temporary resources
[ ERROR ] b'[WARNING]: Skipping plugin (/usr/share/ovirt-hosted-engine-\n'
[ ERROR ] b'setup/he_ansible/callback_plugins/2_ovirt_logger.py), cannot load: cannot\n'
[ ERROR ] b"import name 'Callable' from 'collections'\n"
[ ERROR ] b'(/usr/lib64/python3.11/collections/init.py)\n'
[ ERROR ] b"ERROR! Unexpected Exception, this is probably a bug: cannot import name 'Callable' from 'collections' (/usr/lib64/python3.11/collections/init.py)\n"
[ ERROR ] Failed to execute stage 'Clean up': Failed executing ansible-playbook
[ INFO ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20230522102503.conf'
[ INFO ] Stage: Pre-termination
[ INFO ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed
Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20230522102455-imqfjz.log

centos stream 8 or 9 makes not difference.

Steps I did after OS install:

• dnf module enable virt:rhel (not needed on stream 9)
• dnf update
• dnf install centos-release-ovirt45
• dnf install ovirt-hosted-engine-setup
• hosted-engine --deploy --4

Is there anything I am missing?
Please let me know if you need the whole log file.

[1848]

Update:
I solved this one with changing
import Callable from collections
to
from collections.abc import Callable
in /usr/share/ovirt-hosted-engine-setup/he_ansible/callback_plugins/2_ovirt_logger.py but I am running into the next problem:

The ipaddr filter requires python's netaddr be installed on the ansible controller.

It seems that there is a problem with the python dependencies. When installing the ovirt-hosted-engine-setup, python3.11 is installed too. It seems that there are some modules missing.
I solved this by installing python3.11-pip and running pip3.11 install netaddr but I ran into the same error when deploying the hosted engine because the module was missing there too.

[1848]

Update:
Managed to install the host and the engine.
Had to manually install jmespath and netaddr with pip3.11 on the host.
When deploying the engine vm itself, I had to do the same there before the engine-setup starts.

[aviallon]

I can confirm this bug still exists

hosted-engine --deploy --4 --6

[ INFO  ] Stage: Initializing
[ INFO  ] Stage: Environment setup
          During customization use CTRL-D to abort.
          Continuing will configure this host for serving as hypervisor and will create a local VM with a running engine.
          The locally running engine will be used to configure a new storage domain and create a VM there.
          At the end the disk of the local VM will be moved to the shared storage.
          Are you sure you want to continue? (Yes, No)[Yes]:
          Configuration files:
          Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20230824150418-lnzws4.log
          Version: otopi-1.10.3 (otopi-1.10.3-1.el9)
[ INFO  ] Stage: Environment packages setup
[ INFO  ] Stage: Programs detection
[ INFO  ] Stage: Environment setup (late)
[ INFO  ] Stage: Environment customization
         
          --== STORAGE CONFIGURATION ==--
         
         
          --== HOST NETWORK CONFIGURATION ==--
         
          Please indicate the gateway IP address [192.168.1.254]:
[ INFO  ] Checking available network interfaces:
[ ERROR ] b'[WARNING]: Skipping plugin (/usr/share/ovirt-hosted-engine-\n'
[ ERROR ] b'setup/he_ansible/callback_plugins/2_ovirt_logger.py), cannot load: cannot\n'
[ ERROR ] b"import name 'Callable' from 'collections'\n"
[ ERROR ] b'(/usr/lib64/python3.11/collections/__init__.py)\n'
[ ERROR ] b"ERROR! Unexpected Exception, this is probably a bug: cannot import name 'Callable' from 'collections' (/usr/lib64/python3.11/collections/__init__.py)\n"
[ ERROR ] Failed to execute stage 'Environment customization': Failed executing ansible-playbook
[ INFO  ] Stage: Clean up
[ INFO  ] Cleaning temporary resources
[ ERROR ] b'[WARNING]: Skipping plugin (/usr/share/ovirt-hosted-engine-\n'
[ ERROR ] b'setup/he_ansible/callback_plugins/2_ovirt_logger.py), cannot load: cannot\n'
[ ERROR ] b"import name 'Callable' from 'collections'\n"
[ ERROR ] b'(/usr/lib64/python3.11/collections/__init__.py)\n'
[ ERROR ] b"ERROR! Unexpected Exception, this is probably a bug: cannot import name 'Callable' from 'collections' (/usr/lib64/python3.11/collections/__init__.py)\n"
[ ERROR ] Failed to execute stage 'Clean up': Failed executing ansible-playbook
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20230824150423.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20230824150418-lnzws4.log

/etc/os-release

NAME="Rocky Linux"
VERSION="9.2 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.2"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.2 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2032-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.2"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.2"

dnf list --installed ovirt*

dnf list --installed ovirt*
Installed Packages
ovirt-ansible-collection.noarch                                          3.1.2-1.el9                                        @centos-ovirt45
ovirt-host.x86_64                                                        4.5.0-3.el9                                        @centos-ovirt45
ovirt-host-dependencies.x86_64                                           4.5.0-3.el9                                        @centos-ovirt45
ovirt-hosted-engine-ha.noarch                                            2.5.0-1.el9                                        @centos-ovirt45
ovirt-hosted-engine-setup.noarch                                         2.7.0-1.el9                                        @centos-ovirt45
ovirt-imageio-client.x86_64                                              2.5.0-1.el9                                        @centos-ovirt45
ovirt-imageio-common.x86_64                                              2.5.0-1.el9                                        @centos-ovirt45
ovirt-imageio-daemon.x86_64                                              2.5.0-1.el9                                        @centos-ovirt45
ovirt-openvswitch.noarch                                                 2.17-1.el9                                         @centos-ovirt45
ovirt-openvswitch-ipsec.noarch                                           2.17-1.el9                                         @centos-ovirt45
ovirt-openvswitch-ovn.noarch                                             2.17-1.el9                                         @centos-ovirt45
ovirt-openvswitch-ovn-common.noarch                                      2.17-1.el9                                         @centos-ovirt45
ovirt-openvswitch-ovn-host.noarch                                        2.17-1.el9                                         @centos-ovirt45
ovirt-provider-ovn-driver.noarch                                         1.2.36-1.el9                                       @centos-ovirt45
ovirt-python-openvswitch.noarch                                          2.17-1.el9                                         @centos-ovirt45
ovirt-vmconsole.noarch                                                   1.0.9-1.el9                                        @centos-ovirt45
ovirt-vmconsole-host.noarch                                              1.0.9-1.el9                                        @centos-ovirt45

[PatDubois]

Holy cow. The 'collections.abc' bug is still there. Is there anyway this can be patched soon ? Can I help do so ?