Add openstack charm func test runner #220

dosaboy · 2024-08-28T16:41:41Z

Also includes tool to identify func test targets from the charm.

openstack/tools/charmed_openstack_functest_runner.sh

brianphaley · 2024-08-28T20:18:31Z

openstack/tools/charmed_openstack_functest_runner.sh

+FIP_MIN=$(ipcalc $CIDR| awk '$1=="HostMin:" {print $2}')
+FIP_MIN_ABC=${FIP_MIN%.*}
+FIP_MIN_D=${FIP_MIN##*.}
+FIP_MIN=${FIP_MIN_ABC}.$(($FIP_MIN_D + 64))


This turns out to be gateway_ip + 64 since it's always .1 by default, and as long as this is a /25 or larger it's fine, smaller and it will be greater than the max. It's only until they get the gate fixed anyways...

brianphaley · 2024-08-28T20:21:42Z

openstack/tools/charmed_openstack_functest_runner.sh

+# We need to set TEST_JUJU3 as well as the constraints file
+# Ref: https://github.com/openstack-charmers/zaza/blob/e96ab098f00951079fccb34bc38d4ae6ebb38606/setup.py#L47
+export TEST_JUJU3=1
+export NET_ID=$(openstack network show net_${OS_USERNAME}-psd -f value -c id)


Just making sure this wasn't supposed to be -psd-extra network

ah well spotted, this is the "external" network used for fips so for us thats *-psd-extra

xtrusia · 2024-08-30T01:22:43Z

While Im testing mysql-router charm zaza test, I realized that I need below env variable

export TEST_ZAZA_BUG_LP1987332=1

related code is here

https://github.com/openstack-charmers/zaza/blob/bef1bdbac7ee15ae742466641c65bcab363c093e/zaza/utilities/juju.py#L511

if I didn't set it, there was python array index error.

just FYI

xtrusia · 2024-08-30T05:19:01Z

Is there way to use PR test ? like below in commit msg

func-test-pr: openstack-charmers/zaza-openstack-tests#1252

dosaboy · 2024-08-30T08:34:21Z

While Im testing mysql-router charm zaza test, I realized that I need below env variable

export TEST_ZAZA_BUG_LP1987332=1

related code is here

https://github.com/openstack-charmers/zaza/blob/bef1bdbac7ee15ae742466641c65bcab363c093e/zaza/utilities/juju.py#L511

if I didn't set it, there was python array index error.

just FYI

I beleive this is not needed with juju3 but can add it anyway for those wanted to stick with j2

dosaboy · 2024-08-30T08:34:52Z

Is there way to use PR test ? like below in commit msg

func-test-pr: openstack-charmers/zaza-openstack-tests#1252

I have a need for this myself so would like to find a way but do not yet know how

dosaboy · 2024-08-30T11:34:03Z

With the current version I can now do a successful build and deploy but hit the following error once the tests start:

2024-08-30 11:29:25 [ERROR] Model default_alias (zaza-e680c75cf4a9)
Traceback (most recent call last):
  File "/home/ubuntu/jammy/octavia/src/.tox/func-target/bin/functest-run-suite", line 8, in <module>
    sys.exit(main())
  File "/home/ubuntu/jammy/octavia/src/.tox/func-target/lib/python3.10/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 405, in main
    func_test_runner(
  File "/home/ubuntu/jammy/octavia/src/.tox/func-target/lib/python3.10/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 313, in func_test_runner
    run_env_deployment(env_deployment, keep_model=preserve_model,
  File "/home/ubuntu/jammy/octavia/src/.tox/func-target/lib/python3.10/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 174, in run_env_deployment
    configure.configure(
  File "/home/ubuntu/jammy/octavia/src/.tox/func-target/lib/python3.10/site-packages/zaza/charm_lifecycle/configure.py", line 58, in configure
    run_configure_list(functions)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/ubuntu/jammy/octavia/src/.tox/func-target/lib/python3.10/site-packages/zaza/charm_lifecycle/configure.py", line 44, in run_configure_list
    utils.get_class(func)()
  File "/home/ubuntu/jammy/octavia/src/.tox/func-target/lib/python3.10/site-packages/zaza/openstack/charm_tests/neutron/setup.py", line 157, in basic_overcloud_network
    undercloud_and_charm_setup(limit_gws=limit_gws)
  File "/home/ubuntu/jammy/octavia/src/.tox/func-target/lib/python3.10/site-packages/zaza/openstack/charm_tests/neutron/setup.py", line 107, in undercloud_and_charm_setup
    network.setup_gateway_ext_port(network_config,
  File "/home/ubuntu/jammy/octavia/src/.tox/func-target/lib/python3.10/site-packages/zaza/openstack/configure/network.py", line 315, in setup_gateway_ext_port
    openstack_utils.configure_gateway_ext_port(
  File "/home/ubuntu/jammy/octavia/src/.tox/func-target/lib/python3.10/site-packages/zaza/openstack/utilities/openstack.py", line 1125, in configure_gateway_ext_port
    net_id = get_admin_net(neutronclient)['id']
TypeError: 'NoneType' object is not subscriptable

dosaboy · 2024-08-30T12:25:08Z

I have hopefully fixed the above issue which might have been caused by the fact that had forgotten to export CIDR_EXT but I also notice that both the zaza and zaza-openstack-tests repos have duplicate code for e.g. https://github.com/openstack-charmers/zaza/blob/master/zaza/utilities/generic.py#L92 https://github.com/openstack-charmers/zaza-openstack-tests/blob/master/zaza/openstack/utilities/generic.py#L153 which are slightly different in which env vars they look for and it is not clear which is actually used when a test is run so I am now setting both kinds.

dosaboy · 2024-08-30T15:23:47Z

works now but new issue is that nova-compute vms use a flavor too small to be able to boot any vms

dosaboy · 2024-08-30T15:46:34Z

works now but new issue is that nova-compute vms use a flavor too small to be able to boot any vms

2024-08-30 13:40:04.351 230827 INFO nova.scheduler.manager [None req-f758d68f-0f19-461c-9068-cb4e96e272e7 184d6a2c973043d8b5ac655217b4c3cf ac3a5e14642c4a9dad8fcde1b0c3530e - - c8b548e494dd4827aa0418706da3f6a7 c8b548e494dd4827aa0418706da3f6a7] Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up.

dosaboy · 2024-08-31T12:59:06Z

works now but new issue is that nova-compute vms use a flavor too small to be able to boot any vms

2024-08-30 13:40:04.351 230827 INFO nova.scheduler.manager [None req-f758d68f-0f19-461c-9068-cb4e96e272e7 184d6a2c973043d8b5ac655217b4c3cf ac3a5e14642c4a9dad8fcde1b0c3530e - - c8b548e494dd4827aa0418706da3f6a7 c8b548e494dd4827aa0418706da3f6a7] Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up.

I had to make the following change to nova-compute constraints to get things working:

-    constraints: mem=7168M
+    constraints: "root-disk=80G mem=8G"

dosaboy · 2024-08-31T13:06:43Z

Putting this up for final review since I am now able to successfully run tests.

rodrigogansobarbieri · 2024-09-02T15:56:47Z

I tried running the script but I got the same error:

Juju.errors.JujuConnectionError: juju server-version 3.5.3 not supported

I changed my juju snap version to 2.9 and pointed the credentials to my juju 2.9 controller and it worked. It is not running the tests. I guess whether it supports juju 3 or not varies from charm to charm. I am running nova-compute charm tests.

One thing I noticed is that the timeout is set to 3600 instead of 7200 as Trent suggested in his notes: export TEST_DEPLOY_TIMEOUT=7200

Also one thing I don't like is that there is currently to option to specify the bundle we want to run. For example, for my nova-compute tests I am running for the 2023.2 branch, so it started running jammy-antelope, and then later it will run jammy-bobcat. Assuming jammy-bobcat fails for some reason and I have to run it again. If I have already collected a successful output from jammy-antelope I would not want to let it run again, but instead specify that I want only to run jammy-bobcat.

rodrigogansobarbieri

-1 for lacking an option to specify the bundle to run (example: run just jammy-bobcat instead of the whole suite that would include jammy-antelope, jammy-bobcat and so on).

dosaboy · 2024-09-03T09:30:56Z

-1 for lacking an option to specify the bundle to run (example: run just jammy-bobcat instead of the whole suite that would include jammy-antelope, jammy-bobcat and so on).

done.

lathiat

These are nits, and not really related to the tool's core functionality. But thought they are worth adding/fixing.

lathiat · 2024-09-04T08:34:39Z

openstack/tools/charmed_openstack_functest_runner.sh

+
+usage () { echo "USAGE: --func-test-target | --func-test-pr | --skip-build"; }
+
+while (($# > 0))


If you call the script without any arguments and you're not inside a charm directory it eventually fails. It should check for the current directory being a charm somehow.

i will fix this as a followup patch but have put info in the help message about how to run the tool

lathiat · 2024-09-04T08:35:20Z

openstack/tools/charmed_openstack_functest_runner.sh

+FUNC_TEST_TARGET=
+SKIP_BUILD=false
+
+usage () { echo "USAGE: --func-test-target | --func-test-pr | --skip-build"; }


Ideally this would make it more clear how to use the tool, e.g. that you have to call it from within a charm directory, etc.

i have put in a much nicer help message

lathiat · 2024-09-04T08:37:53Z

openstack/tools/charmed_openstack_functest_runner.sh

+    read -p "Destroy model and run next test? [ENTER]"
+    # cleanup before next run
+    model=`juju list-models| egrep -o "^zaza-\S+"|tr -d '*'`
+    juju destroy-model --no-prompt $model --force --no-wait --destroy-storage


This will fail if you have multiple matching models. Ideally it would also print the name of all the models it will destroy before it does, in case you had created another model with a name starting with zaza.

You can also just tell zaza to destroy the model itself automatically by passing an argument, perhaps we should use that by default and then have an option to switch to manual cleanup behaviour for trying to debug the tests?

do you happen to know what the argument is?

we can fix this as a future patch

lathiat · 2024-09-05T08:01:40Z

Tested working OK for me with glance-simplestreams-sync master

Supports running Openstack charm functional tests manually in a similar way to OSCI. Includes tool to identify func test targets from the charm.

brianphaley reviewed Aug 28, 2024

View reviewed changes

dosaboy force-pushed the add-charm-openstack-func-test-runner branch 7 times, most recently from 2f92996 to bd35bb2 Compare August 29, 2024 10:14

dosaboy force-pushed the add-charm-openstack-func-test-runner branch 3 times, most recently from 081c7e1 to 26d7a43 Compare August 30, 2024 10:45

dosaboy force-pushed the add-charm-openstack-func-test-runner branch from 26d7a43 to 7168957 Compare August 30, 2024 12:03

dosaboy force-pushed the add-charm-openstack-func-test-runner branch from 7168957 to 4a41ec5 Compare August 31, 2024 12:55

dosaboy force-pushed the add-charm-openstack-func-test-runner branch from 4a41ec5 to 2d7f740 Compare August 31, 2024 13:06

dosaboy marked this pull request as ready for review August 31, 2024 13:06

rodrigogansobarbieri closed this Sep 2, 2024

rodrigogansobarbieri reopened this Sep 2, 2024

rodrigogansobarbieri requested changes Sep 2, 2024

View reviewed changes

dosaboy force-pushed the add-charm-openstack-func-test-runner branch from 2d7f740 to 14a7994 Compare September 3, 2024 08:56

dosaboy force-pushed the add-charm-openstack-func-test-runner branch from 14a7994 to 62bf204 Compare September 3, 2024 09:21

lathiat suggested changes Sep 4, 2024

View reviewed changes

Add openstack charm func test runner

faa48ba

Supports running Openstack charm functional tests manually in a similar way to OSCI. Includes tool to identify func test targets from the charm.

dosaboy force-pushed the add-charm-openstack-func-test-runner branch from 62bf204 to faa48ba Compare September 5, 2024 10:56

dosaboy merged commit b7b76dc into canonical:main Sep 5, 2024
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add openstack charm func test runner #220

Add openstack charm func test runner #220

dosaboy commented Aug 28, 2024

brianphaley Aug 28, 2024

brianphaley Aug 28, 2024

dosaboy Aug 30, 2024

xtrusia commented Aug 30, 2024

xtrusia commented Aug 30, 2024

dosaboy commented Aug 30, 2024

dosaboy commented Aug 30, 2024

dosaboy commented Aug 30, 2024

dosaboy commented Aug 30, 2024

dosaboy commented Aug 30, 2024

dosaboy commented Aug 30, 2024

dosaboy commented Aug 31, 2024

dosaboy commented Aug 31, 2024

rodrigogansobarbieri commented Sep 2, 2024

rodrigogansobarbieri left a comment

dosaboy commented Sep 3, 2024

lathiat left a comment

lathiat Sep 4, 2024

dosaboy Sep 5, 2024

lathiat Sep 4, 2024

dosaboy Sep 5, 2024

lathiat Sep 4, 2024

dosaboy Sep 5, 2024

dosaboy Sep 5, 2024

lathiat commented Sep 5, 2024 •

edited

Loading


		usage () { echo "USAGE: --func-test-target \| --func-test-pr \| --skip-build"; }

		while (($# > 0))

Add openstack charm func test runner #220

Add openstack charm func test runner #220

Conversation

dosaboy commented Aug 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xtrusia commented Aug 30, 2024

xtrusia commented Aug 30, 2024

dosaboy commented Aug 30, 2024

dosaboy commented Aug 30, 2024

dosaboy commented Aug 30, 2024

dosaboy commented Aug 30, 2024

dosaboy commented Aug 30, 2024

dosaboy commented Aug 30, 2024

dosaboy commented Aug 31, 2024

dosaboy commented Aug 31, 2024

rodrigogansobarbieri commented Sep 2, 2024

rodrigogansobarbieri left a comment

Choose a reason for hiding this comment

dosaboy commented Sep 3, 2024

lathiat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lathiat commented Sep 5, 2024 • edited Loading

lathiat commented Sep 5, 2024 •

edited

Loading