Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[tempest]Enable live migration testing #585

Merged

Conversation

gibizer
Copy link
Contributor

@gibizer gibizer commented Dec 4, 2023

No description provided.

@openshift-ci openshift-ci bot requested review from frenzyfriday and viroel December 4, 2023 13:03
@gibizer
Copy link
Contributor Author

gibizer commented Dec 4, 2023

/hold for the dependencies to land

@gibizer
Copy link
Contributor Author

gibizer commented Dec 4, 2023

recheck

4 similar comments
@gibizer
Copy link
Contributor Author

gibizer commented Dec 4, 2023

recheck

@gibizer
Copy link
Contributor Author

gibizer commented Dec 4, 2023

recheck

@gibizer
Copy link
Contributor Author

gibizer commented Dec 4, 2023

recheck

@gibizer
Copy link
Contributor Author

gibizer commented Dec 4, 2023

recheck

Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/25c93ffb05994b449780e07cc85724e1

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 23m 44s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 08m 53s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 10m 19s
openstack-operator-tempest-multinode FAILURE in 1h 23m 06s

@gibizer
Copy link
Contributor Author

gibizer commented Dec 5, 2023

recheck now with the EDPM depends on

Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/dcf6e54d94f1476883f40c4e55ff1f20

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 18m 16s
podified-multinode-edpm-deployment-crc FAILURE in 45m 30s
cifmw-crc-podified-edpm-baremetal FAILURE in 44m 50s
openstack-operator-tempest-multinode FAILURE in 50m 01s

@gibizer
Copy link
Contributor Author

gibizer commented Dec 5, 2023

recheck
known issue already WAd in ci-framework

The error was: TypeError: can't intern AnsibleUnsafeText

Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/c9b076d005c640c795cedad131dc81c6

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 12m 43s
podified-multinode-edpm-deployment-crc RETRY_LIMIT in 4s
cifmw-crc-podified-edpm-baremetal RETRY_LIMIT in 4s
openstack-operator-tempest-multinode FAILURE in 1h 53m 50s

@gibizer
Copy link
Contributor Author

gibizer commented Dec 7, 2023

recheck dependencies are now landed

@gibizer
Copy link
Contributor Author

gibizer commented Dec 7, 2023

/unhold

Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/dd3505544bff4192b1c6782de462a321

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 44m 50s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 09m 34s
cifmw-crc-podified-edpm-baremetal FAILURE in 1h 15m 26s
openstack-operator-tempest-multinode FAILURE in 1h 26m 16s

@gibizer
Copy link
Contributor Author

gibizer commented Dec 7, 2023

The live migration test failed with:

nova.exception.MigrationPreCheckError: Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility.

It seem we did not bumped to the nova-operator that added skip_cpu_compare_on_dest = true in openstack-k8s-operators/nova-operator#574 as it is missing from the compute config during the job run https://review.rdoproject.org/zuul/build/b2ab5f0df27c43dfa579cf77874015b9/log/controller/ci-framework-data/logs/192.168.122.100/openstack/config/nova/01-nova.conf

Copy link

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/openstack-k8s-operators/openstack-operator for 585,2684683b348e33d778a9771cc01120ba766357fb

@gibizer
Copy link
Contributor Author

gibizer commented Dec 7, 2023

The live migration test failed with:

nova.exception.MigrationPreCheckError: Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility.

It seem we did not bumped to the nova-operator that added skip_cpu_compare_on_dest = true in openstack-k8s-operators/nova-operator#574 as it is missing from the compute config during the job run https://review.rdoproject.org/zuul/build/b2ab5f0df27c43dfa579cf77874015b9/log/controller/ci-framework-data/logs/192.168.122.100/openstack/config/nova/01-nova.conf

now nova-operator is bumped as wel

Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/82f7157f2b4045a9a6e6031d44b3137c

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 48m 51s
podified-multinode-edpm-deployment-crc RETRY_LIMIT in 3s
cifmw-crc-podified-edpm-baremetal RETRY_LIMIT in 3s
openstack-operator-tempest-multinode FAILURE in 1h 32m 23s

@gibizer
Copy link
Contributor Author

gibizer commented Dec 8, 2023

/hold
Sadly we don't have scenario tests that use live-migration with non-shared storage.

Details: {'code': 400, 'message': 'compute-0.ci-rdo.local is not on shared storage: Shared storage live-migration requires either shared storage or boot-from-volume with no local disks.'}

I will check if we have some non scenario test that we can enable here to test live migration until we have ceph or cinder available in the CI job.

Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/2b4c61842e374d4596a852c5f4ac86f1

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 47m 13s
podified-multinode-edpm-deployment-crc NODE_FAILURE Node request 099-0006758916 failed in 0s
cifmw-crc-podified-edpm-baremetal FAILURE in 41m 04s
openstack-operator-tempest-multinode POST_FAILURE in 1h 29m 57s

Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/09a290110bf84066a783d6fd6703d947

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 51m 12s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 10m 09s
cifmw-crc-podified-edpm-baremetal FAILURE in 1h 14m 37s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 33m 58s

Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/7cb34acdc8724748a32b60cc23a4a925

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 40m 07s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 05m 39s
cifmw-crc-podified-edpm-baremetal RETRY_LIMIT in 12m 54s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 23m 42s

@gibizer
Copy link
Contributor Author

gibizer commented Dec 19, 2023

Live migration test passes now test_live_block_migration[id-1dce86b8-eb04-4c03-a9d8-9c1dc3ee0c7b,multinode] pass

@gibizer
Copy link
Contributor Author

gibizer commented Dec 19, 2023

recheck cifmw-crc-podified-edpm-baremetal RETRY_LIMIT

Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/dfc10f34b9b349d9bec86a93d99e4857

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 38m 06s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 14m 08s
cifmw-crc-podified-edpm-baremetal FAILURE in 22m 12s
openstack-operator-tempest-multinode POST_FAILURE in 1h 21m 56s

@gibizer
Copy link
Contributor Author

gibizer commented Dec 20, 2023

recheck
cleanup failed

Task 
Detach port
  failed running on host 
controller

@gibizer
Copy link
Contributor Author

gibizer commented Dec 20, 2023

/unhold

Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/c9480ce3786147b6b8ef53969e977e5b

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 46m 28s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 12m 27s
cifmw-crc-podified-edpm-baremetal FAILURE in 42m 32s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 29m 46s

@gibizer
Copy link
Contributor Author

gibizer commented Dec 20, 2023

recheck come on baremetal job

Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/864aaf0f1abe4016a5d8c5495a7084fd

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 09m 47s
podified-multinode-edpm-deployment-crc FAILURE in 35m 35s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 10m 18s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 50m 22s

@gibizer
Copy link
Contributor Author

gibizer commented Dec 20, 2023

recheck this is a gamble, each job passed before, but not at the same time

Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/7f7219745bab46558c93b7472c91df39

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 38m 32s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 18m 55s
cifmw-crc-podified-edpm-baremetal RETRY_LIMIT in 33m 32s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 20m 26s

@gibizer
Copy link
Contributor Author

gibizer commented Dec 20, 2023

recheck baremetal job this time :)

Copy link
Contributor

openshift-ci bot commented Jan 4, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dprince, gibizer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved label Jan 4, 2024
@openshift-merge-bot openshift-merge-bot bot merged commit a7920d9 into openstack-k8s-operators:main Jan 4, 2024
7 checks passed
@gibizer gibizer deleted the live-migration branch July 5, 2024 15:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants