Details on the expected variables for running ods-ci tests in the docs

[GunaKKIBM]

The readme doesn't explain about the variables expected - https://github.com/red-hat-data-services/ods-ci/tree/master?tab=readme-ov-file

It would be helpful if there are more details provided on:

1. What are these test users and AWS credentials are necessary - what is it referring to exactly? and from which resource on the cluster should we extract the same? https://github.com/red-hat-data-services/ods-ci/blob/master/ods_ci/test-variables.yml.example
2. I would want to understand more on what the users (TEST_USER.NAME, etc) here, is this users of type user.openshift.io/v1 resource in openshift? I understand these values are used for testing, but I am unclear about where these values should be extracted from.
3. The details of test cases which need these variables.

[GunaKKIBM]

We are trying to run these tests on our openshift cluster, where RHODS operator, notebooks, etc is installed. Hence, we would need a better understanding of the same. Could we have call scheduled for a KT session to understand these tests and test parameters better? Anytime between 6 pm - 9 pm IST on 17th July would be convenient.

[bdattoma]

hey @GunaKKIBM !

What are these test users and AWS credentials are necessary

Test users are used to log in the OCP cluster and interact with OCP and RHOAI resources. In RHOAI there is also a distinction between RHOAI admin users and RHOAI users. We configure our test clusters before running the test (automatically). You can either use your own user Identity Provider or can re-use this https://github.com/red-hat-data-services/ods-ci/blob/master/ods_ci/docs/ODS-CI-IMAGE-README.md#user-configuration-json-file (SET_ENVIRONMENT = 1 when running the container image)

AWS are used for different tests which need to store artifacts/inputs in a S3-like object storage (e.g., model serving, pipelines, etc)

I would want to understand more on what the users (TEST_USER.NAME, etc) here, is this users of type user.openshift.io/v1 resource in openshift?

Yes they are. We use htpasswd and LDAP identity provider as source for the users

The details of test cases which need these variables.

What kind of details you looking for? A list of the tests?

https://github.com/red-hat-data-services/ods-ci/tree/master?tab=readme-ov-file

could you pls review this link? It gives empty set of issues. Did you mean https://github.com/red-hat-data-services/ods-ci/blob/master/README.md?

[bdattoma]

We are trying to run these tests on our openshift cluster, where RHODS operator, notebooks, etc is installed. Hence, we would need a better understanding of the same. Could we have call scheduled for a KT session to understand these tests and test parameters better? Anytime between 6 pm - 9 pm IST on 17th July would be convenient.

we can arrange something, I'll take care

[bdattoma]

update: we had a meeting and clarified some doubts.
We'll work together to make things working fine on your end.

[GunaKKIBM]

I ran tests with and without providing the test variables (test user name and password, cos credentials). I see the tests pass in both the cases. Elaborating here:

Note: Outbound traffic is enabled on this cluster.
Without test user names and password, cos credentials. Though I see Errors which says - Certain variables are not defined.
All the test cases pass. Ideally, this isn't expected, correct me if I am wrong

With test user names and password, cos credentials. All the test cases pass

[GunaKKIBM]

Going through one of the test cases:

Data-Science-Pipelines-Kfp
Based on what I understand, it is creating datasciencepipelinesapplications Custom Resource and then checking whether it is getting created or not. While the test suite was running, I monitored the cluster, I don't see any CR getting created newly. Yet the testcase passed. I think there's a bug in validation of this created CR.

Note: A CR was already created on the cluster manually by us. But, that shouldn't affect the test cases.

[GunaKKIBM]

Could we please get an update on this issue?

[GunaKKIBM]

Update:

The tests triggered inside the pod take hardly few seconds to run. The ods-ci pod is going to completed state quickly. The reports generated say all tests have passed. Ideally, this isn't expected. Also, I see the test case for creating datasciencepipelinesapplications has passed, but I monitored the events and this resource, nothing got created, the test case should have failed.

[GunaKKIBM]

Update:

After checking the values that's supposed to be passed against ROBOT_EXTRA_ARGS in pod yaml file. Passed the following value instead of dry-run.

- name: ROBOT_EXTRA_ARGS
   value: "-L DEBUG --variablefile /tmp/ods-ci-test-variables/test-variables.yaml"

The tests are running against the cluster now.

[bdattoma]

Update:

After checking the values that's supposed to be passed against ROBOT_EXTRA_ARGS in pod yaml file. Passed the following value instead of dry-run.

- name: ROBOT_EXTRA_ARGS
   value: "-L DEBUG --variablefile /tmp/ods-ci-test-variables/test-variables.yaml"

The tests are running against the cluster now.

I was out of office last week and couldn't read you

If you pass --dryrun option, it will not actually execute the tests but will just check the tests don't have syntax or library import issues. More details here

[GunaKKIBM]

Here are the test results - attached
test-without-dryon.tar.gz

[bdattoma]

Here are the test results - attached test-without-dryon.tar.gz

I'll ping in Slack about it