Create Openshift ProjectRequest attempts to patch

[erikgb]

SUMMARY

We have a role to create/delete Openshift projects that has been working fine for many Ansible releases. But when attempting to upgrade to Ansible 7, it suddenly fails when attempting to create a new project. The Openshift project API is very confusing, and not idempotent, but this used to work. To create an Openshift project, a user must CREATE a ProjectRequest, which will make the cluster create a new Project (Namespace). This may only happen once, and a user is not allowed to modify (PATCH) any of the project resources.

With Ansible 7, the attempt to create the project fails with the following error (example):

TASK [openshift_project : Create ProjectRequest] *******************************
fatal: [kafka-rev-ansible-7-g5cfyt.stas-test.mycompany.com]: FAILED! => changed=false 
  msg: 'Failed to patch object: b''{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"projectrequests.project.openshift.io \\"kafka-rev-ansible-7-g5cfyt\\" is forbidden: User \\"system:serviceaccount:kafka:gitlab\\" cannot patch resource \\"projectrequests\\" in API group \\"project.openshift.io\\" at the cluster scope","reason":"Forbidden","details":{"name":"kafka-rev-ansible-7-g5cfyt","group":"project.openshift.io","kind":"projectrequests"},"code":403}\n'''
  reason: Forbidden

ISSUE TYPE

• Bug Report

COMPONENT NAME

kubernetes.core.k8s

ANSIBLE VERSION

ansible [core 2.14.5]
  config file = None
  configured module search path = ['/home/ansible/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.11/site-packages/ansible
  ansible collection location = /home/ansible/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.11.3 (main, May  4 2023, 05:53:32) [GCC 10.2.1 20210110] (/usr/local/bin/python)
  jinja version = 3.1.2
  libyaml = True

COLLECTION VERSION

# /usr/local/lib/python3.11/site-packages/ansible_collections
Collection      Version
--------------- -------
kubernetes.core 2.4.0

CONFIGURATION

ANSIBLE_FORCE_COLOR(env: ANSIBLE_FORCE_COLOR) = True
CONFIG_FILE() = None
DEFAULT_HOST_LIST(env: ANSIBLE_INVENTORY) = ['/builds/kafka/provisioning/k8s/inventories/review.yml']
DEFAULT_LOAD_CALLBACK_PLUGINS(env: ANSIBLE_LOAD_CALLBACK_PLUGINS) = True
DEFAULT_ROLES_PATH(env: ANSIBLE_ROLES_PATH) = ['/builds/kafka/provisioning/k8s/roles']
DEFAULT_STDOUT_CALLBACK(env: ANSIBLE_STDOUT_CALLBACK) = yaml
HOST_KEY_CHECKING(env: ANSIBLE_HOST_KEY_CHECKING) = False
INTERPRETER_PYTHON(env: ANSIBLE_PYTHON_INTERPRETER) = auto

OS / ENVIRONMENT

N/A

STEPS TO REPRODUCE

Pre-requirements: An Openshift cluster with self-provisioner access. The project/namespace that we attempt to create must NOT already exist.

- name: Create ProjectRequest
  kubernetes.core.k8s:
    api_version: project.openshift.io/v1
    kind: ProjectRequest
    name: "{{ kubernetes_namespace }}"
    resource_definition:
      description: "{{ kubernetes_namespace_description }}"
      displayName: "{{ kubernetes_namespace_display_name }}"

EXPECTED RESULTS

The project request is CREATED in the Openshift API without error.

ACTUAL RESULTS

The attempt to create the project request fails with error (example): msg: 'Failed to patch object: b''{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"projectrequests.project.openshift.io \\"kafka-rev-ansible-7-g5cfyt\\" is forbidden: User \\"system:serviceaccount:kafka:gitlab\\" cannot patch resource \\"projectrequests\\" in API group \\"project.openshift.io\\" at the cluster scope","reason":"Forbidden","details":{"name":"kafka-rev-ansible-7-g5cfyt","group":"project.openshift.io","kind":"projectrequests"},"code":403}\n''' reason: Forbidden. Note: the project is actually created by this failing task.

TASK [openshift_project : Create ProjectRequest] *******************************
task path: /builds/kafka/provisioning/k8s/roles/openshift_project/tasks/main.yml:37
redirecting (type: filter) ansible.builtin.json_query to community.general.json_query
<kafka-rev-ansible-7-g5cfyt.stas-test.mycompany.com> ESTABLISH LOCAL CONNECTION FOR USER: ansible
<kafka-rev-ansible-7-g5cfyt.stas-test.mycompany.com> EXEC /bin/sh -c 'echo ~ansible && sleep 0'
<kafka-rev-ansible-7-g5cfyt.stas-test.mycompany.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/ansible/.ansible/tmp `"&& mkdir "` echo /home/ansible/.ansible/tmp/ansible-tmp-1684583776.8747976-142-179954988027677 `" && echo ansible-tmp-1684583776.8747976-142-179954988027677="` echo /home/ansible/.ansible/tmp/ansible-tmp-1684583776.8747976-142-179954988027677 `" ) && sleep 0'
Loading collection cloud.common from /usr/local/lib/python3.11/site-packages/ansible_collections/cloud/common
Using module file /usr/local/lib/python3.11/site-packages/ansible_collections/kubernetes/core/plugins/modules/k8s.py
<kafka-rev-ansible-7-g5cfyt.stas-test.mycompany.com> PUT /home/ansible/.ansible/tmp/ansible-local-96xec_0y7q/tmp9_a4lciu TO /home/ansible/.ansible/tmp/ansible-tmp-1684583776.8747976-142-179954988027677/AnsiballZ_k8s.py
<kafka-rev-ansible-7-g5cfyt.stas-test.mycompany.com> EXEC /bin/sh -c 'chmod u+x /home/ansible/.ansible/tmp/ansible-tmp-1684583776.8747976-142-179954988027677/ /home/ansible/.ansible/tmp/ansible-tmp-1684583776.8747976-142-179954988027677/AnsiballZ_k8s.py && sleep 0'
<kafka-rev-ansible-7-g5cfyt.stas-test.mycompany.com> EXEC /bin/sh -c '/usr/bin/python3 /home/ansible/.ansible/tmp/ansible-tmp-1684583776.8747976-142-179954988027677/AnsiballZ_k8s.py && sleep 0'
<kafka-rev-ansible-7-g5cfyt.stas-test.mycompany.com> EXEC /bin/sh -c 'rm -f -r /home/ansible/.ansible/tmp/ansible-tmp-1684583776.8747976-142-179954988027677/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
  File "/tmp/ansible_kubernetes.core.k8s_payload__jodf5af/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/runner.py", line 68, in run_module
    result = perform_action(svc, definition, module.params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ansible_kubernetes.core.k8s_payload__jodf5af/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/runner.py", line 159, in perform_action
    instance = svc.update(resource, definition, existing)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ansible_kubernetes.core.k8s_payload__jodf5af/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/service.py", line 426, in update
    raise exception
  File "/tmp/ansible_kubernetes.core.k8s_payload__jodf5af/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/service.py", line 413, in update
    k8s_obj = self.patch_resource(
              ^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ansible_kubernetes.core.k8s_payload__jodf5af/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/service.py", line 165, in patch_resource
    raise CoreException(msg) from e
fatal: [kafka-rev-ansible-7-g5cfyt.stas-test.mycompany.com]: FAILED! => changed=false 
  invocation:
    module_args:
      api_key: VALUE_SPECIFIED_IN_NO_LOG_PARAMETER
      api_version: project.openshift.io/v1
      append_hash: false
      apply: false
      ca_cert: null
      client_cert: null
      client_key: null
      context: null
      continue_on_error: false
      delete_options: null
      force: false
      generate_name: null
      host: https://api.stas-test.mycompany.com
      impersonate_groups: null
      impersonate_user: null
      kind: ProjectRequest
      kubeconfig: null
      label_selectors: null
      merge_type: null
      name: kafka-rev-ansible-7-g5cfyt
      namespace: kafka-rev-ansible-7-g5cfyt
      no_proxy: null
      password: null
      persist_config: null
      proxy: null
      proxy_headers: null
      resource_definition:
        apiVersion: project.openshift.io/v1
        description: ''
        displayName: ''
        kind: ProjectRequest
        metadata:
          name: kafka-rev-ansible-7-g5cfyt
          namespace: kafka-rev-ansible-7-g5cfyt
      server_side_apply: null
      src: null
      state: present
      template: null
      username: null
      validate: null
      validate_certs: null
      wait: false
      wait_condition: null
      wait_sleep: 5
      wait_timeout: 120
  msg: 'Failed to patch object: b''{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"projectrequests.project.openshift.io \\"kafka-rev-ansible-7-g5cfyt\\" is forbidden: User \\"system:serviceaccount:kafka:gitlab\\" cannot patch resource \\"projectrequests\\" in API group \\"project.openshift.io\\" at the cluster scope","reason":"Forbidden","details":{"name":"kafka-rev-ansible-7-g5cfyt","group":"project.openshift.io","kind":"projectrequests"},"code":403}\n'''
  reason: Forbidden
PLAY RECAP *********************************************************************
kafka-rev-ansible-7-g5cfyt.stas-test.mycompany.com : ok=1    changed=0    unreachable=0    failed=1    skipped=2    rescued=0    ignored=0   
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

[erikgb]

Somehow creating the project request works if I using community.okd.k8s instead of kubernetes.core.k8s, but I would prefer to avoid this.

[erikgb]

This seems broken since Ansible version 7.4.0 (kubernetes.core collection version 2.4.0). Only major change noted in this release is #481. @gravesm any idea if this could be related?

[erikgb]

I am not a python programmer, but I think the code flow ends up in the wrong branch here:

kubernetes.core/plugins/module_utils/k8s/runner.py

Lines 175 to 195 in 54d8193

	if params.get("apply"):
	instance = svc.apply(resource, definition, existing)
	result["method"] = "apply"
	elif not existing:
	if state == "patched":
	result.setdefault("warnings", []).append(
	"resource 'kind={kind},name={name}' was not found but will not be "
	"created as 'state' parameter has been set to '{state}'".format(
	kind=kind, name=definition["metadata"].get("name"), state=state
	)
	)
	return result
	instance = svc.create(resource, definition)
	result["method"] = "create"
	result["changed"] = True
	elif params.get("force", False):
	instance = svc.replace(resource, definition, existing)
	result["method"] = "replace"
	else:
	instance = svc.update(resource, definition, existing)
	result["method"] = "update"

A regular user can only CREATE projectrequests in the (dumb) Openshift API, and I think the variables used in this if-else do not take that correctly into account...

[gravesm]

Yes, it looks like the refactoring introduced a bug here.

[erikgb]

Thanks for the feedback @gravesm! Is this something that will be fixed? Our upgrade to Ansible 7 is currently blocked by this, and I expect other Openshift users in multi-tenant environments to be affected as well.

[gravesm]

I can't give a date for when this would be fixed. We would certainly consider a PR for this if someone wants to submit one.

[tenstad]

The problem appears to be that a ForbiddenError in K8sService.retrieve always returns the project as existing, meaning a non-existant project is later attempted patched.

https://github.com/ansible-collections/kubernetes.core/blob/main/plugins/module_utils/k8s/service.py#L199-L204

Returning None avoids the error, but would then again cause the following warning:

X was not found, but creating it returned a 409 Conflict error. This can happen if the resource you are creating does not directly create a resource of the same kind

So I would say the issue lies in correctly finding out if the project exists or not.

[erikgb]

Would it be possible to remove the special handling for Openshift custom resources? IMO it does not make sense to adapt a generic collection like kubernetes.core to Openshift. It would be nice if the collection could be used with Openshift - even for tricky resource types as ProjectRequest and Project. But special handling as in the referenced code does not make sense to me.

[erikgb]

The culprit here is the Openshift APIs. There is no way to write idempotent code without tailoring the logic around the (broken) ProjectRequest/Project APIs. This is the RBAC granted as self-provisioner:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: self-provisioner
  annotations:
    openshift.io/description: A user that can request projects.
    rbac.authorization.kubernetes.io/autoupdate: 'true'
rules:
  - verbs:
      - create
    apiGroups:
      - ''
      - project.openshift.io
    resources:
      - projectrequests

Does this collection support state: created? 😉 There seems to be no way to declaratively express a Project resource as present with the RBAC granted to self-provisioners.

$ k auth can-i --list | grep project
projectrequests                                                      []                       []                                                            [create list]
projectrequests.project.openshift.io                                 []                       []                                                            [create list]
projecthelmchartrepositories.helm.openshift.io                       []                       []                                                            [get list update create watch patch delete]
projects                                                             []                       []                                                            [list watch get delete patch update]
projects.project.openshift.io                                        []                       []                                                            [list watch get delete patch update]

[erikgb]

Trying to exploit the Openshift-adoption in the referenced code (#623 (comment)), declaring Project (even if the RBAC to create that resource is no available) as present in the playbook, seems to work using Ansible 5 and 6. On Ansible versions 7 and 8 we're getting an internal error ('''dict'' object has no attribute ''to_dict'''), which seems like a code bug. With verbosity set to 3, this gives the following log (sorry for the garbled output, but we are forced to use yet another "fantastic" tool - GitLab):

TASK [openshift_project : Create/Patch Project] ********************************
�[1;30mtask path: /builds/stas/ansible-roles/openshift_project/tasks/main.yml:18�[0m
�[34m<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: ansible�[0m
�[34m<127.0.0.1> EXEC /bin/sh -c 'echo ~ansible && sleep 0'�[0m
�[34m<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/ansible/.ansible/tmp `"&& mkdir "` echo /home/ansible/.ansible/tmp/ansible-tmp-1694022261.46392-228-215735704715761 `" && echo ansible-tmp-1694022261.46392-228-215735704715761="` echo /home/ansible/.ansible/tmp/ansible-tmp-1694022261.46392-228-215735704715761 `" ) && sleep 0'�[0m
�[34mUsing module file /usr/local/lib/python3.11/site-packages/ansible_collections/kubernetes/core/plugins/modules/k8s.py�[0m
�[34m<127.0.0.1> PUT /home/ansible/.ansible/tmp/ansible-local-179x09r55h5/tmpda3izmu3 TO /home/ansible/.ansible/tmp/ansible-tmp-1694022261.46392-228-215735704715761/AnsiballZ_k8s.py�[0m
�[34m<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/ansible/.ansible/tmp/ansible-tmp-1694022261.46392-228-215735704715761/ /home/ansible/.ansible/tmp/ansible-tmp-1694022261.46392-228-215735704715761/AnsiballZ_k8s.py && sleep 0'�[0m
�[34m<127.0.0.1> EXEC /bin/sh -c '/usr/local/bin/python /home/ansible/.ansible/tmp/ansible-tmp-1694022261.46392-228-215735704715761/AnsiballZ_k8s.py && sleep 0'�[0m
�[34m<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/ansible/.ansible/tmp/ansible-tmp-1694022261.46392-228-215735704715761/ > /dev/null 2>&1 && sleep 0'�[0m
�[31mThe full traceback is:�[0m
�[31m  File "/tmp/ansible_kubernetes.core.k8s_payload_leyh9b3j/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/runner.py", line 68, in run_module�[0m
�[31m    result = perform_action(svc, definition, module.params)�[0m
�[31m             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^�[0m
�[31m  File "/tmp/ansible_kubernetes.core.k8s_payload_leyh9b3j/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/runner.py", line 177, in perform_action�[0m
�[31m    existing = existing.to_dict()�[0m
�[31m               ^^^^^^^^^^^^^^^^�[0m
�[31mfatal: [localhost]: FAILED! => changed=false �[0m
�[31m  invocation:�[0m
�[31m    module_args:�[0m
�[31m      api_key: VALUE_SPECIFIED_IN_NO_LOG_PARAMETER�[0m
�[31m      api_version: project.openshift.io/v1�[0m
�[31m      append_hash: false�[0m
�[31m      apply: false�[0m
�[31m      ca_cert: null�[0m
�[31m      client_cert: null�[0m
�[31m      client_key: null�[0m
�[31m      context: null�[0m
�[31m      continue_on_error: false�[0m
�[31m      delete_options: null�[0m
�[31m      force: false�[0m
�[31m      generate_name: null�[0m
�[31m      host: <REDACTED>�[0m
�[31m      impersonate_groups: null�[0m
�[31m      impersonate_user: null�[0m
�[31m      kind: Project�[0m
�[31m      kubeconfig: null�[0m
�[31m      label_selectors: null�[0m
�[31m      merge_type: null�[0m
�[31m      name: ansible-openshift-project-7-test-recent-ansible�[0m
�[31m      namespace: null�[0m
�[31m      no_proxy: null�[0m
�[31m      password: null�[0m
�[31m      persist_config: null�[0m
�[31m      proxy: null�[0m
�[31m      proxy_headers: null�[0m
�[31m      resource_definition:�[0m
�[31m        apiVersion: project.openshift.io/v1�[0m
�[31m        description: ''�[0m
�[31m        displayName: ''�[0m
�[31m        kind: Project�[0m
�[31m        metadata:�[0m
�[31m          name: ansible-openshift-project-7-test-recent-ansible�[0m
�[31m      server_side_apply: null�[0m
�[31m      src: null�[0m
�[31m      state: present�[0m
�[31m      template: null�[0m
�[31m      username: null�[0m
�[31m      validate: null�[0m
�[31m      validate_certs: null�[0m
�[31m      wait: false�[0m
�[31m      wait_condition: null�[0m
�[31m      wait_sleep: 5�[0m
�[31m      wait_timeout: 120�[0m
�[31m  msg: '''dict'' object has no attribute ''to_dict'''�[0m

PLAY RECAP *********************************************************************
�[31mlocalhost�[0m                  : �[32mok=1   �[0m changed=0    unreachable=0    �[31mfailed=1   �[0m �[36mskipped=1   �[0m rescued=0    ignored=0

[erikgb]

I can't give a date for when this would be fixed. We would certainly consider a PR for this if someone wants to submit one.

@gravesm Now there's an open PR for some time that could fix this. Please take a look. 🙏