PermissionDeniedException raised when the subscription already exists and the user/service account doesn't have permission to create new ones

[dolfandringa]

I am using rele with GCP. I want to use a minimum permission principle, where I allow subscribers to only subscribe to the topics that I allow them to. For that reason, I don't want to grant the roles/pubsub.editor role at the project level to service accounts, since that allows that service account to create subscriptions and attach them to any topic. Instead I make the subscriptions already in terraform, and attach the right policy to it to allow my service account to update and consume that subscription.

This causes an error in rele though, since rele first tries to create the subscription, and only if it throws an error that it already exists, update an existing one. If the user doesn't have permission to create new subscriptions in the first place, this causes a crash in rele at startup:

2023-10-04T12:54:27.509667Z [debug    ] None                           [rele.management.discover] message=Autodiscovering subs... severity=DEBUG
2023-10-04T12:54:27.509932Z [debug    ] None                           [rele.management.discover] message= * Discovered subs module: simulation_api.home.subs severity=DEBUG
Configuring worker with 1 subscription(s)...
Configuring worker with 1 subscription(s)...
  myapp-home-simulation-response - store_simulation_response
Traceback (most recent call last):
  File "/home/me/myapp/.venv/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 75, in error_remapped_callable
    return callable_(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/myapp/.venv/lib64/python3.11/site-packages/grpc/_channel.py", line 1161, in __call__
    return _end_unary_response_blocking(state, call, False, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/myapp/.venv/lib64/python3.11/site-packages/grpc/_channel.py", line 1004, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.PERMISSION_DENIED
        details = "User not authorized to perform this action."
        debug_error_string = "UNKNOWN:Error received from peer ipv4:142.251.130.10:443 {created_time:"2023-10-04T12:54:31.228021606+00:00", grpc_status:7, grpc_message:"User not authorized to perform this action."}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/me/myapp/manage.py", line 22, in <module>
    main()
  File "/home/me/myapp/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/home/me/myapp/.venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/home/me/myapp/.venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/me/myapp/.venv/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/me/myapp/.venv/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/myapp/.venv/lib/python3.11/site-packages/rele/management/commands/runrele.py", line 30, in handle
    create_and_run(subs, self.config)
  File "/home/me/myapp/.venv/lib/python3.11/site-packages/rele/worker.py", line 172, in create_and_run
    worker.run_forever()
  File "/home/me/myapp/.venv/lib/python3.11/site-packages/rele/worker.py", line 75, in run_forever
    self.setup()
  File "/home/me/myapp/.venv/lib/python3.11/site-packages/rele/worker.py", line 53, in setup
    self._subscriber.update_or_create_subscription(subscription)
  File "/home/me/myapp/.venv/lib/python3.11/site-packages/rele/client.py", line 78, in update_or_create_subscription
    self._create_subscription(subscription_path, topic_path, subscription)
  File "/home/me/myapp/.venv/lib/python3.11/site-packages/rele/client.py", line 116, in _create_subscription
    self._client.create_subscription(request=request)
  File "/home/me/myapp/.venv/lib/python3.11/site-packages/google/pubsub_v1/services/subscriber/client.py", line 680, in create_subscription
    response = rpc(
               ^^^^
  File "/home/me/myapp/.venv/lib/python3.11/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
    return wrapped_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/myapp/.venv/lib/python3.11/site-packages/google/api_core/retry.py", line 366, in retry_wrapped_func
    return retry_target(
           ^^^^^^^^^^^^^
  File "/home/me/myapp/.venv/lib/python3.11/site-packages/google/api_core/retry.py", line 204, in retry_target
    return target()
           ^^^^^^^^
  File "/home/me/myapp/.venv/lib/python3.11/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/myapp/.venv/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 77, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.PermissionDenied: 403 User not authorized to perform this action.

I also found the culprit in rele/client.py:

    def update_or_create_subscription(self, subscription):
        """Handles creating the subscription when it does not exists or updates it
        if the subscription contains any parameter that allows it.

        This makes it easier to deploy a worker and forget about the
        subscription side of things. The subscription must
        have a topic to subscribe to. Which means that the topic must be
        created manually before the worker is started.

        :param subscription: obj :class:`~rele.subscription.Subscription`.
        """
        subscription_path = self._client.subscription_path(
            self._gc_project_id, subscription.name
        )
        topic_path = self._client.topic_path(self._gc_project_id, subscription.topic)

        try:
            self._create_subscription(subscription_path, topic_path, subscription)
        except exceptions.NotFound:
            logger.warning(
                "Cannot subscribe to a topic that does not exist."
                f"Creating {topic_path}..."
            )
            topic = self._create_topic(topic_path)
            logger.info(f"Topic {topic.name} created.")
            self._create_subscription(subscription_path, topic_path, subscription)
        except exceptions.AlreadyExists:
            self._update_subscription(subscription_path, topic_path, subscription)

In the line before last it catches exceptions.AlreadyExists but self._create_subscription can also throw exceptions.PermissionDenied. If both exceptions are caught in that except statement, the issue disappears. So I will create a PR for that. The only workaround for now is to provide the PubSub Editor role at the project level, so it can create subscriptions.