feat: add configuration options for Kafka consumer groups

[Masterchen09]

This PR adds configurable Kafka consumer groups to the Helm chart. With the exception of one consumer group, it is already possible today to configure the consumer groups via the extra environment variables (extraEnvs), unfortunately one of the environment variables (DATAHUB_UPGRADE_HISTORY_KAFKA_CONSUMER_GROUP_ID) is already present in the corresponding Deployment and can therefore not specified via the extra environment variables. Nonetheless similar to the configurable Kafka topics it would be great to have dedicated options for the consumer groups too.

Additionally this PR already uses three new environment variables added by another PR in the datahub-actions repository to also configure the consumer groups used by the default action pipelines. Of course the other PR has to be merged first and there must be a new release for the datahub-actions image, which must be used by the Helm chart so this will have any effect (this PR expects that this new release will be version v0.0.16, therefore this PR should not be merged until there is a new release!).

For the default values I have used the currently defined default values e.g. defined
here or here (I have simply searched for "KAFKA_CONSUMER_GROUP_ID" to get all consumer groups resp. the corresponding environment variables).

DATAHUB_UPGRADE_HISTORY_KAFKA_CONSUMER_GROUP_ID is a bit special in here: The three pods (datahub-gms, mae-consumer and mce-consumer) which are using this environment variable have different values defined, therefore there are three "subkeys" to configure each pod individually. Additionally the previous default value for each pod was defined using the release name of the chart, to be consistent with previous versions this logic is kept and the default value "next" to the Kafka listener (e.g., here) is not used.

This PR also solves #319.

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable)

[github-actions]

This PR is stale. We will close it in 30 days if there is no comment or activity. If you want feedback but not able to get it on github please head to #contribute channel in slack at https://slack.datahubproject.io.

[Masterchen09]

To make sure that this PR is not automatically closed - here’s a comment.

[github-actions]

This PR is stale. We will close it in 30 days if there is no comment or activity. If you want feedback but not able to get it on github please head to #contribute channel in slack at https://slack.datahubproject.io.

[Masterchen09]

To make sure that this PR is not automatically closed - here’s a comment...again...

@darnaut Here's a reminder regarding this PR. :-)

[github-actions]

This PR is stale. We will close it in 30 days if there is no comment or activity. If you want feedback but not able to get it on github please head to #contribute channel in slack at https://slack.datahubproject.io.

[oelazzou]

Hello, it is possible to push this PR, the fact of being able to configure prefixes allows me to solve problems on kafka which requires prefix namespace. @darnaut thank you

[schoeberlt]

Hi, we also have this issue so it would be really great to merge this

[geoHeil]

Hi I also would need this functionality for my organization

[geoHeil]

LGTM