feat(datahub-gms): Enable autoscaling via HPA

[7onn]

Summary

datahub-project/datahub#11761
The company where I work for, started crashing datahub-gms during Snowflake ingestion, and I thought it would be handy to have an autoscaler for this workload. Hence, this pull request.

How to test it?

Edit the values.yaml of both datahub chart, and datahub-gms subchart. Make sure to enable datahub-gms.hpa.enabled and global.datahub_standalone_consumers_enabled. Then, navige to the subchart folder and:

cd charts/datahub/subcharts/datahub-gms
helm template test . --values values.yaml --values ../../values.yaml | yq '. | select(.kind == "HorizontalPodAutoscaler")'

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable)

[7onn]

the root chart has a similar template. I had to copy this so I could helm template from within the subchart folder.

[david-leifker]

This looks good to me. I would be interested to know how this works in production, can GMS scale quick enough to help with load spikes. I can see this being helpful for long running ingestion runs for sure.

Thank you!

[7onn]

This looks good to me. I would be interested to know how this works in production, can GMS scale quick enough to help with load spikes. I can see this being helpful for long running ingestion runs for sure.

Thank you!

Hi David, first of all, thanks a lot for your time reviewing my contribution. I highly appreciate it.

In regards to production, I suppose the default setting isn't the most helpful thing with targetCPUUtilizationPercentage: 100 as this would require the app to be under some spike for some time and perhaps even get throttled for a while which could cause the health check to fail and the pod to restart. What I had planned was to release this with targetCPUUtilizationPercentage: 60 to scale out precociously and not even let GMS get throttled by Kubernetes.