Add abitility to set QPS and Burst limits for api client

[Demch1k]

We have a problem in our big kubernetes clusters with 300+ postgres clusters.
When operator starts reconcile it can stack creating new clusters for 10-15 minutes and We can see in the logs messages like that:

I0618 03:13:56.195420 1 request.go:697] Waited for 17.19394453s due to client-side throttling, not priority and fairness, request: POST:https://10.141.64.1:443/apis/apps/v1/namespaces/postgres-ns/statefulsets

Especially in working hours when our developers create new environments.
This PR adds ability to set kubernetes qps and burst limits for client.

[Demch1k]

Any updates ?

[FxKu]

Can you not make these variables also configurable similar to NoDatabaseAccess and NoTeamsAPI. We can use 5 and 10 as default values.

[Demch1k]

@FxKu I have reverted my previous changes because we can't change qps and burst limits for kube client after initialization.
Without kube client we can't obtain crd with limits. Only one way to pass it is used cli args.

[FxKu]

Config changes always require a replacement of the operator pod. Doesn't this initialize the client as well each time?

[Demch1k]

@FxKu I mean that you can't change kubernetes client parameters after client has been initilized.
To be able to read it from operator configs you need initilize kubernetes client first and read custom resource. After that you must reinitilize kubernetes client to apply new values like QPS or Burst but I don't think that is possible. If you know how to do it, help me please with it.

[Demch1k]

@FxKu We have been testing this fix in our k8s cluster with 200+ postgres clusters for 2 weeks now and it really helps.

[Demch1k]

@FxKu any updates?

[FxKu]

Suggested change

	flag.IntVar(&config.KubeQPS, "kubeqps", 5, "Kubernetes api requests per second.")
	flag.IntVar(&config.KubeBurst, "kubeburst", 10, "Kubernetes api requests burst limit.")
	flag.IntVar(&config.KubeQPS, "kubeqps", 50, "Kubernetes api requests per second.")
	flag.IntVar(&config.KubeBurst, "kubeburst", 100, "Kubernetes api requests burst limit.")

I read, that default is 50 and 100. Can we use these values instead? For some reason the owner references e2e test keeps failing and I don't know why.

[FxKu]

Sorry, these were values from kubelet and not REST config. 5 and 10 are probably right. But maybe we can still increase to see if e2e tests succeed.

[Demch1k]

No problem, you can increase it for tests.
But for default values I recommend use smaller like default values because it can affect on api servers.

[FxKu]

ok then better keep the defaults here. Buty any idea, why the owner references check e2e test starts failing with this change. Maybe we add these 2 new config values to the config of our kind cluster, too.

[FxKu]

@FxKu We have been testing this fix in our k8s cluster with 200+ postgres clusters for 2 weeks now and it really helps.

Can you add a chapter in the docs on how to do this (administrator.md is a good place, I think)? I guess many users would be interested.