Add details for flags/settings that will migrate over to MKE 4 for Custom Kube Config

[moshiurH]

The purpose of this issue ticket is to improve the writeup for the migration for Custom Kube Config. Either this writeup can go in https://mirantis.github.io/mke-docs/docs/migrate-from-mke-3/ or https://mirantis.github.io/mke-docs/docs/configuration/kubernetes/. The development epic is based on https://mirantis.jira.com/browse/BOP-515.

The following should migrate over from MKE 3 to 4 with respect to Custom Kube Config (confirmed as per the testing effort):

1. CustomKubeAPIServerFlags
2. CustomKubeSchedulerFlags
3. CustomKubeControllerManagerFlags
4. MKE 3 Node/Worker Profiles
5. MKE 3 Manager Settings (there are about 20~30 toml values which were coded for)

---

MKE 3 Toml Snippets relevant for points 1 to 3:

  custom_kube_api_server_flags = [
      "--enable-admission-plugins=PodSecurity"
  ]
  custom_kube_scheduler_flags = [
    "--authentication-token-webhook-cache-ttl=20s",
    "--authorization-webhook-cache-authorized-ttl=20s",
    "--v=2",
    "--tls-min-version=VersionTLS13"
  ]
  custom_kube_controller_manager_flags = [
    "--attach-detach-reconcile-sync-period=1m30s",
    "--authentication-token-webhook-cache-ttl=20s",
    "--cluster-name=my_mke"
  ]

All possible options and values for this section can be found in:
https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/
https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/
https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/

---

MKE 3 Toml Snippet relevant for point 4:
https://docs.mirantis.com/mke/3.7/ops/administer-cluster/custom-kubelet-profiles/add-kubelet-node-profiles.html

---

MKE 3 Toml Snippet relevant for point 5 (see https://docs.mirantis.com/mke/3.7/ops/administer-cluster/configure-an-mke-cluster/configuration-options.html):
etcd_storage_quota = "100Mi"
kube_api_server_profiling_enabled = true
kube_controller_manager_profiling_enabled = true
kube_scheduler_profiling_enabled = true
kube_scheduler_bind_to_all = true
cipher_suites_for_kube_api_server = ["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"]
cipher_suites_for_kubelet = ["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"]
cipher_suites_for_etcd_server = ["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"]
kube_api_server_auditing = true
kube_api_server_audit_log_maxage = 30
kube_api_server_audit_log_maxbackup = 10
kube_api_server_audit_log_maxsize = 100
kube_api_server_request_timeout = "30s"
kube_manager_terminated_pod_gc_threshold = 100
k8s_event_rate_limit = 5
k8s_always_pull_images_ac_enabled = true
unauthenticated_http2_dos_mitigation = true
cipher_suites_for_kubelet = "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
kubelet_max_pods = 110
kubelet_pods_per_core = 10
kubelet_data_root = "/var/lib/kubelet"
windows_kubelet_data_root = "C:\var\lib\kubelet"
kubelet_event_record_qps = 5
custom_kubelet_flags = ["--cgroup-driver=systemd", "--rotate-certificates=true"]
custom_kubelet_flags_profiles = { "high-performance" = "--cpu-manager-policy=static", "default" = "--memory-manager-policy=none" }
and
cipher_suites_for_kubelet = "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
kubelet_max_pods = 110
kubelet_pods_per_core = 10
kubelet_data_root = "/var/lib/kubelet"
windows_kubelet_data_root = "C:\var\lib\kubelet"
kubelet_event_record_qps = 5
custom_kubelet_flags = ["--cgroup-driver=systemd", "--rotate-certificates=true"]
custom_kubelet_flags_profiles = { "high-performance" = "--cpu-manager-policy=static", "default" = "--memory-manager-policy=none" }

Point 5 confirmed using: https://github.com/MirantisContainers/mke/pull/270, https://github.com/MirantisContainers/mke/pull/273, https://github.com/Mirantis/orca/blob/master/types/node_config.go, and https://docs.mirantis.com/mke/3.7/ops/administer-cluster/configure-an-mke-cluster/configuration-options.html

[moshiurH]

@nsteph Please take a look and suggest any improvements if you notice any for the migration aspects of MKE 4 Custom Kube Param. Thank you.

[nsteph]

Thanks, Moshiur. I'll be adding a sub-page to the migration docs to cover this BOP-1572