EMQX 5, Azure AKS and K8s Operator (v2.1.0) - override bootstrapconfig

[klemen-df]

Hi,

there is a tutorial (here, click!) but is for emqx-ee and for emqx4 (and an older operator).

The instructions here and here say that I need to configure :

•     # If the EMQX cluster is deployed behind HAProxy or Nginx, and you need to get the real source IP address and port of the client, you need to open this configuration.
  

•     listener.tcp.external.proxy_protocol: "on"
  

•     # Use the client certificate to override the value of the Username field
  

•     listener.tcp.external.peer_cert_as_username: "cn"
  

In the v2 of the operator I see that I can override bootstrapConfig (click!) but this means that I have to also include default values that are already written in example? Is there a way just to add these two parameters?

[Rory-Z]

this means that I have to also include default values that are already written in example?

No, the EMQX Operator will merge default value in bootstrap config, So you can fill in only what you care about
Just like this:

apiVersion: apps.emqx.io/v2alpha1
kind: EMQX
metadata:
  name: emqx
spec:
  image: emqx/emqx:5.0.17
  bootstrapConfig: |
    listeners.tcp.default {
      proxy_protocol = true
      peer_cert_as_username = cn
    }

The EMQX Operator only enables port 1883 (MQTT protocol) by default, if you want to enable other listening ports (SSL/WS/WSS/QUIC) you need to fill in the appropriate configuration.
More info: https://www.emqx.io/docs/en/v5.0/configuration/configuration-manual.html#root-config-keys

[klemen-df]

Hi,

I (and probably someone else as well) use Terraform.

This is my flow :

• I install emxq5 operator with helm

resource "helm_release" "emqx5" {
  name       = "emqx5"
  namespace  = "default"
  repository = "https://repos.emqx.io/charts"
  chart      = "emqx-operator"
  version    = "2.1.0"

  depends_on = [module.cert_manager]
}

• I apply (terraform apply)
• I create a CRD

resource "kubernetes_manifest" "this_emqx5" {
  manifest = {
    "apiVersion" = "apps.emqx.io/v2alpha1"
    "kind"       = "EMQX"
    "metadata" = {
      "name"      = "emqx"
      "namespace" = var.emqx.namespace
    }
    "spec" = {
      "image"           = "emqx/emqx:5.0.17"
    }
  }

  depends_on = [
    helm_release.emqx5
  ]
}

• I decide to override the bootstrapConfig with just the delta of parameters

resource "kubernetes_manifest" "this_emqx5" {
  manifest = {
    "apiVersion" = "apps.emqx.io/v2alpha1"
    "kind"       = "EMQX"
    "metadata" = {
      "name"      = "emqx"
      "namespace" = var.emqx.namespace
    }
    "spec" = {
      "image"           = "emqx/emqx:5.0.17"
      "bootstrapConfig" = <<EOF
      {
        listeners.tcp.default {
          proxy_protocol = true
          #peer_cert_as_username = "cn"
        }
      }
      EOF
    }
  }

  depends_on = [
    helm_release.emqx5
  ]
}

• Now terraform want's to delete the old config and fails :

      ~ object   = {
          ~ spec       = {
              ~ bootstrapConfig          = <<-EOT
                  - {node:{data_dir:data, etc_dir:etc}, cluster:{discovery_strategy:dns, dns:{record_type:srv, name:"emqx-headless.default.svc.cluster.local"}}, dashboard:{listeners:{http:{bind:18083}}, default_username:admin, default_password:public}, listeners:{tcp:{default:{bind:"0.0.0.0:1883", max_connections:1024000}}}}
                  +       {
                  +         listeners.tcp.default {
                  +           proxy_protocol = true
                  +           #peer_cert_as_username = "cn"
                  +         }
                  +       }
                EOT
                # (7 unchanged elements hidden)
            }
            # (3 unchanged elements hidden)
        }

• and fails

kubernetes_manifest.this_emqx5: Modifying...
╷
│ Error: API response status: Failure
│
│   with kubernetes_manifest.this_emqx5,
│   on emqx5.tf line 18, in resource "kubernetes_manifest" "this_emqx5":
│   18: resource "kubernetes_manifest" "this_emqx5" {
│
│ admission webhook "validator.apps.emqx.io" denied the request: bootstrap config cannot be updated, old bootstrap config: {node:{data_dir:data, etc_dir:etc}, cluster:{discovery_strategy:dns, dns:{record_type:srv, name:"emqx-headless.default.svc.cluster.local"}},
│ dashboard:{listeners:{http:{bind:18083}}, default_username:admin, default_password:public}, listeners:{tcp:{default:{bind:"0.0.0.0:1883", max_connections:1024000}}}}, new bootstrap config:       {
│         listeners.tcp.default {
│           proxy_protocol = true
│           #peer_cert_as_username = "cn"
│         }
│       }

If I override the bootstrapConfig with full configuration then all is good.

Probably not a bug in Terraform but just some incompatibility with this operator

[Rory-Z]

Hi, @KlemenDanfoss The bootstrapConfig cannot be updated, so you need to set the right content at the beginning, if you want change EMQX config when EMQX running, you can via EMQX Dashboard

[klemen-df]

Oh, ok, thanks.

[Rory-Z]

Hello, the EMQX Operator 2.2.0 and EMQX 5.1 is released, now can update EMQX's config via spec.config.data in apps.emqx.io/v2beta1 EMQX