Error on deployment emqx 5.1.0 operator 2.2.0-alpha.2 #876

alexa22013 · 2023-07-23T00:37:48Z

alexa22013
Jul 23, 2023

error in operator:

{"level":"info","ts":"2023-07-23T00:29:52Z","msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"emqx","controllerGroup":"apps.emqx.io","controllerKind":"EMQX","eMQX":{"name":"emqx","namespace":"emqx"},"namespace":"emqx","name":"emqx","reconcileID":"10af23e4-b032-48ec-ae03-54d99dba94c6"}

emqx nodes give these errors:

ReconcilerFailed connection refused

failed to get node statues by API: failed to get API http://10.121.214.248:18083/api/v5/nodes: failed to request API: Get "http://10.121.214.248:18083/api/v5/nodes": dial tcp 10.121.214.248:18083: connect: connection refused

full error in operator

{"level":"info","ts":"2023-07-23T07:49:59Z","msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"emqx","controllerGroup":"apps.emqx.io","controllerKind":"EMQX","eMQX":{"name":"emqx","namespace":"emqx"},"namespace":"emqx","name":"emqx","reconcileID":"95165206-4f3e-4974-8b66-b61066b6eef2"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x15cc4a7]

goroutine 492 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118 +0x1f4
	/go/pkg/mod/github.com/rory-z/[email protected]/config.go:133 +0x1e
github.com/emqx/emqx-operator/apis/apps/v2alpha2.GetListenersServicePorts({0xc000d47500, 0x3293})
	/workspace/apis/apps/v2alpha2/util.go:174 +0xb48
github.com/emqx/emqx-operator/controllers/apps/v2alpha2.(*addSvc).reconcile(0xc0003841f0, {0x1c7c1c0, 0xc000440b40}, 0xc000970000, {0x1c7cf00?, 0xc001e91470?})
	/workspace/controllers/apps/v2alpha2/add_svc.go:35 +0x2d4
github.com/emqx/emqx-operator/controllers/apps/v2alpha2.(*EMQXReconciler).Reconcile(0xc0007a8a20, {0x1c7c1c0, 0xc000440b40}, {{{0xc00069d6b0?, 0x0?}, {0xc00069d6ac?, 0x40dec7?}}})
	/workspace/controllers/apps/v2alpha2/emqx_controller.go:125 +0x5b1
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1c7c118?, {0x1c7c1c0?, 0xc000440b40?}, {{{0xc00069d6b0?, 0x1951440?}, {0xc00069d6ac?, 0xc000574908?}}})
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121 +0xc8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0002dc3c0, {0x1c7c118, 0xc000253090}, {0x1870020?, 0xc000030d40?})
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320 +0x309
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0002dc3c0, {0x1c7c118, 0xc000253090})
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:230 +0x587
I0723 07:50:43.050203       1 request.go:601] Waited for 1.040104117s due to client-side throttling, not priority and fairness, request: GET:https://10.121.224.1:443/apis/monitoring.coreos.com/v1?timeout=32s

This issue is causing livenessProbe and readinessProbe to fail in core nodes, and replicant nodes are not deploying

Anyway to resolve this?

Thanks in advance!

Rory-Z · 2023-07-24T05:41:33Z

Rory-Z
Jul 24, 2023
Maintainer

Hi, @alexa22013 Please try EMQX 5.1.1

3 replies

alexa22013 Jul 24, 2023
Author

Thanks! @Rory-Z for the reply. You mean there is no issue operator?
Also when deploying replicant nodes doesnt come up, and no error logs are seen in the operator, any reason for it? (This happens even when using operator:2.1.4 with emqx:5.1.0)

alexa22013 Jul 24, 2023
Author

@Rory-Z still same error using emqx:5.1.1 and replicants are to deploying

{"level":"error","ts":"2023-07-24T15:56:49Z","msg":"Reconciler error","controller":"emqx","controllerGroup":"apps.emqx.io","controllerKind":"EMQX","eMQX":{"name":"emqx","namespace":"emqx"},"namespace":"emqx","name":"emqx","reconcileID":"53a79b53-0da4-489b-9662-1b1c92cdafc9","error":"failed to get emqx config: failed to get API http://10.121.211.91:18083/api/v5/configs: failed to request API: Get \"http://10.121.211.91:18083/api/v5/configs\": dial tcp 10.121.211.91:18083: connect: connection refused","errorVerbose":"Get \"http://10.121.211.91:18083/api/v5/configs\": dial tcp 10.121.211.91:18083: connect: connection refused\nfailed to request API\ngithub.com/emqx/emqx-operator/internal/requester.(*Requester).Request\n\t/workspace/internal/requester/requester.go:89\ngithub.com/emqx/emqx-operator/controllers/apps/v2alpha2.getEMQXConfigsByAPI\n\t/workspace/controllers/apps/v2alpha2/sync_emqx_config.go:79\ngithub.com/emqx/emqx-operator/controllers/apps/v2alpha2.(*syncConfig).reconcile\n\t/workspace/controllers/apps/v2alpha2/sync_emqx_config.go:46\ngithub.com/emqx/emqx-operator/controllers/apps/v2alpha2.(*EMQXReconciler).Reconcile\n\t/workspace/controllers/apps/v2alpha2/emqx_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598\nfailed to get API http://10.121.211.91:18083/api/v5/configs\ngithub.com/emqx/emqx-operator/controllers/apps/v2alpha2.getEMQXConfigsByAPI\n\t/workspace/controllers/apps/v2alpha2/sync_emqx_config.go:83\ngithub.com/emqx/emqx-operator/controllers/apps/v2alpha2.(*syncConfig).reconcile\n\t/workspace/controllers/apps/v2alpha2/sync_emqx_config.go:46\ngithub.com/emqx/emqx-operator/controllers/apps/v2alpha2.(*EMQXReconciler).Reconcile\n\t/workspace/controllers/apps/v2alpha2/emqx_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598\nfailed to get emqx config\ngithub.com/emqx/emqx-operator/controllers/apps/v2alpha2.(*syncConfig).reconcile\n\t/workspace/controllers/apps/v2alpha2/sync_emqx_config.go:48\ngithub.com/emqx/emqx-operator/controllers/apps/v2alpha2.(*EMQXReconciler).Reconcile\n\t/workspace/controllers/apps/v2alpha2/emqx_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234"}

Rory-Z Nov 1, 2023
Maintainer

Hi @alexa22013 Sorry for delay. Could you please try EMQX Operator 2.2.3 and EMQX 5.3.0 ?

swechencheng · 2023-10-31T16:44:54Z

swechencheng
Oct 31, 2023

Would be nice to see if there is any update on this.
Have tried to use operator 2.2.3 to deploy EMQX 5.1.6 but have the same issue.
Environment GKE 1.26.6-gke.1700

4 replies

Rory-Z Nov 1, 2023
Maintainer

Hi, @rtechencheng Could you please try the latest of the EMQX ? I think it is EMQX 5.3.0

swechencheng Nov 1, 2023

Hi, thanks. Turns out to be a wrong indentation in our emqx.yaml file and some config like persistent_session_store which has been hidden.

Rory-Z Nov 2, 2023
Maintainer

Great, so your problem has been resolved ?

swechencheng Mar 18, 2024

Yes thanks.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error on deployment emqx 5.1.0 operator 2.2.0-alpha.2 #876

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 7 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Error on deployment emqx 5.1.0 operator 2.2.0-alpha.2 #876

alexa22013 Jul 23, 2023

Replies: 2 comments · 7 replies

Rory-Z Jul 24, 2023 Maintainer

alexa22013 Jul 24, 2023 Author

alexa22013 Jul 24, 2023 Author

Rory-Z Nov 1, 2023 Maintainer

swechencheng Oct 31, 2023

Rory-Z Nov 1, 2023 Maintainer

swechencheng Nov 1, 2023

Rory-Z Nov 2, 2023 Maintainer

swechencheng Mar 18, 2024

alexa22013
Jul 23, 2023

Replies: 2 comments 7 replies

Rory-Z
Jul 24, 2023
Maintainer

alexa22013 Jul 24, 2023
Author

alexa22013 Jul 24, 2023
Author

Rory-Z Nov 1, 2023
Maintainer

swechencheng
Oct 31, 2023

Rory-Z Nov 1, 2023
Maintainer

Rory-Z Nov 2, 2023
Maintainer