Increase amount of replicas in running cluster

[arcarmona]

Hello,

Just a quick question, when using EMQx in GKE with EMQx Operator, if I wanted to run some updates on a runnning cluster like, increasing the replicas from 2 to 3 what would be the most effective way to achieve this, reapplying the .yaml with the increased replicas?

Thanks

[Rory-Z]

Yes, you can modify .spec.coreTemplate.spec.replicas to change your emqx core node replicas, also you can modify .spec.replicantTemplate.spec.replicas to change your emqx replicant node replicas

[arcarmona]

Hey @Rory-Z, I had a running cluster and did kubectl apply with the modified .yaml, I was expecting to see the new node spawn within the cluster without interrupting the service however it acted as a blue green deployment and generated both a new node in the running cluster and a new stateful set which later took over.

I had blue green deployment at some point but I thought I removed it by reapplying the .yaml with those lines commented, but as shown by results it might have not been the case. So theoretically should applying the change in replicas work by only adding a node? How can I avoid this blue green behavior if I added it in the .yaml (with type=recreate)?

[arcarmona]

Additional context as to the reason behind this post, I have been attempting to increase the number of replicas from 2 to 3 in my running cluster and end up having only two after CrashLoopBackOff error on one of the nodes because of inconsistent database and failure to merge schema, further details on the logs below:

Se analizó hasta el 17/6/24, 8:35 a.m.. Se analizaron 954.4 KB.
No se encontraron entradas más recientes que coincidan con el filtro actual.
2024-06-17 08:56:49.251 ∅∅∅
EMQX_NODE__DATA_DIR [node.data_dir]: data
2024-06-17 08:56:49.252 ∅∅∅
EMQX_NODE__COOKIE [node.cookie]: ******
2024-06-17 08:56:49.252 ∅∅∅
EMQX_NODE__NAME [node.name]: emqx@emqx-core-79796fcbbb-1.emqx-headless.cloudservices.svc.cluster.local
2024-06-17 08:56:52.112 ∅∅∅
2024-06-17T06:56:52.111502+00:00 [error] Mnesia('emqx@emqx-core-79796fcbbb-1.emqx-headless.cloudservices.svc.cluster.local'): ** ERROR ** mnesia_event got {inconsistent_database, starting_partitioned_network, 'emqx@emqx-core-79796fcbbb-2.emqx-headless.cloudservices.svc.cluster.local'}
2024-06-17 08:56:52.197 ∅∅∅
2024-06-17T06:56:52.196714+00:00 [error] Mnesia('emqx@emqx-core-79796fcbbb-1.emqx-headless.cloudservices.svc.cluster.local'): ** ERROR ** (core dumped to file: "/opt/emqx/MnesiaCore.emqx@emqx-core-79796fcbbb-1.emqx-headless.cloudservices.svc.cluster.local_1718_607412_194808"), ** FATAL ** Failed to merge schema: {aborted,function_clause}
2024-06-17 08:57:02.197 ∅∅∅
2024-06-17T06:57:02.195969+00:00 [error] Generic server mnesia_subscr terminating. Reason: killed. Last message: {'EXIT',<0.2208.0>,killed}. State: {state,<0.2208.0>,#Ref<0.1144114197.2730622979.171884>}.
2024-06-17 08:57:02.197 ∅∅∅
2024-06-17T06:57:02.196003+00:00 [error] Generic server mnesia_monitor terminating. Reason: killed. Last message: {'EXIT',<0.2208.0>,killed}. State: {state,<0.2208.0>,[],[],true,[],undefined,[],[]}.
2024-06-17 08:57:02.197 ∅∅∅
2024-06-17T06:57:02.196504+00:00 [error] crasher: initial call: gen_event:init_it/6, pid: <0.2206.0>, registered_name: mnesia_event, exit: {killed,[{gen_event,terminate_server,4,[{file,"gen_event.erl"},{line,592}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]}, ancestors: [mnesia_sup,<0.2204.0>], message_queue_len: 0, messages: [], links: [], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 28, reductions: 13157; neighbours:
2024-06-17 08:57:02.197 ∅∅∅
2024-06-17T06:57:02.196709+00:00 [error] Generic server mnesia_recover terminating. Reason: killed. Last message: {'EXIT',<0.2208.0>,killed}. State: {state,<0.2208.0>,undefined,undefined,undefined,0,false,true,[]}.
2024-06-17 08:57:02.197 ∅∅∅
2024-06-17T06:57:02.197137+00:00 [error] crasher: initial call: mnesia_recover:init/1, pid: <0.2212.0>, registered_name: mnesia_recover, exit: {killed,[{gen_server,decode_msg,9,[{file,"gen_server.erl"},{line,1049}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]}, ancestors: [mnesia_kernel_sup,mnesia_sup,<0.2204.0>], message_queue_len: 0, messages: [], links: [], dictionary: [], trap_exit: true, status: running, heap_size: 1598, stack_size: 28, reductions: 8430; neighbours:
2024-06-17 08:57:02.198 ∅∅∅
2024-06-17T06:57:02.196420+00:00 [error] crasher: initial call: application_master:init/4, pid: <0.2203.0>, registered_name: [], exit: {{killed,{mnesia_app,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,142}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]}, ancestors: [<0.2202.0>], message_queue_len: 1, messages: [{'EXIT',<0.2204.0>,normal}], links: [<0.2202.0>,<0.2021.0>], dictionary: [], trap_exit: true, status: running, heap_size: 233, stack_size: 28, reductions: 169; neighbours:
2024-06-17 08:57:02.198 ∅∅∅
2024-06-17T06:57:02.196549+00:00 [error] crasher: initial call: mnesia_subscr:init/1, pid: <0.2210.0>, registered_name: mnesia_subscr, exit: {killed,[{gen_server,decode_msg,9,[{file,"gen_server.erl"},{line,1049}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]}, ancestors: [mnesia_kernel_sup,mnesia_sup,<0.2204.0>], message_queue_len: 0, messages: [], links: [], dictionary: [], trap_exit: true, status: running, heap_size: 2586, stack_size: 28, reductions: 3959; neighbours:
2024-06-17 08:57:02.199 ∅∅∅
2024-06-17T06:57:02.198212+00:00 [error] crasher: initial call: application_master:init/4, pid: <0.2195.0>, registered_name: [], exit: {{bad_return,{{mria_app,start,[normal,[]]},{'EXIT',{{badmatch,{error,{killed,{mnesia_app,start,[normal,[]]}}}},[{mria_mnesia,ensure_started,0,[{file,"mria_mnesia.erl"},{line,113}]},{mria_app,start,2,[{file,"mria_app.erl"},{line,36}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}},[{application_master,init,4,[{file,"application_master.erl"},{line,142}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]}, ancestors: [<0.2194.0>], message_queue_len: 1, messages: [{'EXIT',<0.2196.0>,normal}], links: [<0.2194.0>,<0.2021.0>], dictionary: [], trap_exit: true, status: running, heap_size: 376, stack_size: 28, reductions: 171; neighbours:
2024-06-17 08:57:02.200 ∅∅∅
2024-06-17T06:57:02.199388+00:00 [error] crasher: initial call: application_master:init/4, pid: <0.2192.0>, registered_name: [], exit: {{bad_return,{{emqx_machine_app,start,[normal,[]]},{'EXIT',{{badmatch,{error,{mria,{bad_return,{{mria_app,start,[normal,[]]},{'EXIT',{{badmatch,{error,{killed,{mnesia_app,start,[normal,[]]}}}},[{mria_mnesia,ensure_started,0,[{file,"mria_mnesia.erl"},{line,113}]},{mria_app,start,2,[{file,"mria_app.erl"},{line,36}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}}},[{mria,start,0,[{file,"mria.erl"},{line,126}]},{ekka,start,0,[{file,"ekka.erl"},{line,94}]},{emqx_machine,start,0,[{file,"emqx_machine.erl"},{line,54}]},{emqx_machine_app,start,2,[{file,"emqx_machine_app.erl"},{line,29}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}},[{application_master,init,4,[{file,"application_master.erl"},{line,142}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]}, ancestors: [<0.2191.0>], message_queue_len: 1, messages: [{'EXIT',<0.2193.0>,normal}], links: [<0.2191.0>,<0.2021.0>], dictionary: [], trap_exit: true, status: running, heap_size: 610, stack_size: 28, reductions: 188; neighbours:
2024-06-17 08:57:02.201 ∅∅∅
2024-06-17T06:57:02.200339+00:00 [error] crasher: initial call: mnesia_monitor:init/1, pid: <0.2209.0>, registered_name: mnesia_monitor, exit: {killed,[{gen_server,decode_msg,9,[{file,"gen_server.erl"},{line,1049}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]}, ancestors: [mnesia_kernel_sup,mnesia_sup,<0.2204.0>], message_queue_len: 1, messages: [{'$gen_call',{<0.2213.0>,#Ref<0.1144114197.2730491906.171201>},{close_log,latest_log}}], links: [<62246.2209.0>,<62247.2209.0>,<0.2247.0>], dictionary: [], trap_exit: true, status: running, heap_size: 4185, stack_size: 28, reductions: 12190; neighbours:
2024-06-17 08:57:03.846 ∅∅∅
Kernel pid terminated (application_controller) ("{application_start_failure,emqx_machine,{bad_return,{{emqx_machine_app,start,[normal,[]]},{'EXIT',{{badmatch,{error,{mria,{bad_return,{{mria_app,start,[normal,[]]},{'EXIT',{{badmatch,{error,{killed,{mnesia_app,start,[normal,[]]}}}},[{mria_mnesia,ensure_started,0,[{file,\"mria_mnesia.erl\"},{line,113}]},{mria_app,start,2,[{file,\"mria_app.erl\"},{line,36}]},{application_master,start_it_old,4,[{file,\"application_master.erl\"},{line,293}]}]}}}}}}},[{mria,start,0,[{file,\"mria.erl\"},{line,126}]},{ekka,start,0,[{file,\"ekka.erl\"},{line,94}]},{emqx_machine,start,0,[{file,\"emqx_machine.erl\"},{line,54}]},{emqx_machine_app,start,2,[{file,\"emqx_machine_app.erl\"},{line,29}]},{application_master,start_it_old,4,[{file,\"application_master.erl\"},{line,293}]}]}}}}}")
2024-06-17 08:57:03.852 ∅∅∅
{}
2024-06-17 08:57:04.165 ∅∅∅
Crash dump is being written to: /opt/emqx/log/erl_crash.dump...done

I imagine this to be a split brain issue simialr to what I have found in this post so I might have solved it through deleting data/mnesia and the joining the node again to the cluster however I am curious as to what would be the correct way to increase replicas without having something like this occur.

[Rory-Z]

If emqx is working normally, increasing the number of replicas will not cause blue-green deployment. Only modifications that result in Pod restarts will trigger blue-green deployment.
@id could you please take a look this crash log ?