Race condition on asterisk custom configuration

[juansalvatella]

I have been able to successfully deploy Kamailio and Asterisk with asterisk-config as a sidecar in GCP picking up the custom configuration files from a kubernetes secret. The overall setup is working fine (thanks for that Sean!) but occasionally asterisk-config container fails and the pod is restarted. Once the pod is restarted, asterisk fails to load the custom modules.conf stored in the K8s secret and instead, it loads the default modules.conf that is shipped with the cycore image hosted at Dockerhub.

These are the startup logs for asterisk-config. It seems to be detecting lots of changes even though nothing has changed and maybe this is causing the issue.

2018/10/08 21:23:34 WARNING: unhandled cloud
2018/10/08 21:23:34 running service
2018/10/08 21:23:34 change detected
2018/10/08 21:23:34 service exited: failed to reload asterisk modules: failed to contact ARI to reload module res_pjsip.so: Put http://127.0.0.1:8088/ari/asterisk/modules/res_pjsip.so: dial tcp 127.0.0.1:8088: connect: connection refused
2018/10/08 21:23:34 running service
2018/10/08 21:23:35 change detected
2018/10/08 21:23:35 service exited: failed to reload asterisk modules: failed to contact ARI to reload module res_pjsip.so: Put http://127.0.0.1:8088/ari/asterisk/modules/res_pjsip.so: dial tcp 127.0.0.1:8088: connect: connection refused
2018/10/08 21:23:35 running service
2018/10/08 21:23:35 change detected
2018/10/08 21:23:35 service exited: failed to reload asterisk modules: module res_pjsip.so not already loaded
2018/10/08 21:23:35 running service
2018/10/08 21:23:35 change detected
2018/10/08 21:23:35 service exited: failed to reload asterisk modules: module res_pjsip.so reload failed: 503 Service Unavailable
2018/10/08 21:23:35 running service
2018/10/08 21:23:35 change detected
2018/10/08 21:23:35 reloads complete
2018/10/08 21:23:35 change detected
2018/10/08 21:23:35 reloads complete
2018/10/08 21:23:35 change detected
2018/10/08 21:23:35 reloads complete
2018/10/08 21:23:35 change detected
2018/10/08 21:23:35 reloads complete
2018/10/08 21:23:35 change detected
2018/10/08 21:23:35 reloads complete
2018/10/08 21:23:35 change detected
2018/10/08 21:23:35 reloads complete

I've noticed that the first warning indicates that no cloud has been selected but, as you identified in a previous issue from last year's Astricon repo, GKE generates k8s nodes with very long hostnames and includes the word "default" which is problematic for Kamailio. Netdiscover in best-effort mode is able to pick up the same public IP and private IP as if the environment variable was set to GCP. The only difference is in the hostname, when set-up to GCP it gets something such as gke--cluster-default-pool--..*****.internal and in best-effort ....bc.googleusercontent.com.

If I can provide any additional information to debug this issue, please let me know.

[Ulexus]

Unfortunately, I don't have a really fantastic solution to this race condition. What I have been doing is adding a check in the startup of Asterisk for the /etc/asterisk/manager.conf (arbitrary choice). If it doesn't exist, I have Asterisk die immediately.

You can see this in my entrypoint script for this year's demo:

https://github.com/CyCoreSystems/asterisk-k8s-demo/blob/master/live-demo/asterisk/entrypoint.sh#L12

[juansalvatella]

Perfect, this worked!