watchdog: restart worker if failing #7
Conversation
| if [[ $FAIL -eq 1 ]]; then | ||
| #/bin/systemd-notify --pid=$WORKER_PID "WATCHDOG=1"; | ||
| /bin/systemctl restart concourse-worker | ||
| sleep 1 |
There was a problem hiding this comment.
I think the sleep should be bigger here to give time to the concourse-worker to clean his mess before being ready.
With a sleep 1 we might up in situations where the watchdog restarts the concourse-worker in a loop.
The logic might need to be different than the systemd watchdog implementation which allowed the healthcheck to fail several times before deciding to restart the concourse-worker.
Here restarting at the first healthcheck failure is brutal when sometimes it might be a false alarm.
Ideally we would want a to retry 2 or 3 times the healthcheck to be sure the concourse-worker is KO, restart the process if it is, then wait some time for the concourse-worker to recover/clean up old things before expecting the healthcheck to be successful.
WDYT ?
There was a problem hiding this comment.
Not sure for the sleep it should be good just after a worker restart but I added 3 retry (with 15sec between each) to be sure
talset
force-pushed
the
fl-watch
branch
2 times, most recently
from
96e806a to
f735086
Compare
Workaround of #5 This commit will need to be revert when real watchdog will be unblocked
Workaround of #5
This commit will need to be revert when real watchdog will be unblocked