fix red herring in the readinessProbe

[nammn]

All Submissions:

• Have you opened an Issue before filing this PR?
• Have you signed our CLA?
• Have you checked to ensure there aren't other open Pull Requests for the same update/change?
• Put closes #XXXX in your comment to auto-close the issue that your PR fixes (if such).

Summary

• fixes the red herring that the agent-health-status file is missing
• continue of fix: remove panic of missing health status.json #1224, we missed a few places. This PR instead suggests to test for the file as early as possible in the code
• it can be missing, since the agent requires some time until it populates that file
• while it is still populating the probe should just return unready and it will eventually work

[lsierant]

can we log something here to indicate what happened?

[nammn]

I was thinking about the same, but then I would need to move the logger initialization out to fit the logger:

mongodb-kubernetes-operator/pkg/readiness/config/config.go

Lines 59 to 64 in a7812ff

	logger := &lumberjack.Logger{
	Filename: readinessProbeLogFilePath(),
	MaxBackups: readIntOrDefault(readinessProbeLoggerBackups, 5),
	MaxSize: readInt(readinessProbeLoggerMaxSize),
	MaxAge: readInt(readinessProbeLoggerMaxAge),
	}

I could do this 🤔

[slaskawi]

How about panicing with an error? This way it should be visible in the kubectl get events and kubectl describe pod.

[nammn]

ahh so the root cause of most red herrings are that exact error/panic.
Note the comment:

// In that case, we don't want to panic to show the message
// in the kubernetes description. That would be a red herring, since that will solve itself with enough time

That error almost always happens because we are calling the probe before the agent was able to create the health status file.

Regarding adding the log, I am not sure whether this effort is worth to log it 🤔
@lsierant @slaskawi

[slaskawi]

Thanks for the explanation @nammn ! I believe it's still worth logging an info why this happened. Without it, investigating corner cases might be simply impossible.

[nammn]

Done