More error tolerant web ui check

[haoming29]

Should fix #1033

The suspicion is that there's a racing condition at the registry start somehow, and the web engine is not "ready-ready" to serve requests, although it could be other issues. However, this PR will make the web ui check more robust in case of the error code returned

[bbockelm]

I don't understand what this code is trying to do.

WaitUntilWorking tries a URL every 50ms until it succeeds or 10 seconds has passed. It seems the code adds another set of loops to retry on top of retries (which appears strange?) ... no?

[haoming29]

I don't understand what this code is trying to do.

WaitUntilWorking tries a URL every 50ms until it succeeds or 10 seconds has passed. It seems the code adds another set of loops to retry on top of retries (which appears strange?) ... no?

My guessing of William hitting that 404 error is that the gin server wasn't fully started before we hit the health check endpoint (since the error is intermittent/random). Since 404 instead of 200 was returned, WaitUntilWorking returns immediately and we never get a change to check it a bit later. I don't want to chance how WaitUntilWorking works because in most situations it makes sense to fail immediately if the status code is different from expected, but for the registry's case, it would probably worth waiting and checking again if the status code doesn't match

[bbockelm]

Gotcha. I would rather add another option to WaitUntilWorking to make the wrong response code non-fatal. Layers of retries are a bit scary.

[haoming29]

Gotcha. I would rather add another option to WaitUntilWorking to make the wrong response code non-fatal. Layers of retries are a bit scary.

Sounds good. I'll refactor the code

[jhiemstrawisc]

@haoming29 is this ready for review or should I wait a bit longer?

[haoming29]

@haoming29 is this ready for review or should I wait a bit longer?

Sorry I haven't get the chance to work on the refactoring. I will let you know when its' review for another round of review

[haoming29]

@jhiemstrawisc Should be good for the review. The Ubuntu test failure should be a random one and re-running it will likely fix the issue

[bbockelm]

LGTM

[jhiemstrawisc]

A few small typos, otherwise LGTM

[haoming29]

A few small typos, otherwise LGTM

Thanks for catching them!

[jhiemstrawisc]

LGTM. Thanks!