You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
TL;DR: Follow up from lookit/lookit-api#483 - either reorder container initialization for the builder statefulset, or populate the client cert some other way.
Narrative
As a developer, I want Sentry alerts to be as meaningful and sparse as possible.
Right now, on every deploy, we are seeing this: TLSParameterError in celery workers
Which indicates that the Docker-in-Docker container hasn't booted up yet, resulting in a lack of expected client certs in the shared volume.
This only causes errors for a short bit, as the DinD container eventually boots up and puts the certs in the right place, in time for about the 3rd or 4th retry from the celery process to pick them up. As such, this amounts to deployment noise, which can become problematic if a real deployment issue starts happening and the meaningful alert gets buried in a sea of non-useful alerts.
Acceptance Criteria
The TLSParameterError no longer occurs on deploys.
Implementation Notes
This is going to be a bit tricky, because it runs into the architectural flaw of kustomize that doesn't allow us to target containers by name with JsonPatches6902, which restricts us to hardcoding an index into the containers array and assuming that the first container is a django-based container that should receive the add-lookit-env-vars patch.
The text was updated successfully, but these errors were encountered:
TL;DR: Follow up from lookit/lookit-api#483 - either reorder container initialization for the builder statefulset, or populate the client cert some other way.
Narrative
As a developer, I want Sentry alerts to be as meaningful and sparse as possible.
Right now, on every deploy, we are seeing this:
TLSParameterError in celery workers
Which indicates that the Docker-in-Docker container hasn't booted up yet, resulting in a lack of expected client certs in the shared volume.
This only causes errors for a short bit, as the DinD container eventually boots up and puts the certs in the right place, in time for about the 3rd or 4th retry from the celery process to pick them up. As such, this amounts to deployment noise, which can become problematic if a real deployment issue starts happening and the meaningful alert gets buried in a sea of non-useful alerts.
Acceptance Criteria
Implementation Notes
This is going to be a bit tricky, because it runs into the architectural flaw of kustomize that doesn't allow us to target containers by name with JsonPatches6902, which restricts us to hardcoding an index into the containers array and assuming that the first container is a django-based container that should receive the
add-lookit-env-vars
patch.The text was updated successfully, but these errors were encountered: