Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Builder starts up before docker daemon has a chance to load client certs #1

Open
1 task
Datamance opened this issue Apr 7, 2020 · 0 comments
Open
1 task
Assignees
Labels
bug Something isn't working scoping Still in the scoping stage - need to figure out what actual work needs to be done

Comments

@Datamance
Copy link
Contributor

TL;DR: Follow up from lookit/lookit-api#483 - either reorder container initialization for the builder statefulset, or populate the client cert some other way.

Narrative
As a developer, I want Sentry alerts to be as meaningful and sparse as possible.

Right now, on every deploy, we are seeing this:
TLSParameterError in celery workers
Which indicates that the Docker-in-Docker container hasn't booted up yet, resulting in a lack of expected client certs in the shared volume.

This only causes errors for a short bit, as the DinD container eventually boots up and puts the certs in the right place, in time for about the 3rd or 4th retry from the celery process to pick them up. As such, this amounts to deployment noise, which can become problematic if a real deployment issue starts happening and the meaningful alert gets buried in a sea of non-useful alerts.

Acceptance Criteria

  • The TLSParameterError no longer occurs on deploys.

Implementation Notes
This is going to be a bit tricky, because it runs into the architectural flaw of kustomize that doesn't allow us to target containers by name with JsonPatches6902, which restricts us to hardcoding an index into the containers array and assuming that the first container is a django-based container that should receive the add-lookit-env-vars patch.

@Datamance Datamance self-assigned this Apr 7, 2020
@Datamance Datamance transferred this issue from lookit/lookit-api Apr 9, 2020
@Datamance Datamance added the bug Something isn't working label Apr 9, 2020
@Datamance Datamance added this to the Deployment Stability milestone Sep 3, 2020
@Datamance Datamance added the scoping Still in the scoping stage - need to figure out what actual work needs to be done label Sep 3, 2020
@mekline mekline moved this to Tech Debt in Issue Clusters Jan 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working scoping Still in the scoping stage - need to figure out what actual work needs to be done
Projects
Status: Tech Debt
Development

No branches or pull requests

1 participant