work in progress - remote run #3986
Conversation
| def remote_run(args) -> int: | ||
| """Run stuff, but remotely!""" | ||
| system_paasta_config = load_system_paasta_config( | ||
| "/nail/home/qlo/paasta_config/paasta/" |
There was a problem hiding this comment.
I was using this to override the API endpoint
paasta_tools/paasta_remote_run_2.py
Outdated
| def wait_until_deployment_gone(kube_client, namespace, deployment_name): | ||
| for retry in range(10): | ||
| pod = find_pod(kube_client, namespace, deployment_name, 1) | ||
| if not pod: | ||
| return | ||
| sleep(5) | ||
| raise Exception("Pod still exists!") |
There was a problem hiding this comment.
This doesn't work. The request times out before the deployment terminates. I think we'd need to return and the client needs to wait and retry.
paasta_tools/paasta_remote_run_2.py
Outdated
|
|
||
| # Create the app with a new name | ||
| formatted_application = deployment.format_kubernetes_app() | ||
| formatted_application.metadata.name += f"-remote-run-{user}" |
There was a problem hiding this comment.
there's the risk of the pod name being truncated, so we'll definitely also want to indicators of this being a remote-run in the pod's labels
paasta_tools/kubernetes_tools.py
Outdated
| selector=V1LabelSelector( | ||
| match_labels={ | ||
| "paasta.yelp.com/service": self.get_service(), | ||
| "paasta.yelp.com/instance": self.get_instance(), | ||
| } |
There was a problem hiding this comment.
todo: This was copied over from the Deployment function below and is causing errors.
There was a problem hiding this comment.
mmm, yeah, I don't see why you would need a select at this stage
| kind="Job", | ||
| metadata=self.get_kubernetes_metadata(git_sha), | ||
| spec=V1JobSpec( | ||
| active_deadline_seconds=3600, |
There was a problem hiding this comment.
I guess we'll need to propagate this somehow from a parameter
paasta_tools/kubernetes_tools.py
Outdated
| selector=V1LabelSelector( | ||
| match_labels={ | ||
| "paasta.yelp.com/service": self.get_service(), | ||
| "paasta.yelp.com/instance": self.get_instance(), | ||
| } |
There was a problem hiding this comment.
mmm, yeah, I don't see why you would need a select at this stage
paasta_tools/kubernetes_tools.py
Outdated
| # DO NOT ADD LABELS AFTER THIS LINE | ||
| config_hash = get_config_hash( | ||
| self.sanitize_for_config_hash(complete_config), | ||
| force_bounce=self.get_force_bounce(), | ||
| ) | ||
| complete_config.metadata.labels["yelp.com/paasta_config_sha"] = config_hash | ||
| complete_config.metadata.labels["paasta.yelp.com/config_sha"] = config_hash | ||
|
|
||
| complete_config.spec.template.metadata.labels[ | ||
| "yelp.com/paasta_config_sha" | ||
| ] = config_hash | ||
| complete_config.spec.template.metadata.labels[ | ||
| "paasta.yelp.com/config_sha" | ||
| ] = config_hash |
There was a problem hiding this comment.
this stuff is very related to deployment bounces, so I'd say we don't need it in this case
paasta_tools/kubernetes_tools.py
Outdated
| audience = "remote-run-" + user | ||
| service_account = "default" | ||
| token_spec = V1TokenRequestSpec( | ||
| expiration_seconds=600, audiences=[audience] # minimum allowed by k8s |
There was a problem hiding this comment.
The audience will need to be the kubernetes one, otherwise it won't be usable for doing execs. You should be able to just leave it unset to achieve that.
paasta_tools/kubernetes_tools.py
Outdated
| def create_temp_exec_token(kube_client: KubeClient, namespace: str, user: str): | ||
| """Create a short lived token for exec""" | ||
| audience = "remote-run-" + user | ||
| service_account = "default" |
There was a problem hiding this comment.
This will need to come from the main paasta config. We'll have to create a user that can just exec in to places.
paasta_tools/kubernetes_tools.py
Outdated
| @@ -4354,6 +4452,19 @@ def get_kubernetes_secret_env_variables( | |||
| return decrypted_secrets | |||
|
|
|||
|
|
|||
| def create_temp_exec_token(kube_client: KubeClient, namespace: str, user: str): | |||
There was a problem hiding this comment.
The user parameter is not used here. Is the plan to include it in the service account name if we go for creating ephemeral service accounts?
Work in progress, not fully functional!
Create a new API, /service/{service}/{instance}/remote_run
Currently accepts additional a username, whether it's interactive, and whether it should recreate the deployment
Launches a deployment with
-remote-run-{username}. If interactive, replaces command with sleep. If deployment exists, maybe terminate it and relaunch. Returns the name of the pod and the namespace.Created CLI cmd to interact with this api. Gives you a kubectl command to run to get a shell in the container.