Pipeline generator plugins #33
Conversation
Signed-off-by: kevin <[email protected]>
Signed-off-by: kevin <[email protected]>
Signed-off-by: kevin <[email protected]>
Signed-off-by: kevin <[email protected]>
Signed-off-by: kevin <[email protected]>
Signed-off-by: kevin <[email protected]>
Signed-off-by: kevin <[email protected]>
Signed-off-by: kevin <[email protected]>
|
could you rebase this on top of master? |
| f"{HF_HOME}:{HF_HOME}" | ||
| ] | ||
|
|
||
| class KubernetesPodContainerConfig(BaseModel): |
There was a problem hiding this comment.
we cancelled the plan to move back to k8s stack? why are we adding this?
There was a problem hiding this comment.
This is for the A100 node on EKS (roblox aws)
Signed-off-by: kevin <[email protected]>
.github/workflows/unit-test.yml
Outdated
| # This workflow will install Python dependencies, run tests and lint with a single version of Python | ||
| # For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python | ||
|
|
||
| name: Pytest |
There was a problem hiding this comment.
this file is already on the main branch head?
| containers: List[KubernetesPodContainerConfig] | ||
| priority_class_name: str = Field(default="ci", alias="priorityClassName") | ||
| node_selector: Dict[str, Any] = Field( | ||
| default={"nvidia.com/gpu.product": "NVIDIA-A100-SXM4-80GB"}, |
There was a problem hiding this comment.
where does this NVIDIA-A100-SXM4-80GB come from?
| DOCKER_PLUGIN_NAME = "docker#v5.2.0" | ||
| KUBERNETES_PLUGIN_NAME = "kubernetes" | ||
|
|
||
| class DockerPluginConfig(BaseModel): |
There was a problem hiding this comment.
could you add some doc / comments, like for everything?
like maybe some link pointers to where the golden spec of these structs are defined?
scripts/pipeline_generator/plugin.py
Outdated
| class KubernetesPluginConfig(BaseModel): | ||
| pod_spec: KubernetesPodSpec = Field(alias="podSpec") | ||
|
|
||
| def get_kubernetes_plugin_config(docker_image_path: str, test_bash_command: List[str], num_gpus: int) -> Dict: |
There was a problem hiding this comment.
nit: docker_image_path -> image or container_image ?
k8s does not always use docker. and the image does not have to be docker image.
Signed-off-by: kevin <[email protected]>
Signed-off-by: kevin <[email protected]>
| [ | ||
| ( | ||
| "test_image:latest", | ||
| ["bash", "-c", "echo A", "pytest -v -s a.py"], |
There was a problem hiding this comment.
this command looks weird and not real.. "echo A" is the arg of -c, waht is the pytest after it? why does the command for a docker plugin look like this?
Signed-off-by: kevin <[email protected]>
Define Buildkite plugin class for Docker & Kubernetes, along with helper functions to generate these plugin configs.