Proposal of dynamic GPU slice plugin

[sailorvii]

NVIDIA official GPU sharing includes time-slice, MPS and MIG. Currently the MPS and MIG dynamic is not supported, we want to add this into volcano scheduler plugin

[volcano-sh-bot]

Welcome @sailorvii!

It looks like this is your first PR to volcano-sh/volcano.

Thank you, and welcome to Volcano. 😃

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign lowang-bh
You can assign the PR to them by writing /assign @lowang-bh in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Monokaix]

Hi, please squash to one commit and sign off.

[JesseStutler]

I have reviewd it, please take a look~

[JesseStutler]

Where is the logic of this AddPod, in the mig-agent of nos? I'm wondering whether our dynamic GPU slice plugin is strongly dependent on the nos project. You can see that the annotation has the watermark of nos, and nos project is not updated frequently.

[sailorvii]

1. AddPod is in volcano/pkg/scheduler/api/node_info.go addResource.
2. 3 functions can be reused from nos project: mig agent, mps agent and mps device plugin. They are not the most important part. If needed, we could rewrite them.

[JesseStutler]

/cc @Monokaix , I think we'd better rewrite them as part of volcano and evolve with us.

[archlitchi]

A nice feature, but i have a few recommends:

1. please add user guide for using dynamic MIG and MPS
2. please clarify if annotations 'dynamicgpuslice' is a pod annotation or a node annotation?

[sailorvii]

archlitchi

Thanks for your time and review.

1. Add the usage part.
2. They're all node annotations. (the title has said “Node labels and annotations”)

[Monokaix]

How about use the deviceshare plugin?

[Monokaix]

We should clarify which dp the user should deploy and the relationship between dynamic mig slice and vgpu. The semantics of vgpu and dynamic mig slice are not completely consistent. Whether to use nvidia dp or hami needs to be discussed again.

[JesseStutler]

Let’s discuss it again how to evolve this feature at the weekly meeting? Currently, it seems that there are three repos: volcano does the scheduling, hami does the dp, and nos does the mig/mps agent. It is too fragmented. @sailorvii @archlitchi @Monokaix

[sailorvii]

Let’s discuss it again how to evolve this feature at the weekly meeting? Currently, it seems that there are three repos: volcano does the scheduling, hami does the dp, and nos does the mig/mps agent. It is too fragmented. @sailorvii @archlitchi @Monokaix

Thank you all for your time. It's good to discuss the details in the meeting.