-
Notifications
You must be signed in to change notification settings - Fork 3
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
5 changed files
with
73 additions
and
98 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,82 +1,65 @@ | ||
Nomad Skeleton Device Plugin | ||
Nomad Nvidia Virtual Device Plugin | ||
================== | ||
|
||
Skeleton project for [Nomad device plugins](https://www.nomadproject.io/docs/internals/plugins/devices.html). | ||
|
||
This project is intended for bootstrapping development of a new device plugin. | ||
|
||
- Website: https://www.nomadproject.io | ||
- Mailing list: [Google Groups](http://groups.google.com/group/nomad-tool) | ||
|
||
Requirements | ||
------------ | ||
|
||
- [Nomad](https://www.nomadproject.io/downloads.html) 0.9+ | ||
- [Go](https://golang.org/doc/install) 1.11 or later (to build the plugin) | ||
|
||
Building the Skeleton Plugin | ||
--------------------- | ||
[Generate](https://github.com/hashicorp/nomad-skeleton-device-plugin/generate) | ||
a new repository in your account from this template by clicking the `Use this | ||
template` button above. | ||
|
||
Clone the repository somewhere in your computer. This project uses | ||
[Go modules](https://blog.golang.org/using-go-modules) so you will need to set | ||
the environment variable `GO111MODULE=on` or work outside your `GOPATH` if it | ||
is set to `auto` or not declared. | ||
|
||
```sh | ||
$ git clone [email protected]:<ORG>/<REPO>git | ||
``` | ||
|
||
Enter the plugin directory and update the paths in `go.mod` and `main.go` to | ||
match your repository path. | ||
|
||
```diff | ||
// go.mod | ||
|
||
- module github.com/hashicorp/nomad-skeleton-device-plugin | ||
+ module github.com/<ORG>/<REPO> | ||
... | ||
This repo contains a device plugin for [Nomad](https://www.nomadproject.io/) to support exposing a number of virtual GPUs for each physical GPU present on the machine. This enables running workloads which don't consume the whole GPU. | ||
|
||
Installation requirements | ||
----------------------- | ||
|
||
This plugin needs the following dependencies to function: | ||
|
||
* [Nomad](https://www.nomadproject.io/downloads.html) 0.9+ | ||
* GNU/Linux x86_64 with kernel version > 3.10 | ||
* NVIDIA GPU with Architecture > Fermi (2.1) | ||
* NVIDIA drivers >= 340.29 with binary nvidia-smi | ||
* Docker v19.03+ | ||
|
||
Copy the plugin binary to the [plugins directory](https://www.nomadproject.io/docs/configuration/index.html#plugin_dir) and [configure the plugin](https://www.nomadproject.io/docs/configuration/plugin.html) in the client config. Also, see the requirements for the official [nvidia-plugin](https://www.nomadproject.io/plugins/devices/nvidia#installation-requirements). | ||
|
||
```hcl | ||
plugin "nvidia-vgpu" { | ||
config { | ||
ignored_gpu_ids = ["uuid1", "uuid2"] | ||
fingerprint_period = "5s" | ||
vgpus = 16 | ||
} | ||
} | ||
``` | ||
|
||
```diff | ||
// main.go | ||
|
||
package main | ||
|
||
import ( | ||
log "github.com/hashicorp/go-hclog" | ||
"github.com/hashicorp/nomad/plugins" | ||
|
||
- "github.com/hashicorp/nomad-skeleton-device-plugin/device" | ||
+ "github.com/<REPO>/<ORG>/device" | ||
) | ||
... | ||
``` | ||
|
||
Build the skeleton plugin. | ||
|
||
```sh | ||
$ make build | ||
``` | ||
|
||
Running the Plugin in Development | ||
--------------------- | ||
|
||
You can test this plugin (and your own device plugins) in development using the | ||
[plugin launcher](https://github.com/hashicorp/nomad/tree/master/plugins/shared/cmd/launcher). The makefile provides | ||
a target for this: | ||
|
||
```sh | ||
$ make eval | ||
Usage | ||
-------------- | ||
|
||
Then use the [device stanza](https://www.nomadproject.io/docs/job-specification/device.html) in the job file to schedule with device support. | ||
|
||
```hcl | ||
job "gpu-test" { | ||
datacenters = ["dc1"] | ||
type = "batch" | ||
group "smi" { | ||
task "smi" { | ||
driver = "docker" | ||
config { | ||
image = "nvidia/cuda:11.0-base" | ||
command = "nvidia-smi" | ||
} | ||
resources { | ||
device "nvidia-vgpu/gpu" { | ||
count = 1 | ||
# Add an affinity for a particular model | ||
affinity { | ||
attribute = "${device.model}" | ||
value = "Tesla K80" | ||
weight = 50 | ||
} | ||
} | ||
} | ||
} | ||
} | ||
} | ||
``` | ||
|
||
Deploying Device Plugins in Nomad | ||
---------------------- | ||
|
||
Copy the plugin binary to the | ||
[plugins directory](https://www.nomadproject.io/docs/configuration/index.html#plugin_dir) and | ||
[configure the plugin](https://www.nomadproject.io/docs/configuration/plugin.html) in the client config. Then use the | ||
[device stanza](https://www.nomadproject.io/docs/job-specification/device.html) in the job file to schedule with | ||
device support. (Note, the skeleton plugin is not intended for use in Nomad.) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters