feat(gres)!: add mappings for managing GRESName and GRESNode objects
#40
+674
−125
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
`jsonschema` is now included as a dependency to help validate Slurm data models as they're exchanged between classes and read in. `typing-extensions` is added because the `Self` type annotation is not available within Python 3.10 add is still a support version of Python for security maintenance and Jammy Jellyfish. Signed-off-by: Jason C. Nucciarone <[email protected]>
Changes: * Change `names` and `nodes` property on `GRESConfig` to return `GRESNameMapping` and `GRESNodeMapping` respectively. * Add `_slice` method to enable quick slicing of the internal data store dictionaries. * Add `BaseMapping` ABC that can be used to create other mappings that can be used within slurm.conf files. * Add `expand` method for recursively expanding dictionaries with nested Slurm data model objects. * Introduce JSON schemas for validating `GRESNameMapping` and `GRESNodeMapping` objects. * Define JSON decoders that can be used to parse dictionary objects into Slurm data models. Signed-off-by: Jason C. Nucciarone <[email protected]>
Changes: * Add addition test case where the same `NodeName` can have multiple generic resource configurations. Signed-off-by: Jason C. Nucciarone <[email protected]>
Signed-off-by: Jason C. Nucciarone <[email protected]>
Signed-off-by: Jason C. Nucciarone <[email protected]>
NucciTheBoss
added
the
Type: Enhancement
dsloanm
approved these changes
Dec 20, 2024
There was a problem hiding this comment.
Looks good and solves the issue I was hitting!
I've not spotted the stricter validation, however. I accidentally generated a few invalid gres.conf files when testing since I can get away with things like:
$ python
Python 3.12.3 (main, Nov 6 2024, 18:32:19) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from slurmutils.editors import gresconfig
>>> with gresconfig.edit("gres-testing.conf") as config:
... config.nodes["123"] = [456]
...
>>>
$ cat gres-testing.conf
456
$| config.names.append(name.dict()) | ||
| config.nodes.updaten(node.dict()) | ||
| config.names[new_gres.name] = [new_gres] | ||
| config.nodes[new_node.node_name] = [new_node] |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds the
GRESNameMappingandGRESNodeMappingobjects to make it easier to manage gres.conf data model objects. Rather than throw lists and dictionaries out there all loosey goosey forGRESConfig, instead these new maps help facilitate interacting with the configured generic resources and easily share them between the slurmctld and slurmd operators. Couple key changes:jsonschema. Helps make sure the input we're putting into the maps are valid.jsonschemaand send "across the wire," but still enable us to take advantage of the data models and the descriptors + methods that they provide.expandfunction is an anamorphism that recursively expands a dictionary with nested data model objects. This makes it easier to punch objects into a JSON schema validator. Custom object hooks are used with the JSON decoder to parse the data models back into Python objects.BaseMappingobject to provide methods necessary forGRESNameMappingandGRESNodeMapping, but in the future I could seeBaseMappingbecoming the newBaseModel. The classes do similar things, butBaseMappingis more flexible and uses JSON schemas to validate the data models.Breaking changes
The
namesandnodesproperties fromGRESConfignow returnGRESNameMappingandGRESNodeMappingobjects respectively rather thanlistanddicttypes.Related issues
Misc.
I updated the annotations for some functions to be compliant with recommendations set forth by Python 3.10. Big thing is that the many of the types provided by the
typingmodule are now deprecated, so their imports are now updated. Another thing is the inclusion of thetyping-extensionsmodule. This is an official module maintained by the Python team. I bring it in as dependency since it provides theSelftype which wasn't introduced into Python until 3.11. Sincepy310is still a supported Python version, figured it doesn't hurt to bringtyping-extensionsin.MutableMappingandSequenceare the recommend annotations for arguments that are either typedictorlistrespectively.