Add benchmark script #81
Conversation
|
@kvhuguenin the times are reported, as far as i'm able to determine, per sample. however, the script allows for asynchronous dispatch, so in principle there could even be some parallelisation benefits.... maybe i should turn that off. let's have a look. |
|
Very nice @sirmarcel and thanks. This will be very useful. Regarding storing. I had the idea of creating an orphan branch maybe called |
|
|
||
| .. code-block:: bash | ||
|
|
||
| pip install .[metatensor] |
There was a problem hiding this comment.
I think also vesin, ase etc are needed.
| Usage | ||
| ----- | ||
|
|
||
| First, make sure to install ``torch-pme`` with the right dependencies: |
There was a problem hiding this comment.
maybe even
| First, make sure to install ``torch-pme`` with the right dependencies: | |
| First, make sure to install ``torch-pme`` with the right dependencies in a fresh enviroment: |
| devices = [] | ||
| if torch.cuda.is_available(): | ||
| devices.append("cuda") | ||
|
|
||
| # run CUDA first! | ||
| devices.append("cpu") |
There was a problem hiding this comment.
| devices = [] | |
| if torch.cuda.is_available(): | |
| devices.append("cuda") | |
| # run CUDA first! | |
| devices.append("cpu") | |
| devices = ["cpu"] | |
| # run CUDA first! | |
| if torch.cuda.is_available(): | |
| devices.insert(0, "cuda") |
There was a problem hiding this comment.
| devices = [] | |
| if torch.cuda.is_available(): | |
| devices.append("cuda") | |
| # run CUDA first! | |
| devices.append("cpu") | |
| devices = ["cuda", "cpu"] if torch.cuda.is_available() else ["cpu"] |
| version = { | ||
| "torch": str(torch.__version__), | ||
| "torch-pme-commit": torch_pme_commit, | ||
| "torch-pme-status": torch_pme_status, | ||
| } |
There was a problem hiding this comment.
maybe also add the torchpme.__version__. Even though useless before we release but useful in the future.
| with open(f"{timestamp}.yaml", "w") as file: | ||
| yaml.dump(out, file) | ||
|
|
||
| print("Have a nice day.") |
There was a problem hiding this comment.
Good old fhi-aims vibes. Like it.
|
Maybe we should also run this code somewhere in CI. Not for benchmark but just that we know we don't break anything. maybe just a subprocess call in der testsuite... EDIT. Maybe one can eve do a |
|
Would be nice to make this a bit more customizable - setting the devices to run on, the sizes to try, etc. We can leave all this as defaults, but e.g. if one is using this in development it makes sense to only run what's being optimized for. |
|
|
||
| import torchpme | ||
|
|
||
| primitive = read("geometry.in") |
There was a problem hiding this comment.
Maybe choose the file based on the location of this script?
| primitive = read("geometry.in") | |
| primitive = read(Path(__file__).parent / "geometry.in") |
| @@ -0,0 +1,344 @@ | |||
| import datetime | |||
There was a problem hiding this comment.
Maybe even make it executable
| import datetime | |
| #!/usr/bin/env python | |
| import datetime |
Fixes #17
As discussed a while ago with @PicoCentauri , this adds a benchmark script, designed to emulate a "typical" training workload. The idea is that we agree to run this before major merges, to make sure we don't cause performance regressions.
The script runs Ewald and PME, with and without Metatensor, for CsCl crystals of various sizes. The results are saved as
.yaml, with some diagnonstic information. An example output can be seen in #80. The script already worked quite nicely to pointint that something was wrong in that case...We could consider making some additional repository for the
.yamland make a nice plot of performance, hopefully, increasing over time.📚 Documentation preview 📚: https://torch-pme--81.org.readthedocs.build/en/81/