Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend benchmarks again #251

Open
turion opened this issue Feb 7, 2023 · 3 comments
Open

Extend benchmarks again #251

turion opened this issue Feb 7, 2023 · 3 comments

Comments

@turion
Copy link
Collaborator

turion commented Feb 7, 2023

At some point in the past, the benchmarks included Anglican and WebPPL. They don't anymore because it can't be expected from a normal developer to install these. It would be nice if the benchmarks were packaged as nix derivations, and the Anglican and WebPPL versions be restored as part of that. If they are implemented in nix, they're less likely to break, and can be run for everyone. In case this proves too hard, we could try to make a STAN or PyMC benchmark for comparison instead.

@turion
Copy link
Collaborator Author

turion commented Feb 8, 2023

Another axis along which it would be very useful to extend the benchmarks: Running them for different commits. This could also be done with nix by providing different sources for the monad-bayes package to the benchmark.

@reubenharry
Copy link
Contributor

I'm sort of ok leaving out the Anglican and WebPPL baselines, for two reasons:

  1. Neither system is under active development (to my knowledge)
  2. Interested readers can simply refer to the benchmark in the paper.

@reubenharry
Copy link
Contributor

Running for different commits would be helpful. Actually something that I would previously have found helpful too is a very simple thing I can run that just tells me: did the changes I commit slow down or speed up the benchmark? One can get this information indirectly via criterion, but I found it cumbersome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants