Performance regressions

[boxed]

I have this little benchmark test suite: https://github.com/boxed/test-benchmarks

Last time I ran it 2 years ago ward beat out pytest with some margin. Today the result is reversed. Just wanted to give a little heads up.

Running the benchmark should be obvious. Create a venv, install from requirements.txt and run python run_bench.py

[hukkin]

I share the performance concern with you. I always thought pytest was a bit too slow.

I've put some effort into improving ward's type annotations and increasing type checker coverage. One fairly straightforward way of increasing performance would be to continue that work, and publish binary wheels built with mypyc. This can typically result in at least a 2x performance increase.

[boxed]

Working on pytest speed I noticed that if no one runs a profiler for a long time you start to get big easy wins :P

I would bet against getting significant speed out of mypyc personally. There's probably some double file access somewhere or something.

[hukkin]

Working on pytest speed I noticed that if no one runs a profiler for a long time you start to get big easy wins :P

Yeah, I expect the same. Having a make profile command set up in ward's Makefile would be a great start IMO.

I would bet against getting significant speed out of mypyc personally. There's probably some double file access somewhere or something.

mypy itself claims a 4x perf boost over regular Python. With strict typing I don't see why something similar wouldn't be possible here. I do agree that profiling and algorithmic improvements probably yield even better results.... but these two paths are not mutually exclusive. We can have both 😸

[adriangb]

I'm curious where ward (and maybe pytest if any of you know) is spending most of it's time. Crawling files? Figuring out fixture dependencies?

[boxed]

Pytest spends a lot of time in crawling files, and also (first time you run a test file) on assertion rewriting. The reason I could beat pytest in speed on my micro benchmarks with hammett, was that I made sure not do do any double work crawling files, and I don't do assertion rewriting.

Not having looked at ward, I'd put a little money on there being some simple mistake somewhere that's causing most of the overhead time right now. There shouldn't be any reason why ward should have such high overhead compared to hammett or pytest.

[darrenburns]

Thanks for posting about this, you're probably correct in that there'll be some work being done twice in the collection phase.

It's something I'll certainly dig into in the future, but isn't too concerning to me at the moment. I haven't put a great deal of effort into optimisation, focussing more on the APIs etc. at the moment. I suspect you're right that there'll be a lot of quick wins possible when it comes to performance if someone was to run a profiler on ward 👍