Multithread api call in Wildbench

[jmercat]

#25

[neginraoof]

From what I see, the threadpool in eval.py probably could be initialized better to use a higher number of threads:
https://github.com/mlfoundations/dcft_private/blob/70c7f006492028c541e370b620239237e8a8d065/eval/eval.py#L123

Basically max_workers would be 1 if you are testing only one benchmark. Would this fix the issue?

Also, since we are going to use curator soon, maybe it's worth it to just change the backend directly?

[jmercat]

I think there is a confusion. I did not write that line. I wrote only the threadpool in WildBench that is by default set to 32 workers.

[neginraoof]

Yeah but my question is that fixing this from eval.py#L123 cannot solve this issue? @jmercat

[jmercat]

I think those serve different purpose. This one is supposed to parallelize the task including generation (I don't know if it happens or if it works I haven't tested that). Mine only parallelizes API calls which needs a bigger number of workers to be fast and it can handle it because it's not an intensive operation.

[neginraoof]

i think this one does not parallelize generations, but just parallelizes the evaluation methods. But in any case, i'll merge because with curator we need to change both of these multithreading logics anyways. Thanks!