fix: Evaluations no longer associate incorrect inputs with predictions

[tssweeney]

I introduced a bug: #2434 which resulted in the predict_and_score (and child calls) having an incorrect input refs getting assigned to the input. The result is that the inputs are effectively all the same (however the outputs are different). This results in a number of odd effects since the system thinks all the predictions are against the same row:

So, what was the issue here? Well a classic case of mutating a closure variable before executing the function! duh!

For more details, consider this basic example:

for ndx, row in enumerate(self._prefetched_rows):
  next_id_future = wc.future_executor.defer(    # <--- this `defer` schedules the function to be ran in a new thread.
    lambda: cached_table_ref.row_digests[ndx]   # <---- this `ndx` is mutated by the outer iterator
  )

Since the execution of the function is delayed, by the time the inner function executes, ndx has been changed!. Changing parallelism to 0 fixes this (since the defer becomes an immediate blocking call).

I double checked everywhere and there are no other such cases of this pattern. Interestingly, @andrewtruong and I basically discussed this exact thing in extensive detail when he reviewed the original PR. We wrote tests and everything. I just missed this one call site!

[circle-job-mirror]

Preview this PR with FeatureBee: https://beta.wandb.ai/?betaVersion=c900c22c9e59ce4f36beee782a5ad39a4e3701dd