[FEAT]: shuffle_join_default_partitions param

[universalmind303]

addresses #2817

[codspeed-hq]

CodSpeed Performance Report

Merging #2844 will degrade performances by 77.08%

_{Comparing universalmind303:default_join_partitions (22a0720) with main (78a92a2)}

Summary

❌ 1 regressions
✅ 15 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	main	universalmind303:default_join_partitions	Change
❌	test_explain[100 Small Files]	33 ms	143.9 ms	-77.08%

[jaychia]

LGTM, we should see if this affects our local TPC-H benchmarks too

[jaychia]

Just chatted with @samster25

We should revert this PR since this quite negatively impacts all of our benchmarks:

• It's not possible a priori to know what a good default here would be, since a user could have a query that may, in fact, benefit from having a smaller partitioning than 200 (as in the case of our benchmarks).
• The best way to solve this would be through AQE, but we don't yet have support for that.

cc @samster25 for thoughts.

I'll create a PR for the revert? LMK what you all think @universalmind303 @colin-ho

[jaychia]

To be clear, we probably need a more sophisticated mechanism here for determining the number of partitions. This includes:

• Statistics which can tell us estimated cardinality/partition sizes at planning-time
• AQE which can give us the actual cardinality/partition sizes at runtime

Both of which we don't currently have in Daft... And thus this approach might be too naive for us to use right now 😭

Sorry @universalmind303 I know this was quite a bit of wasted effort on your end.