Allow daft.read_parquet take in custom statistics from the user

[raghumdani]

Is your feature request related to a problem? Please describe.
Yes, we have seen a tremendous amount of OOMs especially due to our lack of ability to accurately estimate file sizes when reading. We have devised a tracking system in our catalog that helps us hints what the in-memory size of the parquet file could be. Moreover, the execution plan changes the memory requirement in ways not very easy to estimate.

Describe the solution you'd like
It would be ideal if Daft allows users to specify hints on what inflation to use on top of size represented in total_uncompressed_size parquet metadata and to override exact in-memory size. The reason is that, in-memory size can be very much different from total_uncompressed_size that parquet calculates which is compute agnostic. Also, similar functionality to be supported for read_csv as well. The exact size data is already available in our catalog and is compliant with Iceberg: https://iceberg.apache.org/spec/#manifests.

# use-case 1
daft.read_parquet(file, inflation_multiplier=3.2)

# use-case 2
daft.read_parquet({name: 's3://file', stats_override: { in_memory_bytes_for_compute: "10202929239"}})

Describe alternatives you've considered

• Store the custom sizes in the schema of the parquet so that Daft can interpret. The problem with that is only Daft can interpret it.
• Use total_uncompressed_size which is not accurate.

Additional context
Let me know if this use-case makes sense as it will avoid OOMs and potentially allow us to just leverage Daft reader without having to perform the execution of a plan ourselves.

[samster25]

Work in progress PR to improve the in memory estimate (that can be configured)
#1901