Api, Spark: Make StrictMetricsEvaluator not fail on nested column predicates

[zhongyujiang]

Currently, the StrictMetricsEvaluator fails when evaluating expressions with nested columns, causing Spark's DELETE FROM statement to throw an exception if the WHERE condition uses nested columns as predicates.

The StrictMetricsEvaluator requires null count data for columns during evaluation, but the null count data for nested columns collected in the current metadata might be incorrect (see #8611). Therefore, the StrictMetricsEvaluator cannot support the evaluation of filters on nested columns.

However, I think we can at least return ROWS_MIGHT_NOT_MATCH when encountering filters with nested columns instead of directly throwing exceptions and causing job failures. We are currently alreadt doing this in the evaluation of startsWith:

iceberg/api/src/main/java/org/apache/iceberg/expressions/StrictMetricsEvaluator.java

Lines 464 to 467 in 5922b4b

	public <T> Boolean startsWith(BoundReference<T> ref, Literal<T> lit) {
	return ROWS_MIGHT_NOT_MATCH;
	}

This fixes #7065.

@aokolnychyi @szehon-ho @RussellSpitzer @Fokko can you please take a look when you have time? Thanks.

[blakewhatley82]

This issue is still around in spark 3.5 and would really be a big capability to have for data that is all in structured format

[Leonti]

Same here, we have a big dataset and almost all of the data is in nested structs. Now we need to delete data based on the nested struct value and this issue is blocking us.
Would love to see this merged!

[amogh-jahagirdar]

I think this change is logically sound, thank you @zhongyujiang!
I'll want to double check though why this wasn't done originally, that's a bit intriguing to me. I'll take a look with fresh eyes tomorrow.

[amogh-jahagirdar]

Overall @zhongyujiang I think the change is good, just some cleanup in tests would be great before we get this in!

[zhongyujiang]

@amogh-jahagirdar Thanks for reviewing, tests updated.

[amogh-jahagirdar]

Thanks @zhongyujiang, from my side the changes look good. I'll give some time for others to review before merging

[nastra]

LGTM, thanks for fixing this @zhongyujiang

[blakewhatley82]

@zhongyujiang @nastra @amogh-jahagirdar great to see this merged! Is it known the timeline for 1.6.2 iceberg release and will this be for spark 3.3x or just spark 3.5x?

[zhongyujiang]

@blakewhatley82 I am not clear on the timeline for the 1.6.2 release. This fix is effective for all Spark versions.

[nastra]

The community is planning a 1.7.0 release end of October, so this will be shipped with that