[GLUTEN-6995][Core] Limit soft affinity duplicate reading detection max cache items

[zhli1142015]

What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)

Fixes: #6995

How was this patch tested?

UT.

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

• Other pull requests

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

#6995

[jackylee-ch]

This patch doesn't fix the performance problem for SoftAffinityListener, the Spark Events still can be discarded, which may cause user problem if they also use Listeners. We need a comment to explain this problem so that users can know the possible problems with this configuration when using it. BTW, maybe it's a better choice to change it to false by default, and users can turn it on as needed.

[zhli1142015]

This patch doesn't fix the performance problem for SoftAffinityListener, the Spark Events still can be discarded, which may cause user problem if they also use Listeners. We need a comment to explain this problem so that users can know the possible problems with this configuration when using it. BTW, maybe it's a better choice to change it to false by default, and users can turn it on as needed.

Do you mean is the SoftAffinityListener cause of the slowness, I don't observe this, can you help tp share how this is repro in your env?

[jackylee-ch]

Do you mean is the SoftAffinityListener cause of the slowness, I don't observe this, can you help tp share how this is repro in your env?

This is how I reproduced the problem locally.

1. Start spark-sql with --conf spark.hadoop.parquet.page.size=1024 --conf spark.hadoop.parquet.block.size=2048 --conf spark.sql.files.maxPartitionBytes=2048
2. Then, run below sqls.

create table test(a string) using parquet;
create table test1(a string) using parquet;
insert into test values(0); // make sure there is 10000 values in it
insert into test1 select /*+ REPARTITION(10000) */ * from test;

3. finally, restart spark-sql with SoftAffinity and run bellow sql

select count(*) from test1;

[zhli1142015]

Do you mean is the SoftAffinityListener cause of the slowness, I don't observe this, can you help tp share how this is repro in your env?

This is how I reproduced the problem locally.

1. Start spark-sql with --conf spark.hadoop.parquet.page.size=1024 --conf spark.hadoop.parquet.block.size=2048 --conf spark.sql.files.maxPartitionBytes=2048
2. Then, run below sqls.

create table test(a string) using parquet;
create table test1(a string) using parquet;
insert into test values(0); // make sure there is 10000 values in it
insert into test1 select /*+ REPARTITION(10000) */ * from test;

3. finally, restart spark-sql with SoftAffinity and run bellow sql

select count(*) from test1;

From above I actually can't repro the issue. I think maybe this is because of the hardware differences, but with more partitions, we actually observe the latency increasing. From the code, there are only the cache get / put operations in the event handling logic. I thought the memory pressure (GC) is more like the cause.
10K values

120k values

[jackylee-ch]

From above I actually can't repro the issue. I think maybe this is because of the hardware differences, but with more partitions, we actually observe the latency increasing.

Did you final get 10000+ FilePartitions for sql or only have one partition? Oh, there is one more information, sorry for I missed, that you need run select count(*) from test for 30 times in one application.

[github-actions]

Run Gluten Clickhouse CI

[zhli1142015]

From above I actually can't repro the issue. I think maybe this is because of the hardware differences, but with more partitions, we actually observe the latency increasing.

Did you final get 10000+ FilePartitions for sql or only have one partition?

6k+ partitions for 10K values, and 60k+ partitions for 120k values.

[jackylee-ch]

From above I actually can't repro the issue. I think maybe this is because of the hardware differences, but with more partitions, we actually observe the latency increasing.

Did you final get 10000+ FilePartitions for sql or only have one partition?

6k+ partitions for 10K values, and 60k+ partitions for 120k values.

Greate, could you try 30 sqls in one Application to check if you can reproduce?

[zhli1142015]

From above I actually can't repro the issue. I think maybe this is because of the hardware differences, but with more partitions, we actually observe the latency increasing.

Did you final get 10000+ FilePartitions for sql or only have one partition?

6k+ partitions for 10K values, and 60k+ partitions for 120k values.

Greate, could you try 30 sqls in one Application to check if you can reproduce?

OOM happens with 120K values, but there is no event dropped logs.

[zhli1142015]

OOM is not related, even SA is disable, it still happens.

[jackylee-ch]

OOM happens with 120K values, but there is no event dropped logs.

Hm, perhaps it is related to hardware performance. I tried ./bin/spark-sql --master local[192] with the default the gluten configs and above sqls, the problem can be stably reproduced in our environment.

BTW, maybe it's a better choice to change it to false by default, and users can turn it on as needed.

Since this pr has already closed the duplicate reading detection by default, It's ok to me now.

[jackylee-ch]

Thanks for your fix.