[SPARK-50310][PYTHON] Add a flag to disable DataFrameQueryContext for PySpark

[itholic]

What changes were proposed in this pull request?

We disabled the DataFrameQueryContext from #48827, and we also need adding a flag for PySpark for the same performant reason.

Why are the changes needed?

To avoid the performance slowdown for the case when the DataFrameQueryContext too much stacked

Does this PR introduce any user-facing change?

No API changes, but the DataFrameQueryContext would no longer displayed when the flag is disabled

How was this patch tested?

Manually tested:

1. FLAG ON (almost 25sec)

>>> spark.conf.get("spark.sql.dataFrameDebugging.enabled")
'true'
>>> import time
>>> import pyspark.sql.functions as F
>>>
>>> c = F.col("name")
>>> start = time.time()
>>> for i in range(10000):
...   _ = c.alias("a")
...
>>> print(time.time() - start)
24.78217577934265

2. FLAG OFF (only 1 sec)

>>> spark.conf.get("spark.sql.dataFrameDebugging.enabled")
'false'
>>> import time
>>> import pyspark.sql.functions as F
>>>
>>> c = F.col("name")
>>> start = time.time()
>>> for i in range(10000):
...   _ = c.alias("a")
...
>>> print(time.time() - start)
1.0222370624542236

Was this patch authored or co-authored using generative AI tooling?

[HyukjinKwon]

spark.sql.dataFrameQueryContext.enabled has to be StaticSQLConf

[itholic]

I see. Should we update the existing conf to static conf or should we add a new one for PySpark specific?

The existing conf here is defined as SQLConf below:

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

Lines 5223 to 5230 in e55511c

	val DATA_FRAME_QUERY_CONTEXT_ENABLED = buildConf("spark.sql.dataFrameQueryContext.enabled")
	.internal()
	.doc(
	"Enable the DataFrame query context. This feature is enabled by default, but has a " +
	"non-trivial performance overhead because of the stack trace collection.")
	.version("4.0.0")
	.booleanConf
	.createWithDefault(true)

[HyukjinKwon]

I think so

[xinrong-meng]

A comment for the PR description #48827 doesn’t seem to disable “DataFrameQueryContext” if I understand correctly :)

[itholic]

A comment for the PR description #48827 doesn’t seem to disable “DataFrameQueryContext” if I understand correctly :)

Yes it doesn't disable the DataFrameQueryContext but just provide an option to disable the DataFrameQueryContext to PySpark users. Default behavior still collects the stack traces. Please let me know if I don't get the context clearly.

[itholic]

We add separate static conf for PySpark since we cannot change the behavior after enabling the config from Python side. also cc @HyukjinKwon @vladimirg-db