[VL] Add write IO metrics for WriteFiles

[Yohahaha]

No description provided.

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

• Other pull requests

[Yohahaha]

depends on #7002

[zhztheplayer]

Perhaps you missed the merge code?

[Yohahaha]

cc @PHILO-HE @liujiayi771 thank you!

[Yohahaha]

@JkSelf

[zhztheplayer]

@Yohahaha Can you confirm, there is a list for merging metric values https://github.com/Yohahaha/gluten/blob/f58163433b83453b772d76a158503bb1a77e28ad/gluten-data/src/main/scala/org/apache/gluten/metrics/MetricsUtil.scala#L129-L153 and whether we should append some code into that list.

[Yohahaha]

@Yohahaha Can you confirm, there is a list for merging metric values https://github.com/Yohahaha/gluten/blob/f58163433b83453b772d76a158503bb1a77e28ad/gluten-data/src/main/scala/org/apache/gluten/metrics/MetricsUtil.scala#L129-L153 and whether we should append some code into that list.

I see, TableWriter count physicalWrittenBytes, writeIONanos, numWrittenFiles, but only numWrittenFiles is merged.

why we need merge those metrics?

[zhztheplayer]

Sorry, I am not aside a dev PC now, so I can't actually reason about the code, which looks too magical to me by the way.

But what's clear is only input and output metrics were being handled separately when the code was firstly written .

TableWriter count physicalWrittenBytes, writeIONanos, numWrittenFiles, but only numWrittenFiles is merged

After all, I think we should make these 4 metrics consistent. Either merge them all, or none of them. And seems to be safe to merge them all by taking the first look. cc @JkSelf

[GlutenPerfBot]

===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====

query	log/native_master_08_28_2024_time.csv	log/native_master_08_26_2024_5db9de5601_time.csv	difference	percentage
q1	13.95	13.96	0.011	100.08%
q2	12.54	14.04	1.503	111.99%
q3	4.44	4.49	0.044	100.98%
q4	71.23	71.87	0.635	100.89%
q5	7.56	9.59	2.025	126.77%
q6	4.16	2.12	-2.044	50.89%
q7	7.17	4.91	-2.260	68.47%
q8	3.48	5.23	1.753	150.45%
q9	26.24	24.12	-2.126	91.90%
q10	10.18	9.91	-0.266	97.39%
q11	38.65	37.86	-0.795	97.94%
q12	1.50	1.38	-0.120	92.02%
q13	6.74	6.40	-0.335	95.02%
q14a	46.11	45.57	-0.544	98.82%
q14b	44.02	42.03	-1.995	95.47%
q15	2.68	2.77	0.094	103.52%
q16	45.81	46.12	0.309	100.67%
q17	4.98	4.94	-0.047	99.06%
q18	6.87	6.87	-0.001	99.99%
q19	2.06	2.20	0.140	106.82%
q20	2.37	1.69	-0.675	71.48%
q21	1.59	1.21	-0.381	76.01%
q22	7.66	8.08	0.419	105.47%
q23a	103.22	106.12	2.904	102.81%
q23b	127.17	129.49	2.319	101.82%
q24a	110.61	106.37	-4.241	96.17%
q24b	117.06	106.39	-10.667	90.89%
q25	4.22	4.25	0.023	100.54%
q26	3.24	3.12	-0.117	96.37%
q27	4.20	3.81	-0.392	90.67%
q28	33.62	33.27	-0.352	98.95%
q29	11.12	11.32	0.199	101.79%
q30	5.93	5.39	-0.538	90.92%
q31	7.24	7.27	0.034	100.47%
q32	1.15	1.33	0.173	115.03%
q33	4.16	4.62	0.452	110.85%
q34	3.74	4.36	0.613	116.37%
q35	7.91	8.97	1.057	113.36%
q36	4.75	5.49	0.741	115.59%
q37	4.54	5.67	1.128	124.83%
q38	13.80	13.29	-0.505	96.34%
q39a	3.64	3.15	-0.491	86.51%
q39b	3.18	2.81	-0.372	88.29%
q40	3.89	3.81	-0.079	97.97%
q41	0.65	0.64	-0.009	98.69%
q42	0.93	1.09	0.154	116.51%
q43	4.82	4.66	-0.164	96.60%
q44	9.90	9.90	0.002	100.02%
q45	3.36	3.25	-0.119	96.46%
q46	3.83	3.75	-0.085	97.78%
q47	18.72	18.83	0.114	100.61%
q48	5.36	5.30	-0.057	98.94%
q49	8.71	8.27	-0.443	94.91%
q50	21.70	21.62	-0.080	99.63%
q51	10.08	10.08	0.001	100.01%
q52	1.14	1.11	-0.036	96.87%
q53	2.42	2.39	-0.033	98.64%
q54	4.30	3.78	-0.514	88.04%
q55	1.09	1.06	-0.032	97.04%
q56	4.15	4.28	0.139	103.35%
q57	10.68	10.63	-0.043	99.59%
q58	2.40	2.42	0.024	101.01%
q59	11.03	11.09	0.058	100.52%
q60	4.20	4.04	-0.162	96.15%
q61	4.05	4.11	0.061	101.50%
q62	4.63	4.85	0.225	104.87%
q63	2.28	2.36	0.074	103.26%
q64	62.45	61.07	-1.379	97.79%
q65	17.13	17.84	0.703	104.11%
q66	3.94	3.89	-0.055	98.61%
q67	387.25	393.73	6.472	101.67%
q68	3.76	3.65	-0.111	97.05%
q69	5.37	8.32	2.942	154.74%
q70	12.39	11.28	-1.108	91.06%
q71	2.55	2.50	-0.048	98.10%
q72	217.13	215.87	-1.256	99.42%
q73	2.21	2.33	0.119	105.38%
q74	23.52	24.63	1.108	104.71%
q75	26.63	27.07	0.437	101.64%
q76	11.56	12.00	0.442	103.82%
q77	2.41	2.28	-0.133	94.50%
q78	49.47	50.11	0.636	101.29%
q79	4.15	3.86	-0.292	92.96%
q80	12.37	12.67	0.296	102.39%
q81	5.18	4.90	-0.282	94.56%
q82	7.12	7.21	0.081	101.14%
q83	1.72	1.61	-0.112	93.48%
q84	2.57	2.70	0.126	104.92%
q85	8.43	8.10	-0.330	96.08%
q86	4.04	4.06	0.025	100.61%
q87	15.33	14.83	-0.507	96.69%
q88	21.40	22.06	0.657	103.07%
q89	3.91	3.80	-0.115	97.07%
q90	3.28	3.25	-0.031	99.07%
q91	2.37	2.56	0.187	107.89%
q92	1.29	1.42	0.120	109.28%
q93	38.96	40.04	1.076	102.76%
q94	24.42	24.90	0.489	102.00%
q9	89.52	90.24	0.722	100.81%
q5	3.19	2.69	-0.497	84.40%
q96	17.51	17.69	0.186	101.06%
q97	1.95	1.95	0.003	100.13%
q98	10.61	10.41	-0.195	98.16%
q99	10.61	10.41	-0.195	98.16%
total	2197.98	2194.66	-3.318	99.85%