Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

agg spill query kill and hang forever #57546

Open
wshwsh12 opened this issue Nov 20, 2024 · 6 comments · May be fixed by #57641
Open

agg spill query kill and hang forever #57546

wshwsh12 opened this issue Nov 20, 2024 · 6 comments · May be fixed by #57641
Labels
affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. impact/leak severity/major sig/execution SIG execution type/bug The issue is confirmed as a bug.

Comments

@wshwsh12
Copy link
Contributor

wshwsh12 commented Nov 20, 2024 •
edited
Loading

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Run all sqls some times.

drop table if exists t1, t2;
create table t1(a int, b int);
create table t2(a int, b int);
insert into t1 value(5,5);
insert into t2 value(1,1);
set tidb_mem_quota_query=1000;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=1100;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=1200;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=1300;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=1400;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=1500;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=1600;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=1700;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=1800;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=1900;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=2000;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=2100;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=2200;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=2300;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=2400;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=2500;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=2600;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=2700;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=2800;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=2900;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=3000;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=3100;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=3200;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=3300;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=3400;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=3500;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=3600;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=3700;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=3800;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=3900;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=4000;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=4100;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=4200;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=4300;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=4400;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=4500;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=4600;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=4700;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=4800;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=4900;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=5000;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=5100;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=5200;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=5300;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=5400;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=5500;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=5600;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=5700;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=5800;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=5900;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=6000;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=6100;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=6200;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=6300;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=6400;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=6500;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=6600;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=6700;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=6800;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=6900;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);
set tidb_mem_quota_query=7000;
update t2 set b=2 where a in (with recursive cte1 as (select 1 as a union all select cte1.a+1 from t1 join cte1 on t1.a > cte1.a) select * from cte1);

2. What did you expect to see? (Required)

Can kill the query.

3. What did you see instead (Required)

hang forever.

goroutine 30792 [semacquire, 6 minutes]:
sync.runtime_Semacquire(0x0?)
	/gosdk/go1.23.0/src/runtime/sema.go:71 +0x25
sync.(*WaitGroup).Wait(0xc057bfb4f0?)
	/gosdk/go1.23.0/src/sync/waitgroup.go:118 +0x48
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggFinalWorker).run(0xc0599d2af8, {0x735b800, 0xc011ddbb08}, 0xc059a16300, 0xc059a162f0)
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_final_worker.go:228 +0x9c
created by github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggExec).prepare4ParallelExec in goroutine 30777
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_executor.go:590 +0x357

goroutine 30793 [semacquire, 6 minutes]:
sync.runtime_Semacquire(0x0?)
	/gosdk/go1.23.0/src/runtime/sema.go:71 +0x25
sync.(*WaitGroup).Wait(0xc059a1a310?)
	/gosdk/go1.23.0/src/sync/waitgroup.go:118 +0x48
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggFinalWorker).run(0xc0599d2b70, {0x735b800, 0xc011ddbb08}, 0xc059a16300, 0xc059a162f0)
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_final_worker.go:228 +0x9c
created by github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggExec).prepare4ParallelExec in goroutine 30777
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_executor.go:590 +0x357

goroutine 30794 [semacquire, 6 minutes]:
sync.runtime_Semacquire(0x0?)
	/gosdk/go1.23.0/src/runtime/sema.go:71 +0x25
sync.(*WaitGroup).Wait(0xc012f67f60?)
	/gosdk/go1.23.0/src/sync/waitgroup.go:118 +0x48
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggFinalWorker).run(0xc0599d2be8, {0x735b800, 0xc011ddbb08}, 0xc059a16300, 0xc059a162f0)
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_final_worker.go:228 +0x9c
created by github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggExec).prepare4ParallelExec in goroutine 30777
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_executor.go:590 +0x357

goroutine 1271 [select, 2 minutes]:
github.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue.(*AnalysisPriorityQueue).run(0xc01136f320)
	/home/wshwsh12/project/tidb/pkg/statistics/handle/autoanalyze/priorityqueue/queue.go:314 +0x190
github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run.func1()
	/home/wshwsh12/project/tidb/pkg/util/wait_group_wrapper.go:157 +0x4c
created by github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run in goroutine 551
	/home/wshwsh12/project/tidb/pkg/util/wait_group_wrapper.go:155 +0x73

goroutine 30783 [semacquire, 6 minutes]:
sync.runtime_Semacquire(0xc0599affb0?)
	/gosdk/go1.23.0/src/runtime/sema.go:71 +0x25
sync.(*WaitGroup).Wait(0xc01367a240?)
	/gosdk/go1.23.0/src/sync/waitgroup.go:118 +0x48
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggExec).spillIfNeed(0xc0599b6848)
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_executor.go:512 +0x45
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggExec).fetchChildData(0xc0599b6848, {0x72c16e8, 0xc0599dbe60}, 0xc057a591c0?)
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_executor.go:496 +0x232
created by github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggExec).prepare4ParallelExec in goroutine 30777
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_executor.go:558 +0xc5

goroutine 30787 [chan receive, 6 minutes]:
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggPartialWorker).finalizeWorkerProcess(0xc059a14440, 0x0, 0x5, 0x0)
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_partial_worker.go:174 +0x6c
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggPartialWorker).run.func1()
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_partial_worker.go:201 +0xa5
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggPartialWorker).run(0xc059a14440, {0x735b800, 0xc011ddbb08}, 0xc059a162f0, 0x5)
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_partial_worker.go:216 +0xff
created by github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggExec).prepare4ParallelExec in goroutine 30777
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_executor.go:576 +0x168

goroutine 1803 [chan receive, 6 minutes]:
github.com/pingcap/tidb/pkg/util/channel.Clear[...](...)
	/home/wshwsh12/project/tidb/pkg/util/channel/channel.go:20
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggExec).Close(0xc0599b6848)
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_executor.go:199 +0x591
github.com/pingcap/tidb/pkg/executor/internal/exec.Close({0x72fa700, 0xc0599b6848})
	/home/wshwsh12/project/tidb/pkg/executor/internal/exec/executor.go:480 +0xdb
github.com/pingcap/tidb/pkg/executor/internal/exec.(*BaseExecutorV2).Close(0xc0599b4b00?)
	/home/wshwsh12/project/tidb/pkg/executor/internal/exec/executor.go:315 +0x52
github.com/pingcap/tidb/pkg/executor/join.(*HashJoinV2Exec).Close(0xc0599a0480)
	/home/wshwsh12/project/tidb/pkg/executor/join/hash_join_v2.go:715 +0x3ea
github.com/pingcap/tidb/pkg/executor/internal/exec.Close({0x72fb5a0, 0xc0599a0480})
	/home/wshwsh12/project/tidb/pkg/executor/internal/exec/executor.go:480 +0xdb
github.com/pingcap/tidb/pkg/executor.(*UpdateExec).Close(0xc0599b3400)
	/home/wshwsh12/project/tidb/pkg/executor/update.go:475 +0x19e
github.com/pingcap/tidb/pkg/executor/internal/exec.Close({0x72fa160, 0xc0599b3400})
	/home/wshwsh12/project/tidb/pkg/executor/internal/exec/executor.go:480 +0xdb
github.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelayExecutor.func1()
	/home/wshwsh12/project/tidb/pkg/executor/adapter.go:996 +0x25
github.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelayExecutor(0xc0599850e0, {0x72c16e8?, 0xc0599da7e0?}, {0x72fa160, 0xc0599b3400})
	/home/wshwsh12/project/tidb/pkg/executor/adapter.go:1017 +0x3dc
github.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelay(0xc0599850e0, {0x72c16e8, 0xc0599da7e0}, {0x72fa160, 0xc0599b3400}, 0x0)
	/home/wshwsh12/project/tidb/pkg/executor/adapter.go:848 +0x239
github.com/pingcap/tidb/pkg/executor.(*ExecStmt).Exec(0xc0599850e0, {0x72c16e8, 0xc0599da7e0})
	/home/wshwsh12/project/tidb/pkg/executor/adapter.go:611 +0xef8
github.com/pingcap/tidb/pkg/session.runStmt({0x72c16e8?, 0xc05986abd0?}, 0xc011ddbb08, {0x72d5260, 0xc0599850e0})
	/home/wshwsh12/project/tidb/pkg/session/session.go:2288 +0x31b
github.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt(0xc011ddbb08, {0x72c16e8?, 0xc05986abd0?}, {0x72dc7b0, 0xc0599aa0b0})
	/home/wshwsh12/project/tidb/pkg/session/session.go:2150 +0x10de
github.com/pingcap/tidb/pkg/server.(*TiDBContext).ExecuteStmt(0xc013664438, {0x72c16e8, 0xc05986abd0}, {0x72dc7b0, 0xc0599aa0b0})
	/home/wshwsh12/project/tidb/pkg/server/driver_tidb.go:291 +0x97
github.com/pingcap/tidb/pkg/server.(*clientConn).handleStmt(0xc01363b6c0, {0x72c1720, 0xc059825900}, {0x72dc7b0, 0xc0599aa0b0}, {0xc0177c5660, 0x0, 0x1}, 0x1)
	/home/wshwsh12/project/tidb/pkg/server/conn.go:2026 +0x2a5
github.com/pingcap/tidb/pkg/server.(*clientConn).handleQuery(0xc01363b6c0, {0x72c1720, 0xc059825900}, {0xc05015b361, 0x95})
	/home/wshwsh12/project/tidb/pkg/server/conn.go:1779 +0xb36
github.com/pingcap/tidb/pkg/server.(*clientConn).dispatch(0xc01363b6c0, {0x72c16e8, 0xc0136c0ab0}, {0xc05015b360, 0x96, 0x96})
	/home/wshwsh12/project/tidb/pkg/server/conn.go:1378 +0xf85
github.com/pingcap/tidb/pkg/server.(*clientConn).Run(0xc01363b6c0, {0x72c16e8, 0xc0136c0ab0})
	/home/wshwsh12/project/tidb/pkg/server/conn.go:1147 +0x528
github.com/pingcap/tidb/pkg/server.(*Server).onConn(0xc001d18500, 0xc01363b6c0)
	/home/wshwsh12/project/tidb/pkg/server/server.go:741 +0x81f
created by github.com/pingcap/tidb/pkg/server.(*Server).startNetworkListener in goroutine 623
	/home/wshwsh12/project/tidb/pkg/server/server.go:560 +0x674

goroutine 30790 [semacquire, 6 minutes]:
sync.runtime_Semacquire(0x0?)
	/gosdk/go1.23.0/src/runtime/sema.go:71 +0x25
sync.(*WaitGroup).Wait(0xc057bfb3b0?)
	/gosdk/go1.23.0/src/sync/waitgroup.go:118 +0x48
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggFinalWorker).run(0xc0599d2a08, {0x735b800, 0xc011ddbb08}, 0xc059a16300, 0xc059a162f0)
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_final_worker.go:228 +0x9c
created by github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggExec).prepare4ParallelExec in goroutine 30777
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_executor.go:590 +0x357

goroutine 30785 [chan receive, 6 minutes]:
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggPartialWorker).finalizeWorkerProcess(0xc059a14170, 0x0, 0x5, 0x0)
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_partial_worker.go:174 +0x6c
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggPartialWorker).run.func1()
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_partial_worker.go:201 +0xa5
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggPartialWorker).run(0xc059a14170, {0x735b800, 0xc011ddbb08}, 0xc059a162f0, 0x5)
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_partial_worker.go:216 +0xff
created by github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggExec).prepare4ParallelExec in goroutine 30777
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_executor.go:576 +0x168

goroutine 30786 [chan receive, 6 minutes]:
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggPartialWorker).finalizeWorkerProcess(0xc059a142d8, 0x0, 0x5, 0x0)
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_partial_worker.go:174 +0x6c
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggPartialWorker).run.func1()
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_partial_worker.go:201 +0xa5
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggPartialWorker).run(0xc059a142d8, {0x735b800, 0xc011ddbb08}, 0xc059a162f0, 0x5)
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_partial_worker.go:216 +0xff
created by github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggExec).prepare4ParallelExec in goroutine 30777
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_executor.go:576 +0x168

goroutine 30796 [semacquire, 6 minutes]:
sync.runtime_Semacquire(0xc056fb8380?)
	/gosdk/go1.23.0/src/runtime/sema.go:71 +0x25
sync.(*WaitGroup).Wait(0xc012f677d0?)
	/gosdk/go1.23.0/src/sync/waitgroup.go:118 +0x48
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggExec).waitAllWorkersAndCloseFinalOutputCh(...)
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_executor.go:550
created by github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggExec).prepare4ParallelExec in goroutine 30777
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_executor.go:602 +0x529

goroutine 30795 [semacquire, 6 minutes]:
sync.runtime_Semacquire(0xab1c060?)
	/gosdk/go1.23.0/src/runtime/sema.go:71 +0x25
sync.(*WaitGroup).Wait(0xc012f66768?)
	/gosdk/go1.23.0/src/sync/waitgroup.go:118 +0x48
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggExec).prepare4ParallelExec.func2()
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_executor.go:594 +0x3b
created by github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggExec).prepare4ParallelExec in goroutine 30777
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_executor.go:593 +0x4b1

goroutine 101574 [IO wait]:
internal/poll.runtime_pollWait(0x70f0d9677a00, 0x72)
	/gosdk/go1.23.0/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc015754800?, 0xc015750a31?, 0x0)
	/gosdk/go1.23.0/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/gosdk/go1.23.0/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc015754800, {0xc015750a31, 0x1, 0x1})
	/gosdk/go1.23.0/src/internal/poll/fd_unix.go:165 +0x27a
net.(*netFD).Read(0xc015754800, {0xc015750a31?, 0xc02f8ab6d8?, 0x0?})
	/gosdk/go1.23.0/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc04ffc6b88, {0xc015750a31?, 0x0?, 0x0?})
	/gosdk/go1.23.0/src/net/net.go:189 +0x45
github.com/soheilhy/cmux.(*bufferedReader).Read(0xc01574ac50, {0xc015750a31, 0x58e3f82?, 0x1})
	/home/wshwsh12/go/pkg/mod/github.com/soheilhy/[email protected]/buffer.go:53 +0x12f
github.com/soheilhy/cmux.(*MuxConn).Read(0xc02f8ab7a0?, {0xc015750a31?, 0x72c1720?, 0xc0158a0d20?})
	/home/wshwsh12/go/pkg/mod/github.com/soheilhy/[email protected]/cmux.go:297 +0x1e
net/http.(*connReader).backgroundRead(0xc015750a20)
	/gosdk/go1.23.0/src/net/http/server.go:690 +0x37
created by net/http.(*connReader).startBackgroundRead in goroutine 101573
	/gosdk/go1.23.0/src/net/http/server.go:686 +0xb6

goroutine 30788 [chan receive, 6 minutes]:
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggPartialWorker).finalizeWorkerProcess(0xc059a145a8, 0x0, 0x5, 0x0)
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_partial_worker.go:174 +0x6c
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggPartialWorker).run.func1()
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_partial_worker.go:201 +0xa5
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggPartialWorker).run(0xc059a145a8, {0x735b800, 0xc011ddbb08}, 0xc059a162f0, 0x5)
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_partial_worker.go:216 +0xff
created by github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggExec).prepare4ParallelExec in goroutine 30777
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_executor.go:576 +0x168

goroutine 30789 [semacquire, 6 minutes]:
sync.runtime_Semacquire(0xc057bfb270?)
	/gosdk/go1.23.0/src/runtime/sema.go:71 +0x25
sync.(*WaitGroup).Wait(0x0?)
	/gosdk/go1.23.0/src/sync/waitgroup.go:118 +0x48
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggExec).waitPartialWorkerAndCloseOutputChs(0xc0599b6848, 0xc0577cea88?)
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_executor.go:538 +0x1f
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggExec).prepare4ParallelExec.func1()
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_executor.go:580 +0x3f
created by github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggExec).prepare4ParallelExec in goroutine 30777
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_executor.go:579 +0x2e5

goroutine 30791 [semacquire, 6 minutes]:
sync.runtime_Semacquire(0x0?)
	/gosdk/go1.23.0/src/runtime/sema.go:71 +0x25
sync.(*WaitGroup).Wait(0xc057bfb450?)
	/gosdk/go1.23.0/src/sync/waitgroup.go:118 +0x48
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggFinalWorker).run(0xc0599d2a80, {0x735b800, 0xc011ddbb08}, 0xc059a16300, 0xc059a162f0)
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_final_worker.go:228 +0x9c
created by github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggExec).prepare4ParallelExec in goroutine 30777
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_executor.go:590 +0x357

goroutine 30784 [chan receive, 6 minutes]:
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggPartialWorker).finalizeWorkerProcess(0xc059a14008, 0x0, 0x5, 0x0)
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_partial_worker.go:174 +0x6c
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggPartialWorker).run.func1()
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_partial_worker.go:201 +0xa5
panic({0x6429be0?, 0xc058ff4360?})
	/gosdk/go1.23.0/src/runtime/panic.go:785 +0x132
github.com/pingcap/tidb/pkg/util/memory.(*PanicOnExceed).Action(0xc059878640, 0xc01367a240)
	/home/wshwsh12/project/tidb/pkg/util/memory/action.go:180 +0x479
github.com/pingcap/tidb/pkg/executor/join.(*hashJoinSpillAction).triggerFallBackAction(0xc0590c6160?, 0xc01367a240)
	/home/wshwsh12/project/tidb/pkg/executor/join/hash_join_spill.go:60 +0x2d
github.com/pingcap/tidb/pkg/executor/join.(*hashJoinSpillAction).Action(0xc0590c6160, 0xc01367a240)
	/home/wshwsh12/project/tidb/pkg/executor/join/hash_join_spill.go:54 +0xb3
github.com/pingcap/tidb/pkg/util/chunk.(*baseSpillDiskAction).action(0xc058af2f60, 0xc01367a240, {0x729af98, 0xc0598738c0})
	/home/wshwsh12/project/tidb/pkg/util/chunk/row_container.go:468 +0x198
github.com/pingcap/tidb/pkg/util/chunk.(*SpillDiskAction).Action(0xc058af3020?, 0xa?)
	/home/wshwsh12/project/tidb/pkg/util/chunk/row_container.go:405 +0x25
github.com/pingcap/tidb/pkg/util/chunk.(*baseSpillDiskAction).action(0xc058af3020, 0xc01367a240, {0x729af98, 0xc0590c6120})
	/home/wshwsh12/project/tidb/pkg/util/chunk/row_container.go:468 +0x198
github.com/pingcap/tidb/pkg/util/chunk.(*SpillDiskAction).Action(0xc058af3080?, 0x211ae05?)
	/home/wshwsh12/project/tidb/pkg/util/chunk/row_container.go:405 +0x25
github.com/pingcap/tidb/pkg/util/chunk.(*baseSpillDiskAction).action(0xc058af3080, 0xc01367a240, {0x729af98, 0xc058accd80})
	/home/wshwsh12/project/tidb/pkg/util/chunk/row_container.go:468 +0x198
github.com/pingcap/tidb/pkg/util/chunk.(*SpillDiskAction).Action(0xc058f2f860?, 0xc04a35ff80?)
	/home/wshwsh12/project/tidb/pkg/util/chunk/row_container.go:405 +0x25
github.com/pingcap/tidb/pkg/util/chunk.(*baseSpillDiskAction).action(0xc058f2f860, 0xc01367a240, {0x729af98, 0xc058f369a0})
	/home/wshwsh12/project/tidb/pkg/util/chunk/row_container.go:468 +0x198
github.com/pingcap/tidb/pkg/util/chunk.(*SpillDiskAction).Action(0xffffffffffffff?, 0x0?)
	/home/wshwsh12/project/tidb/pkg/util/chunk/row_container.go:405 +0x25
github.com/pingcap/tidb/pkg/util/memory.(*Tracker).Consume.func1(0xc01367a248, 0xc01367a240)
	/home/wshwsh12/project/tidb/pkg/util/memory/tracker.go:485 +0xf7
github.com/pingcap/tidb/pkg/util/memory.(*Tracker).Consume(0xc02ec37e40?, 0x104)
	/home/wshwsh12/project/tidb/pkg/util/memory/tracker.go:512 +0x242
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggPartialWorker).getChildInput(0xc059a14008)
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_partial_worker.go:85 +0x1ab
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggPartialWorker).fetchChunkAndProcess(0xc059a14008, {0x735b800, 0xc011ddbb08}, 0xc018905f46, 0xc018905f45)
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_partial_worker.go:102 +0x99
github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggPartialWorker).run(0xc059a14008, {0x735b800, 0xc011ddbb08}, 0xc059a162f0, 0x5)
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_partial_worker.go:214 +0xec
created by github.com/pingcap/tidb/pkg/executor/aggregate.(*HashAggExec).prepare4ParallelExec in goroutine 30777
	/home/wshwsh12/project/tidb/pkg/executor/aggregate/agg_hash_executor.go:576 +0x168

4. What is your TiDB version? (Required)

master
@wshwsh12 wshwsh12 added the type/bug The issue is confirmed as a bug. label Nov 20, 2024
@wshwsh12 wshwsh12 changed the title kill cte query and hang agg spill query kill and hang forever Nov 20, 2024
@xzhangxian1008 xzhangxian1008 linked a pull request Nov 22, 2024 that will close this issue
Open
13 tasks
@xzhangxian1008
Copy link
Contributor

/remove-label may-affects-5.4

@ti-chi-bot ti-chi-bot bot removed the may-affects-5.4 This bug maybe affects 5.4.x versions. label Nov 22, 2024
@xzhangxian1008
Copy link
Contributor

/remove-label may-affects-6.1

@xzhangxian1008
Copy link
Contributor

/remove-label may-affects-6.5

@xzhangxian1008
Copy link
Contributor

/remove-label may-affects-7.1

@xzhangxian1008
Copy link
Contributor

/remove-label may-affects-7.5

@xzhangxian1008
Copy link
Contributor

/label affects-8.1

@ti-chi-bot ti-chi-bot bot added affects-8.1 This bug affects the 8.1.x(LTS) versions. and removed may-affects-8.1 labels Nov 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. impact/leak severity/major sig/execution SIG execution type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

executor: fix hang in hash agg when exceeding memory limit leads to panic xzhangxian1008/tidb
3 participants
@jebter @wshwsh12 @xzhangxian1008