Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: [benchmark][retrieve] standalone Milvus panic in concurrent(=100) query retrieve varchar(length=100) scene #38731

Closed
1 task done
wangting0128 opened this issue Dec 25, 2024 · 2 comments
Closed
1 task done

[Bug]: [benchmark][retrieve] standalone Milvus panic in concurrent(=100) query retrieve varchar(length=100) scene #38731

wangting0128 opened this issue Dec 25, 2024 · 2 comments
Assignees
@sunby
Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.5.2

Comments

@wangting0128
Copy link
Contributor

wangting0128 commented Dec 25, 2024 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:sunby-no_unlink-e76862d-20241217
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):rocksmq    
- SDK version(e.g. pymilvus v2.0.0rc2):2.5.0rc124
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: fouramf-9n89w

server:

NAME                                                              READY   STATUS                   RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
wt-test-unlink-1-etcd-0                                           1/1     Running                  0               44h     10.104.18.54    4am-node25   <none>           <none>
wt-test-unlink-1-milvus-standalone-56b457d5f5-wt2x2               1/1     Running                  3 (12h ago)     44h     10.104.26.153   4am-node32   <none>           <none>
wt-test-unlink-1-minio-b964b9cbf-rpkpk                            1/1     Running                  0               44h     10.104.18.53    4am-node25   <none>           <none>
截屏2024-12-25 11 33 42

panic logs:

[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803070079746180] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069945200271] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069156933482] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069612036020] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069560779528] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803070073339238] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803070150222579] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069913165532] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803070252733648] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803070201478102] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069676107120] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069906758589] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069304295705] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069682514217] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069291481477] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069599221722] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069970828067] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069329924097] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069233818333] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803070002862819] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069515929789] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069714549708] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [delegator/exclude_info.go:49] ["add exclude info"] [segmentID=454803069105677191] [ts=454833573753257986]
[2024/12/24 15:00:25.812 +00:00] [INFO] [delegator/distribution.go:313] ["Update readable segment version"] [partitions="[454803069067241171]"] [oldVersion=1735052405813399982] [newVersion=1735052415813767711] [growingSegmentNum=0] [sealedSegmentNum=188]
[2024/12/24 15:00:25.812 +00:00] [INFO] [observers/target_observer.go:521] ["observer trigger update current target"] [collectionID=454803069067241170]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [meta/target_manager.go:106] ["start to update current target for collection"] [collectionID=454803069067241170]
[2024/12/24 15:00:25.812 +00:00] [DEBUG] [meta/target_manager.go:130] ["finish to update current target for collection"] [collectionID=454803069067241170] [segments="[454803069458265969,454803069791433526,454803069535151301,454803069361959643,454803069772212597,454803069496708165,454803070054118408,454803069265853083,454803069701735536,454803069285074351,454803070137408681,454803069740177788,454803070047711461,454803069644071661,454803069240225227,454803069785026553,454803069586407511,454803069746584722,454803069919572481,454803069131305277,454803070009269776,454803069541558452,454803069509322613,454803069759398647,454803069714549708,454803069394195486,454803069951607218,454803070086153121,454803069554372542,454803070022083691,454803069547965530,454803069221004309,454803070259140594,454803069663292934,454803069733770796,454803069842689155,454803069272260161,454803070073339238,454803069407009616,454803069958014167,454803070092560057,454803069278667240,454803070066932297,454803069727363777,454803069169747650,454803069432637868,454803069932386377,454803070268554016,454803069983641983,454803069368366794,454803069823468329,454803069490301073,454803070233512808,454803069445451944,454803069099270190,454803069086456189,454803069246632109,454803069387788339,454803069336331192,454803070002862819,454803069503115352,454803069477486998,454803069612036020,454803069214597230,454803069964421113,454803069253039010,454803069708142629,454803070028490636,454803070111780873,454803069881130783,454803069310702815,454803070239919760,454803069342738302,454803070227105868,454803069144119317,454803069304295705,454803070214291983,454803070246326701,454803069483894006,454803069893944691,454803069182561794,454803069669700036,454803069092863188,454803069874723841,454803069631257373,454803069451858958,454803069291481477,454803069945200271,454803069567186516,454803070195071179,454803069188968859,454803069580000472,454803069624850259,454803070265547546,454803069605628826,454803070220698929,454803069156933482,454803069913165532,454803069829875270,454803069259446029,454803070163036448,454803069163340560,454803069124898265,454803069105677191,454803069528744074,454803070124594776,454803069073642201,454803070182257290,454803070175850341,454803069650478757,454803070201478102,454803069381381163,454803070143815628,454803069695328438,454803070079746180,454803069938793327,454803069977235026,454803069849096082,454803069317109891,454803069150526383,454803070034897583,454803069137712288,454803069419823657,454803069208190166,454803070150222579,454803069323516988,454803069195375964,454803070015676733,454803070131001737,454803069560779528,454803069112084217,454803069439044903,454803069861909980,454803069688921321,454803069297888583,454803069201783059,454803069900351640,454803069810654417,454803069797840486,454803070207885039,454803069656885836,454803069720956758,454803069515929789,454803069592814599,454803069329924097,454803070060525355,454803070156629503,454803069426230676,454803069682514217,454803069804247462,454803070118187824,454803070041304522,454803070105373933,454803069413416639,454803069522336793,454803069778619572,454803070188664238,454803069599221722,454803069176154721,454803069400602573,454803070098966994,454803069573593499,454803069080049195,454803069676107120,454803069118491242,454803069887537737,454803069471079995,454803069374973981,454803069464672969,454803069906758589,454803069765805621,454803069355552538,454803069990048927,454803069996455873,454803069752991678,454803069868316892,454803070252733648,454803069618443135,454803069227411367,454803069637664509,454803069925979430,454803069233818333,454803069855503030,454803069836282209,454803070169443392,454803069970828067,454803069817061372,454803069349145425]"] [channels="[by-dev-rootcoord-dml_1_454803069067241170v0]"] [version=1735052415813767711] [partStatsVersion="partitionStats:by-dev-rootcoord-dml_1_454803069067241170v0:[454803069067241171:0,],"]
[2024/12/24 15:00:25.813 +00:00] [INFO] [datacoord/services.go:859] ["get recovery info request received"] [traceID=053fd2194be6baddbd5b6ee232b4e6c9] [collectionID=454803069067241170] [partitionIDs="[]"]
[2024/12/24 15:00:25.813 +00:00] [INFO] [datacoord/handler.go:244] [GetQueryVChanPositions] [collectionID=454803069067241170] [channel=by-dev-rootcoord-dml_1_454803069067241170v0] [numOfSegments=241] ["result flushed"=188] ["result growing"=0] ["result L0"=0] ["partition stats"="{\"454803069067241171\":0}"]
[2024/12/24 15:00:25.813 +00:00] [INFO] [datacoord/handler.go:348] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_1_454803069067241170v0] [posTs=454833573753257986] [posTime=2024/12/24 14:59:50.111 +00:00]
[2024/12/24 15:00:25.813 +00:00] [INFO] [datacoord/services.go:874] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=053fd2194be6baddbd5b6ee232b4e6c9] [collectionID=454803069067241170] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_1_454803069067241170v0] ["# of unflushed segments"=0] ["# of flushed segments"=188] ["# of dropped segments"=53] ["# of indexed segments"=0] ["# of l0 segments"=0]
[2024/12/24 15:00:25.813 +00:00] [DEBUG] [meta/target_manager.go:201] ["finish to update next targets for collection"] [collectionID=454803069067241170] [PartitionIDs="[454803069067241171]"]
operator()
	/go/src/github.com/milvus-io/milvus/internal/core/src/segcore/segment_c.cpp:225 pc=0x7f32ee7d3a0f
operator()
	/go/src/github.com/milvus-io/milvus/internal/core/src/futures/Future.h:181 pc=0x7f32ee7d3a0f
makeTryWithNoUnwrap<const milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda()> >
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/Try-inl.h:254 pc=0x7f32ee7d3a0f
makeTryWith<const milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda()> >
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/Try-inl.h:276 pc=0x7f32ee7d3a0f
setWith<const milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda()> >
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/SharedPromise-inl.h:81 pc=0x7f32ee7d3a0f
operator()<folly::Try<folly::Unit> >
	/go/src/github.com/milvus-io/milvus/internal/core/src/futures/Future.h:187 pc=0x7f32ee7d3a0f
operator()
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/Future-inl.h:941 pc=0x7f32ee7d3a0f
invoke<folly::Executor::KeepAlive<folly::Executor>, folly::Try<folly::Unit> >
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/Future-inl.h:139 pc=0x7f32ee7d3a0f
invoke<folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void>, folly::futures::detail::CoreCallbackState<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)> >, folly::Unit>
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/Future-inl.h:341 pc=0x7f32ee7d3a0f
operator()
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/Future-inl.h:388 pc=0x7f32ee7d3a0f
makeTryWithNoUnwrap<folly::futures::detail::FutureBase<folly::Unit>::thenImplementation<folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void> >(folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>&&, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void>, folly::futures::detail::InlineContinuation)::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)> mutable::<lambda()> >
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/Try-inl.h:265 pc=0x7f32ee7d3a0f
makeTryWith<folly::futures::detail::FutureBase<folly::Unit>::thenImplementation<folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void> >(folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>&&, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void>, folly::futures::detail::InlineContinuation)::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)> mutable::<lambda()> >
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/Try-inl.h:276 pc=0x7f32ee7d3a0f
operator()
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/Future-inl.h:387 pc=0x7f32ee7d3a0f
operator()
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/detail/Core.h:629 pc=0x7f32ee7d3a0f
callBig<folly::futures::detail::Core<folly::Unit>::setCallback<folly::futures::detail::FutureBase<folly::Unit>::thenImplementation<folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void> >(folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>&&, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void>, folly::futures::detail::InlineContinuation)::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)> >(folly::futures::detail::FutureBase<folly::Unit>::thenImplementation<folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void> >(folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>&&, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void>, folly::futures::detail::InlineContinuation)::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>&&, std::shared_ptr<folly::RequestContext>&&, folly::futures::detail::InlineContinuation)::<lambda(folly::futures::detail::CoreBase&, folly::Executor::KeepAlive<>&&, folly::exception_wrapper*)> >
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/Function.h:359 pc=0x7f32ee7d3a0f
_ZN5folly6detail8function14FunctionTraitsIFvRNS_7futures6detail8CoreBaseEONS_8Executor9KeepAliveIS7_EEPNS_17exception_wrapperEEEclES6_SA_SC_
	/root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Function.h:374 pc=0x7f32e9f52d3f
operator()
	/root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/futures/detail/Core.cpp:616 pc=0x7f32e9f52d3f
operator()
	/root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Executor.h:187 pc=0x7f32e9f52d3f
callSmall<folly::Executor::KeepAlive<>::add<folly::futures::detail::CoreBase::doCallback(folly::Executor::KeepAlive<>&&, folly::futures::detail::State)::<lambda(folly::Executor::KeepAlive<>&&)> >(folly::futures::detail::CoreBase::doCallback(folly::Executor::KeepAlive<>&&, folly::futures::detail::State)::<lambda(folly::Executor::KeepAlive<>&&)>&&) &&::<lambda()> >
	/root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Function.h:345 pc=0x7f32e9f52d3f
_ZN5folly6detail8function14FunctionTraitsIFvvEEclEv
	/root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Function.h:374 pc=0x7f32e9e72e75
_ZN5folly15catch_exceptionIRNS_8FunctionIFvvEEERDoFvPKcEJRS6_EvEET2_OT_OT0_DpOT1_
	/root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/lang/Exception.h:286 pc=0x7f32e9e72e75
_ZN5folly8Executor18invokeCatchingExnsINS_8FunctionIFvvEEEEEvPKcT_
	/root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Executor.h:234 pc=0x7f32e9e72e75
_ZN5folly18ThreadPoolExecutor7runTaskERKSt10shared_ptrINS0_6ThreadEEONS0_4TaskE
	/root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/executors/ThreadPoolExecutor.cpp:102 pc=0x7f32e9e72e75
_ZN5folly21CPUThreadPoolExecutor9threadRunESt10shared_ptrINS_18ThreadPoolExecutor6ThreadEE
	/root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/executors/CPUThreadPoolExecutor.cpp:333 pc=0x7f32e9e35e74
_ZSt13__invoke_implIvRMN5folly18ThreadPoolExecutorEFvSt10shared_ptrINS1_6ThreadEEERPS1_JRS4_EET_St21__invoke_memfun_derefOT0_OT1_DpOT2_
	/usr/include/c++/12/bits/invoke.h:74 pc=0x7f32e9e7544b
_ZSt8__invokeIRMN5folly18ThreadPoolExecutorEFvSt10shared_ptrINS1_6ThreadEEEJRPS1_RS4_EENSt15__invoke_resultIT_JDpT0_EE4typeEOSC_DpOSD_
	/usr/include/c++/12/bits/invoke.h:96 pc=0x7f32e9e7544b
_ZNSt5_BindIFMN5folly18ThreadPoolExecutorEFvSt10shared_ptrINS1_6ThreadEEEPS1_S4_EE6__callIvJEJLm0ELm1EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
	/usr/include/c++/12/functional:495 pc=0x7f32e9e7544b
_ZNSt5_BindIFMN5folly18ThreadPoolExecutorEFvSt10shared_ptrINS1_6ThreadEEEPS1_S4_EEclIJEvEET0_DpOT_
	/usr/include/c++/12/functional:580 pc=0x7f32e9e7544b
_ZN5folly6detail8function14FunctionTraitsIFvvEE9callSmallISt5_BindIFMNS_18ThreadPoolExecutorEFvSt10shared_ptrINS7_6ThreadEEEPS7_SA_EEEEvRNS1_4DataE
	/root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Function.h:345 pc=0x7f32e9e7544b
(null)
	(null):0 pc=0x7f32ebc1c252
(null)
	(null):0 pc=0x7f32ebe00ac2
(null)
	(null):0 pc=0x7f32ebe9284f
(null)
	(null):0 pc=0xffffffffffffffff

client error log: 2024-12-24 15:00:26,770 ~ 2024-12-24 15:09:08,556

Expected Behavior

No response

Steps To Reproduce

1. create a collection with fields: 'id'(INT64, primary key), 'float_vector'(3dim), 'varchar_100'
2. build FLAT index on field float_vector
3. insert 900m data
4. flush collection
5. load collection
6. concurrent requests query retrieve varchar_100 <- concurrent 100, and milvus panic

Milvus Log

No response

Anything else?

test result:

{'server': {'deploy_tool': '',
            'deploy_mode': '',
            'config_name': '',
            'config': {},
            'host': 'wt-test-unlink-1-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_concurrent_locust_custom_parameters',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 3,
                                                    'max_length': 101,
                                                    'varchar_filled': True,
                                                    'dataset_name': 'local',
                                                    'dataset_size': '900m',
                                                    'ni_per': 50000},
                                 'collection_params': {'other_fields': ['varchar_100'], 'shards_num': 1, 'collection_name': 'scene_varchar_100'},
                                 'index_params': {'index_type': 'FLAT', 'index_param': {}},
                                 'concurrent_params': {'concurrent_number': 100, 'during_time': '30m', 'interval': 20, 'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'query',
                                                       'weight': 1,
                                                       'params': {'expr': '',
                                                                  'output_fields': ['varchar_100'],
                                                                  'limit': 10,
                                                                  'timeout': 600,
                                                                  'random_data': True,
                                                                  'random_count': 10,
                                                                  'random_range': [0, 900000000],
                                                                  'field_name': 'id',
                                                                  'field_type': 'int64'}}]},
            'run_id': 2024122424787695,
            'datetime': '2024-12-24 03:54:38.131492',
            'client_version': '2.2'},
 'result': {'test_result': {'index': {'RT': 19.6642},
                            'insert': {'total_time': 2382.3471, 'VPS': 377778.704, 'batch_time': 0.1324, 'batch': 50000},
                            'flush': {'RT': 3.0173},
                            'load': {'RT': 469.0147},
                            'Locust': {'Aggregated': {'Requests': 2421,
                                                      'Fails': 1211,
                                                      'RPS': 1.35,
                                                      'fail_s': 0.5,
                                                      'RT_max': 189903.26,
                                                      'RT_avg': 72074.93,
                                                      'TP50': 101000.0,
                                                      'TP99': 188000.0},
                                       'query': {'Requests': 2421,
                                                 'Fails': 1211,
                                                 'RPS': 1.35,
                                                 'fail_s': 0.5,
                                                 'RT_max': 189903.26,
                                                 'RT_avg': 72074.93,
                                                 'TP50': 101000.0,
                                                 'TP99': 188000.0}}}}}
@wangting0128 wangting0128 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 25, 2024
@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 25, 2024
@yanliang567 yanliang567 added this to the 2.5.1 milestone Dec 25, 2024
@yanliang567 yanliang567 removed their assignment Dec 25, 2024
@yanliang567 yanliang567 modified the milestones: 2.5.1, 2.5.2 Dec 30, 2024
@chyezh
Copy link
Contributor

chyezh commented Dec 30, 2024

Addressed by ASAN.

2024-12-27 15:48:27.719	=================================================================
2024-12-27 15:48:27.719	==8==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000314ce8 at pc 0x7f1ae38a31e3 bp 0x7f186e4990a0 sp 0x7f186e499090
2024-12-27 15:48:27.719	READ of size 8 at 0x603000314ce8 thread T1241
2024-12-27 15:48:29.953	    #0 0x7f1ae38a31e2 in milvus::segcore::SegmentInternalInterface::FillTargetEntry(milvus::tracer::TraceContext*, milvus::query::RetrievePlan const*, std::unique_ptr<milvus::proto::segcore::RetrieveResults, std::default_delete<milvus::proto::segcore::RetrieveResults> > const&, long const*, long, bool, bool) const /workspace/source/internal/core/src/segcore/SegmentInterface.cpp:157
2024-12-27 15:48:29.953	    #1 0x7f1ae38ae118 in milvus::segcore::SegmentInternalInterface::Retrieve(milvus::tracer::TraceContext*, milvus::query::RetrievePlan const*, long const*, long) const /workspace/source/internal/core/src/segcore/SegmentInterface.cpp:245
2024-12-27 15:48:29.953	    #2 0x7f1ae3b336ae in operator() /workspace/source/internal/core/src/segcore/segment_c.cpp:225
2024-12-27 15:48:29.953	    #3 0x7f1ae3b336ae in operator() /workspace/source/internal/core/src/futures/Future.h:181
2024-12-27 15:48:29.953	    #4 0x7f1ae3b336ae in makeTryWithNoUnwrap<const milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda()> > /root/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/Try-inl.h:254
2024-12-27 15:48:29.953	    #5 0x7f1ae3b336ae in makeTryWith<const milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda()> > /root/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/Try-inl.h:276
2024-12-27 15:48:29.953	    #6 0x7f1ae3b336ae in setWith<const milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda()> > /root/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/SharedPromise-inl.h:81
2024-12-27 15:48:29.953	    #7 0x7f1ae3b336ae in operator()<folly::Try<folly::Unit> > /workspace/source/internal/core/src/futures/Future.h:187
2024-12-27 15:48:29.953	    #8 0x7f1ae3b336ae in operator() /root/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/Future-inl.h:941
2024-12-27 15:48:29.953	    #9 0x7f1ae3b336ae in invoke<folly::Executor::KeepAlive<folly::Executor>, folly::Try<folly::Unit> > /root/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/Future-inl.h:139
2024-12-27 15:48:29.953	    #10 0x7f1ae3b336ae in invoke<folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void>, folly::futures::detail::CoreCallbackState<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)> >, folly::Unit> /root/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/Future-inl.h:341
2024-12-27 15:48:29.953	    #11 0x7f1ae3b336ae in operator() /root/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/Future-inl.h:388
2024-12-27 15:48:29.953	    #12 0x7f1ae3b336ae in makeTryWithNoUnwrap<folly::futures::detail::FutureBase<folly::Unit>::thenImplementation<folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void> >(folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>&&, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void>, folly::futures::detail::InlineContinuation)::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)> mutable::<lambda()> > /root/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/Try-inl.h:265
2024-12-27 15:48:29.953	    #13 0x7f1ae3b336ae in makeTryWith<folly::futures::detail::FutureBase<folly::Unit>::thenImplementation<folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void> >(folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>&&, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void>, folly::futures::detail::InlineContinuation)::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)> mutable::<lambda()> > /root/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/Try-inl.h:276
2024-12-27 15:48:29.953	    #14 0x7f1ae3b336ae in operator() /root/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/Future-inl.h:387
2024-12-27 15:48:29.953	    #15 0x7f1ae3b336ae in operator() /root/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/detail/Core.h:629
2024-12-27 15:48:29.953	    #16 0x7f1ae3b336ae in callBig<folly::futures::detail::Core<folly::Unit>::setCallback<folly::futures::detail::FutureBase<folly::Unit>::thenImplementation<folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void> >(folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>&&, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void>, folly::futures::detail::InlineContinuation)::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)> >(folly::futures::detail::FutureBase<folly::Unit>::thenImplementation<folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void> >(folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>&&, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<CProto>::asyncProduce<AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncRetrieveByOffsets(CTraceContext, CSegmentInterface, CRetrievePlan, int64_t*, int64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void>, folly::futures::detail::InlineContinuation)::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>&&, std::shared_ptr<folly::RequestContext>&&, folly::futures::detail::InlineContinuation)::<lambda(folly::futures::detail::CoreBase&, folly::Executor::KeepAlive<>&&, folly::exception_wrapper*)> > /root/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/Function.h:359
2024-12-27 15:48:29.953	    #17 0x7f1ada857d3f in folly::detail::function::FunctionTraits<void (folly::futures::detail::CoreBase&, folly::Executor::KeepAlive<folly::Executor>&&, folly::exception_wrapper*)>::operator()(folly::futures::detail::CoreBase&, folly::Executor::KeepAlive<folly::Executor>&&, folly::exception_wrapper*) /root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Function.h:374
2024-12-27 15:48:29.953	    #18 0x7f1ada857d3f in operator() /root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/futures/detail/Core.cpp:616
2024-12-27 15:48:29.953	    #19 0x7f1ada857d3f in operator() /root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Executor.h:187
2024-12-27 15:48:29.953	    #20 0x7f1ada857d3f in callSmall<folly::Executor::KeepAlive<>::add<folly::futures::detail::CoreBase::doCallback(folly::Executor::KeepAlive<>&&, folly::futures::detail::State)::<lambda(folly::Executor::KeepAlive<>&&)> >(folly::futures::detail::CoreBase::doCallback(folly::Executor::KeepAlive<>&&, folly::futures::detail::State)::<lambda(folly::Executor::KeepAlive<>&&)>&&) &&::<lambda()> > /root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Function.h:345
2024-12-27 15:48:29.953	    #21 0x7f1ada777e75 in folly::detail::function::FunctionTraits<void ()>::operator()() /root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Function.h:374
2024-12-27 15:48:29.953	    #22 0x7f1ada777e75 in void folly::catch_exception<folly::Function<void ()>&, void (&)(char const*) noexcept, char const*&, void>(folly::Function<void ()>&, void (&)(char const*) noexcept, char const*&) /root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/lang/Exception.h:286
2024-12-27 15:48:29.953	    #23 0x7f1ada777e75 in void folly::Executor::invokeCatchingExns<folly::Function<void ()> >(char const*, folly::Function<void ()>) /root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Executor.h:234
2024-12-27 15:48:29.953	    #24 0x7f1ada777e75 in folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&) /root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/executors/ThreadPoolExecutor.cpp:102
2024-12-27 15:48:29.953	    #25 0x7f1ada73ae74 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) /root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/executors/CPUThreadPoolExecutor.cpp:333
2024-12-27 15:48:29.953	    #26 0x7f1ada77a44b in void std::__invoke_impl<void, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&>(std::__invoke_memfun_deref, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&) /usr/include/c++/12/bits/invoke.h:74
2024-12-27 15:48:29.953	    #27 0x7f1ada77a44b in std::__invoke_result<void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&>::type std::__invoke<void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&>(void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&) /usr/include/c++/12/bits/invoke.h:96
2024-12-27 15:48:29.953	    #28 0x7f1ada77a44b in void std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) /usr/include/c++/12/functional:495
2024-12-27 15:48:29.953	    #29 0x7f1ada77a44b in void std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)>::operator()<, void>() /usr/include/c++/12/functional:580
2024-12-27 15:48:29.953	    #30 0x7f1ada77a44b in void folly::detail::function::FunctionTraits<void ()>::callSmall<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) /root/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Function.h:345
2024-12-27 15:48:29.953	    #31 0x7f1ae4eaa139 in folly::detail::function::FunctionTraits<void ()>::operator()() /root/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/Function.h:374
2024-12-27 15:48:29.953	    #32 0x7f1ae4eaa139 in folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}::operator()() /root/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/executors/thread_factory/NamedThreadFactory.h:40
2024-12-27 15:48:29.953	    #33 0x7f1ae4eaa139 in void std::__invoke_impl<void, folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>(std::__invoke_other, folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}&&) /usr/include/c++/12/bits/invoke.h:61
2024-12-27 15:48:29.953	    #34 0x7f1ae4eaa139 in std::__invoke_result<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>::type std::__invoke<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>(folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}&&) /usr/include/c++/12/bits/invoke.h:96
2024-12-27 15:48:29.953	    #35 0x7f1ae4eaa139 in void std::thread::_Invoker<std::tuple<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/include/c++/12/bits/std_thread.h:279
2024-12-27 15:48:29.953	    #36 0x7f1ae4eaa139 in std::thread::_Invoker<std::tuple<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}> >::operator()() /usr/include/c++/12/bits/std_thread.h:286
2024-12-27 15:48:29.953	    #37 0x7f1ae4eaa139 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}> > >::_M_run() /usr/include/c++/12/bits/std_thread.h:231
2024-12-27 15:48:29.953	    #38 0x7f1adfe53252  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc252)
2024-12-27 15:48:29.953	    #39 0x7f1adfadbac2  (/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
2024-12-27 15:48:29.953	    #40 0x7f1adfb6d84f  (/lib/x86_64-linux-gnu/libc.so.6+0x12684f)
2024-12-27 15:48:29.953	
2024-12-27 15:48:29.953	0x603000314ce8 is located 8 bytes inside of 32-byte region [0x603000314ce0,0x603000314d00)
2024-12-27 15:48:29.953	freed by thread T1237 here:
2024-12-27 15:48:30.025	    #0 0x7f1ae89d8540 in operator delete(void*, unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:164
2024-12-27 15:48:30.025	    #1 0x7f1ae3a2e489 in std::__new_allocator<fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable> >::deallocate(fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable>*, unsigned long) /usr/include/c++/12/bits/new_allocator.h:158
2024-12-27 15:48:30.025	    #2 0x7f1ae3a2e489 in std::allocator_traits<std::allocator<fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable> > >::deallocate(std::allocator<fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable> >&, fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable>*, unsigned long) /usr/include/c++/12/bits/alloc_traits.h:496
2024-12-27 15:48:30.025	    #3 0x7f1ae3a2e489 in std::_Vector_base<fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable>, std::allocator<fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable> > >::_M_deallocate(fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable>*, unsigned long) /usr/include/c++/12/bits/stl_vector.h:387
2024-12-27 15:48:30.025	    #4 0x7f1ae3a2e489 in std::_Vector_base<fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable>, std::allocator<fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable> > >::~_Vector_base() /usr/include/c++/12/bits/stl_vector.h:366
2024-12-27 15:48:30.025	    #5 0x7f1ae3a2e489 in std::vector<fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable>, std::allocator<fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable> > >::~vector() /usr/include/c++/12/bits/stl_vector.h:733
2024-12-27 15:48:30.025	    #6 0x7f1ae3a2e489 in milvus::query::RetrievePlan::~RetrievePlan() /workspace/source/internal/core/src/query/PlanImpl.h:96
2024-12-27 15:48:30.025	    #7 0x7f1ae3a2e489 in DeleteRetrievePlan /workspace/source/internal/core/src/segcore/plan_c.cpp:171
2024-12-27 15:48:30.025	    #8 0x2018943 in runtime.asmcgocall /go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:918
2024-12-27 15:48:30.025	
2024-12-27 15:48:30.025	previously allocated by thread T1237 here:
2024-12-27 15:48:30.211	    #0 0x7f1ae89d7498 in operator new(unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:95
2024-12-27 15:48:30.211	    #1 0x7f1ae3417196 in std::__new_allocator<fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable> >::allocate(unsigned long, void const*) /usr/include/c++/12/bits/new_allocator.h:137
2024-12-27 15:48:30.211	    #2 0x7f1ae3417196 in std::allocator_traits<std::allocator<fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable> > >::allocate(std::allocator<fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable> >&, unsigned long) /usr/include/c++/12/bits/alloc_traits.h:464
2024-12-27 15:48:30.211	    #3 0x7f1ae3417196 in std::_Vector_base<fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable>, std::allocator<fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable> > >::_M_allocate(unsigned long) /usr/include/c++/12/bits/stl_vector.h:378
2024-12-27 15:48:30.211	    #4 0x7f1ae3417196 in void std::vector<fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable>, std::allocator<fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable> > >::_M_realloc_insert<fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable> const&>(__gnu_cxx::__normal_iterator<fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable>*, std::vector<fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable>, std::allocator<fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable> > > >, fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable> const&) /usr/include/c++/12/bits/vector.tcc:453
2024-12-27 15:48:30.211	    #5 0x7f1ae33ff010 in std::vector<fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable>, std::allocator<fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable> > >::push_back(fluent::NamedType<long, milvus::impl::FieldIdTag, fluent::Comparable, fluent::Hashable> const&) /usr/include/c++/12/bits/stl_vector.h:1287
2024-12-27 15:48:30.211	    #6 0x7f1ae33ff010 in milvus::query::ProtoParser::CreateRetrievePlan(milvus::proto::plan::PlanNode const&) /workspace/source/internal/core/src/query/PlanProto.cpp:271
2024-12-27 15:48:30.211	    #7 0x7f1ae33c291e in milvus::query::CreateRetrievePlanByExpr(milvus::Schema const&, void const*, long) /workspace/source/internal/core/src/query/Plan.cpp:119
2024-12-27 15:48:30.211	    #8 0x7f1ae3a2e24e in CreateRetrievePlanByExpr /workspace/source/internal/core/src/segcore/plan_c.cpp:145
2024-12-27 15:48:30.211	    #9 0x60eda47 in _cgo_64992fa11e30_Cfunc_CreateRetrievePlanByExpr /tmp/go-build/cgo-gcc-prolog:57
2024-12-27 15:48:30.211	    #10 0x2018943 in runtime.asmcgocall /go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:918

The reduce operation in retrieve operation doesn't promise all future ready.

func MergeSegcoreRetrieveResults(ctx context.Context, retrieveResults []*segcorepb.RetrieveResults, param *mergeParam, segments []Segment, plan *RetrievePlan, manager *Manager) (*segcorepb.RetrieveResults, error) {
...
		if err := conc.AwaitAll(futures...); err != nil {
			return nil, err
		}

This was referenced Dec 30, 2024
Merged
Merged
Merged
sre-ci-robot pushed a commit that referenced this issue Dec 30, 2024
@chyezh
fix: the retrieve plan on heap is used after free when reduce (#38841)
04f07f0 
issue: #38731
pr: #38840

Signed-off-by: chyezh <[email protected]>
sre-ci-robot pushed a commit that referenced this issue Dec 30, 2024
@chyezh
fix: the retrieve plan on heap is used after free when reduce (#38842)
253860b 
issue: #38731
pr: #38840

Signed-off-by: chyezh <[email protected]>
sre-ci-robot pushed a commit that referenced this issue Dec 30, 2024
@chyezh
fix: the retrieve plan on heap is used after free when reduce (#38840)
07c1f43 
issue: #38731

Signed-off-by: chyezh <[email protected]>
@wangting0128
Copy link
Contributor Author

verification passed

image: master-20241231-ba0c7b8f-amd64

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sunby sunby

Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.5.2
Development

No branches or pull requests
4 participants
@sunby @chyezh @wangting0128 @yanliang567