Skip to content
This repository has been archived by the owner on Aug 16, 2023. It is now read-only.

Sanity check crash if enable binary range search ut #988

Open
cydrain opened this issue Jul 12, 2023 · 3 comments
Open

Sanity check crash if enable binary range search ut #988

cydrain opened this issue Jul 12, 2023 · 3 comments
Assignees

Comments

@cydrain
Copy link
Collaborator

cydrain commented Jul 12, 2023

When enable binary range search ut in file test_search.cc, sanity check will crash with following error messages:

Address 0x7ffca61774d0 is located in stack of thread T0 at offset 3296 in frame
    #0 0x555e4af2f8fd in operator() /home/caiyd/vec/knowhere/tests/ut/test_search.cc:374

  This frame has 2 object(s):
    [48, 56) 'distrib' (line 376)
    [80, 5080) 'rng' (line 375) <== Memory access at offset 3296 is inside this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: unknown-crash (/home/caiyd/vec/knowhere/build/Debug/libknowhere.so+0x6fedbf) in knowhere::FlatIndexNode<faiss::IndexBinaryFlat>::RangeSearch(knowhere::DataSet const&, knowhere::Config const&, knowhere::BitsetView const&) const::{lambda()#1}::operator()() const
Shadow bytes around the buggy address:
  0x100014c26e40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100014c26e50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100014c26e60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100014c26e70: f1 f1 f1 f1 00 f2 f2 f2 00 f2 f2 f2 00 00 f2 f2
  0x100014c26e80: 00 00 00 00 00 f2 f2 f2 f2 f2 00 00 00 00 00 00
=>0x100014c26e90: 00 00 00 f3 f3 f3 f3 f3 00 00[00]00 00 00 00 00
  0x100014c26ea0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100014c26eb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100014c26ec0: 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1
  0x100014c26ed0: f1 f1 f8 f2 f8 f2 01 f2 00 f2 f2 f2 00 f2 f2 f2
  0x100014c26ee0: 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
Thread T9 created by T0 here:
    #0 0x7fe4a4fcf815 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cc:208
    #1 0x7fe4a1f7e0a8 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib/x86_64-linux-gnu/libstdc++.so.6+0xd70a8)
    #2 0x7fe4a42dc9aa in ctpl::thread_pool::set_thread(int) /home/caiyd/vec/knowhere/include/knowhere/comp/ctpl/ctpl-std.h:251
    #3 0x7fe4a42dba9b in ctpl::thread_pool::resize(int) /home/caiyd/vec/knowhere/include/knowhere/comp/ctpl/ctpl-std.h:115
    #4 0x7fe4a42db6fc in ctpl::thread_pool::thread_pool(int) /home/caiyd/vec/knowhere/include/knowhere/comp/ctpl/ctpl-std.h:78
    #5 0x7fe4a42e7beb in std::_MakeUniq<ctpl::thread_pool>::__single_object std::make_unique<ctpl::thread_pool, unsigned int&>(unsigned int&) /usr/include/c++/9/bits/unique_ptr.h:857
    #6 0x7fe4a42dcbb8 in knowhere::ThreadPool::ThreadPool(unsigned int) /home/caiyd/vec/knowhere/include/knowhere/comp/thread_pool.h:27
    #7 0x7fe4a431c444 in void __gnu_cxx::new_allocator<knowhere::ThreadPool>::construct<knowhere::ThreadPool, unsigned int&>(knowhere::ThreadPool*, unsigned int&) /usr/include/c++/9/ext/new_allocator.h:146
    #8 0x7fe4a431a6d1 in void std::allocator_traits<std::allocator<knowhere::ThreadPool> >::construct<knowhere::ThreadPool, unsigned int&>(std::allocator<knowhere::ThreadPool>&, knowhere::ThreadPool*, unsigned int&) /usr/include/c++/9/bits/alloc_traits.h:483
    #9 0x7fe4a4317342 in std::_Sp_counted_ptr_inplace<knowhere::ThreadPool, std::allocator<knowhere::ThreadPool>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<unsigned int&>(std::allocator<knowhere::ThreadPool>, unsigned int&) (/home/caiyd/vec/knowhere/build/Debug/libknowhere.so+0x6bb342)
    #10 0x7fe4a4310600 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<knowhere::ThreadPool, std::allocator<knowhere::ThreadPool>, unsigned int&>(knowhere::ThreadPool*&, std::_Sp_alloc_shared_tag<std::allocator<knowhere::ThreadPool> >, unsigned int&) /usr/include/c++/9/bits/shared_ptr_base.h:679
    #11 0x7fe4a430ae2c in std::__shared_ptr<knowhere::ThreadPool, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<knowhere::ThreadPool>, unsigned int&>(std::_Sp_alloc_shared_tag<std::allocator<knowhere::ThreadPool> >, unsigned int&) /usr/include/c++/9/bits/shared_ptr_base.h:1344
    #12 0x7fe4a42ffac0 in std::shared_ptr<knowhere::ThreadPool>::shared_ptr<std::allocator<knowhere::ThreadPool>, unsigned int&>(std::_Sp_alloc_shared_tag<std::allocator<knowhere::ThreadPool> >, unsigned int&) (/home/caiyd/vec/knowhere/build/Debug/libknowhere.so+0x6a3ac0)
    #13 0x7fe4a42f3826 in std::shared_ptr<knowhere::ThreadPool> std::allocate_shared<knowhere::ThreadPool, std::allocator<knowhere::ThreadPool>, unsigned int&>(std::allocator<knowhere::ThreadPool> const&, unsigned int&) /usr/include/c++/9/bits/shared_ptr.h:702
    #14 0x7fe4a42e7d79 in std::shared_ptr<knowhere::ThreadPool> std::make_shared<knowhere::ThreadPool, unsigned int&>(unsigned int&) /usr/include/c++/9/bits/shared_ptr.h:718
    #15 0x7fe4a42dd0ab in knowhere::ThreadPool::GetGlobalThreadPool() /home/caiyd/vec/knowhere/include/knowhere/comp/thread_pool.h:91
    #16 0x7fe4a42b1b9b in knowhere::BruteForce::Search(std::shared_ptr<knowhere::DataSet>, std::shared_ptr<knowhere::DataSet>, nlohmann::json_abi_v3_11_2::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_11_2::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> > > const&, knowhere::BitsetView const&) /home/caiyd/vec/knowhere/src/common/comp/brute_force.cc:56
    #17 0x555e4ad745da in CATCH2_INTERNAL_TEST_0 /home/caiyd/vec/knowhere/tests/ut/test_bruteforce.cc:41
    #18 0x555e4afcf747 in Catch::TestInvokerAsFunction::invoke() const src/catch2/internal/catch_test_case_registry_impl.cpp:149
    #19 0x555e4afc37e8 in Catch::TestCaseHandle::invoke() const src/catch2/../catch2/catch_test_case_info.hpp:115
    #20 0x555e4afc28dd in Catch::RunContext::invokeActiveTestCase() src/catch2/internal/catch_run_context.cpp:541
    #21 0x555e4afc25ef in Catch::RunContext::runCurrentTest(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) src/catch2/internal/catch_run_context.cpp:504
    #22 0x555e4afc0bc8 in Catch::RunContext::runTest(Catch::TestCaseHandle const&) src/catch2/internal/catch_run_context.cpp:235
    #23 0x555e4af6e82e in execute src/catch2/catch_session.cpp:110
    #24 0x555e4af6fc9f in Catch::Session::runInternal() src/catch2/catch_session.cpp:332
    #25 0x555e4af6f7b8 in Catch::Session::run() src/catch2/catch_session.cpp:263
    #26 0x555e4af6a4ae in int Catch::Session::run<char>(int, char const* const*) src/catch2/../catch2/catch_session.hpp:41
    #27 0x555e4af6a39c in main src/catch2/internal/catch_main.cpp:36
    #28 0x7fe4a1b6d082 in __libc_start_main ../csu/libc-start.c:308

==2767259==ABORTING
@cydrain
Copy link
Collaborator Author

cydrain commented Jul 12, 2023

crash.log

@cydrain
Copy link
Collaborator Author

cydrain commented Jul 12, 2023

/assign @hhy3

@hhy3
Copy link
Member

hhy3 commented Jul 18, 2023

It seems to be caused by exception handling problem of thread pool.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants