Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash in BeaconResponseHandler #155

Open
mdavidsaver opened this issue Sep 30, 2019 · 2 comments
Open

Crash in BeaconResponseHandler #155

mdavidsaver opened this issue Sep 30, 2019 · 2 comments
Assignees
@mdavidsaver
Labels
bug

Comments

@mdavidsaver
Copy link
Member

mdavidsaver commented Sep 30, 2019 •
edited
Loading

@bhill-slac reports a crash in the Beacon handling code. This was observed in a gateway process, but does not appear to be gateway specific (though I'm not certain of this).

In the captured stack trace, three threads are not idle.


Thread 1 (LWP 1214):
#0  0x00007f7e14d32207 in setlocale () from /lib64/libc.so.6
#1  0x00007f7cfc001930 in ?? ()
#2  0x00007f7db4000d00 in ?? ()
#3  0x00007f7dfc515ab0 in ?? ()
#4  0x00007f7e0ce42f47 in __cxxabiv1::__terminate (handler=<optimized out>)
    at /home/nwani/m3/conda-bld/compilers_linux-64_1560109574129/work/.build/x86_64-conda_cos6-linux-gnu/src/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:47
#5  0x00007f7e0ce42f7d in std::terminate ()
    at /home/nwani/m3/conda-bld/compilers_linux-64_1560109574129/work/.build/x86_64-conda_cos6-linux-gnu/src/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:57
#6  0x00007f7e0ce43988 in __cxxabiv1::__cxa_pure_virtual ()
    at /home/nwani/m3/conda-bld/compilers_linux-64_1560109574129/work/.build/x86_64-conda_cos6-linux-gnu/src/gcc/libstdc++-v3/libsupc++/pure.cc:50
#7  0x00007f7e0d2f4589 in epics::pvData::compare (a=..., b=...) at ../../src/factory/Compare.cpp:86
#8  0x00007f7e0d2b2cfa in epics::pvData::FieldCreate::Helper::cache<epics::pvData::Structure> (
    create=create@entry=0x55992b468310, ent=...) at ../../src/factory/FieldCreateFactory.cpp:63
#9  0x00007f7e0d2abf8a in epics::pvData::FieldCreate::createStructure (this=this@entry=0x55992b468310, fieldNames=..., 
    fields=...) at ../../src/factory/FieldCreateFactory.cpp:1305
#10 0x00007f7e0d2af789 in deserializeStructureField (control=0x55992b987d80, buffer=0x55992b987ea0, 
    fieldCreate=0x55992b468310) at ../../src/factory/FieldCreateFactory.cpp:251
#11 epics::pvData::FieldCreate::deserialize (this=0x55992b468310, buffer=buffer@entry=0x55992b987ea0, 
    control=control@entry=0x55992b987d80) at ../../src/factory/FieldCreateFactory.cpp:1468
#12 0x00007f7e0d664039 in (anonymous namespace)::BeaconResponseHandler::handleResponse (this=<optimized out>, 
    responseFrom=0x7f7dfc515d90, transport=..., version=<optimized out>, command=<optimized out>, 
    payloadSize=<optimized out>, payloadBuffer=0x55992b987ea0) at ../../src/remoteClient/clientContextImpl.cpp:2776
#13 0x00007f7e0d65eb8a in (anonymous namespace)::ClientResponseHandler::handleResponse (this=<optimized out>, 
    responseFrom=<optimized out>, transport=..., version=<optimized out>, command=<optimized out>, 
    payloadSize=<optimized out>, payloadBuffer=0x55992b987ea0) at ../../src/remoteClient/clientContextImpl.cpp:3024
#14 0x00007f7e0d63b51f in epics::pvAccess::BlockingUDPTransport::processBuffer (this=this@entry=0x55992b987d80, 
    transport=..., fromAddress=..., receiveBuffer=receiveBuffer@entry=0x55992b987ea0)
    at ../../src/remote/blockingUDPTransport.cpp:422
#15 0x00007f7e0d63c58b in epics::pvAccess::BlockingUDPTransport::run (this=0x55992b987d80)
    at ../../src/remote/blockingUDPTransport.cpp:271
#16 0x00007f7e0cfd2949 in epicsThreadCallEntryPoint (pPvt=0x55992b9880e0) at ../../src/osi/epicsThread.cpp:83
#17 0x00007f7e0cfd88dc in start_routine (arg=0x55992b988810) at ../../src/osi/os/posix/osdThread.c:403
#18 0x00007f7e150d0dd5 in start_thread () from /lib64/libpthread.so.0
#19 0x00007f7e14df9ead in tdestroy_recurse () from /lib64/libc.so.6
#20 0x0000000000000000 in ?? ()



Thread 70 (LWP 1212):
#0  0x00007f7e150d74ed in __lll_timedwait_tid () from /lib64/libpthread.so.0
#1  0x0000000037efb61d in ?? ()
#2  0x00000000317c54fa in ?? ()
#3  0x00007f7dfc917a30 in ?? ()
#4  0x00007f7dfc9179c0 in ?? ()
#5  0x00007f7dfc917a30 in ?? ()
#6  0x00007f7e0cfda3a5 in epicsThreadOnce (id=0x55992b4683a0, func=
    0x7f7e0cfd5979 <epicsTimeFromTimespec(epicsTimeStamp*, timespec const*)+25>, arg=0x7f7e150d2de6 <_L_lock_870+15>)
    at ../../src/osi/os/posix/osdThread.c:510
#7  0x00007f7e0d2a578b in epics::pvData::FieldCreate::getFieldCreate () at ../../src/factory/FieldCreateFactory.cpp:1592
#8  0x00007f7e0d2a835f in Lock (m=..., this=<synthetic pointer>) at ../../src/misc/pv/lock.h:45
#9  epics::pvData::Field::~Field (this=0x7f7e0cfd3acc <epicsMutex::lock()+12>, __vtt_parm=<optimized out>, 
    __in_chrg=<optimized out>) at ../../src/factory/FieldCreateFactory.cpp:92
#10 0x00007f7e0d2a9a9e in epics::pvData::Structure::~Structure (this=this@entry=0x7f7cfc001930, 
    __in_chrg=<optimized out>, __vtt_parm=<optimized out>) at ../../src/factory/FieldCreateFactory.cpp:527
#11 0x00007f7e0d2a9b09 in epics::pvData::Structure::~Structure (this=0x7f7cfc001930, __in_chrg=<optimized out>, 
    __vtt_parm=<optimized out>) at ../../src/factory/FieldCreateFactory.cpp:527
#12 0x00007f7e0d2b4a5a in _M_release (this=0x7f7cfc00f8a0) at /usr/include/c++/4.8.2/tr1/shared_ptr.h:141
#13 ~__shared_count (this=0x7f7cfc00edf0, __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/tr1/shared_ptr.h:341
#14 ~__shared_ptr (this=0x7f7cfc00ede8, __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/tr1/shared_ptr.h:541
#15 ~shared_ptr (this=0x7f7cfc00ede8, __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/tr1/shared_ptr.h:985
#16 epics::pvData::PVField::~PVField (this=0x7f7cfc00edc0, __vtt_parm=<optimized out>, __in_chrg=<optimized out>)
    at ../../src/factory/PVField.cpp:38
#17 0x00007f7e0d2b6d28 in epics::pvData::PVStructure::~PVStructure (this=this@entry=0x7f7cfc00edc0, 
    __in_chrg=<optimized out>, __vtt_parm=<optimized out>) at ../../src/factory/PVStructure.cpp:63
#18 0x00007f7e0d2b6ea9 in epics::pvData::PVStructure::~PVStructure (this=0x7f7cfc00edc0, __in_chrg=<optimized out>, 
    __vtt_parm=<optimized out>) at ../../src/factory/PVStructure.cpp:63
#19 0x00007f7e0d948f89 in std::tr1::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7f7cfc000b70)
    at /usr/include/c++/4.8.2/tr1/shared_ptr.h:141
#20 0x00007f7e0d664111 in ~__shared_count (this=0x7f7dfc917bf8, __in_chrg=<optimized out>)
    at /usr/include/c++/4.8.2/tr1/shared_ptr.h:341
#21 ~__shared_ptr (this=0x7f7dfc917bf0, __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/tr1/shared_ptr.h:541
#22 ~shared_ptr (this=0x7f7dfc917bf0, __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/tr1/shared_ptr.h:985
#23 (anonymous namespace)::BeaconResponseHandler::handleResponse (this=<optimized out>, responseFrom=0x7f7dfc917d90, 
    transport=..., version=<optimized out>, command=<optimized out>, payloadSize=<optimized out>, 
    payloadBuffer=0x55992b986c20) at ../../src/remoteClient/clientContextImpl.cpp:2775
#24 0x00007f7e0d65eb8a in (anonymous namespace)::ClientResponseHandler::handleResponse (this=<optimized out>, 
    responseFrom=<optimized out>, transport=..., version=<optimized out>, command=<optimized out>, 
    payloadSize=<optimized out>, payloadBuffer=0x55992b986c20) at ../../src/remoteClient/clientContextImpl.cpp:3024
#25 0x00007f7e0d63b51f in epics::pvAccess::BlockingUDPTransport::processBuffer (this=this@entry=0x55992b986b00, 
    transport=..., fromAddress=..., receiveBuffer=receiveBuffer@entry=0x55992b986c20)
    at ../../src/remote/blockingUDPTransport.cpp:422
#26 0x00007f7e0d63c58b in epics::pvAccess::BlockingUDPTransport::run (this=0x55992b986b00)
    at ../../src/remote/blockingUDPTransport.cpp:271
#27 0x00007f7e0cfd2949 in epicsThreadCallEntryPoint (pPvt=0x55992b986ec0) at ../../src/osi/epicsThread.cpp:83
#28 0x00007f7e0cfd88dc in start_routine (arg=0x55992b987610) at ../../src/osi/os/posix/osdThread.c:403
#29 0x00007f7e150d0dd5 in start_thread () from /lib64/libpthread.so.0
#30 0x00007f7e14df9ead in tdestroy_recurse () from /lib64/libc.so.6
#31 0x0000000000000000 in ?? ()



Thread 7 (LWP 1264):
#0  0x00005599291d4ccf in address_in_range () at /tmp/build/80754af9/python_1565725737370/work/Objects/obmalloc.c:1365
#1  pymalloc_free.isra.0 (p=0x7f7dfedd6170) at /tmp/build/80754af9/python_1565725737370/work/Objects/obmalloc.c:1635
#2  _PyObject_Free () at /tmp/build/80754af9/python_1565725737370/work/Objects/obmalloc.c:1840
#3  0x00005599291eb7b2 in list_dealloc (op=0x7f7de00c1af0)
    at /tmp/build/80754af9/python_1565725737370/work/Objects/listobject.c:324
#4  0x00005599292ae2ec in _PyEval_EvalFrameDefault ()
    at /tmp/build/80754af9/python_1565725737370/work/Python/ceval.c:1314
#5  0x00007f7dfee43020 in __Pyx_PyFunction_FastCallNoKw (co=co@entry=0x7f7e0a490270, args=<optimized out>, 
    args@entry=0x7f7de1231990, na=na@entry=3, globals=globals@entry=0x7f7e1546e370) at _gw.cpp:9157
#6  0x00007f7dfee4399f in __Pyx_PyFunction_FastCallDict (func=0x7f7dfedd0b00, args=0x7f7de1231990, nargs=3, kwargs=0x0)
    at _gw.cpp:9189
#7  0x00007f7dfee4878f in GWProvider_testChannel (__pyx_v_provider=<optimized out>, __pyx_v_name=<optimized out>, 
    __pyx_v_peer=<optimized out>) at _gw.cpp:6368
#8  0x00007f7dfee5b628 in GWProvider::channelFind (this=0x55992beb2310, name=..., requester=...)
    at ../gwchannel.cpp:1109
#9  0x00007f7e0d693443 in epics::pvAccess::ServerSearchHandler::handleResponse (this=0x55992beb8718, 
    responseFrom=<optimized out>, transport=..., version=<optimized out>, command=<optimized out>, 
    payloadSize=<optimized out>, payloadBuffer=0x55992befa1b0) at ../../src/server/responseHandlers.cpp:361
#10 0x00007f7e0d683b92 in epics::pvAccess::ServerResponseHandler::handleResponse (this=<optimized out>, 
    responseFrom=<optimized out>, transport=..., version=<optimized out>, command=<optimized out>, 
    payloadSize=<optimized out>, payloadBuffer=0x55992befa1b0) at ../../src/server/responseHandlers.cpp:171
#11 0x00007f7e0d63b51f in epics::pvAccess::BlockingUDPTransport::processBuffer (this=this@entry=0x55992befa090, 
    transport=..., fromAddress=..., receiveBuffer=receiveBuffer@entry=0x55992befa1b0)
    at ../../src/remote/blockingUDPTransport.cpp:422
#12 0x00007f7e0d63c58b in epics::pvAccess::BlockingUDPTransport::run (this=0x55992befa090)
    at ../../src/remote/blockingUDPTransport.cpp:271
#13 0x00007f7e0cfd2949 in epicsThreadCallEntryPoint (pPvt=0x55992bf1a420) at ../../src/osi/epicsThread.cpp:83
#14 0x00007f7e0cfd88dc in start_routine (arg=0x55992bf1aba0) at ../../src/osi/os/posix/osdThread.c:403
#15 0x00007f7e150d0dd5 in start_thread () from /lib64/libpthread.so.0
#16 0x00007f7e14df9ead in tdestroy_recurse () from /lib64/libc.so.6
#17 0x0000000000000000 in ?? ()
@mdavidsaver mdavidsaver self-assigned this Sep 30, 2019
@mdavidsaver
Copy link
Member Author

The crash appears to happen when two different Beacon receiver threads are processing the PVField payload at the end of Beacon messages.

This payload is a full un-cached serialization, including Structure. This does break an assumption I made in epics-base/pvDataCPP#55 that Structure creation was relatively rare. In fact it happens for every Beacon...

Even so, this should only manifest as a performance hit. This crash suggests that there is also a problem with locking.

@mdavidsaver
Copy link
Member Author

While it is being deserialized, the Beacon payload is currently uninteresting, and unused.

pvAccessCPP/src/server/beaconServerStatusProvider.cpp

Lines 30 to 34 in 866b75a

PVField::shared_pointer DefaultBeaconServerStatusProvider::getServerStatusData()
{
//TODO implement (fill data)
return _status;
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mdavidsaver mdavidsaver

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mdavidsaver