Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]: DFP duo and azure inference pipelines segfault inside the release container #2014

Open
2 tasks done
AnuradhaKaruppiah opened this issue Oct 29, 2024 · 1 comment
Open
2 tasks done

[BUG]: DFP duo and azure inference pipelines segfault inside the release container #2014

AnuradhaKaruppiah opened this issue Oct 29, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@AnuradhaKaruppiah
Copy link
Contributor

Version

24.10

Which installation method(s) does this occur on?

Docker

Describe the bug.

To repro -

  1. Build release container locally and use it for building the morpheus_pipeline docker service using these instructions

  2. Start mlflow service and run duo inference pipeline inside the container -

python dfp_duo_pipeline.py --train_users none --start_time "2022-08-30" --input_file="../../data/dfp/duo-inference-data/*.json"

The pipeline segfaults. See call stack in the snippet below.

  1. This problem doesn't happen outside the container. Inside the container it happens frequently but not always.

stack trace -

(gdb) bt
#0  0x00007f45f1c173dc in mrc::node::RxSinkBase<std::shared_ptr<morpheus::MessageMeta> >::RxSinkBase() ()
   from /opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/_lib/../../../../libmorpheus.so
#1  0x00007f45f1ca313f in mrc::node::RxNode<std::shared_ptr<morpheus::MessageMeta>, std::shared_ptr<morpheus::MessageMeta>, mrc::runnable::Context>::RxNode<mrc::pymrc::PythonNode<std::shared_ptr<morpheus::MessageMeta>, std::shared_ptr<morpheus::MessageMeta>, mrc::runnable::Context>::op_factory_from_sub_fn(std::function<rxcpp::subscription (rxcpp::observable<std::shared_ptr<morpheus::MessageMeta>, rxcpp::dynamic_observable<std::shared_ptr<morpheus::MessageMeta> > >, rxcpp::subscriber<std::shared_ptr<morpheus::MessageMeta>, rxcpp::observer<std::shared_ptr<morpheus::MessageMeta>, void, void, void, void> >)>)::{lambda(rxcpp::observable<std::shared_ptr<morpheus::MessageMeta>, rxcpp::dynamic_observable<std::shared_ptr<morpheus::MessageMeta> > >)#1}>(mrc::pymrc::PythonNode<std::shared_ptr<morpheus::MessageMeta>, std::shared_ptr<morpheus::MessageMeta>, mrc::runnable::Context>::op_factory_from_sub_fn(std::function<rxcpp::subscription (rxcpp::observable<std::shared_ptr<morpheus::MessageMeta>, rxcpp::dynamic_observable<std::shared_ptr<morpheus::MessageMeta> > >, rxcpp::subscriber<std::shared_ptr<morpheus::MessageMeta>, rxcpp::observer<std::shared_ptr<morpheus::MessageMeta>, void, void, void, void> >)>)::{lambda(rxcpp::observable<std::shared_ptr<morpheus::MessageMeta>, rxcpp::dynamic_observable<std::shared_ptr<morpheus::MessageMeta> > >)#1}&&) ()
   from /opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/_lib/../../../../libmorpheus.so
#2  0x00007f45f1ca34ae in mrc::pymrc::PythonNode<std::shared_ptr<morpheus::MessageMeta>, std::shared_ptr<morpheus::MessageMeta>, mrc::runnable::Context>::RxNode<mrc::pymrc::PythonNode<std::shared_ptr<morpheus::MessageMeta>, std::shared_ptr<morpheus::MessageMeta>, mrc::runnable::Context>::op_factory_from_sub_fn(std::function<rxcpp::subscription (rxcpp::observable<std::shared_ptr<morpheus::MessageMeta>, rxcpp::dynamic_observable<std::shared_ptr<morpheus::MessageMeta> > >, rxcpp::subscriber<std::shared_ptr<morpheus::MessageMeta>, rxcpp::observer<std::shared_ptr<morpheus::MessageMeta>, void, void, void, void> >)>)::{lambda(rxcpp::observable<std::shared_ptr<morpheus::MessageMeta>, rxcpp::dynamic_observable<std::shared_ptr<morpheus::MessageMeta> > >)#1}>(int, void const**, mrc::pymrc::PythonNode<std::shared_ptr<morpheus::MessageMeta>, std::shared_ptr<morpheus::MessageMeta>, mrc::runnable::Context>::op_factory_from_sub_fn(std::function<rxcpp::subscription (rxcpp::observable<std::shared_ptr<morpheus::MessageMeta>, rxcpp::dynamic_observable<std::shared_ptr<morpheus::MessageMeta> > >, rxcpp::subscriber<std::shared_ptr<morpheus::MessageMeta>, rxcpp::observer<std::shared_ptr<morpheus::MessageMeta>, void, void, void, void> >)>)::{lambda(rxcpp::observable<std::shared_ptr<morpheus::MessageMeta>, rxcpp::dynamic_observable<std::shared_ptr<morpheus::MessageMeta> > >)#1}&&) ()
   from /opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/_lib/../../../../libmorpheus.so
#3  0x00007f45f1ca3d7d in morpheus::WriteToFileStage::WriteToFileStage(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::_Ios_Openmode, morpheus::FileTypes, bool, bool) ()
   from /opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/_lib/../../../../libmorpheus.so
#4  0x00007f45f1ca4162 in std::shared_ptr<mrc::segment::Object<morpheus::WriteToFileStage> > mrc::segment::IBuilder::construct_object<morpheus::WriteToFileStage, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::_Ios_Openmode&, morpheus::FileTypes&, bool&, bool&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::_Ios_Openmode&, morpheus::FileTypes&, bool&, bool&) () from /opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/_lib/../../../../libmorpheus.so
#5  0x00007f45f1ca4435 in morpheus::WriteToFileStageInterfaceProxy::init(mrc::segment::IBuilder&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, morpheus::FileTypes, bool, bool) () from /opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/_lib/../../../../libmorpheus.so
--Type <RET> for more, q to quit, c to continue without paging--
#6  0x00007f45bf0660bc in ?? ()
   from /opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/_lib/stages.cpython-310-x86_64-linux-gnu.so
#7  0x00007f45bf040963 in ?? ()
   from /opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/_lib/stages.cpython-310-x86_64-linux-gnu.so

#8  0x000055ee905dac46 in cfunction_call (func=0x7f45bf572b10, args=<optimized out>, kwargs=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Objects/methodobject.c:543
#9  0x000055ee905d3f73 in _PyObject_MakeTpCall (tstate=0x7f44e4040f70, callable=0x7f45bf572b10, args=<optimized out>, nargs=8,
    keywords=0x0) at /usr/local/src/conda/python-3.10.15/Objects/call.c:215
#10 0x000055ee905e692b in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7f44ee7b4e30,
    callable=0x7f45bf572b10, tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:112
#11 _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7f44ee7b4e30, callable=0x7f45bf572b10,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:99
#12 method_vectorcall (method=<optimized out>, args=0x7f44ee797fb8, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Objects/classobject.c:83
#13 0x000055ee905e4534 in slot_tp_init (self=<optimized out>, args=<optimized out>, kwds=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Objects/typeobject.c:7737
#14 0x000055ee905d427b in type_call (type=<optimized out>, args=0x7f44ee797fa0, kwds=0x0)
    at /usr/local/src/conda/python-3.10.15/Objects/typeobject.c:1135
#15 0x00007f45f26f2687 in ?? ()
   from /opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/_lib/common.cpython-310-x86_64-linux-gnu.so
#16 0x000055ee905d3f73 in _PyObject_MakeTpCall (tstate=0x7f44e4040f70, callable=0x55ee93a15160, args=<optimized out>, nargs=7,
    keywords=0x0) at /usr/local/src/conda/python-3.10.15/Objects/call.c:215
#17 0x000055ee905d0176 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=<optimized out>,
    callable=0x55ee93a15160, tstate=<optimized out>) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:112
#18 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f44ec3007a8, callable=0x55ee93a15160,
    tstate=<optimized out>) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:99
#19 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f44ec3007a8, callable=0x55ee93a15160)
    at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:123
#20 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7f44e56186f0,
    tstate=<optimized out>) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5893
#21 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x7f44ec300610, throwflag=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Python/ceval.c:4181
#22 0x000055ee905db0cc in _PyEval_EvalFrame (throwflag=0, f=0x7f44ec300610, tstate=0x7f44e4040f70)
    at /usr/local/src/conda/python-3.10.15/Include/internal/pycore_ceval.h:46
#23 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7f45beb0e060,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5067
#24 _PyFunction_Vectorcall (func=0x7f45beb0e050, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Objects/call.c:342
--Type <RET> for more, q to quit, c to continue without paging--
#25 0x000055ee905cb680 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f44ee80b800,
    callable=0x7f45beb0e050, tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#26 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f44ee80b800, callable=0x7f45beb0e050)
    at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:123
#27 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7f44e56188a0,
    tstate=<optimized out>) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5893
#28 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x7f44ee80b680, throwflag=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Python/ceval.c:4198
#29 0x000055ee905e64e2 in _PyEval_EvalFrame (throwflag=0, f=0x7f44ee80b680, tstate=0x7f44e4040f70)
    at /usr/local/src/conda/python-3.10.15/Include/internal/pycore_ceval.h:46
#30 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=0x7f44ec3005c8, locals=0x0, con=0x7f45bef14440,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5067
#31 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=0x7f44ec3005c8, func=0x7f45bef14430)
    at /usr/local/src/conda/python-3.10.15/Objects/call.c:342
#32 _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7f44ec3005c8, callable=0x7f45bef14430,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#33 method_vectorcall (method=<optimized out>, args=0x7f44ec3005d0, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Objects/classobject.c:53
#34 0x000055ee905cc2be in _PyObject_VectorcallTstate (kwnames=0x7f45beef8240, nargsf=<optimized out>, args=<optimized out>,
    callable=0x7f44ee7c6e80, tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#35 PyObject_Vectorcall (kwnames=0x7f45beef8240, nargsf=<optimized out>, args=<optimized out>, callable=0x7f44ee7c6e80)
    at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:123
#36 call_function (kwnames=0x7f45beef8240, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7f44e5618ab0,
    tstate=<optimized out>) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5893
#37 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x7f44ec300420, throwflag=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Python/ceval.c:4231
#38 0x000055ee905e64e2 in _PyEval_EvalFrame (throwflag=0, f=0x7f44ec300420, tstate=0x7f44e4040f70)
    at /usr/local/src/conda/python-3.10.15/Include/internal/pycore_ceval.h:46
#39 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=0x7f44e562ce90, locals=0x0, con=0x7f45beeeb770,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5067
#40 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=0x7f44e562ce90, func=0x7f45beeeb760)
    at /usr/local/src/conda/python-3.10.15/Objects/call.c:342
#41 _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7f44e562ce90, callable=0x7f45beeeb760,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#42 method_vectorcall (method=<optimized out>, args=0x7f44e562ce98, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Objects/classobject.c:53
#43 0x000055ee905cc2be in _PyObject_VectorcallTstate (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>,
    callable=0x7f44ee7b50c0, tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
--Type <RET> for more, q to quit, c to continue without paging--
#44 PyObject_Vectorcall (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>, callable=0x7f44ee7b50c0)
    at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:123
#45 call_function (kwnames=0x7f45beed0e80, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7f44e5618cc0,
    tstate=<optimized out>) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5893
#46 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x7f44e562cce0, throwflag=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Python/ceval.c:4231
#47 0x000055ee905e64e2 in _PyEval_EvalFrame (throwflag=0, f=0x7f44e562cce0, tstate=0x7f44e4040f70)
    at /usr/local/src/conda/python-3.10.15/Include/internal/pycore_ceval.h:46
#48 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=0x7f44ec3003e0, locals=0x0, con=0x7f45beeeb770,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5067
#49 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=0x7f44ec3003e0, func=0x7f45beeeb760)
    at /usr/local/src/conda/python-3.10.15/Objects/call.c:342
#50 _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7f44ec3003e0, callable=0x7f45beeeb760,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#51 method_vectorcall (method=<optimized out>, args=0x7f44ec3003e8, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Objects/classobject.c:53
#52 0x000055ee905cc2be in _PyObject_VectorcallTstate (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>,
    callable=0x7f44ee7b4a80, tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#53 PyObject_Vectorcall (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>, callable=0x7f44ee7b4a80)
    at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:123
#54 call_function (kwnames=0x7f45beed0e80, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7f44e5618ed0,
    tstate=<optimized out>) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5893
#55 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x7f44ec300230, throwflag=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Python/ceval.c:4231
#56 0x000055ee905e64e2 in _PyEval_EvalFrame (throwflag=0, f=0x7f44ec300230, tstate=0x7f44e4040f70)
    at /usr/local/src/conda/python-3.10.15/Include/internal/pycore_ceval.h:46
#57 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=0x7f44e5629510, locals=0x0, con=0x7f45beeeb770,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5067
#58 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=0x7f44e5629510, func=0x7f45beeeb760)
    at /usr/local/src/conda/python-3.10.15/Objects/call.c:342
#59 _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7f44e5629510, callable=0x7f45beeeb760,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#60 method_vectorcall (method=<optimized out>, args=0x7f44e5629518, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Objects/classobject.c:53
#61 0x000055ee905cc2be in _PyObject_VectorcallTstate (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>,
    callable=0x7f44ee7fd980, tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#62 PyObject_Vectorcall (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>, callable=0x7f44ee7fd980)
    at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:123
--Type <RET> for more, q to quit, c to continue without paging--
#63 call_function (kwnames=0x7f45beed0e80, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7f44e56190e0,
    tstate=<optimized out>) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5893
#64 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x7f44e5629360, throwflag=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Python/ceval.c:4231
#65 0x000055ee905e64e2 in _PyEval_EvalFrame (throwflag=0, f=0x7f44e5629360, tstate=0x7f44e4040f70)
    at /usr/local/src/conda/python-3.10.15/Include/internal/pycore_ceval.h:46
#66 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=0x7f44e5624d20, locals=0x0, con=0x7f45beeeb770,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5067
#67 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=0x7f44e5624d20, func=0x7f45beeeb760)
    at /usr/local/src/conda/python-3.10.15/Objects/call.c:342
#68 _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7f44e5624d20, callable=0x7f45beeeb760,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#69 method_vectorcall (method=<optimized out>, args=0x7f44e5624d28, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Objects/classobject.c:53
#70 0x000055ee905cc2be in _PyObject_VectorcallTstate (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>,
    callable=0x7f44ee7fd840, tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#71 PyObject_Vectorcall (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>, callable=0x7f44ee7fd840)
    at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:123
#72 call_function (kwnames=0x7f45beed0e80, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7f44e56192f0,
    tstate=<optimized out>) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5893
#73 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x7f44e5624b70, throwflag=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Python/ceval.c:4231
#74 0x000055ee905e64e2 in _PyEval_EvalFrame (throwflag=0, f=0x7f44e5624b70, tstate=0x7f44e4040f70)
    at /usr/local/src/conda/python-3.10.15/Include/internal/pycore_ceval.h:46
#75 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=0x7f44e5626cf0, locals=0x0, con=0x7f45beeeb770,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5067
#76 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=0x7f44e5626cf0, func=0x7f45beeeb760)
    at /usr/local/src/conda/python-3.10.15/Objects/call.c:342
#77 _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7f44e5626cf0, callable=0x7f45beeeb760,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#78 method_vectorcall (method=<optimized out>, args=0x7f44e5626cf8, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Objects/classobject.c:53
#79 0x000055ee905cc2be in _PyObject_VectorcallTstate (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>,
    callable=0x7f44ee7fce00, tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#80 PyObject_Vectorcall (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>, callable=0x7f44ee7fce00)
    at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:123
#81 call_function (kwnames=0x7f45beed0e80, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7f44e5619500,
    tstate=<optimized out>) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5893
--Type <RET> for more, q to quit, c to continue without paging--
#82 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x7f44e5626b40, throwflag=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Python/ceval.c:4231
#83 0x000055ee905e64e2 in _PyEval_EvalFrame (throwflag=0, f=0x7f44e5626b40, tstate=0x7f44e4040f70)
    at /usr/local/src/conda/python-3.10.15/Include/internal/pycore_ceval.h:46
#84 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=0x7f44e5621100, locals=0x0, con=0x7f45beeeb770,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5067
#85 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=0x7f44e5621100, func=0x7f45beeeb760)
    at /usr/local/src/conda/python-3.10.15/Objects/call.c:342
#86 _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7f44e5621100, callable=0x7f45beeeb760,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#87 method_vectorcall (method=<optimized out>, args=0x7f44e5621108, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Objects/classobject.c:53
#88 0x000055ee905cc2be in _PyObject_VectorcallTstate (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>,
    callable=0x7f44ee7ff740, tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#89 PyObject_Vectorcall (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>, callable=0x7f44ee7ff740)
    at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:123
#90 call_function (kwnames=0x7f45beed0e80, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7f44e5619710,
    tstate=<optimized out>) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5893
#91 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x7f44e5620f50, throwflag=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Python/ceval.c:4231
#92 0x000055ee905e64e2 in _PyEval_EvalFrame (throwflag=0, f=0x7f44e5620f50, tstate=0x7f44e4040f70)
    at /usr/local/src/conda/python-3.10.15/Include/internal/pycore_ceval.h:46
#93 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=0x7f44e561eb20, locals=0x0, con=0x7f45beeeb770,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5067
#94 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=0x7f44e561eb20, func=0x7f45beeeb760)
    at /usr/local/src/conda/python-3.10.15/Objects/call.c:342
#95 _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7f44e561eb20, callable=0x7f45beeeb760,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#96 method_vectorcall (method=<optimized out>, args=0x7f44e561eb28, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Objects/classobject.c:53
#97 0x000055ee905cc2be in _PyObject_VectorcallTstate (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>,
    callable=0x7f44ee7ffd00, tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#98 PyObject_Vectorcall (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>, callable=0x7f44ee7ffd00)
    at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:123
#99 call_function (kwnames=0x7f45beed0e80, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7f44e5619920,
    tstate=<optimized out>) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5893
#100 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x7f44e561e970, throwflag=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Python/ceval.c:4231
--Type <RET> for more, q to quit, c to continue without paging--
#101 0x000055ee905e64e2 in _PyEval_EvalFrame (throwflag=0, f=0x7f44e561e970, tstate=0x7f44e4040f70)
    at /usr/local/src/conda/python-3.10.15/Include/internal/pycore_ceval.h:46
#102 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=0x7f44e561af20, locals=0x0, con=0x7f45beeeb770,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5067
#103 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=0x7f44e561af20, func=0x7f45beeeb760)
    at /usr/local/src/conda/python-3.10.15/Objects/call.c:342
#104 _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7f44e561af20,
    callable=0x7f45beeeb760, tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#105 method_vectorcall (method=<optimized out>, args=0x7f44e561af28, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Objects/classobject.c:53
#106 0x000055ee905cc2be in _PyObject_VectorcallTstate (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>,
    callable=0x7f44ee7ff5c0, tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#107 PyObject_Vectorcall (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>, callable=0x7f44ee7ff5c0)
    at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:123
#108 call_function (kwnames=0x7f45beed0e80, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7f44e5619b30,
    tstate=<optimized out>) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5893
#109 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x7f44e561ad70, throwflag=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Python/ceval.c:4231
#110 0x000055ee905e64e2 in _PyEval_EvalFrame (throwflag=0, f=0x7f44e561ad70, tstate=0x7f44e4040f70)
    at /usr/local/src/conda/python-3.10.15/Include/internal/pycore_ceval.h:46
#111 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=0x7f44e4d32030, locals=0x0, con=0x7f45beeeb770,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5067
#112 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=0x7f44e4d32030, func=0x7f45beeeb760)
    at /usr/local/src/conda/python-3.10.15/Objects/call.c:342
#113 _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7f44e4d32030,
    callable=0x7f45beeeb760, tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#114 method_vectorcall (method=<optimized out>, args=0x7f44e4d32038, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Objects/classobject.c:53
#115 0x000055ee905cc2be in _PyObject_VectorcallTstate (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>,
    callable=0x7f44f0657b80, tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#116 PyObject_Vectorcall (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>, callable=0x7f44f0657b80)
    at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:123
#117 call_function (kwnames=0x7f45beed0e80, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7f44e5619d40,
    tstate=<optimized out>) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5893
#118 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x7f44e4d31e80, throwflag=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Python/ceval.c:4231
#119 0x000055ee905e64e2 in _PyEval_EvalFrame (throwflag=0, f=0x7f44e4d31e80, tstate=0x7f44e4040f70)
    at /usr/local/src/conda/python-3.10.15/Include/internal/pycore_ceval.h:46
--Type <RET> for more, q to quit, c to continue without paging--
#120 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=0x7f44ec3001f0, locals=0x0, con=0x7f45beeeb770,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5067
#121 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=0x7f44ec3001f0, func=0x7f45beeeb760)
    at /usr/local/src/conda/python-3.10.15/Objects/call.c:342
#122 _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7f44ec3001f0,
    callable=0x7f45beeeb760, tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#123 method_vectorcall (method=<optimized out>, args=0x7f44ec3001f8, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Objects/classobject.c:53
#124 0x000055ee905cc2be in _PyObject_VectorcallTstate (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>,
    callable=0x7f44f0656240, tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#125 PyObject_Vectorcall (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>, callable=0x7f44f0656240)
    at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:123
#126 call_function (kwnames=0x7f45beed0e80, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7f44e5619f50,
    tstate=<optimized out>) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5893
#127 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x7f44ec300040, throwflag=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Python/ceval.c:4231
#128 0x000055ee905e64e2 in _PyEval_EvalFrame (throwflag=0, f=0x7f44ec300040, tstate=0x7f44e4040f70)
    at /usr/local/src/conda/python-3.10.15/Include/internal/pycore_ceval.h:46
#129 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=0x7f44ee7fbe00, locals=0x0, con=0x7f45beeeb770,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5067
#130 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=0x7f44ee7fbe00, func=0x7f45beeeb760)
    at /usr/local/src/conda/python-3.10.15/Objects/call.c:342
#131 _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7f44ee7fbe00,
    callable=0x7f45beeeb760, tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#132 method_vectorcall (method=<optimized out>, args=0x7f44ee7fbe08, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Objects/classobject.c:53
#133 0x000055ee905cc2be in _PyObject_VectorcallTstate (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>,
    callable=0x7f44f28bc1c0, tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#134 PyObject_Vectorcall (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>, callable=0x7f44f28bc1c0)
    at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:123
#135 call_function (kwnames=0x7f45beed0e80, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7f44e561a160,
    tstate=<optimized out>) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5893
#136 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x7f44ee7fbc50, throwflag=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Python/ceval.c:4231
#137 0x000055ee905e64e2 in _PyEval_EvalFrame (throwflag=0, f=0x7f44ee7fbc50, tstate=0x7f44e4040f70)
    at /usr/local/src/conda/python-3.10.15/Include/internal/pycore_ceval.h:46
#138 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=0x7f44ee7fbc10, locals=0x0, con=0x7f45beeeb770,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5067
--Type <RET> for more, q to quit, c to continue without paging--
#139 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=0x7f44ee7fbc10, func=0x7f45beeeb760)
    at /usr/local/src/conda/python-3.10.15/Objects/call.c:342
#140 _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7f44ee7fbc10,
    callable=0x7f45beeeb760, tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#141 method_vectorcall (method=<optimized out>, args=0x7f44ee7fbc18, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Objects/classobject.c:53
#142 0x000055ee905cc2be in _PyObject_VectorcallTstate (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>,
    callable=0x7f44f0657c00, tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#143 PyObject_Vectorcall (kwnames=0x7f45beed0e80, nargsf=<optimized out>, args=<optimized out>, callable=0x7f44f0657c00)
    at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:123
#144 call_function (kwnames=0x7f45beed0e80, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7f44e561a370,
    tstate=<optimized out>) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5893
#145 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x7f44ee7fba60, throwflag=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Python/ceval.c:4231
#146 0x000055ee905db0cc in _PyEval_EvalFrame (throwflag=0, f=0x7f44ee7fba60, tstate=0x7f44e4040f70)
    at /usr/local/src/conda/python-3.10.15/Include/internal/pycore_ceval.h:46
#147 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7f45beeeb770,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5067
#148 _PyFunction_Vectorcall (func=0x7f45beeeb760, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Objects/call.c:342
#149 0x000055ee905cb680 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f44ee803fc0,
    callable=0x7f45beeeb760, tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:114
#150 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f44ee803fc0, callable=0x7f45beeeb760)
    at /usr/local/src/conda/python-3.10.15/Include/cpython/abstract.h:123
#151 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7f44e561a520,
    tstate=<optimized out>) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5893
#152 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x7f44ee803e20, throwflag=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Python/ceval.c:4198
#153 0x000055ee905db0cc in _PyEval_EvalFrame (throwflag=0, f=0x7f44ee803e20, tstate=0x7f44e4040f70)
    at /usr/local/src/conda/python-3.10.15/Include/internal/pycore_ceval.h:46
#154 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x7f44ee7bd6d0,
    tstate=0x7f44e4040f70) at /usr/local/src/conda/python-3.10.15/Python/ceval.c:5067
#155 _PyFunction_Vectorcall (func=0x7f44ee7bd6c0, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Objects/call.c:342
#156 0x000055ee905e6e7c in PyVectorcall_Call (kwargs=<optimized out>, tuple=<optimized out>, callable=0x7f44ee7bd6c0)
    at /usr/local/src/conda/python-3.10.15/Objects/call.c:267
#157 _PyObject_Call (kwargs=<optimized out>, args=<optimized out>, callable=0x7f44ee7bd6c0, tstate=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Objects/call.c:290
--Type <RET> for more, q to quit, c to continue without paging--
#158 PyObject_Call (callable=0x7f44ee7bd6c0, args=<optimized out>, kwargs=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Objects/call.c:317
#159 0x000055ee9069e38d in partial_call (pto=0x7f44ee7e1530, args=0x7f44f0631210, kwargs=<optimized out>)
    at /usr/local/src/conda/python-3.10.15/Modules/_functoolsmodule.c:323
#160 0x000055ee905d3f73 in _PyObject_MakeTpCall (tstate=0x7f44e4040f70, callable=0x7f44ee7e1530, args=<optimized out>,
    nargs=1, keywords=0x0) at /usr/local/src/conda/python-3.10.15/Objects/call.c:215
#161 0x00007f45bf342284 in ?? ()
   from /opt/conda/envs/morpheus/lib/python3.10/site-packages/mrc/core/pipeline.cpython-310-x86_64-linux-gnu.so
#162 0x00007f45bf33e249 in ?? ()
   from /opt/conda/envs/morpheus/lib/python3.10/site-packages/mrc/core/pipeline.cpython-310-x86_64-linux-gnu.so
#163 0x00007f45c978c009 in std::_Function_handler<void (mrc::segment::IBuilder&), mrc::pymrc::Pipeline::make_segment(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void (mrc::segment::IBuilder&)> const&)::{lambda(mrc::segment::IBuilder&)#1}>::_M_invoke(std::_Any_data const&, mrc::segment::IBuilder&) ()
   from /opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/_lib/../../../.././libmrc_pymrc.so
#164 0x00007f45c94cdfda in mrc::segment::BuilderDefinition::initialize() ()
   from /opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/_lib/../../../.././libmrc.so
#165 0x00007f45c94d1d9e in boost::fibers::detail::task_object<std::_Bind<mrc::segment::SegmentInstance::SegmentInstance(std::shared_ptr<mrc::segment::SegmentDefinition const>, unsigned short, mrc::pipeline::PipelineResources&, unsigned long)::{lambda()#1} ()>, std::allocator<boost::fibers::packaged_task<std::unique_ptr<mrc::segment::BuilderDefinition, std::default_delete<mrc::segment::BuilderDefinition> > ()> >, std::unique_ptr<mrc::segment::BuilderDefinition, std::default_delete<mrc::segment::BuilderDefinition> >>::run() () from /opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/_lib/../../../.././libmrc.so
#166 0x00007f45c94d1a84 in boost::fibers::detail::task_object<mrc::core::FiberTaskQueue::enqueue<mrc::segment::SegmentInstance::SegmentInstance(std::shared_ptr<mrc::segment::SegmentDefinition const>, unsigned short, mrc::pipeline::PipelineResources&, unsigned long)::{lambda()#1}>(mrc::FiberMetaData&&, mrc::segment::SegmentInstance::SegmentInstance(std::shared_ptr<mrc::segment::SegmentDefinition const>, unsigned short, mrc::pipeline::PipelineResources&, unsigned long)::{lambda()#1}&&)::{lambda()#1}, std::allocator<boost::fibers::packaged_task<void ()> >, void>::run() ()
   from /opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/_lib/../../../.././libmrc.so
#167 0x00007f45c94e3187 in boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>::run_(boost::context::fiber&&)
    () from /opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/_lib/../../../.././libmrc.so
#168 0x00007f45c94e0edc in void boost::context::detail::fiber_entry<boost::context::detail::fiber_record<boost::context::fiber, boost::context::basic_fixedsize_stack<boost::context::stack_traits>, std::_Bind<boost::context::fiber (boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>::*(boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>*, std::_Placeholder<1>))(boost::context::fiber&&)> > >(boost::context::detail::transfer_t) ()
   from /opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/_lib/../../../.././libmrc.so
#169 0x00007f45f163711f in make_fcontext ()
   from /opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/_lib/../../../.././libboost_context.so.1.84.0
#170 0x0000000000000000 in ?? ()
(gdb)

Minimum reproducible example

No response

Relevant log output

(morpheus) root@d383fc37ec2f:/workspace/examples/digital_fingerprinting/production# python dfp_duo_pipeline.py --train_users none --start_time "2022-08-30" --input_file="../../../data/dfp/duo-inference-data/*.json"
*** Aborted at 1730153290 (unix time) try "date -d @1730153290" if you are using GNU date ***
PC: @     0x7f567540b3dc mrc::node::RxSinkBase<std::shared_ptr<morpheus::MessageMeta> >::RxSinkBase()
*** SIGSEGV (@0xfeabda8a3180) received by PID 32 (TID 0x7f55715ff640) from PID 18446744073081074048; stack trace: ***
    @     0x7f593a33cee8 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x99ee7)
    @     0x7f5934e8b917 google::(anonymous namespace)::FailureSignalHandler(int, siginfo*, void*)
    @     0x7f593a2e5520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4251f)
    @     0x7f567540b3dc mrc::node::RxSinkBase<std::shared_ptr<morpheus::MessageMeta> >::RxSinkBase()
    @     0x7f567549713f mrc::node::RxNode<std::shared_ptr<morpheus::MessageMeta>, std::shared_ptr<morpheus::MessageMeta>, mrc::runnable::Context>::RxNode<mrc::pymrc::PythonNode<std::shared_ptr<morpheus::MessageMeta>, std::shared_ptr<morpheus::MessageMeta>, mrc::runnable::Context>z
    @     0x7f56754974ae mrc::pymrc::PythonNode<std::shared_ptr<morpheus::MessageMeta>, std::shared_ptr<morpheus::MessageMeta>, mrc::runnable::Context>::RxNode<mrc::pymrc::PythonNode<std::shared_ptr<morpheus::MessageMeta>, std::shared_ptr<morpheus::MessageMeta>, mrc::runnable::Conz
    @     0x7f5675497d7d morpheus::WriteToFileStage::WriteToFileStage(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::_Ios_Openmode, morpheus::FileTypes, bool, bool)
    @     0x7f5675498162 std::shared_ptr<mrc::segment::Object<morpheus::WriteToFileStage> > mrc::segment::IBuilder::construct_object<morpheus::WriteToFileStage, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::_Ios_Openmode&, morpheus::Fz
    @     0x7f5675498435 morpheus::WriteToFileStageInterfaceProxy::init(mrc::segment::IBuilder&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cz
    @     0x7f564285b0bc (/opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/_lib/stages.cpython-310-x86_64-linux-gnu.so+0x790bb)
    @     0x7f5642835963 (/opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/_lib/stages.cpython-310-x86_64-linux-gnu.so+0x53962)
    @     0x5564431e6c46 cfunction_call
    @     0x5564431dff73 _PyObject_MakeTpCall.localalias
    @     0x5564431f292b method_vectorcall
    @     0x5564431f0534 slot_tp_init
    @     0x5564431e027b type_call
    @     0x7f5675ee6687 (/opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/_lib/common.cpython-310-x86_64-linux-gnu.so+0x1b686)
    @     0x5564431dff73 _PyObject_MakeTpCall.localalias
    @     0x5564431dc176 _PyEval_EvalFrameDefault
    @     0x5564431e70cc _PyFunction_Vectorcall
    @     0x5564431d7680 _PyEval_EvalFrameDefault
    @     0x5564431f24e2 method_vectorcall
    @     0x5564431d82be _PyEval_EvalFrameDefault
    @     0x5564431f24e2 method_vectorcall
    @     0x5564431d82be _PyEval_EvalFrameDefault
    @     0x5564431f24e2 method_vectorcall
    @     0x5564431d82be _PyEval_EvalFrameDefault
    @     0x5564431f24e2 method_vectorcall
    @     0x5564431d82be _PyEval_EvalFrameDefault
    @     0x5564431f24e2 method_vectorcall
    @     0x5564431d82be _PyEval_EvalFrameDefault
    @     0x5564431f24e2 method_vectorcall
Segmentation fault (core dumped)
(morpheus) root@d383fc37ec2f:/workspace/examples/digital_fingerprinting/production# python dfp_duo_pipeline.py --train_users none --start_time "2022-08-30" --input_file="../../data/dfp/duo-inference-data/*.json"
*** Aborted at 1730153355 (unix time) try "date -d @1730153355" if you are using GNU date ***
PC: @     0x7efd852583dc mrc::node::RxSinkBase<std::shared_ptr<morpheus::MessageMeta> >::RxSinkBase()
*** SIGSEGV (@0xfdf9fa139450) received by PID 294 (TID 0x7efc813ff640) from PID 18446744073610171472; stack trace: ***
    @     0x7f004a18fee8 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x99ee7)
    @     0x7f0044cde917 google::(anonymous namespace)::FailureSignalHandler(int, siginfo*, void*)
    @     0x7f004a138520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4251f)
    @     0x7efd852583dc mrc::node::RxSinkBase<std::shared_ptr<morpheus::MessageMeta> >::RxSinkBase()
    @     0x7efd852e413f mrc::node::RxNode<std::shared_ptr<morpheus::MessageMeta>, std::shared_ptr<morpheus::MessageMeta>, mrc::runnable::Context>::RxNode<mrc::pymrc::PythonNode<std::shared_ptr<morpheus::MessageMeta>, std::shared_ptr<morpheus::MessageMeta>, mrc::runnable::Context>z
    @     0x7efd852e44ae mrc::pymrc::PythonNode<std::shared_ptr<morpheus::MessageMeta>, std::shared_ptr<morpheus::MessageMeta>, mrc::runnable::Context>::RxNode<mrc::pymrc::PythonNode<std::shared_ptr<morpheus::MessageMeta>, std::shared_ptr<morpheus::MessageMeta>, mrc::runnable::Conz
    @     0x7efd852e4d7d morpheus::WriteToFileStage::WriteToFileStage(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::_Ios_Openmode, morpheus::FileTypes, bool, bool)
    @     0x7efd852e5162 std::shared_ptr<mrc::segment::Object<morpheus::WriteToFileStage> > mrc::segment::IBuilder::construct_object<morpheus::WriteToFileStage, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::_Ios_Openmode&, morpheus::Fz
    @     0x7efd852e5435 morpheus::WriteToFileStageInterfaceProxy::init(mrc::segment::IBuilder&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cz
    @     0x7efd526a60bc (/opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/_lib/stages.cpython-310-x86_64-linux-gnu.so+0x790bb)
    @     0x7efd52680963 (/opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/_lib/stages.cpython-310-x86_64-linux-gnu.so+0x53962)
    @     0x56503b33bc46 cfunction_call
    @     0x56503b334f73 _PyObject_MakeTpCall.localalias
    @     0x56503b34792b method_vectorcall
    @     0x56503b345534 slot_tp_init
    @     0x56503b33527b type_call
    @     0x7efd85d33687 (/opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/_lib/common.cpython-310-x86_64-linux-gnu.so+0x1b686)
    @     0x56503b334f73 _PyObject_MakeTpCall.localalias
    @     0x56503b331176 _PyEval_EvalFrameDefault
    @     0x56503b33c0cc _PyFunction_Vectorcall
    @     0x56503b32c680 _PyEval_EvalFrameDefault
    @     0x56503b3474e2 method_vectorcall
    @     0x56503b32d2be _PyEval_EvalFrameDefault
    @     0x56503b3474e2 method_vectorcall
    @     0x56503b32d2be _PyEval_EvalFrameDefault
    @     0x56503b3474e2 method_vectorcall
    @     0x56503b32d2be _PyEval_EvalFrameDefault
    @     0x56503b3474e2 method_vectorcall
    @     0x56503b32d2be _PyEval_EvalFrameDefault
    @     0x56503b3474e2 method_vectorcall
    @     0x56503b32d2be _PyEval_EvalFrameDefault
    @     0x56503b3474e2 method_vectorcall
Segmentation fault (core dumped)
(morpheus) root@d383fc37ec2f:/

Full env printout

Click here to see environment details
[Paste the results of print_env.sh here, it will be hidden by default]

Other/Misc.

No response

Code of Conduct

  • I agree to follow Morpheus' Code of Conduct
  • I have searched the open bugs and have found no duplicates for this bug report
@AnuradhaKaruppiah AnuradhaKaruppiah added the bug Something isn't working label Oct 29, 2024
@AnuradhaKaruppiah
Copy link
Contributor Author

This problem is not reproducible on the latest 24.10.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Morpheus Boards
Status: Todo
Milestone
No milestone
Development

No branches or pull requests
1 participant
@AnuradhaKaruppiah