[BUG]: DFP AttributeError: 'int' object has no attribute 'item'

[dagardner-nv]

Version

24.10

Which installation method(s) does this occur on?

Source, Conda, Docker

Describe the bug.

Occurs with DFP Azure inference pipeline

Minimum reproducible example

python dfp_azure_pipeline.py --train_users none  --start_time "2022-08-30" --input_file="../../data/dfp/azure-inference-data/*.json" --log_level=DEBUG --tracking_uri="http://localhost:5000"

Relevant log output

Click here to see error details

azure inf

Running training pipeline with the following options:

Train generic_user: False

Skipping users: []

Start Time: 2022-08-30 00:00:00+00:00

Duration: 60 days, 0:00:00

Cache Dir: ./.cache/dfp

Tracking URI: http://localhost:5000

Updating list of available models...

Updating list of available models... Done.

====Pipeline Pre-build====

====Pre-Building Segment: linear_segment_0====

====Pre-Building Segment Complete!====

====Pipeline Pre-build Complete!====

====Registering Pipeline====

====Building Pipeline====

====Building Pipeline Complete!====

====Registering Pipeline Complete!====

====Starting Pipeline====

====Building Segment: linear_segment_0====

Added source: <from-multi-file-0; MultiFileSource(filenames=['../../data/dfp/azure-inference-data/*.json'], watch=False, watch_interval=1.0)>

└─> fsspec.OpenFiles

Added stage: <dfp-file-batcher-1; DFPFileBatcherStage(date_conversion_func=functools.partial(<function date_extractor at 0x7f4ba5265630>, filename_regex=re.compile('(?P\d{4})-(?P\d{1,2})-(?P\d{1,2})(?:T(?P\d{1,2})(?::||\.)(?P\d{1,2})(?::||\.)(?P\d{1,2})(?:\.(?P\d{0,6}))?)?(?PZ)?')), period=D, sampling_rate_s=0, start_time=2022-08-30 00:00:00+00:00, end_time=2022-10-29 00:00:00+00:00, sampling=None)>

└─ fsspec.OpenFiles -> Tuple[fsspec.core.OpenFiles, int]

Added stage: <dfp-file-to-df-2; DFPFileToDataFrameStage(schema=DataFrameInputSchema(json_columns=['properties'], column_info=[DateTimeColumn(name='timestamp', dtype='datetime64[ns]', input_name='time'), RenameColumn(name='username', dtype='str', input_name='properties.userPrincipalName'), RenameColumn(name='appDisplayName', dtype='str', input_name='properties.appDisplayName'), ColumnInfo(name='category', dtype='str'), RenameColumn(name='clientAppUsed', dtype='str', input_name='properties.clientAppUsed'), RenameColumn(name='deviceDetailbrowser', dtype='str', input_name='properties.deviceDetail.browser'), RenameColumn(name='deviceDetaildisplayName', dtype='str', input_name='properties.deviceDetail.displayName'), RenameColumn(name='deviceDetailoperatingSystem', dtype='str', input_name='properties.deviceDetail.operatingSystem'), StringCatColumn(name='location', dtype='str', input_columns=['properties.location.city', 'properties.location.countryOrRegion'], sep=', '), RenameColumn(name='statusfailureReason', dtype='str', input_name='properties.status.failureReason')], preserve_columns=None, row_filter=None), filter_null=True, file_type=FileTypes.JSON, parser_kwargs={'lines': False, 'orient': 'records'}, cache_dir=./.cache/dfp)>

└─ Tuple[fsspec.core.OpenFiles, int] -> pandas.DataFrame

Added stage: <monitor-3; MonitorStage(description=Input data rate, smoothing=0.05, unit=messages, delayed_start=False, determine_count_fn=None, log_level=LogLevels.INFO)>

└─ pandas.DataFrame -> pandas.DataFrame

Added stage: <dfp-split-users-4; DFPSplitUsersStage(include_generic=False, include_individual=True, skip_users=[], only_users=[])>

└─ pandas.DataFrame -> morpheus.ControlMessage

Added stage: <dfp-rolling-window-5; DFPRollingWindowStage(min_history=1, min_increment=0, max_history=1d, cache_dir=./.cache/dfp)>

└─ morpheus.ControlMessage -> morpheus.ControlMessage

Added stage: <dfp-preproc-6; DFPPreprocessingStage(input_schema=DataFrameInputSchema(json_columns=[], column_info=[ColumnInfo(name='timestamp', dtype='datetime64[ns]'), ColumnInfo(name='username', dtype='str'), ColumnInfo(name='appDisplayName', dtype='str'), ColumnInfo(name='clientAppUsed', dtype='str'), ColumnInfo(name='deviceDetailbrowser', dtype='str'), ColumnInfo(name='deviceDetaildisplayName', dtype='str'), ColumnInfo(name='deviceDetailoperatingSystem', dtype='str'), ColumnInfo(name='statusfailureReason', dtype='str'), IncrementColumn(name='logcount', dtype='int', input_name='timestamp', groupby_column='username', period='D'), DistinctIncrementColumn(name='locincrement', dtype='int', input_name='location', groupby_column='username', period='D', timestamp_column='timestamp'), DistinctIncrementColumn(name='appincrement', dtype='int', input_name='appDisplayName', groupby_column='username', period='D', timestamp_column='timestamp')], preserve_columns=re.compile('(_batch_id)'), row_filter=None))>

└─ morpheus.ControlMessage -> morpheus.ControlMessage

Added stage: <dfp-inference-7; DFPInferenceStage(model_name_formatter=DFP-azure-{user_id})>

└─ morpheus.ControlMessage -> morpheus.ControlMessage

Added stage: <monitor-8; MonitorStage(description=Inference rate, smoothing=0.001, unit=messages, delayed_start=False, determine_count_fn=None, log_level=LogLevels.INFO)>

└─ morpheus.ControlMessage -> morpheus.ControlMessage

Added stage: <filter-9; FilterDetectionsStage(threshold=2.0, copy=True, filter_source=FilterSource.DATAFRAME, field_name=mean_abs_z)>

└─ morpheus.ControlMessage -> morpheus.ControlMessage

Added stage: <dfp-postproc-10; DFPPostprocessingStage()>

└─ morpheus.ControlMessage -> morpheus.ControlMessage

Added stage: <serialize-11; SerializeStage(include=None, exclude=['batch_count', 'origin_hash', '_row_hash', '_batch_id'], fixed_columns=True)>

└─ morpheus.ControlMessage -> morpheus.MessageMeta

Added stage: <to-file-12; WriteToFileStage(filename=dfp_detections_azure.csv, overwrite=True, file_type=FileTypes.Auto, include_index_col=True, flush=False)>

└─ morpheus.MessageMeta -> morpheus.MessageMeta

====Pipeline Started====

====Building Segment Complete!====

Creating dask cluster...

Creating dask cluster... Done. Dashboard: http://192.168.4.51:8787/status

S3 objects to DF complete. Rows: 119, Cache: miss, Duration: 833.8134288787842 ms, Rate: 142.71777819651746 rows/s

Preallocating column event_time[TypeId.STRING]

S3 objects to DF complete. Rows: 209, Cache: miss, Duration: 162.492036819458 ms, Rate: 1286.2168761673913 rows/s

Preallocating column event_time[TypeId.STRING]

Stopping dask cluster...

Stopping dask cluster... Done.

Batch split users complete. Input: 119 rows from 2022-08-30 00:17:05.561523 to 2022-08-30 23:58:05.567378. Output: 16 users, rows/user min: 1, max: 18, avg: 7.44. Duration: 233.89 ms

Rolling window complete for [email protected] in 153.25 ms. Input: 13 rows from 2022-08-30T01:54:26.639083000 to 2022-08-30T23:58:05.567378000. Output: 13 rows from 2022-08-30T01:54:26.639083000 to 2022-08-30T23:58:05.567378000

Batch split users complete. Input: 209 rows from 2022-08-31 00:21:46.153050 to 2022-08-31 23:54:50.435683. Output: 17 users, rows/user min: 1, max: 106, avg: 12.29. Duration: 283.07 ms

Preprocessed 13 data for logs in 2022-08-30T01:54:26.639083000 to 2022-08-30T23:58:05.567378000 in 119.91143226623535 ms

Rolling window complete for [email protected] in 199.34 ms. Input: 18 rows from 2022-08-30T01:23:39.156080000 to 2022-08-30T23:05:47.146155000. Output: 18 rows from 2022-08-30T01:23:39.156080000 to 2022-08-30T23:05:47.146155000

Preprocessed 18 data for logs in 2022-08-30T01:23:39.156080000 to 2022-08-30T23:05:47.146155000 in 114.43161964416504 ms

Rolling window complete for [email protected] in 178.14 ms. Input: 5 rows from 2022-08-30T05:42:19.778470000 to 2022-08-30T18:59:24.984779000. Output: 5 rows from 2022-08-30T05:42:19.778470000 to 2022-08-30T18:59:24.984779000

Preprocessed 5 data for logs in 2022-08-30T05:42:19.778470000 to 2022-08-30T18:59:24.984779000 in 100.68392753601074 ms

Downloading artifacts: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 21.18it/s]

Rolling window complete for [email protected] in 248.02 ms. Input: 13 rows from 2022-08-30T00:17:05.561523000 to 2022-08-30T23:08:41.474570000. Output: 13 rows from 2022-08-30T00:17:05.561523000 to 2022-08-30T23:08:41.474570000█████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 21.21it/s]

Preprocessed 13 data for logs in 2022-08-30T00:17:05.561523000 to 2022-08-30T23:08:41.474570000 in 86.13181114196777 ms

Rolling window complete for [email protected] in 294.65 ms. Input: 4 rows from 2022-08-30T03:48:15.868637000 to 2022-08-30T23:49:02.282976000. Output: 4 rows from 2022-08-30T03:48:15.868637000 to 2022-08-30T23:49:02.282976000

Preprocessed 4 data for logs in 2022-08-30T03:48:15.868637000 to 2022-08-30T23:49:02.282976000 in 156.81219100952148 ms

Rolling window complete for [email protected] in 263.97 ms. Input: 4 rows from 2022-08-30T08:20:52.146591000 to 2022-08-30T19:07:44.917975000. Output: 4 rows from 2022-08-30T08:20:52.146591000 to 2022-08-30T19:07:44.917975000

Preprocessed 4 data for logs in 2022-08-30T08:20:52.146591000 to 2022-08-30T19:07:44.917975000 in 221.08125686645508 ms

Rolling window complete for [email protected] in 321.56 ms. Input: 3 rows from 2022-08-30T03:46:30.304629000 to 2022-08-30T06:53:19.650392000. Output: 3 rows from 2022-08-30T03:46:30.304629000 to 2022-08-30T06:53:19.650392000

Preprocessed 3 data for logs in 2022-08-30T03:46:30.304629000 to 2022-08-30T06:53:19.650392000 in 197.83735275268555 ms

Rolling window complete for [email protected] in 294.62 ms. Input: 11 rows from 2022-08-30T02:36:39.981855000 to 2022-08-30T23:30:09.312791000. Output: 11 rows from 2022-08-30T02:36:39.981855000 to 2022-08-30T23:30:09.312791000

Preprocessed 11 data for logs in 2022-08-30T02:36:39.981855000 to 2022-08-30T23:30:09.312791000 in 119.9648380279541 ms

Completed inference for user [email protected]. Model load: 422.1477508544922 ms, Model infer: 1408.827781677246 ms. Start: 2022-08-30 01:54:26.639083, End: 2022-08-30 23:58:05.567378

Rolling window complete for [email protected] in 234.98 ms. Input: 5 rows from 2022-08-30T01:33:19.402330000 to 2022-08-30T18:39:54.214210000. Output: 5 rows from 2022-08-30T01:33:19.402330000 to 2022-08-30T18:39:54.214210000

Preprocessed 5 data for logs in 2022-08-30T01:33:19.402330000 to 2022-08-30T18:39:54.214210000 in 122.68924713134766 ms

Rolling window complete for [email protected] in 200.19 ms. Input: 2 rows from 2022-08-30T05:54:21.257941000 to 2022-08-30T08:11:17.157376000. Output: 2 rows from 2022-08-30T05:54:21.257941000 to 2022-08-30T08:11:17.157376000

Preprocessed 2 data for logs in 2022-08-30T05:54:21.257941000 to 2022-08-30T08:11:17.157376000 in 113.14010620117188 ms

Rolling window complete for [email protected] in 208.18 ms. Input: 9 rows from 2022-08-30T07:42:51.522461000 to 2022-08-30T23:03:40.411836000. Output: 9 rows from 2022-08-30T07:42:51.522461000 to 2022-08-30T23:03:40.411836000

Preprocessed 9 data for logs in 2022-08-30T07:42:51.522461000 to 2022-08-30T23:03:40.411836000 in 78.69076728820801 ms

Rolling window complete for [email protected] in 153.71 ms. Input: 9 rows from 2022-08-30T00:50:13.640088000 to 2022-08-30T23:43:17.639540000. Output: 9 rows from 2022-08-30T00:50:13.640088000 to 2022-08-30T23:43:17.639540000

Input data rate[Complete]: 328 messages [00:01, 141.04 messages/s]E20241025 11:36:06.734444 139952843122368 context.cpp:124] /linear_segment_0/dfp-rolling-window-5; rank: 0; size: 1; tid: 139952843122368: set_exception issued; issuing kill to current runnable. Exception msg: AttributeError: 'int' object has no attribute 'item'
At:

/home/dagardner/work/morpheus-2410reltest/python/morpheus_dfp/morpheus_dfp/stages/dfp_rolling_window_stage.py(155): _build_window

/home/dagardner/work/morpheus-2410reltest/python/morpheus_dfp/morpheus_dfp/stages/dfp_rolling_window_stage.py(178): on_data

Preprocessed 9 data for logs in 2022-08-30T00:50:13.640088000 to 2022-08-30T23:43:17.639540000 in 190.80257415771484 ms

Completed inference for user [email protected]. Model load: 72.72815704345703 ms, Model infer: 806.1537742614746 ms. Start: 2022-08-30 01:23:39.156080, End: 2022-08-30 23:05:47.146155

Input data rate[Complete]: 328 messages [00:01, 141.04 messagesE20241025 11:36:06.852890 139972469520192 runner.cpp:189] Runner::await_join - an exception was caught while awaiting on one or more contexts/instances - rethrowing

E20241025 11:36:06.853118 139972469520192 segment_instance.cpp:273] segment::SegmentInstance - an exception was caught while awaiting on one or more nodes - rethrowing

E20241025 11:36:06.853158 139972469520192 service.cpp:224] Service[segment::SegmentInstance]: caught exception in service_await_join: AttributeError: 'int' object has no attribute 'item'
At:

/home/dagardner/work/morpheus-2410reltest/python/morpheus_dfp/morpheus_dfp/stages/dfp_rolling_window_stage.py(155): _build_window

/home/dagardner/work/morpheus-2410reltest/python/morpheus_dfp/morpheus_dfp/stages/dfp_rolling_window_stage.py(178): on_data

E20241025 11:36:06.853219 139972469520192 pipeline_instance.cpp:230] pipeline::PipelineInstance - an exception was caught while awaiting on segments - rethrowing

E20241025 11:36:06.853259 139972469520192 service.cpp:224] Service[pipeline::PipelineInstance]: caught exception in service_await_join: AttributeError: 'int' object has no attribute 'item'
At:

/home/dagardner/work/morpheus-2410reltest/python/morpheus_dfp/morpheus_dfp/stages/dfp_rolling_window_stage.py(155): _build_window

/home/dagardner/work/morpheus-2410reltest/python/morpheus_dfp/morpheus_dfp/stages/dfp_rolling_window_stage.py(178): on_data

E20241025 11:36:06.853353 139972469520192 service.cpp:224] Service[pipeline::Manager]: caught exception in service_await_join: AttributeError: 'int' object has no attribute 'item'
At:

/home/dagardner/work/morpheus-2410reltest/python/morpheus_dfp/morpheus_dfp/stages/dfp_rolling_window_stage.py(155): _build_window

/home/dagardner/work/morpheus-2410reltest/python/morpheus_dfp/morpheus_dfp/stages/dfp_rolling_window_stage.py(178): on_data

E20241025 11:36:06.853393 139972469520192 service.cpp:224] Service[ExecutorDefinition]: caught exception in service_await_join: AttributeError: 'int' object has no attribute 'item'
At:

/home/dagardner/work/morpheus-2410reltest/python/morpheus_dfp/morpheus_dfp/stages/dfp_rolling_window_stage.py(155): _build_window

/home/dagardner/work/morpheus-2410reltest/python/morpheus_dfp/morpheus_dfp/stages/dfp_rolling_window_stage.py(178): on_data

Input data rate[Complete]: 328 messages [00:01, 326.02 messages/s]

Exception occurred in pipeline. Rethrowing

Traceback (most recent call last):

File "/home/dagardner/work/morpheus-2410reltest/python/morpheus/morpheus/pipeline/pipeline.py", line 408, in post_start

await executor.join_async()

File "/home/dagardner/work/morpheus-2410reltest/python/morpheus_dfp/morpheus_dfp/stages/dfp_rolling_window_stage.py", line 178, in on_data

result = self._build_window(message)

File "/home/dagardner/work/morpheus-2410reltest/python/morpheus_dfp/morpheus_dfp/stages/dfp_rolling_window_stage.py", line 155, in _build_window

first_row_idx = match.index[0].item()

AttributeError: 'int' object has no attribute 'item'

Inference rate[Complete]: 13 messages [00:03,  4.01 messages/s]

====Pipeline Complete====

Traceback (most recent call last):

File "/home/dagardner/work/morpheus-2410reltest/examples/digital_fingerprinting/production/dfp_azure_pipeline.py", line 460, in 

run_pipeline(obj={}, auto_envvar_prefix='DFP', show_default=True, prog_name="dfp")

File "/home/dagardner/work/conda/envs/morpheus/envs/morpheus-2410reltest/lib/python3.10/site-packages/click/core.py", line 1157, in call

return self.main(*args, **kwargs)

File "/home/dagardner/work/conda/envs/morpheus/envs/morpheus-2410reltest/lib/python3.10/site-packages/click/core.py", line 1078, in main

rv = self.invoke(ctx)

File "/home/dagardner/work/conda/envs/morpheus/envs/morpheus-2410reltest/lib/python3.10/site-packages/click/core.py", line 1434, in invoke

return ctx.invoke(self.callback, **ctx.params)

File "/home/dagardner/work/conda/envs/morpheus/envs/morpheus-2410reltest/lib/python3.10/site-packages/click/core.py", line 783, in invoke

return __callback(*args, **kwargs)

File "/home/dagardner/work/morpheus-2410reltest/examples/digital_fingerprinting/production/dfp_azure_pipeline.py", line 455, in run_pipeline

pipeline.run()

File "/home/dagardner/work/morpheus-2410reltest/python/morpheus/morpheus/pipeline/pipeline.py", line 654, in run

asyncio.run(self.run_async())

File "/home/dagardner/work/conda/envs/morpheus/envs/morpheus-2410reltest/lib/python3.10/asyncio/runners.py", line 44, in run

return loop.run_until_complete(main)

File "/home/dagardner/work/conda/envs/morpheus/envs/morpheus-2410reltest/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete

return future.result()

File "/home/dagardner/work/morpheus-2410reltest/python/morpheus/morpheus/pipeline/pipeline.py", line 635, in run_async

await self.join()

File "/home/dagardner/work/morpheus-2410reltest/python/morpheus/morpheus/pipeline/pipeline.py", line 452, in join

await self._post_start_future

File "/home/dagardner/work/morpheus-2410reltest/python/morpheus/morpheus/pipeline/pipeline.py", line 408, in post_start

await executor.join_async()

File "/home/dagardner/work/morpheus-2410reltest/python/morpheus_dfp/morpheus_dfp/stages/dfp_rolling_window_stage.py", line 178, in on_data

result = self._build_window(message)

File "/home/dagardner/work/morpheus-2410reltest/python/morpheus_dfp/morpheus_dfp/stages/dfp_rolling_window_stage.py", line 155, in _build_window

first_row_idx = match.index[0].item()

AttributeError: 'int' object has no attribute 'item'

Full env printout

Click here to see environment details

[Paste the results of print_env.sh here, it will be hidden by default]

Other/Misc.

No response

Code of Conduct

• I agree to follow Morpheus' Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report