[BUG]: --stop_after flag not always able to halt vdb_upload pipeline

[dagardner-nv]

Version

24.03

Which installation method(s) does this occur on?

Source

Describe the bug.

Using the --stop_after=1024 should cause the source to stop after emitting 1024 records, instead at 1024 this is logged:

Stop limit reached... preparing to halt the source.
RSS source exhausted, stopping.

Yet the pipeline continues to execute. It appears that the feed parser continues to endlessly retry failed urls.

Minimum reproducible example

python examples/llm/main.py vdb_upload pipeline --stop_after=1024

Relevant log output

Click here to see error details

Stop limit reached... preparing to halt the source.

RSS source exhausted, stopping.

Full env printout

Click here to see environment details

[Paste the results of print_env.sh here, it will be hidden by default]

Other/Misc.

No response

Code of Conduct

• I agree to follow Morpheus' Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report

[dagardner-nv]

Invalid issue, source completes as expected, additional logs are being generated bu the web scraper stage.