Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]: DFP file_batcher module always uses default date regex pattern #1576

Closed
2 tasks done
efajardo-nv opened this issue Mar 22, 2024 · 0 comments · Fixed by #1580
Closed
2 tasks done

[BUG]: DFP file_batcher module always uses default date regex pattern #1576

efajardo-nv opened this issue Mar 22, 2024 · 0 comments · Fixed by #1580
Labels
bug Something isn't working

Comments

@efajardo-nv
Copy link
Contributor

Version

23.11

Which installation method(s) does this occur on?

Conda

Describe the bug.

This line in the file_batcher looks for batch_iso_date_regex_pattern in the module's config which does not exists as key so will always use DEFAULT_ISO_DATE_REGEX_PATTERN

iso_date_regex_pattern = config.get("batch_iso_date_regex_pattern", DEFAULT_ISO_DATE_REGEX_PATTERN)

This results in the default value being used if the pattern doesn't match filenames. Consequently, the program falls back to the last modified time for all files, potentially causing entire datasets to be loaded at once and leading to CUDF overflow errors for large files.

Minimum reproducible example

Run DFP pipeline on input files with names that don't follow default regex pattern.

Relevant log output

Click here to see error details
[Paste the error here, it will be hidden by default]

Full env printout

Click here to see environment details
[Paste the results of print_env.sh here, it will be hidden by default]

Other/Misc.

No response

Code of Conduct

  • I agree to follow Morpheus' Code of Conduct
  • I have searched the open bugs and have found no duplicates for this bug report
@efajardo-nv efajardo-nv added the bug Something isn't working label Mar 22, 2024
@efajardo-nv efajardo-nv mentioned this issue Mar 25, 2024
Merged
@jarmak-nv jarmak-nv moved this from Todo to Review - Ready for Review in Morpheus Boards Mar 25, 2024
rapids-bot bot pushed a commit that referenced this issue Mar 27, 2024
@efajardo-nv
Fix iso_date_regex_pattern config in file_batcher module and allo…
1fc9588 
…w override (#1580)

- Update `file_batcher`module to use documented module config key `iso_date_regex_pattern` instead of `batch_iso_date_regex_pattern`.
- Allow override of `iso_date_regex_pattern` with control message.
- Add tests for `file_batcher` module

Closes #1576 
Closes #1577

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md).
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - Eli Fajardo (https://github.com/efajardo-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #1580
@github-project-automation github-project-automation bot moved this from Review - Ready for Review to Done in Morpheus Boards Mar 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Morpheus Boards
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix iso_date_regex_pattern config in file_batcher module and allow override efajardo-nv/Morpheus
1 participant
@efajardo-nv