Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]: WebScraperStage ignores value for _cache_dir #1355

Closed
2 tasks done
dagardner-nv opened this issue Nov 9, 2023 · 1 comment
Closed
2 tasks done

[BUG]: WebScraperStage ignores value for _cache_dir #1355

dagardner-nv opened this issue Nov 9, 2023 · 1 comment
Labels
bug Something isn't working
Milestone
23.11 - Sherlock

Comments

@dagardner-nv
Copy link
Contributor

Version

23.11

Which installation method(s) does this occur on?

Source

Describe the bug.

This value should be exposed as a constructor argument, it also hard-codes the name of the cache db from the RSSSourceStage.

        self._cache_dir = "./.cache/llm/rss/"

        # Ensure the directory exists
        os.makedirs(self._cache_dir, exist_ok=True)
        ...
        self._session = requests_cache.CachedSession(os.path.join("./.cache/http", "RSSDownloadStage.sqlite"),
                                                     backend="sqlite")

Minimum reproducible example

No response

Relevant log output

Click here to see error details
[Paste the error here, it will be hidden by default]

Full env printout

Click here to see environment details
[Paste the results of print_env.sh here, it will be hidden by default]

Other/Misc.

No response

Code of Conduct

  • I agree to follow Morpheus' Code of Conduct
  • I have searched the open bugs and have found no duplicates for this bug report
@dagardner-nv dagardner-nv added the bug Something isn't working label Nov 9, 2023
@dagardner-nv dagardner-nv added this to the 23.11 - Sherlock milestone Nov 9, 2023
@cwharris cwharris mentioned this issue Nov 13, 2023
Merged
rapids-bot bot pushed a commit that referenced this issue Nov 14, 2023
@cwharris
Add cache_path arg to WebScraperStage (#1358)
264d950 
Allows injecting an optional cache path to the WebScraperStage. If `cache_path` is not None, it uses the path as the path for the response caching system's sqllite database. If `cache_path` is None, the response caching system is bypassed.

Closes #1355

Authors:
  - Christopher Harris (https://github.com/cwharris)

Approvers:
  - Devin Robison (https://github.com/drobison00)

URL: #1358
@cwharris
Copy link
Contributor

Closed by #1358

@github-project-automation github-project-automation bot moved this from Todo to Done in Morpheus Boards Nov 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Morpheus Boards
Status: Done
Milestone
23.11 - Sherlock
Development

No branches or pull requests
2 participants
@cwharris @dagardner-nv