Integrate scraping #15
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Closed
freddyheppell
added a commit
that referenced
this pull request
Jul 11, 2024
* migrate to poetry (#3) * migrate to poetry * move poetry install first * update lockfile * fix script declaration * update docs * bump workflow versions * Support more Python versions (#4) * test up to py3.12 * drop py3.8 as it's nearly eol * bump package versions * Fix removed numpy NaN * update py version in readme * Change linting to Ruff (#5) * Install ruff * remove makefile * ruff check fix * manual ruff fixes * undo .at to .loc with noqa * update lint workflow * delete old flake8 config * fix CI lint commands * convert pickle fixtures to json tables (#6) * Remove yoast plugin requirement (#12) * better handling without yoast plugin * run ruff * Better support no scrape (#13) * Allow empty scrape properly * ruff lint * Integrate scraping (#15) * incorporate wp-json-scraper module * ruff autofix * docconvert * manual fixes * fix noqa comment * refactor cli to support subcommands * Basic integration of downloader * swap progress bar implementation * ruff * remove object display code * add media dl support * lint * remove totally unused modules * remove unused csv export code * ruff * implement proxy/cookies/auth on download client * improve subparser documentation * ruff * add NOTICE file * remove plugin list * remove console util * dl utils cleanup * implement request robustness features * make media dl use request session * Add docs (#18) * docs * ruff * fix broken test * fix typing union * add deploy dir to gitignore * change project name to match package name * hotfix: remove docconvert * hotfix: fix package name for version get * Packaging improvements (#19) * package meta and readme updates * relax dependency constraints * regen lock * readme tweaks * put back gh markdown admonition * rename main package for consistency (#20) * rename main package * ruff * Feature/prefix consistency (#21) * wip * make prefix behaviour consistent between commands * update cli docs * Some tests for downloader (#22) * add downloader tests * test no prefix downloader too * test wpapi and exporter * ruff * hotfix: print and wrong command name * fix dl input directory path * Prerelease restructure (#23) * expose dl api a bit better * improve docs for dl module * enable pyupgrade rules * enable ruff specific rules * remove some doc noqas * ruff reformat * fix export decorator usage * change dl media to use pathlib * fix build system in pyproject * minor cli and docs fixes * add changelog for 1.0.0 * clean up naming of translation pickers * set version to 1.0.0rc1 * change pytest to use importlib mode (#25) * Change CLI to use click (#26) * swap to click for cli * ruff * remove old argparse support * use download instead of dl consistently * update docs for cli * ruff * Feature/packaging meta (#27) * add classifiers * update license docs to match new path * rename repo * Remove print statements (#28) * fix name in notice file * replace prints with logs * add cli version * Ruff * prepare release 1.0.0 * add publish workflow * Add building to CI test
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Integrate wp-json-scraper into this codebase.
Closes #10
remove or refactor out utilspartial save/resume?made separate issue Download saving and resuming #16HTML scrape (or command to generate list to pass towget?)