DEV-458 end-to-end automated test for metadata workflow #46

moseshll · 2024-11-13T22:07:28Z

README updates on standard directory locations
README updates on daily script inputs and outputs

spec/unit/post_zephir_verifier_spec.rb

- README updates on standard directory locations - README updates on daily script inputs and outputs

* accepts empty lines * accepts 9 digit catalog record IDs * rejects all other non-whitespace lines * sets up empty journal * sets up logging to string for testing

- `with_test_environment` helper.

* Fix regex for verifying digitization source

* Add optional 'gzipped' argument * Make 'errmsg' a keyword argument

* verify lines have the expected number of columns * verify fields by regex as best we can Still to do: test hathifiles line counts against catalog; directly test HathifileContentsVerifier

- Add `database` to `Services` canister - Add `PopulateRightsVerifier` with unit tests

* Squelch logging output from tests -- since we now have an 'errors' method to check this in tests To do: integration test for hathifiles

* pull repeated uses to :around blocks * override TMPDIR by default

* Use env vars directly in derivative, or extract logic from directory_for where needed * Move filename logic from Verifier to Derivative

* move helpers out of spec helper to derivatives * use ClimateControl * move test for all missing files to derivatives integration * raise exception if requesting nonexistent 'full' deletes file

* clean up requires

* Reduces level of nested ifs * Separate catalog full verification from catalog update -- rather different behavior * Also: update documentation regarding derivatives / paths

lib/verifier/post_zephir_verifier.rb

lib/verifier/hathifiles_redirects_verifier.rb

* prep bin/verify.rb for testing * try running bin/verify.rb runs * use run_for_date by default; remove requirement for journal

lib/verifier.rb

lib/verifier/hathifiles_redirects_verifier.rb

spec/integration/derivatives_integration_spec.rb

spec/unit/hathifile_derivative_spec.rb

* sufficient fixtures to run all verifiers (some fixtures have been moved/renamed and adjusted in tests appropriately) * verify.rb (for now) notes which date is needed for 'today' * extract solr catalog & hathifile database helpers to spec/support * info logging

…n date" and not "datestamp" - `Derivative` subclasses can provide a timestamp delta - PostZephirDerivatives and Dates also changed to "run date" semantics.

…integration_spec.rb` - Move `hathifile_derivative_spec.rb` to `derivative/hathifile_spec.rb` - Rename `HathifileRedirectsVerifier` to `HathifilesRedirectsVerifier`

previous commit changed everything use current date instead of filename date, so we can call the verifiers as they were before.

…n to monthlies) - Requires some wrangling of TMPDIR to find `ht_bib_export_incr...` and dollar dup files - Update .gitignore to just ignore data/ so we don't miss new fixtures

- For now these ENV values are taken from hathifiles secret - `database.yml` appears to be read-write to the rights DB (in practice) so using it for Verifier is not appropriate.

…ng a trailing digit.

moseshll marked this pull request as draft November 13, 2024 22:07

aelkiss reviewed Dec 3, 2024

View reviewed changes

spec/unit/post_zephir_verifier_spec.rb Outdated Show resolved Hide resolved

moseshll and others added 24 commits December 5, 2024 15:58

DEV-458 end-to-end automated test for metadata workflow

890b4ae

- README updates on standard directory locations - README updates on daily script inputs and outputs

Minor README adjustments.

066806b

Add log level and DATA_ROOT to docker-compose.

a6093f3

Add Canister and Climate Control gems; add logger service.

88cafaa

Add Journal class and write it from post_zephir.rb.

270f428

Reconcile README and updated default TMPDIR.

cccf97a

Prototype Verifier.

3f50009

Flesh out enough details for others to evaluate.

9bdc2ee

DEV-1421 (WIP): verify delete files

e0107ce

DEV-1421: verify delete files

787d056

* accepts empty lines * accepts 9 digit catalog record IDs * rejects all other non-whitespace lines * sets up empty journal * sets up logging to string for testing

DEV-1421: rubocop fixes

8429ed7

- spec for Verifier class.

93ddd2e

- `with_test_environment` helper.

added verify_rights_file_format, some tests, and test helpers

48c1b58

more tests for verify_rights_file_format

48e41b3

Use verifier.errors and with_test_environment

8ae1a90

* Fix regex for verifying digitization source

Use extracted methods for checking delete file

eadc0cd

* Add optional 'gzipped' argument * Make 'errmsg' a keyword argument

move with_temp_file to spec_helper and correct

2e1c7d0

DEV-1414: stub out verification & tests for hathifiles

78d572d

DEV-1414: hathifiles field verification

4a6b693

* verify lines have the expected number of columns * verify fields by regex as best we can Still to do: test hathifiles line counts against catalog; directly test HathifileContentsVerifier

fixup: run standardrb

d85c1df

fixup: add hathifile fixture

2c231d6

- Add Dockerfile/Gemfile support for Sequel

abcb0e4

- Add `database` to `Services` canister - Add `PopulateRightsVerifier` with unit tests

DEV-1414: Test HathifilesVerifier methods directly

5565629

DEV-1414: compute catalog file for given hathifile name

a2623bc

moseshll force-pushed the DEV-458_E2E_workflow_test branch from bff9a91 to a2623bc Compare December 5, 2024 21:12

aelkiss and others added 3 commits December 5, 2024 16:43

DEV-1414: Test hathifile line count

62c7b63

* Squelch logging output from tests -- since we now have an 'errors' method to check this in tests To do: integration test for hathifiles

fix regex to disallow uppercase in volid

4b16ab5

dev-1420 dry out some tests

c59adda

moseshll and others added 13 commits December 18, 2024 16:23

Appease standardrb

912ffe4

Batch rights_current checks in PopulateRightsVerifier; refactor tests

fdeadcc

nested conditions

6094687

PostZephirVerifier.verify_catalog_prep now using Derivative::Delete

2970aa5

added, implemented and tested Derivative::IngestBibrecord

5498a33

made and implemented Derivative::HTBibExport

c1aee80

changed require_relative to require where possible

46a86d5

Clean up some uses of ClimateControl

aa73c64

* pull repeated uses to :around blocks * override TMPDIR by default

Remove directory_for; datestamped_derivative

6deabe3

* Use env vars directly in derivative, or extract logic from directory_for where needed * Move filename logic from Verifier to Derivative

dropped zinzout, uzing zlib everywhere for consistency

48d1a64

Refactor derivatives integration spec

1435ea6

* move helpers out of spec helper to derivatives * use ClimateControl * move test for all missing files to derivatives integration * raise exception if requesting nonexistent 'full' deletes file

Rename Derivatives -> PostZephirDerivatives

7732354

* clean up requires

Use guard clauses for verify_file, etc

75fe457

* Reduces level of nested ifs * Separate catalog full verification from catalog update -- rather different behavior * Also: update documentation regarding derivatives / paths

aelkiss reviewed Dec 19, 2024

View reviewed changes

lib/verifier/post_zephir_verifier.rb Outdated Show resolved Hide resolved

Remove extra with_test_environment in hathifiles contents verifier

77ce92d

aelkiss reviewed Dec 19, 2024

View reviewed changes

lib/verifier/hathifiles_redirects_verifier.rb Outdated Show resolved Hide resolved

Add verifier classes to verifier script

3bdde11

* prep bin/verify.rb for testing * try running bin/verify.rb runs * use run_for_date by default; remove requirement for journal

aelkiss reviewed Dec 20, 2024

View reviewed changes

lib/verifier.rb Show resolved Hide resolved

aelkiss reviewed Dec 20, 2024

View reviewed changes

lib/verifier/hathifiles_redirects_verifier.rb Outdated Show resolved Hide resolved

aelkiss reviewed Dec 20, 2024

View reviewed changes

spec/integration/derivatives_integration_spec.rb Outdated Show resolved Hide resolved

aelkiss reviewed Dec 20, 2024

View reviewed changes

spec/unit/hathifile_derivative_spec.rb Outdated Show resolved Hide resolved

aelkiss and others added 9 commits December 20, 2024 14:50

- Change the semantics of run_for_date parameter to always mean "ru…

bd141dd

…n date" and not "datestamp" - `Derivative` subclasses can provide a timestamp delta - PostZephirDerivatives and Dates also changed to "run date" semantics.

- Move derivatives_integration_spec.rb to `post_zephir_derivatives_…

6fa3bcd

…integration_spec.rb` - Move `hathifile_derivative_spec.rb` to `derivative/hathifile_spec.rb` - Rename `HathifileRedirectsVerifier` to `HathifilesRedirectsVerifier`

run_verifiers: use class names instead of lambdas

829d4fe

previous commit changed everything use current date instead of filename date, so we can call the verifiers as they were before.

- Extend CATALOG_ARCHIVE line count check to update files (in additio…

0bbde91

…n to monthlies) - Requires some wrangling of TMPDIR to find `ht_bib_export_incr...` and dollar dup files - Update .gitignore to just ignore data/ so we don't miss new fixtures

- Database connection uses ENV instead of database.yml

a74a806

- For now these ENV values are taken from hathifiles secret - `database.yml` appears to be read-write to the rights DB (in practice) so using it for Verifier is not appropriate.

Address #55 Move WhateverVerifier to Verifier::Whatever

b71f6ff

Allow hathifiles digitization_agent_code to match yale2 by allowi…

58cf99c

…ng a trailing digit.

Add exception handler around Solr results to diagnose testing issue

c8f19b6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DEV-458 end-to-end automated test for metadata workflow #46

DEV-458 end-to-end automated test for metadata workflow #46

moseshll commented Nov 13, 2024

DEV-458 end-to-end automated test for metadata workflow #46

Are you sure you want to change the base?

DEV-458 end-to-end automated test for metadata workflow #46

Conversation

moseshll commented Nov 13, 2024