19910 - Python 3.12.2 upgrade #1435
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
seeker25
changed the title
…flask, upgrade sqlalchemy - still need to fix "tracing", still need to fix sql-continuum, still need to fix exception handling, still need to fix JWT library flask-oidc-jwt (doesn't work with Flask 3.0 without update)
seeker25
commented
Mar 4, 2024
pay-api/migrations/versions/03b2c7caed21_populate_receipt_number_payments.py
Show resolved
Hide resolved
seeker25
commented
Mar 4, 2024
seeker25
commented
Mar 4, 2024
seeker25
commented
Mar 4, 2024
seeker25
commented
Mar 4, 2024
|
|
||
| db.session.add(activity) | ||
| # TODO fix this later | ||
| # if isinstance(obj, VersionedModel) and not current_app.config.get('DISABLE_ACTIVITY_LOGS'): |
There was a problem hiding this comment.
Need SQL-continuum replacement
seeker25
commented
Mar 4, 2024
| @@ -27,6 +27,8 @@ | |||
| db = SQLAlchemy(query_class=CustomQuery) # pylint: disable=invalid-name | |||
|
|
|||
|
|
|||
| activity_plugin = ActivityPlugin() # pylint: disable=invalid-name | |||
| # TODO FIX THIS | |||
| # activity_plugin = ActivityPlugin() # pylint: disable=invalid-name | |||
There was a problem hiding this comment.
TODO: Need to replace this something - Need SQL-continuum replacement
seeker25
commented
Mar 4, 2024
seeker25
commented
Mar 4, 2024
seeker25
commented
Mar 4, 2024
seeker25
commented
Mar 4, 2024
seeker25
commented
Mar 4, 2024
|
|
||
| def publish_to_queue(payload: dict, invoice: Invoice): | ||
| """Publish a 'COMPLETED' invoice's info to the GCP PubSub Queue.""" | ||
| def publish_to_queue(payload: dict): |
There was a problem hiding this comment.
TODO, still needs some work
seeker25
commented
Mar 4, 2024
seeker25
commented
Mar 4, 2024
| @@ -168,7 +168,7 @@ def update_statement_settings(auth_account_id: str, frequency: str): | |||
|
|
|||
| max_frequency = StatementSettings._find_longest_frequency(current_statements_settings.frequency, frequency) | |||
| last_date = StatementSettings._get_end_of(max_frequency) | |||
| current_statements_settings.to_date = last_date | |||
| current_statements_settings.to_date = last_date # TODO Should be date not date time? | |||
There was a problem hiding this comment.
.date() I was thinking about here
seeker25
commented
Mar 4, 2024
seeker25
commented
Mar 4, 2024
seeker25
commented
Mar 4, 2024
| return _app | ||
|
|
||
|
|
||
| @pytest.fixture(autouse=True) | ||
| def mock_queue_publish(monkeypatch): |
There was a problem hiding this comment.
General mock for queue publishing, easier to do this than use it when needed
seeker25
commented
Mar 4, 2024
| # This is the path we'll use in legal_api!! | ||
|
|
||
| # even though this isn't referenced directly, it sets up the internal configs that upgrade needs | ||
| if database_exists(_db.engine.url): |
There was a problem hiding this comment.
Quite a bit different than LEAR, I see LEAR doing alot of this, but it's easier to use the library I think
seeker25
commented
Mar 4, 2024
| # Handle where test DOESN'T session.commit() | ||
| sess2.expire_all() | ||
| nested = sess.begin_nested() | ||
| # When using a SAVEPOINT via the Session.begin_nested() or Connection.begin_nested() methods, |
There was a problem hiding this comment.
Self explainatory.. if you commit or rollback on a session or a connection, it rolls back the outer object.. which is why we needed to patch the commit and rollbacks in nested operations.
seeker25
commented
Mar 4, 2024
| @@ -22,18 +22,6 @@ services: | |||
| retries: 10 | |||
| volumes: | |||
| - ./setup:/tmp/keycloak/test/ | |||
| nats: | |||
There was a problem hiding this comment.
Future: Maybe we need a docker container for pubsub? I think it's mostly HTTP requests (push)?
seeker25
commented
Mar 4, 2024
|
seeker25
added a commit
that referenced
this pull request
May 31, 2024
* 19910 - Python 3.12.2 upgrade (#1435) * Update docker files to python 3.12.2 * Update CI flows * update requirements * Update grcpiostatus and protobuf * Add python 3.12 to test CI * Add in more python 3.12 * Fix flake8 * Get the bulk of the unit tests working, upgrade marshmellow, upgrade flask, upgrade sqlalchemy - still need to fix "tracing", still need to fix sql-continuum, still need to fix exception handling, still need to fix JWT library flask-oidc-jwt (doesn't work with Flask 3.0 without update) * Small cleanup * remove autoflush param, didn't mean for it to be in there * Update requirements to pull from github, where we fixed the library * Clear out some old TODOs * clear out old todo * remove todo * old todo * todo removal * Remove todo * change default database name * Use more modern database * nuke tracer for now * add in ci for queue_python_upgrade * Fix linting * remove setup.cfg changes * fix unit test * Small unit test fix * use utcnow instead of now. * Move created on to utcnow * unit test fixes * Small unit test tweak * use datetime.now(tz=timezone.utc) * up google-auth version * Update prod.txt * Update requirements.txt * Update prod.txt * fix migration bug (#1440) * Changes to upgrade to Python 3.12 * lint fix * Small unit test fixes * Fix pay-admin * Update ftp-poller's requirements * 19939 - EFT - Invoice Reference (#1436) * Squashed commit of the following: commit e1c58f9 Author: Rodrigo Barraza <[email protected]> Date: Tue Mar 5 08:42:01 2024 -0800 Cleanup auto_save commit 9b8a9a0 Author: Rodrigo Barraza <[email protected]> Date: Tue Mar 5 08:33:27 2024 -0800 Remove committing commit a05e210 Author: Rodrigo Barraza <[email protected]> Date: Tue Mar 5 08:30:07 2024 -0800 Reference and Receipt creation commit edb6e51 Author: Rodrigo Barraza <[email protected]> Date: Tue Mar 5 08:18:47 2024 -0800 Updates to EFT Service Handling invoice reference creation * Reverting reconciliation changes * PR feedback updates * 19939 - EFT Reconciliation - Invoice Reference (#1437) * Updates to EFT Service Handling invoice reference creation * Reference and Receipt creation * Remove committing * Cleanup auto_save * Revert EFT service * PR feedback updates Bringing back auto_save * 19724 - EFT - Create CFS account job modification (#1441) * update api and payment-jobs for EFT cfs_account create * add updated requirements.txt from PR#18263 * isort fix * 18263 - New queues use PUB/SUB (#1438) * Update requirements.txt * Fix CI * Changes to convert project to new queues * Remove nats from workflows * remove nats from docker compose * remove q_cli * Remove more NATS * Remove more nats * Gun events-listener, combining the queues for less tech debt * More nats removal * Move payment-reconciliations into pay-queue * Change project structure of queues * Add in tests from events listener * Getting rid of some infrastructure and tech debt we don't need * Enhance gcp_queue_publisher * Wire up ftp poller to new queues * Add in enum, change util func * Small tweaks for gcp_queue_publisher * Put in adhoc test for publishing to queue * Add in PaymentToken dataclass shared across apps * shorten enums * Replace queue messages in mailer in payment-jobs * Add more MessageTypes to enum * More enum * Convert to PUB * missing comma * Add in pub * Rework entire PUB * Clean up clean up everybody clean up! * Small fix for ftp-poller * Fix ref * Holy moly third try it must be friday * fix lint * remove unused receipt method * payment jobs updates, makefile for pay-queue * linting fixes, requirements fixes * Dependencies are being funny with this one - will need to relook at it. This should work for now. * Update old references * Add in pay jobs CI * directory change * Use pip 24.0.0 * Flask-Migrate upgrade + disable compare_type * Small tweaks for jobs, still some failing tests * Use scalars to fix where conditions when there are tuples, as those cause exceptions * Fix CI * Fix unit test for pay-api * Lint fixes and others * Update CI * fix deps * update reqs * Take out tracing for BCOL-API * more tracing removal * 1 more spot for tracing * Fix linting, move pubsub to local, had issues referencing it as a library * Update requirements * More test / lint fixes * Cleanup conftest * more cleanup * Some unit tests working, more to go * More cleanup * 4 unit tests to go * Fix eft reconciliation tests * Lint fixes more unit test fixes * Unit tests should be all passing * Remove redundant comment * Cleanup * Lint fix * Add in code coverage for pay-queue * Put in event listener topic * Some cleanup * Update requirements * 19724_2 - EFT - Create CFS account job (#1442) * update api and payment-jobs for EFT cfs_account create * add updated requirements.txt from PR#18263 * isort fix * EFT - Create CFS account job - 2 * remove redundant function * remove incorrect comment * Pay API CD upgrade. * Rollback the auto formatting. * Fixed flake8. * 20299 - Poetry upgrade (#1443) * Migrate to Poetry * More cleanup, add dev deps * Init poetry, add in deps * Docker file poetry for bcol-api * Revert makefile changes * Fix make files * Docker file + Makefile update * makefile update * Makefile update * Fix makefiles * Fix pay-admin docker * update pay queue dockerfile * minor nudge * one more nudge * Fix missing port * Fix poetry for docker * attempt to fix CI * attempt to fix pay-queue migrations * attempt to fix CI #999 * Update paths * Fix pay queues migration path * Fix path for jobs * rename payment-jobs -> jobs * Update maintainer * CI fix * Add pay api gcp CD flow. (#1446) (#1447) Co-authored-by: Patrick Wei <[email protected]> * First run at pubsub emulation for integration tests + pg8000 upgrade (#1448) * First run at pubsub emulation * Use pg8000 instead of psycopg2 * Remove requirement of keys allow for unauthenticated client * Change gunicorn to match lear * Remove prehooks, no concepts of prehooks anymore * Remove jaeger-client * Fix unit tests * Update dependencies * Use test environment variables * Update secret for linux * Checkout v4 * Add in extra hosts * bind to 0.0.0.0 * Upgrade CD flows for Payment projects. (#1449) * Upgrade CD flows for Payment projects. * fixed linting. * Fixed empty value issue. * Fixed deployment config issue. * Fixed deployment environment variables issue. * Fixed the configuration. * Fixed CD issues. (#1452) * Fixed empty value issue. * Fixed deployment config issue. * Fixed deployment environment variables issue. * Fixed the configuration. * Fixed missing service account. * Fixed deployment config issue. * Fixed deployment environment variables issue. * Fixed missing service account. * 19875 - PAY Jobs - Disbursement Process handle Partial Refunds (#1428) * Fixed typo. * 14521 - Get rid of sqlalchemy-continuum (#1450) * Use new versioning library thanks to Thor/ people working on LEAR. Disable activity log for now, we'll add audit columns and just use the history tables. Still need a migration to move data from verisons -> history tables. * Add in version column * Add in default for version * Small nudge * Another tweak refunds_partial table missing * Migration fixes along the way * Migration query * Easier for copy and paste * Credit for script * Fix multiple head issue * Small tweaks, implement payment_methods again * Fix warnings + depreciated cls.query.get() * remove pytest-asyncio * Implement versioning with a unit test * Remove unused statement, lint cleanup * Change commit -> flush * Add in some checks to see if column has changed before updating it * Rollback conditional checks. This also reverts commit 1e84bec. * Minor touch ups. * 19936 - EFT - PAY-API changes to handle CFS when switching payment methods (#1445) * add eft cfs_account create to pay-api * add unit test for eft payment account switch * pylint fix * lint fix * update eft cfs account creation * Add in logging * move Migrate into __init__.py instead of in wsgi.py * move out check for migration mode * Enhance logging * Small touch ups to get it to work in docker and hopefully gcp * 19721 - Updating EFT service for EFT invoice task (#1455) * Update EFT service * Expose EFT Service * Update logging * Enum support for invoice tasks (#1457) * 19721- Create EFT Invoices task (#1456) * Create EFT Invoices task * Linting * Remove comments, update commit hash * Adding enums, code feedback * timezone fix * Refactoring into smaller functions * enum support * Poetry lock update * Linting * pylint cleanup * Delete enums.py * Increased tests and final fixes * Remove print * Sync from main (#1460) * Add pay api gcp CD flow. (#1446) * Bump pillow from 10.2.0 to 10.3.0 in /report-api (#1458) Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.2.0 to 10.3.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@10.2.0...10.3.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add in TZ conversion (#1459) --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Patrick Wei <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * add verification logic in some .setters to avoid unnecessary db actions (#1462) * 19723 - Exposing eft short names service (#1466) * Include EFTShortNames in service * as EftShortNamesService * Lint fix * 19723 - Linking EFT Job (#1464) * Linking EFTs * Minor cleanup * Tests * Remove print * 20457 - SBC-PAY Common Queue (#1467) * Pub/Sub Integration * fix unit test * unit test fix * queue name change * lint fix * lint fix * unit test fix * pay queue unit test fix * 19496 refactor eft shortnames to support multiple account linking and states (#1488) * 19496 -EFT TDI17 Processing / Multi Account linking (#1489) * 19496 - EFT TDI17 processing update, Multi account linking fixes * update queue poetry.lock * linting * 20454 - Pay-queue JWT Verification (#1491) * 20454 - Fix a jwt condition check issue (#1492) * just for test * pubsub jwt verification * remove extra code * add back DISABLE_PAD_SUCCESS_EMAIL, it deleted by incidence * 20454 - Add test logs (#1493) * 20414 - EFT Short name summaries search (#1494) * 20414 - EFT Short name summaries search * PR feedback * 19722 - Unlinking EFT Task (#1487) * Unlinking EFT * Flake fixes * Unlinking EFT test * Cleanup and fixes * Fixing tests * Updates and fixes * renaming function * Lint fixes * Flake8 fix * Test fix * 20420 - EFT Short name search (#1496) * 20420 - EFT Short name search - refactor for EFT Enabled Accounts Table search - clean up params / schema * version increment * clean up statement filter * 20421 - Add cfs account status to search eft names (#1525) * 20417 - EFT Shortname links / transaction search (#1526) * 20417 - EFT Shortname links / transaction search * fixes after doing some linking tests with the UI * Fix CI * Fixes, PR Feedback --------- Co-authored-by: Travis Semple <[email protected]> * 20419 - Short name linking status (#1540) - Handle basic unlinking for short name still in pending - return whether associated account link has pending payments * Merge branch 'main' of https://github.com/bcgov/sbc-pay into feature-queue-python-upgrade * Fix linting issues. Make units tests pass for pay-api. * Fix lint and test for ftp-poller * pay-queue lint, ci passing * update sbc-common-components, also remove because pay-api already includes it. * remove requirements.txt for pay-admin * poetry update pay-api * Fix lint plus unit tests for payment-jobs * 21464 - Resync EFT branch with main (#1547) * Merge branch 'main' of https://github.com/bcgov/sbc-pay into feature-queue-python-upgrade * Fix linting issues. Make units tests pass for pay-api. * Fix lint and test for ftp-poller * pay-queue lint, ci passing * update sbc-common-components, also remove because pay-api already includes it. * remove requirements.txt for pay-admin * poetry update pay-api * Fix lint plus unit tests for payment-jobs * Revert "21464 - Resync EFT branch with main (#1547)" (#1550) This reverts commit 23fed0f. * Lint fixes * Small lint fixes * Remove duplicate literal * fix hint * remove redundant return * Codecov fixes * more lint fixes * Fix CD * Put back pay-api cd * update poetry, will fix error building * update psycopg2-binary * fix CD * remove requirements.txt * use latest poetry * remove more requirements * Put back gunicorn config, comment for GCP. --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Jia Xu <[email protected]> Co-authored-by: Rodrigo Barraza <[email protected]> Co-authored-by: pwei1018 <[email protected]> Co-authored-by: Patrick Wei <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Odysseus Chiu <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #:
bcgov/entity#19910
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the sbc-pay license (Apache 2.0).