use db to cache gap analysis results instead of redis #435
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
northdpole
added a commit
that referenced
this pull request
Oct 28, 2023
* use db to cache gap analysis results instead of redis * lint * typo * fix 1 test, fix bugs, make cache key into its own function * lint * fix-mock * migration * Fix: bad ref * fix tests * lint * minor changes * lint --------- Co-authored-by: john681611 <[email protected]>
northdpole
added a commit
that referenced
this pull request
Oct 28, 2023
* use db to cache gap analysis results instead of redis * lint * typo * fix 1 test, fix bugs, make cache key into its own function * lint * fix-mock * migration * Fix: bad ref * fix tests * lint * minor changes * lint --------- Co-authored-by: john681611 <[email protected]>
northdpole
added a commit
that referenced
this pull request
Oct 29, 2023
* trigger first staging build * Inital Hack of NEO4j DB creation * Added: Neo4j docker run * Added NEO_DB Class * Create Inital Path API response * Build basic UI for testing * Mock score and WIP UI * implement scoring and basic tests * scoring passing tests * Update the UI * Updated: Dropdowns are now dynamic * Localise neo4j * Added Navigation method * Add share and nav links * readme improvement * Hide table on new search * Optermise query to remove relates to * Get duel running method working and show empty values * Refactor grouping & scoring code locations * Add colour to strength raiting * Add gap analysis tests * Short drop down list * Styling improvements and legends * rm scikit version pin * loosen requirement in sqlalchemy * more deps changes * fix deps * Disable auto db pop (#372) * Add gap analysis tests * Short drop down list * Disable CRE_Graph auto population on startup * revert conflict mess with tests * Migrate Neo4j population to seperate function * Add CLI population command for DB * linting fixes * fix pillow * Use python 3.11.4 for CI * Parse cre_dep object from neo4j Node * Standardise Gap Analysis API Object * Fix bad parse field * Use Standard display format and create GA type * GA: Show all strong links by default (or min of 3) * Better route display * GA add support for tools * Fix existing tests * Add parsing tests * make parse_node static * Use Enum types * Better route display * linting * Minimal GA styling * Fix: GA share link spaces * linting * Added: Direct link grade and removed external link icon * Fix: linked via standard was possible * Added: Per link score penalty * Fix: GA caching not taking queries (#412) * Make Direct Dark Green (#411) * Make Direct Dark Green * Update GapAnalysis.tsx Signed-off-by: John Harvey <[email protected]> --------- Signed-off-by: John Harvey <[email protected]> * GA Reduce related punishment (#416) * Rename gap to map (#419) * GA make strong status <=2 * Rename URL and UI from Gap to Map analysis * GA fix bad links (#417) Fix bad links in GA (use standard function) * Speed up CICD (#418) * Rename share link (#415) * Neo4j ORM (#408) * Make Direct Dark Green * Update GapAnalysis.tsx Signed-off-by: John Harvey <[email protected]> * Trial implementation of Neo4J ORM * Populate Neo4j DB using model * More progress in conversion * get ORM working * Fix tests * Remove old NeoDB driver connection * Fix: id issue * Fix: Tags strings splitting * linting fixes * Remove NeomodelPath reference * Revert rebase bug --------- Signed-off-by: John Harvey <[email protected]> * Remove BOLT from neo4j Setup requirement (#428) * pin neomodel to a decent version * relax six requirements * Sort Base standard alphabetically (#427) * Use allShortestPaths over shortestPath (#431) * Added Cross-cutting concerns to GA blacklist (#430) * Added Cross-cutting concerns to GA blacklist * GA page linting * Remove double brackets * Fix GA loading disappearing * rename variable * Revert "Use allShortestPaths over shortestPath (#431)" This reverts commit e33d13d. * Precompute map analysis (#433) * add time metrics * progress * Revert "Revert "Use allShortestPaths over shortestPath (#431)"" This reverts commit 167828c. * map analysis through redis * new reqs * add worker actually * abort if background job fails * lint * minor cleanup * move hash to utils and make db method write to redis optionally * make ga job results shared among clients * lint * Split ga response (#434) * add time metrics * map analysis through redis * move hash to utils and make db method write to redis optionally * lint * Added: Front end support for weaker links calls * Added: DB seperation for smaller inital response and cache extras * Added: DB side of part split response logic * Refactor and tests --------- Co-authored-by: Spyros <[email protected]> * skip returning ids for failed jobs * add optional tls to redis, heroku requirement * add optional tls to redis, heroku requirement * add optional tls to redis, heroku requirement * add optional tls to redis, heroku requirement * add optional tls to redis, heroku requirement * add optional tls to redis, heroku requirement * use db to cache gap analysis results instead of redis (#435) * use db to cache gap analysis results instead of redis * lint * typo * fix 1 test, fix bugs, make cache key into its own function * lint * fix-mock * migration * Fix: bad ref * fix tests * lint * minor changes * lint --------- Co-authored-by: john681611 <[email protected]> * Get e2e tests reporting correctly (#395) * try pip cacheing * try yarn cache * try background service method * e2e test fixes * tmp increase timeouts for e2e tests due to new neo4j experiments * fix e2e tests * make e2e workflow match makefile * Revert actions simplification * increase timing * balance timings --------- Signed-off-by: Spyros <[email protected]> Co-authored-by: Spyros <[email protected]> Co-authored-by: Spyros <[email protected]> * minor fixes * fix-lint * fix caching * fix e2e? --------- Signed-off-by: John Harvey <[email protected]> Signed-off-by: Spyros <[email protected]> Co-authored-by: john681611 <[email protected]> Co-authored-by: John Harvey <[email protected]>
northdpole
added a commit
that referenced
this pull request
Oct 29, 2023
* trigger first staging build * Inital Hack of NEO4j DB creation * Added: Neo4j docker run * Added NEO_DB Class * Create Inital Path API response * Build basic UI for testing * Mock score and WIP UI * implement scoring and basic tests * scoring passing tests * Update the UI * Updated: Dropdowns are now dynamic * Localise neo4j * Added Navigation method * Add share and nav links * readme improvement * Hide table on new search * Optermise query to remove relates to * Get duel running method working and show empty values * Refactor grouping & scoring code locations * Add colour to strength raiting * Add gap analysis tests * Short drop down list * Styling improvements and legends * rm scikit version pin * loosen requirement in sqlalchemy * more deps changes * fix deps * Disable auto db pop (#372) * Add gap analysis tests * Short drop down list * Disable CRE_Graph auto population on startup * revert conflict mess with tests * Migrate Neo4j population to seperate function * Add CLI population command for DB * linting fixes * fix pillow * Use python 3.11.4 for CI * Parse cre_dep object from neo4j Node * Standardise Gap Analysis API Object * Fix bad parse field * Use Standard display format and create GA type * GA: Show all strong links by default (or min of 3) * Better route display * GA add support for tools * Fix existing tests * Add parsing tests * make parse_node static * Use Enum types * Better route display * linting * Minimal GA styling * Fix: GA share link spaces * linting * Added: Direct link grade and removed external link icon * Fix: linked via standard was possible * Added: Per link score penalty * Fix: GA caching not taking queries (#412) * Make Direct Dark Green (#411) * Make Direct Dark Green * Update GapAnalysis.tsx Signed-off-by: John Harvey <[email protected]> --------- Signed-off-by: John Harvey <[email protected]> * GA Reduce related punishment (#416) * Rename gap to map (#419) * GA make strong status <=2 * Rename URL and UI from Gap to Map analysis * GA fix bad links (#417) Fix bad links in GA (use standard function) * Speed up CICD (#418) * Rename share link (#415) * Neo4j ORM (#408) * Make Direct Dark Green * Update GapAnalysis.tsx Signed-off-by: John Harvey <[email protected]> * Trial implementation of Neo4J ORM * Populate Neo4j DB using model * More progress in conversion * get ORM working * Fix tests * Remove old NeoDB driver connection * Fix: id issue * Fix: Tags strings splitting * linting fixes * Remove NeomodelPath reference * Revert rebase bug --------- Signed-off-by: John Harvey <[email protected]> * Remove BOLT from neo4j Setup requirement (#428) * pin neomodel to a decent version * relax six requirements * Sort Base standard alphabetically (#427) * Use allShortestPaths over shortestPath (#431) * Added Cross-cutting concerns to GA blacklist (#430) * Added Cross-cutting concerns to GA blacklist * GA page linting * Remove double brackets * Fix GA loading disappearing * rename variable * Revert "Use allShortestPaths over shortestPath (#431)" This reverts commit e33d13d. * Precompute map analysis (#433) * add time metrics * progress * Revert "Revert "Use allShortestPaths over shortestPath (#431)"" This reverts commit 167828c. * map analysis through redis * new reqs * add worker actually * abort if background job fails * lint * minor cleanup * move hash to utils and make db method write to redis optionally * make ga job results shared among clients * lint * Split ga response (#434) * add time metrics * map analysis through redis * move hash to utils and make db method write to redis optionally * lint * Added: Front end support for weaker links calls * Added: DB seperation for smaller inital response and cache extras * Added: DB side of part split response logic * Refactor and tests --------- Co-authored-by: Spyros <[email protected]> * skip returning ids for failed jobs * add optional tls to redis, heroku requirement * add optional tls to redis, heroku requirement * add optional tls to redis, heroku requirement * add optional tls to redis, heroku requirement * add optional tls to redis, heroku requirement * add optional tls to redis, heroku requirement * use db to cache gap analysis results instead of redis (#435) * use db to cache gap analysis results instead of redis * lint * typo * fix 1 test, fix bugs, make cache key into its own function * lint * fix-mock * migration * Fix: bad ref * fix tests * lint * minor changes * lint --------- Co-authored-by: john681611 <[email protected]> * Get e2e tests reporting correctly (#395) * try pip cacheing * try yarn cache * try background service method * e2e test fixes * tmp increase timeouts for e2e tests due to new neo4j experiments * fix e2e tests * make e2e workflow match makefile * Revert actions simplification * increase timing * balance timings --------- Signed-off-by: Spyros <[email protected]> Co-authored-by: Spyros <[email protected]> Co-authored-by: Spyros <[email protected]> * minor fixes * fix-lint * fix caching * fix e2e? * add tabs * lint --------- Signed-off-by: John Harvey <[email protected]> Signed-off-by: Spyros <[email protected]> Co-authored-by: john681611 <[email protected]> Co-authored-by: John Harvey <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.