Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Work to merge Zeljko Obrenovic's awesome work #407

Closed
wants to merge 20 commits into from

Conversation

northdpole
Copy link
Collaborator

@northdpole northdpole commented Sep 30, 2023

original work here https://zeljkoobrenovic.github.io/opencre-explorer/

  • navlink
  • react rendering of explorer page
  • make visualisations react
  • [optional if entire pr is less than 3 days] add index db integration and migrate all our "fetch and axios." requests to core requests module and get it to check local db first

northdpole and others added 18 commits September 15, 2023 17:04
#380)

* increase threshold accuracy so we get better matches now that there is a fallback to generic llm

* make threshold configurable via env

* lint
* increase threshold accuracy so we get better matches now that there is a fallback to generic llm

* make threshold configurable via env

* lint

* fix frontend bug that was causing crashes

* improve message
* wip, make returned markdown both pretty and safe while also providing decent syntax highlight

* minor fix

* fix bug and cleanup
)

* fix smartlink functionality in case we don't know about a standard

* lint and tests fix

* lint2
* fix smartlink functionality in case we don't know about a standard

* lint and tests fix

* lint2

* add e2e
* migrate to axios and fix error messages

* Update application/frontend/src/pages/Deeplink/Deeplink.tsx

Co-authored-by: John Harvey <[email protected]>
Signed-off-by: Spyros <[email protected]>

---------

Signed-off-by: Spyros <[email protected]>
Co-authored-by: John Harvey <[email protected]>
* trigger first staging build

* Inital Hack of NEO4j DB creation

* Added: Neo4j docker run

* Added NEO_DB Class

* Create Inital Path API response

* Build basic UI for testing

* Mock score and WIP UI

* implement scoring and basic tests

* scoring passing tests

* Update the UI

* Updated: Dropdowns are now dynamic

* Localise neo4j

* Added Navigation method

* Add share and nav links

* readme improvement

* Hide table on new search

* Optermise query to remove relates to

* Get duel running method working and show empty values

* Refactor grouping & scoring code locations

* Add colour to strength raiting

* Add gap analysis tests

* Short drop down list

* Styling improvements and legends

* rm scikit version pin

* loosen requirement in sqlalchemy

* more deps changes

* fix deps

* Disable auto db pop (#372)

* Add gap analysis tests

* Short drop down list

* Disable CRE_Graph auto population on startup

* revert conflict mess with tests

* Migrate Neo4j population to seperate function

* Add CLI population command for DB

* linting fixes

* fix pillow

* Use python 3.11.4 for CI

* Parse cre_dep object from neo4j Node

* Standardise Gap Analysis API Object

* Fix bad parse field

* Use Standard display format and create GA type

* GA: Show all strong links by default (or min of 3)

* Better route display

* GA add support for tools

* Fix existing tests

* Add parsing tests

* make parse_node static

* Use Enum types

* Better route display

* linting

* Minimal GA styling

* Fix: GA share link spaces

* linting

* Added: Direct link grade and removed external link icon

* Fix: linked via standard was possible

* Added: Per link score penalty

* Fix: GA caching not taking queries (#412)

* Make Direct Dark Green (#411)

* Make Direct Dark Green

* Update GapAnalysis.tsx

Signed-off-by: John Harvey <[email protected]>

---------

Signed-off-by: John Harvey <[email protected]>

* GA Reduce related punishment (#416)

* Rename gap to map (#419)

* GA make strong status <=2

* Rename URL and UI from Gap to Map analysis

* GA fix bad links (#417)

Fix bad links in GA (use standard function)

* Speed up CICD (#418)

* Rename share link (#415)

* Neo4j ORM (#408)

* Make Direct Dark Green

* Update GapAnalysis.tsx

Signed-off-by: John Harvey <[email protected]>

* Trial implementation of Neo4J ORM

* Populate Neo4j DB using model

* More progress in conversion

* get ORM working

* Fix tests

* Remove old NeoDB driver connection

* Fix: id issue

* Fix: Tags strings splitting

* linting fixes

* Remove  NeomodelPath reference

* Revert rebase bug

---------

Signed-off-by: John Harvey <[email protected]>

* Remove BOLT from neo4j Setup requirement (#428)

* pin neomodel to a decent version

* relax six requirements

* Sort Base standard alphabetically (#427)

* Use allShortestPaths over shortestPath (#431)

* Added Cross-cutting concerns to GA blacklist (#430)

* Added Cross-cutting concerns to GA blacklist

* GA page linting

* Remove double brackets

* Fix GA loading disappearing

* rename variable

* Revert "Use allShortestPaths over shortestPath (#431)"

This reverts commit e33d13d.

* Precompute map analysis (#433)

* add time metrics

* progress

* Revert "Revert "Use allShortestPaths over shortestPath (#431)""

This reverts commit 167828c.

* map analysis through redis

* new reqs

* add worker actually

* abort if background job fails

* lint

* minor cleanup

* move hash to utils and make db method write to redis optionally

* make ga job results shared among clients

* lint

* Split ga response (#434)

* add time metrics

* map analysis through redis

* move hash to utils and make db method write to redis optionally

* lint

* Added: Front end support for weaker links calls

* Added: DB seperation for smaller inital response and cache extras

* Added: DB side of part split response logic

* Refactor and tests

---------

Co-authored-by: Spyros <[email protected]>

* skip returning ids for failed jobs

* add optional tls to redis, heroku requirement

* add optional tls to redis, heroku requirement

* add optional tls to redis, heroku requirement

* add optional tls to redis, heroku requirement

* add optional tls to redis, heroku requirement

* add optional tls to redis, heroku requirement

* use db to cache gap analysis results instead of redis (#435)

* use db to cache gap analysis results instead of redis

* lint

* typo

* fix 1 test, fix bugs, make cache key into its own function

* lint

* fix-mock

* migration

* Fix: bad ref

* fix tests

* lint

* minor changes

* lint

---------

Co-authored-by: john681611 <[email protected]>

* Get e2e tests reporting correctly (#395)

* try pip cacheing

* try yarn cache

* try background service method

* e2e test fixes

* tmp increase timeouts for e2e tests due to new neo4j experiments

* fix e2e tests

* make e2e workflow match makefile

* Revert actions simplification

* increase timing

* balance timings

---------

Signed-off-by: Spyros <[email protected]>
Co-authored-by: Spyros <[email protected]>
Co-authored-by: Spyros <[email protected]>

* minor fixes

* fix-lint

* fix caching

* fix e2e?

---------

Signed-off-by: John Harvey <[email protected]>
Signed-off-by: Spyros <[email protected]>
Co-authored-by: john681611 <[email protected]>
Co-authored-by: John Harvey <[email protected]>
* trigger first staging build

* Inital Hack of NEO4j DB creation

* Added: Neo4j docker run

* Added NEO_DB Class

* Create Inital Path API response

* Build basic UI for testing

* Mock score and WIP UI

* implement scoring and basic tests

* scoring passing tests

* Update the UI

* Updated: Dropdowns are now dynamic

* Localise neo4j

* Added Navigation method

* Add share and nav links

* readme improvement

* Hide table on new search

* Optermise query to remove relates to

* Get duel running method working and show empty values

* Refactor grouping & scoring code locations

* Add colour to strength raiting

* Add gap analysis tests

* Short drop down list

* Styling improvements and legends

* rm scikit version pin

* loosen requirement in sqlalchemy

* more deps changes

* fix deps

* Disable auto db pop (#372)

* Add gap analysis tests

* Short drop down list

* Disable CRE_Graph auto population on startup

* revert conflict mess with tests

* Migrate Neo4j population to seperate function

* Add CLI population command for DB

* linting fixes

* fix pillow

* Use python 3.11.4 for CI

* Parse cre_dep object from neo4j Node

* Standardise Gap Analysis API Object

* Fix bad parse field

* Use Standard display format and create GA type

* GA: Show all strong links by default (or min of 3)

* Better route display

* GA add support for tools

* Fix existing tests

* Add parsing tests

* make parse_node static

* Use Enum types

* Better route display

* linting

* Minimal GA styling

* Fix: GA share link spaces

* linting

* Added: Direct link grade and removed external link icon

* Fix: linked via standard was possible

* Added: Per link score penalty

* Fix: GA caching not taking queries (#412)

* Make Direct Dark Green (#411)

* Make Direct Dark Green

* Update GapAnalysis.tsx

Signed-off-by: John Harvey <[email protected]>

---------

Signed-off-by: John Harvey <[email protected]>

* GA Reduce related punishment (#416)

* Rename gap to map (#419)

* GA make strong status <=2

* Rename URL and UI from Gap to Map analysis

* GA fix bad links (#417)

Fix bad links in GA (use standard function)

* Speed up CICD (#418)

* Rename share link (#415)

* Neo4j ORM (#408)

* Make Direct Dark Green

* Update GapAnalysis.tsx

Signed-off-by: John Harvey <[email protected]>

* Trial implementation of Neo4J ORM

* Populate Neo4j DB using model

* More progress in conversion

* get ORM working

* Fix tests

* Remove old NeoDB driver connection

* Fix: id issue

* Fix: Tags strings splitting

* linting fixes

* Remove  NeomodelPath reference

* Revert rebase bug

---------

Signed-off-by: John Harvey <[email protected]>

* Remove BOLT from neo4j Setup requirement (#428)

* pin neomodel to a decent version

* relax six requirements

* Sort Base standard alphabetically (#427)

* Use allShortestPaths over shortestPath (#431)

* Added Cross-cutting concerns to GA blacklist (#430)

* Added Cross-cutting concerns to GA blacklist

* GA page linting

* Remove double brackets

* Fix GA loading disappearing

* rename variable

* Revert "Use allShortestPaths over shortestPath (#431)"

This reverts commit e33d13d.

* Precompute map analysis (#433)

* add time metrics

* progress

* Revert "Revert "Use allShortestPaths over shortestPath (#431)""

This reverts commit 167828c.

* map analysis through redis

* new reqs

* add worker actually

* abort if background job fails

* lint

* minor cleanup

* move hash to utils and make db method write to redis optionally

* make ga job results shared among clients

* lint

* Split ga response (#434)

* add time metrics

* map analysis through redis

* move hash to utils and make db method write to redis optionally

* lint

* Added: Front end support for weaker links calls

* Added: DB seperation for smaller inital response and cache extras

* Added: DB side of part split response logic

* Refactor and tests

---------

Co-authored-by: Spyros <[email protected]>

* skip returning ids for failed jobs

* add optional tls to redis, heroku requirement

* add optional tls to redis, heroku requirement

* add optional tls to redis, heroku requirement

* add optional tls to redis, heroku requirement

* add optional tls to redis, heroku requirement

* add optional tls to redis, heroku requirement

* use db to cache gap analysis results instead of redis (#435)

* use db to cache gap analysis results instead of redis

* lint

* typo

* fix 1 test, fix bugs, make cache key into its own function

* lint

* fix-mock

* migration

* Fix: bad ref

* fix tests

* lint

* minor changes

* lint

---------

Co-authored-by: john681611 <[email protected]>

* Get e2e tests reporting correctly (#395)

* try pip cacheing

* try yarn cache

* try background service method

* e2e test fixes

* tmp increase timeouts for e2e tests due to new neo4j experiments

* fix e2e tests

* make e2e workflow match makefile

* Revert actions simplification

* increase timing

* balance timings

---------

Signed-off-by: Spyros <[email protected]>
Co-authored-by: Spyros <[email protected]>
Co-authored-by: Spyros <[email protected]>

* minor fixes

* fix-lint

* fix caching

* fix e2e?

* add tabs

* lint

---------

Signed-off-by: John Harvey <[email protected]>
Signed-off-by: Spyros <[email protected]>
Co-authored-by: john681611 <[email protected]>
Co-authored-by: John Harvey <[email protected]>
* add dbg logging

* nits and minor fixes

* lint
@john681611 john681611 force-pushed the integrate-zeljkos-magic branch from edd0a47 to 07b4e62 Compare November 1, 2023 11:18
@john681611 john681611 changed the base branch from main to staging November 1, 2023 11:19
@john681611 john681611 force-pushed the integrate-zeljkos-magic branch from c20e137 to 512e65a Compare November 5, 2023 14:25
@northdpole northdpole force-pushed the integrate-zeljkos-magic branch from b7661f2 to bfd169e Compare November 5, 2023 16:52
@northdpole
Copy link
Collaborator Author

continued here #441

@northdpole northdpole closed this Nov 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants