Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[backend] add exclusion cache handling (search tree version) (#8941) #9137

Draft
wants to merge 1 commit into
base: release/6.5.0
Choose a base branch
from

Conversation

JeremyCloarec
Copy link
Contributor

@JeremyCloarec JeremyCloarec commented Nov 25, 2024

Proposed changes

  • add cache file containing local exclusion list cache tree
  • add cache build manager, building new cache when an exclusion list is changed and pushing it to redis
  • add search algorithm to check if a value is in current cache tree

Related issues

Checklist

  • I consider the submitted work as finished
  • I tested the code for its functionality
  • I wrote test cases for the relevant uses case (coverage and e2e)
  • I added/update the relevant documentation (either on github or on notion)
  • Where necessary I refactored code to improve the overall quality

Further comments

First testing of cache perf:

  • For 1 million IP entries, cache build takes ~30s, cache check takes <1ms

@JeremyCloarec JeremyCloarec changed the base branch from master to release/6.5.0 November 25, 2024 14:39
@github-actions github-actions bot added the filigran team use to identify PR from the Filigran team label Nov 25, 2024
@JeremyCloarec JeremyCloarec force-pushed the issue/8941-chunk3bis-treetest branch 3 times, most recently from 5683ae6 to 012b75e Compare November 26, 2024 10:50
Copy link

codecov bot commented Nov 26, 2024

Codecov Report

Attention: Patch coverage is 67.92453% with 34 lines in your changes missing coverage. Please review.

Project coverage is 64.97%. Comparing base (0042e77) to head (ccdcad6).

Files with missing lines Patch % Lines
...cti-graphql/src/database/exclusionListCacheTree.ts 31.25% 22 Missing ⚠️
...phql/src/manager/exclusionListCacheBuildManager.ts 0.00% 6 Missing ⚠️
...atform/opencti-graphql/src/utils/exclusionLists.ts 93.65% 4 Missing ⚠️
...cti-platform/opencti-graphql/src/initialization.js 0.00% 1 Missing ⚠️
...-graphql/src/modules/indicator/indicator-domain.ts 75.00% 1 Missing ⚠️
Additional details and impacted files
@@                Coverage Diff                @@
##           release/6.5.0    #9137      +/-   ##
=================================================
+ Coverage          64.93%   64.97%   +0.03%     
=================================================
  Files                626      625       -1     
  Lines              59722    59665      -57     
  Branches            6618     6616       -2     
=================================================
- Hits               38782    38765      -17     
+ Misses             20940    20900      -40     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@JeremyCloarec JeremyCloarec marked this pull request as draft November 26, 2024 15:09
@JeremyCloarec JeremyCloarec force-pushed the issue/8941-chunk3bis-treetest branch 4 times, most recently from b1552ae to f0cd64e Compare November 27, 2024 09:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
filigran team use to identify PR from the Filigran team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant