Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance regression: Preparing IntegrityMetaComponents after BOM processing takes too long #1306

Open
2 tasks done
Tracked by #860
nscuro opened this issue Jun 6, 2024 · 2 comments · May be fixed by DependencyTrack/hyades-apiserver#846
Open
2 tasks done
Tracked by #860

Performance regression: Preparing IntegrityMetaComponents after BOM processing takes too long #1306

nscuro opened this issue Jun 6, 2024 · 2 comments · May be fixed by DependencyTrack/hyades-apiserver#846
Assignees
@nscuro
Labels
component/api-server defect Something isn't working p2 Non-critical bugs, and features that help organizations to identify and reduce risk performance size/L High effort
Milestone
0.7.0

Comments

@nscuro
Copy link
Member

nscuro commented Jun 6, 2024 •
edited
Loading

Current Behavior

While porting #3357 I noticed that the section that is "preparing" IntegrityMetaComponents is executing a transaction for every single component of the processed BOM:

https://github.com/DependencyTrack/hyades-apiserver/blob/0dbf1d376c9e1a984e7c0bc735db1504c6de5aed/src/main/java/org/dependencytrack/tasks/BomUploadProcessingTask.java#L410-L425

Profiling the informWithBloatedBomTest confirmed my suspicion that this ends up being super expensive:

Untitled

Querying and creating the IntegrityMetaComponents takes more CPU time than the actual BOM processing itself. And this is with API server and database residing on the same machine, it will be even more impactfull when that is not the case.

Steps to Reproduce

  1. Run informWithBloatedBomTest with profiling

Expected Behavior

Preparing IntegrityMetaComponents should be a quick operation. It should not create a transaction for every component in the BOM.

It's likely that the entire operation can be achieved with a single batch insert using JDBI.

Hyades Version

hyades-apiserver 5.5.0-SNAPSHOT

Repository Type

Hyades apiserver

Browser

N/A

Checklist
@nscuro nscuro added defect Something isn't working p2 Non-critical bugs, and features that help organizations to identify and reduce risk size/S Small effort component/api-server performance labels Jun 6, 2024
@nscuro
Copy link
Member Author

nscuro commented Jun 10, 2024

Even more clear with DependencyTrack/hyades-apiserver#705 in place:

image

@nscuro nscuro added this to the 0.6.0 milestone Jul 19, 2024
@nscuro nscuro self-assigned this Aug 2, 2024
@nscuro
Copy link
Member Author

nscuro commented Aug 2, 2024

Batch inserts into INTEGRITY_META_COMPONENT won't work well because potentially many threads will do it in parallel, so lots of contention. Did some testing locally and am running into deadlocks with just a few consecutive BOM uploads.

Potentially need an entirely different approach to populating that table.

@nscuro nscuro added size/L High effort and removed size/S Small effort labels Aug 9, 2024
This was referenced Aug 11, 2024
Draft
Draft
@nscuro nscuro modified the milestones: 0.6.0, 0.7.0 Aug 22, 2024
@nscuro nscuro mentioned this issue Oct 2, 2024
Open
34 tasks
@nscuro nscuro added this to Hyades Oct 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nscuro nscuro

Labels
component/api-server defect Something isn't working p2 Non-critical bugs, and features that help organizations to identify and reduce risk performance size/L High effort
Projects
Hyades
Status: In Progress
Milestone
0.7.0
Development

Successfully merging a pull request may close this issue.

Improve efficiency of repository and integrity meta analysis DependencyTrack/hyades-apiserver
1 participant
@nscuro