Skip to content

Commit

Permalink
Optimise data transfer between principle and worker procs
Browse files Browse the repository at this point in the history
SearchResultMinimal is used to transder results between
the worker processes and the main collector. As a result
it must be a small as possible in order to keep transfer
fast and memory footprint low. There were several
unnecessary variables being stored and duplicated in this
object that have now been removed maing the transfer
faster and using less memory. Transfers are also batched
so as to reduce interruption on searches.

Also removes unnecssary use of multiprocessing.Queue for
single thread usecase.
  • Loading branch information
dosaboy committed May 11, 2024
1 parent f2c5882 commit 2e83b4c
Show file tree
Hide file tree
Showing 2 changed files with 276 additions and 136 deletions.
Loading

0 comments on commit 2e83b4c

Please sign in to comment.