Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement file-level parallelization #100

Merged
merged 1 commit into from
Oct 26, 2023
Merged

Conversation

drdavella
Copy link
Member

@drdavella drdavella commented Oct 25, 2023

Overview

Implement (optional) per-codemod file-level parallelization

Description

  • This PR enables files to be processed in parallel for each codemod
  • There is no parallelism between codemods since we need to preserve strict ordering
  • The default behavior for now is to use a single thread, so there is no difference from the previous behavior
  • All per-file codemod state is now managed by FileContext and then aggregated at the end of each codemod pass
  • This will enable us to perform testing in more resource constrained environments where more threads might be useful. In particular, we suspect that environments that are highly limited by IO bandwidth will benefit from parallelization

@codecov
Copy link

codecov bot commented Oct 25, 2023

Codecov Report

Merging #100 (15716b7) into main (73f5b4a) will decrease coverage by 0.10%.
The diff coverage is 100.00%.

❗ Current head 15716b7 differs from pull request most recent head 43b88ab. Consider uploading reports for the commit 43b88ab to get more accurate results

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #100      +/-   ##
==========================================
- Coverage   95.63%   95.54%   -0.10%     
==========================================
  Files          60       60              
  Lines        2451     2446       -5     
==========================================
- Hits         2344     2337       -7     
- Misses        107      109       +2     
Files Coverage Δ
src/codemodder/cli.py 100.00% <100.00%> (ø)
src/codemodder/codemodder.py 98.14% <100.00%> (+0.08%) ⬆️
src/codemodder/codemods/api/__init__.py 95.31% <100.00%> (-0.08%) ⬇️
src/codemodder/codemods/base_codemod.py 100.00% <100.00%> (ø)
src/codemodder/context.py 97.53% <100.00%> (+0.23%) ⬆️
src/codemodder/file_context.py 100.00% <100.00%> (ø)
...codemodder/project_analysis/python_repo_manager.py 100.00% <100.00%> (ø)
src/core_codemods/order_imports.py 92.85% <ø> (ø)
src/core_codemods/sql_parameterization.py 91.32% <100.00%> (-0.04%) ⬇️
src/core_codemods/upgrade_sslcontext_tls.py 100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

@drdavella drdavella marked this pull request as ready for review October 26, 2023 13:23
Copy link
Contributor

@clavedeluna clavedeluna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How much of a PITA would it be to add an integration test with --max-workers used?

src/codemodder/cli.py Outdated Show resolved Hide resolved
src/codemodder/codemodder.py Outdated Show resolved Hide resolved
src/codemodder/codemodder.py Show resolved Hide resolved
@drdavella drdavella force-pushed the parallel-file-processing branch from e49878a to 15716b7 Compare October 26, 2023 13:55
@drdavella drdavella requested a review from clavedeluna October 26, 2023 14:14
@drdavella
Copy link
Member Author

@clavedeluna since the default is to use one worker and since this is purely to enable performance testing I'm going to skip the integration test for now. Once we decide we want to use --max-workers > 1 in production environments we can revisit with an integration test.

@drdavella drdavella force-pushed the parallel-file-processing branch from 15716b7 to 43b88ab Compare October 26, 2023 14:37
@drdavella drdavella merged commit b2b91af into main Oct 26, 2023
9 checks passed
@drdavella drdavella deleted the parallel-file-processing branch October 26, 2023 14:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants