Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve fast pandas and flexABLE startegies for better performance #502

Closed
wants to merge 13 commits into from

Conversation

nick-harder
Copy link
Member

@nick-harder nick-harder commented Nov 27, 2024

Pull Request

Related Issue

Closes #[issue number] (if applicable)

Description

After the latest commit with the FastSeries I have noticed a huge slowßdown for long and large simulations. Here I took the effort to find the reasons and came upon multiple places. At the end the problem is now fixed and the speed up for large simulations compared to current main is 4x. For small simulations for one year (example_01a) the speed up is 2x. Still some further testing is needed. This results in a total speed up of 10x compared to the last stable release fr large simulations and 8x for small simulations.

Changes Proposed

-change how average operation time and down time is calculated
-implement static methods for loc and at methods in fast series
-optimize FastIndex to remove overhead of the idx_from_date method
-do not create FastIndex constantly, instead use the slice
-for demand unit, compute the volume for the specific index instead of the whole index
-optimize write_dispatch and write_market_dispatch in the output role to collect the values in a buffer and process them at once resulting in a 2x speed up
-optimize separate_orders function to improve performance
-revise calculate_reward functions for flexable startegies to remove unrequired parameters and calculations
-remove save_frequency_hours: null from all configs and replace with 720 (one month) to avoid confusion for users

Testing

[Describe the testing you've done, including any specific test cases or scenarios]

Checklist

Please check all applicable items:

  • Code changes are sufficiently documented (docstrings, inline comments, doc folder updates)
  • New unit tests added for new features or bug fixes
  • Existing tests pass with the changes
  • Reinforcement learning examples are operational (for DRL-related changes)
  • Code tested with both local and Docker databases
  • Code follows project style guidelines and best practices
  • Changes are backwards compatible, or deprecation notices added
  • New dependencies added to pyproject.toml
  • A note for the release notes doc/release_notes.rst of the upcoming release is included
  • Consent to release this PR's code under the GNU Affero General Public License v3.0

Additional Notes (if applicable)

[Any additional information, concerns, or areas you want reviewers to focus on]

Screenshots (if applicable)

[Add screenshots to demonstrate visual changes]

…ry to fix this

-add dedicated loc and at classes
-don't create new fastindex
-store columns in forecaster when a non existent column is called
-bring several other fixes
-collect market_dispatch and unit_dispatch in a buffer for speed up
@nick-harder nick-harder marked this pull request as draft November 27, 2024 16:22
Copy link
Member

@maurerle maurerle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some remarks.

I would change some things in the output role, so that we do not need to bookkeep two types of buffers.
Makes sense to move the pandas conversion into the writing, quite interesting that this has such a large impact.

assume/strategies/flexable.py Outdated Show resolved Hide resolved
assume/common/base.py Outdated Show resolved Hide resolved
assume/common/units_operator.py Outdated Show resolved Hide resolved
assume/strategies/flexable.py Outdated Show resolved Hide resolved
@nick-harder nick-harder marked this pull request as ready for review November 28, 2024 12:31
Copy link

codecov bot commented Nov 28, 2024

Codecov Report

Attention: Patch coverage is 50.00000% with 84 lines in your changes missing coverage. Please review.

Project coverage is 75.43%. Comparing base (fd8963d) to head (027c577).

Files with missing lines Patch % Lines
assume/strategies/learning_advanced_orders.py 2.22% 44 Missing ⚠️
assume/strategies/flexable.py 17.94% 32 Missing ⚠️
assume/common/base.py 80.00% 4 Missing ⚠️
assume/common/fast_pandas.py 92.85% 2 Missing ⚠️
assume/common/forecasts.py 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #502      +/-   ##
==========================================
- Coverage   75.93%   75.43%   -0.50%     
==========================================
  Files          50       50              
  Lines        6706     6680      -26     
==========================================
- Hits         5092     5039      -53     
- Misses       1614     1641      +27     
Flag Coverage Δ
pytest 75.43% <50.00%> (-0.50%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

-move more parts of outputs into buffer for bath processing
@nick-harder nick-harder marked this pull request as draft November 28, 2024 16:11
tests/test_utils.py Outdated Show resolved Hide resolved
@nick-harder nick-harder changed the title Improve fast pandas, base_classes and output role for better performance Improve fast pandas and flexABLE startegies for better performance Nov 29, 2024
@nick-harder nick-harder marked this pull request as ready for review November 29, 2024 11:19
@nick-harder nick-harder marked this pull request as draft November 29, 2024 11:27
@nick-harder
Copy link
Member Author

will create a new branch with more focused changes

@nick-harder nick-harder deleted the improve-fast-pandas branch November 29, 2024 11:40
@nick-harder nick-harder restored the improve-fast-pandas branch November 29, 2024 11:42
@nick-harder nick-harder deleted the improve-fast-pandas branch November 29, 2024 11:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants