-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve fast pandas and flexABLE startegies for better performance #502
Conversation
…ry to fix this -add dedicated loc and at classes -don't create new fastindex -store columns in forecaster when a non existent column is called
-bring several other fixes
-collect market_dispatch and unit_dispatch in a buffer for speed up
-reverse example.py
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I left some remarks.
I would change some things in the output role, so that we do not need to bookkeep two types of buffers.
Makes sense to move the pandas conversion into the writing, quite interesting that this has such a large impact.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #502 +/- ##
==========================================
- Coverage 75.93% 75.43% -0.50%
==========================================
Files 50 50
Lines 6706 6680 -26
==========================================
- Hits 5092 5039 -53
- Misses 1614 1641 +27
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
-move more parts of outputs into buffer for bath processing
-reverse tests to original state
will create a new branch with more focused changes |
Pull Request
Related Issue
Closes #[issue number] (if applicable)
Description
After the latest commit with the FastSeries I have noticed a huge slowßdown for long and large simulations. Here I took the effort to find the reasons and came upon multiple places. At the end the problem is now fixed and the speed up for large simulations compared to current main is 4x. For small simulations for one year (example_01a) the speed up is 2x. Still some further testing is needed. This results in a total speed up of 10x compared to the last stable release fr large simulations and 8x for small simulations.
Changes Proposed
-change how average operation time and down time is calculated
-implement static methods for loc and at methods in fast series
-optimize FastIndex to remove overhead of the idx_from_date method
-do not create FastIndex constantly, instead use the slice
-for demand unit, compute the volume for the specific index instead of the whole index
-optimize write_dispatch and write_market_dispatch in the output role to collect the values in a buffer and process them at once resulting in a 2x speed up
-optimize separate_orders function to improve performance
-revise calculate_reward functions for flexable startegies to remove unrequired parameters and calculations
-remove save_frequency_hours: null from all configs and replace with 720 (one month) to avoid confusion for users
Testing
[Describe the testing you've done, including any specific test cases or scenarios]
Checklist
Please check all applicable items:
doc
folder updates)pyproject.toml
doc/release_notes.rst
of the upcoming release is includedAdditional Notes (if applicable)
[Any additional information, concerns, or areas you want reviewers to focus on]
Screenshots (if applicable)
[Add screenshots to demonstrate visual changes]