Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add PricesCsv #274

Merged
merged 11 commits into from
Jan 31, 2024
Merged

Add PricesCsv #274

merged 11 commits into from
Jan 31, 2024

Conversation

maread99
Copy link
Owner

Adds functionality to get and serve prices from local .csv files.

Aside from new PricesCsv class the main change required was to support fixed right limits (as opposed to 'now').

Updates docs, tutorials and tests to reflect changes.

New prices.csv module:

  • Adds prices.csv module with functionality to locate and parse .csv files, provide parsing warning advices and compile DataFrame that consolidate data at a specific interval for all symbols.

base.PricesBase:

  • adds support for fixed right limit, of note:
    • adds _bis_available_any, _bis_available_any_no_partial_indicies and _bis_any to evaluate bis for which ANY data is available over the requested period. This provides for returning prices when strict False and end is to the right of the right limit.
  • overhauls implementation of _price_at_from_daily to FIX bug where daily data could return inaccurate prices at a specific minute when sessions of underlying symbols overlapped.
  • removes all dependence on either daily or intraday intervals being defined (i.e. can now define one or the other or both).
  • implements changes to make defining BaseInterval, BASE_LIMITS and BASE_LIMITS_RIGHT as class attributes optional if instead subclass fully defines via analogous update methods prior to executing PricesBase constructor .
  • adds verification that subclass defines base intervals and limits.

daterange module:

  • removes hard_strict from daterange._Getter.get_start and changes behaviour to enforce consistency when period defined with start or end and a duration and one extreme is evaluated to lie outside of limits.
  • changes end_accuracy implementation so that when end represents end of a live indice accuracy is now the right side of the 'now' minute.. Before was end of live indice,
  • adds 'limit' parameter to Getter* get_start and get_end methods and 'strict' parameter to get_end.

parsing module:

  • revises parse_start_end and dependents to support a right limit.

data module:

  • adds data.Data.available_any method to query if data is available for any timestamp within a range.

prices.yahoo module:

  • refactors some methods from PricesYahoo to functions on the prices.base module.
  • refactors prices_for_symbols from PricesYahoo to PricesBase

To TEST modules:

  • adds new test_csv test module.
  • adds new test_limits test module to test effect of a right limit on PricesBase.get and PricesBase.price_at.
  • revisions to existing tests to support changes that implement a right limit.
  • tests added to test_base_prices to ensure get directly raising errors when daily / intraday prices requested although daily intraday interval not defined.
  • tests added to test_base to verify errors raised if base intervals or base limits not defined.
  • adds 2 days of daily data to right extreme of some daily .csv resource files to bring the availability of daily data in line with that of intraday data.

@maread99
Correct PricesBase cls doc for typos etc
1fce433
@maread99
Add functions to extract csv files
9ba23d9 
Adds `prices.csv` module with functions to extract csv files
per relevant symbol per interval.

Adds tests to new `test_csv` test module.
@maread99
Add functionality to find and parse csv files.
bf6d0d6 
Adds functionality to csv module to:
- find csv files
- parse csv files
- raise warnings and errors relating to finding and parsing csv
- compile DataFrame consolidating data at a specific interval for all
symbols

Adds WIP `PricesCSV` class to get price data from local csv files.

Refactors following methods from `PricesYahoo`  to functions on the
`base` module:
- fill_reindexed
- fill_reindexed_daily
- adjust_high_low

Also provides:
- tests for all new functionality
- option to dynamically define base intervals within the constructor.
- definition of right limits.

Yet to povide for consideration of right limits beyond the PricesBase
class - WIP, TODO
@maread99
Remove hard_strict option from daterange._Getter
687b537 
Removes `hard_strict` option from `daterange._Getter.get_start` and
clients. Also amends affected tests.

`hard_strict` was  provided to allow for raising an error, even when
strict passed as False, if period defined from a duration and 'start'
value that was before the left limit. However, this treatment was
incoherent with fact that could return partial prices by passing strict
False and defining end within available range and a duration that
resulted in the evaluated start falling before the left limit (i.e. the
same situation evaluated from other end point).

A review of code, tests and documentation found no justification
for forcing the raising of the error when strict False, which in
turn left the hard_strict redundant.
@maread99
Change 'end_accuracy' to now when end is None
ae48422 
Changes to daterange module
- end_accuracy changed when end represents end of a live indice. Before
was end of live indice, now the right side of the 'now' minute.
- updates tests on `test_daterange` to reflect changes.

Changes to `base.PricesBase`
- simplifies `_bis_available` and `bis_available_end` implementations to
include removing 'live indice when end is None' hack which is no longer
required given changes to 'end_accuracy'.
- removes similar hack from `_get_bi_table_intraday`.
@maread99
Implement option for right limit ealier than now
e609900 
To `base.PricesBase`:
- adds `live_prices` method.
- adds `limit_right_daily` property.
- adds `limit_right_intraday` property.
- adds `latest_requestable_minute` property.
- adds `_bis_available_any`, `_bis_available_any_no_partial_indicies`
and `_bis_any` to evaluate bis for which ANY data is available over
the requested period. This provides for returning prices when strict
False and end is to the right of the right limit.
- revises `_get_bi_table_intraday` to accommodate returning data when
end is to the right of the right limit and strict is False.
- renames `_bis_available` as `_bis_available_all`.
- revises `_bis_valid` to raise `EndOutOfBoundsError`.
- revises `_bis_available_all` and  `_bis_available_end` to also
raise `PricesUnavailableIntervalError` when start is right of right
limit or end is left of left limit.
- revises `GetPricesParams` to support right limits and adds
`request_all_availabla_data` property to query if the params are
requesting all available data.
- tidies `get` to make code paths clearer.
- revises `price_at` and dependents to support right limit.
  - overhauls implementation of `_price_at_from_daily` to FIX bug where
daily data could return inaccurate prices at a specific minute when
sessions of underlying symbols overlapped.

To `daterange._Getter`:
- adds support for a right limit.
- adds 'limit' parameter to `get_start`, `get_end` and
`_get_start_end`.
- adds 'strict' parameter to `get_end` to override instance stict.
- enforces consistency of behaviour when evaluated period is
partially out-of-limits when strict is False, only start or end is
defined and the other is evaluated from a duration. Now always
evaluates period independent of limits and returns prices for the
part of the period for which prices are available.

To `parsing` module:
- revises `parse_start_end` and dependents to support a right limit.

To `data` module:
- adds `data.Data.available_any` method to query if data is avaialble
for any timestamp within a range.

To `errors` module:
- revises and reworks messages for `PricesIntradayUnavailableError`.
- adds:
  - `EndTooLateError`
  - `PriceAtUnavailableError`
  - `PriceAtUnavailableLimitError`
  - `PriceAtUnavailableLivePricesError`
  - `EndOutOfBoundsRightError`

To `csv` module:
  - implements `_request_data` and calls from constructor to get all
data upfront, after which further calls to method will fail.

To TEST modules:
  - revisions to existing tests to support changes that implemented a
right limit.
  - new `test_limits` test module to test effect of a right limit on
`PricesBase.get` and `PricesBase.price_at`.
  - adds  `test_parsing.test_start_end_right_limit_now_now` to test
support for right limit when parsing start and end params.
  - adds 2 days of daily data to right extreme of some daily .csv
resource files to bring the availability of daily data in line with
that of intraday data.
@maread99
Support daily only or intraday only intervals
aefe424 
To `prices.base.PricesBase`:
- removes all dependence on either daily or intraday intervals being
defined (i.e. can now define one or the other or both).
- revises `get` implementation to raise new errors
`PricesDailyIntervalError` / `PricesIntradayIntervalError` when
request requires daily / intraday data although daily / intraday
intervals are not defined for the prices subclass.
- `session_prices` and `close_at` now raise new
`MethodUnavailableNoDailyInterval` error if no daily interval
defined.
- `price_at` revised to raise errors in possible edge cases where
daily data required although daily interval not defined.
- adds verification that subclass defines base intervals and limits.

Tests:
- tests added to `test_base_prices` to ensure `get` directly raising
errors when  daily / intraday prices requested although daily /
intraday interval not defined. Fixtures revised to provide Prices
test classes defined with only daily and only intraday intervals.
- tests added to `test_base` to verify errors raised if base
intervals or base limits not defined. PriceMock class deconstructed
to provide for defining new `PricesMockIntradayOnly` and
`PricesMocDailyOnly` classes.
- tests added to `test_base` to verify error raised if
`session_prices` or `close_at` called when no daily interval defined.
@maread99
Add and revise tests to cover fixed right limit
d9703e1 
Principally adds and revises tests to cover new properties and
methods that provide for implementing a fixed right limit.

To `test_base`:
- Adds mock Prices fixtures for FixedLimits, including classes
defined with IntradayOnly and DailyOnly intervals.
- revises `test_limits` to include tests for `limit_right_intraday`
and `limit_right_daily` properties.
- adds `test_limits_fixed` to test right limit properties and setup.
- adds `test_live_prices`.
- adds `test_latest_requestable_minute`.
- adds `test__minute_to_earliest_previous_trading_minute`.
- revises `test__bis_available` to include `bis_available_any`.
- adds `test__bis_available_fixed_right` to test `bis_available`
methods when right limit defined.
- revises `test_get_prices_params_cls` to test
`gpp.request_all_available_data`.

To `test_daterange`, for both `TestGetterDaily` and
`TestGetterIntraday` test classes:
- revises `test_constructor_properties` to cover new `limit_right`
and `end_limit` properties.
- revises `test_get_start` to verify effect of new `limit` property
and defining a fixed right limit.
- revises `test_get_end` to verify effect of new `limit` and `stict`
properties and defining a fixed right limit.
- For `TestIntradayGetter`, specifically revises
`test_get_start_too_late`, `test_get_end_ool` and
`test_end_now_and_get_end_none`.

To `test_data` revises `test_pre_requests` to `verify_available_any`.

Also:

To `prices.base` module:
- Fixes `set_base_limits_right` to consider BASE_LIMITS_RIGHT attr.
- Revises `_update_base_limits` to provide for not setting
BASE_LIMITS attr and relying on update method at runtime.
- Revises `PricesBase` class doc to cover changes to optionally
provide BaseInterval, BASE_LIMITS and BASE_LIMITS_RIGHT dynamically
via update methods.
- Further changes to provide for defining only intraday or daily
intervals.
- Refactors `prices_for_symbols` from `PricesYahoo` to base. Moves
test to `test_base_prices` and revises.

To prices.csv module:
- implements `prices_for_symbols`. Includes test to `test_csv`.
@maread99 maread99 added the enhancement New feature or request label Jan 30, 2024
@maread99 maread99 force-pushed the dev-csv branch 6 times, most recently from 9189ec1 to c49d663 Compare January 31, 2024 00:06
@maread99
Revise tutorial and README for PricesCsv
052ba9a 
Revises `prices.ipynb` tutorial to add `PricesCsv` and adds example
.csv files to `docs/tutorials/resources`.

Updates README to cover getting price data from locally stored .csv
files.

Also adds `PricesCsv` to package `__init__` so that directly
accessible from `market_prices`.
@maread99 maread99 changed the title Add functionality to retrieve and serve prices from local .csv files Add PricesCsv Jan 31, 2024
@maread99
Revise test regex matches
2bb4c7a 
Revises test regex matches for compatibility across py39 through
py311.
@maread99 maread99 force-pushed the dev-csv branch 3 times, most recently from d932193 to 4d20ced Compare January 31, 2024 14:35
@maread99
Revise tests matching csv consolidated errors
c3522e2 
Revises tests mathcing csv conolidated errors to only match when
running on Windows platform (different platforms walk the directory
tree in different manners, with the consequence that ordering of
the errors included in the consoldiated message changes).
@maread99 maread99 merged commit b0820c5 into master Jan 31, 2024
7 checks passed
@maread99 maread99 deleted the dev-csv branch January 31, 2024 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@maread99