Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PR - Add Delta Lake Backend for Cryptofeed #1054

Open
wants to merge 90 commits into
base: master
Choose a base branch
from
Open

PR - Add Delta Lake Backend for Cryptofeed #1054

wants to merge 90 commits into from
+637 −5

Commits on Aug 31, 2024

  1. feat(deltalake): Implement Delta Lake backend and add dependencies 

    - Add DeltaLakeCallback class with support for various data types
- Implement partitioning, Z-ordering, and time travel features
- Add schema documentation for each data type
- Include Delta Lake dependencies in setup.py
- Create demo file for Delta Lake usage with S3 configuration
- Update extras_require in setup.py to include deltalake option
    @tommy-ca
    tommy-ca committed Aug 31, 2024
    Configuration menu
    Copy the full SHA
    07cca17 View commit details
    Browse the repository at this point in the history

  2. feat(deltalake): optimize Delta Lake implementation

    @tommy-ca
    tommy-ca committed Aug 31, 2024
    Configuration menu
    Copy the full SHA
    c003c69 View commit details
    Browse the repository at this point in the history

  3. fix(deltalake): fix book table name

    @tommy-ca
    tommy-ca committed Aug 31, 2024
    Configuration menu
    Copy the full SHA
    f8256dc View commit details
    Browse the repository at this point in the history

  4. fix(deltalake): Fix book name

    @tommy-ca
    tommy-ca committed Aug 31, 2024
    Configuration menu
    Copy the full SHA
    cf2c926 View commit details
    Browse the repository at this point in the history

  5. fix(deltalake): Fix numeric type

    @tommy-ca
    tommy-ca committed Aug 31, 2024
    Configuration menu
    Copy the full SHA
    b02ab52 View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2024

  1. fix: Ensure timestamp columns have nanosecond precision in DeltaLake … 

    …backend
    @tommy-ca
    tommy-ca committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    42eea55 View commit details
    Browse the repository at this point in the history

  2. feat: Refactor timestamp column handling in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    090c4f4 View commit details
    Browse the repository at this point in the history

  3. fix: Handle null values in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    2a60f20 View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2024

  1. feat: Implement DeltaLake backend for Cryptofeed

    @tommy-ca
    tommy-ca committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    2f01815 View commit details
    Browse the repository at this point in the history

  2. fix: Refactor DeltaLakeCallback class

    @tommy-ca
    tommy-ca committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    2081b11 View commit details
    Browse the repository at this point in the history

  3. fix: Add debug logging for DataFrame schema in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    a9ba6b2 View commit details
    Browse the repository at this point in the history

  4. fix: Add DataFrame schema logging when timestamp-related error occurs… 

    … during Delta Lake write
    @tommy-ca
    tommy-ca committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    4ab9fd8 View commit details
    Browse the repository at this point in the history

  5. fix: convert timestamp columns to datetime64[ms]

    @tommy-ca
    tommy-ca committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    797a789 View commit details
    Browse the repository at this point in the history

  6. fix: Ensure all partition columns are present in the DataFrame

    @tommy-ca
    tommy-ca committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    ceb8f76 View commit details
    Browse the repository at this point in the history

  7. fix: convert timestamp column to datetime64[ms] format

    @tommy-ca
    tommy-ca committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    84c436d View commit details
    Browse the repository at this point in the history

  8. feat: Convert timestamp column to date string format in DeltaLakeCall… 

    …back
    @tommy-ca
    tommy-ca committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    84fd533 View commit details
    Browse the repository at this point in the history

  9. refactor: Simplify null value handling in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    5e4c4df View commit details
    Browse the repository at this point in the history

  10. fix: Ensure empty string is a category in categorical columns and han… 

    …dle null values correctly
    @tommy-ca
    tommy-ca committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    5bdd670 View commit details
    Browse the repository at this point in the history

  11. refactor: Refactor DeltaLakeCallback class to improve code readabilit… 

    …y and maintainability
    @tommy-ca
    tommy-ca committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    099c6c4 View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2024

  1. fix: Improve error handling and logging in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    d3a9c71 View commit details
    Browse the repository at this point in the history

  2. fix: Optimize Delta Lake table by filling null values with empty strings

    @tommy-ca
    tommy-ca committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    fd82534 View commit details
    Browse the repository at this point in the history

  3. fix: optimize handling of missing data in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    1b37d26 View commit details
    Browse the repository at this point in the history

  4. feat: Add custom transformations and improve column validation in Del… 

    …taLakeCallback
    @tommy-ca
    tommy-ca committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    81870dc View commit details
    Browse the repository at this point in the history

  5. fix: Add logging configuration to deltalake backend

    @tommy-ca
    tommy-ca committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    880c079 View commit details
    Browse the repository at this point in the history

  6. fix: Initialize DeltaLakeCallback and add logging for writer method a… 

    …nd _write_batch
    @tommy-ca
    tommy-ca committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    e45dee7 View commit details
    Browse the repository at this point in the history

  7. fix: Change logging levels from DEBUG to WARNING in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    fcaa65f View commit details
    Browse the repository at this point in the history

  8. fix: Improve logging and error handling in DeltaLakeCallback writer m… 

    …ethod
    @tommy-ca
    tommy-ca committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    4145588 View commit details
    Browse the repository at this point in the history

  9. fix: Refactor field transformation in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    274ca48 View commit details
    Browse the repository at this point in the history

  10. fix: Reorder columns to prioritize exchange and symbol

    @tommy-ca
    tommy-ca committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    43bbd3a View commit details
    Browse the repository at this point in the history

  11. fix: Ensure datetime columns have millisecond precision in DeltaLakeC… 

    …allback
    @tommy-ca
    tommy-ca committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    e59f57d View commit details
    Browse the repository at this point in the history

  12. feat: Ensure datetime columns are in millisecond precision

    @tommy-ca
    tommy-ca committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    4bacc42 View commit details
    Browse the repository at this point in the history

Commits on Sep 8, 2024

  1. chore: Convert datetime columns to microsecond precision

    @tommy-ca
    tommy-ca committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    d520c02 View commit details
    Browse the repository at this point in the history

  2. fix: Change log levels from warning to debug for non-critical messages

    @tommy-ca
    tommy-ca committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    b11f9a4 View commit details
    Browse the repository at this point in the history

  3. refactor: Simplify datetime column handling in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 8, 2024
    Configuration menu
    Copy the full SHA
    11893db View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2024

  1. feat: Add batch processing and flush interval to DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    313a8f8 View commit details
    Browse the repository at this point in the history

  2. fix: Update copyright year in demo_deltalake.py

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    f5d3f8c View commit details
    Browse the repository at this point in the history

  3. Merge branch 'bmoscon:master' into master

    @tommy-ca
    tommy-ca authored Sep 9, 2024
    Configuration menu
    Copy the full SHA
    959dc7e View commit details
    Browse the repository at this point in the history

  4. feat(deltalake): Implement Delta Lake backend and add dependencies 

    - Add DeltaLakeCallback class with support for various data types
- Implement partitioning, Z-ordering, and time travel features
- Add schema documentation for each data type
- Include Delta Lake dependencies in setup.py
- Create demo file for Delta Lake usage with S3 configuration
- Update extras_require in setup.py to include deltalake option
    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    85ad668 View commit details
    Browse the repository at this point in the history

  5. feat(deltalake): optimize Delta Lake implementation

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    b7a20b9 View commit details
    Browse the repository at this point in the history

  6. fix(deltalake): fix book table name

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    6f81da6 View commit details
    Browse the repository at this point in the history

  7. fix(deltalake): Fix book name

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    d12441b View commit details
    Browse the repository at this point in the history

  8. fix(deltalake): Fix numeric type

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    d6a2ae8 View commit details
    Browse the repository at this point in the history

  9. fix: Ensure timestamp columns have nanosecond precision in DeltaLake … 

    …backend
    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    e257d83 View commit details
    Browse the repository at this point in the history

  10. feat: Refactor timestamp column handling in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    265e761 View commit details
    Browse the repository at this point in the history

  11. fix: Handle null values in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    00d81ec View commit details
    Browse the repository at this point in the history

  12. feat: Implement DeltaLake backend for Cryptofeed

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    d6ed8b9 View commit details
    Browse the repository at this point in the history

  13. fix: Refactor DeltaLakeCallback class

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    16f2825 View commit details
    Browse the repository at this point in the history

  14. fix: Add debug logging for DataFrame schema in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    d51babc View commit details
    Browse the repository at this point in the history

  15. fix: Add DataFrame schema logging when timestamp-related error occurs… 

    … during Delta Lake write
    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    2a7712e View commit details
    Browse the repository at this point in the history

  16. fix: convert timestamp columns to datetime64[ms]

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    f611215 View commit details
    Browse the repository at this point in the history

  17. fix: Ensure all partition columns are present in the DataFrame

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    a16f737 View commit details
    Browse the repository at this point in the history

  18. fix: convert timestamp column to datetime64[ms] format

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    4173cc2 View commit details
    Browse the repository at this point in the history

  19. feat: Convert timestamp column to date string format in DeltaLakeCall… 

    …back
    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    b880bfa View commit details
    Browse the repository at this point in the history

  20. refactor: Simplify null value handling in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    b02e46c View commit details
    Browse the repository at this point in the history

  21. fix: Ensure empty string is a category in categorical columns and han… 

    …dle null values correctly
    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    358c162 View commit details
    Browse the repository at this point in the history

  22. refactor: Refactor DeltaLakeCallback class to improve code readabilit… 

    …y and maintainability
    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    90a1559 View commit details
    Browse the repository at this point in the history

  23. fix: Improve error handling and logging in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    1365432 View commit details
    Browse the repository at this point in the history

  24. fix: Optimize Delta Lake table by filling null values with empty strings

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    ac5e61a View commit details
    Browse the repository at this point in the history

  25. fix: optimize handling of missing data in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    3338d3c View commit details
    Browse the repository at this point in the history

  26. feat: Add custom transformations and improve column validation in Del… 

    …taLakeCallback
    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    7c93b11 View commit details
    Browse the repository at this point in the history

  27. fix: Add logging configuration to deltalake backend

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    bdea1ff View commit details
    Browse the repository at this point in the history

  28. fix: Initialize DeltaLakeCallback and add logging for writer method a… 

    …nd _write_batch
    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    11c7b22 View commit details
    Browse the repository at this point in the history

  29. fix: Change logging levels from DEBUG to WARNING in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    d0ccd92 View commit details
    Browse the repository at this point in the history

  30. fix: Improve logging and error handling in DeltaLakeCallback writer m… 

    …ethod
    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    a8d6310 View commit details
    Browse the repository at this point in the history

  31. fix: Refactor field transformation in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    4946bb8 View commit details
    Browse the repository at this point in the history

  32. fix: Reorder columns to prioritize exchange and symbol

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    b1cad2b View commit details
    Browse the repository at this point in the history

  33. fix: Ensure datetime columns have millisecond precision in DeltaLakeC… 

    …allback
    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    6cfd5f7 View commit details
    Browse the repository at this point in the history

  34. feat: Ensure datetime columns are in millisecond precision

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    d213cf0 View commit details
    Browse the repository at this point in the history

  35. chore: Convert datetime columns to microsecond precision

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    adac33b View commit details
    Browse the repository at this point in the history

  36. fix: Change log levels from warning to debug for non-critical messages

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    4e2aff6 View commit details
    Browse the repository at this point in the history

  37. refactor: Simplify datetime column handling in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    99b3a36 View commit details
    Browse the repository at this point in the history

  38. feat: Add batch processing and flush interval to DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    1b25acd View commit details
    Browse the repository at this point in the history

  39. fix: Update copyright year in demo_deltalake.py

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    d649fea View commit details
    Browse the repository at this point in the history

  40. Merge remote-tracking branch 'my/master'

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    9c9158b View commit details
    Browse the repository at this point in the history

  41. feat: Add common configuration for Delta Lake callbacks

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    bb4cb4c View commit details
    Browse the repository at this point in the history

  42. Merge remote-tracking branch 'my/master'

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    9343c30 View commit details
    Browse the repository at this point in the history

  43. fix: Fix logging message in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    87a77f5 View commit details
    Browse the repository at this point in the history

  44. refactor: Improve logging and error handling in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    98762fd View commit details
    Browse the repository at this point in the history

  45. refactor: Optimize DeltaLakeCallback batch processing

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    53a87c7 View commit details
    Browse the repository at this point in the history

  46. fix: Convert datetime columns to UTC and microsecond precision

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    acfd7cc View commit details
    Browse the repository at this point in the history

  47. feat: Convert datetime columns to UTC and microsecond precision

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    b600513 View commit details
    Browse the repository at this point in the history

  48. feat: Convert datetime columns to UTC and microsecond precision

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    fae08d4 View commit details
    Browse the repository at this point in the history

  49. feat: Increase optimize_interval, enable time_travel, change numeric_… 

    …type and batch_size in DeltaLakeCallback
    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    b313e5e View commit details
    Browse the repository at this point in the history

  50. fix: Ensure datetime columns are in microsecond precision and remove … 

    …timezone info after conversion
    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    2261007 View commit details
    Browse the repository at this point in the history

  51. feat: Change numeric_type to float in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    1da540f View commit details
    Browse the repository at this point in the history

  52. fix: Ensure datetime columns are in microsecond precision and convert… 

    … timestamp columns to UTC datetime
    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    3d1a84c View commit details
    Browse the repository at this point in the history

  53. fix: Convert timestamp columns to datetime64[ns] instead of datetime6… 

    …4[us, UTC]
    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    fabc822 View commit details
    Browse the repository at this point in the history

  54. fix: Update logging levels in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    47ca6a4 View commit details
    Browse the repository at this point in the history

  55. fix: Remove unnecessary logging statements in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    cf0004a View commit details
    Browse the repository at this point in the history

  56. fix: Convert integer columns in DeltaLakeCallback

    @tommy-ca
    tommy-ca committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    c4d7b23 View commit details
    Browse the repository at this point in the history