Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PR - Add Delta Lake Backend for Cryptofeed #1054

Open
wants to merge 90 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
90 commits
Select commit Hold shift + click to select a range
07cca17
feat(deltalake): Implement Delta Lake backend and add dependencies
tommy-ca Aug 31, 2024
c003c69
feat(deltalake): optimize Delta Lake implementation
tommy-ca Aug 31, 2024
f8256dc
fix(deltalake): fix book table name
tommy-ca Aug 31, 2024
cf2c926
fix(deltalake): Fix book name
tommy-ca Aug 31, 2024
b02ab52
fix(deltalake): Fix numeric type
tommy-ca Aug 31, 2024
42eea55
fix: Ensure timestamp columns have nanosecond precision in DeltaLake …
tommy-ca Sep 1, 2024
090c4f4
feat: Refactor timestamp column handling in DeltaLakeCallback
tommy-ca Sep 1, 2024
2a60f20
fix: Handle null values in DeltaLakeCallback
tommy-ca Sep 1, 2024
2f01815
feat: Implement DeltaLake backend for Cryptofeed
tommy-ca Sep 2, 2024
2081b11
fix: Refactor DeltaLakeCallback class
tommy-ca Sep 2, 2024
a9ba6b2
fix: Add debug logging for DataFrame schema in DeltaLakeCallback
tommy-ca Sep 2, 2024
4ab9fd8
fix: Add DataFrame schema logging when timestamp-related error occurs…
tommy-ca Sep 2, 2024
797a789
fix: convert timestamp columns to datetime64[ms]
tommy-ca Sep 2, 2024
ceb8f76
fix: Ensure all partition columns are present in the DataFrame
tommy-ca Sep 2, 2024
84c436d
fix: convert timestamp column to datetime64[ms] format
tommy-ca Sep 2, 2024
84fd533
feat: Convert timestamp column to date string format in DeltaLakeCall…
tommy-ca Sep 2, 2024
5e4c4df
refactor: Simplify null value handling in DeltaLakeCallback
tommy-ca Sep 2, 2024
5bdd670
fix: Ensure empty string is a category in categorical columns and han…
tommy-ca Sep 2, 2024
099c6c4
refactor: Refactor DeltaLakeCallback class to improve code readabilit…
tommy-ca Sep 2, 2024
d3a9c71
fix: Improve error handling and logging in DeltaLakeCallback
tommy-ca Sep 3, 2024
fd82534
fix: Optimize Delta Lake table by filling null values with empty strings
tommy-ca Sep 3, 2024
1b37d26
fix: optimize handling of missing data in DeltaLakeCallback
tommy-ca Sep 3, 2024
81870dc
feat: Add custom transformations and improve column validation in Del…
tommy-ca Sep 3, 2024
880c079
fix: Add logging configuration to deltalake backend
tommy-ca Sep 3, 2024
e45dee7
fix: Initialize DeltaLakeCallback and add logging for writer method a…
tommy-ca Sep 3, 2024
fcaa65f
fix: Change logging levels from DEBUG to WARNING in DeltaLakeCallback
tommy-ca Sep 3, 2024
4145588
fix: Improve logging and error handling in DeltaLakeCallback writer m…
tommy-ca Sep 3, 2024
274ca48
fix: Refactor field transformation in DeltaLakeCallback
tommy-ca Sep 3, 2024
43bbd3a
fix: Reorder columns to prioritize exchange and symbol
tommy-ca Sep 3, 2024
e59f57d
fix: Ensure datetime columns have millisecond precision in DeltaLakeC…
tommy-ca Sep 3, 2024
4bacc42
feat: Ensure datetime columns are in millisecond precision
tommy-ca Sep 3, 2024
d520c02
chore: Convert datetime columns to microsecond precision
tommy-ca Sep 8, 2024
b11f9a4
fix: Change log levels from warning to debug for non-critical messages
tommy-ca Sep 8, 2024
11893db
refactor: Simplify datetime column handling in DeltaLakeCallback
tommy-ca Sep 8, 2024
313a8f8
feat: Add batch processing and flush interval to DeltaLakeCallback
tommy-ca Sep 9, 2024
f5d3f8c
fix: Update copyright year in demo_deltalake.py
tommy-ca Sep 9, 2024
959dc7e
Merge branch 'bmoscon:master' into master
tommy-ca Sep 9, 2024
85ad668
feat(deltalake): Implement Delta Lake backend and add dependencies
tommy-ca Aug 31, 2024
b7a20b9
feat(deltalake): optimize Delta Lake implementation
tommy-ca Aug 31, 2024
6f81da6
fix(deltalake): fix book table name
tommy-ca Aug 31, 2024
d12441b
fix(deltalake): Fix book name
tommy-ca Aug 31, 2024
d6a2ae8
fix(deltalake): Fix numeric type
tommy-ca Aug 31, 2024
e257d83
fix: Ensure timestamp columns have nanosecond precision in DeltaLake …
tommy-ca Sep 1, 2024
265e761
feat: Refactor timestamp column handling in DeltaLakeCallback
tommy-ca Sep 1, 2024
00d81ec
fix: Handle null values in DeltaLakeCallback
tommy-ca Sep 1, 2024
d6ed8b9
feat: Implement DeltaLake backend for Cryptofeed
tommy-ca Sep 2, 2024
16f2825
fix: Refactor DeltaLakeCallback class
tommy-ca Sep 2, 2024
d51babc
fix: Add debug logging for DataFrame schema in DeltaLakeCallback
tommy-ca Sep 2, 2024
2a7712e
fix: Add DataFrame schema logging when timestamp-related error occurs…
tommy-ca Sep 2, 2024
f611215
fix: convert timestamp columns to datetime64[ms]
tommy-ca Sep 2, 2024
a16f737
fix: Ensure all partition columns are present in the DataFrame
tommy-ca Sep 2, 2024
4173cc2
fix: convert timestamp column to datetime64[ms] format
tommy-ca Sep 2, 2024
b880bfa
feat: Convert timestamp column to date string format in DeltaLakeCall…
tommy-ca Sep 2, 2024
b02e46c
refactor: Simplify null value handling in DeltaLakeCallback
tommy-ca Sep 2, 2024
358c162
fix: Ensure empty string is a category in categorical columns and han…
tommy-ca Sep 2, 2024
90a1559
refactor: Refactor DeltaLakeCallback class to improve code readabilit…
tommy-ca Sep 2, 2024
1365432
fix: Improve error handling and logging in DeltaLakeCallback
tommy-ca Sep 3, 2024
ac5e61a
fix: Optimize Delta Lake table by filling null values with empty strings
tommy-ca Sep 3, 2024
3338d3c
fix: optimize handling of missing data in DeltaLakeCallback
tommy-ca Sep 3, 2024
7c93b11
feat: Add custom transformations and improve column validation in Del…
tommy-ca Sep 3, 2024
bdea1ff
fix: Add logging configuration to deltalake backend
tommy-ca Sep 3, 2024
11c7b22
fix: Initialize DeltaLakeCallback and add logging for writer method a…
tommy-ca Sep 3, 2024
d0ccd92
fix: Change logging levels from DEBUG to WARNING in DeltaLakeCallback
tommy-ca Sep 3, 2024
a8d6310
fix: Improve logging and error handling in DeltaLakeCallback writer m…
tommy-ca Sep 3, 2024
4946bb8
fix: Refactor field transformation in DeltaLakeCallback
tommy-ca Sep 3, 2024
b1cad2b
fix: Reorder columns to prioritize exchange and symbol
tommy-ca Sep 3, 2024
6cfd5f7
fix: Ensure datetime columns have millisecond precision in DeltaLakeC…
tommy-ca Sep 3, 2024
d213cf0
feat: Ensure datetime columns are in millisecond precision
tommy-ca Sep 3, 2024
adac33b
chore: Convert datetime columns to microsecond precision
tommy-ca Sep 8, 2024
4e2aff6
fix: Change log levels from warning to debug for non-critical messages
tommy-ca Sep 8, 2024
99b3a36
refactor: Simplify datetime column handling in DeltaLakeCallback
tommy-ca Sep 8, 2024
1b25acd
feat: Add batch processing and flush interval to DeltaLakeCallback
tommy-ca Sep 9, 2024
d649fea
fix: Update copyright year in demo_deltalake.py
tommy-ca Sep 9, 2024
9c9158b
Merge remote-tracking branch 'my/master'
tommy-ca Sep 9, 2024
bb4cb4c
feat: Add common configuration for Delta Lake callbacks
tommy-ca Sep 9, 2024
9343c30
Merge remote-tracking branch 'my/master'
tommy-ca Sep 9, 2024
87a77f5
fix: Fix logging message in DeltaLakeCallback
tommy-ca Sep 9, 2024
98762fd
refactor: Improve logging and error handling in DeltaLakeCallback
tommy-ca Sep 9, 2024
53a87c7
refactor: Optimize DeltaLakeCallback batch processing
tommy-ca Sep 9, 2024
acfd7cc
fix: Convert datetime columns to UTC and microsecond precision
tommy-ca Sep 9, 2024
b600513
feat: Convert datetime columns to UTC and microsecond precision
tommy-ca Sep 9, 2024
fae08d4
feat: Convert datetime columns to UTC and microsecond precision
tommy-ca Sep 9, 2024
b313e5e
feat: Increase optimize_interval, enable time_travel, change numeric_…
tommy-ca Sep 9, 2024
2261007
fix: Ensure datetime columns are in microsecond precision and remove …
tommy-ca Sep 9, 2024
1da540f
feat: Change numeric_type to float in DeltaLakeCallback
tommy-ca Sep 9, 2024
3d1a84c
fix: Ensure datetime columns are in microsecond precision and convert…
tommy-ca Sep 9, 2024
fabc822
fix: Convert timestamp columns to datetime64[ns] instead of datetime6…
tommy-ca Sep 9, 2024
47ca6a4
fix: Update logging levels in DeltaLakeCallback
tommy-ca Sep 9, 2024
cf0004a
fix: Remove unnecessary logging statements in DeltaLakeCallback
tommy-ca Sep 9, 2024
c4d7b23
fix: Convert integer columns in DeltaLakeCallback
tommy-ca Sep 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -108,3 +108,5 @@ ENV/

# PyCharm
.idea/
.aider*
.trunk/
Loading
Loading