Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test: Add support for fetching Redshift query results using Redshift unload command #24359

Closed
wants to merge 4 commits into from
Closed

Test: Add support for fetching Redshift query results using Redshift unload command #24359

wants to merge 4 commits into from
+1,609 −14

Conversation

ebyhr
Copy link
Member

@ebyhr ebyhr commented Dec 4, 2024

Description

Run CI with secrets: #24117

@cla-bot cla-bot bot added the cla-signed label Dec 4, 2024
@github-actions github-actions bot added docs hudi Hudi connector iceberg Iceberg connector delta-lake Delta Lake connector hive Hive connector labels Dec 4, 2024
@ebyhr ebyhr force-pushed the ebi/mayank/redshift-unload branch from 54df82d to 2e98524 Compare December 4, 2024 03:52
@mayankvadariya
Copy link
Contributor

@ebyhr please help me pull latest commit from #24117

@ebyhr ebyhr force-pushed the ebi/mayank/redshift-unload branch from 2e98524 to 6f2b682 Compare December 5, 2024 04:03
@ebyhr
Copy link
Member Author

ebyhr commented Dec 5, 2024

Updated.

@ebyhr ebyhr force-pushed the ebi/mayank/redshift-unload branch from 6f2b682 to 012f238 Compare December 15, 2024 23:10
@ebyhr
Copy link
Member Author

ebyhr commented Dec 15, 2024

Updated.

@mayankvadariya
Copy link
Contributor

@ebyhr can you please pull the latest changes from parent PR?

@mayankvadariya
Copy link
Contributor

https://github.com/trinodb/trino/actions/runs/12343063858/job/34443666325?pr=24359#step:19:179 test passes locally with these commits. 🤔

@ebyhr ebyhr force-pushed the ebi/mayank/redshift-unload branch from 012f238 to 634aa48 Compare December 18, 2024 21:45
@ebyhr
Copy link
Member Author

ebyhr commented Dec 18, 2024

Updated.

@mayankvadariya
Copy link
Contributor

https://github.com/trinodb/trino/actions/runs/12401862084/job/34622195723?pr=24359#step:19:181 failed again.

@mayankvadariya
Copy link
Contributor

@ebyhr please update the PR with the latest commits from source PR.

@ebyhr ebyhr force-pushed the ebi/mayank/redshift-unload branch from 634aa48 to 3ae7328 Compare December 23, 2024 07:13
@ebyhr
Copy link
Member Author

ebyhr commented Dec 23, 2024

Updated.

@mayankvadariya
Copy link
Contributor

Finally redshift tests are passing https://github.com/trinodb/trino/actions/runs/12462869117/job/34784444202?pr=24359 with b93a8f0 commit. TY @raunaqmorarka for helping me put the right fix.

@mayankvadariya
Copy link
Contributor

@ebyhr please update the PR with the latest commits from source PR.

mayankvadariya and others added 4 commits December 27, 2024 14:20
@mayankvadariya
Correctly merge multiple splits info
f33c4bb 
Previously SplitOperatorInfo wasn't Mergeable and hence base
OperatorInfo(OperatorStats#getMergeableInfoOrNull) was null.
@nineinchnick @mayankvadariya
Prepare to implement a page source provider for Redshift
9712af9
@nineinchnick @mayankvadariya
Fetch Redshift query results unloaded to S3
e37ef6f 
Co-authored-by: Mayank Vadariya <[email protected]>
@mayankvadariya
REMOVE - run connector test
9582983
@ebyhr ebyhr force-pushed the ebi/mayank/redshift-unload branch from 3ae7328 to 9582983 Compare December 27, 2024 23:20
@mayankvadariya
Copy link
Contributor

It seems unload tests are passing but Redshift job is timing out. @ebyhr can you please rerun redshift job?
https://github.com/trinodb/trino/actions/runs/12521687438/job/34929074733?pr=24359#step:19:453

[INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 826.6 s -- in io.trino.plugin.redshift.TestRedshiftUnload
[INFO] Tests run: 41, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 622.9 s -- in io.trino.plugin.redshift.TestRedshiftUnloadTypeMapping

@mayankvadariya
Copy link
Contributor

@hashhar @Praveen2112 @wendigo can you please help me rerun failing jobs including Redshift one?
https://github.com/trinodb/trino/actions/runs/12521687438/job/34929074733?pr=24359

@wendigo
Copy link
Contributor

wendigo commented Dec 30, 2024

@mayankvadariya I did and it failed again

@mayankvadariya
Copy link
Contributor

Thank you @wendigo. Let's wait for the third run to finish https://github.com/trinodb/trino/actions/runs/12521687438/job/34987834854?pr=24359
If it fails again with the timeout then, can you please push a temporary commit with increased timeout in ci.yml?

@wendigo
Copy link
Contributor

wendigo commented Dec 30, 2024

@mayankvadariya the failure was due to a deadlock, not a timeout so I guess it won't help

@mayankvadariya
Copy link
Contributor

Closing as source PR #24117 has been merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
cla-signed delta-lake Delta Lake connector docs hive Hive connector hudi Hudi connector iceberg Iceberg connector
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@ebyhr @mayankvadariya @wendigo @nineinchnick