Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

change to 9 parameters constructor #823

Merged
merged 5 commits into from
Sep 3, 2024

Conversation

XavierZYXue
Copy link
Contributor

@XavierZYXue XavierZYXue commented Aug 1, 2024

fixes #821 to switch the 9 parameters constructor, could you please review it?

@cerveada
Copy link
Contributor

cerveada commented Aug 2, 2024

Hello, thanks for your contribution.

Please correct me if I am wrong, but this fixes the issues for the new version of BigQuery, but also it will break it for the older versions.

Another thing I notice is that in the linked Big Query code the method has 8 params, but here you use 9. How come?

@XavierZYXue
Copy link
Contributor Author

XavierZYXue commented Aug 2, 2024

@cerveada hi, first of appreciated your response, I revised this logic because it was changed to 9 parameters after this PR, and affetced all versions after 0.26.0
that is to say, it will be applicable to all versions after 0.26.0

@cerveada
Copy link
Contributor

cerveada commented Aug 2, 2024

That is ok, but we should keep it compatible with the old version as well. We already do this in other plugins. Usually some combination of retries and reflection is sufficient to support both.

You can see some examples of it here:
core/src/main/scala/za/co/absa/spline/harvester/plugin/embedded/DataSourceV2Plugin.scala

@XavierZYXue
Copy link
Contributor Author

@cerveada I've added different constructors separately are compatible with all Spark BigQuery connector
please review again?

Copy link
Contributor

@wajda wajda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The build for the Spark 3.4 is failing due to timeout.
Also please see my comment below.

@XavierZYXue
Copy link
Contributor Author

any comment for this after I changed? and why the pipeline always timeout? do you know the reason

@XavierZYXue XavierZYXue force-pushed the bigfix-#821-incompatibleversion branch from e1d7aee to 4925c68 Compare September 2, 2024 14:19
Copy link

sonarcloud bot commented Sep 2, 2024

Copy link
Contributor

@wajda wajda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
Regarding timeout - I think there is some non-deterministic behaviour in tests. It also happens to other branches, so no relation to this PR.

@wajda wajda requested a review from cerveada September 3, 2024 12:46
Copy link
Contributor

@cerveada cerveada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wajda wajda merged commit fd58cd0 into AbsaOSS:develop Sep 3, 2024
13 of 14 checks passed
@rkrumins
Copy link

rkrumins commented Sep 4, 2024

Amazing, thank you very much @cerveada and @cerveada

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Incompatible version between Spline and Spark BigQuery connector
4 participants