-
Notifications
You must be signed in to change notification settings - Fork 95
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
change to 9 parameters constructor #823
change to 9 parameters constructor #823
Conversation
Hello, thanks for your contribution. Please correct me if I am wrong, but this fixes the issues for the new version of BigQuery, but also it will break it for the older versions. Another thing I notice is that in the linked Big Query code the method has 8 params, but here you use 9. How come? |
That is ok, but we should keep it compatible with the old version as well. We already do this in other plugins. Usually some combination of retries and reflection is sufficient to support both. You can see some examples of it here: |
@cerveada I've added different constructors separately are compatible with all Spark BigQuery connector |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The build for the Spark 3.4 is failing due to timeout.
Also please see my comment below.
core/src/main/scala/za/co/absa/spline/harvester/plugin/embedded/BigQueryPlugin.scala
Outdated
Show resolved
Hide resolved
any comment for this after I changed? and why the pipeline always timeout? do you know the reason |
e1d7aee
to
4925c68
Compare
Quality Gate passedIssues Measures |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
Regarding timeout - I think there is some non-deterministic behaviour in tests. It also happens to other branches, so no relation to this PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
fixes #821 to switch the 9 parameters constructor, could you please review it?