Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support stateless queries. #184

Merged
merged 6 commits into from
Dec 18, 2023
Merged

Support stateless queries. #184

merged 6 commits into from
Dec 18, 2023

Conversation

goomrw
Copy link
Contributor

@goomrw goomrw commented Dec 13, 2023

From
https://cloud.google.com/java/docs/reference/google-cloud-bigquery/2.34.1/com.google.cloud.bigquery.BigQuery

Stateless queries: query execution without corresponding job metadata

Stateless queries are currently in preview.

This PR introduces support for stateless queries via a new JDBC query parameter. Set jobcreationmode to a string that matches one of the standard JobCreationMode enum values:

  • JOB_CREATION_MODE_UNSPECIFIED: Unspecified JobCreationMode, defaults to JOB_CREATION_REQUIRED.
  • JOB_CREATION_REQUIRED: Default. Job creation is always required.
  • JOB_CREATION_OPTIONAL: Job creation is optional. Returning immediate results is prioritized. BigQuery will automatically determine if a Job needs to be created. The conditions under which BigQuery can decide to not create a Job are subject to change. If Job creation is required, JOB_CREATION_REQUIRED mode should be used, which is the default.

See
https://github.com/googleapis/java-bigquery/blob/v2.34.0/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/QueryJobConfiguration.java#L98-L111

For example, the following create a connection with optional job creation:

DriverManager.getConnection(
    "jdbc:BQDriver:my-project?jobcreationmode=JOB_CREATION_OPTIONAL",
    driverProperties);

Note that this will cause queries to fail if the provided project does not have stateless queries preview enabled.

@goomrw
Support stateless queries.
efef4cc 
From
https://cloud.google.com/java/docs/reference/google-cloud-bigquery/2.34.1/com.google.cloud.bigquery.BigQuery

> Stateless queries: query execution without corresponding job metadata

Stateless queries are currently in preview.

This PR introduces support for stateless queries via a new JDBC query
parameter. Set `jobcreationmode` to a string that matches one of the
standard `JobCreationMode` enum values:

- `JOB_CREATION_MODE_UNSPECIFIED`: Unspecified JobCreationMode,
   defaults to `JOB_CREATION_REQUIRED`.
- `JOB_CREATION_REQUIRED`: Default. Job creation is always
   required.
- `JOB_CREATION_OPTIONAL`: Job creation is optional. Returning immediate
   results is prioritized. BigQuery will automatically determine if a Job
   needs to be created. The conditions under which BigQuery can decide to
   not create a Job are subject to change. If Job creation is required,
   JOB_CREATION_REQUIRED mode should be used, which is the default.

See
https://github.com/googleapis/java-bigquery/blob/v2.34.0/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/QueryJobConfiguration.java#L98-L111

For example, the following create a connection with optional job
creation:

    DriverManager.getConnection(
        "jdbc:BQDriver:my-project?jobcreationmode=JOB_CREATION_OPTIONAL",
        driverProperties);

Note that this will cause queries to fail if the provided project does
not have stateless queries preview enabled.
@goomrw goomrw force-pushed the goomrw-bq-stateless-query branch from 7ca2597 to efef4cc Compare December 13, 2023 19:29
fzakaria
fzakaria reviewed Dec 13, 2023
View reviewed changes
Copy link
Contributor

@fzakaria fzakaria left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stopped right before the start of the tests

src/main/java/net/starschema/clouddb/jdbc/BQConnection.java Show resolved Hide resolved
src/main/java/net/starschema/clouddb/jdbc/BQConnection.java Show resolved Hide resolved
src/main/java/net/starschema/clouddb/jdbc/BQConnection.java Outdated Show resolved Hide resolved
src/main/java/net/starschema/clouddb/jdbc/BQForwardOnlyResultSet.java Show resolved Hide resolved
@goomrw
Add microbenchmark for stateless queries.
71bc3a9 
This adds a Java Microbenchmark Harness (JMH) benchmark that compares
a small query with and without job creation.

Documentation on running the microbenchmark can be found in its
class' Javadoc.

A small query with job creation takes almost twice as long as one
with optional job creation.
@goomrw
Address review comment about docs; fix Javadoc.
40e7b13 
Method references need # not .
@goomrw
Copy link
Contributor Author

goomrw commented Dec 14, 2023

I think we have a flaky test:

syncQueryCancel(net.starschema.clouddb.jdbc.CancelTest)  Time elapsed: 1.007 sec  <<< FAILURE!
junit.framework.AssertionFailedError: expected:<1> but was:<0>
	at junit.framework.Assert.fail(Assert.java:57)
	at junit.framework.Assert.failNotEquals(Assert.java:329)
	at junit.framework.Assert.assertEquals(Assert.java:78)
	at junit.framework.Assert.assertEquals(Assert.java:234)
	at junit.framework.Assert.assertEquals(Assert.java:241)
	at net.starschema.clouddb.jdbc.CancelTest.syncQueryCancel(CancelTest.java:111)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:[189](https://github.com/looker-open-source/bqjdbc/actions/runs/7202650361/job/19621182281?pr=184#step:6:190))
	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)```

@goomrw
Copy link
Contributor Author

goomrw commented Dec 14, 2023

71bc3a9 added a microbenchmark. Here's a representative summary:

Benchmark                                                     Mode  Cnt   Score    Error  Units
StatelessSmallQueryBenchmark.benchmarkSmallQueryOptionalJob  thrpt    5  67.994 ± 10.326  ops/s
StatelessSmallQueryBenchmark.benchmarkSmallQueryRequiredJob  thrpt    5  37.171 ±  3.041  ops/s

Queries with optional jobs are between 1.4x and 2.3x faster than required jobs!

fzakaria
fzakaria approved these changes Dec 18, 2023
View reviewed changes
Copy link
Contributor

@fzakaria fzakaria left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

More can be improved in subsequent commits and are outside the scope of this change.

@goomrw goomrw merged commit fd1165e into main Dec 18, 2023
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fzakaria fzakaria fzakaria approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@goomrw @fzakaria