Refactoring/180 remove jdbc adapter properties (#192)

* #180: Refactored property handling and query rewriting.

.gitignore

@@ -19,3 +19,4 @@ dependency-reduced-pom.xml
 **/local
 Scripts
 .dbeaver*
+**/*.log

README.md

@@ -43,7 +43,6 @@ This means an user familiar with SQL will immediately feel at home when accessin
 
 This is an open source project which is officially supported by Exasol. For any question, you can contact our support team.
 
-
 # Table of Contents
 
 ## Information for Users
@@ -73,3 +72,28 @@ This is an open source project which is officially supported by Exasol. For any
 * [Remote Debugging](doc/development/remote_debugging.md)
 * [Troubleshooting](doc/development/troubleshooting.md)
 * [Versioning](doc/development/versioning.md)
+
+## Dependencies
+
+### Run Time Dependencies
+
+Running the Virtual Schema requires a Java Runtime version 8 or later.
+
+| Dependency                                                                          | Purpose                                                | License                       |
+|-------------------------------------------------------------------------------------|--------------------------------------------------------|-------------------------------|
+| [JSON-P](https://javaee.github.io/jsonp/)                                           | JSON Processing                                        | CDDL-1.0                      |
+| [Exasol Script API](https://docs.exasol.com/database_concepts/udf_scripts.htm)      | Accessing Exasol features                              | MIT License                   |
+| JDBC driver(s), depending on data source                                            | Connecting to the data source                          | Check driver documentation    |
+
+### Build Time Dependencies
+
+| Dependency                                                                          | Purpose                                                | License                       |
+|-------------------------------------------------------------------------------------|--------------------------------------------------------|-------------------------------|
+| [Apache Derby](https://db.apache.org/derby/)                                        | Pure-Java embedded database                            | Apache License 2.0            |
+| [Apache Maven](https://maven.apache.org/)                                           | Build tool                                             | Apache License 2.0            |
+| [Java Hamcrest](http://hamcrest.org/JavaHamcrest/)                                  | Checking for conditions in code via matchers           | BSD License                   |
+| [JSONassert](http://jsonassert.skyscreamer.org/)                                    | Compare JSON documents for semantic equality           | Apache License 2.0            |
+| [JUnit](https://junit.org/junit5)                                                   | Unit testing framework                                 | Eclipse Public License 1.0    |
+| [Mockito](http://site.mockito.org/)                                                 | Mocking framework                                      | MIT License                   |
+| [JUnit 5 System Extensions](https://github.com/itsallcode/junit5-system-extensions) | Capturing `STDOUT` and `STDERR`                        | Eclipse Public License 2.0    |
+| [Equals Verifier](https://jqno.nl/equalsverifier/)                                  | Testing `equals(...)` and `hashCode()` contracts       | Apache License 2.0            |

doc/development/step_by_step_guide_to_writing_your_own_dialect.md

@@ -451,4 +451,40 @@ As mentioned before, if the `BaseTableMetadataReader` and `BaseColumnMetadataRea
 
 If not, you must implement your own variants.
 
-Take a look at the `PostgreSQLColumnMetadataReader` and its unit test for an example of how to instantiate a different sub-reader from the top-level reader.
+Take a look at the `PostgreSQLColumnMetadataReader` and its unit test for an example of how to instantiate a different sub-reader from the top-level reader.
+
+## Implementing Query Rewriting
+
+At its very core the Virtual Schema JDBC adapter rewrites queries. When a user issues a query at the Virtual Schema frontend, it is first parsed and interpreted by the database core, then pushed down in parts or as a whole to the Virtual Schema backend. In the case of the JDBC adapter we have remote data source that speak SQL — or a driver that makes it like as if they did. But before the Virtual Schema backend pushes a query further down to the remote data source, it rewrites it to accommodate dialect differences.
+
+If possible the backend does not even execute that query itself, but aims to construct an [IMPORT](https://docs.exasol.com/sql/import.htm) statement and gives that back to the Virtual Schema frontend. That way the Exasol database can directly run `IMPORT` on the remote data source, effectively remove the need for mashalling, transferring and unmarshalling data payload for the communication between Virtual Schema backend and frontend.
+
+###  Pre-Requisites for Using IMPORT
+
+If you plan to use IMPORT there are some things you need to bare in mind:
+
+1. Remote data source must offer a JDBC driver
+   (always the case if you write a dialect for the JDBC adapter instead of an adapter from scratch)
+1. The driver must support switching Auto-Commit off
+
+The ExaLoader needs control over committing during an import, so that it is for example able to roll back incomplete imports.
+
+### Overloading Rewriting
+
+The `AbstractSqlDialect` has a base implementation for query rewriting that should work with most databases that support JDBC. Sometimes you need more though and in that case you have different mechanisms for doing that.
+
+#### Variant a) Implementing Your own QueryRewriter
+
+Our recommended way is to implement a specialized `QueryRewriter` for your dialect. In most cases deriving that from the `BaseQueryRewriter` and adding some minimal changes does the trick.
+
+Check the `ExasolQueryRewriter` or `OracleQueryRewriter` for examples on how to extend the base implementation. In those two examples we added support for the specialized `IMPORT FROM EXA` and `IMPORT FROM ORA` variants which are more efficient than a standard `IMPORT`.
+
+Also take a look at the corresponding unit tests to learn how to verify the behavior of the rewriters without being forced to run slow and cumbersome integration tests against the remote data source during development.
+
+This variant is less effort than implementing query rewriting from scratch.
+
+Make sure to override the method `createQueryRewriter()` in the dialect to instantiate the right implementation for your use case. 
+
+#### Variant b) Override `rewriteQuery(...)` in the Dialect
+
+In those very rare case where the `QueryRewriter` does not work for you, override the method `rewriteQuery(...)` in your dialect.

doc/dialects/athena.md

@@ -43,7 +43,7 @@ You install the adapter script via the special SQL command `CREATE JAVA ADAPTER
 ```sql
 CREATE OR REPLACE JAVA ADAPTER SCRIPT ADAPTER.JDBC_ADAPTER AS
     %scriptclass com.exasol.adapter.jdbc.JdbcAdapter;
-    %jar /buckets/bucketfs1/jdbc/virtualschema-jdbc-adapter-dist-1.16.2.jar;
+    %jar /buckets/bucketfs1/jdbc/virtualschema-jdbc-adapter-dist-1.16.3.jar;
     %jar /buckets/bucketfs1/jdbc/AthenaJDBC42-<JDBC driver version>.jar;
 /
 ```

doc/dialects/db2.md

@@ -46,7 +46,7 @@ CREATE or replace JAVA ADAPTER SCRIPT adapter.jdbc_adapter AS
 
   // This will add the adapter jar to the classpath so that it can be used inside the adapter script
   // Replace the names of the bucketfs and the bucket with the ones you used.
-  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.2.jar;
+  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.3.jar;
 
   // DB2 Driver files
   %jar /buckets/bucketfs1/bucket1/db2jcc4.jar;

doc/dialects/exasol.md

@@ -17,7 +17,7 @@ After uploading the adapter jar, the adapter script can be created as follows:
 CREATE SCHEMA adapter;
 CREATE JAVA ADAPTER SCRIPT adapter.jdbc_adapter AS
   %scriptclass com.exasol.adapter.jdbc.JdbcAdapter;
-  %jar /buckets/your-bucket-fs/your-bucket/virtualschema-jdbc-adapter-dist-1.16.2.jar;
+  %jar /buckets/your-bucket-fs/your-bucket/virtualschema-jdbc-adapter-dist-1.16.3.jar;
 /
 ```
 

doc/dialects/hive.md

@@ -23,7 +23,7 @@ CREATE SCHEMA adapter;
 CREATE  JAVA  ADAPTER SCRIPT jdbc_adapter AS
   %scriptclass com.exasol.adapter.jdbc.JdbcAdapter;
 
-  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.2.jar;
+  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.3.jar;
 
   %jar /buckets/bucketfs1/bucket1/HiveJDBC41.jar;
 /

doc/dialects/impala.md

@@ -22,7 +22,7 @@ CREATE SCHEMA adapter;
 CREATE  JAVA  ADAPTER SCRIPT jdbc_adapter AS
   %scriptclass com.exasol.adapter.jdbc.JdbcAdapter;
 
-  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.2.jar;
+  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.3.jar;
 
   %jar /buckets/bucketfs1/bucket1/hive_metastore.jar;
   %jar /buckets/bucketfs1/bucket1/hive_service.jar;

doc/dialects/oracle.md

@@ -32,7 +32,7 @@ CREATE JAVA ADAPTER SCRIPT adapter.jdbc_oracle AS
 
   // You need to replace `your-bucket-fs` and `your-bucket` to match the actual location
   // of the adapter jar.
-  %jar /buckets/your-bucket-fs/your-bucket/virtualschema-jdbc-adapter-dist-1.16.2.jar;
+  %jar /buckets/your-bucket-fs/your-bucket/virtualschema-jdbc-adapter-dist-1.16.3.jar;
 
   // Add the oracle jdbc driver to the classpath
   %jar /buckets/bucketfs1/bucket1/ojdbc7-12.1.0.2.jar

doc/dialects/postgresql.md

@@ -15,7 +15,7 @@ CREATE OR REPLACE JAVA ADAPTER SCRIPT adapter.jdbc_adapter
 
   // This will add the adapter jar to the classpath so that it can be used inside the adapter script
   // Replace the names of the bucketfs and the bucket with the ones you used.
-  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.2.jar;
+  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.3.jar;
 
   // You have to add all files of the data source jdbc driver here (e.g. MySQL or Hive)
   %jar /buckets/bucketfs1/bucket1/postgresql-42.0.0.jar;

doc/dialects/redshift.md

@@ -45,7 +45,7 @@ You install the adapter script via the special SQL command `CREATE JAVA ADAPTER
 ```sql
 CREATE OR REPLACE JAVA ADAPTER SCRIPT ADAPTER.JDBC_ADAPTER AS
     %scriptclass com.exasol.adapter.jdbc.JdbcAdapter;
-    %jar /buckets/bucketfs1/jdbc/virtualschema-jdbc-adapter-dist-1.16.2.jar;
+    %jar /buckets/bucketfs1/jdbc/virtualschema-jdbc-adapter-dist-1.16.3.jar;
     %jar /buckets/bucketfs1/jdbc/RedshiftJDBC42-<JDBC driver version>.jar;
 /
 ```

doc/dialects/sql_server.md

@@ -17,7 +17,7 @@ CREATE OR REPLACE JAVA ADAPTER SCRIPT adapter.sql_server_jdbc_adapter
 
   // This will add the adapter jar to the classpath so that it can be used inside the adapter script
   // Replace the names of the bucketfs and the bucket with the ones you used.
-  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.2.jar;
+  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.3.jar;
 
   // You have to add all files of the data source jdbc driver here 
   %jar /buckets/bucketfs1/bucket1/jtds.jar;

doc/dialects/sybase.md

@@ -18,7 +18,7 @@ CREATE OR REPLACE JAVA ADAPTER SCRIPT adapter.jdbc_adapter
   AS
 
   %scriptclass com.exasol.adapter.jdbc.JdbcAdapter;
-  %jar /buckets/bucketfs1/virtualschema/virtualschema-jdbc-adapter-dist-1.16.2.jar;
+  %jar /buckets/bucketfs1/virtualschema/virtualschema-jdbc-adapter-dist-1.16.3.jar;
   %jar /buckets/bucketfs1/virtualschema/jtds-1.3.1.jar;
 /
 ```

doc/dialects/teradata.md

@@ -22,7 +22,7 @@ CREATE OR REPLACE JAVA ADAPTER SCRIPT adapter.jdbc_adapter
 
   // This will add the adapter jar to the classpath so that it can be used inside the adapter script
   // Replace the names of the bucketfs and the bucket with the ones you used.
-  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.2.jar;
+  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.3.jar;
 
   // You have to add all files of the data source jdbc driver here (e.g. MySQL or Hive)
   %jar /buckets/bucketfs1/bucket1/terajdbc4.jar;

doc/user-guide/deploying_the_virtual_schema_adapter.md

@@ -23,7 +23,7 @@ cd virtual-schemas/jdbc-adapter/
 mvn clean -DskipTests package
 ```
 
-The resulting fat JAR is stored in `virtualschema-jdbc-adapter-dist/target/virtualschema-jdbc-adapter-dist-1.16.2.jar`.
+The resulting fat JAR is stored in `virtualschema-jdbc-adapter-dist/target/virtualschema-jdbc-adapter-dist-1.16.3.jar`.
 
 ## Uploading the Adapter JAR Archive
 
@@ -42,8 +42,8 @@ Following steps are required to upload a file to a bucket:
 1. Now upload the file into this bucket, e.g. using curl (adapt the hostname, BucketFS port, bucket name and bucket write password).
 
 ```bash
-curl -X PUT -T virtualschema-jdbc-adapter-dist/target/virtualschema-jdbc-adapter-dist-1.16.2.jar \
- http://w:[email protected]:2580/bucket1/virtualschema-jdbc-adapter-dist-1.16.2.jar
+curl -X PUT -T virtualschema-jdbc-adapter-dist/target/virtualschema-jdbc-adapter-dist-1.16.3.jar \
+ http://w:[email protected]:2580/bucket1/virtualschema-jdbc-adapter-dist-1.16.3.jar
 ```
 
 See chapter 3.6.4. "The synchronous cluster file system BucketFS" in the EXASolution User Manual for more details about BucketFS.
@@ -75,7 +75,7 @@ CREATE JAVA ADAPTER SCRIPT adapter.jdbc_adapter AS
 
   // This will add the adapter jar to the classpath so that it can be used inside the adapter script
   // Replace the names of the bucketfs and the bucket with the ones you used.
-  %jar /buckets/your-bucket-fs/your-bucket/virtualschema-jdbc-adapter-dist-1.16.2.jar;
+  %jar /buckets/your-bucket-fs/your-bucket/virtualschema-jdbc-adapter-dist-1.16.3.jar;
 
   // You have to add all files of the data source jdbc driver here (e.g. Hive JDBC driver files)
   %jar /buckets/your-bucket-fs/your-bucket/name-of-data-source-jdbc-driver.jar;

doc/user-guide/dialects/db2.md

@@ -46,7 +46,7 @@ CREATE or replace JAVA ADAPTER SCRIPT adapter.jdbc_adapter AS
 
   // This will add the adapter jar to the classpath so that it can be used inside the adapter script
   // Replace the names of the bucketfs and the bucket with the ones you used.
-  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.2.jar;
+  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.3.jar;
 
   // DB2 Driver files
   %jar /buckets/bucketfs1/bucket1/db2jcc4.jar;

doc/user-guide/dialects/exasol.md

@@ -17,7 +17,7 @@ After uploading the adapter jar, the adapter script can be created as follows:
 CREATE SCHEMA adapter;
 CREATE JAVA ADAPTER SCRIPT adapter.jdbc_adapter AS
   %scriptclass com.exasol.adapter.jdbc.JdbcAdapter;
-  %jar /buckets/your-bucket-fs/your-bucket/virtualschema-jdbc-adapter-dist-1.16.2.jar;
+  %jar /buckets/your-bucket-fs/your-bucket/virtualschema-jdbc-adapter-dist-1.16.3.jar;
 /
 ```
 

doc/user-guide/dialects/hive.md

@@ -23,7 +23,7 @@ CREATE SCHEMA adapter;
 CREATE  JAVA  ADAPTER SCRIPT jdbc_adapter AS
   %scriptclass com.exasol.adapter.jdbc.JdbcAdapter;
 
-  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.2.jar;
+  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.3.jar;
 
   %jar /buckets/bucketfs1/bucket1/HiveJDBC41.jar;
 /

doc/user-guide/dialects/impala.md

@@ -22,7 +22,7 @@ CREATE SCHEMA adapter;
 CREATE  JAVA  ADAPTER SCRIPT jdbc_adapter AS
   %scriptclass com.exasol.adapter.jdbc.JdbcAdapter;
 
-  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.2.jar;
+  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.3.jar;
 
   %jar /buckets/bucketfs1/bucket1/hive_metastore.jar;
   %jar /buckets/bucketfs1/bucket1/hive_service.jar;

doc/user-guide/dialects/oracle.md

@@ -32,7 +32,7 @@ CREATE JAVA ADAPTER SCRIPT adapter.jdbc_oracle AS
 
   // You need to replace `your-bucket-fs` and `your-bucket` to match the actual location
   // of the adapter jar.
-  %jar /buckets/your-bucket-fs/your-bucket/virtualschema-jdbc-adapter-dist-1.16.2.jar;
+  %jar /buckets/your-bucket-fs/your-bucket/virtualschema-jdbc-adapter-dist-1.16.3.jar;
 
   // Add the oracle jdbc driver to the classpath
   %jar /buckets/bucketfs1/bucket1/ojdbc7-12.1.0.2.jar

doc/user-guide/dialects/postgresql.md

@@ -15,7 +15,7 @@ CREATE OR REPLACE JAVA ADAPTER SCRIPT adapter.jdbc_adapter
 
   // This will add the adapter jar to the classpath so that it can be used inside the adapter script
   // Replace the names of the bucketfs and the bucket with the ones you used.
-  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.2.jar;
+  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.3.jar;
 
   // You have to add all files of the data source jdbc driver here (e.g. MySQL or Hive)
   %jar /buckets/bucketfs1/bucket1/postgresql-42.0.0.jar;

doc/user-guide/dialects/redshift.md

@@ -21,7 +21,7 @@ CREATE OR REPLACE JAVA ADAPTER SCRIPT adapter.jdbc_adapter
 
   // This will add the adapter jar to the classpath so that it can be used inside the adapter script
   // Replace the names of the bucketfs and the bucket with the ones you used.
-  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.2.jar;
+  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.3.jar;
 
   // You have to add all files of the data source jdbc driver here (e.g. MySQL or Hive)
 

doc/user-guide/dialects/sql_server.md

@@ -17,7 +17,7 @@ CREATE OR REPLACE JAVA ADAPTER SCRIPT adapter.sql_server_jdbc_adapter
 
   // This will add the adapter jar to the classpath so that it can be used inside the adapter script
   // Replace the names of the bucketfs and the bucket with the ones you used.
-  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.2.jar;
+  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.3.jar;
 
   // You have to add all files of the data source jdbc driver here 
   %jar /buckets/bucketfs1/bucket1/jtds.jar;

doc/user-guide/dialects/sybase.md

@@ -18,7 +18,7 @@ CREATE OR REPLACE JAVA ADAPTER SCRIPT adapter.jdbc_adapter
   AS
 
   %scriptclass com.exasol.adapter.jdbc.JdbcAdapter;
-  %jar /buckets/bucketfs1/virtualschema/virtualschema-jdbc-adapter-dist-1.16.2.jar;
+  %jar /buckets/bucketfs1/virtualschema/virtualschema-jdbc-adapter-dist-1.16.3.jar;
   %jar /buckets/bucketfs1/virtualschema/jtds-1.3.1.jar;
 /
 ```

doc/user-guide/dialects/teradata.md

@@ -22,7 +22,7 @@ CREATE OR REPLACE JAVA ADAPTER SCRIPT adapter.jdbc_adapter
 
   // This will add the adapter jar to the classpath so that it can be used inside the adapter script
   // Replace the names of the bucketfs and the bucket with the ones you used.
-  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.2.jar;
+  %jar /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.3.jar;
 
   // You have to add all files of the data source jdbc driver here (e.g. MySQL or Hive)
   %jar /buckets/bucketfs1/bucket1/terajdbc4.jar;

jdbc-adapter/integration-test-data/integration-test-db2.yaml

@@ -5,7 +5,7 @@ general:
   debugAddress:    '192.168.0.12:3000'  # Address which will be defined as DEBUG_ADDRESS in the virtual schemas
   bucketFsUrl:  http://exasol-host:2580/bucket1
   bucketFsPassword: bucket1
-  jdbcAdapterPath:  /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.2.jar
+  jdbcAdapterPath:  /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.3.jar
 
 exasol:
   runIntegrationTests: true

jdbc-adapter/integration-test-data/integration-test-sample.yaml

@@ -5,7 +5,7 @@ general:
   debugAddress:    '192.168.0.12:3000'  # Address which will be defined as DEBUG_ADDRESS in the virtual schemas
   bucketFsUrl:  http://exasol-host:2580/bucket1
   bucketFsPassword: bucket1
-  jdbcAdapterPath:  /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.2.jar
+  jdbcAdapterPath:  /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-1.16.3.jar
 
 exasol:
   runIntegrationTests: true

jdbc-adapter/integration-test-data/integration-test-travis.yaml

@@ -4,7 +4,7 @@ general:
   debug:                   false
   debugAddress:            ''
   bucketFsUrl:             http://127.0.0.1:6594/default
-  jdbcAdapterPath:         /buckets/bfsdefault/default/virtualschema-jdbc-adapter-dist-1.16.2.jar
+  jdbcAdapterPath:         /buckets/bfsdefault/default/virtualschema-jdbc-adapter-dist-1.16.3.jar
   additionalJDBCDriverDir: /vagrant/drivers/
 
 exasol:

jdbc-adapter/integration-test-data/run_integration_tests.sh

@@ -88,7 +88,7 @@ cleanup_remote_dbs() {
 
 prepare_docker() {
 	docker pull "$docker_image"
-	git clone "$docker_db_repository" "$1/$docker_name"
+	git clone --branch "$exasol_docker_image_version" "$docker_db_repository" "$1/$docker_name"
 	pushd "$1/$docker_name"
 	pipenv install -r exadt_requirements.txt
 	pipenv run ./exadt create-cluster --root "$1"/cluster --create-root "$docker_name"