Skip to content

Commit

Permalink
Feature/257 mysql dialect (#266)
Browse files Browse the repository at this point in the history
* #191: implemented MySQL dialect
* #257: changed the version to 2.1.0
  • Loading branch information
AnastasiiaSergienko authored Sep 23, 2019
1 parent 5fd7cf1 commit 7fcead9
Show file tree
Hide file tree
Showing 34 changed files with 602 additions and 30 deletions.
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ Supported dialects:
1. [Exasol](doc/dialects/exasol.md)
1. [Hive](doc/dialects/hive.md)
1. [Impala](doc/dialects/impala.md)
1. [MySQL](doc/dialects/mysql.md))
1. [Oracle](doc/dialects/oracle.md)
1. [PostgreSQL](doc/dialects/postgresql.md)
1. [Redshift](doc/dialects/redshift.md)
Expand Down
2 changes: 1 addition & 1 deletion doc/dialects/athena.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ The SQL statement below creates the adapter script, defines the Java class that
```sql
CREATE OR REPLACE JAVA ADAPTER SCRIPT ADAPTER.JDBC_ADAPTER AS
%scriptclass com.exasol.adapter.RequestDispatcher;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.0.1.jar;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.1.0.jar;
%jar /buckets/<BFS service>/<bucket>/AthenaJDBC42-<JDBC driver version>.jar;
/
;
Expand Down
2 changes: 1 addition & 1 deletion doc/dialects/aurora.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ The SQL statement below creates the adapter script, defines the Java class that
```sql
CREATE OR REPLACE JAVA ADAPTER SCRIPT ADAPTER.JDBC_ADAPTER AS
%scriptclass com.exasol.adapter.RequestDispatcher;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.0.1.jar;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.1.0.jar;
%jar /buckets/<BFS service>/<bucket>/postgresql-<JDBC driver version>.jar;
/
```
Expand Down
2 changes: 1 addition & 1 deletion doc/dialects/bigquery.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Please remember to check the versions of your JAR files after downloading driver
```sql
CREATE JAVA ADAPTER SCRIPT ADAPTER.JDBC_ADAPTER AS
%scriptclass com.exasol.adapter.RequestDispatcher;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.0.1.jar;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.1.0.jar;
%jar /buckets/<BFS service>/<bucket>/avro-1.8.2.jar;
%jar /buckets/<BFS service>/<bucket>/gax-1.40.0.jar;
%jar /buckets/<BFS service>/<bucket>/google-api-client-1.28.0.jar;
Expand Down
4 changes: 2 additions & 2 deletions doc/dialects/db2.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ The SQL statement below creates the adapter script, defines the Java class that
```sql
CREATE OR REPLACE JAVA ADAPTER SCRIPT ADAPTER.JDBC_ADAPTER AS
%scriptclass com.exasol.adapter.RequestDispatcher;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.0.1.jar;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.1.0.jar;
%jar /buckets/<BFS service>/<bucket>/db2jcc4.jar;
%jar /buckets/<BFS service>/<bucket>/db2jcc_license_cu.jar;
/
Expand All @@ -68,7 +68,7 @@ CREATE OR REPLACE JAVA ADAPTER SCRIPT ADAPTER.JDBC_ADAPTER AS
```sql
CREATE OR REPLACE JAVA ADAPTER SCRIPT ADAPTER.JDBC_ADAPTER AS
%scriptclass com.exasol.adapter.RequestDispatcher;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.0.1.jar;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.1.0.jar;
%jar /buckets/<BFS service>/<bucket>/db2jcc4.jar;
%jar /buckets/<BFS service>/<bucket>/db2jcc_license_cu.jar;
%jar /buckets/<BFS service>/<bucket>/db2jcc_license_cisuz.jar;
Expand Down
2 changes: 1 addition & 1 deletion doc/dialects/exasol.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ The SQL statement below creates the adapter script, defines the Java class that
```sql
CREATE JAVA ADAPTER SCRIPT ADAPTER.JDBC_ADAPTER AS
%scriptclass com.exasol.adapter.RequestDispatcher;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.0.1.jar;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.1.0.jar;
/
```

Expand Down
2 changes: 1 addition & 1 deletion doc/dialects/hive.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ The SQL statement below creates the adapter script, defines the Java class that
```sql
CREATE OR REPLACE JAVA ADAPTER SCRIPT ADAPTER.JDBC_ADAPTER AS
%scriptclass com.exasol.adapter.RequestDispatcher;
%jar /buckets/<BFS service>/<bucket>/jars/virtualschema-jdbc-adapter-dist-2.0.1.jar;
%jar /buckets/<BFS service>/<bucket>/jars/virtualschema-jdbc-adapter-dist-2.1.0.jar;
%jar /buckets/<BFS service>/<bucket>/jars/HiveJDBC41.jar;
/
```
Expand Down
2 changes: 1 addition & 1 deletion doc/dialects/impala.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ The SQL statement below creates the adapter script, defines the Java class that
```sql
CREATE OR REPLACE JAVA ADAPTER SCRIPT ADAPTER.JDBC_ADAPTER AS
%scriptclass com.exasol.adapter.RequestDispatcher;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.0.1.jar;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.1.0.jar;
%jar /buckets/<BFS service>/<bucket>/ImpalaJDBC41.jar;
/
;
Expand Down
87 changes: 87 additions & 0 deletions doc/dialects/mysql.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
# MySQL SQL Dialect

[MySQL](https://www.mysql.com/) is an open-source relational database management system.

## Registering the JDBC Driver in EXAOperation

First download the [MySQL JDBC driver](https://dev.mysql.com/downloads/connector/j/).
Select Operating System -> Platform Independent -> Download.

Now register the driver in EXAOperation:

1. Click "Software"
1. Switch to tab "JDBC Drivers"
1. Click "Browse..."
1. Select JDBC driver file
1. Click "Upload"
1. Click "Add"
1. In dialog "Add EXACluster JDBC driver" configure the JDBC driver (see below)

You need to specify the following settings when adding the JDBC driver via EXAOperation.

| Parameter | Value |
|-----------|-----------------------------------------------------|
| Name | `MYSQL` |
| Main | `com.mysql.jdbc.Driver` |
| Prefix | `jdbc:mysql:` |
| Files | `mysql-connector-java-<version>.jar` |

IMPORTANT: Currently you have to **Disable Security Manager** for the driver if you want to connect to MySQL using Virtual Schemas.
It is necessary because JDBC driver requires a JAVA permission which we do not grant by default.

## Uploading the JDBC Driver to EXAOperation

1. [Create a bucket in BucketFS](https://docs.exasol.com/administration/on-premise/bucketfs/create_new_bucket_in_bucketfs_service.htm)
1. Upload the driver to BucketFS

This step is necessary since the UDF container the adapter runs in has no access to the JDBC drivers installed via EXAOperation but it can access BucketFS.

## Installing the Adapter Script

Upload the latest available release of [Virtual Schema JDBC Adapter](https://github.com/exasol/virtual-schemas/releases) to Bucket FS.

Then create a schema to hold the adapter script.

```sql
CREATE SCHEMA ADAPTER;
```

The SQL statement below creates the adapter script, defines the Java class that serves as entry point and tells the UDF framework where to find the libraries (JAR files) for Virtual Schema and database driver.

```sql
CREATE OR REPLACE JAVA ADAPTER SCRIPT ADAPTER.JDBC_ADAPTER AS
%scriptclass com.exasol.adapter.RequestDispatcher;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.1.0.jar;
%jar /buckets/<BFS service>/<bucket>/mysql-connector-java-<version>.jar;
/
;
```

## Defining a Named Connection

Define the connection to MySQL as shown below.

```sql
CREATE OR REPLACE CONNECTION MYSQL_CONNECTION
TO 'jdbc:mysql://<host>:<port>/'
USER '<user>'
IDENTIFIED BY '<password>';
```

## Creating a Virtual Schema

Below you see how a MySQL Virtual Schema is created.

```sql
CREATE VIRTUAL SCHEMA <virtual schema name>
USING ADAPTER.JDBC_ADAPTER
WITH
SQL_DIALECT = 'MYSQL'
CONNECTION_NAME = 'MYSQL_CONNECTION'
SCHEMA_NAME = '<schema name>';
```

## Data Types Mapping and Limitations

- `TIME` is casted to `TIMESTAMP` with a format `1970-01-01 hh:mm:ss`.
- Unsupported data types: `BINARY`, `VARBINARY`, `BLOB`, `TINYBLOB`, `MEDIUMBLOB`, `LONGBLOB`.
2 changes: 1 addition & 1 deletion doc/dialects/oracle.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ The SQL statement below creates the adapter script, defines the Java class that
```sql
CREATE JAVA ADAPTER SCRIPT ADAPTER.JDBC_ADAPTER AS
%scriptclass com.exasol.adapter.RequestDispatcher;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.0.1.jar;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.1.0.jar;
%jar /buckets/<BFS service>/<bucket>/ojdbc<JDBC driver version>.jar;
/
;
Expand Down
2 changes: 1 addition & 1 deletion doc/dialects/postgresql.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ The SQL statement below creates the adapter script, defines the Java class that
```sql
CREATE OR REPLACE JAVA ADAPTER SCRIPT ADAPTER.JDBC_ADAPTER AS
%scriptclass com.exasol.adapter.RequestDispatcher;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.0.1.jar;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.1.0.jar;
%jar /buckets/<BFS service>/<bucket>/postgresql-<version>.jar;
/
```
Expand Down
2 changes: 1 addition & 1 deletion doc/dialects/redshift.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ The SQL statement below creates the adapter script, defines the Java class that
```sql
CREATE OR REPLACE JAVA ADAPTER SCRIPT ADAPTER.JDBC_ADAPTER AS
%scriptclass com.exasol.adapter.RequestDispatcher;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.0.1.jar;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.1.0.jar;
%jar /buckets/<BFS service>/<bucket>/RedshiftJDBC42-<JDBC driver version>.jar;
/
;
Expand Down
2 changes: 1 addition & 1 deletion doc/dialects/saphana.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ The SQL statement below creates the adapter script, defines the Java class that
```sql
CREATE JAVA ADAPTER SCRIPT ADAPTER.JDBC_ADAPTER AS
%scriptclass com.exasol.adapter.RequestDispatcher;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.0.1.jar;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.1.0.jar;
%jar /buckets/<BFS service>/<bucket>/ngdbc-<JDBC driver version>.jar;
/
;
Expand Down
2 changes: 1 addition & 1 deletion doc/dialects/sql_server.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ The SQL statement below creates the adapter script, defines the Java class that
```sql
CREATE OR REPLACE JAVA ADAPTER SCRIPT ADAPTER.JDBC_ADAPTER AS
%scriptclass com.exasol.adapter.RequestDispatcher;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.0.1.jar;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.1.0.jar;
%jar /buckets/<BFS service>/<bucket>/jtds.jar;
/
```
Expand Down
2 changes: 1 addition & 1 deletion doc/dialects/sybase.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ The SQL statement below creates the adapter script, defines the Java class that
```sql
CREATE OR REPLACE JAVA ADAPTER SCRIPT ADAPTER.JDBC_ADAPTER AS
%scriptclass com.exasol.adapter.RequestDispatcher;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.0.1.jar;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.1.0.jar;
%jar /buckets/<BFS service>/<bucket>/jtds-<version>.jar;
/
```
Expand Down
2 changes: 1 addition & 1 deletion doc/dialects/teradata.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ The SQL statement below creates the adapter script, defines the Java class that
```sql
CREATE OR REPLACE JAVA ADAPTER SCRIPT ADAPTER.JDBC_ADAPTER AS
%scriptclass com.exasol.adapter.RequestDispatcher;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.0.1.jar;
%jar /buckets/<BFS service>/<bucket>/virtualschema-jdbc-adapter-dist-2.1.0.jar;
%jar /buckets/<BFS service>/<bucket>/terajdbc4.jar;
%jar /buckets/<BFS service>/<bucket>/tdgssconfig.jar;
/
Expand Down
8 changes: 4 additions & 4 deletions doc/user-guide/deploying_the_virtual_schema_adapter.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ cd virtual-schemas/jdbc-adapter/
mvn clean -DskipTests package
```

The resulting fat JAR is stored in `virtualschema-jdbc-adapter-dist/target/virtualschema-jdbc-adapter-dist-2.0.1.jar`.
The resulting fat JAR is stored in `virtualschema-jdbc-adapter-dist/target/virtualschema-jdbc-adapter-dist-2.1.0.jar`.

## Uploading the Adapter JAR Archive

Expand All @@ -46,8 +46,8 @@ Following steps are required to upload a file to a bucket:
1. Now upload the file into this bucket, e.g. using curl (adapt the hostname, BucketFS port, bucket name and bucket write password).

```bash
curl -X PUT -T virtualschema-jdbc-adapter-dist/target/virtualschema-jdbc-adapter-dist-2.0.1.jar \
http://w:[email protected]:2580/bucket1/virtualschema-jdbc-adapter-dist-2.0.1.jar
curl -X PUT -T virtualschema-jdbc-adapter-dist/target/virtualschema-jdbc-adapter-dist-2.1.0.jar \
http://w:[email protected]:2580/bucket1/virtualschema-jdbc-adapter-dist-2.1.0.jar
```

If you later need to change the bucket passwords, select the bucket and click "Edit".
Expand Down Expand Up @@ -84,7 +84,7 @@ CREATE SCHEMA ADAPTER;

CREATE JAVA ADAPTER SCRIPT ADAPTER.JDBC_ADAPTER AS
%scriptclass com.exasol.adapter.RequestDispatcher;
%jar /buckets/your-bucket-fs/your-bucket/virtualschema-jdbc-adapter-dist-2.0.1.jar;
%jar /buckets/your-bucket-fs/your-bucket/virtualschema-jdbc-adapter-dist-2.1.0.jar;
%jar /buckets/your-bucket-fs/your-bucket/<JDBC driver>.jar;
/
```
Expand Down
11 changes: 7 additions & 4 deletions doc/user-guide/user_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,17 +17,18 @@ Please note that the syntax for creating adapter scripts is not recognized by al
1. [Athena](../dialects/athena.md)
1. [Aurora](../dialects/aurora.md)
1. [Big Query](../dialects/bigquery.md)
1. [DB2](../dialects/db2.md)
1. [EXASOL](../dialects/exasol.md)
1. [Hive](../dialects/hive.md)
1. [Impala](../dialects/impala.md)
1. [DB2](../dialects/db2.md)
1. [MySQL](doc/dialects/mysql.md))
1. [Oracle](../dialects/oracle.md)
1. [Teradata](../dialects/teradata.md)
1. [PostgresSQL](../dialects/postgresql.md)
1. [Redshift](../dialects/redshift.md)
1. [SAP HANA](.../dialects/saphana.md)
1. [SQL Server](../dialects/sql_server.md)
1. [Sybase ASE](../dialects/sybase.md)
1. [PostgresSQL](../dialects/postgresql.md)
1. [Teradata](../dialects/teradata.md)
1. Generic


Expand Down Expand Up @@ -178,7 +179,9 @@ There are a few important things you need to know about those data types.

1. Columns of an unrecognized / unsupported data type are not mapped in a Virtual Schema. From Exasol's perspective those columns do not exist on a table. This is done so that tables containing those columns can still be mapped and do not have to be rejected as a whole.
2. You can't query columns of an unrecognized / unsupported data type. If the source table contains them, you have to *explicitly* exclude them from the query. You can for example not use the asterisk (`*`) on a table that contains one ore more of those columns. This will result in an error issued by the Virtual schema.
3. You can't use functions that result in an unsupported / unknown data type.
3. If you want to query all columns except unsupported, add `1` to the columns list. Otherwise you will see the same error as if you query with the asterisk (`*`).
For example, a table contains 3 columns: `bool_column` BOOLEAN, `timestamp_column` TIMESTAMP, `blob_column` BLOB. The column BLOB is not supported. If you want to query two other columns, use: `SELECT "bool_column", "timestamp_column", 1 FROM table_name;` .
4. You can't use functions that result in an unsupported / unknown data type.

## See Also

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ general:
debugAddress: '192.168.0.12:3000' # Address which will be defined as DEBUG_ADDRESS in the virtual schemas
bucketFsUrl: http://exasol-host:2580/bucket1
bucketFsPassword: bucket1
jdbcAdapterPath: /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-2.0.1.jar
jdbcAdapterPath: /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-2.1.0.jar

exasol:
runIntegrationTests: true
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ general:
debugAddress: '192.168.0.12:3000' # Address which will be defined as DEBUG_ADDRESS in the virtual schemas
bucketFsUrl: http://exasol-host:2580/bucket1
bucketFsPassword: bucket1
jdbcAdapterPath: /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-2.0.1.jar
jdbcAdapterPath: /buckets/bucketfs1/bucket1/virtualschema-jdbc-adapter-dist-2.1.0.jar

exasol:
runIntegrationTests: true
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ general:
debug: true
debugAddress: ''
bucketFsUrl: http://127.0.0.1:6594/default
jdbcAdapterPath: /buckets/bfsdefault/default/virtualschema-jdbc-adapter-dist-2.0.1.jar
jdbcAdapterPath: /buckets/bfsdefault/default/virtualschema-jdbc-adapter-dist-2.1.0.jar
additionalJDBCDriverDir: /vagrant/drivers/

exasol:
Expand Down
2 changes: 1 addition & 1 deletion jdbc-adapter/local/integration-test-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ general:
debugAddress: '10.44.1.228:3000' # Address which will be defined as DEBUG_ADDRESS in the virtual schemas
bucketFsUrl: http://localhost:2580/jars
bucketFsPassword: public
jdbcAdapterPath: /buckets/bfsdefault/jars/virtualschema-jdbc-adapter-dist-2.0.1.jar
jdbcAdapterPath: /buckets/bfsdefault/jars/virtualschema-jdbc-adapter-dist-2.1.0.jar

exasol:
runIntegrationTests: true
Expand Down
2 changes: 1 addition & 1 deletion jdbc-adapter/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
<module>virtualschema-jdbc-adapter-dist</module>
</modules>
<properties>
<product.version>2.0.1</product.version>
<product.version>2.1.0</product.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>9</java.version>
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
package com.exasol.adapter.dialects.mysql;

import java.sql.Connection;
import java.sql.Types;

import com.exasol.adapter.AdapterProperties;
import com.exasol.adapter.dialects.IdentifierConverter;
import com.exasol.adapter.dialects.JdbcTypeDescription;
import com.exasol.adapter.jdbc.BaseColumnMetadataReader;
import com.exasol.adapter.metadata.DataType;

/**
* This class implements MySQL-specific reading of column metadata.
*/
public class MySqlColumnMetadataReader extends BaseColumnMetadataReader {
/**
* Create a new instance of the {@link MySqlColumnMetadataReader}.
*
* @param connection JDBC connection through which the column metadata is read from the remote database
* @param properties user-defined adapter properties
* @param identifierConverter converter between source and Exasol identifiers
*/
public MySqlColumnMetadataReader(final Connection connection, final AdapterProperties properties,
final IdentifierConverter identifierConverter) {
super(connection, properties, identifierConverter);
}

@Override
public DataType mapJdbcType(final JdbcTypeDescription jdbcTypeDescription) {
switch (jdbcTypeDescription.getJdbcType()) {
case Types.TIME:
return DataType.createTimestamp(false);
case Types.BINARY:
return DataType.createUnsupported();
default:
return super.mapJdbcType(jdbcTypeDescription);
}
}
}
Loading

0 comments on commit 7fcead9

Please sign in to comment.