Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to spark 3.5 #269

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Upgrade to spark 3.5 #269

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 13 additions & 8 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ This library contains the source code for the Apache Spark Connector for SQL Ser

[Apache Spark](https://spark.apache.org/) is a unified analytics engine for large-scale data processing.

There are three version sets of the connector available through Maven, a 2.4.x, a 3.0.x and a 3.1.x compatible version. All versions can be found [here](https://search.maven.org/search?q=spark-mssql-connector) and can be imported using the coordinates below:
All available versions can be found [here](https://search.maven.org/search?q=spark-mssql-connector) and can be imported using the coordinates below:

| Connector | Maven Coordinate | Scala Version |
|----------------------------------|--------------------------------------------------------|---------------|
Expand All @@ -18,6 +18,7 @@ There are three version sets of the connector available through Maven, a 2.4.x,
| Spark 3.1.x compatible connector | `com.microsoft.azure:spark-mssql-connector_2.12:1.2.0` | 2.12 |
| Spark 3.3.x compatible connector | `com.microsoft.azure:spark-mssql-connector_2.12:1.3.0` | 2.12 |
| Spark 3.4.x compatible connector | `com.microsoft.azure:spark-mssql-connector_2.12:1.4.0` | 2.12 |
| Spark 3.5.x compatible connector | `com.microsoft.azure:spark-mssql-connector_2.12:1.5.0` | 2.12 |

## Current Releases

Expand All @@ -27,6 +28,10 @@ The latest Spark 3.0.x compatible connector is on v1.1.0.

The latest Spark 3.1.x compatible connector is on v1.2.0.

The latest Spark 3.4.x compatible connector is on v1.4.0.

The latest Spark 3.5.x compatible connector is on v1.5.0.

For main changes from previous releases and known issues please refer to [CHANGELIST](docs/CHANGELIST.md)

## Supported Features
Expand All @@ -37,13 +42,13 @@ For main changes from previous releases and known issues please refer to [CHANGE
* Reliable connector support for Sql Server Single Instance


| Component | Versions Supported |
|--------------------------------------|----------------------------|
| Apache Spark | 2.4.x, 3.0.x, 3.1.x, 3.3.x |
| Scala | 2.11, 2.12 |
| Microsoft JDBC Driver for SQL Server | 8.4.1 |
| Microsoft SQL Server | SQL Server 2008 or later |
| Azure SQL Databases | Supported |
| Component | Versions Supported |
|--------------------------------------|------------------------------------------|
| Apache Spark | 2.4.x, 3.0.x, 3.1.x, 3.3.x, 3.4.x, 3.5.x |
| Scala | 2.11, 2.12 |
| Microsoft JDBC Driver for SQL Server | 8.4.1 |
| Microsoft SQL Server | SQL Server 2008 or later |
| Azure SQL Databases | Supported |

*Note: Azure Synapse (Azure SQL DW) use is not tested with this connector. While it may work, there may be unintended consequences.*

Expand Down
12 changes: 6 additions & 6 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<groupId>com.microsoft.azure</groupId>
<artifactId>spark-mssql-connector</artifactId>
<artifactId>spark-mssql-connector_${scala.binary.version}</artifactId>
<packaging>jar</packaging>
<version>1.4.0</version>
<version>1.5.0</version>
<name>${project.groupId}:${project.artifactId}</name>
<description>The Apache Spark Connector for SQL Server and Azure SQL is a high-performance connector that enables you to use transactional data in big data analytics and persists results for ad-hoc queries or reporting.</description>
<url>https://github.com/microsoft/sql-spark-connector</url>
Expand Down Expand Up @@ -201,20 +201,20 @@
</build>
<profiles>
<profile>
<id>spark34</id>
<id>spark35</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<scala.binary.version>2.12</scala.binary.version>
<scala.version>2.12.11</scala.version>
<spark.version>3.4.0</spark.version>
<scala.version>2.12.20</scala.version>
<spark.version>3.5.0</spark.version>
</properties>
<dependencies>
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_${scala.binary.version}</artifactId>
<version>3.2.6</version>
<version>3.2.19</version>
<scope>test</scope>
</dependency>
<dependency>
Expand Down
6 changes: 3 additions & 3 deletions test/scala_test/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,14 +123,14 @@
</build>
<profiles>
<profile>
<id>spark34</id>
<id>spark35</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<scala.binary.version>2.12</scala.binary.version>
<scala.version>2.12.11</scala.version>
<spark.version>3.4.0</spark.version>
<scala.version>2.12.20</scala.version>
<spark.version>3.5.0</spark.version>
</properties>
</profile>
</profiles>
Expand Down