Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade to spark 3.5 #269

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Upgrade to spark 3.5 #269

wants to merge 1 commit into from

Conversation

grihabor
Copy link

Changes similar to those of the previous upgrade to 3.4 at #235

  • Bumped spark to 3.5.0
  • Bumped the version of the library to 1.5.0
  • Updated the readme accordingly
    • Added the 1.5.0 artifact to the version table

@grihabor
Copy link
Author

@microsoft-github-policy-service agree company="PulsePoint"

@grihabor
Copy link
Author

Tests passed

grihabor@grihabor-Vivobook:~/projects/sql-spark-connector$ mvn test
[INFO] Scanning for projects...
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for com.microsoft.azure:spark-mssql-connector_2.12:jar:1.5.0
[WARNING] 'artifactId' contains an expression but should be a constant. @ com.microsoft.azure:spark-mssql-connector_${scala.binary.version}:1.5.0, /home/grihabor/projects/sql-spark-connector/pom.xml, line 5
, column 17
[WARNING] 
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING] 
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING] 
[INFO] 
[INFO] -----------< com.microsoft.azure:spark-mssql-connector_2.12 >-----------
[INFO] Building com.microsoft.azure:spark-mssql-connector_2.12 1.5.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ spark-mssql-connector_2.12 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/grihabor/projects/sql-spark-connector/src/main/resources
[INFO] 
[INFO] --- scala-maven-plugin:4.4.0:add-source (scala-compile-first) @ spark-mssql-connector_2.12 ---
[INFO] Add Source directory: /home/grihabor/projects/sql-spark-connector/src/main/scala
[INFO] Add Test Source directory: /home/grihabor/projects/sql-spark-connector/src/test/scala
[INFO] 
[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ spark-mssql-connector_2.12 ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- scala-maven-plugin:4.4.0:compile (scala-compile) @ spark-mssql-connector_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: /home/grihabor/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.5-bin_2.12.20__52.0-1.3.5_20200330T000931.jar
[INFO] compile in 1.5 s
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ spark-mssql-connector_2.12 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/grihabor/projects/sql-spark-connector/src/test/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.7.0:testCompile (default-testCompile) @ spark-mssql-connector_2.12 ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/grihabor/projects/sql-spark-connector/target/test-classes
[INFO] 
[INFO] --- scala-maven-plugin:4.4.0:testCompile (test-compile) @ spark-mssql-connector_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: /home/grihabor/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.5-bin_2.12.20__52.0-1.3.5_20200330T000931.jar
[INFO] compile in 0.2 s
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ spark-mssql-connector_2.12 ---
[INFO] Surefire report directory: /home/grihabor/projects/sql-spark-connector/target/surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running com.microsoft.sqlserver.jdbc.spark.DataSourceUtilsTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.205 sec

Results :

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] --- scalatest-maven-plugin:1.0:test (scala-test) @ spark-mssql-connector_2.12 ---
Discovery starting.
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
Discovery completed in 642 milliseconds.
Run starting. Expected test count is: 7
DataSourceTest:
24/12/12 17:08:21 WARN Utils: Your hostname, grihabor-Vivobook resolves to a loopback address: 127.0.1.1; using 172.17.0.1 instead (on interface docker0)
24/12/12 17:08:21 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
24/12/12 17:08:21 INFO SparkContext: Running Spark version 3.5.0
24/12/12 17:08:21 INFO SparkContext: OS info Linux, 6.8.0-49-generic, amd64
24/12/12 17:08:21 INFO SparkContext: Java version 1.8.0_392
24/12/12 17:08:21 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
24/12/12 17:08:22 INFO ResourceUtils: ==============================================================
24/12/12 17:08:22 INFO ResourceUtils: No custom resources configured for spark.driver.
24/12/12 17:08:22 INFO ResourceUtils: ==============================================================
24/12/12 17:08:22 INFO SparkContext: Submitted application: test-sql-context
24/12/12 17:08:22 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: 
, offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
24/12/12 17:08:22 INFO ResourceProfile: Limiting resource is cpu
24/12/12 17:08:22 INFO ResourceProfileManager: Added ResourceProfile id: 0
24/12/12 17:08:22 INFO SecurityManager: Changing view acls to: grihabor
24/12/12 17:08:22 INFO SecurityManager: Changing modify acls to: grihabor
24/12/12 17:08:22 INFO SecurityManager: Changing view acls groups to: 
24/12/12 17:08:22 INFO SecurityManager: Changing modify acls groups to: 
24/12/12 17:08:22 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: grihabor; groups with view permissions: EMPTY; users with modify permissions:
 grihabor; groups with modify permissions: EMPTY
24/12/12 17:08:22 INFO Utils: Successfully started service 'sparkDriver' on port 36287.
24/12/12 17:08:22 INFO SparkEnv: Registering MapOutputTracker
24/12/12 17:08:22 INFO SparkEnv: Registering BlockManagerMaster
24/12/12 17:08:22 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
24/12/12 17:08:22 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
24/12/12 17:08:22 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
24/12/12 17:08:22 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-08ba5e45-e84b-48c9-b174-f9c9ca478747
24/12/12 17:08:22 INFO MemoryStore: MemoryStore started with capacity 1872.3 MiB
24/12/12 17:08:22 INFO SparkEnv: Registering OutputCommitCoordinator
24/12/12 17:08:22 INFO JettyUtils: Start Jetty 0.0.0.0:4040 for SparkUI
24/12/12 17:08:22 INFO Utils: Successfully started service 'SparkUI' on port 4040.
24/12/12 17:08:22 INFO Executor: Starting executor ID driver on host 172.17.0.1
24/12/12 17:08:22 INFO Executor: OS info Linux, 6.8.0-49-generic, amd64
24/12/12 17:08:22 INFO Executor: Java version 1.8.0_392
24/12/12 17:08:22 INFO Executor: Starting executor with user classpath (userClassPathFirst = false): ''
24/12/12 17:08:22 INFO Executor: Created or updated repl class loader org.apache.spark.util.MutableURLClassLoader@41f4fe5 for default.
24/12/12 17:08:22 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 34553.
24/12/12 17:08:22 INFO NettyBlockTransferService: Server created on 172.17.0.1:34553
24/12/12 17:08:22 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
24/12/12 17:08:22 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 172.17.0.1, 34553, None)
24/12/12 17:08:22 INFO BlockManagerMasterEndpoint: Registering block manager 172.17.0.1:34553 with 1872.3 MiB RAM, BlockManagerId(driver, 172.17.0.1, 34553, None)
24/12/12 17:08:22 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 172.17.0.1, 34553, None)
24/12/12 17:08:22 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 172.17.0.1, 34553, None)
24/12/12 17:08:23 INFO DataSourceTest: 

===== TEST OUTPUT FOR com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Schema validation between Spark DataFrame and SQL Server ResultSet' =====

24/12/12 17:08:23 INFO DataSourceTest: 

===== FINISHED com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Schema validation between Spark DataFrame and SQL Server ResultSet' =====

- Schema validation between Spark DataFrame and SQL Server ResultSet
24/12/12 17:08:23 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir.
24/12/12 17:08:23 INFO SharedState: Warehouse path is 'file:/home/grihabor/projects/sql-spark-connector/spark-warehouse/com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest'.
24/12/12 17:08:23 INFO DataSourceTest: 

===== TEST OUTPUT FOR com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'JdbcBulkOptions should have proper Bulk configurations' =====

24/12/12 17:08:23 INFO DataSourceTest: 

===== FINISHED com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'JdbcBulkOptions should have proper Bulk configurations' =====

- JdbcBulkOptions should have proper Bulk configurations
24/12/12 17:08:23 INFO DataSourceTest: 

===== TEST OUTPUT FOR com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Data pool URL generation' =====

24/12/12 17:08:23 INFO DataSourceTest: 

===== FINISHED com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Data pool URL generation' =====

- Data pool URL generation
24/12/12 17:08:23 INFO DataSourceTest: 

===== TEST OUTPUT FOR com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Multi part tablename test' =====

24/12/12 17:08:23 INFO DataSourceTest: 

===== FINISHED com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Multi part tablename test' =====

- Multi part tablename test
24/12/12 17:08:23 INFO DataSourceTest: 

===== TEST OUTPUT FOR com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Data pool options test' =====

24/12/12 17:08:23 INFO DataSourceTest: 

===== FINISHED com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Data pool options test' =====

- Data pool options test
24/12/12 17:08:23 INFO DataSourceTest: 

===== TEST OUTPUT FOR com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Default AAD options are correct.' =====

24/12/12 17:08:23 INFO DataSourceTest: 

===== FINISHED com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Default AAD options are correct.' =====

- Default AAD options are correct.
24/12/12 17:08:23 INFO DataSourceTest: 

===== TEST OUTPUT FOR com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Correct AAD options are set when accessToken is specified' =====

24/12/12 17:08:23 INFO DataSourceTest: 

===== FINISHED com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest: 'Correct AAD options are set when accessToken is specified' =====

- Correct AAD options are set when accessToken is specified
24/12/12 17:08:23 INFO SparkContext: SparkContext is stopping with exitCode 0.
24/12/12 17:08:23 INFO SparkUI: Stopped Spark web UI at http://172.17.0.1:4040
24/12/12 17:08:23 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
24/12/12 17:08:23 INFO MemoryStore: MemoryStore cleared
24/12/12 17:08:23 INFO BlockManager: BlockManager stopped
24/12/12 17:08:23 INFO BlockManagerMaster: BlockManagerMaster stopped
24/12/12 17:08:23 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
24/12/12 17:08:23 INFO SparkContext: Successfully stopped SparkContext
24/12/12 17:08:23 WARN DataSourceTest: 

===== POSSIBLE THREAD LEAK IN SUITE com.microsoft.sqlserver.jdbc.spark.unit.bulkwrite.DataSourceTest, threads: rpc-boss-3-1 (daemon=true), shuffle-boss-6-1 (daemon=true) =====

Run completed in 3 seconds, 112 milliseconds.
Total number of tests run: 7
Suites: completed 2, aborted 0
Tests: succeeded 7, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
24/12/12 17:08:23 INFO ShutdownHookManager: Shutdown hook called
24/12/12 17:08:23 INFO ShutdownHookManager: Deleting directory /tmp/spark-93dff511-fb8d-4106-b2d9-38c5bf129734
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  8.387 s
[INFO] Finished at: 2024-12-12T17:08:24+01:00
[INFO] ------------------------------------------------------------------------

@grihabor grihabor mentioned this pull request Dec 12, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@grihabor