Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spark 3.5 (Scala 2.12) #1272

Open
wants to merge 30 commits into
base: master
Choose a base branch
from
Open

Spark 3.5 (Scala 2.12) #1272

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
1c97b40
Merge branch 'master' into Spark3.3
ghislainfourny May 4, 2023
1b41e02
Merge branch 'master' into Spark3.3
ghislainfourny May 16, 2023
3bebc9e
Fix parsing.
May 16, 2023
b73a128
Update to Spark 3.5.
Dec 21, 2023
9d0bfc4
Upgrade to AntLR 4.9.3.
Dec 21, 2023
570b548
Change jackson version to align with Spark.
Dec 21, 2023
c9931b3
Fix test.
Dec 21, 2023
1803a5c
Fix tests.
Dec 21, 2023
1d1d9f6
Fix tests.
Dec 21, 2023
524667d
Fix tests.
Dec 21, 2023
f745e31
Merge branch 'master' into Spark3.5.0
ghislainfourny Feb 27, 2024
c52ff1f
Fix test.
Feb 27, 2024
4aef769
Fix test.
Feb 27, 2024
debaef5
Merge branch 'master' into Spark3.5.0
ghislainfourny Feb 27, 2024
4b619b4
Merge branch 'master' into Spark3.5.0
ghislainfourny Feb 27, 2024
5521e0f
Merge branch 'master' into Spark3.5.0
ghislainfourny Jul 9, 2024
c4a0e34
Minor update.
Jul 10, 2024
ce49b4e
Merge branch 'master' of github.com:RumbleDB/rumble into Spark3.5.0
Jul 10, 2024
287869a
Merge branch 'master' of github.com:RumbleDB/rumble into Spark3.5.0
Jul 10, 2024
fb040cf
Merge branch 'master' into Spark3.5.0
ghislainfourny Jul 10, 2024
8e4460f
Merge branch 'master' into Spark3.5.0
ghislainfourny Sep 26, 2024
3ce80c1
Update pom.xml
ghislainfourny Oct 17, 2024
eeba107
Merge branch 'master' into Spark3.5.0
ghislainfourny Oct 24, 2024
e151cc8
Merge pull request #1270 from RumbleDB/delta-lake-functions
ghislainfourny Oct 24, 2024
023abbd
Merge branch 'master' into Spark3.5.0
ghislainfourny Oct 24, 2024
f913a12
Merge branch 'master' into Spark3.5-Scala2.12
ghislainfourny Oct 28, 2024
6a189dc
Merge branch 'master' into Spark3.5-Scala2.12
ghislainfourny Oct 28, 2024
92c9211
Merge branch 'master' into Spark3.5-Scala2.12
ghislainfourny Nov 18, 2024
5a8383c
Delete src/test/resources/test_files/runtime-spark/FileInputs/Delta.jq
ghislainfourny Nov 18, 2024
9875e81
Update maven.yml
ghislainfourny Nov 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .github/workflows/maven.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,5 +136,3 @@ jobs:
run: mvn install -DskipTests -Dgpg.skip --quiet
- name: Compile with Maven
run: mvn clean compile assembly:single
- name: DeltaUpdateRuntimeTests
run: mvn -Dtest=DeltaUpdateRuntimeTests test
72 changes: 36 additions & 36 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -220,31 +220,31 @@
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>3.4.2</version>
<version>3.5.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>3.4.2</version>
<version>3.5.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.12</artifactId>
<version>3.4.2</version>
<version>3.5.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-aws</artifactId>
<version>3.3.2</version>
<version>3.3.6</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-avro_2.12</artifactId>
<version>3.4.2</version>
<version>3.5.1</version>
</dependency>
<dependency>
<groupId>org.antlr</groupId>
Expand Down Expand Up @@ -287,37 +287,37 @@
<artifactId>commons-io</artifactId>
<version>2.11.0</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.13</version>
</dependency>
<!--<dependency>
<groupId>edu.vanderbilt.accre</groupId>
<artifactId>laurelin</artifactId>
<version>1.0.1</version>
</dependency>-->
<dependency>
<groupId>org.jgrapht</groupId>
<artifactId>jgrapht-core</artifactId>
<version>1.4.0</version>
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.10.6</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-yaml</artifactId>
<version>2.13.4</version>
</dependency>
<dependency>
<groupId>io.delta</groupId>
<artifactId>delta-core_2.12</artifactId>
<version>2.4.0</version>
</dependency>
</dependencies>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.13</version>
</dependency>
<!--<dependency>
<groupId>edu.vanderbilt.accre</groupId>
<artifactId>laurelin</artifactId>
<version>1.0.1</version>
</dependency>-->
<dependency>
<groupId>org.jgrapht</groupId>
<artifactId>jgrapht-core</artifactId>
<version>1.4.0</version>
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.10.6</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-yaml</artifactId>
<version>2.15.2</version>
</dependency>
<dependency>
<groupId>io.delta</groupId>
<artifactId>delta-spark_2.12</artifactId>
<version>3.2.1</version>
</dependency>
</dependencies>

<distributionManagement>
<snapshotRepository>
Expand Down
4 changes: 2 additions & 2 deletions src/main/java/org/rumbledb/parser/XQueryLexer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Generated from ./src/main/java/org/rumbledb/parser/XQueryLexer.g4 by ANTLR 4.8
// Generated from ./src/main/java/org/rumbledb/parser/XQueryLexer.g4 by ANTLR 4.9.3

// Java header
package org.rumbledb.parser;
Expand All @@ -18,7 +18,7 @@

@SuppressWarnings({"all", "warnings", "unchecked", "unused", "cast"})
public class XQueryLexer extends Lexer {
static { RuntimeMetaData.checkVersion("4.8", RuntimeMetaData.VERSION); }
static { RuntimeMetaData.checkVersion("4.9.3", RuntimeMetaData.VERSION); }

protected static final DFA[] _decisionToDFA;
protected static final PredictionContextCache _sharedContextCache =
Expand Down
4 changes: 2 additions & 2 deletions src/main/java/org/rumbledb/parser/XQueryParser.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Generated from ./src/main/java/org/rumbledb/parser/XQueryParser.g4 by ANTLR 4.8
// Generated from ./src/main/java/org/rumbledb/parser/XQueryParser.g4 by ANTLR 4.9.3

// Java header
package org.rumbledb.parser;
Expand All @@ -25,7 +25,7 @@

@SuppressWarnings({"all", "warnings", "unchecked", "unused", "cast"})
public class XQueryParser extends Parser {
static { RuntimeMetaData.checkVersion("4.8", RuntimeMetaData.VERSION); }
static { RuntimeMetaData.checkVersion("4.9.3", RuntimeMetaData.VERSION); }

protected static final DFA[] _decisionToDFA;
protected static final PredictionContextCache _sharedContextCache =
Expand Down
2 changes: 1 addition & 1 deletion src/main/java/org/rumbledb/parser/XQueryParserBaseVisitor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Generated from ./src/main/java/org/rumbledb/parser/XQueryParser.g4 by ANTLR 4.8
// Generated from ./src/main/java/org/rumbledb/parser/XQueryParser.g4 by ANTLR 4.9.3

// Java header
package org.rumbledb.parser;
Expand Down
2 changes: 1 addition & 1 deletion src/main/java/org/rumbledb/parser/XQueryParserVisitor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Generated from ./src/main/java/org/rumbledb/parser/XQueryParser.g4 by ANTLR 4.8
// Generated from ./src/main/java/org/rumbledb/parser/XQueryParser.g4 by ANTLR 4.9.3

// Java header
package org.rumbledb.parser;
Expand Down
3 changes: 2 additions & 1 deletion src/test/resources/test_files/runtime-spark/DataFrames/GroupbyClause5.jq
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
(:JIQS: ShouldRun; Output="([ 1, { "guess" : "Bulgarian", "choices" : [ "Albanian", "Bulgarian", "Russian", "Ukrainian" ] }, { "guess" : "Bulgarian", "choices" : [ "Arabic", "Bulgarian", "Hebrew", "Spanish" ] } ], [ 2, { "guess" : "Arabic", "choices" : [ "Arabic", "Cantonese", "Maltese", "Samoan" ] }, { "guess" : "Arabic", "choices" : [ "Albanian", "Arabic", "Czech", "Dutch" ] } ])" :)
(:JIQS: ShouldRun; Output="([ 1, { "guess" : "Arabic", "choices" : [ "Arabic", "Cantonese", "Maltese", "Samoan" ] }, { "guess" : "Arabic", "choices" : [ "Albanian", "Arabic", "Czech", "Dutch" ] } ], [ 2, { "guess" : "Bulgarian", "choices" : [ "Albanian", "Bulgarian", "Russian", "Ukrainian" ] }, { "guess" : "Bulgarian", "choices" : [ "Arabic", "Bulgarian", "Hebrew", "Spanish" ] } ])" :)
for $i in parallelize((
{"guess": "Arabic", "choices": ["Arabic", "Cantonese", "Maltese", "Samoan"]},
{"guess": "Arabic", "choices": ["Albanian", "Arabic", "Czech", "Dutch"]},
{"guess": "Bulgarian", "choices": ["Albanian", "Bulgarian", "Russian", "Ukrainian"]},
{"guess": "Bulgarian", "choices": ["Arabic", "Bulgarian", "Hebrew", "Spanish"]}))
group by $guess := $i.guess
order by $guess
count $group_index
return [$group_index, $i]

Expand Down
3 changes: 0 additions & 3 deletions src/test/resources/test_files/runtime-spark/FileInputs/Delta.jq
View file
Open in desktop

This file was deleted.

Loading