Skip to content

Commit

Permalink
Merge branch 'master' into ghislainfourny-patch-1
Browse files Browse the repository at this point in the history
  • Loading branch information
mschoeb authored Oct 31, 2024
2 parents 6073b9f + 3e283a5 commit 1b2bae6
Show file tree
Hide file tree
Showing 39 changed files with 301 additions and 48 deletions.
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ target/
.ipynb_checkpoints/
*.iml
/bin/
*.jar
rumble*.jar
*.*~

# for now ignore jsound package
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,4 @@ The documentation also contains an introduction specific to RumbleDB and how you

[The documentation of the current master (for the adventurous and curious) is available here.](http://sparksoniq.readthedocs.io/en/latest/)

RumbleDB is an effort involving many researchers and ETH Zurich students: code and support by Stefan Irimescu, Ghislain Fourny, Gustavo Alonso, Renato Marroquin, Rodrigo Bruno, Falko Noé, Ioana Stefan, Andrea Rinaldi, Stevan Mihajlovic, Mario Arduini, Can Berker Çıkış, Elwin Stephan, David Dao, Zirun Wang, Ingo Müller, Dan-Ovidiu Graur, Thomas Zhou, Olivier Goerens, Alexandru Meterez, Remo Röthlisberger, Dominik Bruggisser, David Loughlin, David Buzatu.
RumbleDB is an effort involving many researchers and ETH Zurich students: code and support by Stefan Irimescu, Ghislain Fourny, Gustavo Alonso, Renato Marroquin, Rodrigo Bruno, Falko Noé, Ioana Stefan, Andrea Rinaldi, Stevan Mihajlovic, Mario Arduini, Can Berker Çıkış, Elwin Stephan, David Dao, Zirun Wang, Ingo Müller, Dan-Ovidiu Graur, Thomas Zhou, Olivier Goerens, Alexandru Meterez, Remo Röthlisberger, Dominik Bruggisser, David Loughlin, David Buzatu, Marco Schöb.
10 changes: 6 additions & 4 deletions docs/Getting started.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,11 +78,11 @@ If you use Linux, Florian Kellner also kindly contributed an [installation scrip

RumbleDB requires an Apache Spark installation on Linux, Mac or Windows.

It is straightforward to directly [download it](https://spark.apache.org/downloads.html), unpack it and put it at a location of your choosing. We recommend to pick Spark 3.2.2. Let us call this location SPARK_HOME (it is a good idea, in fact to also define an environment variable SPARK_HOME pointing to the absolute path of this location).
It is straightforward to directly [download it](https://spark.apache.org/downloads.html), unpack it and put it at a location of your choosing. We recommend to pick Spark 3.4.3. Let us call this location SPARK_HOME (it is a good idea, in fact to also define an environment variable SPARK_HOME pointing to the absolute path of this location).

What you need to do then is to add the subdirectory "bin" within the unpacked directory to the PATH variable. On macOS this is done by adding

export SPARK_HOME=/path/to/spark-3.2.2-bin-hadoop3.2
export SPARK_HOME=/path/to/spark-3.4.3-bin-hadoop3.2
export PATH=$SPARK_HOME/bin:$PATH

(with SPARK_HOME appropriately set to match your unzipped Spark directory) to the file .zshrc in your home directory, then making sure to force the change with
Expand Down Expand Up @@ -111,9 +111,11 @@ Like Spark, RumbleDB is just a download and no installation is required.

In order to run RumbleDB, you simply need to download one of the small .jar files from the [download page](https://github.com/RumbleDB/rumble/releases) and put it in a directory of your choice, for example, right besides your data.

If you use Spark 3.2+, use rumbledb-1.22.0-for-spark-3.2.jar.
If you use Spark 3.4+, use rumbledb-1.22.0-for-spark-3.4.jar.

If you use Spark 3.3+, use rumbledb-1.22.0-for-spark-3.3.jar.
If you use Spark 3.5+, use rumbledb-1.22.0-for-spark-3.5.jar.

If you use Spark 4.0+ (preview), use rumbledb-1.22.0-for-spark-4.0.jar.

These jars do not embed Spark, since you chose to set it up separately. They will work with your Spark installation with the spark-submit command.

Expand Down
4 changes: 2 additions & 2 deletions docs/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@ We show here how to install RumbleDB from the github repository if you wish to d
The following software is required:

- [Java SE](http://www.oracle.com/technetwork/java/javase/downloads/index.html) 8 (last tested on OpenJDK 8u251). The version of Java is important, as Spark only works with Java 8 or java 11.
- [Spark](https://spark.apache.org/), version 3.1.2 (for example)
- [Spark](https://spark.apache.org/), version 3.4.4 (for example)
- [Ant](https://ant.apache.org/), version 1.11.1
- [ANTLR](http://www.ant.org/), version 4.8 (supplied in our repository)
- [ANTLR](http://www.ant.org/), version 4.9.3 (supplied in our repository)
- [Maven](https://maven.apache.org/) 3.6.0

Important: the ANTLR version varies with the Spark version, because Spark is also shipped with an ANTLR runtime (example: Spark 3.0 and 3.1 is with ANTLR 4.7, Spark 3.2 with ANTLR 4.8). The ANTLR runtime MUST match the ANTLR generator used to generate the RumbleDB jar file.
Expand Down
16 changes: 13 additions & 3 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,10 @@
<compilerId>eclipse</compilerId>
<source>1.8</source>
<target>1.8</target>
<compilerArguments>
<properties>${project.basedir}/org.eclipse.jdt.core.prefs</properties>
</compilerArguments>
<compilerArgs>
<arg>-properties</arg>
<arg>${project.basedir}/org.eclipse.jdt.core.prefs</arg>
</compilerArgs>
</configuration>
<dependencies>
<dependency>
Expand All @@ -63,6 +64,14 @@
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-resources-plugin</artifactId>
<version>3.3.1</version>
<configuration>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
Expand Down Expand Up @@ -150,6 +159,7 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-release-plugin</artifactId>
<version>3.0.1</version>
<configuration>
<localCheckout>true</localCheckout>
<pushChanges>false</pushChanges>
Expand Down
8 changes: 4 additions & 4 deletions src/main/java/org/rumbledb/cli/Main.java
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,12 @@ public class Main {

public static void main(String[] args) throws IOException {
String javaVersion = System.getProperty("java.version");
if (!javaVersion.startsWith("1.8") && !javaVersion.startsWith("11.")) {
System.err.println("[Error] RumbleDB requires Java 8 or Java 11.");
if (!javaVersion.startsWith("11.") && !javaVersion.startsWith("17.")) {
System.err.println("[Error] RumbleDB requires Java 11 or 17.");
System.err.println("Your Java version: " + System.getProperty("java.version"));
System.err.println("You can download Java 8 or 11 from https://adoptium.net/");
System.err.println("You can download Java 11 or 17 from https://adoptium.net/");
System.err.println(
"If you do have Java 8 or 11, but the wrong version appears above, then it means you need to set your JAVA_HOME environment variable properly to point to Java 8 or 11."
"If you do have Java 11 or 17, but the wrong version appears above, then it means you need to set your JAVA_HOME environment variable properly to point to Java 11 or 17."
);
System.exit(43);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -656,7 +656,11 @@ public StaticContext visitProlog(Prolog prolog, StaticContext argument) {
@Override
public StaticContext visitProgram(Program program, StaticContext argument) {
visitDescendants(program, argument);
ExecutionMode mergedExecutionMode = getHighestExecutionModeFromStatements(exitStatementChildren);
ExecutionMode mergedExecutionMode = program.getStatementsAndOptionalExpr().getHighestExecutionMode();
for (Statement statement : this.exitStatementChildren) {
ExecutionMode statementExecMode = statement.getHighestExecutionMode(this.visitorConfig);
mergedExecutionMode = getHighestExecutionMode(mergedExecutionMode, statementExecMode);
}
program.setHighestExecutionMode(mergedExecutionMode);
return argument;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ public class SequentialClassificationVisitor extends AbstractNodeVisitor<Descend
public SequentialClassificationVisitor(Prolog prolog) {
this.prolog = prolog;
this.blockLevel = 0;
variableBlockLevel = new HashMap<>();
this.variableBlockLevel = new HashMap<>();
}

protected DescendentSequentialProperties defaultAction(Node node, DescendentSequentialProperties argument) {
Expand Down
27 changes: 27 additions & 0 deletions src/main/java/org/rumbledb/context/BuiltinFunctionCatalogue.java
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@
import org.rumbledb.runtime.functions.datetime.components.TimezoneFromTimeFunctionIterator;
import org.rumbledb.runtime.functions.datetime.components.YearFromDateFunctionIterator;
import org.rumbledb.runtime.functions.datetime.components.YearFromDateTimeFunctionIterator;
import org.rumbledb.runtime.functions.delta_lake.CreateDeltaLakeTableFunctionIterator;
import org.rumbledb.runtime.functions.delta_lake.DeleteDeltaLakeTableFunctionIterator;
import org.rumbledb.runtime.functions.durations.components.DaysFromDurationFunctionIterator;
import org.rumbledb.runtime.functions.durations.components.HoursFromDurationFunctionIterator;
import org.rumbledb.runtime.functions.durations.components.ImplicitTimezoneIterator;
Expand Down Expand Up @@ -2688,6 +2690,28 @@ private static BuiltinFunction createBuiltinFunction(
BuiltinFunction.BuiltinFunctionExecutionMode.LOCAL
);

/**
* function that creates a delta lake table at a given path location
*/
static final BuiltinFunction create_delta_lake_table = createBuiltinFunction(
new Name(Name.JN_NS, "jn", "create-delta-lake-table"),
"string",
"boolean",
CreateDeltaLakeTableFunctionIterator.class,
BuiltinFunction.BuiltinFunctionExecutionMode.LOCAL
);

/**
* function that deletes a delta lake table at a given path location
*/
static final BuiltinFunction delete_delta_lake_table = createBuiltinFunction(
new Name(Name.JN_NS, "jn", "delete-delta-lake-table"),
"string",
"boolean",
DeleteDeltaLakeTableFunctionIterator.class,
BuiltinFunction.BuiltinFunctionExecutionMode.LOCAL
);

static {
builtinFunctions = new HashMap<>();

Expand Down Expand Up @@ -2718,6 +2742,9 @@ private static BuiltinFunction createBuiltinFunction(
builtinFunctions.put(avro_file2.getIdentifier(), avro_file2);
builtinFunctions.put(parse_json.getIdentifier(), parse_json);

builtinFunctions.put(create_delta_lake_table.getIdentifier(), create_delta_lake_table);
builtinFunctions.put(delete_delta_lake_table.getIdentifier(), delete_delta_lake_table);

builtinFunctions.put(count.getIdentifier(), count);
builtinFunctions.put(boolean_function.getIdentifier(), boolean_function);
builtinFunctions.put(not_function.getIdentifier(), not_function);
Expand Down
2 changes: 1 addition & 1 deletion src/main/java/org/rumbledb/context/GlobalVariables.java
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ public class GlobalVariables implements Serializable, KryoSerializable {
private Set<Name> globalVariables;

public GlobalVariables() {
globalVariables = new HashSet<>();
this.globalVariables = new HashSet<>();
}

public void addGlobalVariable(Name globalVariable) {
Expand Down
2 changes: 1 addition & 1 deletion src/main/java/org/rumbledb/context/InScopeVariable.java
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,6 @@ public void read(Kryo kryo, Input input) {
}

public boolean isAssignable() {
return isAssignable;
return this.isAssignable;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -166,12 +166,12 @@ public void serializeToJSONiq(StringBuffer sb, int indent) {

@Nullable
public List<Annotation> getAnnotations() {
return annotations;
return this.annotations;
}


public boolean isAssignable() {
return isAssignable;
return this.isAssignable;
}
}

Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ public StatementsAndOptionalExpr getBody() {

@Nullable
public List<Annotation> getAnnotations() {
return annotations;
return this.annotations;
}

@Override
Expand Down Expand Up @@ -252,7 +252,7 @@ public void serializeToJSONiq(StringBuffer sb, int indent) {
}

public boolean hasSequentialPropertyAnnotation() {
return hasSequentialPropertyAnnotation;
return this.hasSequentialPropertyAnnotation;
}

public void setHasExitStatement(boolean hasExitStatement) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,11 @@ public Annotation(Name annotationName, List<Expression> literals) {
}

public Name getAnnotationName() {
return annotationName;
return this.annotationName;
}

public List<Expression> getLiterals() {
return literals;
return this.literals;
}

public static boolean checkAssignable(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ public <T> T accept(AbstractNodeVisitor<T> visitor, T argument) {
@Override
public List<Node> getChildren() {
List<Node> result = new ArrayList<>();
blockStatements.forEach(statement -> {
this.blockStatements.forEach(statement -> {
if (statement != null) {
result.add(statement);
}
Expand All @@ -49,6 +49,6 @@ public void serializeToJSONiq(StringBuffer sb, int indent) {
}

public List<Statement> getBlockStatements() {
return blockStatements;
return this.blockStatements;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -74,14 +74,14 @@ public void serializeToJSONiq(StringBuffer sb, int indent) {
}

public Expression getTestCondition() {
return testCondition;
return this.testCondition;
}

public List<SwitchCaseStatement> getCases() {
return cases;
return this.cases;
}

public Statement getDefaultStatement() {
return defaultStatement;
return this.defaultStatement;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -23,18 +23,18 @@ public <T> T accept(AbstractNodeVisitor<T> visitor, T argument) {

@Override
public List<Node> getChildren() {
return new ArrayList<>(variables);
return new ArrayList<>(this.variables);
}

@Override
public void serializeToJSONiq(StringBuffer sb, int indent) {
indentIt(sb, indent);
for (VariableDeclStatement variableDeclStatement : variables) {
for (VariableDeclStatement variableDeclStatement : this.variables) {
variableDeclStatement.serializeToJSONiq(sb, 0);
}
}

public List<VariableDeclStatement> getVariables() {
return variables;
return this.variables;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -103,10 +103,10 @@ public Expression getVariableExpression() {
}

public List<Annotation> getAnnotations() {
return annotations;
return this.annotations;
}

public boolean isAssignable() {
return isAssignable;
return this.isAssignable;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ public void serializeToJSONiq(StringBuffer sb, int indent) {
}

public Expression getExitExpression() {
return exitExpression;
return this.exitExpression;
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,6 @@ public void serializeToJSONiq(StringBuffer sb, int indent) {
}

public ReturnStatementClause getReturnStatementClause() {
return returnStatementClause;
return this.returnStatementClause;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,6 @@ public void serializeToJSONiq(StringBuffer sb, int indent) {
}

public Statement getReturnStatement() {
return returnStatement;
return this.returnStatement;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@ public <T> T accept(AbstractNodeVisitor<T> visitor, T argument) {
@Override
public List<Node> getChildren() {
List<Node> result = new ArrayList<>();
result.add(testCondition);
result.add(statement);
result.add(this.testCondition);
result.add(this.statement);
return result;
}

Expand All @@ -43,10 +43,10 @@ public void serializeToJSONiq(StringBuffer sb, int indent) {
}

public Expression getTestCondition() {
return testCondition;
return this.testCondition;
}

public Statement getStatement() {
return statement;
return this.statement;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ protected void closeLocal() {

public void setNextResult() {
this.nextResult = null;
if (!encounteredExitStatement) {
if (!this.encounteredExitStatement) {
try {
if (this.userDefinedFunctionCallIterator.hasNext()) {
this.nextResult = this.userDefinedFunctionCallIterator.next();
Expand Down
Loading

0 comments on commit 1b2bae6

Please sign in to comment.