Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exception when running crossmatch with Spark 3.0.0 version #27

Closed
stargaser opened this issue Oct 16, 2020 · 2 comments
Closed

Exception when running crossmatch with Spark 3.0.0 version #27

stargaser opened this issue Oct 16, 2020 · 2 comments

Comments

@stargaser
Copy link

I've installed at IPAC the automated build version that @stevenstetzler mentioned in #26:

Yeah, the automated builds seem to work still. I recently used it to make a new Spark 3 release with an updated version of Scala (I was running into similar issues as @stargaser using Spark 3 with AXS until I updated Scala).

When I run crossmatch, an exception is raised that I think is the same as the one I encountered when I built axs and axs-spark myself from the instructions in #20. Does this mean I need to upgrade Scala, and if so, what version should I have?

The informational part of the exception:

Py4JJavaError: An error occurred while calling o234.crossmatch.
: java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;
	at org.dirac.axs.FrameFunctions$.crossmatch(FrameFunctions.scala:42)
	at org.dirac.axs.FrameFunctions.crossmatch(FrameFunctions.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:745)
@stargaser
Copy link
Author

I think I found my problem. I had spark.jars in the $SPARK_HOME/conf/spark-defaults.conf file pointing to the wrong jar for AxsUtilities. I can run a crossmatch with a local cluster now.

@stargaser
Copy link
Author

Now that I have fixed spark.jars, I can crossmatch. I see that the upgraded Scala is in one of the jar files. I am closing this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant