SparkDSLR and GpuKMeans not working in cluster mode

[a-agrz]

Hi !
I was testing the new merge of GPUDataset in GPUEnabler and got the following problems
I did my tests in a node that has 3 GPUS attached to it.

In Local mode, both examples SparkDSLR and GpuKMeans works fine

SparkDSLR in local mode:

GpuKMeans in local mode:

But I got this errors when executing SparkDSLR example in cluster mode,

and this errors when executing GpuKMeans example in cluster mode

PS: SparkGPULR (RDD version) works fine in sluster mode

So why datasets don't work in cluster mode? what I'm I missing here? is there some configurations that I need to do ??

Best regards

Aguerzaa

[josiahsams]

The exception is due to the fact that, the kernel file(*.ptx) file, which is part of the examples.jar file, is not part of the driver/executor classpath and getClass.getResource returns NULL. So to make it part of the classpath, add the following entries in the Spark's conf/spark-default.conf file,

 spark.executor.extraClassPath	  /home/joe/GPUEnabler/gpu-enabler/target/gpu-enabler_2.11-1.0.0.jar:/home/joe/GPUEnabler/examples/target/gpu-enabler-examples_2.11-1.0.0.jar
 spark.driver.extraClassPath	  /home/joe/GPUEnabler/gpu-enabler/target/gpu-enabler_2.11-1.0.0.jar:/home/joe/GPUEnabler/examples/target/gpu-enabler-examples_2.11-1.0.0.jar
 spark.jars			  /home/joe/GPUEnabler/gpu-enabler/target/gpu-enabler_2.11-1.0.0.jar

[a-agrz]

Thank YOU @josiahsams; It's working now, here are the results I found:

SparkDSLR in cluster mode:

GpuKMeans in cluster mode: