Failed to add /home/vcap/app/ to Spark environment

[robachmann]

Hi Adrian

I'm trying to store tracing-information of Zipkin in an Elasticsearch database. Zipkin-Server and ES are running in Cloud Foundry. I've already figured out that I need an additional app running zipkin-dependencies in order to see any dependencies between applications.

Unfortunately, the required environment-variables are not read but I guess that's already part of #45. Therefore, I set the connection details manually during deployment:

env:
  STORAGE_TYPE: elasticsearch
  ES_HTTP_LOGGING: BASIC
  ES_HOSTS: https://urlofelasticsearch.com
  ES_USERNAME: username
  ES_PASSWORD: password
  ES_NODES_WAN_ONLY: true

When I now push the application, I see the following exception during startup:

2018-06-12T15:56:13.65+0200 [APP/PROC/WEB/0] OUT JVM Memory Configuration: -Xmx1406944K -Xss1M -XX:ReservedCodeCacheSize=240M -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=178207K
2018-06-12T15:56:14.02+0200 [APP/PROC/WEB/0] ERR 18/06/12 13:56:14 INFO ElasticsearchDependenciesJob: Processing spans from zipkin-2018-06-12/span
2018-06-12T15:56:14.25+0200 [APP/PROC/WEB/0] ERR 18/06/12 13:56:14 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-06-12T15:56:14.84+0200 [APP/PROC/WEB/0] ERR 18/06/12 13:56:14 ERROR SparkContext: Failed to add /home/vcap/app/ to Spark environment
2018-06-12T15:56:14.84+0200 [APP/PROC/WEB/0] ERR java.lang.IllegalArgumentException: Directory /home/vcap/app is not allowed for addJar
2018-06-12T15:56:14.84+0200 [APP/PROC/WEB/0] ERR 	at org.apache.spark.SparkContext.addJarFile$1(SparkContext.scala:1817)
2018-06-12T15:56:14.84+0200 [APP/PROC/WEB/0] ERR 	at org.apache.spark.SparkContext.addJar(SparkContext.scala:1840)
2018-06-12T15:56:14.84+0200 [APP/PROC/WEB/0] ERR 	at org.apache.spark.SparkContext$$anonfun$12.apply(SparkContext.scala:466)
2018-06-12T15:56:14.84+0200 [APP/PROC/WEB/0] ERR 	at org.apache.spark.SparkContext$$anonfun$12.apply(SparkContext.scala:466)
2018-06-12T15:56:14.84+0200 [APP/PROC/WEB/0] ERR 	at scala.collection.immutable.List.foreach(List.scala:381)
2018-06-12T15:56:14.84+0200 [APP/PROC/WEB/0] ERR 	at org.apache.spark.SparkContext.<init>(SparkContext.scala:466)
2018-06-12T15:56:14.84+0200 [APP/PROC/WEB/0] ERR 	at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
2018-06-12T15:56:14.84+0200 [APP/PROC/WEB/0] ERR 	at zipkin.dependencies.elasticsearch.ElasticsearchDependenciesJob.run(ElasticsearchDependenciesJob.java:179)
2018-06-12T15:56:14.84+0200 [APP/PROC/WEB/0] ERR 	at zipkin.dependencies.elasticsearch.ElasticsearchDependenciesJob.run(ElasticsearchDependenciesJob.java:165)
2018-06-12T15:56:14.84+0200 [APP/PROC/WEB/0] ERR 	at zipkin.dependencies.ZipkinDependenciesJob.main(ZipkinDependenciesJob.java:72)
2018-06-12T15:56:14.95+0200 [APP/PROC/WEB/0] ERR 18/06/12 13:56:14 WARN Java7Support: Unable to load JDK7 types (annotations, java.nio.file.Path): no Java7 support added
2018-06-12T15:56:16.22+0200 [APP/PROC/WEB/0] ERR 18/06/12 13:56:16 INFO ElasticsearchDependenciesJob: No spans found at zipkin-2018-06-12/span
2018-06-12T15:56:16.24+0200 [APP/PROC/WEB/0] ERR 18/06/12 13:56:16 INFO ElasticsearchDependenciesJob: Processing spans from zipkin:span-2018-06-12/span
2018-06-12T15:56:16.31+0200 [APP/PROC/WEB/0] ERR 18/06/12 13:56:16 ERROR SparkContext: Failed to add /home/vcap/app/ to Spark environment
2018-06-12T15:56:16.31+0200 [APP/PROC/WEB/0] ERR java.lang.IllegalArgumentException: Directory /home/vcap/app is not allowed for addJar
2018-06-12T15:56:16.31+0200 [APP/PROC/WEB/0] ERR 	at org.apache.spark.SparkContext.addJarFile$1(SparkContext.scala:1817)
2018-06-12T15:56:16.31+0200 [APP/PROC/WEB/0] ERR 	at org.apache.spark.SparkContext.addJar(SparkContext.scala:1840)
2018-06-12T15:56:16.31+0200 [APP/PROC/WEB/0] ERR 	at org.apache.spark.SparkContext$$anonfun$12.apply(SparkContext.scala:466)
2018-06-12T15:56:16.31+0200 [APP/PROC/WEB/0] ERR 	at org.apache.spark.SparkContext$$anonfun$12.apply(SparkContext.scala:466)
2018-06-12T15:56:16.31+0200 [APP/PROC/WEB/0] ERR 	at scala.collection.immutable.List.foreach(List.scala:381)
2018-06-12T15:56:16.31+0200 [APP/PROC/WEB/0] ERR 	at org.apache.spark.SparkContext.<init>(SparkContext.scala:466)
2018-06-12T15:56:16.31+0200 [APP/PROC/WEB/0] ERR 	at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
2018-06-12T15:56:16.31+0200 [APP/PROC/WEB/0] ERR 	at zipkin.dependencies.elasticsearch.ElasticsearchDependenciesJob.run(ElasticsearchDependenciesJob.java:179)
2018-06-12T15:56:16.31+0200 [APP/PROC/WEB/0] ERR 	at zipkin.dependencies.elasticsearch.ElasticsearchDependenciesJob.run(ElasticsearchDependenciesJob.java:170)
2018-06-12T15:56:16.31+0200 [APP/PROC/WEB/0] ERR 	at zipkin.dependencies.ZipkinDependenciesJob.main(ZipkinDependenciesJob.java:72)
2018-06-12T15:56:17.76+0200 [APP/PROC/WEB/0] ERR [Stage 0:>                                                          (0 + 0) / 5]
2018-06-12T15:56:17.76+0200 [APP/PROC/WEB/0] ERR                                                                                 
2018-06-12T15:56:17.76+0200 [APP/PROC/WEB/0] ERR 18/06/12 13:56:17 INFO ElasticsearchDependenciesJob: Saving dependency links to zipkin:dependency-2018-06-12/dependency

In case this has anything to do with Java versions (as I read Java 7), note that the app was pushed with the Java Buildpack of Cloud Foundry having Open Jdk JRE 1.8.0_172 shipped with.

Could you please point me into a direction how I can solve this issue? As a work-around I simply use Zipkin's InMemory-DB but this will cause Zipkin periodically to crash when all memory is used up
¯\(ツ)/¯

Thanks and best regards

EDIT: Looks like there's something like a ring buffer in place taking care of memory-usage.

[codefromthecrypt]

#107 should do it