Skip to content
This repository has been archived by the owner on Dec 20, 2018. It is now read-only.

Spark1.6 with databricks 2.0.1 : Exception in thread "main" java.lang.UnsupportedOperationException: This mix of union types is not supported (see README): ArrayBuffer(RECORD) at #290

Open
sagarsvp opened this issue Aug 6, 2018 · 0 comments

Comments

@sagarsvp
Copy link

sagarsvp commented Aug 6, 2018

For databricks 2.0.1 with spark 1.6, While Reading a Complex AVRO with below Code

**SparkConf conf = new SparkConf().setAppName("SearchAVRO").setMaster("local[*]");
JavaSparkContext sc = new JavaSparkContext(conf);      
SQLContext sqlContext = new SQLContext(sc);

// Creates a DataFrame from a specified file
DataFrame df = sqlContext.read().format("com.databricks.spark.avro").load(inputPath);**

We get an exception:

Exception in thread "main" java.lang.UnsupportedOperationException: This mix of union types is not supported (see README): ArrayBuffer(RECORD) at com.databricks.spark.avro.SchemaConverters$.toSqlType(SchemaConverters.scala:88) at com.databricks.spark.avro.SchemaConverters$.toSqlType(SchemaConverters.scala:63) at com.databricks.spark.avro.SchemaConverters$.toSqlType(SchemaConverters.scala:79) at com.databricks.spark.avro.SchemaConverters$$anonfun$1.apply(SchemaConverters.scala:56)

==========================================================
Similar code works properly for databricks 4.0.0 with Spark 2.2

Can you please add support for the same in databricks 2.0.1??

@sagarsvp sagarsvp changed the title Exception in thread "main" java.lang.UnsupportedOperationException: This mix of union types is not supported (see README): ArrayBuffer(RECORD) at Spark1.6 with databricks 2.0.1 : Exception in thread "main" java.lang.UnsupportedOperationException: This mix of union types is not supported (see README): ArrayBuffer(RECORD) at Aug 6, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant