Error running on Spark 1.5.2 (on Databricks)

[zakipatel]

On Spark 2.0, I am able to successfully install the python library (%sh pip install spark-df-profiling) , run the import command (import spark_df_profiling) and (report = spark_df_profiling.ProfileReport(df)

However, this does not work on Spark 1.5.2. As per the documentation on Github, it says that Spark 1.5 + is supported and is compatible with 2.0. Is there something that I need to do in addition to the pip install and setting up the Six===1.10.0 library ?

See screenshot below - this works just fine in Spark 2.0 (on Databricks, but get the error in 1.5.2)

[julioasotodv]

Hi,

This may happen if you don't "reload" the notebook after installing the library.

Moreover, the recommended way to install Python libraries in Databricks is by attaching them rather than using pip with the %sh command.

Let me know if that helped you.

[SaurabhCK]

use dbutils
dbutils.library.installPyPI("spark-df-profiling","1.1.2")
this should work