PySpark Arrow Stream Serializer

[BryanCutler]

Enable UDF evaluation with Arrow using stream format to load as Pandas Series, modified PythonRDD to support this and maintain backwards compatibility.

[BryanCutler]

@icexelloss , here is what I had so far. Feel free to use what you like and let me know if you have any questions.

[icexelloss]

Thanks Bryan! This is quite a bit a change. I will take a look this week.

[BryanCutler]

Sure, no problem!

[icexelloss]

Bryan,

You mentioned a ~2.5x speed up comparing this and the original udf methods. How did you run the experiments? I am trying to reproduce your results.

[BryanCutler]

I was basically using the code below, just manually turning on/off Arrow by commenting a couple lines (I left a note in the code as to what needs to be commented out)

nrows = 1 << 24
df = spark.range(0, nrows, 1, 4).toDF("a").cache()
is_odd = udf(lambda n: n % 2, LongType())
odd = df.withColumn("is_odd", is_odd(col("a")))
flt_odd = odd.filter("is_odd == 1")
t = timeit.repeat(lambda: flt_odd.count(), repeat=10, number=1)
time_df = pd.Series(t)
print(time_df.describe())