Adopt pyarrow and using it for pure python side hdfs handling

[ninjapapa]

pyarrow has a solid hdfs interface, we can easily use it to replace our current Scala HDFS interface.
Some relevant links:

• http://wesmckinney.com/blog/python-hdfs-interfaces/
• https://arrow.apache.org/docs/python/

An obvious side benefit is that we can turn on
spark.sql.execution.arrow.enabled=true
It significantly improved the toPanda performance of Spark DF.

https://arrow.apache.org/blog/2017/07/26/spark-arrow/

[AliTajeldin]

I think arrow integration is only in latest versions of spark. Should we make that support optional to maintain backward compatibility with older versions of spark. That is, fall back to scala hdfs or some other hdfs library (the toPandas will fall back automatically).

[ninjapapa]

The hdfs part does not need spark integration. So we can add pyarrow to requirement.txt and use it purely for hdfs api for now. Turn on spark.sql.execution.arrow.enabled or not is a cluster config choice. SMV does not need to change anything.

[AliTajeldin]

makes sense

[ninjapapa]

Adding pyarrow python package is easy, however its hdfs interface used libhdfs.so or libhdfs3.so. The first one should be part of standard hadoop package, and exists on real cluster environment, and the second could be installed through condo, as in the 2nd link of the main message shows.

However, when we build SMV, none of those 2 libraries exist. To access HDFS fully from python, we may need to use another library, such as
https://hdfscli.readthedocs.io/en/latest/

Or for better performance we may do:

try:
    fs = pyarrow.hdfs.connect(...)
except:
    fs = hdfs.TokenClient(...)

For short term, will still use the SmvHDFS interface on Scala side.

Defer this issue.