etl_config.json not loaded in EMR

[junjchen]

First of all, thanks for the great work! I am new to spark and this repo has really helped me getting started.

I am trying to get my etl job running on aws EMR in cluster mode, but got hit with an issue where the pyspark program failed to load up my config.json passed in through --files s3://path.... I googled a bit and figured this might be an issue with how --files only copies files to executor nodes, not driver node, so the code here https://github.com/AlexIoannides/pyspark-example-project/blob/master/dependencies/spark.py#L93 finds nothing when iterating through the spark root dir. (for reference, I based my guess on this SO post https://stackoverflow.com/questions/47187533/files-option-in-pyspark-not-working).

Could this be the issue and is there a workaround for it? Or am I not making sense at all?

[moongyu123]

In my case
I maked config_json.py in folder 'dependencies'
It returns JSONString

and spark.py's code like this
from dependencies import config
config_json_string = config.get_config_json()
config_dict = json.loads(config_json_string)