redshift_to_pandas write to S3

[Gauravshah]

Expected Behaviour

it would be more performance efficient if redshift_to_pandas first writes to S3 and then reads it from there.

Actual Behaviour

redshift_to_pandas reads over wire directly connecting to redshift

Implications

If there is large dataset ( >100 million rows) being downloaded, redshift's one thread is occupied in serving this user.

[yaojiach]

It sounds like a separate concern. If you are using Python >= 3.6 you can try https://github.com/yaojiach/red-panda/blob/master/red_panda/red_panda.py#L664

[PabTorre]

the syntax from the redshift side shouldn't be too hard to integrate.
https://docs.aws.amazon.com/redshift/latest/dg/t_Unloading_tables.html

[agawronski]

Will look into this.