Timestamp ('9999-12-31 23:59:59') fails to read

[JeyhunBabirzade]

Hi @cryeo ,

Thanks for the great library. I had only one issue so far, and it is as per below:

If you run below command in Jupyter it will fail to execute:

%%sparksql
select cast('9999-12-31 23:59:59' as timestamp)

The same command does not fail using spark.sql() or if you run it with a timestamp less than year 3000, e.g. ('2999-12-31 23:59:59')
spark.sql("select cast('9999-12-31 23:59:59' as timestamp)").show()
The database I am using records 'effective to' dates as '9999-12-31 23:59:59'. That is why, when I do a select * from table it fails.
But the same command executes with spark.sql() command.

Cheers,
Jey

[Wh1isper]

@JeyhunBabirzade
Hi, I have developed a similar module with support for the latest spark server-client mode, maybe you can try it out：

https://github.com/Wh1isper/sparglim#sql-magic

Install with:

pip install sparglim["magic"]

%load_ext sparglim.sql

%sql select cast('9999-12-31 23:59:59' as timestamp)

Output:

CAST(9999-12-31 23:59:59 AS TIMESTAMP)
--
9999-12-31 23:59:59