Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deal with timestamp #1

Open
maykiwo opened this issue Dec 7, 2017 · 1 comment
Open

Deal with timestamp #1

maykiwo opened this issue Dec 7, 2017 · 1 comment

Comments

@maykiwo
Copy link

maykiwo commented Dec 7, 2017

Thanks for your work.
It seems have a problem with timestamp or i've not used correctly your jar

for testing i have got this file
hello|1|2003-03-05 18:09:28.0
world|2|2006-03-05 17:09:28.0

Use the jar like this :
java -jar csv-to-orc.jar -i hello.csv -o hello.orc -s 'structbonjour:string,num:int,create_date:timestamp' -sep '|' -skipcount 0

Dump the orc file :
$ hive --orcfiledump -d -t /user/hdp/test/hello.orc
2017-12-07 09:49:58,366 INFO [main] orc.RecordReaderImpl: Reader schema not provided -- using file schema structbonjour:string,num:int,create_date:timestamp
{"bonjour":"hello","num":1,"create_date":"1970-01-01 01:00:00"}
{"bonjour":"world","num":2,"create_date":"1970-01-01 01:00:00"}

the timstamp is set to linux time. it is the behavior of your jar ?
I have excepted to retreive the timestamp of csv file.

regards

@egor-ryashin
Copy link

Added #2

cat 1.csv
time,tag,number
2018-07-09T21:00:00Z,"a",2
2018-07-09T21:00:01Z,"b",3
java -jar target/csv-to-orc-1.0-SNAPSHOT-jar-with-dependencies.jar -i 1.csv -o 1.orc -s 'struct<time:timestamp,tag:string,number:int>' -sep ',' -skipcount 1
bin/hive --orcfiledump -d -t 1.orc
Processing data file 1.orc [length: 411]
{"time":"2018-07-10 00:00:00.0","tag":"a","number":2}
{"time":"2018-07-10 00:00:01.0","tag":"b","number":3}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants