I have bunch of .csv.lz4 files in s3 that I want to copy into vertica table. Is it possible? If not, what would be the best file format for this purpose? Thanks
We did some research on LZ4 compression in the past. I can update the Jira to let the engineering team know that there is interest in it in the field.
For now though, I think the only way to load LZ4 compressed files is to cat them into a COPY command.
[[email protected] ~]$ cat test.txt
[[email protected] ~]$ lz4 test.txt > test.lz4
[[email protected] ~]$ lz4cat test.lz4
[[email protected] ~]$ lz4cat test.lz4 | vsql -c "COPY test FROM STDIN;"
[[email protected] ~]$ vsql -c "SELECT * FROM test;"
You can write your own lz4 filter.
From what I remember, it took around 2 days of work, around 200 lines of c++.
Can't find what you're looking for? Search the Vertica Documentation, Knowledge Base, or Blog for more information.