I have bunch of .csv.lz4 files in s3 that I want to copy into vertica table. Is it possible? If not, what would be the best file format for this purpose? Thanks
We did some research on LZ4 compression in the past. I can update the Jira to let the engineering team know that there is interest in it in the field.
For now though, I think the only way to load LZ4 compressed files is to cat them into a COPY command.
[dbadmin@vertica ~]$ cat test.txt
[dbadmin@vertica ~]$ lz4 test.txt > test.lz4
[dbadmin@vertica ~]$ lz4cat test.lz4
[dbadmin@vertica ~]$ lz4cat test.lz4 | vsql -c "COPY test FROM STDIN;"
[dbadmin@vertica ~]$ vsql -c "SELECT * FROM test;"
You can write your own lz4 filter.
From what I remember, it took around 2 days of work, around 200 lines of c++.