can COPY from csv.lz4 files?

BHIFP

I have bunch of .csv.lz4 files in s3 that I want to copy into vertica table. Is it possible? If not, what would be the best file format for this purpose? Thanks

Jim_Knicely

We did some research on LZ4 compression in the past. I can update the Jira to let the engineering team know that there is interest in it in the field.

For now though, I think the only way to load LZ4 compressed files is to cat them into a COPY command.

Like this:

[dbadmin@vertica ~]$ cat test.txt
1
2
3

[dbadmin@vertica ~]$ lz4 test.txt > test.lz4

[dbadmin@vertica ~]$ lz4cat test.lz4
1
2
3

[dbadmin@vertica ~]$ lz4cat test.lz4 | vsql -c "COPY test FROM STDIN;"

[dbadmin@vertica ~]$ vsql -c "SELECT * FROM test;"
 c
---
 1
 2
 3
(3 rows)

Sergey_Cherepan_1

You can write your own lz4 filter.
From what I remember, it took around 2 days of work, around 200 lines of c++.