Please take this survey to help us learn more about how you use third party tools. Your input is greatly appreciated!

can COPY from csv.lz4 files?

I have bunch of .csv.lz4 files in s3 that I want to copy into vertica table. Is it possible? If not, what would be the best file format for this purpose? Thanks

Answers

  • Jim_KnicelyJim_Knicely Administrator

    We did some research on LZ4 compression in the past. I can update the Jira to let the engineering team know that there is interest in it in the field.

    For now though, I think the only way to load LZ4 compressed files is to cat them into a COPY command.

    Like this:

    [[email protected] ~]$ cat test.txt
    1
    2
    3
    
    [[email protected] ~]$ lz4 test.txt > test.lz4
    
    [[email protected] ~]$ lz4cat test.lz4
    1
    2
    3
    
    [[email protected] ~]$ lz4cat test.lz4 | vsql -c "COPY test FROM STDIN;"
    
    [[email protected] ~]$ vsql -c "SELECT * FROM test;"
     c
    ---
     1
     2
     3
    (3 rows)
    
  • You can write your own lz4 filter.
    From what I remember, it took around 2 days of work, around 200 lines of c++.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file

Can't find what you're looking for? Search the Vertica Documentation, Knowledge Base, or Blog for more information.