Please take this survey to help us learn more about how you use third party tools. Your input is greatly appreciated!

COPY STREAMERS

Hi,

We have JavaScript programs (WORKERS) that get messages from Message-Queue (RabbitMQ) and load the information into vertica (using NODEJS as our platform).

The Idea is to open connection and using COPY STREAMING (STDIN)  to constatntly loading data into Vertica.

·         We intend not closing the connection, and stream data all the time, when the rate varriant between 10-20 rows per minute to 10K-30K per minute.

·         Also, we’ll have couple of Streamers (4-5)

 

My questions:

·         Is it O.K. to use streamers without closing the connections?

·         What impact (resource wise) those connections have on the DB?

·         Any known timeouts that I should be worried from?

·         Is it possible that overtime the connections will become heavy?

·         If this method is wrong what would you suggest?

 

Thank You.

Gil P.

Cell      : +972 54 5597107

Comments

  • Hi!

    I have implemented same functionality with Python + vsql wrappers, but it is't deployed on production environment, so I can write conclusions from TEST environment only :(
    Is it O.K. to use streamers without closing the connections?
    I did it in little different way: separate server listens on port/s and client sends data to server. Server opens a connection to database only when client opens a connection and database connection closed when some condition is achieved(rows num or time limit for opened connection).
    What impact (resource wise) those connections have on the DB?
    Similar as a single vsql connection.
    Any known timeouts that I should be worried from?
    No with my implementation.
    Is it possible that overtime the connections will become heavy?
    Of cause, you are developer so you are the boss and it depends on your implementation.
    If this method is wrong what would you suggest?
    Can't say about NodeJS. My main pitfall was - its multiply simultaneous connections and data streams to different tables. For example condition isn't achieved(num of rows and timeout) to close COPY STREAM.

    Sorry for poor English, but I can explain on Hebrew ;)
    Regards
  • Thank You for your replay, actually i'm in the process of testing it right now. 
    from your implementation it seems o.k. 
    Thank Again.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file

Can't find what you're looking for? Search the Vertica Documentation, Knowledge Base, or Blog for more information.