COPY STREAMERS

Gil_Peretz

Hi,

We have JavaScript programs (WORKERS) that get messages from Message-Queue (RabbitMQ) and load the information into vertica (using NODEJS as our platform).

The Idea is to open connection and using COPY STREAMING (STDIN)  to constatntly loading data into Vertica.

·         We intend not closing the connection, and stream data all the time, when the rate varriant between 10-20 rows per minute to 10K-30K per minute.

·         Also, we’ll have couple of Streamers (4-5)

 

My questions:

·         Is it O.K. to use streamers without closing the connections?

·         What impact (resource wise) those connections have on the DB?

·         Any known timeouts that I should be worried from?

·         Is it possible that overtime the connections will become heavy?

·         If this method is wrong what would you suggest?

 

Thank You.

Gil P.

Cell      : +972 54 5597107

id10t

Hi!

I have implemented same functionality with Python + vsql wrappers, but it is't deployed on production environment, so I can write conclusions from TEST environment only

Is it O.K. to use streamers without closing the connections?

I did it in little different way: separate server listens on port/s and client sends data to server. Server opens a connection to database only when client opens a connection and database connection closed when some condition is achieved(rows num or time limit for opened connection).

What impact (resource wise) those connections have on the DB?

Similar as a single vsql connection.

Any known timeouts that I should be worried from?

No with my implementation.

Is it possible that overtime the connections will become heavy?

Of cause, you are developer so you are the boss and it depends on your implementation.

If this method is wrong what would you suggest?

Can't say about NodeJS. My main pitfall was - its multiply simultaneous connections and data streams to different tables. For example condition isn't achieved(num of rows and timeout) to close COPY STREAM.

Sorry for poor English, but I can explain on Hebrew
Regards

Gil_Peretz

Thank You for your replay, actually i'm in the process of testing it right now. 
from your implementation it seems o.k. 
Thank Again.