Sessions exceeding the limit in Hadoop

I have hadoop eco system which receives files as input and hadoop after processing loads into vertica database with 200 session limit.

Hadoop is creating vertica session for each input file from hadoop.

Now if the input files exceeds 200 limit then the sessions exceed 200 and causes 'sesssion exceed error'

is there any pooling concept which will help .


  • Options

     Why don't you join the files so you can avoid hifh concurency against Vertica. Vertica was not bild for nigh concurency stuff. Also this way you will avoid you TM to go crazy. 


  • Options

    Thanks a lot for the reply ...Yes, i have that suggestion for the Hadoop team, but they are more crazy then TM at the moment.

    Is there a way in vertica to accept more sessions or any other chance.

  • Options

     Adding more sessions wont solve nothing ! 

    What is your Resource pool Planned Concureny ? How many CPU do you have on your nodes (single node)?


    You can open more session and all of them will be queing !  


  • Options

    Adam Wrote a answered a similar question here:



    -is not Hadop related but it relates to the whole session and loading stuff

  • Options

    I have 12 node cluster with max concurrency 24 planned concurrency 10 and max session 200.

    We are using Vertica-hadoop native connector in map reduce which will create a copy statement.

    i believe it is creating a new session for evry file(say more that 200   files in total).

    i am planning to increase the session limit to 500.


  • Options

    Hi SRA,


      You said you have hadoop feeding data to you so you can load it into Vertica. I assume is aggregated data ! (less important this point:) ).

      Now i guess you have this data in a certain format and the files come with similar of names ? ! 


    Your load design should not create a load session per each finle that is generated.


    Is not a good approach to run copy"load" every time a file is generated by your hadoop ! A smart method would be to have your load process run every lets say 10 min and have him load all files present in the location where hadoop generates the files.



    COPY table_bla_bla FROM '/hadoop_path/file*.csv' ON node01, '/hadoop_path/file*.csv' ON node2, ''/hadoop_path/file*.csv'' ON node3, DELIMITER '|' direct ; 

    This way you run only one copy command. 

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file