[VJDBC](5065) ERROR: Too many ROS containers exist for the following projections
hi
plz help me ...
i don't know how i solve this problem..
............................................................................................................................................................
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : Because of an error, this step can't continue:
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - ERROR (version 6.0.1.0-386, build 1 from 2015-12-03 11.37.25 by buildguy) : org.pentaho.di.core.exception.KettleException:
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - Error batch inserting rows into table [PI_LAP_RSLT].
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - Errors encountered (first 10):
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 -
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 -
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - Error updating batch
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - [Vertica][VJDBC](5065) ERROR: Too many ROS containers exist for the following projections:
CRDW.PI_LAP_RSLT_super (limit = 48128, ROS files = 48128, DV files = 0, new files = 47)
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 -
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 -
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - at org.pentaho.di.trans.steps.tableoutput.TableOutput.writeToTable(TableOutput.java:342)
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - at org.pentaho.di.trans.steps.tableoutput.TableOutput.processRow(TableOutput.java:118)
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - at java.lang.Thread.run(Thread.java:745)
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - Caused by: org.pentaho.di.core.exception.KettleDatabaseBatchException:
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - Error updating batch
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - [Vertica][VJDBC](5065) ERROR: Too many ROS containers exist for the following projections:
CRDW.PI_LAP_RSLT_super (limit = 48128, ROS files = 48128, DV files = 0, new files = 47)
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 -
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - at org.pentaho.di.core.database.Database.createKettleDatabaseBatchException(Database.java:1372)
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - at org.pentaho.di.trans.steps.tableoutput.TableOutput.writeToTable(TableOutput.java:289)
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - ... 3 more
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - Caused by: java.sql.SQLTransientException: [Vertica][VJDBC](5065) ERROR: Too many ROS containers exist for the following projections:
CRDW.PI_LAP_RSLT_super (limit = 48128, ROS files = 48128, DV files = 0, new files = 47)
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - at com.vertica.util.ServerErrorData.buildException(Unknown Source)
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - at com.vertica.dataengine.VQueryExecutor.readCopyStartResponse(Unknown Source)
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - at com.vertica.dataengine.VQueryExecutor.handleExecuteResponse(Unknown Source)
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - at com.vertica.dataengine.VQueryExecutor.execute(Unknown Source)
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - at com.vertica.jdbc.common.SPreparedStatement.executeBatch(Unknown Source)
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - at com.vertica.jdbc.VerticaJdbc4PreparedStatementImpl.executeBatch(Unknown Source)
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - at org.pentaho.di.trans.steps.tableoutput.TableOutput.writeToTable(TableOutput.java:285)
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - ... 3 more
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - Caused by: com.vertica.support.exceptions.TransientException: [Vertica][VJDBC](5065) ERROR: Too many ROS containers exist for the following projections:
CRDW.PI_LAP_RSLT_super (limit = 48128, ROS files = 48128, DV files = 0, new files = 47)
2016/03/15 21:20:22 - Table_Output_PI_LAP_RSLT.0 - ... 10 more
.....................................................................................................................................................................
what mean 'too many ros containers exist for the following projections'
i can't solve ....help me ...
Comments
Hi ,
Few point to check :
Sharing your vertica.log will also be very useful for understing your root problem
I hope you will find it useful
Thanks
Hi,
In general there is a value set to limit number of containers per projection. If small records are loading into db or if mergeout interval is too large, you face the subject error. Please identify the projections for which you are facing this error and perform a mergeout using following command
select DO_TM_TASK ('mergeout', '<schema.projection-name>');
GO through the following url to know more about tuple mover configuration parameters and how to tune them
https://my.vertica.com/docs/7.1.x/HTML/index.htm#Authoring/AdministratorsGuide/TupleMover/TuningTheTupleMover.htm%3FTocPath%3DAdministrator's%2520Guide%7CManaging%2520the%2520Database%7CManaging%2520Tuple%2520Mover%2520Operations%7C_____2
Sruthi
I hope this:
Table_Output_PI_LAP_RSLT.0 - [Vertica][VJDBC](5065) ERROR: Too many ROS containers exist for the following projections:
CRDW.PI_LAP_RSLT_super (limit = 48128, ROS files = 48128, DV files = 0, new files = 47)
Is not indicating that there are 48,128 ROS containers for that projection. Because that is insanely high. Maybe that's a cluster wide total. Even still, that number is really, really high.
A ROS Container is a logical grouping of files. In an ideal situation, a table might exist as a single ROS container. If the table is very large, there might be a dozen or more. But usually not hundreds in most cases. But there are things which can legitimately increase the number of ROS containers:
- turning on local segmentation - it's a way of further subdividing ROS containers, which creates a lot more
- a very high ActivePartitionCount value - which tells the internal MergeOut process to essentially ignore them which allows the number to grow very high
- partitioning - a way of logically dividing your table up by dates to provide additional performance and flexibility. If you have 365 days of data, and you partitioned by day, you'd have a minimum of 365 ROS containers for that table.
The system imposes a limit on the total number of ROS containers because an excessive amount can contribute to problems, and performance issues. Most people do not have a great reason to exceed this value.
But, if you're running into the error, you're either doing something wrong, or something is misconfigured:
- You're loading a huge file without a DIRECT keyword. This will cause it to load to memory, and then spill once it reaches a threshold.
- You're loading lots of little bitty files
- You're doing individual inserts with COMMITs after every one.
- You've turned off MergeOut, or run it too infrequently
- MoveOut runs too aggressively
Pentaho might be doing a bunch of insert statements, in which case you should find a way to do COPY instead. I know Talend has that capability, but I'm not sure about Pentaho. If that's not the case, can you post the results of this query here so we can further diagnose?
select * from configuration_parameters where current_value != default_value ;