Too many ROS containers loading data from HDFS to Vertica
I would like to load data from HDFS to Vertica. I'm testing with pig-connector. It works well when loading smaller size data, like 1G. But it would have "Too many ROS containers" problem when loading larger-size data, like 25G, to Vertica. The only config parameter I've changed is "pig.maxCombinedSplitSize". I modified it to reduce number of mappers.
I actually had the same ROS container problem when loading data from HDFS to Vertica using Sqoop Export.
Does anyone know how to fix this problem? And is there "direct" mode for pig-connector as "COPY", to load directly to ROS? Thank you!
0