Node frequent went down when COPY large ammount of data

Vaibhav_Gupta · July 2014

When we COPY the data one of the node went down in cluster. Below the logs in vertica.log

2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7ed8 due to loss of 450359964152211262014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7ed9 due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7eda due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7edb due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7edc due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7ede due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7edf due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7ee2 due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7ee4 due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7ee5 due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7ee6 due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7ee9 due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7eea due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7ef1 due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7efa due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7f04 due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [Comms] <INFO> nodeSetNotifier: node v_vabs_node0002 left the cluster
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [Recover] <INFO> Node left cluster, reassessing k-safety...
2014-07-29 01:41:06.433 Spread Client:0x79327a0 [Recover] <INFO> Checking Deps:Down bits: 0010 Deps:
0011 - cnt: 676
0110 - cnt: 676
1001 - cnt: 676
1100 - cnt: 676
1111 - cnt: 1399
2014-07-29 01:41:06.433 DistCall Dispatch:0x7f1d1c018860-b0000000886cf1 [Txn] <INFO> Rollback Txn: b0000000886cf1 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.433 DistCall Dispatch:0x7f1d1c0169d0-b0000000886cf2 [Txn] <INFO> Rollback Txn: b0000000886cf2 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.433 DistCall Dispatch:0x7f1d1c017830-b0000000886cf4 [Txn] <INFO> Rollback Txn: b0000000886cf4 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.434 DistCall Dispatch:0x7f1d1c0045a0-b0000000886cdb [Txn] <INFO> Rollback Txn: b0000000886cdb 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.434 DistCall Dispatch:0x7f1d1c006ba0-b0000000886cea [Txn] <INFO> Rollback Txn: b0000000886cea 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.435 DistCall Dispatch:0x7f1d1c005cd0-b0000000886ce9 [Txn] <INFO> Rollback Txn: b0000000886ce9 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.435 DistCall Dispatch:0x7f1d1c017f50-b0000000886ceb [Txn] <INFO> Rollback Txn: b0000000886ceb 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.436 DistCall Dispatch:0x7f1d1c004a70-b0000000886ce5 [Txn] <INFO> Rollback Txn: b0000000886ce5 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.436 DistCall Dispatch:0x7f1d1c05dde0-b0000000886cda [Txn] <INFO> Rollback Txn: b0000000886cda 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.437 DistCall Dispatch:0x7f1d1c00c640-b0000000886cf0 [Txn] <INFO> Rollback Txn: b0000000886cf0 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.437 DistCall Dispatch:0x7f1d1c00cea0-b0000000886cf5 [Txn] <INFO> Rollback Txn: b0000000886cf5 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.437 DistCall Dispatch:0x7f1d1c027b50-b0000000886d14 [Txn] <INFO> Rollback Txn: b0000000886d14 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.438 DistCall Dispatch:0x7f1d1c02abe0-b0000000886d15 [Txn] <INFO> Rollback Txn: b0000000886d15 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.438 DistCall Dispatch:0x7f1d1c06be80-b0000000886cdc [Txn] <INFO> Rollback Txn: b0000000886cdc 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.442 Spread Client:0x79327a0 <LOG> @v_vabs_node0001: 00000/3298: Event Posted: Event Code:3 Event Id:0 Event Severity: Critical [2] PostedTimestamp: 2014-07-29 01:41:06.433086 ExpirationTimestamp: 2082-08-16 03:55:13.433086 EventCodeDescription: Current Fault Tolerance at Critical Level ProblemDescription: Loss of node v_vabs_node0001 will cause shutdown to occur. K=1 total number of nodes=4 DatabaseName: vabs Hostname: bigdata011.vabs.com
2014-07-29 01:41:06.444 Spread Client:0x79327a0 [Comms] <INFO> nodeSetNotifier: node v_vabs_node0003 left the cluster
2014-07-29 01:41:06.444 Spread Client:0x79327a0 [Recover] <INFO> Node left cluster, reassessing k-safety...
2014-07-29 01:41:06.444 Spread Client:0x79327a0 [Recover] <INFO> Cluster partitioned: 4 total nodes, 2 up nodes, 2 down nodes
2014-07-29 01:41:06.444 Spread Client:0x79327a0 [Recover] <INFO> Setting node v_vabs_node0001 to UNSAFE
2014-07-29 01:41:06.445 Init Session:0x7f1cb4013330-a000000047d0c6 [Dist] <WARNING> Node v_vabs_node0002 was in plan recipents list at plan start but is now DOWN
2014-07-29 01:41:06.447 Spread Client:0x79327a0 <LOG> @v_vabs_node0001: 00000/3298: Event Posted: Event Code:6 Event Id:5 Event Severity: Informational [6] PostedTimestamp: 2014-07-29 01:41:06.447401 ExpirationTimestamp: 2082-08-16 03:55:13.447401 EventCodeDescription: Node State Change ProblemDescription: Changing node v_vabs_node0001 startup state to UNSAFE DatabaseName: vabs Hostname: bigdata011.vabs.com
2014-07-29 01:41:06.448 AnalyzeRowCount:0x7f1cac012ef0 <WARNING> @v_vabs_node0001: 01000/4539: Received no response from v_vabs_node0002 in GetRowCountMessage
2014-07-29 01:41:06.449 Spread Client:0x79327a0 <LOG> @v_vabs_node0001: 00000/3293: Event Cleared: Event Code:6 Event Id:6 Event Severity: Informational [6] PostedTimestamp: 2014-07-29 01:41:06.447537 ExpirationTimestamp: 2014-07-29 01:41:06.447537 EventCodeDescription: Node State Change ProblemDescription: Changing node v_vabs_node0001 leaving startup state UP DatabaseName: vabs Hostname: bigdata011.vabs.com
2014-07-29 01:41:06.449 Spread Client:0x79327a0 [Recover] <INFO> Changing node v_vabs_node0001 startup state from UP to UNSAFE
2014-07-29 01:41:06.449 Spread Client:0x79327a0 <LOG> @v_vabs_node0001: 00000/3298: Event Posted: Event Code:2 Event Id:0 Event Severity: Emergency [0] PostedTimestamp: 2014-07-29 01:41:06.44976 ExpirationTimestamp: 2014-07-29 01:51:06.44976 EventCodeDescription: Loss Of K Safety ProblemDescription: System is not K-safe: K=1 total number of nodes=4 DatabaseName: vabs Hostname: bigdata011.vabs.com
2014-07-29 01:41:06.452 AnalyzeRowCount:0x7f1cac012ef0 <WARNING> @v_vabs_node0001: 01000/4539: Received no response from v_vabs_node0003 in GetRowCountMessage
2014-07-29 01:41:06.454 AnalyzeRowCount:0x7f1cac012ef0 <WARNING> @v_vabs_node0001: 01000/4539: Received no response from v_vabs_node0005 in GetRowCountMessage
2014-07-29 01:41:06.464 DistCall Dispatch:0x7f1d1c06b910-c00000004dbcbe [Txn] <INFO> Rollback Txn: c00000004dbcbe 'SELECT BigData_LabelType_Identifier AS id, LabelType_Name AS name FROM AnalyticsNew.Label_Type'
2014-07-29 01:41:06.469 Spread Client:0x79327a0 [Comms] <INFO> nodeSetNotifier: node v_vabs_node0005 left the cluster
2014-07-29 01:41:06.469 Spread Client:0x79327a0 [Recover] <INFO> Node left cluster, reassessing k-safety...
2014-07-29 01:41:06.470 Spread Client:0x79327a0 [Comms] <INFO> stop: disconnecting #r3783-14#N010000001121 from spread daemon
2014-07-29 01:41:06.470 Spread Client:0x79327a0 [Comms] <INFO> connected: false
2014-07-29 01:41:06.470 Spread Client:0x79327a0 [Comms] <INFO> DB Group changed
2014-07-29 01:41:06.470 Spread Client:0x79327a0 [VMPI] <INFO> DistCall: Set current group members called with 0 members
2014-07-29 01:41:06.470 Spread Client:0x79327a0 [VMPI] <INFO> Removing 45035996273704970 from list of initialized nodes for session bigdata011.vabs-29563:0xb09e1
2014-07-29 01:41:06.470 Spread Client:0x79327a0 [VMPI] <INFO> Removing 45035996273704970 from list of initialized nodes for session bigdata011.vabs-29563:0xb0adf
2014-07-29 01:41:06.470 Spread Client:0x79327a0 [VMPI] <INFO> Removing 45035996273704970 from list of initialized nodes for session bigdata011.vabs-29563:0xb0ae3
2014-07-29 01:41:06.470 Spread Client:0x79327a0 [VMPI] <INFO> Removing 45035996273704970 from list of initialized nodes for session bigdata011.vabs-29563:0xb0afd
2014-07-29 01:41:06.471 Spread Client:0x79327a0 [Comms] <INFO> nodeSetNotifier: node v_vabs_node0001 left the cluster
2014-07-29 01:41:06.471 Spread Client:0x79327a0 [Recover] <INFO> Node left cluster, reassessing k-safety...
2014-07-29 01:41:06.471 Spread Client:0x79327a0 [Comms] <INFO> Lost membership of the DB group
2014-07-29 01:41:06.471 Spread Client:0x79327a0 [Comms] <INFO> Removing #r3783-14#N010000001121->v_vabs_node0001 from processToNode and other maps due to departure from Vertica:all
2014-07-29 01:41:06.471 Spread Client:0x79327a0 [Comms] <INFO> nodeToState map:
2014-07-29 01:41:06.471 Spread Client:0x79327a0 [Comms] <INFO> Removing #r4871-14#N010000001124->v_vabs_node0005 from processToNode and other maps due to departure from Vertica:all
2014-07-29 01:41:06.471 Spread Client:0x79327a0 [Comms] <INFO> nodeToState map:
2014-07-29 01:41:06.471 Spread Client:0x79327a0 [Comms] <INFO> Removing #r5560-14#N010000001123->v_vabs_node0003 from processToNode and other maps due to departure from Vertica:all
2014-07-29 01:41:06.471 Spread Client:0x79327a0 [Comms] <INFO> nodeToState map:
2014-07-29 01:41:06.471 Spread Client:0x79327a0 [Comms]

Vaibhav_Gupta · July 2014

/opt/vertica/bin/vnetperf --condense --hosts 10.0.1.121,10.0.1.122,10.0.1.123,10.0.1.124 --identity-file '/root/.ssh/id_rsa'
The maximum recommended rtt latency is 2 milliseconds. The ideal rtt latency is 200 microseconds or less. It is recommended that clock skew be kept to under 1 second.
test | date | node | index | rtt latency (us) | clock skew (us)
-------------------------------------------------------------------------------------------------------------------------
latency | 2014-07-29_09:22:10,146 | 10.0.1.121 | 0 | 99 | -1
latency | 2014-07-29_09:22:10,146 | 10.0.1.122 | 1 | 213 | 593
latency | 2014-07-29_09:22:10,146 | 10.0.1.123 | 2 | 166 | 719
latency | 2014-07-29_09:22:10,146 | 10.0.1.124 | 3 | 242 | 91

The minimum recommended throughput is 100 MB/s. Ideal throughput is 800 MB/s or more. Note: UDP numbers may be lower, multiple network switches may reduce performance results.
date | test | rate limit (MB/s) | node | MB/s (sent) | MB/s (rec) | bytes (sent) | bytes (rec) | duration (s)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2014-07-29_09:22:10,150 | udp-throughput | 32 | 10.0.1.121 | 30.5479 | 30.5479 | 32047104 | 32047104 | 1.00048
2014-07-29_09:22:10,150 | udp-throughput | 32 | 10.0.1.122 | 30.5481 | 30.5481 | 32047104 | 32047104 | 1.00047
2014-07-29_09:22:10,150 | udp-throughput | 32 | 10.0.1.123 | 30.5353 | 30.5353 | 32047104 | 32047104 | 1.00089
2014-07-29_09:22:10,150 | udp-throughput | 32 | 10.0.1.124 | 30.5341 | 30.5341 | 32047104 | 32047104 | 1.00093
2014-07-29_09:22:10,150 | udp-throughput | 32 | average | 30.5414 | 30.5414 | 32047104 | 32047104 | 1.00069
2014-07-29_09:22:11,156 | udp-throughput | 64 | 10.0.1.121 | 61.0583 | 61.0895 | 64061440 | 64094208 | 1.00058
2014-07-29_09:22:11,156 | udp-throughput | 64 | 10.0.1.122 | 61.0295 | 61.0608 | 64061440 | 64094208 | 1.00105
2014-07-29_09:22:11,156 | udp-throughput | 64 | 10.0.1.123 | 61.0597 | 61.0597 | 64061440 | 64061440 | 1.00056
2014-07-29_09:22:11,156 | udp-throughput | 64 | 10.0.1.124 | 60.9173 | 60.855 | 64061440 | 63995904 | 1.0029
2014-07-29_09:22:11,156 | udp-throughput | 64 | average | 61.0162 | 61.0162 | 64061440 | 64061440 | 1.00127
2014-07-29_09:22:12,167 | udp-throughput | 128 | 10.0.1.121 | 111.742 | 99.0858 | 120356864 | 106725376 | 1.0272
2014-07-29_09:22:12,167 | udp-throughput | 128 | 10.0.1.122 | 110.775 | 99.0045 | 119963648 | 107216896 | 1.03278
2014-07-29_09:22:12,167 | udp-throughput | 128 | 10.0.1.123 | 109.976 | 97.7302 | 120356864 | 106954752 | 1.04369
2014-07-29_09:22:12,167 | udp-throughput | 128 | 10.0.1.124 | 121.891 | 114.406 | 128057344 | 120193024 | 1.00192
2014-07-29_09:22:12,167 | udp-throughput | 128 | average | 113.596 | 102.557 | 122183680 | 110272512 | 1.0264
2014-07-29_09:22:13,379 | udp-throughput | 256 | 10.0.1.121 | 114.679 | 79.6479 | 120356864 | 83591168 | 1.00089
2014-07-29_09:22:13,379 | udp-throughput | 256 | 10.0.1.122 | 113.666 | 79.7533 | 119275520 | 83689472 | 1.00074
2014-07-29_09:22:13,379 | udp-throughput | 256 | 10.0.1.123 | 114.722 | 79.7714 | 120356864 | 83689472 | 1.00052
2014-07-29_09:22:13,379 | udp-throughput | 256 | 10.0.1.124 | 243.757 | 114.142 | 256049152 | 119898112 | 1.00177
2014-07-29_09:22:13,379 | udp-throughput | 256 | average | 146.706 | 88.3287 | 154009600 | 92717056 | 1.00098
2014-07-29_09:22:14,591 | udp-throughput | 512 | 10.0.1.121 | 114.661 | 78.5112 | 120356864 | 82411520 | 1.00105
2014-07-29_09:22:14,591 | udp-throughput | 512 | 10.0.1.122 | 113.41 | 78.8968 | 118980608 | 82771968 | 1.00052
2014-07-29_09:22:14,591 | udp-throughput | 512 | 10.0.1.123 | 114.659 | 78.4491 | 120258560 | 82280448 | 1.00025
2014-07-29_09:22:14,591 | udp-throughput | 512 | 10.0.1.124 | 488.299 | 113.559 | 512032768 | 119078912 | 1.00003
2014-07-29_09:22:14,591 | udp-throughput | 512 | average | 207.757 | 87.3541 | 217907200 | 91635712 | 1.00046
2014-07-29_09:22:15,647 | udp-throughput | 640 | 10.0.1.121 | 110.366 | 79.1763 | 115834880 | 83099648 | 1.00093
2014-07-29_09:22:15,647 | udp-throughput | 640 | 10.0.1.122 | 113.044 | 78.5907 | 118587392 | 82444288 | 1.00044
2014-07-29_09:22:15,647 | udp-throughput | 640 | 10.0.1.123 | 119.677 | 76.0161 | 125566976 | 79757312 | 1.00061
2014-07-29_09:22:15,647 | udp-throughput | 640 | 10.0.1.124 | 606.729 | 113.661 | 640024576 | 119898112 | 1.00601
2014-07-29_09:22:15,647 | udp-throughput | 640 | average | 237.454 | 86.861 | 250003456 | 91299840 | 1.002
2014-07-29_09:22:16,864 | udp-throughput | 768 | 10.0.1.121 | 163.601 | 80.1924 | 171671552 | 84148224 | 1.00072
2014-07-29_09:22:16,864 | udp-throughput | 768 | 10.0.1.122 | 79.1396 | 108.915 | 83001344 | 114229248 | 1.00021
2014-07-29_09:22:16,864 | udp-throughput | 768 | 10.0.1.123 | 99.095 | 97.1581 | 103940096 | 101908480 | 1.0003
2014-07-29_09:22:16,864 | udp-throughput | 768 | 10.0.1.124 | 309.095 | 113.089 | 325287936 | 119013376 | 1.00364
2014-07-29_09:22:16,864 | udp-throughput | 768 | average | 162.733 | 99.8385 | 170975232 | 104824832 | 1.00122
2014-07-29_09:22:17,912 | udp-throughput | 1024 | 10.0.1.121 | 104.581 | 83.972 | 113737728 | 91324416 | 1.03718
2014-07-29_09:22:17,912 | udp-throughput | 1024 | 10.0.1.122 | 106.361 | 85.0227 | 115638272 | 92438528 | 1.03686
2014-07-29_09:22:17,912 | udp-throughput | 1024 | 10.0.1.123 | 119.675 | 80.0546 | 130056192 | 86999040 | 1.0364
2014-07-29_09:22:17,912 | udp-throughput | 1024 | 10.0.1.124 | 584.633 | 114.02 | 614268928 | 119799808 | 1.00202
2014-07-29_09:22:17,912 | udp-throughput | 1024 | average | 228.812 | 90.7673 | 243425280 | 97640448 | 1.02811
2014-07-29_09:22:18,956 | udp-throughput | 2048 | 10.0.1.121 | 141.049 | 70.5395 | 152010752 | 76021760 | 1.02779
2014-07-29_09:22:18,956 | udp-throughput | 2048 | 10.0.1.122 | 82.8515 | 91.0393 | 89194496 | 98009088 | 1.02669
2014-07-29_09:22:18,956 | udp-throughput | 2048 | 10.0.1.123 | 94.2356 | 87.2067 | 101482496 | 93913088 | 1.02701
2014-07-29_09:22:18,956 | udp-throughput | 2048 | 10.0.1.124 | 318.022 | 108.668 | 334200832 | 114196480 | 1.00219
2014-07-29_09:22:18,956 | udp-throughput | 2048 | average | 159.039 | 89.3634 | 169222144 | 95535104 | 1.02092

The minimum recommended throughput is 100 MB/s. Ideal throughput is 800 MB/s or more. Note: UDP numbers may be lower, multiple network switches may reduce performance results.
date | test | rate limit (MB/s) | node | MB/s (sent) | MB/s (rec) | bytes (sent) | bytes (rec) | duration (s)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2014-07-29_09:22:19,991 | tcp-throughput | 32 | 10.0.1.121 | 30.6102 | 30.6102 | 32145408 | 32145408 | 1.0015
2014-07-29_09:22:19,991 | tcp-throughput | 32 | 10.0.1.122 | 30.6103 | 30.6103 | 32145408 | 32145408 | 1.0015
2014-07-29_09:22:19,991 | tcp-throughput | 32 | 10.0.1.123 | 30.6106 | 30.5794 | 32145408 | 32112640 | 1.00149
2014-07-29_09:22:19,991 | tcp-throughput | 32 | 10.0.1.124 | 30.6103 | 30.6103 | 32145408 | 32145408 | 1.0015
2014-07-29_09:22:19,991 | tcp-throughput | 32 | average | 30.6104 | 30.6026 | 32145408 | 32137216 | 1.0015
2014-07-29_09:22:22,5 | tcp-throughput | 64 | 10.0.1.121 | 61.1268 | 61.1268 | 64192512 | 64192512 | 1.00151
2014-07-29_09:22:22,5 | tcp-throughput | 64 | 10.0.1.122 | 61.1266 | 61.1266 | 64192512 | 64192512 | 1.00151
2014-07-29_09:22:22,5 | tcp-throughput | 64 | 10.0.1.123 | 61.1276 | 61.1276 | 64192512 | 64192512 | 1.00149
2014-07-29_09:22:22,5 | tcp-throughput | 64 | 10.0.1.124 | 61.1271 | 61.1271 | 64192512 | 64192512 | 1.0015
2014-07-29_09:22:22,5 | tcp-throughput | 64 | average | 61.127 | 61.127 | 64192512 | 64192512 | 1.0015
2014-07-29_09:22:24,15 | tcp-throughput | 128 | 10.0.1.121 | 112.547 | 105.169 | 118456320 | 110690304 | 1.00374
2014-07-29_09:22:24,15 | tcp-throughput | 128 | 10.0.1.122 | 121.159 | 118.833 | 128024576 | 125566976 | 1.00772
2014-07-29_09:22:24,15 | tcp-throughput | 128 | 10.0.1.123 | 118.698 | 121.525 | 125206528 | 128188416 | 1.00597
2014-07-29_09:22:24,15 | tcp-throughput | 128 | 10.0.1.124 | 102.704 | 107.97 | 108625920 | 114196480 | 1.00867
2014-07-29_09:22:24,15 | tcp-throughput | 128 | average | 113.777 | 113.374 | 120078336 | 119660544 | 1.00652
2014-07-29_09:22:26,47 | tcp-throughput | 256 | 10.0.1.121 | 194.622 | 169.857 | 205750272 | 179568640 | 1.0082
2014-07-29_09:22:26,47 | tcp-throughput | 256 | 10.0.1.122 | 208.691 | 163.689 | 222461952 | 174489600 | 1.0166
2014-07-29_09:22:26,47 | tcp-throughput | 256 | 10.0.1.123 | 175.217 | 229.875 | 191496192 | 251232256 | 1.04228
2014-07-29_09:22:26,47 | tcp-throughput | 256 | 10.0.1.124 | 99.4907 | 111.342 | 107282432 | 120061952 | 1.02836
2014-07-29_09:22:26,47 | tcp-throughput | 256 | average | 169.505 | 168.691 | 181747712 | 181338112 | 1.02386
2014-07-29_09:22:28,111 | tcp-throughput | 512 | 10.0.1.121 | 280.433 | 165.068 | 300613632 | 176947200 | 1.0223
2014-07-29_09:22:28,111 | tcp-throughput | 512 | 10.0.1.122 | 279.88 | 282.49 | 302120960 | 304939008 | 1.02946
2014-07-29_09:22:28,111 | tcp-throughput | 512 | 10.0.1.123 | 259.644 | 362.537 | 284033024 | 396591104 | 1.04326
2014-07-29_09:22:28,111 | tcp-throughput | 512 | 10.0.1.124 | 107.341 | 113.488 | 116162560 | 122814464 | 1.03205
2014-07-29_09:22:28,111 | tcp-throughput | 512 | average | 231.824 | 230.896 | 250732544 | 250322944 | 1.03177
2014-07-29_09:22:30,199 | tcp-throughput | 640 | 10.0.1.121 | 188.804 | 208.925 | 201392128 | 222855168 | 1.01726
2014-07-29_09:22:30,199 | tcp-throughput | 640 | 10.0.1.122 | 219.586 | 176.938 | 231309312 | 186384384 | 1.00459
2014-07-29_09:22:30,199 | tcp-throughput | 640 | 10.0.1.123 | 251.777 | 270.53 | 267485184 | 287408128 | 1.01317
2014-07-29_09:22:30,199 | tcp-throughput | 640 | 10.0.1.124 | 110.082 | 111.461 | 117637120 | 119111680 | 1.01913
2014-07-29_09:22:30,199 | tcp-throughput | 640 | average | 192.562 | 191.963 | 204455936 | 203939840 | 1.01354
2014-07-29_09:22:32,288 | tcp-throughput | 768 | 10.0.1.121 | 216.375 | 214.319 | 227672064 | 225509376 | 1.00347
2014-07-29_09:22:32,288 | tcp-throughput | 768 | 10.0.1.122 | 182.414 | 169.226 | 193069056 | 179109888 | 1.00938
2014-07-29_09:22:32,288 | tcp-throughput | 768 | 10.0.1.123 | 227.762 | 232.579 | 247889920 | 253132800 | 1.03795
2014-07-29_09:22:32,288 | tcp-throughput | 768 | 10.0.1.124 | 103.602 | 111.219 | 111869952 | 120094720 | 1.02978
2014-07-29_09:22:32,288 | tcp-throughput | 768 | average | 182.538 | 181.836 | 195125248 | 194461696 | 1.02015
2014-07-29_09:22:34,378 | tcp-throughput | 1024 | 10.0.1.121 | 157.504 | 130.241 | 166592512 | 137756672 | 1.0087
2014-07-29_09:22:34,378 | tcp-throughput | 1024 | 10.0.1.122 | 180.39 | 305.503 | 193232896 | 327254016 | 1.02157
2014-07-29_09:22:34,378 | tcp-throughput | 1024 | 10.0.1.123 | 307.718 | 198.837 | 337281024 | 217939968 | 1.0453
2014-07-29_09:22:34,378 | tcp-throughput | 1024 | 10.0.1.124 | 100.45 | 110.48 | 109608960 | 120553472 | 1.04063
2014-07-29_09:22:34,378 | tcp-throughput | 1024 | average | 186.515 | 186.265 | 2016

[Deleted User] · July 2014

It appears, (nodeSetNotifier: node v_vabs_node0002 left the cluster.)
Do you have the logs on that node, maybe it will shed some light on why that node left

Vaibhav_Gupta · July 2014

Below Logs of node v_vabs_node0001
FYI: v_vabs_node0001 went down

,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'2014-07-29 01:39:44.192 Init Session:0x7f51b8017970-b0000000886cdb [Txn] <INFO> Begin Txn: b0000000886cdb 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:39:44.193 Init Session:0x7f51b8024960-b0000000886ce6 [Txn] <INFO> Begin Txn: b0000000886ce6 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:39:44.193 Init Session:0x7f51b800fb10-b0000000886ce8 [Txn] <INFO> Begin Txn: b0000000886ce8 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:39:44.196 Init Session:0x7f51b8015fb0-b0000000886cf5 [Txn] <INFO> Begin Txn: b0000000886cf5 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:39:44.196 Init Session:0x7f51b8012c70-b0000000886ce9 [Txn] <INFO> Begin Txn: b0000000886ce9 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:39:44.196 Init Session:0x7f51b801a420-b0000000886ce7 [Txn] <INFO> Begin Txn: b0000000886ce7 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:39:44.199 Init Session:0x7f51b80214a0-b0000000886ceb [Txn] <INFO> Begin Txn: b0000000886ceb 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:39:44.199 Init Session:0x7f51b8010050-b0000000886cf1 [Txn] <INFO> Begin Txn: b0000000886cf1 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:39:44.200 Init Session:0x7f51b800f5d0-b0000000886cf3 [Txn] <INFO> Begin Txn: b0000000886cf3 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:39:44.201 Init Session:0x7f51b80242d0-b0000000886cf7 [Txn] <INFO> Begin Txn: b0000000886cf7 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:39:44.201 Init Session:0x7f51b801c7c0-b0000000886cf2 [Txn] <INFO> Begin Txn: b0000000886cf2 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:39:44.201 Init Session:0x7f51b800f2e0-b0000000886cf6 [Txn] <INFO> Begin Txn: b0000000886cf6 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:39:44.337 Init Session:0x7f51b8014c60-b0000000886cb8 [Session] <INFO> [Query] TX:b0000000886cb8(bigdata012.vabs-13592:0xb7ec5) COMMIT
2014-07-29 01:39:45.298 DistCall Dispatch:0x7f51c802cfb0-a000000047d0e8 [Catalog] <INFO> applyTransaction: Received catalog update, current catalog (TRANSACTION-a000000047d0e8 version=0)
2014-07-29 01:39:45.301 DistCall Dispatch:0x7f51c802cfb0-a000000047d0e8 [Catalog] <INFO> applyTransaction: Complete. New version = 0x5e39a5b
2014-07-29 01:39:45.301 DistCall Dispatch:0x7f51c802cfb0-a000000047d0e8 [Catalog] <INFO> applyTransaction: globalOidSeq is at 7b7c4cf3
2014-07-29 01:39:45.302 DistCall Dispatch:0x7f51c802cfb0-a000000047d0e8 [Catalog] <INFO> applyTransaction: Received catalog update, current catalog (TRANSACTION-a000000047d0e8 version=0xce)
2014-07-29 01:39:45.302 DistCall Dispatch:0x7f51c802cfb0-a000000047d0e8 [Catalog] <INFO> applyTransaction: Complete. New version = 0x5e39a5b
2014-07-29 01:39:45.302 DistCall Dispatch:0x7f51c802cfb0-a000000047d0e8 [Catalog] <INFO> applyTransaction: globalOidSeq is at 7b7c4cf3
2014-07-29 01:39:45.302 DistCall Dispatch:0x7f51c802cfb0-a000000047d0e8 [Txn] <INFO> Starting Commit: Txn: a000000047d0e8 'do_analyze_row_count'
2014-07-29 01:39:45.690 Init Session:0x7f51b8024fd0 <LOG> @v_vabs_node0002: 00000/2705: Connection received: host=10.0.1.120 port=59753 (connCnt 36)
2014-07-29 01:39:45.690 DistCall Dispatch:0x7f51c802cfb0 [Txn] <INFO> Commit Complete: Txn: a000000047d0e8 at epoch 0x2f856e2
2014-07-29 01:39:45.690 Init Session:0x7f51b8024fd0 <LOG> @v_vabs_node0002: 00000/4686: sendAuthRequest: user=pentaho database=vabs host=10.0.1.120 authType=3
2014-07-29 01:39:51.319 Init Session:0x7f51b8014c60-b0000000886cb8 [Txn] <INFO> Starting Commit: Txn: b0000000886cb8 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:39:52.929 Init Session:0x7f51b8025640-b0000000886cbb [Session] <INFO> [Query] TX:b0000000886cbb(bigdata012.vabs-13592:0xb7ec7) COMMIT
2014-07-29 01:39:52.932 Init Session:0x7f51b8014c60 [Txn] <INFO> Commit Complete: Txn: b0000000886cb8 at epoch 0x2f856e2
2014-07-29 01:39:54.001 RebalanceCluster:0x7f51f0041fe0 [Util] <INFO> Task 'RebalanceCluster' enabled
2014-07-29 01:39:56.948 Init Session:0x7f51b8014c60 <LOG> @v_vabs_node0002: 00000/4719: Session bigdata012.vabs-13592:0xb7ec5 ended; closing connection (connCnt 36)
2014-07-29 01:39:56.988 Init Session:0x7f51b8025640-b0000000886cbb [Txn] <INFO> Starting Commit: Txn: b0000000886cbb 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:39:57.040 Init Session:0x7f51b8025640 [Txn] <INFO> Commit Complete: Txn: b0000000886cbb at epoch 0x2f856e3
2014-07-29 01:39:58.344 Init Session:0x7f51b800eb30 <LOG> @v_vabs_node0002: 00000/2705: Connection received: host=10.0.1.120 port=59755 (connCnt 36)
2014-07-29 01:39:58.345 Init Session:0x7f51b800eb30 <LOG> @v_vabs_node0002: 00000/4686: sendAuthRequest: user=pentaho database=vabs host=10.0.1.120 authType=3
2014-07-29 01:39:59.822 Init Session:0x7f51b8024fd0-b0000000886d0f [Txn] <INFO> Begin Txn: b0000000886d0f 'check_login_history'
2014-07-29 01:39:59.823 Init Session:0x7f51b8024fd0-b0000000886d0f [Txn] <INFO> Rollback Txn: b0000000886d0f 'check_login_history'
2014-07-29 01:39:59.824 Init Session:0x7f51b8024fd0 <LOG> @v_vabs_node0002: 00000/4686: sendAuthRequest: user=pentaho database=vabs host=10.0.1.120 authType=0
2014-07-29 01:39:59.825 Init Session:0x7f51b8024fd0 <LOG> @v_vabs_node0002: 00000/2703: Connection authenticated: user=pentaho database=vabs host=10.0.1.120
2014-07-29 01:39:59.826 Init Session:0x7f51b8024fd0 <LOG> @v_vabs_node0002: 00000/2609: Client pid: 14779
2014-07-29 01:39:59.826 TM Mergeout(02):0x7f51f0041fe0-b0000000886d10 [Txn] <INFO> Begin Txn: b0000000886d10 'Mergeout: Tuple Mover'
2014-07-29 01:39:59.828 Init Session:0x7f51b8024fd0 <LOG> @v_vabs_node0002: 00000/2610: Client requested binary data transfer
2014-07-29 01:39:59.829 TM Mergeout(03):0x7f51f0042be0-b0000000886d11 [Txn] <INFO> Begin Txn: b0000000886d11 'Mergeout: Tuple Mover'
2014-07-29 01:39:59.834 Init Session:0x7f51b800eb30-b0000000886d12 [Txn] <INFO> Begin Txn: b0000000886d12 'check_login_history'
2014-07-29 01:39:59.892 Init Session:0x7f51b800eb30-b0000000886d12 [Txn] <INFO> Rollback Txn: b0000000886d12 'check_login_history'
2014-07-29 01:39:59.943 Init Session:0x7f51b800eb30 <LOG> @v_vabs_node0002: 00000/4686: sendAuthRequest: user=pentaho database=vabs host=10.0.1.120 authType=0
2014-07-29 01:39:59.944 Init Session:0x7f51b800eb30 <LOG> @v_vabs_node0002: 00000/2703: Connection authenticated: user=pentaho database=vabs host=10.0.1.120
2014-07-29 01:39:59.944 Init Session:0x7f51b800eb30 <LOG> @v_vabs_node0002: 00000/2609: Client pid: 14779
2014-07-29 01:39:59.945 Init Session:0x7f51b800eb30 <LOG> @v_vabs_node0002: 00000/2610: Client requested binary data transfer
2014-07-29 01:40:00.000 Init Session:0x7f51b8024fd0 [Session] <INFO> [PQuery] TX:0(bigdata012.vabs-13592:0xb7efa) select version_internal()
2014-07-29 01:40:00.006 Init Session:0x7f51b800eb30 [Session] <INFO> [PQuery] TX:0(bigdata012.vabs-13592:0xb7f04) select version_internal()
2014-07-29 01:40:00.006 TM Mergeout(01):0x7f51f0042e50-b0000000886d13 [Txn] <INFO> Begin Txn: b0000000886d13 'Mergeout: Tuple Mover'
2014-07-29 01:40:00.007 Init Session:0x7f51b8024fd0 <LOG> @v_vabs_node0002: 00000/3316: Executing statement: 'select version_internal()'
2014-07-29 01:40:00.028 Init Session:0x7f51b800eb30 <LOG> @v_vabs_node0002: 00000/3316: Executing statement: 'select version_internal()'
2014-07-29 01:40:00.029 Init Session:0x7f51b8024fd0 [Session] <INFO> [Query] TX:0(bigdata012.vabs-13592:0xb7efa) COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT
2014-07-29 01:40:00.056 Init Session:0x7f51b800eb30 [Session] <INFO> [Query] TX:0(bigdata012.vabs-13592:0xb7f04) COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_uniqu

colin_loghin · July 2014

This message (NETWORK change with 1 VS sets) usually means that one node was not able to heartbeat with the other nodes , such as when network separation occurs. I am guessing the vertica cluster setup did not use 1 private set of NICs for cluster setup ( -s parameter in install_vertica) and COPY statement network traffic is impacting internal heartbeat traffic. This would explain why node goes down in this situation.

Reassess the network setup, reinstall vertica using private ips to seperate traffic,
check if you have large latency ( it may be fluctuating) and use default UDP assuming all hosts are on the same subnet.

Vaibhav_Gupta · July 2014

The vertica cluster is on Vms (Vmware). The below command is use for vertica install
/opt/vertica/sbin/install_vertica -s bigdata-stg-011.vabs-staging.com,bigdata-stg-012.vabs-staging.com,bigdata-stg-013.vabs-staging.com,bigdata-stg-014.vabs-staging.com -l /data/dbadmin -r /tmp/vertica-6.1.2-0.x86_64.RHEL5.rpm -d /data/vertica/ -C -N -T -S default -Y -L vlicense.dat

For latency i tried vnetperf. Output in 2nd comment

Below my spread file

Spread_Segment 10.0.1.121:4803 {
N010000001121 10.0.1.121 {
10.0.1.121
127.0.0.1
}
}
Spread_Segment 10.0.1.122:4803 {
N010000001122 10.0.1.122 {
10.0.1.122
127.0.0.1
}
}
Spread_Segment 10.0.1.123:4803 {
N010000001123 10.0.1.123 {
10.0.1.123
127.0.0.1
}
}
Spread_Segment 10.0.1.124:4803 {
N010000001124 10.0.1.124 {
10.0.1.124
127.0.0.1
}
}
EventLogFile = /dev/null
EventTimeStamp = "[%a %d %b %Y %H:%M:%S]"
DaemonUser = spread
DaemonGroup = verticadba
DebugFlags = { EXIT }

Node frequent went down when COPY large ammount of data

Comments

Leave a Comment