The Vertica Forum recently got a makeover! Let us know what you think by filling out this short, anonymous survey.
Please take this survey to help us learn more about how you use third party tools. Your input is greatly appreciated!

Node frequent went down when COPY large ammount of data

Vaibhav_GuptaVaibhav_Gupta
in General Discussion
When we COPY the data one of the node went down in cluster.  Below the logs in vertica.log

2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7ed8 due to loss of 450359964152211262014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7ed9 due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7eda due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7edb due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7edc due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7ede due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7edf due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7ee2 due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7ee4 due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7ee5 due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7ee6 due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7ee9 due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7eea due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7ef1 due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7efa due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [VMPI] <INFO> Ending session bigdata012.vabs-13592:0xb7f04 due to loss of 45035996415221126
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [Comms] <INFO> nodeSetNotifier: node v_vabs_node0002 left the cluster
2014-07-29 01:41:06.432 Spread Client:0x79327a0 [Recover] <INFO> Node left cluster, reassessing k-safety...
2014-07-29 01:41:06.433 Spread Client:0x79327a0 [Recover] <INFO> Checking Deps:Down bits: 0010 Deps:
0011 - cnt: 676
0110 - cnt: 676
1001 - cnt: 676
1100 - cnt: 676
1111 - cnt: 1399
2014-07-29 01:41:06.433 DistCall Dispatch:0x7f1d1c018860-b0000000886cf1 [Txn] <INFO> Rollback Txn: b0000000886cf1 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.433 DistCall Dispatch:0x7f1d1c0169d0-b0000000886cf2 [Txn] <INFO> Rollback Txn: b0000000886cf2 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.433 DistCall Dispatch:0x7f1d1c017830-b0000000886cf4 [Txn] <INFO> Rollback Txn: b0000000886cf4 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.434 DistCall Dispatch:0x7f1d1c0045a0-b0000000886cdb [Txn] <INFO> Rollback Txn: b0000000886cdb 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.434 DistCall Dispatch:0x7f1d1c006ba0-b0000000886cea [Txn] <INFO> Rollback Txn: b0000000886cea 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.435 DistCall Dispatch:0x7f1d1c005cd0-b0000000886ce9 [Txn] <INFO> Rollback Txn: b0000000886ce9 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.435 DistCall Dispatch:0x7f1d1c017f50-b0000000886ceb [Txn] <INFO> Rollback Txn: b0000000886ceb 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.436 DistCall Dispatch:0x7f1d1c004a70-b0000000886ce5 [Txn] <INFO> Rollback Txn: b0000000886ce5 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.436 DistCall Dispatch:0x7f1d1c05dde0-b0000000886cda [Txn] <INFO> Rollback Txn: b0000000886cda 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.437 DistCall Dispatch:0x7f1d1c00c640-b0000000886cf0 [Txn] <INFO> Rollback Txn: b0000000886cf0 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.437 DistCall Dispatch:0x7f1d1c00cea0-b0000000886cf5 [Txn] <INFO> Rollback Txn: b0000000886cf5 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.437 DistCall Dispatch:0x7f1d1c027b50-b0000000886d14 [Txn] <INFO> Rollback Txn: b0000000886d14 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.438 DistCall Dispatch:0x7f1d1c02abe0-b0000000886d15 [Txn] <INFO> Rollback Txn: b0000000886d15 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.438 DistCall Dispatch:0x7f1d1c06be80-b0000000886cdc [Txn] <INFO> Rollback Txn: b0000000886cdc 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
2014-07-29 01:41:06.442 Spread Client:0x79327a0 <LOG> @v_vabs_node0001: 00000/3298: Event Posted: Event Code:3 Event Id:0 Event Severity: Critical [2] PostedTimestamp: 2014-07-29 01:41:06.433086 ExpirationTimestamp: 2082-08-16 03:55:13.433086 EventCodeDescription: Current Fault Tolerance at Critical Level ProblemDescription: Loss of node v_vabs_node0001 will cause shutdown to occur. K=1 total number of nodes=4 DatabaseName: vabs Hostname: bigdata011.vabs.com
2014-07-29 01:41:06.444 Spread Client:0x79327a0 [Comms] <INFO> nodeSetNotifier: node v_vabs_node0003 left the cluster
2014-07-29 01:41:06.444 Spread Client:0x79327a0 [Recover] <INFO> Node left cluster, reassessing k-safety...
2014-07-29 01:41:06.444 Spread Client:0x79327a0 [Recover] <INFO> Cluster partitioned: 4 total nodes, 2 up nodes, 2 down nodes
2014-07-29 01:41:06.444 Spread Client:0x79327a0 [Recover] <INFO> Setting node v_vabs_node0001 to UNSAFE
2014-07-29 01:41:06.445 Init Session:0x7f1cb4013330-a000000047d0c6 [Dist] <WARNING> Node v_vabs_node0002 was in plan recipents list at plan start but is now DOWN
2014-07-29 01:41:06.447 Spread Client:0x79327a0 <LOG> @v_vabs_node0001: 00000/3298: Event Posted: Event Code:6 Event Id:5 Event Severity: Informational [6] PostedTimestamp: 2014-07-29 01:41:06.447401 ExpirationTimestamp: 2082-08-16 03:55:13.447401 EventCodeDescription: Node State Change ProblemDescription: Changing node v_vabs_node0001 startup state to UNSAFE DatabaseName: vabs Hostname: bigdata011.vabs.com
2014-07-29 01:41:06.448 AnalyzeRowCount:0x7f1cac012ef0 <WARNING> @v_vabs_node0001: 01000/4539: Received no response from v_vabs_node0002 in GetRowCountMessage
2014-07-29 01:41:06.449 Spread Client:0x79327a0 <LOG> @v_vabs_node0001: 00000/3293: Event Cleared: Event Code:6 Event Id:6 Event Severity: Informational [6] PostedTimestamp: 2014-07-29 01:41:06.447537 ExpirationTimestamp: 2014-07-29 01:41:06.447537 EventCodeDescription: Node State Change ProblemDescription: Changing node v_vabs_node0001 leaving startup state UP DatabaseName: vabs Hostname: bigdata011.vabs.com
2014-07-29 01:41:06.449 Spread Client:0x79327a0 [Recover] <INFO> Changing node v_vabs_node0001 startup state from UP to UNSAFE
2014-07-29 01:41:06.449 Spread Client:0x79327a0 <LOG> @v_vabs_node0001: 00000/3298: Event Posted: Event Code:2 Event Id:0 Event Severity: Emergency [0] PostedTimestamp: 2014-07-29 01:41:06.44976 ExpirationTimestamp: 2014-07-29 01:51:06.44976 EventCodeDescription: Loss Of K Safety ProblemDescription: System is not K-safe: K=1 total number of nodes=4 DatabaseName: vabs Hostname: bigdata011.vabs.com
2014-07-29 01:41:06.452 AnalyzeRowCount:0x7f1cac012ef0 <WARNING> @v_vabs_node0001: 01000/4539: Received no response from v_vabs_node0003 in GetRowCountMessage
2014-07-29 01:41:06.454 AnalyzeRowCount:0x7f1cac012ef0 <WARNING> @v_vabs_node0001: 01000/4539: Received no response from v_vabs_node0005 in GetRowCountMessage
2014-07-29 01:41:06.464 DistCall Dispatch:0x7f1d1c06b910-c00000004dbcbe [Txn] <INFO> Rollback Txn: c00000004dbcbe 'SELECT BigData_LabelType_Identifier AS id, LabelType_Name AS name FROM AnalyticsNew.Label_Type'
2014-07-29 01:41:06.469 Spread Client:0x79327a0 [Comms] <INFO> nodeSetNotifier: node v_vabs_node0005 left the cluster
2014-07-29 01:41:06.469 Spread Client:0x79327a0 [Recover] <INFO> Node left cluster, reassessing k-safety...
2014-07-29 01:41:06.470 Spread Client:0x79327a0 [Comms] <INFO> stop: disconnecting #r3783-14#N010000001121 from spread daemon
2014-07-29 01:41:06.470 Spread Client:0x79327a0 [Comms] <INFO> connected: false
2014-07-29 01:41:06.470 Spread Client:0x79327a0 [Comms] <INFO> DB Group changed
2014-07-29 01:41:06.470 Spread Client:0x79327a0 [VMPI] <INFO> DistCall: Set current group members called with 0 members
2014-07-29 01:41:06.470 Spread Client:0x79327a0 [VMPI] <INFO> Removing 45035996273704970 from list of initialized nodes for session bigdata011.vabs-29563:0xb09e1
2014-07-29 01:41:06.470 Spread Client:0x79327a0 [VMPI] <INFO> Removing 45035996273704970 from list of initialized nodes for session bigdata011.vabs-29563:0xb0adf
2014-07-29 01:41:06.470 Spread Client:0x79327a0 [VMPI] <INFO> Removing 45035996273704970 from list of initialized nodes for session bigdata011.vabs-29563:0xb0ae3
2014-07-29 01:41:06.470 Spread Client:0x79327a0 [VMPI] <INFO> Removing 45035996273704970 from list of initialized nodes for session bigdata011.vabs-29563:0xb0afd
2014-07-29 01:41:06.471 Spread Client:0x79327a0 [Comms] <INFO> nodeSetNotifier: node v_vabs_node0001 left the cluster
2014-07-29 01:41:06.471 Spread Client:0x79327a0 [Recover] <INFO> Node left cluster, reassessing k-safety...
2014-07-29 01:41:06.471 Spread Client:0x79327a0 [Comms] <INFO> Lost membership of the DB group
2014-07-29 01:41:06.471 Spread Client:0x79327a0 [Comms] <INFO> Removing #r3783-14#N010000001121->v_vabs_node0001 from processToNode and other maps due to departure from Vertica:all
2014-07-29 01:41:06.471 Spread Client:0x79327a0 [Comms] <INFO> nodeToState map:
2014-07-29 01:41:06.471 Spread Client:0x79327a0 [Comms] <INFO> Removing #r4871-14#N010000001124->v_vabs_node0005 from processToNode and other maps due to departure from Vertica:all
2014-07-29 01:41:06.471 Spread Client:0x79327a0 [Comms] <INFO> nodeToState map:
2014-07-29 01:41:06.471 Spread Client:0x79327a0 [Comms] <INFO> Removing #r5560-14#N010000001123->v_vabs_node0003 from processToNode and other maps due to departure from Vertica:all
2014-07-29 01:41:06.471 Spread Client:0x79327a0 [Comms] <INFO> nodeToState map:
2014-07-29 01:41:06.471 Spread Client:0x79327a0 [Comms]

Comments

  • Vaibhav_GuptaVaibhav_Gupta
    /opt/vertica/bin/vnetperf --condense --hosts 10.0.1.121,10.0.1.122,10.0.1.123,10.0.1.124 --identity-file '/root/.ssh/id_rsa'
    The maximum recommended rtt latency is 2 milliseconds. The ideal rtt latency is 200 microseconds or less. It is recommended that clock skew be kept to under 1 second.
    test                          | date                    | node             | index | rtt latency (us)  | clock skew (us) 
    -------------------------------------------------------------------------------------------------------------------------
    latency                       | 2014-07-29_09:22:10,146 | 10.0.1.121       | 0     | 99                | -1              
    latency                       | 2014-07-29_09:22:10,146 | 10.0.1.122       | 1     | 213               | 593             
    latency                       | 2014-07-29_09:22:10,146 | 10.0.1.123       | 2     | 166               | 719             
    latency                       | 2014-07-29_09:22:10,146 | 10.0.1.124       | 3     | 242               | 91              

    The minimum recommended throughput is 100 MB/s. Ideal throughput is 800 MB/s or more. Note: UDP numbers may be lower, multiple network switches may reduce performance results.
    date                    | test                     | rate limit (MB/s) | node             | MB/s (sent) | MB/s (rec)  | bytes (sent)        | bytes (rec)         | duration (s)
    ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    2014-07-29_09:22:10,150 | udp-throughput           | 32                | 10.0.1.121       | 30.5479     | 30.5479     | 32047104            | 32047104            | 1.00048     
    2014-07-29_09:22:10,150 | udp-throughput           | 32                | 10.0.1.122       | 30.5481     | 30.5481     | 32047104            | 32047104            | 1.00047     
    2014-07-29_09:22:10,150 | udp-throughput           | 32                | 10.0.1.123       | 30.5353     | 30.5353     | 32047104            | 32047104            | 1.00089     
    2014-07-29_09:22:10,150 | udp-throughput           | 32                | 10.0.1.124       | 30.5341     | 30.5341     | 32047104            | 32047104            | 1.00093     
    2014-07-29_09:22:10,150 | udp-throughput           | 32                | average          | 30.5414     | 30.5414     | 32047104            | 32047104            | 1.00069     
    2014-07-29_09:22:11,156 | udp-throughput           | 64                | 10.0.1.121       | 61.0583     | 61.0895     | 64061440            | 64094208            | 1.00058     
    2014-07-29_09:22:11,156 | udp-throughput           | 64                | 10.0.1.122       | 61.0295     | 61.0608     | 64061440            | 64094208            | 1.00105     
    2014-07-29_09:22:11,156 | udp-throughput           | 64                | 10.0.1.123       | 61.0597     | 61.0597     | 64061440            | 64061440            | 1.00056     
    2014-07-29_09:22:11,156 | udp-throughput           | 64                | 10.0.1.124       | 60.9173     | 60.855      | 64061440            | 63995904            | 1.0029      
    2014-07-29_09:22:11,156 | udp-throughput           | 64                | average          | 61.0162     | 61.0162     | 64061440            | 64061440            | 1.00127     
    2014-07-29_09:22:12,167 | udp-throughput           | 128               | 10.0.1.121       | 111.742     | 99.0858     | 120356864           | 106725376           | 1.0272      
    2014-07-29_09:22:12,167 | udp-throughput           | 128               | 10.0.1.122       | 110.775     | 99.0045     | 119963648           | 107216896           | 1.03278     
    2014-07-29_09:22:12,167 | udp-throughput           | 128               | 10.0.1.123       | 109.976     | 97.7302     | 120356864           | 106954752           | 1.04369     
    2014-07-29_09:22:12,167 | udp-throughput           | 128               | 10.0.1.124       | 121.891     | 114.406     | 128057344           | 120193024           | 1.00192     
    2014-07-29_09:22:12,167 | udp-throughput           | 128               | average          | 113.596     | 102.557     | 122183680           | 110272512           | 1.0264      
    2014-07-29_09:22:13,379 | udp-throughput           | 256               | 10.0.1.121       | 114.679     | 79.6479     | 120356864           | 83591168            | 1.00089     
    2014-07-29_09:22:13,379 | udp-throughput           | 256               | 10.0.1.122       | 113.666     | 79.7533     | 119275520           | 83689472            | 1.00074     
    2014-07-29_09:22:13,379 | udp-throughput           | 256               | 10.0.1.123       | 114.722     | 79.7714     | 120356864           | 83689472            | 1.00052     
    2014-07-29_09:22:13,379 | udp-throughput           | 256               | 10.0.1.124       | 243.757     | 114.142     | 256049152           | 119898112           | 1.00177     
    2014-07-29_09:22:13,379 | udp-throughput           | 256               | average          | 146.706     | 88.3287     | 154009600           | 92717056            | 1.00098     
    2014-07-29_09:22:14,591 | udp-throughput           | 512               | 10.0.1.121       | 114.661     | 78.5112     | 120356864           | 82411520            | 1.00105     
    2014-07-29_09:22:14,591 | udp-throughput           | 512               | 10.0.1.122       | 113.41      | 78.8968     | 118980608           | 82771968            | 1.00052     
    2014-07-29_09:22:14,591 | udp-throughput           | 512               | 10.0.1.123       | 114.659     | 78.4491     | 120258560           | 82280448            | 1.00025     
    2014-07-29_09:22:14,591 | udp-throughput           | 512               | 10.0.1.124       | 488.299     | 113.559     | 512032768           | 119078912           | 1.00003     
    2014-07-29_09:22:14,591 | udp-throughput           | 512               | average          | 207.757     | 87.3541     | 217907200           | 91635712            | 1.00046     
    2014-07-29_09:22:15,647 | udp-throughput           | 640               | 10.0.1.121       | 110.366     | 79.1763     | 115834880           | 83099648            | 1.00093     
    2014-07-29_09:22:15,647 | udp-throughput           | 640               | 10.0.1.122       | 113.044     | 78.5907     | 118587392           | 82444288            | 1.00044     
    2014-07-29_09:22:15,647 | udp-throughput           | 640               | 10.0.1.123       | 119.677     | 76.0161     | 125566976           | 79757312            | 1.00061     
    2014-07-29_09:22:15,647 | udp-throughput           | 640               | 10.0.1.124       | 606.729     | 113.661     | 640024576           | 119898112           | 1.00601     
    2014-07-29_09:22:15,647 | udp-throughput           | 640               | average          | 237.454     | 86.861      | 250003456           | 91299840            | 1.002       
    2014-07-29_09:22:16,864 | udp-throughput           | 768               | 10.0.1.121       | 163.601     | 80.1924     | 171671552           | 84148224            | 1.00072     
    2014-07-29_09:22:16,864 | udp-throughput           | 768               | 10.0.1.122       | 79.1396     | 108.915     | 83001344            | 114229248           | 1.00021     
    2014-07-29_09:22:16,864 | udp-throughput           | 768               | 10.0.1.123       | 99.095      | 97.1581     | 103940096           | 101908480           | 1.0003      
    2014-07-29_09:22:16,864 | udp-throughput           | 768               | 10.0.1.124       | 309.095     | 113.089     | 325287936           | 119013376           | 1.00364     
    2014-07-29_09:22:16,864 | udp-throughput           | 768               | average          | 162.733     | 99.8385     | 170975232           | 104824832           | 1.00122     
    2014-07-29_09:22:17,912 | udp-throughput           | 1024              | 10.0.1.121       | 104.581     | 83.972      | 113737728           | 91324416            | 1.03718     
    2014-07-29_09:22:17,912 | udp-throughput           | 1024              | 10.0.1.122       | 106.361     | 85.0227     | 115638272           | 92438528            | 1.03686     
    2014-07-29_09:22:17,912 | udp-throughput           | 1024              | 10.0.1.123       | 119.675     | 80.0546     | 130056192           | 86999040            | 1.0364      
    2014-07-29_09:22:17,912 | udp-throughput           | 1024              | 10.0.1.124       | 584.633     | 114.02      | 614268928           | 119799808           | 1.00202     
    2014-07-29_09:22:17,912 | udp-throughput           | 1024              | average          | 228.812     | 90.7673     | 243425280           | 97640448            | 1.02811     
    2014-07-29_09:22:18,956 | udp-throughput           | 2048              | 10.0.1.121       | 141.049     | 70.5395     | 152010752           | 76021760            | 1.02779     
    2014-07-29_09:22:18,956 | udp-throughput           | 2048              | 10.0.1.122       | 82.8515     | 91.0393     | 89194496            | 98009088            | 1.02669     
    2014-07-29_09:22:18,956 | udp-throughput           | 2048              | 10.0.1.123       | 94.2356     | 87.2067     | 101482496           | 93913088            | 1.02701     
    2014-07-29_09:22:18,956 | udp-throughput           | 2048              | 10.0.1.124       | 318.022     | 108.668     | 334200832           | 114196480           | 1.00219     
    2014-07-29_09:22:18,956 | udp-throughput           | 2048              | average          | 159.039     | 89.3634     | 169222144           | 95535104            | 1.02092     

    The minimum recommended throughput is 100 MB/s. Ideal throughput is 800 MB/s or more. Note: UDP numbers may be lower, multiple network switches may reduce performance results.
    date                    | test                     | rate limit (MB/s) | node             | MB/s (sent) | MB/s (rec)  | bytes (sent)        | bytes (rec)         | duration (s)
    ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    2014-07-29_09:22:19,991 | tcp-throughput           | 32                | 10.0.1.121       | 30.6102     | 30.6102     | 32145408            | 32145408            | 1.0015      
    2014-07-29_09:22:19,991 | tcp-throughput           | 32                | 10.0.1.122       | 30.6103     | 30.6103     | 32145408            | 32145408            | 1.0015      
    2014-07-29_09:22:19,991 | tcp-throughput           | 32                | 10.0.1.123       | 30.6106     | 30.5794     | 32145408            | 32112640            | 1.00149     
    2014-07-29_09:22:19,991 | tcp-throughput           | 32                | 10.0.1.124       | 30.6103     | 30.6103     | 32145408            | 32145408            | 1.0015      
    2014-07-29_09:22:19,991 | tcp-throughput           | 32                | average          | 30.6104     | 30.6026     | 32145408            | 32137216            | 1.0015      
    2014-07-29_09:22:22,5   | tcp-throughput           | 64                | 10.0.1.121       | 61.1268     | 61.1268     | 64192512            | 64192512            | 1.00151     
    2014-07-29_09:22:22,5   | tcp-throughput           | 64                | 10.0.1.122       | 61.1266     | 61.1266     | 64192512            | 64192512            | 1.00151     
    2014-07-29_09:22:22,5   | tcp-throughput           | 64                | 10.0.1.123       | 61.1276     | 61.1276     | 64192512            | 64192512            | 1.00149     
    2014-07-29_09:22:22,5   | tcp-throughput           | 64                | 10.0.1.124       | 61.1271     | 61.1271     | 64192512            | 64192512            | 1.0015      
    2014-07-29_09:22:22,5   | tcp-throughput           | 64                | average          | 61.127      | 61.127      | 64192512            | 64192512            | 1.0015      
    2014-07-29_09:22:24,15  | tcp-throughput           | 128               | 10.0.1.121       | 112.547     | 105.169     | 118456320           | 110690304           | 1.00374     
    2014-07-29_09:22:24,15  | tcp-throughput           | 128               | 10.0.1.122       | 121.159     | 118.833     | 128024576           | 125566976           | 1.00772     
    2014-07-29_09:22:24,15  | tcp-throughput           | 128               | 10.0.1.123       | 118.698     | 121.525     | 125206528           | 128188416           | 1.00597     
    2014-07-29_09:22:24,15  | tcp-throughput           | 128               | 10.0.1.124       | 102.704     | 107.97      | 108625920           | 114196480           | 1.00867     
    2014-07-29_09:22:24,15  | tcp-throughput           | 128               | average          | 113.777     | 113.374     | 120078336           | 119660544           | 1.00652     
    2014-07-29_09:22:26,47  | tcp-throughput           | 256               | 10.0.1.121       | 194.622     | 169.857     | 205750272           | 179568640           | 1.0082      
    2014-07-29_09:22:26,47  | tcp-throughput           | 256               | 10.0.1.122       | 208.691     | 163.689     | 222461952           | 174489600           | 1.0166      
    2014-07-29_09:22:26,47  | tcp-throughput           | 256               | 10.0.1.123       | 175.217     | 229.875     | 191496192           | 251232256           | 1.04228     
    2014-07-29_09:22:26,47  | tcp-throughput           | 256               | 10.0.1.124       | 99.4907     | 111.342     | 107282432           | 120061952           | 1.02836     
    2014-07-29_09:22:26,47  | tcp-throughput           | 256               | average          | 169.505     | 168.691     | 181747712           | 181338112           | 1.02386     
    2014-07-29_09:22:28,111 | tcp-throughput           | 512               | 10.0.1.121       | 280.433     | 165.068     | 300613632           | 176947200           | 1.0223      
    2014-07-29_09:22:28,111 | tcp-throughput           | 512               | 10.0.1.122       | 279.88      | 282.49      | 302120960           | 304939008           | 1.02946     
    2014-07-29_09:22:28,111 | tcp-throughput           | 512               | 10.0.1.123       | 259.644     | 362.537     | 284033024           | 396591104           | 1.04326     
    2014-07-29_09:22:28,111 | tcp-throughput           | 512               | 10.0.1.124       | 107.341     | 113.488     | 116162560           | 122814464           | 1.03205     
    2014-07-29_09:22:28,111 | tcp-throughput           | 512               | average          | 231.824     | 230.896     | 250732544           | 250322944           | 1.03177     
    2014-07-29_09:22:30,199 | tcp-throughput           | 640               | 10.0.1.121       | 188.804     | 208.925     | 201392128           | 222855168           | 1.01726     
    2014-07-29_09:22:30,199 | tcp-throughput           | 640               | 10.0.1.122       | 219.586     | 176.938     | 231309312           | 186384384           | 1.00459     
    2014-07-29_09:22:30,199 | tcp-throughput           | 640               | 10.0.1.123       | 251.777     | 270.53      | 267485184           | 287408128           | 1.01317     
    2014-07-29_09:22:30,199 | tcp-throughput           | 640               | 10.0.1.124       | 110.082     | 111.461     | 117637120           | 119111680           | 1.01913     
    2014-07-29_09:22:30,199 | tcp-throughput           | 640               | average          | 192.562     | 191.963     | 204455936           | 203939840           | 1.01354     
    2014-07-29_09:22:32,288 | tcp-throughput           | 768               | 10.0.1.121       | 216.375     | 214.319     | 227672064           | 225509376           | 1.00347     
    2014-07-29_09:22:32,288 | tcp-throughput           | 768               | 10.0.1.122       | 182.414     | 169.226     | 193069056           | 179109888           | 1.00938     
    2014-07-29_09:22:32,288 | tcp-throughput           | 768               | 10.0.1.123       | 227.762     | 232.579     | 247889920           | 253132800           | 1.03795     
    2014-07-29_09:22:32,288 | tcp-throughput           | 768               | 10.0.1.124       | 103.602     | 111.219     | 111869952           | 120094720           | 1.02978     
    2014-07-29_09:22:32,288 | tcp-throughput           | 768               | average          | 182.538     | 181.836     | 195125248           | 194461696           | 1.02015     
    2014-07-29_09:22:34,378 | tcp-throughput           | 1024              | 10.0.1.121       | 157.504     | 130.241     | 166592512           | 137756672           | 1.0087      
    2014-07-29_09:22:34,378 | tcp-throughput           | 1024              | 10.0.1.122       | 180.39      | 305.503     | 193232896           | 327254016           | 1.02157     
    2014-07-29_09:22:34,378 | tcp-throughput           | 1024              | 10.0.1.123       | 307.718     | 198.837     | 337281024           | 217939968           | 1.0453      
    2014-07-29_09:22:34,378 | tcp-throughput           | 1024              | 10.0.1.124       | 100.45      | 110.48      | 109608960           | 120553472           | 1.04063     
    2014-07-29_09:22:34,378 | tcp-throughput           | 1024              | average          | 186.515     | 186.265     | 2016
  • [Deleted User][Deleted User]
    It appears, (nodeSetNotifier: node v_vabs_node0002 left the cluster.)
    Do you have the logs on that node, maybe it will shed some light on why that node left
  • Vaibhav_GuptaVaibhav_Gupta
    Below Logs of node v_vabs_node0001 
    FYI: v_vabs_node0001 went down 



    ,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'2014-07-29 01:39:44.192 Init Session:0x7f51b8017970-b0000000886cdb [Txn] <INFO> Begin Txn: b0000000886cdb 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
    2014-07-29 01:39:44.193 Init Session:0x7f51b8024960-b0000000886ce6 [Txn] <INFO> Begin Txn: b0000000886ce6 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
    2014-07-29 01:39:44.193 Init Session:0x7f51b800fb10-b0000000886ce8 [Txn] <INFO> Begin Txn: b0000000886ce8 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
    2014-07-29 01:39:44.196 Init Session:0x7f51b8015fb0-b0000000886cf5 [Txn] <INFO> Begin Txn: b0000000886cf5 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
    2014-07-29 01:39:44.196 Init Session:0x7f51b8012c70-b0000000886ce9 [Txn] <INFO> Begin Txn: b0000000886ce9 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
    2014-07-29 01:39:44.196 Init Session:0x7f51b801a420-b0000000886ce7 [Txn] <INFO> Begin Txn: b0000000886ce7 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
    2014-07-29 01:39:44.199 Init Session:0x7f51b80214a0-b0000000886ceb [Txn] <INFO> Begin Txn: b0000000886ceb 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
    2014-07-29 01:39:44.199 Init Session:0x7f51b8010050-b0000000886cf1 [Txn] <INFO> Begin Txn: b0000000886cf1 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
    2014-07-29 01:39:44.200 Init Session:0x7f51b800f5d0-b0000000886cf3 [Txn] <INFO> Begin Txn: b0000000886cf3 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
    2014-07-29 01:39:44.201 Init Session:0x7f51b80242d0-b0000000886cf7 [Txn] <INFO> Begin Txn: b0000000886cf7 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
    2014-07-29 01:39:44.201 Init Session:0x7f51b801c7c0-b0000000886cf2 [Txn] <INFO> Begin Txn: b0000000886cf2 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
    2014-07-29 01:39:44.201 Init Session:0x7f51b800f2e0-b0000000886cf6 [Txn] <INFO> Begin Txn: b0000000886cf6 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
    2014-07-29 01:39:44.337 Init Session:0x7f51b8014c60-b0000000886cb8 [Session] <INFO> [Query] TX:b0000000886cb8(bigdata012.vabs-13592:0xb7ec5) COMMIT
    2014-07-29 01:39:45.298 DistCall Dispatch:0x7f51c802cfb0-a000000047d0e8 [Catalog] <INFO> applyTransaction: Received catalog update, current catalog (TRANSACTION-a000000047d0e8 version=0)
    2014-07-29 01:39:45.301 DistCall Dispatch:0x7f51c802cfb0-a000000047d0e8 [Catalog] <INFO> applyTransaction: Complete. New version = 0x5e39a5b
    2014-07-29 01:39:45.301 DistCall Dispatch:0x7f51c802cfb0-a000000047d0e8 [Catalog] <INFO> applyTransaction: globalOidSeq is at 7b7c4cf3
    2014-07-29 01:39:45.302 DistCall Dispatch:0x7f51c802cfb0-a000000047d0e8 [Catalog] <INFO> applyTransaction: Received catalog update, current catalog (TRANSACTION-a000000047d0e8 version=0xce)
    2014-07-29 01:39:45.302 DistCall Dispatch:0x7f51c802cfb0-a000000047d0e8 [Catalog] <INFO> applyTransaction: Complete. New version = 0x5e39a5b
    2014-07-29 01:39:45.302 DistCall Dispatch:0x7f51c802cfb0-a000000047d0e8 [Catalog] <INFO> applyTransaction: globalOidSeq is at 7b7c4cf3
    2014-07-29 01:39:45.302 DistCall Dispatch:0x7f51c802cfb0-a000000047d0e8 [Txn] <INFO> Starting Commit: Txn: a000000047d0e8 'do_analyze_row_count'
    2014-07-29 01:39:45.690 Init Session:0x7f51b8024fd0 <LOG> @v_vabs_node0002: 00000/2705: Connection received: host=10.0.1.120 port=59753 (connCnt 36)
    2014-07-29 01:39:45.690 DistCall Dispatch:0x7f51c802cfb0 [Txn] <INFO> Commit Complete: Txn: a000000047d0e8 at epoch 0x2f856e2
    2014-07-29 01:39:45.690 Init Session:0x7f51b8024fd0 <LOG> @v_vabs_node0002: 00000/4686: sendAuthRequest: user=pentaho database=vabs host=10.0.1.120 authType=3
    2014-07-29 01:39:51.319 Init Session:0x7f51b8014c60-b0000000886cb8 [Txn] <INFO> Starting Commit: Txn: b0000000886cb8 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
    2014-07-29 01:39:52.929 Init Session:0x7f51b8025640-b0000000886cbb [Session] <INFO> [Query] TX:b0000000886cbb(bigdata012.vabs-13592:0xb7ec7) COMMIT
    2014-07-29 01:39:52.932 Init Session:0x7f51b8014c60 [Txn] <INFO> Commit Complete: Txn: b0000000886cb8 at epoch 0x2f856e2
    2014-07-29 01:39:54.001 RebalanceCluster:0x7f51f0041fe0 [Util] <INFO> Task 'RebalanceCluster' enabled
    2014-07-29 01:39:56.948 Init Session:0x7f51b8014c60 <LOG> @v_vabs_node0002: 00000/4719: Session bigdata012.vabs-13592:0xb7ec5 ended; closing connection (connCnt 36)
    2014-07-29 01:39:56.988 Init Session:0x7f51b8025640-b0000000886cbb [Txn] <INFO> Starting Commit: Txn: b0000000886cbb 'COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT'
    2014-07-29 01:39:57.040 Init Session:0x7f51b8025640 [Txn] <INFO> Commit Complete: Txn: b0000000886cbb at epoch 0x2f856e3
    2014-07-29 01:39:58.344 Init Session:0x7f51b800eb30 <LOG> @v_vabs_node0002: 00000/2705: Connection received: host=10.0.1.120 port=59755 (connCnt 36)
    2014-07-29 01:39:58.345 Init Session:0x7f51b800eb30 <LOG> @v_vabs_node0002: 00000/4686: sendAuthRequest: user=pentaho database=vabs host=10.0.1.120 authType=3
    2014-07-29 01:39:59.822 Init Session:0x7f51b8024fd0-b0000000886d0f [Txn] <INFO> Begin Txn: b0000000886d0f 'check_login_history'
    2014-07-29 01:39:59.823 Init Session:0x7f51b8024fd0-b0000000886d0f [Txn] <INFO> Rollback Txn: b0000000886d0f 'check_login_history'
    2014-07-29 01:39:59.824 Init Session:0x7f51b8024fd0 <LOG> @v_vabs_node0002: 00000/4686: sendAuthRequest: user=pentaho database=vabs host=10.0.1.120 authType=0
    2014-07-29 01:39:59.825 Init Session:0x7f51b8024fd0 <LOG> @v_vabs_node0002: 00000/2703: Connection authenticated: user=pentaho database=vabs host=10.0.1.120
    2014-07-29 01:39:59.826 Init Session:0x7f51b8024fd0 <LOG> @v_vabs_node0002: 00000/2609: Client pid: 14779
    2014-07-29 01:39:59.826 TM Mergeout(02):0x7f51f0041fe0-b0000000886d10 [Txn] <INFO> Begin Txn: b0000000886d10 'Mergeout: Tuple Mover'
    2014-07-29 01:39:59.828 Init Session:0x7f51b8024fd0 <LOG> @v_vabs_node0002: 00000/2610: Client requested binary data transfer
    2014-07-29 01:39:59.829 TM Mergeout(03):0x7f51f0042be0-b0000000886d11 [Txn] <INFO> Begin Txn: b0000000886d11 'Mergeout: Tuple Mover'
    2014-07-29 01:39:59.834 Init Session:0x7f51b800eb30-b0000000886d12 [Txn] <INFO> Begin Txn: b0000000886d12 'check_login_history'
    2014-07-29 01:39:59.892 Init Session:0x7f51b800eb30-b0000000886d12 [Txn] <INFO> Rollback Txn: b0000000886d12 'check_login_history'
    2014-07-29 01:39:59.943 Init Session:0x7f51b800eb30 <LOG> @v_vabs_node0002: 00000/4686: sendAuthRequest: user=pentaho database=vabs host=10.0.1.120 authType=0
    2014-07-29 01:39:59.944 Init Session:0x7f51b800eb30 <LOG> @v_vabs_node0002: 00000/2703: Connection authenticated: user=pentaho database=vabs host=10.0.1.120
    2014-07-29 01:39:59.944 Init Session:0x7f51b800eb30 <LOG> @v_vabs_node0002: 00000/2609: Client pid: 14779
    2014-07-29 01:39:59.945 Init Session:0x7f51b800eb30 <LOG> @v_vabs_node0002: 00000/2610: Client requested binary data transfer
    2014-07-29 01:40:00.000 Init Session:0x7f51b8024fd0 [Session] <INFO> [PQuery] TX:0(bigdata012.vabs-13592:0xb7efa) select version_internal()
    2014-07-29 01:40:00.006 Init Session:0x7f51b800eb30 [Session] <INFO> [PQuery] TX:0(bigdata012.vabs-13592:0xb7f04) select version_internal()
    2014-07-29 01:40:00.006 TM Mergeout(01):0x7f51f0042e50-b0000000886d13 [Txn] <INFO> Begin Txn: b0000000886d13 'Mergeout: Tuple Mover'
    2014-07-29 01:40:00.007 Init Session:0x7f51b8024fd0 <LOG> @v_vabs_node0002: 00000/3316: Executing statement: 'select version_internal()'
    2014-07-29 01:40:00.028 Init Session:0x7f51b800eb30 <LOG> @v_vabs_node0002: 00000/3316: Executing statement: 'select version_internal()'
    2014-07-29 01:40:00.029 Init Session:0x7f51b8024fd0 [Session] <INFO> [Query] TX:0(bigdata012.vabs-13592:0xb7efa) COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_unique,impressions_unique,story_adds,story_adds_unique,negative_feedback,negative_feedback_unique,date_retrieved,object_id,parent_object_id,parent_object_type,object_type,location_id,posts_added,posts_removed,views_unique,views_unique_added,videos_view_added,videos_view_unique,videos_view_unique_added,videos_play_added,visitors_added,impressions_viral,impressions_viral_unique,external_Shares,is_public_data,photo_views,link_clicks,other_clicks) FROM STDIN NATIVE VARCHAR RETURNREJECTED AUTO NO COMMIT
    2014-07-29 01:40:00.056 Init Session:0x7f51b800eb30 [Session] <INFO> [Query] TX:0(bigdata012.vabs-13592:0xb7f04) COPY ANALYTICSNEW_STAGING.SocialObject_Stats (posts,medias,images,notes,topics,listed,views,likes,dislikes,comments,shares,mentions,visitors,reach,period,stories,storytellers,channel_id,segment_type,segment_value,stats_date,videos,owned_videos,favorites,favorites_added,favorites_removed,videos_view,videos_play,video_play_duration_minutes,average_view_duration_minutes,average_view_percentage,click_through_rate,close_rate,viewer_percentage,customer_id,medias_added,medias_removed,videos_added,videos_removed,owned_videos_added,owned_videos_removed,images_added,images_removed,topics_added,topics_removed,listed_added,listed_removed,notes_added,notes_removed,views_added,likes_added,likes_removed,dislikes_added,comments_added,comments_removed,shares_added,mentions_added,mentions_removed,followers_reach,fb_shares,twitter_shares,consumptions,consumptions_unique,engaged_users,impressions,impressions_followers,impressions_followers_paid,impressions_followers_paid_unique,impressions_followers_unique,impressions_organic,impressions_organic_unique,impressions_paid,impressions_paid_uniqu
  • colin_loghincolin_loghin ✭
    This message (NETWORK change with 1 VS sets) usually means that one node was not able to heartbeat with the other nodes , such as when network separation occurs. I am guessing the vertica cluster setup did not use 1 private set of NICs for cluster setup ( -s parameter in install_vertica) and COPY statement network traffic is impacting internal heartbeat traffic. This would explain why node goes down in this situation. 

    Reassess the network setup, reinstall vertica using private ips to seperate traffic, 
    check if you have large latency ( it may be fluctuating) and use default UDP assuming all hosts are on the same subnet. 

  • Vaibhav_GuptaVaibhav_Gupta
    The vertica cluster is on Vms (Vmware). The below command is use for vertica install
    /opt/vertica/sbin/install_vertica -s bigdata-stg-011.vabs-staging.com,bigdata-stg-012.vabs-staging.com,bigdata-stg-013.vabs-staging.com,bigdata-stg-014.vabs-staging.com -l /data/dbadmin -r /tmp/vertica-6.1.2-0.x86_64.RHEL5.rpm -d /data/vertica/ -C -N -T -S default -Y -L vlicense.dat

    For latency i tried vnetperf. Output in 2nd comment 

    Below my spread file

    Spread_Segment 10.0.1.121:4803 {
      N010000001121    10.0.1.121 {
        10.0.1.121
        127.0.0.1
      }
    }
    Spread_Segment 10.0.1.122:4803 {
      N010000001122    10.0.1.122 {
        10.0.1.122
        127.0.0.1
      }
    }
    Spread_Segment 10.0.1.123:4803 {
      N010000001123    10.0.1.123 {
        10.0.1.123
        127.0.0.1
      }
    }
    Spread_Segment 10.0.1.124:4803 {
      N010000001124    10.0.1.124 {
        10.0.1.124
        127.0.0.1
      }
    }
    EventLogFile = /dev/null
    EventTimeStamp = "[%a %d %b %Y %H:%M:%S]"
    DaemonUser = spread
    DaemonGroup = verticadba
    DebugFlags = { EXIT }


Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Format
SpoilerCodeQuote
Heading 2Heading 1
Emoji
:):D:(;):/:o:s:p:'(:|B):#:*<3o:)>:)
Url
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file
Home • General DiscussionComment As ...