Failed to create new database
Hi all,
I have 10 nodes Vertica environment with active database. I want to create other database ... so I stopped the current database and tried to create a new one. (I tried to create from the admintools, console manager & by script ... same failure)
this is want I'm doing:
/opt/vertica/bin/admintools --tool create_db -s 172.19.224.198,172.19.224.193,172.19.224.194,172.19.224.199,172.19.224.195,172.19.224.200,172.19.224.196,172.19.224.201,172.19.224.197,172.19.224.202 -d vertica_polonium_mtc -c /db/catalog -D /db/data -p PassworD
Distributing changes to cluster.
172.19.224.198 OK [vertica][(7, 1, 1)][010][x86_64]
172.19.224.193 OK [vertica][(7, 1, 1)][010][x86_64]
172.19.224.194 OK [vertica][(7, 1, 1)][010][x86_64]
172.19.224.199 OK [vertica][(7, 1, 1)][010][x86_64]
172.19.224.195 OK [vertica][(7, 1, 1)][010][x86_64]
172.19.224.200 OK [vertica][(7, 1, 1)][010][x86_64]
172.19.224.196 OK [vertica][(7, 1, 1)][010][x86_64]
172.19.224.201 OK [vertica][(7, 1, 1)][010][x86_64]
172.19.224.197 OK [vertica][(7, 1, 1)][010][x86_64]
172.19.224.202 OK [vertica][(7, 1, 1)][010][x86_64]
Checking full connectivity
Creating database vertica_polonium_mtc
Starting bootstrap node v_vertica_polonium_mtc_node0001 (172.19.224.198)
Starting nodes:
v_vertica_polonium_mtc_node0001 (172.19.224.198)
Starting Vertica on all nodes. Please wait, databases with large catalogs may take a while to initialize.
Node Status: v_vertica_polonium_mtc_node0001: (INITIALIZING)
Node Status: v_vertica_polonium_mtc_node0001: (INITIALIZING)
Node Status: v_vertica_polonium_mtc_node0001: (UP)
Creating database nodes
Creating node v_vertica_polonium_mtc_node0002 (host 172.19.224.193)
Creating node v_vertica_polonium_mtc_node0003 (host 172.19.224.194)
Creating node v_vertica_polonium_mtc_node0004 (host 172.19.224.199)
Creating node v_vertica_polonium_mtc_node0005 (host 172.19.224.195)
Creating node v_vertica_polonium_mtc_node0006 (host 172.19.224.200)
Creating node v_vertica_polonium_mtc_node0007 (host 172.19.224.196)
Creating node v_vertica_polonium_mtc_node0008 (host 172.19.224.201)
Creating node v_vertica_polonium_mtc_node0009 (host 172.19.224.197)
Creating node v_vertica_polonium_mtc_node0010 (host 172.19.224.202)
Generating new configuration information
Starting all nodes
Starting nodes:
v_vertica_polonium_mtc_node0002 (172.19.224.193)
v_vertica_polonium_mtc_node0003 (172.19.224.194)
v_vertica_polonium_mtc_node0004 (172.19.224.199)
v_vertica_polonium_mtc_node0005 (172.19.224.195)
v_vertica_polonium_mtc_node0006 (172.19.224.200)
v_vertica_polonium_mtc_node0007 (172.19.224.196)
v_vertica_polonium_mtc_node0008 (172.19.224.201)
v_vertica_polonium_mtc_node0009 (172.19.224.197)
v_vertica_polonium_mtc_node0010 (172.19.224.202)
Starting Vertica on all nodes. Please wait, databases with large catalogs may take a while to initialize.
Node Status: v_vertica_polonium_mtc_node0001: (UP) v_vertica_polonium_mtc_node0002: (DOWN) v_vertica_polonium_mtc_node0003: (DOWN) v_vertica_polonium_mtc_node0004: (DOWN) v_vertica_polonium_mtc_node0005: (DOWN) v_vertica_polonium_mtc_node0006: (DOWN) v_vertica_polonium_mtc_node0007: (DOWN) v_vertica_polonium_mtc_node0008: (DOWN) v_vertica_polonium_mtc_node0009: (DOWN) v_vertica_polonium_mtc_node0010: (DOWN)
Node Status: v_vertica_polonium_mtc_node0001: (UP) v_vertica_polonium_mtc_node0002: (DOWN) v_vertica_polonium_mtc_node0003: (DOWN) v_vertica_polonium_mtc_node0004: (DOWN) v_vertica_polonium_mtc_node0005: (DOWN) v_vertica_polonium_mtc_node0006: (DOWN) v_vertica_polonium_mtc_node0007: (DOWN) v_vertica_polonium_mtc_node0008: (DOWN) v_vertica_polonium_mtc_node0009: (DOWN) v_vertica_polonium_mtc_node0010: (DOWN)
Node Status: v_vertica_polonium_mtc_node0001: (UP) v_vertica_polonium_mtc_node0002: (DOWN) v_vertica_polonium_mtc_node0003: (DOWN) v_vertica_polonium_mtc_node0004: (DOWN) v_vertica_polonium_mtc_node0005: (DOWN) v_vertica_polonium_mtc_node0006: (DOWN) v_vertica_polonium_mtc_node0007: (DOWN) v_vertica_polonium_mtc_node0008: (DOWN) v_vertica_polonium_mtc_node0009: (DOWN) v_vertica_polonium_mtc_node0010: (DOWN)
Node Status: v_vertica_polonium_mtc_node0001: (UP) v_vertica_polonium_mtc_node0002: (DOWN) v_vertica_polonium_mtc_node0003: (DOWN) v_vertica_polonium_mtc_node0004: (DOWN) v_vertica_polonium_mtc_node0005: (DOWN) v_vertica_polonium_mtc_node0006: (DOWN) v_vertica_polonium_mtc_node0007: (DOWN) v_vertica_polonium_mtc_node0008: (DOWN) v_vertica_polonium_mtc_node0009: (DOWN) v_vertica_polonium_mtc_node0010: (DOWN)
Node Status: v_vertica_polonium_mtc_node0001: (UP) v_vertica_polonium_mtc_node0002: (DOWN) v_vertica_polonium_mtc_node0003: (DOWN) v_vertica_polonium_mtc_node0004: (DOWN) v_vertica_polonium_mtc_node0005: (DOWN) v_vertica_polonium_mtc_node0006: (DOWN) v_vertica_polonium_mtc_node0007: (DOWN) v_vertica_polonium_mtc_node0008: (DOWN) v_vertica_polonium_mtc_node0009: (DOWN) v_vertica_polonium_mtc_node0010: (DOWN)
Node Status: v_vertica_polonium_mtc_node0001: (UP) v_vertica_polonium_mtc_node0002: (DOWN) v_vertica_polonium_mtc_node0003: (DOWN) v_vertica_polonium_mtc_node0004: (DOWN) v_vertica_polonium_mtc_node0005: (DOWN) v_vertica_polonium_mtc_node0006: (DOWN) v_vertica_polonium_mtc_node0007: (DOWN) v_vertica_polonium_mtc_node0008: (DOWN) v_vertica_polonium_mtc_node0009: (DOWN) v_vertica_polonium_mtc_node0010: (DOWN)
Node Status: v_vertica_polonium_mtc_node0001: (UP) v_vertica_polonium_mtc_node0002: (DOWN) v_vertica_polonium_mtc_node0003: (DOWN) v_vertica_polonium_mtc_node0004: (DOWN) v_vertica_polonium_mtc_node0005: (DOWN) v_vertica_polonium_mtc_node0006: (DOWN) v_vertica_polonium_mtc_node0007: (DOWN) v_vertica_polonium_mtc_node0008: (DOWN) v_vertica_polonium_mtc_node0009: (DOWN) v_vertica_polonium_mtc_node0010: (DOWN)
Node Status: v_vertica_polonium_mtc_node0001: (UP) v_vertica_polonium_mtc_node0002: (DOWN) v_vertica_polonium_mtc_node0003: (DOWN) v_vertica_polonium_mtc_node0004: (DOWN) v_vertica_polonium_mtc_node0005: (DOWN) v_vertica_polonium_mtc_node0006: (DOWN) v_vertica_polonium_mtc_node0007: (DOWN) v_vertica_polonium_mtc_node0008: (DOWN) v_vertica_polonium_mtc_node0009: (DOWN) v_vertica_polonium_mtc_node0010: (DOWN)
Node Status: v_vertica_polonium_mtc_node0001: (UP) v_vertica_polonium_mtc_node0002: (DOWN) v_vertica_polonium_mtc_node0003: (DOWN) v_vertica_polonium_mtc_node0004: (DOWN) v_vertica_polonium_mtc_node0005: (DOWN) v_vertica_polonium_mtc_node0006: (DOWN) v_vertica_polonium_mtc_node0007: (DOWN) v_vertica_polonium_mtc_node0008: (DOWN) v_vertica_polonium_mtc_node0009: (DOWN) v_vertica_polonium_mtc_node0010: (DOWN)
Node Status: v_vertica_polonium_mtc_node0001: (UP) v_vertica_polonium_mtc_node0002: (DOWN) v_vertica_polonium_mtc_node0003: (DOWN) v_vertica_polonium_mtc_node0004: (DOWN) v_vertica_polonium_mtc_node0005: (DOWN) v_vertica_polonium_mtc_node0006: (DOWN) v_vertica_polonium_mtc_node0007: (DOWN) v_vertica_polonium_mtc_node0008: (DOWN) v_vertica_polonium_mtc_node0009: (DOWN) v_vertica_polonium_mtc_node0010: (DOWN)
Nodes UP: v_vertica_polonium_mtc_node0001
Nodes DOWN: v_vertica_polonium_mtc_node0006, v_vertica_polonium_mtc_node0007, v_vertica_polonium_mtc_node0004, v_vertica_polonium_mtc_node0005, v_vertica_polonium_mtc_node0002, v_vertica_polonium_mtc_node0003, v_vertica_polonium_mtc_node0010, v_vertica_polonium_mtc_node0008, v_vertica_polonium_mtc_node0009 (may be still initializing).
It is suggested that you continue waiting.
Do you want to continue waiting? (yes/no) [yes]
If I check the admintools.conf (when vertica_polonium_lcd is the old database, vertica_polonium_mtc is the new database)
[root@fmvvrtcbi-l01 other]# vi /opt/vertica/config/admintools.conf
default_base = /data/server/vertica
show_hostnames = False
format = 3
last_port = 5433
controlmode = pt2pt
controlsubnet = 192.168.1.255
spreadlog = False
largecluster = 10
tmp_dir = /tmp
[Cluster]
hosts = 172.19.224.198,172.19.224.193,172.19.224.194,172.19.224.199,172.19.224.195,172.19.224.200,172.19.224.196,172.19.224.201,172.19.224.197,172.19.224.202
spread_hosts = fmvvrtcbi-l01.5min.aol.com,fmvvrtcbi-l02.5min.aol.com,fmvvrtcbi-l03.5min.aol.com,fmvvrtcbi-l04.5min.aol.com,fmvvrtcbi-l05.5min.aol.com,fmvvrtcbi-l06.5min.aol.com,fmvvrtcbi-l07.5min.aol.com,fmvvrtcbi-l08.5min.aol.com,fmvvrtcbi-l09.5min.aol.com,fmvvrtcbi-l10.5min.aol.com
[Nodes]
node1 = 172.19.224.198,/db/catalog,/db/data
node2 = 172.19.224.193,/db/data,/db/catalog
node3 = 172.19.224.194,/db/data,/db/catalog
node4 = 172.19.224.199,/db/data,/db/catalog
node5 = 172.19.224.195,/db/data,/db/catalog
node6 = 172.19.224.200,/db/data,/db/catalog
node7 = 172.19.224.196,/db/data,/db/catalog
node8 = 172.19.224.201,/db/data,/db/catalog
node9 = 172.19.224.197,/db/data,/db/catalog
node10 = 172.19.224.202,/db/data,/db/catalog
v_vertica_polonium_lcd_node0001 = 172.19.224.198,/db/catalog,/db/data
v_vertica_polonium_lcd_node0002 = 172.19.224.193,/db/catalog,/db/data
v_vertica_polonium_lcd_node0003 = 172.19.224.194,/db/catalog,/db/data
v_vertica_polonium_lcd_node0004 = 172.19.224.199,/db/catalog,/db/data
v_vertica_polonium_lcd_node0005 = 172.19.224.195,/db/catalog,/db/data
v_vertica_polonium_lcd_node0006 = 172.19.224.200,/db/catalog,/db/data
v_vertica_polonium_lcd_node0007 = 172.19.224.196,/db/catalog,/db/data
v_vertica_polonium_lcd_node0008 = 172.19.224.201,/db/catalog,/db/data
v_vertica_polonium_lcd_node0009 = 172.19.224.197,/db/catalog,/db/data
v_vertica_polonium_lcd_node0010 = 172.19.224.202,/db/catalog,/db/data
v_vertica_polonium_mtc_node0001 = 172.19.224.198,/db/catalog,/db/data
v_vertica_polonium_mtc_node0002 = 172.19.224.193,/db/catalog,/db/data
v_vertica_polonium_mtc_node0003 = 172.19.224.194,/db/catalog,/db/data
v_vertica_polonium_mtc_node0004 = 172.19.224.199,/db/catalog,/db/data
v_vertica_polonium_mtc_node0005 = 172.19.224.195,/db/catalog,/db/data
v_vertica_polonium_mtc_node0006 = 172.19.224.200,/db/catalog,/db/data
v_vertica_polonium_mtc_node0007 = 172.19.224.196,/db/catalog,/db/data
v_vertica_polonium_mtc_node0008 = 172.19.224.201,/db/catalog,/db/data
v_vertica_polonium_mtc_node0009 = 172.19.224.197,/db/catalog,/db/data
v_vertica_polonium_mtc_node0010 = 172.19.224.202,/db/catalog,/db/data
[Database:vertica_polonium_lcd]
restartpolicy = ksafe
port = 5433
path = /db/catalog/vertica_polonium_lcd
nodes = v_vertica_polonium_lcd_node0001,v_vertica_polonium_lcd_node0002,v_vertica_polonium_lcd_node0003,v_vertica_polonium_lcd_node0004,v_vertica_polonium_lcd_node0005,v_vertica_polonium_lcd_node0006,v_vertica_polonium_lcd_node0007,v_vertica_polonium_lcd_node0008,v_vertica_polonium_lcd_node0009,v_vertica_polonium_lcd_node0010
And the Vertica.log tail:
[root@fmvvrtcbi-l01 other]# vi /db/catalog/vertica_polonium_mtc/v_vertica_polonium_mtc_node0001_catalog/vertica.log
2015-11-15 11:37:49.510 Init Session:0x7fd9a0011290 <LOG> @v_vertica_polonium_mtc_node0001: 08V01/3533: Incomplete startup packet
HINT: This is often caused by load balancer health checks. These messages may be disabled by running the command: select set_config_parameter('WarnOnIncompleteStartupPacket', 0);
2015-11-15 11:37:49.854 Init Session:0x7fd9a0010fc0 <LOG> @v_vertica_polonium_mtc_node0001: 00000/2705: Connection received: host=172.29.24.161 port=46454 (connCnt 1)
2015-11-15 11:37:49.854 Init Session:0x7fd9a0010fc0 <LOG> @v_vertica_polonium_mtc_node0001: 08V01/3533: Incomplete startup packet
HINT: This is often caused by load balancer health checks. These messages may be disabled by running the command: select set_config_parameter('WarnOnIncompleteStartupPacket', 0);
2015-11-15 11:37:50.887 Init Session:0x7fd9a0011290 <LOG> @v_vertica_polonium_mtc_node0001: 00000/2705: Connection received: host=172.29.108.145 port=63610 (connCnt 1)
2015-11-15 11:37:50.887 Init Session:0x7fd9a0011290 <LOG> @v_vertica_polonium_mtc_node0001: 08V01/3533: Incomplete startup packet
HINT: This is often caused by load balancer health checks. These messages may be disabled by running the command: select set_config_parameter('WarnOnIncompleteStartupPacket', 0);
2015-11-15 11:37:52.003 AnalyzeRowCount:0x7fd9b00149e0-a0000000000051 [Txn] <INFO> Begin Txn: a0000000000051 'getRowCountsForProj'
2015-11-15 11:37:52.004 AnalyzeRowCount:0x7fd9b00149e0-a0000000000051 [Txn] <INFO> Rollback Txn: a0000000000051 'getRowCountsForProj'
2015-11-15 11:37:52.004 AnalyzeRowCount:0x7fd9b00149e0 <WARNING> @v_vertica_polonium_mtc_node0001: 01000/4539: Received no response from v_vertica_polonium_mtc_node0002 in GetRowCountMessage
2015-11-15 11:37:52.004 AnalyzeRowCount:0x7fd9b00149e0 <WARNING> @v_vertica_polonium_mtc_node0001: 01000/4539: Received no response from v_vertica_polonium_mtc_node0003 in GetRowCountMessage
2015-11-15 11:37:52.004 AnalyzeRowCount:0x7fd9b00149e0 <WARNING> @v_vertica_polonium_mtc_node0001: 01000/4539: Received no response from v_vertica_polonium_mtc_node0004 in GetRowCountMessage
2015-11-15 11:37:52.004 AnalyzeRowCount:0x7fd9b00149e0 <WARNING> @v_vertica_polonium_mtc_node0001: 01000/4539: Received no response from v_vertica_polonium_mtc_node0005 in GetRowCountMessage
2015-11-15 11:37:52.004 AnalyzeRowCount:0x7fd9b00149e0 <WARNING> @v_vertica_polonium_mtc_node0001: 01000/4539: Received no response from v_vertica_polonium_mtc_node0006 in GetRowCountMessage
2015-11-15 11:37:52.004 AnalyzeRowCount:0x7fd9b00149e0 <WARNING> @v_vertica_polonium_mtc_node0001: 01000/4539: Received no response from v_vertica_polonium_mtc_node0007 in GetRowCountMessage
2015-11-15 11:37:52.004 AnalyzeRowCount:0x7fd9b00149e0 <WARNING> @v_vertica_polonium_mtc_node0001: 01000/4539: Received no response from v_vertica_polonium_mtc_node0008 in GetRowCountMessage
2015-11-15 11:37:52.005 AnalyzeRowCount:0x7fd9b00149e0 <WARNING> @v_vertica_polonium_mtc_node0001: 01000/4539: Received no response from v_vertica_polonium_mtc_node0009 in GetRowCountMessage
2015-11-15 11:37:52.005 AnalyzeRowCount:0x7fd9b00149e0 <WARNING> @v_vertica_polonium_mtc_node0001: 01000/4539: Received no response from v_vertica_polonium_mtc_node0010 in GetRowCountMessage
2015-11-15 11:37:52.005 AnalyzeRowCount:0x7fd9b00149e0 [Util] <INFO> Task 'AnalyzeRowCount' enabled
2015-11-15 11:37:53.030 Init Session:0x7fd9a0010fc0 <LOG> @v_vertica_polonium_mtc_node0001: 00000/2705: Connection received: host=172.19.224.198 port=4078 (connCnt 1)
2015-11-15 11:37:53.030 Init Session:0x7fd9a0010fc0 <LOG> @v_vertica_polonium_mtc_node0001: 00000/4540: Received SSL negotiation startup packet
2015-11-15 11:37:53.030 Init Session:0x7fd9a0010fc0 <LOG> @v_vertica_polonium_mtc_node0001: 00000/4691: Sending SSL negotiation response 'N'
2015-11-15 11:37:53.030 Init Session:0x7fd9a0010fc0 <FATAL> @v_vertica_polonium_mtc_node0001: {SessionRun} 57V03/5785: Cluster Status Request by 172.19.224.198:4078
HINT: Cluster State: vertica_polonium_mtc
UP: 1 of 10 (v_vertica_polonium_mtc_node0001)
----
LOCATION: initSession, /scratch_a/release/7003/vbuild/vertica/Session/ClientSession.cpp:436
2015-11-15 11:37:53.112 Init Session:0x7fd9a0011290 <LOG> @v_vertica_polonium_mtc_node0001: 00000/2705: Connection received: host=172.19.224.198 port=4088 (connCnt 1)
2015-11-15 11:37:53.112 Init Session:0x7fd9a0011290 <LOG> @v_vertica_polonium_mtc_node0001: 00000/4540: Received SSL negotiation startup packet
2015-11-15 11:37:53.112 Init Session:0x7fd9a0011290 <LOG> @v_vertica_polonium_mtc_node0001: 00000/4691: Sending SSL negotiation response 'N'
2015-11-15 11:37:53.113 Init Session:0x7fd9a0011290 <FATAL> @v_vertica_polonium_mtc_node0001: {SessionRun} 57V03/5785: Cluster Status Request by 172.19.224.198:4088
HINT: Cluster State: vertica_polonium_mtc
UP: 1 of 10 (v_vertica_polonium_mtc_node0001)
----
LOCATION: initSession, /scratch_a/release/7003/vbuild/vertica/Session/ClientSession.cpp:436
2015-11-15 11:37:54.990 Init Session:0x7fd9a0010fc0 <LOG> @v_vertica_polonium_mtc_node0001: 00000/2705: Connection received: host=172.19.224.198 port=4110 (connCnt 1)
2015-11-15 11:37:54.990 Init Session:0x7fd9a0010fc0 <LOG> @v_vertica_polonium_mtc_node0001: 00000/4540: Received SSL negotiation startup packet
2015-11-15 11:37:54.990 Init Session:0x7fd9a0010fc0 <LOG> @v_vertica_polonium_mtc_node0001: 00000/4691: Sending SSL negotiation response 'N'
2015-11-15 11:37:54.991 Init Session:0x7fd9a0010fc0 <FATAL> @v_vertica_polonium_mtc_node0001: {SessionRun} 57V03/5785: Cluster Status Request by 172.19.224.198:4110
HINT: Cluster State: vertica_polonium_mtc
UP: 1 of 10 (v_vertica_polonium_mtc_node0001)
----
LOCATION: initSession, /scratch_a/release/7003/vbuild/vertica/Session/ClientSession.cpp:436
2015-11-15 11:37:55.084 Init Session:0x7fd9a0011290 <LOG> @v_vertica_polonium_mtc_node0001: 00000/2705: Connection received: host=172.29.24.161 port=27916 (connCnt 1)
2015-11-15 11:37:55.084 Init Session:0x7fd9a0011290 <LOG> @v_vertica_polonium_mtc_node0001: 08V01/3533: Incomplete startup packet
HINT: This is often caused by load balancer health checks. These messages may be disabled by running the command: select set_config_parameter('WarnOnIncompleteStartupPacket', 0);
2015-11-15 11:37:55.886 Init Session:0x7fd9a0010fc0 <LOG> @v_vertica_polonium_mtc_node0001: 00000/2705: Connection received: host=172.29.108.145 port=61381 (connCnt 1)
2015-11-15 11:37:55.886 Init Session:0x7fd9a0010fc0 <LOG> @v_vertica_polonium_mtc_node0001: 08V01/3533: Incomplete startup packet
HINT: This is often caused by load balancer health checks. These messages may be disabled by running the command: select set_config_parameter('WarnOnIncompleteStartupPacket', 0);
2015-11-15 11:37:59.510 Init Session:0x7fd9a0011290 <LOG> @v_vertica_polonium_mtc_node0001: 00000/2705: Connection received: host=172.29.24.161 port=13585 (connCnt 1)
2015-11-15 11:37:59.510 Init Session:0x7fd9a0011290 <LOG> @v_vertica_polonium_mtc_node0001: 08V01/3533: Incomplete startup packet
HINT: This is often caused by load balancer health checks. These messages may be disabled by running the command: select set_config_parameter('WarnOnIncompleteStartupPacket', 0);
2015-11-15 11:38:00.875 Init Session:0x7fd9a0010fc0 <LOG> @v_vertica_polonium_mtc_node0001: 00000/2705: Connection received: host=172.29.108.145 port=63800 (connCnt 1)
2015-11-15 11:38:00.875 Init Session:0x7fd9a0010fc0 <LOG> @v_vertica_polonium_mtc_node0001: 08V01/3533: Incomplete startup packet
HINT: This is often caused by load balancer health checks. These messages may be disabled by running the command: select set_config_parameter('WarnOnIncompleteStartupPacket', 0);
Please please please ... I need your help 
Thank
Comments
Some more details...: this is the vertica.log from one of the nodes: 2015-11-15 11:54:50.002 nameless:0x75e8850 [Catalog] getLocalStorageLocations: no local node 2015-11-15 11:54:51.002 nameless:0x75e8850 [Catalog] getLocalStorageLocations: no local node 2015-11-15 11:54:52.002 nameless:0x75e8850 [Catalog] getLocalStorageLocations: no local node 2015-11-15 11:54:53.002 nameless:0x75e8850 [Catalog] getLocalStorageLocations: no local node 2015-11-15 11:54:53.015 Init Session:0x7f8688010ff0 @[initializing]: 00000/2705: Connection received: host=172.19.224.198 port=30072 (connCnt 1) 2015-11-15 11:54:53.015 Init Session:0x7f8688010ff0 @[initializing]: 00000/4540: Received SSL negotiation startup packet 2015-11-15 11:54:53.015 Init Session:0x7f8688010ff0 @[initializing]: 00000/4691: Sending SSL negotiation response 'N' 2015-11-15 11:54:53.016 Init Session:0x7f8688010ff0 @[initializing]: {SessionRun} 57V03/5785: Cluster Status Request by 172.19.224.198:30072 HINT: Cluster State: vertica_polonium_mtc --Waiting for cluster invitation ---- LOCATION: initSession, /scratch_a/release/7003/vbuild/vertica/Session/ClientSession.cpp:436 2015-11-15 11:54:53.098 Init Session:0x7f8688010ff0 @[initializing]: 00000/2705: Connection received: host=172.19.224.198 port=30082 (connCnt 1) 2015-11-15 11:54:53.098 Init Session:0x7f8688010ff0 @[initializing]: 00000/4540: Received SSL negotiation startup packet 2015-11-15 11:54:53.098 Init Session:0x7f8688010ff0 @[initializing]: 00000/4691: Sending SSL negotiation response 'N' 2015-11-15 11:54:53.098 Init Session:0x7f8688010ff0 @[initializing]: {SessionRun} 57V03/5785: Cluster Status Request by 172.19.224.198:30082 HINT: Cluster State: vertica_polonium_mtc --Waiting for cluster invitation ---- LOCATION: initSession, /scratch_a/release/7003/vbuild/vertica/Session/ClientSession.cpp:436 2015-11-15 11:54:53.507 Init Session:0x7f8688010ff0 @[initializing]: 00000/2705: Connection received: host=172.29.24.161 port=54205 (connCnt 1) 2015-11-15 11:54:53.507 Init Session:0x7f8688010ff0 @[initializing]: 08V01/3533: Incomplete startup packet HINT: This is often caused by load balancer health checks. These messages may be disabled by running the command: select set_config_parameter('WarnOnIncompleteStartupPacket', 0); 2015-11-15 11:54:54.002 nameless:0x75e8850 [Catalog] getLocalStorageLocations: no local node 2015-11-15 11:54:54.880 Init Session:0x7f8688010ff0 @[initializing]: 00000/2705: Connection received: host=172.29.108.145 port=50292 (connCnt 1) 2015-11-15 11:54:54.880 Init Session:0x7f8688010ff0 @[initializing]: 08V01/3533: Incomplete startup packet HINT: This is often caused by load balancer health checks. These messages may be disabled by running the command: select set_config_parameter('WarnOnIncompleteStartupPacket', 0); 2015-11-15 11:54:54.967 Init Session:0x7f8688010ff0 @[initializing]: 00000/2705: Connection received: host=172.19.224.198 port=30104 (connCnt 1) 2015-11-15 11:54:54.967 Init Session:0x7f8688010ff0 @[initializing]: 00000/4540: Received SSL negotiation startup packet 2015-11-15 11:54:54.967 Init Session:0x7f8688010ff0 @[initializing]: 00000/4691: Sending SSL negotiation response 'N' 2015-11-15 11:54:54.967 Init Session:0x7f8688010ff0 @[initializing]: {SessionRun} 57V03/5785: Cluster Status Request by 172.19.224.198:30104 HINT: Cluster State: vertica_polonium_mtc --Waiting for cluster invitation ---- LOCATION: initSession, /scratch_a/release/7003/vbuild/vertica/Session/ClientSession.cpp:436 And the startup.log [root@fmvvrtcbi-l02 catalog]# vi /db/catalog/vertica_polonium_mtc/v_vertica_polonium_mtc_node0002_catalog/startup.log { "node" : "v_vertica_polonium_mtc_node0002", "stage" : "Connecting to Spread", "text" : "Connecting to spread 4803", "timestamp" : "2015-11-15 11:24:26.767" } { "node" : "v_vertica_polonium_mtc_node0002", "stage" : "Waiting for Cluster Invite", "text" : "Prepare to be invited", "timestamp" : "2015-11-15 11:24:28.000" } { "node" : "v_vertica_polonium_mtc_node0002", "stage" : "Waiting for Cluster Invite", "text" : "Prepare to be invited", "timestamp" : "2015-11-15 11:24:30.000" } { "node" : "v_vertica_polonium_mtc_node0002", "stage" : "Waiting for Cluster Invite", "text" : "Prepare to be invited", "timestamp" : "2015-11-15 11:24:32.000" } { "node" : "v_vertica_polonium_mtc_node0002", "stage" : "Waiting for Cluster Invite", "text" : "Prepare to be invited", "timestamp" : "2015-11-15 11:24:34.000" } { "node" : "v_vertica_polonium_mtc_node0002", "stage" : "Waiting for Cluster Invite", "text" : "Ready to be invited", "timestamp" : "2015-11-15 11:24:35.778" }
Ok ... thanks to SruthiA from HP who answer in other post :
"Try killing from admintools and check with ps that it is dead.
1)In the admintools menu - Option 7 then option 3 (select all hosts)
2)ps -ef | grep vertica (check if they are dead)
3)If not kill -9 them.
Then start the DB from admintools. "
WORKS