Failure attempting to Create Database after new install
Hello,
Immediately following a fresh install on 3 hosts, I try to create a DB via admintools and get the following output. Hosts 02 and 03 never move from the (DOWN) state. Any ideas on what the cause could be?
Thanks,
Matt
*** Creating database: MattTestDB ***
10.0.0.239 OK [vertica][(6, 1, 3)][000][all]
10.0.0.240 OK [vertica][(6, 1, 3)][000][all]
10.0.0.241 OK [vertica][(6, 1, 3)][000][all]
Checking full connectivity
Creating database MattTestDB
Node Status: v_matttestdb_node0001: (DOWN)
Node Status: v_matttestdb_node0001: (INITIALIZING)
Node Status: v_matttestdb_node0001: (VALIDATING LICENSE)
Node Status: v_matttestdb_node0001: (UP)
Creating database nodes
Creating node v_matttestdb_node0002 (host 10.0.0.240)
Creating node v_matttestdb_node0003 (host 10.0.0.241)
Node Status: v_matttestdb_node0001: (UP) v_matttestdb_node0002: (DOWN) v_matttestdb_node0003: (DOWN)
Node Status: v_matttestdb_node0001: (UP) v_matttestdb_node0002: (DOWN) v_matttestdb_node0003: (DOWN)
Node Status: v_matttestdb_node0001: (UP) v_matttestdb_node0002: (DOWN) v_matttestdb_node0003: (DOWN)
Node Status: v_matttestdb_node0001: (UP) v_matttestdb_node0002: (DOWN) v_matttestdb_node0003: (DOWN)
Node Status: v_matttestdb_node0001: (UP) v_matttestdb_node0002: (DOWN) v_matttestdb_node0003: (DOWN)
Node Status: v_matttestdb_node0001: (UP) v_matttestdb_node0002: (DOWN) v_matttestdb_node0003: (DOWN)
Node Status: v_matttestdb_node0001: (UP) v_matttestdb_node0002: (DOWN) v_matttestdb_node0003: (DOWN)
Immediately following a fresh install on 3 hosts, I try to create a DB via admintools and get the following output. Hosts 02 and 03 never move from the (DOWN) state. Any ideas on what the cause could be?
Thanks,
Matt
*** Creating database: MattTestDB ***
10.0.0.239 OK [vertica][(6, 1, 3)][000][all]
10.0.0.240 OK [vertica][(6, 1, 3)][000][all]
10.0.0.241 OK [vertica][(6, 1, 3)][000][all]
Checking full connectivity
Creating database MattTestDB
Node Status: v_matttestdb_node0001: (DOWN)
Node Status: v_matttestdb_node0001: (INITIALIZING)
Node Status: v_matttestdb_node0001: (VALIDATING LICENSE)
Node Status: v_matttestdb_node0001: (UP)
Creating database nodes
Creating node v_matttestdb_node0002 (host 10.0.0.240)
Creating node v_matttestdb_node0003 (host 10.0.0.241)
Node Status: v_matttestdb_node0001: (UP) v_matttestdb_node0002: (DOWN) v_matttestdb_node0003: (DOWN)
Node Status: v_matttestdb_node0001: (UP) v_matttestdb_node0002: (DOWN) v_matttestdb_node0003: (DOWN)
Node Status: v_matttestdb_node0001: (UP) v_matttestdb_node0002: (DOWN) v_matttestdb_node0003: (DOWN)
Node Status: v_matttestdb_node0001: (UP) v_matttestdb_node0002: (DOWN) v_matttestdb_node0003: (DOWN)
Node Status: v_matttestdb_node0001: (UP) v_matttestdb_node0002: (DOWN) v_matttestdb_node0003: (DOWN)
Node Status: v_matttestdb_node0001: (UP) v_matttestdb_node0002: (DOWN) v_matttestdb_node0003: (DOWN)
Node Status: v_matttestdb_node0001: (UP) v_matttestdb_node0002: (DOWN) v_matttestdb_node0003: (DOWN)
0
Comments
dbadmin@host01:/data/MattTestDB$ more dbLog
Connected to spread at host 10.0.0.239(10.0.0.239) port 4803
dbadmin@host02:/data/MattTestDB# more dbLog
Connected to spread at host 10.0.0.240(10.0.0.240) port 4803
dbadmin@host03:/data/MattTestDB# more dbLog
Connected to spread at host 10.0.0.241(10.0.0.241) port 4803
Node Status: v_testdb_node0001: (DOWN) v_testdb_node0002: (UP) v_testdb_node0003: (DOWN)
Node Status: v_testdb_node0001: (DOWN) v_testdb_node0002: (UP) v_testdb_node0003: (DOWN)
Node Status: v_testdb_node0001: (DOWN) v_testdb_node0002: (UP) v_testdb_node0003: (DOWN)
Node Status: v_testdb_node0001: (DOWN) v_testdb_node0002: (UP) v_testdb_node0003: (DOWN)
Node Status: v_testdb_node0001: (DOWN) v_testdb_node0002: (UP) v_testdb_node0003: (DOWN)
Node Status: v_testdb_node0001: (DOWN) v_testdb_node0002: (UP) v_testdb_node0003: (DOWN)
Node Status: v_testdb_node0001: (DOWN) v_testdb_node0002: (UP) v_testdb_node0003: (DOWN)
Node Status: v_testdb_node0001: (DOWN) v_testdb_node0002: (UP) v_testdb_node0003: (DOWN)
Node Status: v_testdb_node0001: (DOWN) v_testdb_node0002: (UP) v_testdb_node0003: (DOWN)
Node Status: v_testdb_node0001: (DOWN) v_testdb_node0002: (UP) v_testdb_node0003: (DOWN)
Hm... Yes, that does sound like a connectivity issue of some sort. Do you have any sort of firewall between your nodes? In particular, is UDP traffic able to get through smoothly? We rely on UDP for a lot of our initialization and control messages; it's also, unfortunately, often blocked or lossy on certain types of networks, and so needs a little more configuration...
Have you run Vertica's "vnetperf" tool? It may give you some pointers about network configuration:
https://my.vertica.com/docs/6.1.x/HTML/index.htm#19284.htm
Adam