Spread not communicating between nodes

Farzam_Farrokhi

Fresh install. I've installed on a 3 node cluster with the nodes not on the same subnet. But I installed using -N -T options. Everything installed fine. Once I create a database on a node the database is created on other nodes as well, but they never come up. I tested the spread on the nodes and spread doesn't seem to be communicating properly. Ports 4803 & 5433 are open. vspread.conf is identical on the three nodes as well. I tested spread using: /opt/vertica/spread/bin/spuser -r Any input on how I can troubleshoot this? Nothing shows up in the /opt/vertica/log/spread_xxx.log that is helpful. Thanks in advance.

Bhawana

Hi,

<!-- @page { margin: 0.79in } P { margin-bottom: 0.08in } -->

Verify this 2 things in spread :

1- Check the /opt/vertica/config/vspread.conf files on all nodes – they need to be identical.

2- Do spread test with spuser to see if nodes at least are communicating between them.

You can do spuser -r test like this
/opt/vertica/spread/bin/spuser -r
At the prompt User >
enter in “j test”
j is for join and you are creating a grouping called test.

Repeat this on all nodes to confirm that they are connecting to each other. You should see the new connections come in with each node connecting.

If the spuser -r is working is fine, it is not the spread error. Can you check if there is a dbLog file or if there is any additional info in /opt/vertica/log/adminTools-dbadmin.log file?


Also Vertica recommends removing firewalls between nodes in the cluster. If you are running your cluster on private interfaces, a firewall is not needed. A firewall on the public side should only need port 5433 open. A database cluster should depend preferably on an external firewall provided by a gateway rather than Linux. Vertica is designed for the server processes in the cluster machines to work together as though they were a single powerful machine. Firewalls can slow network throughput even with the necessary ports open. If you must use firewalls, open the following ports on each host. Vertica 5433 TCP (All client connections) 32768-61000 TCP (server server) Dynamic ports , range depends on OS configuration, check /proc/sys/net/ipv4/ip_local_port_range ) Spread 4803 TCP (Client connections) 4803 UDP (Daemon Daemon) 4804 UDP (Daemon Daemon) 4805 UDP (Monitor to Daemon) (optional and only if "DangerousMonitor = yes" in config file) Administration tools (adminTools, install_vertica) 22 TCP (SSH connections within cluster) Again, Vertica does not recommend having firewall rules for the private network interfaces.



Regards,

Bhawana

 

Farzam_Farrokhi

Hi Bhawan,
Thanks for your reply. As I mentioned I checked vspread.conf file on all servers and they are identical. And I have tested with spuser and the nodes can't communicate with each other; i.e. I don't see any new connections coming. So it seem the problem is with spread not being able to communicate between nodes. 

I don't have firewall enabled. And not seeing any warnings or errors in the /opt/vertica/log/adminTools-dbadmin.log file. I've also checked /opt/vertica/log/spread_xxxxx.log file and there is no helpful information. just messages as:
[Thu 07 Nov 2013 17:22:50] Send_join: State is 4
[Thu 07 Nov 2013 17:22:51] Send_join: State is 4
[Thu 07 Nov 2013 17:22:52] Send_join: State is 4

Any ideas on how I can further troubleshoot?

Ruben_Silva

Farzam are you using Amazon?

I was having the same issue until I read some documentation on hew spread works. That and this:

http://aws.amazon.com/vpc/faqs/#R4

Led me to using the -T option of the install_vertica script:

  [-T] direct transmit only: disallow UDP broadcast traffic

Now everything looks ok.