Avoid Vertica broadcast storm

Juri_Dmitrijev

For last  4 hours I am trying to find out how to reconfigure already installed Vertica cluster to use only point-to-point transfer. Setting controlmode = pt2pt, controlsubnet = default has no effect. (install_vertica with -T flag as well)

Abhi_1

What is the vertica version you are using?

Juri_Dmitrijev

7.0.2-1.x86_64

Abhi_1

From 7.x version you need to execute the below command inside Vsql to switch the spread modes.
Would suggest to do these changes when all the nodes are up and running.

select set_control_mode('pt2pt');
select reload_spread(true);

Below steps are not mandatory but suggested to keep track of the changes

Edit /opt/vertica/config/admintools.conf file to modify
from
controlmode = broadcast
to
controlmode = pt2pt

Use admintools distribute database configuration and admintools metadata to all nodes.

Juri_Dmitrijev

thx, that was usefull. topic closed

bertcord

late to the post but I am curious as to how many nodes you had when you determined you needed this change.  I am working on a project that might need 180 nodes however I can split this into multiple clusters.  I am leaning towards clusters of 64 nodes.

Curtis_1

180 nodes would be huge. You should definitely do that hand in hand with Vertica support and/or engineering. Some clients at that scale have found better performance with smaller clusters in the range of 32-64 nodes, depending on traffic and volume.

We consider something to be a "large cluster" which requires that flag be specified during the installation, at somewhere around 30-40 nodes. Other environmental factors can drive that as well - such as the number of routers or switches, or latency between nodes. Some small clusters require this type of configuration, and Amazon clusters require it be default.

bertcord

thanks for the response.  Our current system runs on 35 data nodes and we are using 10 ephemeral nodes to handle data loads.  For the most part we have had no issues.  The one major issue we had was due to Global Catalog lock timeouts.  This was mostly our fault due ot some bad logic and the use of sequence numbers.