Vertica Network Traffic - Where Can I Find More Info?

Alec_Isaacson

I'm new with Vertica (we're running 6.0.2) and I'm trying to understand how it uses the network and best practices for network configuration. The docs talk about using high speed networks and configuring public and private interfaces - that makes sense to me. However, I want to better understand the high levels of UDP traffic I'm now seeing on Vertica's subnet. Is that due to replication, or something else? Is that something that can be reduced? (Does Vertica really broadcast all database updates?) In general, is there any documentation on the network traffic that Vertica generates? I've looked in the Concepts and Admin guides but don't see much. Thanks.

[Deleted User]

Hi Alec, So, the simple answer is, Vertica's control messages go over UDP and its data messages go over TCP. (There are some cases during node recovery when Vertica may send larger amounts of data over UDP. But they're temporary / will stop after the cluster has fully recovered; and they only contain metadata, not real data.) Vertica uses UDP for its control messages because UDP Broadcast is a useful tool for efficiently coordinating a bunch of machines. Actual data always goes over TCP, in the simple way that you might expect. Under typical workloads and in the steady state, Vertica's traffic is primarily TCP traffic; if you're seeing lots of UDP traffic, that probably means that a node is still recovering (see the NODES system table). It can also mean that you're doing a ton of short transactions that don't require shipping any data around; then the only traffic will be control messages. Adam

Alec_Isaacson

Adam: Thanks for the quick response on this. I spent some additional time with the docs last night. One of the things I saw was a note for the -T option to install_vertica. It said that option configured the spread daemon to use point-to-point communication and that it should be used in virtual environments (my Vertica nodes are virtual). I'm definitely going to use that option. Does spread's p2p mode turn off the use of UDP for control messages and have it use TCP, or is this something different? Is there anywhere I can find details on what p2p mode does?

[Deleted User]

Hi Alec, Unfortunately, I don't think that this level of detail is currently thoroughly addressed in our docs. (I could be wrong; I'm not on our doc team, and I haven't had to use the docs to set up a network with this particular configuration... Others here may well correct me.) -T instructs Vertica to use unicast (poinT-to-poinT) UDP rather than broadcast UDP for its control messages. It still uses UDP. If you'd really like some technical depth: You may have noticed that Vertica uses a utility called "Spread" to send many of these UDP messages. Spread (the original one, anyway) is actually an open-source distribued-agreement protocol and library; it's documented elsewhere online. Out of curiosity, if you're running in a virtualized environment, are you running on Amazon? If so, we'd strongly encourage using our preinstalled/configured Amazon images. If not, you're welcome to take a look at those images (and/or their documentation); they might be useful for your own configuration. Also, a random note for virtual deployments (and, really, for anything complex enough that UDP Broadcast is an issue): Vertica is intended for interactive use; a keystroke can push a distributed atomic operation around the whole cluster (and if you've studied database theory, you may know that atomicity guarantees require round-trip packets between a bunch of nodes), and users expect that to appear to happen instantaneously. As such, we're highly sensitive to latency and to things like dropped packets, etc. Adam

Alec_Isaacson

Adam: Thanks again - this is great stuff. I saw that spread was responsible for messaging between nodes, I didn't know it was OSS, so I'll look more into that. We're not running in Amazon, we're running on VMware that we host and manage. The original driver was that we were seeing a lot of UDP traffic between the nodes, so we ended up putting them on their own VMware VLAN to segment the traffic. (Then, I saw in the docs how we could define separate public / database networks, so we may go back and revisit that configuration.) I will take a look at the Amazon images though and glean what I can from them.