Inequal nodes perform worse together than separately

Maxim_Fridental

I'm evaluating Vertica CE 7.0.1. After having installed it on a single node (24 CPU cores, 128 Gb RAM, SSD RAID) and executed a set of test queries, I wanted to see how it performs when scaled horizontally.

To avoid provisioning second hardware of that size, I've installed the second node in a virtual machine with 2 Gb RAM running on my desktop PC. I wanted to learn how easy is to add a new node to a cluster, and I was expected to see maybe 5% performance improvements at least on some of my test queries.

Instead, I see severe performance degradation (the performance of the 2-node-cluster is about only 20% of the performance of the single-node installation). But it puzzles me more that some queries can't be executed at all and are rejected with the hint that they don't fit into memory.

I've executed

select * from RESOURCE_POOL_STATUS where pool_name='general';

which gaves me

-[ RECORD 1 ]--------------+------------------
node_name                  | v_test_node0001
pool_oid                   | 45035996273718864
pool_name                  | general
is_internal                | t
memory_size_kb             | 120941936
memory_size_actual_kb      | 120941936
memory_inuse_kb            | 0
general_memory_borrowed_kb | 0
queueing_threshold_kb      | 114894840
max_memory_size_kb         | 120941936
running_query_count        | 0
planned_concurrency        | 24
max_concurrency            |
is_standalone              | t
queue_timeout_in_seconds   | 300
execution_parallelism      | AUTO
priority                   | 0
runtime_priority           | MEDIUM
runtime_priority_threshold | 2
single_initiator           | false
query_budget_kb            | 4787285
cpu_affinity_set           |
cpu_affinity_mask          | ffffff
cpu_affinity_mode          | ANY
-[ RECORD 2 ]--------------+------------------
node_name                  | v_test_node0003
pool_oid                   | 45035996273718864
pool_name                  | general
is_internal                | t
memory_size_kb             | 765086
memory_size_actual_kb      | 765086
memory_inuse_kb            | 0
general_memory_borrowed_kb | 0
queueing_threshold_kb      | 726831
max_memory_size_kb         | 765086
running_query_count        | 0
planned_concurrency        | 4
max_concurrency            |
is_standalone              | t
queue_timeout_in_seconds   | 300
execution_parallelism      | AUTO
priority                   | 0
runtime_priority           | MEDIUM
runtime_priority_threshold | 2
single_initiator           | false

So I suppose the first node is still having 120 Gb of RAM that was enough to process all queries when it was the single node. How can it be the cluster doesn't have enough memory now? It is possible to tune it somehow?

Thanks,

Maxim

kesten_broughto

I'm not surprised by this.  I've found with most clustered software, if the nodes aren't equal, performance is often limited by the weakest link.  Load balancers should be smart enough to divvy the work but often don't do a good job for unequal nodes.  The vertica installer even warns if there is even a slight difference in RAM between nodes.  
I'd say, you've tested adding a node fine.  Testing for multi-node performance would require homogenous nodes.

[Deleted User]

Also, please ensure the communications is point to point vs broadcast if using VMs.

[Deleted User]

Vertica is a fast as the slowest node. Vertica assumes that all the nodes are the same architecture, when it plans queries, it plan it that all the nodes will use the same amount of cpu, mem, etc.  So if the query gets planned in the 2gb, some queries won't even start because they may request more resources that it is available. If the queries are planned in the big machine, the queries may fail in the execution of the small query, because it assume that can allocate more than it has available. Does make sense? The key here is that the vertica nodes should be the same.  Can you install VM in the big cluster and have nodes with similar architecture?