Marge two Vertica clusters

ChenSha

Hi all,

We have two environments of Vertica 7.1.1-10, 10 node each, database of 40TB, K-safety 1.
One as the production and the other is DR.
We want to close the DR site and merge the two clusters to one 20 nodes production cluster.

Several questions:
1. Do I need to uninstall Vertica from the DR site before I'm adding the nodes to the production site? or I need only to drop the existing database on the DR site?
2. How it's work with the downtime when I'm adding the nodes? There will be downtime? Can I still work with the production environment while I'm adding the nodes (at the re-balancing time)?
3. Can I add 10 nodes together? or I need to add them in batches, 5 nodes each time for example?
4. What is the overhead on the system?
5. How can I determent the re-balancing time?

 

If you have any more insights and things that I need to check or to give more attention... I'd love to hear

 

Thanks
Chen

Adrian_Oprea_1

So you wana decomision on Vertica Cluster and reuse the hosts in ass new nodes for the left cluster ? 

 

Before you add any new nodes you need to

1. Disable local segmentation(for better performance)

SELECT DISABLE_LOCAL_SEGMENTS();

 2. Purge your data.(less work to be done during the rebalance).

--look for delete vectors

SELECT schema_name
          ,projection_name
          ,count(*) num_ros
          ,sum(total_row_count) num_rows
          ,sum(deleted_row_count) num_deld_rows
          ,sum(delete_vector_count) Num_dv
          ,(sum(deleted_row_count) / sum(total_row_count) * 100)::INT por_del_rows
FROM storage_containers
WHERE node_name = ( SELECT local_node_name())
GROUP BY 1, 2
HAVING sum(deleted_row_count) > 0
ORDER BY 5 DESC;

 3. Drop any unused projections(less work to be done during the rebalance).

- i use this sql to look for such

SELECT
    p.table_schema,
    p.basename                                 AS "Table Name",
    MAX(pu.last_used)                          AS last_used,
    extract(DAY from (getdate()-MAX(pu.last_used)))        AS days_last_used,
   -- extract(DAY FROM GETDATE() - pu.last_used) AS "Days without Using",
    ROUND(SUM(sc.used_bytes/ (1024^2)),3.0)    AS used_mb,
    ROUND(SUM(sc.used_bytes)/( 1024^3),3.0)    AS used_gb
FROM
    (
        SELECT
            projection_oid,
            MAX(TIME) AS last_used
        FROM
            v_internal.dc_projections_used
        WHERE
            table_oid IN
            (
                SELECT
                    table_id
                FROM
                    v_catalog.tables
                WHERE
                    NOT is_system_table)
        GROUP BY
            projection_oid) pu
JOIN
    (
        SELECT
            oid,
            name,
            basename,
            "schemaname" AS table_schema
        FROM
            v_internal.vs_projections) p
ON
    p.oid = pu.projection_oid
JOIN
    (
        SELECT
            projection_id,
            used_bytes
        FROM
            v_monitor.storage_containers) sc
ON
    sc.projection_id = pu.projection_oid
GROUP BY
    p.table_schema,
    p.basename
ORDER BY
    last_used ASC ;

 

 4. Optimize your REFRESH resource pool for the rebalance task.

- see if you can increase the PLANNEDCONCURRENCY value for this pool, will improve prerformance.

and after rebalance you can put it back to default.

 

 

To add a new node to an existing Vertica Cluster :

 

1- Make sure you have a host with the same operation system installed as the running Vertica O.S.

2- Prepare the box to receive  Vertica software and create the same O.S. layout(partitions).

3- Setup the sshd password-less for root user between the two hosts(running and the host that will be added)

3- Choose the way you want to add the new host to the cluster

A – Using Vertica Management Console.

B – Using the update_vertica utility.

-- run this from one of the nodes that are part of the receveing cluster (as root user)

 

/opt/vertica/sbin/update_vertica --add-hosts host_name1,host_name2 --rpm /tmp/vertica-version 7.x.x

Once done - list avilable nodes in the cluster

 

 

admintools -t list_host

- his command will list all hosts and not all nodes.

 

 

This will show you all nodes the run the db

 

 admintools -t list_allnodes

Add the hosts to your db 

 

 

admintools -t db_add_node -s host_name1,host_name2 -d database_name -p password

Rebalance the data to apply to your required K-safety

admintools -t rebalance_data -d <db_name> -p test -k 0,1,2

 

 

1. Do I need to uninstall Vertica from the DR site before I'm adding the nodes to the production site? or I need only to drop the existing database on the DR site?

 - i would do a clean install

2. How it's work with the downtime when I'm adding the nodes? There will be downtime? Can I still work with the production environment while I'm adding the nodes (at the re-balancing time)?

- no downtime is required(some lock might happen on the tables), it`s just resource intensive so ideal is to do it when you less have access to the db.

3. Can I add 10 nodes together? or I need to add them in batches, 5 nodes each time for example?

- i sudgesst doing in two nodes at a time. Or actually i don`t think i know the right answer, never added more then 2 at a time in the past so i don`t actually know ! 

4. What is the overhead on the system?

- it is resoirce intensive as you basecly replicate the objects across the cluster.(network perf will impact the times if network is slow)

vioperf: Measures the speed and consistency of your hard drives.

vnetperf: Measures the latency and throughput of your network between nodes.

5. How can I determent the re-balancing time?

What do you mean ? 

   How long is going to take ? (it all depends on size/number of projecitons/Disk speed/Network latency)

or 

 When should i do it ? (after you add or remove a node)

 

 

hope this small walk thru helped you ! 

 

ChenSha

Does anyone know if I can stop the "add nodes" command while it running?

for example, if it takes to much time and it start to overhead on the system and I want to stop the process... can it be done?

thanks