Marge two Vertica clusters

Hi all,

We have two environments of Vertica 7.1.1-10, 10 node each, database of 40TB, K-safety 1.
One as the production and the other is DR.
We want to close the DR site and merge the two clusters to one 20 nodes production cluster.

Several questions:
1. Do I need to uninstall Vertica from the DR site before I'm adding the nodes to the production site? or I need only to drop the existing database on the DR site?
2. How it's work with the downtime when I'm adding the nodes? There will be downtime? Can I still work with the production environment while I'm adding the nodes (at the re-balancing time)?
3. Can I add 10 nodes together? or I need to add them in batches, 5 nodes each time for example?
4. What is the overhead on the system?
5. How can I determent the re-balancing time?

 

If you have any more insights and things that I need to check or to give more attention... I'd love to hear :)

 

Thanks :)
Chen

Comments

  • So you wana decomision on Vertica Cluster and reuse the hosts in ass new nodes for the left cluster ? 

     

    Before you add any new nodes you need to

    1. Disable local segmentation(for better performance)
    SELECT DISABLE_LOCAL_SEGMENTS();

     2. Purge your data.(less work to be done during the rebalance).

    --look for delete vectors

    SELECT schema_name
    ,projection_name
    ,count(*) num_ros
    ,sum(total_row_count) num_rows
    ,sum(deleted_row_count) num_deld_rows
    ,sum(delete_vector_count) Num_dv
    ,(sum(deleted_row_count) / sum(total_row_count) * 100)::INT por_del_rows
    FROM storage_containers
    WHERE node_name = ( SELECT local_node_name())
    GROUP BY 1, 2
    HAVING sum(deleted_row_count) > 0
    ORDER BY 5 DESC;

     3. Drop any unused projections(less work to be done during the rebalance).

    - i use this sql to look for such

    SELECT
    p.table_schema,
    p.basename AS "Table Name",
    MAX(pu.last_used) AS last_used,
    extract(DAY from (getdate()-MAX(pu.last_used))) AS days_last_used,
    -- extract(DAY FROM GETDATE() - pu.last_used) AS "Days without Using",
    ROUND(SUM(sc.used_bytes/ (1024^2)),3.0) AS used_mb,
    ROUND(SUM(sc.used_bytes)/( 1024^3),3.0) AS used_gb
    FROM
    (
    SELECT
    projection_oid,
    MAX(TIME) AS last_used
    FROM
    v_internal.dc_projections_used
    WHERE
    table_oid IN
    (
    SELECT
    table_id
    FROM
    v_catalog.tables
    WHERE
    NOT is_system_table)
    GROUP BY
    projection_oid) pu
    JOIN
    (
    SELECT
    oid,
    name,
    basename,
    "schemaname" AS table_schema
    FROM
    v_internal.vs_projections) p
    ON
    p.oid = pu.projection_oid
    JOIN
    (
    SELECT
    projection_id,
    used_bytes
    FROM
    v_monitor.storage_containers) sc
    ON
    sc.projection_id = pu.projection_oid
    GROUP BY
    p.table_schema,
    p.basename
    ORDER BY
    last_used ASC ;

     

     4. Optimize your REFRESH resource pool for the rebalance task.

    - see if you can increase the PLANNEDCONCURRENCY value for this pool, will improve prerformance.

    and after rebalance you can put it back to default.

     

     

    To add a new node to an existing Vertica Cluster :

     

    1- Make sure you have a host with the same operation system installed as the running Vertica O.S.

    2- Prepare the box to receive  Vertica software and create the same O.S. layout(partitions).

    3- Setup the sshd password-less for root user between the two hosts(running and the host that will be added)

    3- Choose the way you want to add the new host to the cluster

    A – Using Vertica Management Console.

    B – Using the update_vertica utility.

    -- run this from one of the nodes that are part of the receveing cluster (as root user)

     

    /opt/vertica/sbin/update_vertica --add-hosts host_name1,host_name2 --rpm /tmp/vertica-version 7.x.x

    Once done - list avilable nodes in the cluster

     

     

    admintools -t list_host

    - his command will list all hosts and not all nodes.

     

     

    This will show you all nodes the run the db

     

     admintools -t list_allnodes

    Add the hosts to your db 

     

     

    admintools -t db_add_node -s host_name1,host_name2 -d database_name -p password

    Rebalance the data to apply to your required K-safety

    admintools -t rebalance_data -d <db_name> -p test -k 0,1,2

     

     

    1. Do I need to uninstall Vertica from the DR site before I'm adding the nodes to the production site? or I need only to drop the existing database on the DR site?

     - i would do a clean install


    2. How it's work with the downtime when I'm adding the nodes? There will be downtime? Can I still work with the production environment while I'm adding the nodes (at the re-balancing time)?

    - no downtime is required(some lock might happen on the tables), it`s just resource intensive so ideal is to do it when you less have access to the db.


    3. Can I add 10 nodes together? or I need to add them in batches, 5 nodes each time for example?

    - i sudgesst doing in two nodes at a time. Or actually i don`t think i know the right answer, never added more then 2 at a time in the past so i don`t actually know ! 


    4. What is the overhead on the system?

    - it is resoirce intensive as you basecly replicate the objects across the cluster.(network perf will impact the times if network is slow)

    vioperf: Measures the speed and consistency of your hard drives.

    vnetperf: Measures the latency and throughput of your network between nodes.


    5. How can I determent the re-balancing time?

    What do you mean ? 

       How long is going to take ? (it all depends on size/number of projecitons/Disk speed/Network latency)

    or 

     When should i do it ? (after you add or remove a node)

     

     

    hope this small walk thru helped you ! 

     

  • Does anyone know if I can stop the "add nodes" command while it running?

    for example, if it takes to much time and it start to overhead on the system and I want to stop the process... can it be done?

    thanks

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file

Can't find what you're looking for? Search the Vertica Documentation, Knowledge Base, or Blog for more information.