vbr -t replicate requires REBALANCE

lybhrwn742lybhrwn742 Vertica Customer ✭
in General Discussion

Hello,
After we replicate one of our tables to the secondary cluster it seems that the table needs to be rebalanced.
In REBALANCE_STATUS the progress is 0% and it's take some time to finish the rebalance.
In the DDL the projections are "ALL NODES" , all nodes are UP and the clusters have the same number of nodes.
We don't know how to fix this problem,
Thank you !

Tagged:

Answers

  • Jim_KnicelyJim_Knicely - Select Field - Administrator
    edited October 2021

    Hi,

    That's weird. What version of Vertica is this? Are both clusters using the same version? Can you post the output of:

    SELECT export_objects('', 'table_schema.table_name');

    From both the source and the target?

  • lybhrwn742lybhrwn742 Vertica Customer ✭

    **CLUSTER 1
    **
    dbadmin=> select version();

    version

    Vertica Analytic Database V10.0.1-3

    dbadmin=> CREATE TABLE test (id number);
    CREATE TABLE
    dbadmin=> insert into test values (1);

    OUTPUT

    1

    dbadmin=>commit;
    COMMIT
    dbadmin=>select * from test;

    id

    1

    dbadmin=>select export_objects('','test');

    export objects

    CREATE TABLE public.test
    (
    id numeric(38,0)
    );

    CREATE PROJECTION public.test /+createtype(L)/
    (
    id
    )
    AS
    SELECT test.id
    FROM public.test
    ORDER BY test.id
    SEGMENTED BY hash(test.id) ALL NODES KSAFE 1;

    SELECT MARK_DESIGN_KSAFE(1);

    dbadmin=> select table_name,separated_percent,transferred_percent from rebalance_status where table_name='test';
    table_name | separated_percent | transferred_percent
    -----------+-------------------+--------------------
    (0 rows)

    vbr -t replicate -c clone_TEST.ini --debug 3

    **CLUSTER 2
    **
    dbadmin=> select version();

    version

    Vertica Analytic Database V10.0.1-3

    dbadmin=>select * from test;

    id

    1

    dbadmin=>select export_objects('','test');

    export objects

    CREATE TABLE public.test
    (
    id numeric(38,0)
    );

    CREATE PROJECTION public.test /+createtype(L)/
    (
    id
    )
    AS
    SELECT test.id
    FROM public.test
    ORDER BY test.id
    SEGMENTED BY hash(test.id) ALL NODES KSAFE 1;

    SELECT MARK_DESIGN_KSAFE(1);

    dbadmin=> select table_name,separated_percent,transferred_percent from rebalance_status where table_name='test';
    table_name | separated_percent | transferred_percent
    -----------+-------------------+--------------------
    test | 0.00| 0.00
    (1 row)

  • SruthiASruthiA Administrator

    After you ran vbr replicate, you did not run rebalance right? Still we can see one row from your test table right?

  • lybhrwn742lybhrwn742 Vertica Customer ✭

    Right, after I replicate the table I see a record in REBALANCE_STATUS (in cluster 1 I don't see since it was created in that cluster)

  • SruthiASruthiA Administrator

    Can you share me the output of the following and attach the vbr log? I see that you ran it debug 3 option

    select * from rebalance_status where table_name='test';

  • lybhrwn742lybhrwn742 Vertica Customer ✭

    Hi SruthiA,

    dbadmin=> select * from rebalance_status where table_name='test';
    table_schema | table_name | separated_percent | transferred_percent | TABLE_SIZE_GB
    -------------+------------+-------------------+---------------------+--------------
    public | test | 0.00 | 0.00 | 0

    dbadmin@hostname [~/VbrDir]
    $ vbr -t replicate -c clone_TEST.ini --debug 3

    Participating nodes: V_DatabaseName_node0001,V_DatabaseName_node0002,V_DatabaseName_node0003,V_DatabaseName_node0004,V_DatabaseName_node0005,V_DatabaseName_node0006,V_DatabaseName_node0007,V_DatabaseName_node0008,V_DatabaseName_node0009,V_DatabaseName_node0010,V_DatabaseName_node0011,V_DatabaseName_node0012,V_DatabaseName_node0013,V_DatabaseName_node0014,V_DatabaseName_node0015,V_DatabaseName_node0016,V_DatabaseName_node0017,V_DatabaseName_node0018
    Starting replication of objects ['public.TEST'] from DatabaseName.
    Snapshotting the source database.
    Snapshot complete.
    Copying catalog snapshot from source to destination.
    Preparing destination database for replicating objects.
    Prep complete, start syncing files.
    Approximate bytes to copy: 0 of 164 total.
    Progress: 0 out of 0 bytes (100%)
    Finalizing object replication.
    Complete object replication, removing snapshot from the source database.
    Object replication complete!

  • SruthiASruthiA Administrator

    I meant vbr log file for the replicate you have run.. It should be present in /tmp/vbr.

  • lybhrwn742lybhrwn742 Vertica Customer ✭

    Is there anything specific I should look for? this is closed site so I can't attach the log.
    Also I don't see any errors there.

    Thanks.

  • Jim_KnicelyJim_Knicely - Select Field - Administrator

    @ybbhosale - What are you seeing in the REMOTE_REPLICATION_STATUS table on the source database?

    Example:

    dbadmin=> SELECT * FROM remote_replication_status;
 objects |  snapshot_name  |                replication_point                 |  last_replicated_time  | replicated_epoch | current_epoch
---------+-----------------+--------------------------------------------------+------------------------+------------------+---------------
 test    | backup_snapshot | backup_snapshot_3R7MOX4G59S5FZA3WPSWBPGKXHWCIL01 | 2021-10-13 07:34:49-04 |               19 |            20
(1 row)
  • lybhrwn742lybhrwn742 Vertica Customer ✭

    Hi Jim,

    dbadmin=> select * from remote_replication_status;
    objects | snapshot_name | replication_point | last_replicated_time | replicated_epoch | current_epoch
    ------------+---------------------+--------------------------------------------+----------------------------+------------------------+-------------------
    public.test |snapshot_copy_objects |snapshot_copy_objects_ACF6D.. | 2021-10-17 18:10:41+03 | 2375600 | 2375622

  • Jim_KnicelyJim_Knicely - Select Field - Administrator

    @lybhrwn742 - I tested replication using Vertica 10.0.1-3 between two three node clusters. I did not see any requirements for rebalancing on the target DB.

    Can you run the following commands on both of your clusters?

    SELECT get_node_dependencies();
    SELECT COUNT(*) FROM projections WHERE NOT is_up_to_date;

    Thanks!

  • lybhrwn742lybhrwn742 Vertica Customer ✭ 
        CLUSTER 1

    dbadmin=> select get_node_dependencies();

    000000000000000011 - cnt: 7
    000000000000000110 - cnt: 56
    000000000000001100 - cnt: 56
    000000000000011000 - cnt: 9
    000000000000110000 - cnt: 58
    000000000001001000 - cnt: 2
    000000000001100000 - cnt: 58
    000000000011000000 - cnt: 7
    000000000110000000 - cnt: 56
    000000001100000000 - cnt: 56
    000000011000000000 - cnt: 7
    000000110000000000 - cnt: 56
    000001100000000000 - cnt: 56
    000010000000000001 - cnt: 56
    000011000000000000 - cnt: 56
    000011111111111111 - cnt: 6
    000100000000000001 - cnt: 49
    000100000000000010 - cnt: 49
    001000000000001000 - cnt: 49
    001000000000010000 - cnt: 49
    010000000000100000 - cnt: 49
    010000000001000000 - cnt: 49
    100000000100000000 - cnt: 49
    100000001000000000 - cnt: 49
    111111111111111111 - cnt: 1

    000000000000000001 - name: v_database_name_node0001
    000000000000000010 - name: v_database_name_node0002
    000000000000000100 - name: v_database_name_node0003
    000000000000001000 - name: v_database_name_node0004
    000000000000010000 - name: v_database_name_node0005
    000000000000100000 - name: v_database_name_node0006
    000000000001000000 - name: v_database_name_node0007
    000000000010000000 - name: v_database_name_node0008
    000000000100000000 - name: v_database_name_node0009
    000000001000000000 - name: v_database_name_node0010
    000000010000000000 - name: v_database_name_node0011
    000000100000000000 - name: v_database_name_node0012
    000001000000000000 - name: v_database_name_node0013
    000010000000000000 - name: v_database_name_node0014
    000100000000000000 - name: v_database_name_node0015
    001000000000000000 - name: v_database_name_node0016
    010000000000000000 - name: v_database_name_node0017
    100000000000000000 - name: v_database_name_node0018

    dbadmin=> select count(*) from projections where not is_up_to_date;

    count

      0

    (1 row)

    CLUSTER 2

    dbadmin=> select get_node_dependencies();

    000000000000000011 - cnt: 2
    000000000000000110 - cnt: 8
    000000000000001001 - cnt: 1
    000000000000001100 - cnt: 74
    000000000000011000 - cnt: 8
    000000000000110000 - cnt: 74
    000000000001100000 - cnt: 3
    000000000011000000 - cnt: 73
    000000000110000000 - cnt: 74
    000000001100000000 - cnt: 74
    000000010000000001 - cnt: 6
    000000011000000000 - cnt: 73
    000000011111111111 - cnt: 6
    000000100000000001 - cnt: 71
    000000110000000000 - cnt: 2
    000001000000000010 - cnt: 1
    000001000000100000 - cnt: 71
    000001000001000000 - cnt: 71
    000001100000000000 - cnt: 3
    000010000000000001 - cnt: 65
    000010001000000000 - cnt: 1
    000010010000000000 - cnt: 66
    000100000001000000 - cnt: 1
    000100000010000000 - cnt: 1
    000100100000000000 - cnt: 65
    001000000000000010 - cnt: 66
    001000000000000100 - cnt: 66
    010000000000000001 - cnt: 1
    010000000000001000 - cnt: 65
    010000000000010000 - cnt: 66
    100000000000000001 - cnt: 2
    100000000000000010 - cnt: 71
    100000010000000000 - cnt: 1
    100000100000000000 - cnt: 7
    100001000000000000 - cnt: 2
    100100000000000000 - cnt: 65
    111111111111111111 - cnt: 1

    000000000000000001 - name: v_database_name_node0004
    000000000000000001 - name: v_database_name_node0001
    000000000000000001 - name: v_database_name_node0002
    000000000000000001 - name: v_database_name_node0003
    000000000000000001 - name: v_database_name_node0005
    000000000000000001 - name: v_database_name_node0006
    000000000000000001 - name: v_database_name_node0007
    000000000000000001 - name: v_database_name_node0008
    000000000000000001 - name: v_database_name_node0009
    000000000000000001 - name: v_database_name_node0010
    000000000000000001 - name: v_database_name_node0011
    000000000000000001 - name: v_database_name_node0013
    000000000000000001 - name: v_database_name_node0014
    000000000000000001 - name: v_database_name_node0015
    000000000000000001 - name: v_database_name_node0016
    000000000000000001 - name: v_database_name_node0017
    000000000000000001 - name: v_database_name_node0018
    000000000000000001 - name: v_database_name_node0012

    dbadmin=> select count(*) from projections where not is_up_to_date;

    count

      2

    (1 row)

    Thanks.

  • lybhrwn742lybhrwn742 Vertica Customer ✭

    Correction for cluster 2 -

    000000000000000001 - name: v_database_name_node0004
    000000000000000010 - name: v_database_name_node0001
    000000000000000100 - name: v_database_name_node0002
    000000000000001000 - name: v_database_name_node0003
    000000000000010000 - name: v_database_name_node0005
    000000000000100000 - name: v_database_name_node0006
    000000000001000000 - name: v_database_name_node0007
    000000000010000000 - name: v_database_name_node0008
    000000000100000000 - name: v_database_name_node0009
    000000001000000000 - name: v_database_name_node0010
    000000010000000000 - name: v_database_name_node0011
    000000100000000000 - name: v_database_name_node0013
    000001000000000000 - name: v_database_name_node0014
    000010000000000000 - name: v_database_name_node0015
    000100000000000000 - name: v_database_name_node0016
    001000000000000000 - name: v_database_name_node0017
    010000000000000000 - name: v_database_name_node0018
    100000000000000000 - name: v_database_name_node0012

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Format
SpoilerCodeQuote
Heading 2Heading 1
Emoji
:):D:(;):/:o:s:p:'(:|B):#:*<3o:)>:)
Url
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file
Home • General DiscussionComment As ...