vbr -t replicate requires REBALANCE

lybhrwn742

Hello,
After we replicate one of our tables to the secondary cluster it seems that the table needs to be rebalanced.
In REBALANCE_STATUS the progress is 0% and it's take some time to finish the rebalance.
In the DDL the projections are "ALL NODES" , all nodes are UP and the clusters have the same number of nodes.
We don't know how to fix this problem,
Thank you !

Jim_Knicely

Hi,

That's weird. What version of Vertica is this? Are both clusters using the same version? Can you post the output of:

SELECT export_objects('', 'table_schema.table_name');

From both the source and the target?

lybhrwn742

**CLUSTER 1
**
dbadmin=> select version();

version

Vertica Analytic Database V10.0.1-3

dbadmin=> CREATE TABLE test (id number);
CREATE TABLE
dbadmin=> insert into test values (1);

OUTPUT

1

dbadmin=>commit;
COMMIT
dbadmin=>select * from test;

id

1

dbadmin=>select export_objects('','test');

export objects

CREATE TABLE public.test
(
id numeric(38,0)
);

CREATE PROJECTION public.test /+createtype(L)/
(
id
)
AS
SELECT test.id
FROM public.test
ORDER BY test.id
SEGMENTED BY hash(test.id) ALL NODES KSAFE 1;

SELECT MARK_DESIGN_KSAFE(1);

dbadmin=> select table_name,separated_percent,transferred_percent from rebalance_status where table_name='test';
table_name | separated_percent | transferred_percent
-----------+-------------------+--------------------
(0 rows)

vbr -t replicate -c clone_TEST.ini --debug 3

**CLUSTER 2
**
dbadmin=> select version();

version

Vertica Analytic Database V10.0.1-3

dbadmin=>select * from test;

id

1

dbadmin=>select export_objects('','test');

export objects

CREATE TABLE public.test
(
id numeric(38,0)
);

CREATE PROJECTION public.test /+createtype(L)/
(
id
)
AS
SELECT test.id
FROM public.test
ORDER BY test.id
SEGMENTED BY hash(test.id) ALL NODES KSAFE 1;

SELECT MARK_DESIGN_KSAFE(1);

dbadmin=> select table_name,separated_percent,transferred_percent from rebalance_status where table_name='test';
table_name | separated_percent | transferred_percent
-----------+-------------------+--------------------
test | 0.00| 0.00
(1 row)

SruthiA

After you ran vbr replicate, you did not run rebalance right? Still we can see one row from your test table right?

lybhrwn742

Right, after I replicate the table I see a record in REBALANCE_STATUS (in cluster 1 I don't see since it was created in that cluster)

SruthiA

Can you share me the output of the following and attach the vbr log? I see that you ran it debug 3 option

select * from rebalance_status where table_name='test';

lybhrwn742

Hi SruthiA,

dbadmin=> select * from rebalance_status where table_name='test';
table_schema | table_name | separated_percent | transferred_percent | TABLE_SIZE_GB
-------------+------------+-------------------+---------------------+--------------
public | test | 0.00 | 0.00 | 0

dbadmin@hostname [~/VbrDir]
$ vbr -t replicate -c clone_TEST.ini --debug 3

Participating nodes: V_DatabaseName_node0001,V_DatabaseName_node0002,V_DatabaseName_node0003,V_DatabaseName_node0004,V_DatabaseName_node0005,V_DatabaseName_node0006,V_DatabaseName_node0007,V_DatabaseName_node0008,V_DatabaseName_node0009,V_DatabaseName_node0010,V_DatabaseName_node0011,V_DatabaseName_node0012,V_DatabaseName_node0013,V_DatabaseName_node0014,V_DatabaseName_node0015,V_DatabaseName_node0016,V_DatabaseName_node0017,V_DatabaseName_node0018
Starting replication of objects ['public.TEST'] from DatabaseName.
Snapshotting the source database.
Snapshot complete.
Copying catalog snapshot from source to destination.
Preparing destination database for replicating objects.
Prep complete, start syncing files.
Approximate bytes to copy: 0 of 164 total.
Progress: 0 out of 0 bytes (100%)
Finalizing object replication.
Complete object replication, removing snapshot from the source database.
Object replication complete!

SruthiA

I meant vbr log file for the replicate you have run.. It should be present in /tmp/vbr.

lybhrwn742

Is there anything specific I should look for? this is closed site so I can't attach the log.
Also I don't see any errors there.

Thanks.

Jim_Knicely

@ybbhosale - What are you seeing in the REMOTE_REPLICATION_STATUS table on the source database?

Example:

dbadmin=> SELECT * FROM remote_replication_status;
 objects |  snapshot_name  |                replication_point                 |  last_replicated_time  | replicated_epoch | current_epoch
---------+-----------------+--------------------------------------------------+------------------------+------------------+---------------
 test    | backup_snapshot | backup_snapshot_3R7MOX4G59S5FZA3WPSWBPGKXHWCIL01 | 2021-10-13 07:34:49-04 |               19 |            20
(1 row)

lybhrwn742

Hi Jim,

dbadmin=> select * from remote_replication_status;
objects | snapshot_name | replication_point | last_replicated_time | replicated_epoch | current_epoch
------------+---------------------+--------------------------------------------+----------------------------+------------------------+-------------------
public.test |snapshot_copy_objects |snapshot_copy_objects_ACF6D.. | 2021-10-17 18:10:41+03 | 2375600 | 2375622

Jim_Knicely

@lybhrwn742 - I tested replication using Vertica 10.0.1-3 between two three node clusters. I did not see any requirements for rebalancing on the target DB.

Can you run the following commands on both of your clusters?

SELECT get_node_dependencies();
SELECT COUNT(*) FROM projections WHERE NOT is_up_to_date;

Thanks!

lybhrwn742

    CLUSTER 1

dbadmin=> select get_node_dependencies();

000000000000000011 - cnt: 7
000000000000000110 - cnt: 56
000000000000001100 - cnt: 56
000000000000011000 - cnt: 9
000000000000110000 - cnt: 58
000000000001001000 - cnt: 2
000000000001100000 - cnt: 58
000000000011000000 - cnt: 7
000000000110000000 - cnt: 56
000000001100000000 - cnt: 56
000000011000000000 - cnt: 7
000000110000000000 - cnt: 56
000001100000000000 - cnt: 56
000010000000000001 - cnt: 56
000011000000000000 - cnt: 56
000011111111111111 - cnt: 6
000100000000000001 - cnt: 49
000100000000000010 - cnt: 49
001000000000001000 - cnt: 49
001000000000010000 - cnt: 49
010000000000100000 - cnt: 49
010000000001000000 - cnt: 49
100000000100000000 - cnt: 49
100000001000000000 - cnt: 49
111111111111111111 - cnt: 1

000000000000000001 - name: v_database_name_node0001
000000000000000010 - name: v_database_name_node0002
000000000000000100 - name: v_database_name_node0003
000000000000001000 - name: v_database_name_node0004
000000000000010000 - name: v_database_name_node0005
000000000000100000 - name: v_database_name_node0006
000000000001000000 - name: v_database_name_node0007
000000000010000000 - name: v_database_name_node0008
000000000100000000 - name: v_database_name_node0009
000000001000000000 - name: v_database_name_node0010
000000010000000000 - name: v_database_name_node0011
000000100000000000 - name: v_database_name_node0012
000001000000000000 - name: v_database_name_node0013
000010000000000000 - name: v_database_name_node0014
000100000000000000 - name: v_database_name_node0015
001000000000000000 - name: v_database_name_node0016
010000000000000000 - name: v_database_name_node0017
100000000000000000 - name: v_database_name_node0018

dbadmin=> select count(*) from projections where not is_up_to_date;

count

  0

(1 row)

CLUSTER 2

dbadmin=> select get_node_dependencies();

000000000000000011 - cnt: 2
000000000000000110 - cnt: 8
000000000000001001 - cnt: 1
000000000000001100 - cnt: 74
000000000000011000 - cnt: 8
000000000000110000 - cnt: 74
000000000001100000 - cnt: 3
000000000011000000 - cnt: 73
000000000110000000 - cnt: 74
000000001100000000 - cnt: 74
000000010000000001 - cnt: 6
000000011000000000 - cnt: 73
000000011111111111 - cnt: 6
000000100000000001 - cnt: 71
000000110000000000 - cnt: 2
000001000000000010 - cnt: 1
000001000000100000 - cnt: 71
000001000001000000 - cnt: 71
000001100000000000 - cnt: 3
000010000000000001 - cnt: 65
000010001000000000 - cnt: 1
000010010000000000 - cnt: 66
000100000001000000 - cnt: 1
000100000010000000 - cnt: 1
000100100000000000 - cnt: 65
001000000000000010 - cnt: 66
001000000000000100 - cnt: 66
010000000000000001 - cnt: 1
010000000000001000 - cnt: 65
010000000000010000 - cnt: 66
100000000000000001 - cnt: 2
100000000000000010 - cnt: 71
100000010000000000 - cnt: 1
100000100000000000 - cnt: 7
100001000000000000 - cnt: 2
100100000000000000 - cnt: 65
111111111111111111 - cnt: 1

000000000000000001 - name: v_database_name_node0004
000000000000000001 - name: v_database_name_node0001
000000000000000001 - name: v_database_name_node0002
000000000000000001 - name: v_database_name_node0003
000000000000000001 - name: v_database_name_node0005
000000000000000001 - name: v_database_name_node0006
000000000000000001 - name: v_database_name_node0007
000000000000000001 - name: v_database_name_node0008
000000000000000001 - name: v_database_name_node0009
000000000000000001 - name: v_database_name_node0010
000000000000000001 - name: v_database_name_node0011
000000000000000001 - name: v_database_name_node0013
000000000000000001 - name: v_database_name_node0014
000000000000000001 - name: v_database_name_node0015
000000000000000001 - name: v_database_name_node0016
000000000000000001 - name: v_database_name_node0017
000000000000000001 - name: v_database_name_node0018
000000000000000001 - name: v_database_name_node0012

dbadmin=> select count(*) from projections where not is_up_to_date;

count

  2

(1 row)

Thanks.

lybhrwn742

Correction for cluster 2 -

000000000000000001 - name: v_database_name_node0004
000000000000000010 - name: v_database_name_node0001
000000000000000100 - name: v_database_name_node0002
000000000000001000 - name: v_database_name_node0003
000000000000010000 - name: v_database_name_node0005
000000000000100000 - name: v_database_name_node0006
000000000001000000 - name: v_database_name_node0007
000000000010000000 - name: v_database_name_node0008
000000000100000000 - name: v_database_name_node0009
000000001000000000 - name: v_database_name_node0010
000000010000000000 - name: v_database_name_node0011
000000100000000000 - name: v_database_name_node0013
000001000000000000 - name: v_database_name_node0014
000010000000000000 - name: v_database_name_node0015
000100000000000000 - name: v_database_name_node0016
001000000000000000 - name: v_database_name_node0017
010000000000000000 - name: v_database_name_node0018
100000000000000000 - name: v_database_name_node0012