Vertica Backup fails with rsync code 23 error.

chandeeswarg

Hi, vertica version is 9.2.1-4. Our vbr.py process fails intermittently with the below error. so we are not able to run any backup/listbackup etc commands. but if we re-try multiple times, its working. can someone please help me on this ?

error:

/opt/vertica/bin/vbr.py --task listbackup --config-file test.ini

Error: Error accessing remote storage: failed accessing remote storage on nrusca-clp44004: rsync: rename "/vertica_backup/bmdbix/..index.LUxCP4" (in vbr) -> ".index": No such file or directory (2)

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1042) [sender=3.0.7]

returncode=23
Listbackup FAILED.

Ini file:
[Misc]
snapshotName = test_backup
restorePointLimit = 7
retryCount = 2
retryDelay = 180

[Database]
dbName = test
dbUser = vsd
[Transmission]
[Mapping]
v_test_node0001 = nrusca-clp44001:/vertica_backup/test
v_test_node0002 = nrusca-clp44002:/vertica_backup/test
v_test_node0003 = nrusca-clp44003:/vertica_backup/test
v_test_node0004 = nrusca-clp44004:/vertica_backup/test
v_test_node0005 = nrusca-clp44005:/vertica_backup/test
v_test_node0006 = nrusca-clp44006:/vertica_backup/test

Jim_Knicely

Hi,
In your post it says "Error accessing remote storage: failed accessing remote storage on nrusca-clp44004". Is the issue always with that same remote server (nrusca-clp44004)?

chandeeswarg

Hi,
Thank You for responding.
The error keeps rotating with all the nodes and not just one node. Once its node4, then its node2, sometimes its node3. So, if i keep issuing the command "/opt/vertica/bin/vbr.py --task backup/listbackup --config-file test.ini", sometimes it fails with the error "rsync code 23" sometimes it works fine with no issues.
Thank You.

Bryan_H

Hi, rsync error 23 is "Partial transfer due to error" This could mean some issue with network or the underlying storage hadrware.
In vbr.ini, you can try increasing retries and timeout settings. Also check whether you have optimized the sysctl and ulimit settings as recommended.

chandeeswarg

@Bryan_H said:
Hi, rsync error 23 is "Partial transfer due to error" This could mean some issue with network or the underlying storage hardware.
In vbr.ini, you can try increasing retries and timeout settings. Also check whether you have optimized the sysctl and ulimit settings as recommended.

Hi,
The settings we have in our environment are as below, which is think is within the permissible values as per vertica documentation.
vm.min_free_kbytes = 65032
fs.file-max = 129024
vm.max_map_count = 8257536

id

uid=580(vsd) gid=4021(verticadba) groups=4021(verticadba)

ulimit -n

128956

ulimit -u

132112

and I've increase the retry parameter to 4 and and the retrydelay to 5 mins but it fails.

/opt/vertica/bin/vbr.py --task listbackup --config-file test.ini --debug 3

Error: Error accessing remote storage: failed accessing remote storage on nrusca-clp44002: rsync: rename "/vertica_backup/bmdbix/..index.lDBrne" (in vbr) -> ".index": No such file or directory (2)

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1042) [sender=3.0.7]

returncode=23
Listbackup FAILED.

Let alone the backup, the listbackup command fails.
Thank You.