Backup Fails after running for few hours !! Urgent Help Required
Hi All,
I tried taking a full database backup. But it failed after running for almost 11 hrs.
Below are the errors received:
1. Error received from vbr.py with debug mode =3:
27775: vbr client subproc on 10.80.82.54 terminates with returncode 1. Details in vbr_v_proddb_node0001_client.log on that host.
Error msg: rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (5924682 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(601) [sender=3.0.7]
rsync failed!
27786: vbr client subproc on 10.80.82.47 terminates with returncode 1. Details in vbr_v_proddb_node0007_client.log on that host.
Error msg: rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: close failed on "/vertica_load/VINITA_IMP_DONOTDELETE/FULL_BACKUP/17SEP15/v_ dbproddb_node0007/Fulldbbkp_ dbPRODDB_16sept15~new~/data/ db_DATA/ dbPRODDB/v_ dbproddb_node0007_data/425/72057594049875425/.72057594049875425_0.fdb.vRxS3F": Remote I/O error (121)
rsync error: error in file IO (code 11) at receiver.c(731) [receiver=3.0.7]
rsync: connection unexpectedly closed (5826772 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(601) [sender=3.0.7]
rsync failed!
2. Message in vbr_<node>_client.log:
2015-09-17 09:27:15 rsync failed with code 12
2015-09-17 09:27:15 rsync failed!
2015-09-17 09:44:32 Transfer client process entry: my pid is 21870; task is backup.
2015-09-17 09:44:32 Read lock acquired on .ctlg file
2015-09-17 09:44:32 rsyncOptions
2015-09-17 09:44:32 ['/opt/vertica/bin/rsync', '--stats', '--whole-file', '--progress']
2015-09-17 09:44:32 linking/copying special files at client: .ctlg, .txt, .conf
2015-09-17 09:44:32 Traceback (most recent call last):
File "/tmp/vbr.py", line 3384, in work
remoteClient(args[0], args[1], args[2], args[3], args[4], args[5], args[6] == 'True')
File "/tmp/vbr.py", line 1083, in remoteClient
os.link(os.path.abspath(cDir+'/../'+g["snapshotName"]+'.udfs'), udfsPath)
OSError: [Errno 17] File exists
2015-09-17 10:02:09 Transfer client process entry: my pid is 25686; task is backup.
2015-09-17 10:02:09 Read lock acquired on .ctlg file
2015-09-17 10:02:09 rsyncOptions
2015-09-17 10:02:09 ['/opt/vertica/bin/rsync', '--stats', '--whole-file', '--progress']
2015-09-17 10:02:09 linking/copying special files at client: .ctlg, .txt, .conf
2015-09-17 10:02:09 Traceback (most recent call last):
File "/tmp/vbr.py", line 3384, in work
remoteClient(args[0], args[1], args[2], args[3], args[4], args[5], args[6] == 'True')
File "/tmp/vbr.py", line 1083, in remoteClient
os.link(os.path.abspath(cDir+'/../'+g["snapshotName"]+'.udfs'), udfsPath)
OSError: [Errno 17] File exists
What could be causing this issues ? The backup has run successfully previously.
Regards
Aadu
Comments
I had this problem recently. There could be a number of issues in play. Here are some hints to troubleshoot.
(1) make a test backup with a small table using the object attribute in the configuration file to reduce the iteration cycle. If it fails, it will fail after 1 minute of 2.
(2) check that your backup hosts are running the same version of rsync. It it best to kill the rsync daemon and let it get restarted automatically.
(3) ensure the rsync version and the vertica rpm version is the same on all nodes + backup hosts.
(4) check which rsync version gets invoked by verifying the PATH variable.
After trying all this, let me know if it still does not work. What worked for me was killing the rsync daemon service on the backup host.
I suspect there is a older/zombie version of rsync running.
list processes on vertica nodes and backup location to ensure there are no zombie/old rsync processes still running. If so, kill them and try again.