Vertica Cluster backup failing

rahul_dhar

Hi, I have installed vertica 6.0.1 in cluster (3 nodes) on RHEL 5.6. I have setup passwordless ssh key for each node. Created a snapshot file as below - [dradmin@datarep1 ~]$ /opt/vertica/bin/vbr.py --setupconfig Snapshot name (backup_snapshot): back_test Backup vertica configurations? (n) [y/n]: y Number of restore points (1): 1 Specify objects (no default): daadmin Vertica user name (dradmin): dradmin Save password to avoid runtime prompt? (n) [y/n]: y Password to save in vbr config file (no default): Node v_im22_node0001 Backup host name (no default): datarep1 Backup directory (no default): /Disk1/DB_BACKUP Node v_im22_node0002 Backup host name (no default): datarep2 Backup directory (no default): /Disk1/DB_BACKUP Node v_im22_node0003 Backup host name (no default): datarep3 Backup directory (no default): /Disk1/DB_BACKUP Config file name (back_test.ini): Change advanced settings? (n) [y/n]: n Saved vbr configuration to back_test.ini. [dradmin@datarep1 ~]$ /opt/vertica/bin/vbr.py --task backup --config-file /home/dradmin/back_test.ini Preparing... Found Database port: 5433 Copying... Traceback (most recent call last): File "/opt/vertica/bin/vbr.py", line 1867, in backup remoteCopyAll() File "/opt/vertica/bin/vbr.py", line 1735, in remoteCopyAll printlock = multiprocessing.Lock() File "/opt/vertica/oss/python/lib/python2.7/multiprocessing/__init__.py", line 171, in Lock return Lock() File "/opt/vertica/oss/python/lib/python2.7/multiprocessing/synchronize.py", line 121, in __init__ SemLock.__init__(self, SEMAPHORE, 1, 1) File "/opt/vertica/oss/python/lib/python2.7/multiprocessing/synchronize.py", line 49, in __init__ sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue) OSError: [Errno 13] Permission denied backup failed unexpectedly! cleaning up... Retrying... #1 ERROR 2310: Can't find projection When communicating with vertica, the process failed with code 1 backup failed! Retrying... #2 ERROR 2310: Can't find projection When communicating with vertica, the process failed with code 1 backup failed! Can anybody throw some light, what could be the issue. Regards Rahul

[Deleted User]

Hi Rahul, Well, that error is occurring in a Python system library (specifically, the "multiprocessing" library). I haven't seen it before personally. But from the error message above: sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue) OSError: [Errno 13] Permission denied Plugging those two lines into Google, the first result is: http://stackoverflow.com/questions/2009278/python-multiprocessing-permission-denied I haven't tried it myself, but that result (or others on the same search-results page) look quite promising. (In case that link stops working: The basic issue it cites is that Python's Multiprocessing module uses shared memory for concurrency control; therefore, /dev/shm must exist, must be the magic filesystem type that it's supposed to be, and must allow access to all users like it's supposed to, or at least to the dbadmin user so it can perform the backup. All three are typically the case by default, though sysadmins may change them for various reasons.) Adam

Prasanta_Pal

As suggested by Adam to look the link http://stackoverflow.com/questions/2009278/python-multiprocessing-permission-denied Which in turn may help, running as sudo to avoid permission error as below: [dradmin@datarep1 ~]$ sudo /opt/vertica/bin/vbr.py --task backup --config-file /home/dradmin/back_test.ini

rahul_dhar

Hi Adam / Prasanta, Thanks a lot for your help. Changing access to /dev/shm worked.

rahul_dhar

Hi Adam / Prasanta, Backup worked last time but now when i have scheduled the backup in crontab to run every morning 2 AM, it is failing with following error - [root@datarep1 tmp]# cat backup.log Preparing... Found Database port: 5433 ERROR 2310: Can't find projection ERROR 2310: Can't find projection ERROR 2310: Can't find projection When communicating with vertica, the process failed with code 1 backup failed! Retrying... #1 ERROR 2310: Can't find projection ERROR 2310: Can't find projection ERROR 2310: Can't find projection When communicating with vertica, the process failed with code 1 backup failed! Retrying... #2 ERROR 2310: Can't find projection ROLLBACK 2310: Can't find projection When communicating with vertica, the process failed with code 1 backup failed! Any idea? Rahul

Massimo_Loporch

On Cents 5 I solved without rebooting using : chmod 777 /dev/shm