Copy Cluster Fails with no "vbr_worker.py"

msanjib

Hi Team,

I have been trying setup a copycluster job from a source of 3 nodes cluster to another site of exactly same configuration only different ips. Al pre-requisites are met , but while running the job getting some weird error.

Console Out:
ontinued use of this database is in violation of the current license agreement.
Maximum licensed raw data size: 1.00TB
Current raw data size: 2.39TB
License utilization: 239%
IMMEDIATE ACTION IS REQUIRED, PLEASE CONTACT VERTICA
Error: Worker process exited with error: Authorized users only
/opt/vertica/oss/python/bin/python: can't open file '/appdefender/vertica/bin/vbr_worker.py': [Errno 2] No such file or directory

Copycluster FAILED.

Detail console out:
---------------------verticauser@PE2RASPDB03PROD-0 ~]$ vbr -t copycluster -c copycluster.ini --debug 3
NOTICE 2001:
***** NOTICE OF LICENSE NON-COMPLIANCE *****
Continued use of this database is in violation of the current license agreement.
Maximum licensed raw data size: 1.00TB
Current raw data size: 2.39TB
License utilization: 239%
IMMEDIATE ACTION IS REQUIRED, PLEASE CONTACT VERTICA
NOTICE 2001:
***** NOTICE OF LICENSE NON-COMPLIANCE *****
Continued use of this database is in violation of the current license agreement.
Maximum licensed raw data size: 1.00TB
Current raw data size: 2.39TB
License utilization: 239%
IMMEDIATE ACTION IS REQUIRED, PLEASE CONTACT VERTICA
Error: Worker process exited with error: Authorized users only
/opt/vertica/oss/python/bin/python: can't open file '/appdefender/vertica/bin/vbr_worker.py': [Errno 2] No such file or directory

Copycluster FAILED.

======================================================== The snippet of the debug message from /tmp/cbr/*.log file

BatchMode=yes -o ExitOnForwardFailure=yes -o PasswordAuthentication=no -o StrictHostKeyChecking=no -oNoHostAuthenticationForLocalhost=yes -L42794:127.0.0.1:37210 10.189.199.xx /opt/vertica/oss/python/bin/python /appdefender/vertica/bin/vbr_worker.py DUMMY
2020-09-16 16:06:34 10.189.199.96 vbr Checked.
2020-09-16 16:06:34 10.189.199.95 vbr Checked.
2020-09-16 16:06:34 10.189.199.97 vbr Checked.
2020-09-16 16:06:43 localhost vsql /opt/vertica/bin/vsql -q -t -X -dappdefenderdb -p5433 -Uverticauser -h10.189.199.96
2020-09-16 16:06:43 localhost vsql select log_vbr_invocations('Copy Cluster Task', '/tmp/vbr/vbr_2020-09-16-160620_SDH7Z21A.log', 'AS411Q1B7ZFWQ5LIX6DED3M9PP8H4KC6', 'Fail');
2020-09-16 16:06:43 localhost vsql Warning from vsql: NOTICE 2001:
***** NOTICE OF LICENSE NON-COMPLIANCE *****
Continued use of this database is in violation of the current license agreement.
Maximum licensed raw data size: 1.00TB
Current raw data size: 2.39TB
License utilization: 239%
IMMEDIATE ACTION IS REQUIRED, PLEASE CONTACT VERTICA.
2020-09-16 16:06:43 localhost vsql Read from vsql: Logged vbr invocations
2020-09-16 16:06:43 localhost vbr Traceback (most recent call last):
File "/opt/vertica/bin/vbr", line 10023, in main
if vbr_task.run():
​

Hibiki

Thank you for your confirmation. Did you see any environment variables start with 'VBR'?
$ env | sort | grep -e '^VBR'
vbr_worker.py file has to be located in /opt/vertica/bin directory. Does '/appdefender/vertica/bin/vbr_worker.py' file exist on all nodes? Do you know which tool has moved the files to /appdefender directory and when? Do you know how to move the files to the original directories?

Hibiki

Can you check the following items?

1. Your cluster has had the data more than licensed size. Please run the following statements on vsql and see if all statements works.

=> CREATE TABLE test_20200917 (c1 INT);
=> INSERT INTO test_20200917 VALUES (1);
=> COMMIT;
=> SELECT count(*) FROM test_20200917;
=> DROP TABLE test_20200917;

2. Please check the environment variables set to dbadmin OS user by using the following command. Usually no variables.

$ env | sort | grep -e '^PYTHON' -e '^VBR'

3. Please check the existence of vbr_worker.py file by using the following command.

$ ls -l /opt/vertica/bin/vbr*

4. Have you run any commands related to 'appdefender' on the Vertica nodes?

5. Please check the directories and files under /appdefender directory by using the following command. If this directory exists, the output of this command may be huge. So if necessary, please upload this output as the attached file.

$ find /appdefender

msanjib

Thanks. I will definitely try that out.
I will appreciate if you can guide me how to reduce the raw data size to minimum without any impact in db. I have option to delete the db but the issue is that it won't educe the size to compliance limit immediately.
My disk space is usage is hardly 200 GB but how come the raw size shows 2.3 TB.

msanjib

The given statement to create a table is working.
Any specific files you would like to see as the /appdefender/ list is very long.

I am using a different vertica dba user "verticauser"
env | sort | grep -e '^PYTHON' -e '^VBR'
env for PYTHON is not set.

msanjib

appdefender/vertica/bin/vbr_worker.py is present and owner as root and execute permission is also present.
IN fact, I did a test by running manually .
verticauser$@ $appdefender/vertica/bin/vbr_worker.py /appdefender/vertica/bin/vbr_worker.py
It works .

msanjib

I have fixed the License compliance issue. Now I am having a typical issue .
Error: Worker process exited with error: Authorized users only
/opt/vertica/oss/python/bin/python: can't open file '/appdefender/vertica/bin/vbr_worker.py': [Errno 2] No such file or directory

Please help.

msanjib

You are absolutely correct. Thanks. In fact, I verified the same in source nodes but forgot to verify in target nodes. I used one soft link and then checked the binary was missing in that location.
Once I set it correctly it worked.