Unable to read bootstrap node metadata

Victor_Melnik

Hi, everyone.

I'm trying to create database through adminTools and have a next error:

*** Creating database mvl ***
127.0.0.1 OK [vertica][(7, 0, 1)][000][all]
Creating database mvl
Starting bootstrap node v_mvl_node0001 (127.0.0.1)
ERROR: Unable to read bootstrap node metadata

Any ideas, guys?
I'm using VirtualBox + Ubuntu 12.04 + Vertica Server 7

Rahul_Choudhary

Hi Victor

Did you notice any particular error/warning messages apart from this while creating db?Can you provide the admintools-dbadmin.log file in order to check this issue further?

Regards
Rahul

Siddarth

Try rebooting your VM and then try to create the database.

Victor_Melnik

Hi, Siddarth.

I've tried your variant, but it haven't solved my problem.

Victor_Melnik

Hi, Rahul.

Sorry for long response. I tried to create database again and here is what I have in my admintools-dbadmin.log file after approving creation of database:

Jul 24 15:30:12  [7950] [adminExec.createDB] creating db ooo  Jul 24 15:30:12  [7950]  Running integrity checks  Jul 24 15:30:12  [7950] {"type":"NODEHOST", "state":"VALIDATING", "host":"127.0.0.1", "node":"v_ooo_node0001", "dbname":"ooo", "msg":"Running Integrity Checks", "maxSteps":"6", "currStep":"1"}  Jul 24 15:30:12  [7950]   Will create database on port 5433   Jul 24 15:30:12  [7950]   Checking that nodes are defined and installed  Jul 24 15:30:12  [7950] {"type":"NODEHOST", "state":"VALIDATING", "host":"127.0.0.1", "node":"v_ooo_node0001", "dbname":"ooo", "msg":"Validating RPM", "maxSteps":"6", "currStep":"2"}  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1> (package version check): ["dpkg -l vertica | tail -1 | awk '{print $3}'"]  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) ['7.0.1-0']  Jul 24 15:30:12  [7950] [adminExec.validateRpmVersion]: checking remote nodes ['127.0.0.1'] for version [vertica][(7, 0, 1)][000]  Jul 24 15:30:12  [7950] [adminExec.validateRpmVersion]: checking installed versions on host 127.0.0.1  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1> (package version check): ["dpkg -l vertica | tail -1 | awk '{print $3}'"]  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) ['7.0.1-0']  Jul 24 15:30:12  [7950] [adminExec.validateRpmVersion]: found [vertica][(7, 0, 1)][000][all]  Jul 24 15:30:12  [7950] {"type":"NODEHOST", "state":"VALIDATING", "host":"127.0.0.1", "node":"v_ooo_node0001", "dbname":"ooo", "msg":"Validating Cluster", "maxSteps":"6", "currStep":"3"}  Jul 24 15:30:12  [7950]  Nominated initiator node v_ooo_node0001  Jul 24 15:30:12  [7950]  Creating catalog and data directories  Jul 24 15:30:12  [7950] {"type":"NODEHOST", "state":"VALIDATING", "host":"127.0.0.1", "node":"v_ooo_node0001", "dbname":"ooo", "msg":"Creating Catalog and Data Directories", "maxSteps":"6", "currStep":"4"}  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: ['[ -e /home/dbadmin/ooo ]']  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=1) []  Jul 24 15:30:12  [7950] [makeDir] mkdir -p --mode=770 /home/dbadmin/ooo/v_ooo_node0001_catalog  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: ['mkdir -p --mode=770 /home/dbadmin/ooo/v_ooo_node0001_catalog']  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) []  Jul 24 15:30:12  [7950] [makeDir ret] 0  Jul 24 15:30:12  [7950] [makeDir] mkdir -p --mode=770 /home/dbadmin/ooo/v_ooo_node0001_data  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: ['mkdir -p --mode=770 /home/dbadmin/ooo/v_ooo_node0001_data']  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) []  Jul 24 15:30:12  [7950] [makeDir ret] 0  Jul 24 15:30:12  [7950] [prepareDBs] echo 5433 >/home/dbadmin/ooo/port.dat  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: ['echo 5433 >/home/dbadmin/ooo/port.dat']  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) []  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: ['[ -e /opt/vertica/config/logrotate ]']  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) []  Jul 24 15:30:12  [7950] [prepareDBs] logrotate create file:echo '/home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.log /home/dbadmin/ooo/v_ooo_node0001_catalog/UDxLogs/UDxFencedProcesses.log {     # rotate weekly     weekly     # and keep for 52 weeks     rotate 52     # no complaining if vertica did not start yet     missingok     # compress log after rotation     compress     # no creating a new empty log, vertica will do that     nocreate     # if set, only rotates when log size is greater than X     size 10M     # delete files after 90 days (not all logrotate pkgs support this keyword) #    maxage 90     # we have 2 log files to rotate but only signal vertica once     sharedscripts     # signal vertica to reopen and create the log     postrotate        kill -USR1 'head -1 /home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.pid 2> /dev/null' 2> /dev/null || true     endscript }  /home/dbadmin/ooo/dbLog {     # rotate weekly     weekly     # and keep for 52 weeks     rotate 52     # no complaining if vertica did not start yet     missingok     # compress log after rotation     compress     # this log is stdout, so rotate by copying it aside and truncating     copytruncate } ' > /opt/vertica/config/logrotate/ooo  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: ["echo '/home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.log /home/dbadmin/ooo/v_ooo_node0001_catalog/UDxLogs/UDxFencedProcesses.log {\n    # rotate weekly\n    weekly\n    # and keep for 52 weeks\n    rotate 52\n    # no complaining if vertica did not start yet\n    missingok\n    # compress log after rotation\n    compress\n    # no creating a new empty log, vertica will do that\n    nocreate\n    # if set, only rotates when log size is greater than X\n    size 10M\n    # delete files after 90 days (not all logrotate pkgs support this keyword)\n#    maxage 90\n    # we have 2 log files to rotate but only signal vertica once\n    sharedscripts\n    # signal vertica to reopen and create the log\n    postrotate\n       kill -USR1 'head -1 /home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.pid 2> /dev/null' 2> /dev/null || true\n    endscript\n}\n\n/home/dbadmin/ooo/dbLog {\n    # rotate weekly\n    weekly\n    # and keep for 52 weeks\n    rotate 52\n    # no complaining if vertica did not start yet\n    missingok\n    # compress log after rotation\n    compress\n    # this log is stdout, so rotate by copying it aside and truncating\n    copytruncate\n}\n' > /opt/vertica/config/logrotate/ooo"]  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) []  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: ['touch /opt/vertica/config/admintools.conf']  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) []  Jul 24 15:30:12  [7950]  Starting DB in multi-node mode  Jul 24 15:30:12  [7950] {"type":"NODEHOST", "state":"VALIDATING", "host":"127.0.0.1", "node":"v_ooo_node0001", "dbname":"ooo", "msg":"Creating and Starting Database", "maxSteps":"6", "currStep":"5"}  Jul 24 15:30:12  [7950] [SSH.createDBMultiNodes]  dbName 'ooo', logFile '/home/dbadmin/ooo/dbLog', catalogPath '/home/dbadmin/ooo', dataPath '/home/dbadmin/ooo', nodes '['v_ooo_node0001']', nodeHosts '['127.0.0.1']', nodeCatalogs '['v_ooo_node0001_catalog']', nodeDatas '['v_ooo_node0001_data']', userName 'dbadmin', host '127.0.0.1', spreadInfo '{'largecluster': None, 'logging': 'False', 'controlsubnet': 'default', 'mode': 'broadcast'}', portNo = '5433')  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: ['LANG=C /sbin/ip -oneline -family inet addr']  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) ['1: lo    inet 127.0.0.1/8 scope host lo\\       valid_lft forever preferred_lft forever', '2: eth0    inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0\\       valid_lft forever preferred_lft forever']  Jul 24 15:30:12  [7950] NetworkProfiles: {'127.0.0.1': [u'127.0.0.1', u'127.255.255.255']}  Jul 24 15:30:12  [7950]  Creating database  Jul 24 15:30:12  [7950] [SSH.createDBMultiNodes] bootstrapping with idx 0 node v_ooo_node0001 on host 127.0.0.1  Jul 24 15:30:12  [7950] [SSH.createDBMultiNodes] bootstrap cmd '/opt/vertica/bin/bootstrap-catalog -C 'ooo' -H '127.0.0.1' -s 'v_ooo_node0001' -D '/home/dbadmin/ooo/v_ooo_node0001_catalog' -S '/home/dbadmin/ooo/v_ooo_node0001_data' -L '/opt/vertica/config/share/license.key' -p '5433' -c '127.0.0.1' -B '127.255.255.255' '  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1> (Bootstrapping database ooo): <command hidden>  Jul 24 15:30:12  [7950] [SSH.createDBMultiNodes] cmd output '['0', ['License installed for Vertica Community Edition ', '', 'Catalog successfully bootstrapped']]'  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1> (Reading bootstrap catalog): <command hidden>  Jul 24 15:30:13  [7950] Unable to read bootstrap node metadata.  Expected JSON DB desc, got: Traceback (most recent call last):   File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 162, in _run_module_as_main     "__main__", fname, loader, pkg_name)   File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 72, in _run_code     exec code in run_globals   File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 296, in <module>     readAndOutputJSON(sys.argv[1], len(sys.argv) > 2 and sys.argv[2] == "silent")   File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 288, in readAndOutputJSON     db.loadFromCatalog(ce)   File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 136, in loadFromCatalog     self.name = ce.sendCmd("get singleton Database name").strip()   File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/tools/CatalogEditor.py", line 53, in sendCmd     self.ceproc.stdin.write(cmd+"\n") IOError: [Errno 32] Broken pipe Exception OSError: (32, 'Broken pipe') in <bound method CatalogEditor.__del__ of <vertica.tools.CatalogEditor.CatalogEditor instance at 0x1ff5ab8>> ignored  Jul 24 15:30:13  [7950]  Multi-node start returns: 0  Jul 24 15:30:13  [7950] [killVerticaProcess] [ -e /home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.pid ]  Jul 24 15:30:13  [7950] <BashAdapter: dbadmin@127.0.0.1>: ['[ -e /home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.pid ]']  Jul 24 15:30:13  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=1) []  Jul 24 15:30:13  [7950] [removeCatDataDirs] cat /home/dbadmin/ooo/dbLog >> /opt/vertica/log/adminTools-dbadmin.log  Jul 24 15:30:13  [7950] <BashAdapter: dbadmin@127.0.0.1>: ['cat /home/dbadmin/ooo/dbLog >> /opt/vertica/log/adminTools-dbadmin.log']  Jul 24 15:30:13  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=1) ['cat: /home/dbadmin/ooo/dbLog: No such file or directory']  Jul 24 15:30:13  [7950] [removeCatDataDirs] cat /home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.log >> /opt/vertica/log/adminTools-dbadmin.log  Jul 24 15:30:13  [7950] <BashAdapter: dbadmin@127.0.0.1>: ['cat /home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.log >> /opt/vertica/log/adminTools-dbadmin.log']  Jul 24 15:30:13  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=1) ['cat: /home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.log: No such file or directory']  Jul 24 15:30:13  [7950] <BashAdapter: dbadmin@127.0.0.1>: ['rm -rf /home/dbadmin/ooo']  Jul 24 15:30:13  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) []  Jul 24 15:30:13  [7950]  Multi-node DB creation unsuccessful  Jul 24 15:30:13  [7950]  Create DB complete, signaling FAIL

 

The most interesting moment for me is:

Jul 24 15:30:13  [7950]  Multi-node DB creation unsuccessful 

Why does Vertica think that it's multi node db creation? I ran install_vertica script with one host, like "-s 127.0.0.1".


Also I've tried this https://community.vertica.com/vertica/topics/installing_vertica_7_0_0_1_rhel6_3x64_unable_to_create_... solution, but without success 

Lasantha_Fernan

Hi,

Have you tried increasing the file limits to around 65K. You can use 'ulimit -n' or edit /etc/security/limits.conf.

I recently ran into a similar issue where a perfectly running Vertica did not start working after doing an OS upgrade, and the cause was the limits.conf file was replaced during upgrade, wiping out previous configs for file limits.

Arvind_Kumar

Hi look for your /etc/hosts

put your ip and hostname there....vertica is not able to resolve your hostname and ip 

so correct your /etc/hosts... it will work for you.

eli_revach

For localhost  singel node database , "-s localhost"   is not needed , do you try to run the create without the -s localhost   ?

Arvind_Kumar

I had the same issue for my single node,

i resolved it by putting correct ip and corresponding hostname in /etc/hosts
and it worked

Starting bootstrap node v_mvl_node0001 (127.0.0.1)
ERROR: Unable to read bootstrap node metadata

this is error happened because in /etc/hosts there is no entry for your machine ip and its hostname

do it and let me know....i will try to help

i believe correcting /etc/hosts will work for you.

Thanks  

Aidan_yp

I'm having the same error on a cluster of 3 nodes.  Any suggestions?

Arvind_Kumar

i will suggest to correct /etc/hosts file for all three nodes then install if you will get error even after doing this please share your /etc/hosts, i will try to dig it more.
But please make sure you have correct /etc/hosts file in your nodes.

Aidan_yp

I do have /etc/hosts with the correct host ip and hostname associated with it but it's still not working.

Example from one of the 3 hosts:
10.3.70.219 vertica-dev1 vertica-dev1.domain.com


In /opt/vertica/log/adminTools-dbadmin.log 

Aug 29 20:33:14  [13331] <BashAdapter: vertica@10.3.70.219> (Bootstrapping database testdb): <command hidden> Aug 29 20:33:14  [13331] [SSH.createDBMultiNodes] cmd output '['0', ['License installed for companyx', '', 'Catalog succ
essfully bootstrapped']]' 
Aug 29 20:33:14  [13331] <BashAdapter: vertica@10.3.70.219> (Reading bootstrap catalog): <command hidden> 
Aug 29 20:33:15  [13331] Unable to read bootstrap node metadata.  Expected JSON DB desc, got: Traceback (most recent 
call last):
  File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 382, in <module>
    readAndOutputJSON(sys.argv[1], len(sys.argv) > 2 and sys.argv[2] == "silent")
  File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 373, in readAndOutputJSON
    db.loadFromCatalog(ce)
  File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 221, in loadFromCatalog
    self.name = ce.sendCmd("get singleton Database name").strip()
  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/tools/CatalogEditor.py", line 72, in sendCmd
    self.ceproc.stdin.write(cmd+"\n")
IOError: [Errno 32] Broken pipe 
Aug 29 20:33:15  [13331]        Multi-node start returns: 0 

Aidan_yp

I did a new install with Vertica on version 7.0 and the problem isn't there.

Andrew_Coons

Hello Victor,

I had the same problem as you running on a similar setup. I had the same error in my log file and I eventually found the problem in another thread (https://community.vertica.com/vertica/topics/unable_to_read_bootstrap_node_metadata). I found that for some reason my dbadmin's open file limit was set too low (run ulimit -n or ulimit -a to check all limits on dbadmin), hope this helps, good luck!

-Andrew

Adam_1

Hello,

I have the following problem:
*** Creating database: test ***
            127.0.0.1 OK [vertica][(7, 1, 1)][000][all]
    Creating database test
ERROR: Unable to bootstrap database: Unable to clean up temp directory: Could not remove file [/initrd.img]: Permission denied
Press RETURN to continue

What does it mean?

Prasanta_Pal

Ii looks like a permission issue, if you know the directory where the file "initrd.img" is, please check if the dbadmin ( or the dba-user if you have different one) has the permission to read/write.
Check also the /tmp folder if dbadmin is able to write/read.

Adam_1

Hello Prasanta Pal,

Thank you for your answer. The initrd.img file is in "/" directory. I manually removed the file. Then the program wanted to remove vmlinuz file and I also removed this file manually, but it didn't help. Finally, I gave access to the /tmp directory to everybody, but the result was the same as previously.

I installed Vertica on a virtual machine (ubuntu 14 lts) and suppose that maybe a problem is with not sufficient resources or maybe I did not set a parameter. It was the second time when I installed Vertica. The previous installation worked (however it wasn't on a virtual machine).

Prasanta_Pal

Can you paste the admintools.conf file to review the details?
located in /opt/vertica/config folder

Adam_1

cat admintools.conf

[Configuration]
last_port = 5433
tmp_dir = /tmp
default_base = /home/dbadmin
format = 3
install_opts = -s localhost -d /home/dziedzic/cloud_store/databases/Vertica/data -g dziedzic -u dziedzic -p '*******'
spreadlog = False
controlsubnet = default
controlmode = broadcast

[Cluster]
hosts = 127.0.0.1

[Nodes]
node0001 = 127.0.0.1,/home/dziedzic/cloud_store/databases/Vertica/data,/home/dziedzic/cloud_store/databases/Vertica/data

Adam_1

Hello, the problem is that the configuration file: /etc/security/limits.conf was changed. Here is the solution: https://community.vertica.com/vertica/topics/not_enough_open_file_handles_allowed Basically, check if the command: ulimit -n gives you at least: 65536.

If not then change the configuaration file /etc/security/limits.conf Add a line: <user>   -    nofile    62363. It should work!

Ahmet_Fuat_Sung

When I encounter this problem I change my locale settings and then this error has gone. Please check your locale settings by issuing "locale" in command line. If it is not en_US.UTF-8 it may be the cause of your problem.

 

To change locale settings, issue these commands:

 

export LC_ALL=en_US.UTF-8

export LANG=en_US.UTF-8