Unable to read bootstrap node metadata
Hi, everyone.
I'm trying to create database through adminTools and have a next error:
*** Creating database mvl ***
127.0.0.1 OK [vertica][(7, 0, 1)][000][all]
Creating database mvl
Starting bootstrap node v_mvl_node0001 (127.0.0.1)
ERROR: Unable to read bootstrap node metadata
Any ideas, guys?
I'm using VirtualBox + Ubuntu 12.04 + Vertica Server 7
I'm trying to create database through adminTools and have a next error:
*** Creating database mvl ***
127.0.0.1 OK [vertica][(7, 0, 1)][000][all]
Creating database mvl
Starting bootstrap node v_mvl_node0001 (127.0.0.1)
ERROR: Unable to read bootstrap node metadata
Any ideas, guys?
I'm using VirtualBox + Ubuntu 12.04 + Vertica Server 7
0
Comments
Did you notice any particular error/warning messages apart from this while creating db?Can you provide the admintools-dbadmin.log file in order to check this issue further?
Regards
Rahul
I've tried your variant, but it haven't solved my problem.
Sorry for long response. I tried to create database again and here is what I have in my admintools-dbadmin.log file after approving creation of database:
Jul 24 15:30:12 [7950] [adminExec.createDB] creating db ooo Jul 24 15:30:12 [7950] Running integrity checks Jul 24 15:30:12 [7950] {"type":"NODEHOST", "state":"VALIDATING", "host":"127.0.0.1", "node":"v_ooo_node0001", "dbname":"ooo", "msg":"Running Integrity Checks", "maxSteps":"6", "currStep":"1"} Jul 24 15:30:12 [7950] Will create database on port 5433 Jul 24 15:30:12 [7950] Checking that nodes are defined and installed Jul 24 15:30:12 [7950] {"type":"NODEHOST", "state":"VALIDATING", "host":"127.0.0.1", "node":"v_ooo_node0001", "dbname":"ooo", "msg":"Validating RPM", "maxSteps":"6", "currStep":"2"} Jul 24 15:30:12 [7950] <BashAdapter: dbadmin@127.0.0.1> (package version check): ["dpkg -l vertica | tail -1 | awk '{print $3}'"] Jul 24 15:30:12 [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) ['7.0.1-0'] Jul 24 15:30:12 [7950] [adminExec.validateRpmVersion]: checking remote nodes ['127.0.0.1'] for version [vertica][(7, 0, 1)][000] Jul 24 15:30:12 [7950] [adminExec.validateRpmVersion]: checking installed versions on host 127.0.0.1 Jul 24 15:30:12 [7950] <BashAdapter: dbadmin@127.0.0.1> (package version check): ["dpkg -l vertica | tail -1 | awk '{print $3}'"] Jul 24 15:30:12 [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) ['7.0.1-0'] Jul 24 15:30:12 [7950] [adminExec.validateRpmVersion]: found [vertica][(7, 0, 1)][000][all] Jul 24 15:30:12 [7950] {"type":"NODEHOST", "state":"VALIDATING", "host":"127.0.0.1", "node":"v_ooo_node0001", "dbname":"ooo", "msg":"Validating Cluster", "maxSteps":"6", "currStep":"3"} Jul 24 15:30:12 [7950] Nominated initiator node v_ooo_node0001 Jul 24 15:30:12 [7950] Creating catalog and data directories Jul 24 15:30:12 [7950] {"type":"NODEHOST", "state":"VALIDATING", "host":"127.0.0.1", "node":"v_ooo_node0001", "dbname":"ooo", "msg":"Creating Catalog and Data Directories", "maxSteps":"6", "currStep":"4"} Jul 24 15:30:12 [7950] <BashAdapter: dbadmin@127.0.0.1>: ['[ -e /home/dbadmin/ooo ]'] Jul 24 15:30:12 [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=1) [] Jul 24 15:30:12 [7950] [makeDir] mkdir -p --mode=770 /home/dbadmin/ooo/v_ooo_node0001_catalog Jul 24 15:30:12 [7950] <BashAdapter: dbadmin@127.0.0.1>: ['mkdir -p --mode=770 /home/dbadmin/ooo/v_ooo_node0001_catalog'] Jul 24 15:30:12 [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) [] Jul 24 15:30:12 [7950] [makeDir ret] 0 Jul 24 15:30:12 [7950] [makeDir] mkdir -p --mode=770 /home/dbadmin/ooo/v_ooo_node0001_data Jul 24 15:30:12 [7950] <BashAdapter: dbadmin@127.0.0.1>: ['mkdir -p --mode=770 /home/dbadmin/ooo/v_ooo_node0001_data'] Jul 24 15:30:12 [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) [] Jul 24 15:30:12 [7950] [makeDir ret] 0 Jul 24 15:30:12 [7950] [prepareDBs] echo 5433 >/home/dbadmin/ooo/port.dat Jul 24 15:30:12 [7950] <BashAdapter: dbadmin@127.0.0.1>: ['echo 5433 >/home/dbadmin/ooo/port.dat'] Jul 24 15:30:12 [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) [] Jul 24 15:30:12 [7950] <BashAdapter: dbadmin@127.0.0.1>: ['[ -e /opt/vertica/config/logrotate ]'] Jul 24 15:30:12 [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) [] Jul 24 15:30:12 [7950] [prepareDBs] logrotate create file:echo '/home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.log /home/dbadmin/ooo/v_ooo_node0001_catalog/UDxLogs/UDxFencedProcesses.log { # rotate weekly weekly # and keep for 52 weeks rotate 52 # no complaining if vertica did not start yet missingok # compress log after rotation compress # no creating a new empty log, vertica will do that nocreate # if set, only rotates when log size is greater than X size 10M # delete files after 90 days (not all logrotate pkgs support this keyword) # maxage 90 # we have 2 log files to rotate but only signal vertica once sharedscripts # signal vertica to reopen and create the log postrotate kill -USR1 'head -1 /home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.pid 2> /dev/null' 2> /dev/null || true endscript } /home/dbadmin/ooo/dbLog { # rotate weekly weekly # and keep for 52 weeks rotate 52 # no complaining if vertica did not start yet missingok # compress log after rotation compress # this log is stdout, so rotate by copying it aside and truncating copytruncate } ' > /opt/vertica/config/logrotate/ooo Jul 24 15:30:12 [7950] <BashAdapter: dbadmin@127.0.0.1>: ["echo '/home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.log /home/dbadmin/ooo/v_ooo_node0001_catalog/UDxLogs/UDxFencedProcesses.log {\n # rotate weekly\n weekly\n # and keep for 52 weeks\n rotate 52\n # no complaining if vertica did not start yet\n missingok\n # compress log after rotation\n compress\n # no creating a new empty log, vertica will do that\n nocreate\n # if set, only rotates when log size is greater than X\n size 10M\n # delete files after 90 days (not all logrotate pkgs support this keyword)\n# maxage 90\n # we have 2 log files to rotate but only signal vertica once\n sharedscripts\n # signal vertica to reopen and create the log\n postrotate\n kill -USR1 'head -1 /home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.pid 2> /dev/null' 2> /dev/null || true\n endscript\n}\n\n/home/dbadmin/ooo/dbLog {\n # rotate weekly\n weekly\n # and keep for 52 weeks\n rotate 52\n # no complaining if vertica did not start yet\n missingok\n # compress log after rotation\n compress\n # this log is stdout, so rotate by copying it aside and truncating\n copytruncate\n}\n' > /opt/vertica/config/logrotate/ooo"] Jul 24 15:30:12 [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) [] Jul 24 15:30:12 [7950] <BashAdapter: dbadmin@127.0.0.1>: ['touch /opt/vertica/config/admintools.conf'] Jul 24 15:30:12 [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) [] Jul 24 15:30:12 [7950] Starting DB in multi-node mode Jul 24 15:30:12 [7950] {"type":"NODEHOST", "state":"VALIDATING", "host":"127.0.0.1", "node":"v_ooo_node0001", "dbname":"ooo", "msg":"Creating and Starting Database", "maxSteps":"6", "currStep":"5"} Jul 24 15:30:12 [7950] [SSH.createDBMultiNodes] dbName 'ooo', logFile '/home/dbadmin/ooo/dbLog', catalogPath '/home/dbadmin/ooo', dataPath '/home/dbadmin/ooo', nodes '['v_ooo_node0001']', nodeHosts '['127.0.0.1']', nodeCatalogs '['v_ooo_node0001_catalog']', nodeDatas '['v_ooo_node0001_data']', userName 'dbadmin', host '127.0.0.1', spreadInfo '{'largecluster': None, 'logging': 'False', 'controlsubnet': 'default', 'mode': 'broadcast'}', portNo = '5433') Jul 24 15:30:12 [7950] <BashAdapter: dbadmin@127.0.0.1>: ['LANG=C /sbin/ip -oneline -family inet addr'] Jul 24 15:30:12 [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) ['1: lo inet 127.0.0.1/8 scope host lo\\ valid_lft forever preferred_lft forever', '2: eth0 inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0\\ valid_lft forever preferred_lft forever'] Jul 24 15:30:12 [7950] NetworkProfiles: {'127.0.0.1': [u'127.0.0.1', u'127.255.255.255']} Jul 24 15:30:12 [7950] Creating database Jul 24 15:30:12 [7950] [SSH.createDBMultiNodes] bootstrapping with idx 0 node v_ooo_node0001 on host 127.0.0.1 Jul 24 15:30:12 [7950] [SSH.createDBMultiNodes] bootstrap cmd '/opt/vertica/bin/bootstrap-catalog -C 'ooo' -H '127.0.0.1' -s 'v_ooo_node0001' -D '/home/dbadmin/ooo/v_ooo_node0001_catalog' -S '/home/dbadmin/ooo/v_ooo_node0001_data' -L '/opt/vertica/config/share/license.key' -p '5433' -c '127.0.0.1' -B '127.255.255.255' ' Jul 24 15:30:12 [7950] <BashAdapter: dbadmin@127.0.0.1> (Bootstrapping database ooo): <command hidden> Jul 24 15:30:12 [7950] [SSH.createDBMultiNodes] cmd output '['0', ['License installed for Vertica Community Edition ', '', 'Catalog successfully bootstrapped']]' Jul 24 15:30:12 [7950] <BashAdapter: dbadmin@127.0.0.1> (Reading bootstrap catalog): <command hidden> Jul 24 15:30:13 [7950] Unable to read bootstrap node metadata. Expected JSON DB desc, got: Traceback (most recent call last): File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 296, in <module> readAndOutputJSON(sys.argv[1], len(sys.argv) > 2 and sys.argv[2] == "silent") File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 288, in readAndOutputJSON db.loadFromCatalog(ce) File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 136, in loadFromCatalog self.name = ce.sendCmd("get singleton Database name").strip() File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/tools/CatalogEditor.py", line 53, in sendCmd self.ceproc.stdin.write(cmd+"\n") IOError: [Errno 32] Broken pipe Exception OSError: (32, 'Broken pipe') in <bound method CatalogEditor.__del__ of <vertica.tools.CatalogEditor.CatalogEditor instance at 0x1ff5ab8>> ignored Jul 24 15:30:13 [7950] Multi-node start returns: 0 Jul 24 15:30:13 [7950] [killVerticaProcess] [ -e /home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.pid ] Jul 24 15:30:13 [7950] <BashAdapter: dbadmin@127.0.0.1>: ['[ -e /home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.pid ]'] Jul 24 15:30:13 [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=1) [] Jul 24 15:30:13 [7950] [removeCatDataDirs] cat /home/dbadmin/ooo/dbLog >> /opt/vertica/log/adminTools-dbadmin.log Jul 24 15:30:13 [7950] <BashAdapter: dbadmin@127.0.0.1>: ['cat /home/dbadmin/ooo/dbLog >> /opt/vertica/log/adminTools-dbadmin.log'] Jul 24 15:30:13 [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=1) ['cat: /home/dbadmin/ooo/dbLog: No such file or directory'] Jul 24 15:30:13 [7950] [removeCatDataDirs] cat /home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.log >> /opt/vertica/log/adminTools-dbadmin.log Jul 24 15:30:13 [7950] <BashAdapter: dbadmin@127.0.0.1>: ['cat /home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.log >> /opt/vertica/log/adminTools-dbadmin.log'] Jul 24 15:30:13 [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=1) ['cat: /home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.log: No such file or directory'] Jul 24 15:30:13 [7950] <BashAdapter: dbadmin@127.0.0.1>: ['rm -rf /home/dbadmin/ooo'] Jul 24 15:30:13 [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) [] Jul 24 15:30:13 [7950] Multi-node DB creation unsuccessful Jul 24 15:30:13 [7950] Create DB complete, signaling FAIL
The most interesting moment for me is: Why does Vertica think that it's multi node db creation? I ran install_vertica script with one host, like "-s 127.0.0.1".
Also I've tried this https://community.vertica.com/vertica/topics/installing_vertica_7_0_0_1_rhel6_3x64_unable_to_create_... solution, but without success
Have you tried increasing the file limits to around 65K. You can use 'ulimit -n' or edit /etc/security/limits.conf.
I recently ran into a similar issue where a perfectly running Vertica did not start working after doing an OS upgrade, and the cause was the limits.conf file was replaced during upgrade, wiping out previous configs for file limits.
put your ip and hostname there....vertica is not able to resolve your hostname and ip
so correct your /etc/hosts... it will work for you.
i resolved it by putting correct ip and corresponding hostname in /etc/hosts
and it worked
Starting bootstrap node v_mvl_node0001 (127.0.0.1)
ERROR: Unable to read bootstrap node metadata
this is error happened because in /etc/hosts there is no entry for your machine ip and its hostname
do it and let me know....i will try to help
i believe correcting /etc/hosts will work for you.
Thanks
But please make sure you have correct /etc/hosts file in your nodes.
Example from one of the 3 hosts:
10.3.70.219 vertica-dev1 vertica-dev1.domain.com
In /opt/vertica/log/adminTools-dbadmin.log
Aug 29 20:33:14 [13331] <BashAdapter: vertica@10.3.70.219> (Bootstrapping database testdb): <command hidden> Aug 29 20:33:14 [13331] [SSH.createDBMultiNodes] cmd output '['0', ['License installed for companyx', '', 'Catalog succ
essfully bootstrapped']]'
Aug 29 20:33:14 [13331] <BashAdapter: vertica@10.3.70.219> (Reading bootstrap catalog): <command hidden>
Aug 29 20:33:15 [13331] Unable to read bootstrap node metadata. Expected JSON DB desc, got: Traceback (most recent
call last):
File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 382, in <module>
readAndOutputJSON(sys.argv[1], len(sys.argv) > 2 and sys.argv[2] == "silent")
File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 373, in readAndOutputJSON
db.loadFromCatalog(ce)
File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 221, in loadFromCatalog
self.name = ce.sendCmd("get singleton Database name").strip()
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/tools/CatalogEditor.py", line 72, in sendCmd
self.ceproc.stdin.write(cmd+"\n")
IOError: [Errno 32] Broken pipe
Aug 29 20:33:15 [13331] Multi-node start returns: 0
I had the same problem as you running on a similar setup. I had the same error in my log file and I eventually found the problem in another thread (https://community.vertica.com/vertica/topics/unable_to_read_bootstrap_node_metadata). I found that for some reason my dbadmin's open file limit was set too low (run ulimit -n or ulimit -a to check all limits on dbadmin), hope this helps, good luck!
-Andrew
I have the following problem:
*** Creating database: test ***
127.0.0.1 OK [vertica][(7, 1, 1)][000][all]
Creating database test
ERROR: Unable to bootstrap database: Unable to clean up temp directory: Could not remove file [/initrd.img]: Permission denied
Press RETURN to continue
What does it mean?
Check also the /tmp folder if dbadmin is able to write/read.
Thank you for your answer. The initrd.img file is in "/" directory. I manually removed the file. Then the program wanted to remove vmlinuz file and I also removed this file manually, but it didn't help. Finally, I gave access to the /tmp directory to everybody, but the result was the same as previously.
I installed Vertica on a virtual machine (ubuntu 14 lts) and suppose that maybe a problem is with not sufficient resources or maybe I did not set a parameter. It was the second time when I installed Vertica. The previous installation worked (however it wasn't on a virtual machine).
located in /opt/vertica/config folder
[Configuration]
last_port = 5433
tmp_dir = /tmp
default_base = /home/dbadmin
format = 3
install_opts = -s localhost -d /home/dziedzic/cloud_store/databases/Vertica/data -g dziedzic -u dziedzic -p '*******'
spreadlog = False
controlsubnet = default
controlmode = broadcast
[Cluster]
hosts = 127.0.0.1
[Nodes]
node0001 = 127.0.0.1,/home/dziedzic/cloud_store/databases/Vertica/data,/home/dziedzic/cloud_store/databases/Vertica/data
If not then change the configuaration file /etc/security/limits.conf Add a line: <user> - nofile 62363. It should work!
When I encounter this problem I change my locale settings and then this error has gone. Please check your locale settings by issuing "locale" in command line. If it is not en_US.UTF-8 it may be the cause of your problem.
To change locale settings, issue these commands:
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8