Unable to read bootstrap node metadata

Victor_MelnikVictor_Melnik
in General Discussion
Hi, everyone.

I'm trying to create database through adminTools and have a next error:

*** Creating database mvl ***
127.0.0.1 OK [vertica][(7, 0, 1)][000][all]
Creating database mvl
Starting bootstrap node v_mvl_node0001 (127.0.0.1)
ERROR: Unable to read bootstrap node metadata

Any ideas, guys?
I'm using VirtualBox + Ubuntu 12.04 + Vertica Server 7

Comments

  • Rahul_ChoudharyRahul_Choudhary
    Hi Victor

    Did you notice any particular error/warning messages apart from this while creating db?Can you provide the admintools-dbadmin.log file in order to check this issue further?

    Regards
    Rahul
  • SiddarthSiddarth
    Try rebooting your VM and then try to create the database.
  • Victor_MelnikVictor_Melnik
    Hi, Siddarth.

    I've tried your variant, but it haven't solved my problem.
  • Victor_MelnikVictor_Melnik
    Hi, Rahul.

    Sorry for long response. I tried to create database again and here is what I have in my admintools-dbadmin.log file after approving creation of database: 
    Jul 24 15:30:12  [7950] [adminExec.createDB] creating db ooo  Jul 24 15:30:12  [7950]  Running integrity checks  Jul 24 15:30:12  [7950] {"type":"NODEHOST", "state":"VALIDATING", "host":"127.0.0.1", "node":"v_ooo_node0001", "dbname":"ooo", "msg":"Running Integrity Checks", "maxSteps":"6", "currStep":"1"}  Jul 24 15:30:12  [7950]   Will create database on port 5433   Jul 24 15:30:12  [7950]   Checking that nodes are defined and installed  Jul 24 15:30:12  [7950] {"type":"NODEHOST", "state":"VALIDATING", "host":"127.0.0.1", "node":"v_ooo_node0001", "dbname":"ooo", "msg":"Validating RPM", "maxSteps":"6", "currStep":"2"}  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1> (package version check): ["dpkg -l vertica | tail -1 | awk '{print $3}'"]  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) ['7.0.1-0']  Jul 24 15:30:12  [7950] [adminExec.validateRpmVersion]: checking remote nodes ['127.0.0.1'] for version [vertica][(7, 0, 1)][000]  Jul 24 15:30:12  [7950] [adminExec.validateRpmVersion]: checking installed versions on host 127.0.0.1  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1> (package version check): ["dpkg -l vertica | tail -1 | awk '{print $3}'"]  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) ['7.0.1-0']  Jul 24 15:30:12  [7950] [adminExec.validateRpmVersion]: found [vertica][(7, 0, 1)][000][all]  Jul 24 15:30:12  [7950] {"type":"NODEHOST", "state":"VALIDATING", "host":"127.0.0.1", "node":"v_ooo_node0001", "dbname":"ooo", "msg":"Validating Cluster", "maxSteps":"6", "currStep":"3"}  Jul 24 15:30:12  [7950]  Nominated initiator node v_ooo_node0001  Jul 24 15:30:12  [7950]  Creating catalog and data directories  Jul 24 15:30:12  [7950] {"type":"NODEHOST", "state":"VALIDATING", "host":"127.0.0.1", "node":"v_ooo_node0001", "dbname":"ooo", "msg":"Creating Catalog and Data Directories", "maxSteps":"6", "currStep":"4"}  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: ['[ -e /home/dbadmin/ooo ]']  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=1) []  Jul 24 15:30:12  [7950] [makeDir] mkdir -p --mode=770 /home/dbadmin/ooo/v_ooo_node0001_catalog  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: ['mkdir -p --mode=770 /home/dbadmin/ooo/v_ooo_node0001_catalog']  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) []  Jul 24 15:30:12  [7950] [makeDir ret] 0  Jul 24 15:30:12  [7950] [makeDir] mkdir -p --mode=770 /home/dbadmin/ooo/v_ooo_node0001_data  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: ['mkdir -p --mode=770 /home/dbadmin/ooo/v_ooo_node0001_data']  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) []  Jul 24 15:30:12  [7950] [makeDir ret] 0  Jul 24 15:30:12  [7950] [prepareDBs] echo 5433 >/home/dbadmin/ooo/port.dat  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: ['echo 5433 >/home/dbadmin/ooo/port.dat']  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) []  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: ['[ -e /opt/vertica/config/logrotate ]']  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) []  Jul 24 15:30:12  [7950] [prepareDBs] logrotate create file:echo '/home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.log /home/dbadmin/ooo/v_ooo_node0001_catalog/UDxLogs/UDxFencedProcesses.log {     # rotate weekly     weekly     # and keep for 52 weeks     rotate 52     # no complaining if vertica did not start yet     missingok     # compress log after rotation     compress     # no creating a new empty log, vertica will do that     nocreate     # if set, only rotates when log size is greater than X     size 10M     # delete files after 90 days (not all logrotate pkgs support this keyword) #    maxage 90     # we have 2 log files to rotate but only signal vertica once     sharedscripts     # signal vertica to reopen and create the log     postrotate        kill -USR1 'head -1 /home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.pid 2> /dev/null' 2> /dev/null || true     endscript }  /home/dbadmin/ooo/dbLog {     # rotate weekly     weekly     # and keep for 52 weeks     rotate 52     # no complaining if vertica did not start yet     missingok     # compress log after rotation     compress     # this log is stdout, so rotate by copying it aside and truncating     copytruncate } ' > /opt/vertica/config/logrotate/ooo  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: ["echo '/home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.log /home/dbadmin/ooo/v_ooo_node0001_catalog/UDxLogs/UDxFencedProcesses.log {\n    # rotate weekly\n    weekly\n    # and keep for 52 weeks\n    rotate 52\n    # no complaining if vertica did not start yet\n    missingok\n    # compress log after rotation\n    compress\n    # no creating a new empty log, vertica will do that\n    nocreate\n    # if set, only rotates when log size is greater than X\n    size 10M\n    # delete files after 90 days (not all logrotate pkgs support this keyword)\n#    maxage 90\n    # we have 2 log files to rotate but only signal vertica once\n    sharedscripts\n    # signal vertica to reopen and create the log\n    postrotate\n       kill -USR1 'head -1 /home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.pid 2> /dev/null' 2> /dev/null || true\n    endscript\n}\n\n/home/dbadmin/ooo/dbLog {\n    # rotate weekly\n    weekly\n    # and keep for 52 weeks\n    rotate 52\n    # no complaining if vertica did not start yet\n    missingok\n    # compress log after rotation\n    compress\n    # this log is stdout, so rotate by copying it aside and truncating\n    copytruncate\n}\n' > /opt/vertica/config/logrotate/ooo"]  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) []  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: ['touch /opt/vertica/config/admintools.conf']  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) []  Jul 24 15:30:12  [7950]  Starting DB in multi-node mode  Jul 24 15:30:12  [7950] {"type":"NODEHOST", "state":"VALIDATING", "host":"127.0.0.1", "node":"v_ooo_node0001", "dbname":"ooo", "msg":"Creating and Starting Database", "maxSteps":"6", "currStep":"5"}  Jul 24 15:30:12  [7950] [SSH.createDBMultiNodes]  dbName 'ooo', logFile '/home/dbadmin/ooo/dbLog', catalogPath '/home/dbadmin/ooo', dataPath '/home/dbadmin/ooo', nodes '['v_ooo_node0001']', nodeHosts '['127.0.0.1']', nodeCatalogs '['v_ooo_node0001_catalog']', nodeDatas '['v_ooo_node0001_data']', userName 'dbadmin', host '127.0.0.1', spreadInfo '{'largecluster': None, 'logging': 'False', 'controlsubnet': 'default', 'mode': 'broadcast'}', portNo = '5433')  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: ['LANG=C /sbin/ip -oneline -family inet addr']  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) ['1: lo    inet 127.0.0.1/8 scope host lo\\       valid_lft forever preferred_lft forever', '2: eth0    inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0\\       valid_lft forever preferred_lft forever']  Jul 24 15:30:12  [7950] NetworkProfiles: {'127.0.0.1': [u'127.0.0.1', u'127.255.255.255']}  Jul 24 15:30:12  [7950]  Creating database  Jul 24 15:30:12  [7950] [SSH.createDBMultiNodes] bootstrapping with idx 0 node v_ooo_node0001 on host 127.0.0.1  Jul 24 15:30:12  [7950] [SSH.createDBMultiNodes] bootstrap cmd '/opt/vertica/bin/bootstrap-catalog -C 'ooo' -H '127.0.0.1' -s 'v_ooo_node0001' -D '/home/dbadmin/ooo/v_ooo_node0001_catalog' -S '/home/dbadmin/ooo/v_ooo_node0001_data' -L '/opt/vertica/config/share/license.key' -p '5433' -c '127.0.0.1' -B '127.255.255.255' '  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1> (Bootstrapping database ooo): <command hidden>  Jul 24 15:30:12  [7950] [SSH.createDBMultiNodes] cmd output '['0', ['License installed for Vertica Community Edition ', '', 'Catalog successfully bootstrapped']]'  Jul 24 15:30:12  [7950] <BashAdapter: dbadmin@127.0.0.1> (Reading bootstrap catalog): <command hidden>  Jul 24 15:30:13  [7950] Unable to read bootstrap node metadata.  Expected JSON DB desc, got: Traceback (most recent call last):   File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 162, in _run_module_as_main     "__main__", fname, loader, pkg_name)   File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 72, in _run_code     exec code in run_globals   File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 296, in <module>     readAndOutputJSON(sys.argv[1], len(sys.argv) > 2 and sys.argv[2] == "silent")   File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 288, in readAndOutputJSON     db.loadFromCatalog(ce)   File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 136, in loadFromCatalog     self.name = ce.sendCmd("get singleton Database name").strip()   File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/tools/CatalogEditor.py", line 53, in sendCmd     self.ceproc.stdin.write(cmd+"\n") IOError: [Errno 32] Broken pipe Exception OSError: (32, 'Broken pipe') in <bound method CatalogEditor.__del__ of <vertica.tools.CatalogEditor.CatalogEditor instance at 0x1ff5ab8>> ignored  Jul 24 15:30:13  [7950]  Multi-node start returns: 0  Jul 24 15:30:13  [7950] [killVerticaProcess] [ -e /home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.pid ]  Jul 24 15:30:13  [7950] <BashAdapter: dbadmin@127.0.0.1>: ['[ -e /home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.pid ]']  Jul 24 15:30:13  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=1) []  Jul 24 15:30:13  [7950] [removeCatDataDirs] cat /home/dbadmin/ooo/dbLog >> /opt/vertica/log/adminTools-dbadmin.log  Jul 24 15:30:13  [7950] <BashAdapter: dbadmin@127.0.0.1>: ['cat /home/dbadmin/ooo/dbLog >> /opt/vertica/log/adminTools-dbadmin.log']  Jul 24 15:30:13  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=1) ['cat: /home/dbadmin/ooo/dbLog: No such file or directory']  Jul 24 15:30:13  [7950] [removeCatDataDirs] cat /home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.log >> /opt/vertica/log/adminTools-dbadmin.log  Jul 24 15:30:13  [7950] <BashAdapter: dbadmin@127.0.0.1>: ['cat /home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.log >> /opt/vertica/log/adminTools-dbadmin.log']  Jul 24 15:30:13  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=1) ['cat: /home/dbadmin/ooo/v_ooo_node0001_catalog/vertica.log: No such file or directory']  Jul 24 15:30:13  [7950] <BashAdapter: dbadmin@127.0.0.1>: ['rm -rf /home/dbadmin/ooo']  Jul 24 15:30:13  [7950] <BashAdapter: dbadmin@127.0.0.1>: (rc=0) []  Jul 24 15:30:13  [7950]  Multi-node DB creation unsuccessful  Jul 24 15:30:13  [7950]  Create DB complete, signaling FAIL
     

    The most interesting moment for me is: 
    Jul 24 15:30:13  [7950]  Multi-node DB creation unsuccessful
    Why does Vertica think that it's multi node db creation? I ran install_vertica script with one host, like "-s 127.0.0.1".


    Also I've tried this https://community.vertica.com/vertica/topics/installing_vertica_7_0_0_1_rhel6_3x64_unable_to_create_... solution, but without success 
  • Lasantha_FernanLasantha_Fernan
    Hi,

    Have you tried increasing the file limits to around 65K. You can use 'ulimit -n' or edit /etc/security/limits.conf.

    I recently ran into a similar issue where a perfectly running Vertica did not start working after doing an OS upgrade, and the cause was the limits.conf file was replaced during upgrade, wiping out previous configs for file limits.
  • Arvind_KumarArvind_Kumar Community Edition User ✭
    Hi look for your /etc/hosts

    put your ip and hostname there....vertica is not able to resolve your hostname and ip 

    so correct your /etc/hosts... it will work for you.
  • eli_revacheli_revach
    For localhost  singel node database , "-s localhost"   is not needed , do you try to run the create without the -s localhost   ?
  • Arvind_KumarArvind_Kumar Community Edition User ✭
    I had the same issue for my single node,

    i resolved it by putting correct ip and corresponding hostname in /etc/hosts
    and it worked

    Starting bootstrap node v_mvl_node0001 (127.0.0.1)
    ERROR: Unable to read bootstrap node metadata

    this is error happened because in /etc/hosts there is no entry for your machine ip and its hostname

    do it and let me know....i will try to help

    i believe correcting /etc/hosts will work for you.

    Thanks  
  • Aidan_ypAidan_yp
    I'm having the same error on a cluster of 3 nodes.  Any suggestions?
  • Arvind_KumarArvind_Kumar Community Edition User ✭
    i will suggest to correct /etc/hosts file for all three nodes then install if you will get error even after doing this please share your /etc/hosts, i will try to dig it more.
    But please make sure you have correct /etc/hosts file in your nodes.
  • Aidan_ypAidan_yp
    I do have /etc/hosts with the correct host ip and hostname associated with it but it's still not working.

    Example from one of the 3 hosts:
    10.3.70.219 vertica-dev1 vertica-dev1.domain.com


    In /opt/vertica/log/adminTools-dbadmin.log 

    Aug 29 20:33:14  [13331] <BashAdapter: vertica@10.3.70.219> (Bootstrapping database testdb): <command hidden> Aug 29 20:33:14  [13331] [SSH.createDBMultiNodes] cmd output '['0', ['License installed for companyx', '', 'Catalog succ
    essfully bootstrapped']]' 
    Aug 29 20:33:14  [13331] <BashAdapter: vertica@10.3.70.219> (Reading bootstrap catalog): <command hidden> 
    Aug 29 20:33:15  [13331] Unable to read bootstrap node metadata.  Expected JSON DB desc, got: Traceback (most recent 
    call last):
      File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 162, in _run_module_as_main
        "__main__", fname, loader, pkg_name)
      File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 72, in _run_code
        exec code in run_globals
      File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 382, in <module>
        readAndOutputJSON(sys.argv[1], len(sys.argv) > 2 and sys.argv[2] == "silent")
      File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 373, in readAndOutputJSON
        db.loadFromCatalog(ce)
      File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 221, in loadFromCatalog
        self.name = ce.sendCmd("get singleton Database name").strip()
      File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/tools/CatalogEditor.py", line 72, in sendCmd
        self.ceproc.stdin.write(cmd+"\n")
    IOError: [Errno 32] Broken pipe 
    Aug 29 20:33:15  [13331]        Multi-node start returns: 0 

  • Aidan_ypAidan_yp
    I did a new install with Vertica on version 7.0 and the problem isn't there.
  • Andrew_CoonsAndrew_Coons
    Hello Victor,

    I had the same problem as you running on a similar setup. I had the same error in my log file and I eventually found the problem in another thread (https://community.vertica.com/vertica/topics/unable_to_read_bootstrap_node_metadata). I found that for some reason my dbadmin's open file limit was set too low (run ulimit -n or ulimit -a to check all limits on dbadmin), hope this helps, good luck!

    -Andrew

  • Adam_1Adam_1
    Hello,

    I have the following problem:
    *** Creating database: test ***
                127.0.0.1 OK [vertica][(7, 1, 1)][000][all]
        Creating database test
    ERROR: Unable to bootstrap database: Unable to clean up temp directory: Could not remove file [/initrd.img]: Permission denied
    Press RETURN to continue

    What does it mean?
  • Prasanta_PalPrasanta_Pal - Select Field - Employee
    Ii looks like a permission issue, if you know the directory where the file "initrd.img" is, please check if the dbadmin ( or the dba-user if you have different one) has the permission to read/write.
    Check also the /tmp folder if dbadmin is able to write/read.

  • Adam_1Adam_1
    Hello Prasanta Pal,

    Thank you for your answer. The initrd.img file is in "/" directory. I manually removed the file. Then the program wanted to remove vmlinuz file and I also removed this file manually, but it didn't help. Finally, I gave access to the /tmp directory to everybody, but the result was the same as previously.

    I installed Vertica on a virtual machine (ubuntu 14 lts) and suppose that maybe a problem is with not sufficient resources or maybe I did not set a parameter. It was the second time when I installed Vertica. The previous installation worked (however it wasn't on a virtual machine).
  • Prasanta_PalPrasanta_Pal - Select Field - Employee
    Can you paste the admintools.conf file to review the details?
    located in /opt/vertica/config folder


  • Adam_1Adam_1
    cat admintools.conf

    [Configuration]
    last_port = 5433
    tmp_dir = /tmp
    default_base = /home/dbadmin
    format = 3
    install_opts = -s localhost -d /home/dziedzic/cloud_store/databases/Vertica/data -g dziedzic -u dziedzic -p '*******'
    spreadlog = False
    controlsubnet = default
    controlmode = broadcast

    [Cluster]
    hosts = 127.0.0.1

    [Nodes]
    node0001 = 127.0.0.1,/home/dziedzic/cloud_store/databases/Vertica/data,/home/dziedzic/cloud_store/databases/Vertica/data

  • Adam_1Adam_1
    Hello, the problem is that the configuration file: /etc/security/limits.conf was changed. Here is the solution: https://community.vertica.com/vertica/topics/not_enough_open_file_handles_allowed Basically, check if the command: ulimit -n gives you at least: 65536.

    If not then change the configuaration file /etc/security/limits.conf Add a line: <user>   -    nofile    62363. It should work!




  • Ahmet_Fuat_SungAhmet_Fuat_Sung

    When I encounter this problem I change my locale settings and then this error has gone. Please check your locale settings by issuing "locale" in command line. If it is not en_US.UTF-8 it may be the cause of your problem.

     

    To change locale settings, issue these commands:

     

    export LC_ALL=en_US.UTF-8

    export LANG=en_US.UTF-8

     

     

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Format
SpoilerCodeQuote
Heading 2Heading 1
Emoji
:):D:(;):/:o:s:p:'(:|B):#:*<3o:)>:)
Url
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file
Home • General DiscussionComment As ...