Error starting database after fresh install (Vertica 7.2.3-0 on Ubuntu 14.04)

Emanuel_Pordes_1

Has anyone come across the following error before?

Any ideas what might be going on?

 

sudo -u dbadmin /opt/vertica/bin/adminTools -t create_db --database my_vertica -s 127.0.0.1
Info: no password specified, using none
Database with 1 or 2 nodes cannot be k-safe and it may lose data if it crashes
Distributing changes to cluster.
	Creating database my_vertica
	Starting bootstrap node v_my_vertica_node0001 (127.0.0.1)
Error: Failed to load the startup information from the bootstrap node.
Hint: Are your resource limits for new processes correct?

The dbadmin logs show the following:

 

 

2016-08-11 11:44:12.044 admintools/11486:0x2ae78d224340 [NewSSH.createDBMultiNodes] <ERROR> Failure to load startup information from initiator. More information follows.
2016-08-11 11:44:12.044 admintools/11486:0x2ae78d224340 [NewSSH.createDBMultiNodes] <ERROR> Unsuccessful vdatabase: <ATResult> status=2 host=127.0.0.1 error_type=<class 'vertica.engine.api.errors.ATReceiveFailure_Init'> error_message=Problem json decoding message '{"status": 2, "content": null, "error_type": "<class \'vertica.engine.api.errors.ATError\'>", "error_message": "Could not load from Catalog Editor.\\n Catalog Editor state \\nCatalogEditor instance\\nClosed? False\\nCE subprocess = <subprocess.Popen object at 0x2af57bf98390>\\nCE proc.poll = 1 CE proc.stdin = <open file \'<fdopen>\', mode \'wb\' at 0x2af57bf74d20> CE proc.stdout = <open file \'<fdopen>\', mode \'rb\' at 0x2af57bf74c00> Exception was Broken pipe\\nVertica Catalog Editor: broken pipe detected\\nAdded stdout messages to error.\\nBEGIN CE output\\nEND CE output", "exec_stack": "Traceback (most recent call last):\\n  File \\"/opt/vertica/share/eggs/vertica/engine/api/at_runner.py\\", line 32, in exec_module\\n    result = instance.execute(command)\\n  File \\"/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/engine/api/compute_vdatabase.py\\", line 58, in execute\\n    cmd.arguments[\'catalogpath\'])\\n  File \\"/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/engine/api/compute_vdatabase.py\\", line 86, in _load_vdb\\n    \\" Exception was {1}\\".format(ce, e))\\nATError: Could not load from Catalog Editor.\\n Catalog Editor state \\nCatalogEditor instance\\nClosed? False\\nCE subprocess = <subprocess.Popen object at 0x2af57bf98390>\\nCE proc.poll = 1 CE proc.stdin = <open file \'<fdopen>\', mode \'wb\' at 0x2af57bf74d20> CE proc.stdout = <open file \'<fdopen>\', mode \'rb\' at 0x2af57bf74c00> Exception was Broken pipe\\nVertica Catalog Editor: broken pipe detected\\nAdded stdout messages to error.\\nBEGIN CE output\\nEND CE output\\n"}\nStopping editor\nError stopping editor: [Errno 3] No such process'. Error was: Extra data: line 2 column 1 - line 3 column 49 (char 1543 - 1607).

        Stack Trace from Initiator follows
Traceback (most recent call last):
  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/network/adapters/bash_adapter.py", line 250, in exec_module
    result = self._exec_module_impl(command, timeout)
  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/network/adapters/bash_adapter.py", line 273, in _exec_module_impl
    result.from_wire_message(raw_buffer)
  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/engine/api/at_result.py", line 82, in from_wire_message
    "{0}. Error was: {1}.".format(repr(msg), e))
ATReceiveFailure_Init: Problem json decoding message '{"status": 2, "content": null, "error_type": "<class \'vertica.engine.api.errors.ATError\'>", "error_message": "Could not load from Catalog Editor.\\n Catalog Editor state \\nCatalogEditor instance\\nClosed? False\\nCE subprocess = <subprocess.Popen object at 0x2af57bf98390>\\nCE proc.poll = 1 CE proc.stdin = <open file \'<fdopen>\', mode \'wb\' at 0x2af57bf74d20> CE proc.stdout = <open file \'<fdopen>\', mode \'rb\' at 0x2af57bf74c00> Exception was Broken pipe\\nVertica Catalog Editor: broken pipe detected\\nAdded stdout messages to error.\\nBEGIN CE output\\nEND CE output", "exec_stack": "Traceback (most recent call last):\\n  File \\"/opt/vertica/share/eggs/vertica/engine/api/at_runner.py\\", line 32, in exec_module\\n    result = instance.execute(command)\\n  File \\"/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/engine/api/compute_vdatabase.py\\", line 58, in execute\\n    cmd.arguments[\'catalogpath\'])\\n  File \\"/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/engine/api/compute_vdatabase.py\\", line 86, in _load_vdb\\n    \\" Exception was {1}\\".format(ce, e))\\nATError: Could not load from Catalog Editor.\\n Catalog Editor state \\nCatalogEditor instance\\nClosed? False\\nCE subprocess = <subprocess.Popen object at 0x2af57bf98390>\\nCE proc.poll = 1 CE proc.stdin = <open file \'<fdopen>\', mode \'wb\' at 0x2af57bf74d20> CE proc.stdout = <open file \'<fdopen>\', mode \'rb\' at 0x2af57bf74c00> Exception was Broken pipe\\nVertica Catalog Editor: broken pipe detected\\nAdded stdout messages to error.\\nBEGIN CE output\\nEND CE output\\n"}\nStopping editor\nError stopping editor: [Errno 3] No such process'. Error was: Extra data: line 2 column 1 - line 3 column 49 (char 1543 - 1607).

 

 

kaurora

Can you post your admintools.conf file please? 

 

/opt/vertica/config/admintools.conf

Emanuel_Pordes_1

[Configuration]
last_port = 5433
tmp_dir = /tmp
default_base = /home/dbadmin
format = 3
install_opts = --failure-threshold HALT --accept-eula --dba-user-password-disabled --license CE -s '127.0.0.1'
at_debug = False
spreadlog = False
controlsubnet = default
ipv6 = False
controlmode = broadcast
unreachable_host_caching = True
admintools_config_version = 103

[Cluster]
hosts = 127.0.0.1

[Nodes]
node0001 = 127.0.0.1,/home/dbadmin,/home/dbadmin

[SSHConfig]
ssh_user =
ssh_ident =
ssh_options = -oConnectTimeout=30 -o TCPKeepAlive=no -o ServerAliveInterval=15 -o ServerAliveCountMax=2 -o StrictHostKeyChecking=no -o BatchMode=yes

kaurora

What do you see in the admintools.log. 

 

/opt/vertica/logs

Emanuel_Pordes_1

https://community.dev.hpe.com/t5/Vertica-Forum/unable-to-read-node-metadata/td-p/227822

 

In checking the ulimits I have come across what seems to be a typo in the Vertica install scripts ("no file" should be "nofile"):

 

+sudo cat /etc/security/limits.conf

root hard nofile 72000
root soft nofile 96000
travis hard nofile 64000
travis soft nofile 72000
# The following 1 line added by Vertica tools. 2016-08-12 13:39:58
dbadmin    -    nice    0
# The following 1 line added by Vertica tools. 2016-08-12 13:39:58
dbadmin    -    no file    65536

 

Note that we allowed the install script to create the dbadmin user for us.

 

Emanuel_Pordes_1

In order for the limits to be picked up, we also had to make changes to /etc/pam.d/login as outlined in http://lzone.de/blog/Apply-changes-to-limits.conf-immediately