Error starting database after fresh install (Vertica 7.2.3-0 on Ubuntu 14.04)
Has anyone come across the following error before?
Any ideas what might be going on?
sudo -u dbadmin /opt/vertica/bin/adminTools -t create_db --database my_vertica -s 127.0.0.1
Info: no password specified, using none
Database with 1 or 2 nodes cannot be k-safe and it may lose data if it crashes
Distributing changes to cluster.
Creating database my_vertica
Starting bootstrap node v_my_vertica_node0001 (127.0.0.1)
Error: Failed to load the startup information from the bootstrap node.
Hint: Are your resource limits for new processes correct?
The dbadmin logs show the following:
2016-08-11 11:44:12.044 admintools/11486:0x2ae78d224340 [NewSSH.createDBMultiNodes] <ERROR> Failure to load startup information from initiator. More information follows.
2016-08-11 11:44:12.044 admintools/11486:0x2ae78d224340 [NewSSH.createDBMultiNodes] <ERROR> Unsuccessful vdatabase: <ATResult> status=2 host=127.0.0.1 error_type=<class 'vertica.engine.api.errors.ATReceiveFailure_Init'> error_message=Problem json decoding message '{"status": 2, "content": null, "error_type": "<class \'vertica.engine.api.errors.ATError\'>", "error_message": "Could not load from Catalog Editor.\\n Catalog Editor state \\nCatalogEditor instance\\nClosed? False\\nCE subprocess = <subprocess.Popen object at 0x2af57bf98390>\\nCE proc.poll = 1 CE proc.stdin = <open file \'<fdopen>\', mode \'wb\' at 0x2af57bf74d20> CE proc.stdout = <open file \'<fdopen>\', mode \'rb\' at 0x2af57bf74c00> Exception was Broken pipe\\nVertica Catalog Editor: broken pipe detected\\nAdded stdout messages to error.\\nBEGIN CE output\\nEND CE output", "exec_stack": "Traceback (most recent call last):\\n File \\"/opt/vertica/share/eggs/vertica/engine/api/at_runner.py\\", line 32, in exec_module\\n result = instance.execute(command)\\n File \\"/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/engine/api/compute_vdatabase.py\\", line 58, in execute\\n cmd.arguments[\'catalogpath\'])\\n File \\"/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/engine/api/compute_vdatabase.py\\", line 86, in _load_vdb\\n \\" Exception was {1}\\".format(ce, e))\\nATError: Could not load from Catalog Editor.\\n Catalog Editor state \\nCatalogEditor instance\\nClosed? False\\nCE subprocess = <subprocess.Popen object at 0x2af57bf98390>\\nCE proc.poll = 1 CE proc.stdin = <open file \'<fdopen>\', mode \'wb\' at 0x2af57bf74d20> CE proc.stdout = <open file \'<fdopen>\', mode \'rb\' at 0x2af57bf74c00> Exception was Broken pipe\\nVertica Catalog Editor: broken pipe detected\\nAdded stdout messages to error.\\nBEGIN CE output\\nEND CE output\\n"}\nStopping editor\nError stopping editor: [Errno 3] No such process'. Error was: Extra data: line 2 column 1 - line 3 column 49 (char 1543 - 1607).
Stack Trace from Initiator follows
Traceback (most recent call last):
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/network/adapters/bash_adapter.py", line 250, in exec_module
result = self._exec_module_impl(command, timeout)
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/network/adapters/bash_adapter.py", line 273, in _exec_module_impl
result.from_wire_message(raw_buffer)
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/engine/api/at_result.py", line 82, in from_wire_message
"{0}. Error was: {1}.".format(repr(msg), e))
ATReceiveFailure_Init: Problem json decoding message '{"status": 2, "content": null, "error_type": "<class \'vertica.engine.api.errors.ATError\'>", "error_message": "Could not load from Catalog Editor.\\n Catalog Editor state \\nCatalogEditor instance\\nClosed? False\\nCE subprocess = <subprocess.Popen object at 0x2af57bf98390>\\nCE proc.poll = 1 CE proc.stdin = <open file \'<fdopen>\', mode \'wb\' at 0x2af57bf74d20> CE proc.stdout = <open file \'<fdopen>\', mode \'rb\' at 0x2af57bf74c00> Exception was Broken pipe\\nVertica Catalog Editor: broken pipe detected\\nAdded stdout messages to error.\\nBEGIN CE output\\nEND CE output", "exec_stack": "Traceback (most recent call last):\\n File \\"/opt/vertica/share/eggs/vertica/engine/api/at_runner.py\\", line 32, in exec_module\\n result = instance.execute(command)\\n File \\"/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/engine/api/compute_vdatabase.py\\", line 58, in execute\\n cmd.arguments[\'catalogpath\'])\\n File \\"/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/engine/api/compute_vdatabase.py\\", line 86, in _load_vdb\\n \\" Exception was {1}\\".format(ce, e))\\nATError: Could not load from Catalog Editor.\\n Catalog Editor state \\nCatalogEditor instance\\nClosed? False\\nCE subprocess = <subprocess.Popen object at 0x2af57bf98390>\\nCE proc.poll = 1 CE proc.stdin = <open file \'<fdopen>\', mode \'wb\' at 0x2af57bf74d20> CE proc.stdout = <open file \'<fdopen>\', mode \'rb\' at 0x2af57bf74c00> Exception was Broken pipe\\nVertica Catalog Editor: broken pipe detected\\nAdded stdout messages to error.\\nBEGIN CE output\\nEND CE output\\n"}\nStopping editor\nError stopping editor: [Errno 3] No such process'. Error was: Extra data: line 2 column 1 - line 3 column 49 (char 1543 - 1607).
0
Comments
Can you post your admintools.conf file please?
/opt/vertica/config/admintools.conf
[Configuration]
last_port = 5433
tmp_dir = /tmp
default_base = /home/dbadmin
format = 3
install_opts = --failure-threshold HALT --accept-eula --dba-user-password-disabled --license CE -s '127.0.0.1'
at_debug = False
spreadlog = False
controlsubnet = default
ipv6 = False
controlmode = broadcast
unreachable_host_caching = True
admintools_config_version = 103
[Cluster]
hosts = 127.0.0.1
[Nodes]
node0001 = 127.0.0.1,/home/dbadmin,/home/dbadmin
[SSHConfig]
ssh_user =
ssh_ident =
ssh_options = -oConnectTimeout=30 -o TCPKeepAlive=no -o ServerAliveInterval=15 -o ServerAliveCountMax=2 -o StrictHostKeyChecking=no -o BatchMode=yes
What do you see in the admintools.log.
/opt/vertica/logs
https://community.dev.hpe.com/t5/Vertica-Forum/unable-to-read-node-metadata/td-p/227822
In checking the ulimits I have come across what seems to be a typo in the Vertica install scripts ("no file" should be "nofile"):
Note that we allowed the install script to create the dbadmin user for us.
In order for the limits to be picked up, we also had to make changes to /etc/pam.d/login as outlined in http://lzone.de/blog/Apply-changes-to-limits.conf-immediately