Error starting verticad -- "no spread installed"

Bill_DeStein

I installed Vertica-7.0.0.0-1 and did /sbin/service verticad start.  It fails, and I get this error message:

"Vertica: No Spread installed"

Can someone tell me why spread is not installed?

Chris Consentino replied to this thread

https://community.vertica.com/vertica/topics/_failed_to_create_database_vmart_using_vmware_ce_v_7_0_...

and wrote, "Spread is installed automatically as part of Vertica. It is used for cluster communication when you have multiple nodes installed."

Also, the installation guide says:  "The install script also creates an operating system user called spread, which is responsible for running the spread daemon. User spread is non-configurable and is set to no login."

But when I look in /etc/passwd, there is no "spread" user.

Could this be because I have a single node configuration.

Ben_Vandiver

Vertica 7 does not create a 'spread' user and does not register a 'spreadd' service.  The verticad service has a bug where it refers to the spreadd service.  You can either delete the lines that reference spreadd in /etc/init.d/verticad or wait for a forthcoming bug fix.

Bill_DeStein

Hey Ben, thanks for your attention to this matter.

I deleted the lines in /etc/init.d/verticad that reference spreadd.  Then I still had trouble because the init script expects to find this file 

/opt/vertica/config/users/dbadmin/installed.dat

and that file doesn't exist.  So I changed the init script to ignore that.  With that fix, the init script invokes this command:

    /opt/vertica/bin/adminTools -t command_host -cstart

The command appears to run successfully but there is no verticad running afterward.  And this file

    /opt/vertica/log/adminTools-dbadmin.log 

ends like this ...

    Feb 15 18:08:20  [isOkToRun] owner of admintools.conf (dbauser) is dbadmin.  

    Feb 15 18:08:20  [commandLineCtrl.commandHost] processing start  

    Feb 15 18:08:20  [commandLineCtrl.commandHost] this host's DBs: {}  

    Feb 15 18:08:20  [commandLineCtrl.commandHost] up DBs: [] 

    Feb 15 18:08:20  [commandLineCtrl.commandHost] should have started a DB, but didn't 

    Feb 15 18:08:20  [commandLineCtrl.commandHost] overall status: 0 

So it seems that I can't start verticad because there are no DBs defined, and I can't create a DB with adminTools because verticad isn't running.

Please advise on how to proceed.  Thanks,

Bill

Ben_Vandiver

Do you have a database defined?  I don't see you say anything about creating a database (this is a separate step from installing Vertica).  If you fire up the "graphical" version of adminTools, you can find create database under 6 "Configuration Menu" then 1 "Create Database".

If you have created a database, can you post the contents of admintools.conf?

Bill_DeStein

Hey Ben,

No, I'be been unable to create a database using the graphical version of adminTools.  When I try to do that, admin tools generates these error messages ...

----------------------

    adminTools Last Chance Error Handler running...

    raised error:  <type 'exceptions.ValueError'>

    error message: No JSON object could be decoded

    trace file:  /opt/vertica/log/adminTools-dbadmin.errors

    REPORT THIS INFORMATION TO TECHNICAL SUPPORT

    AND INCLUDE CONTENTS OF THE TRACE FILE IN YOUR REPORT

-----------------------

The backtrace in the adminTools log show the error was coming from line 1197 in SSH.py.  So I added some print statements to SSH.py.  See here

------------------------

    # step 2a: build VDatabase object

    vdcmd = getRemoteVDatabaseCommand(os.path.join(catalogPath, nodeCatalogs[ initiatorOffset ]))

    print "vdcmd: " + vdcmd

    status, ret = connection.execute(vdcmd, info_msg="Reading bootstrap catalog",hide=True)

    print "status: " + str(status)

    print "ret[1]: " + str(ret[1])

    db = VDatabase.VDatabase(jsonString="\n".join(ret[1]))

----------------------------

Here's the output when I try to create a database ....

----------------------------

vdcmd: /opt/vertica/oss/python/bin/python -m vertica.config.VDatabase /home/dbadmin/vmartdb/v_vmartdb_node0001_catalog /opt/vertica
status: False
ret[1]: ['Traceback (most recent call last):', ' 
File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 162, in  _run_module_as_main', '    "__main__", fname, loader, pkg_name)', '  
File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 72, in _run_code', '    exec code in run_globals', '  
File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 283, in <module>', '    db.loadFromCatalog(ce)', '  
File "/opt/vertica/share/eggs/vertica/config/VDatabase.py", line 136,in loadFromCatalog', '    self.name = ce.sendCmd("get singletonDatabase name").strip()', '  
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/tools/CatalogEditor.py", line 53, in sendCmd', '    self.ceproc.stdin.write(cmd+"\\n")', 'IOError: [Errno 32] Broken pipe', "Exception OSError: (32, 'Broken pipe') in <bound method CatalogEditor.__del__ of <vertica.tools.CatalogEditor.CatalogEditor instance at 0x25989e0>> ignored"]


-----------------------------------

So adminTools is failing in sendCmd method in the CatelogEditor class.    

Best,

Bill

Ben_Vandiver

Your database catalog directory is something like /home/dbadmin/mydb/v_mydb_node0001_catalog
(You can find it in /opt/vertica/config/admintools.conf)
Can you do an 'ls' in that directory?
I suspect the database directory is not properly set up.

One of my (running) dbs looks like:
[dbadmin@host01 v_acme_inc_db_node0001_catalog]$ ls
bootstrap-catalog.log  CopyErrorLogs  debug_log.conf  ErrorReport.txt  Snapshots           spread.conf  startup.log  UDxLogs       vertica.log              vertica.log-20140215.gz  vertica.pid
Catalog                DataCollector  editor.log      Libraries        spread.backup.conf  spread.pid   tmp          vertica.conf  vertica.log-20140214.gz  vertica.log-20140216.gz

If you can find the editor.log file in the directory and include any errors you see in that, that would also be helpful.

Bill_DeStein

Hey Ben,

Here's what's in that directory ...

-------------------------
[dbadmin@ip-172-31-21-166 ~]$ ls -laF /home/dbadmin/vmartdb/v_vmartdb_node0001_catalog
drwx------. 9 dbadmin verticadba  4096 Feb 15 23:31 ./
drwxr-xr-x. 4 dbadmin verticadba  4096 Feb 15 23:31 ../
-rw-------. 1 dbadmin verticadba  3102 Feb 15 23:31 bootstrap-catalog.log
drwx------. 4 dbadmin verticadba  4096 Feb 15 23:31 Catalog/
drwx------. 2 dbadmin verticadba  4096 Feb 15 23:31 CopyErrorLogs/
drwxr-x---. 2 dbadmin verticadba 20480 Feb 15 23:31 DataCollector/
-rw-------. 1 dbadmin verticadba  2199 Feb 15 23:31 debug_log.conf
drwx------. 2 dbadmin verticadba  4096 Feb 15 23:31 Libraries/
drwx------. 2 dbadmin verticadba  4096 Feb 15 23:31 Snapshots/
-rw-------. 1 dbadmin verticadba   262 Feb 15 23:31 spread.conf
drwx------. 2 dbadmin verticadba  4096 Feb 15 23:31 tmp/
drwx------. 2 dbadmin verticadba  4096 Feb 15 23:31 UDxLogs/
-rw-------. 1 dbadmin verticadba   921 Feb 15 23:31 vertica.conf
[dbadmin@ip-172-31-21-166 ~]$ 
-------------------------

There is no editor.log to share.

Best,

Bill

Ben_Vandiver

Hmm... what is the output of:
/opt/vertica/oss/python/bin/python -m vertica.config.VDatabase /home/dbadmin/vmartdb/v_vmartdb_node0001_catalog /opt/vertica
If that produces a traceback, can you instead run:
/opt/vertica/bin/vertica -Eversions -D /home/dbadmin/vmartdb/v_vmartdb_node0001_catalog
(all of this should be done as the dbadmin user - I'm currently suspecting some sort of permissions issue)

Bill_DeStein

Hey Ben,

It's working now.  I tried running the vertica subprocess from the command line, and it went like this ...

-----------------------------
[dbadmin@ip-172-31-21-166 ~]$ /opt/vertica/bin/vertica -D
/home/dbadmin/vmartdb/v_vmartdb_node0001_catalog -E
ERROR: Host does not meet minimum requirements:
Not enough open file handles allowed (1024 available/32768 required); see 'ulimit -n'.
[dbadmin@ip-172-31-21-166 ~]$
-----------------------------

So I changed ulimits for the dbadmin user and all is well.

You might want to consider adding the file handle check to the install_vertica script.

Thanks for your help with this.

BIll

Ben_Vandiver

Interesting.  The install_vertica does try that check.  I'll investigate.  Glad to hear that things are working for you now.