create DB failed due to node connectivity issues

Hi,
I installed vertica on 3 hosts and the installation worked well, but when I want to create a database, I get this error :

*** Creating database: test ***
Press RETURN to continue
create DB failed due to node connectivity issues
All nodes must be able to connect to each other via passwordless ssh
Could not reach node 37.XX.XX.53
Could not reach node 94.XX.XX.94
Press RETURN to continue

I am using the user dbadmin and the passwordless connection works perfectly fine between all dbamin account on each host

ssh 37.XX.XX.53
and
ssh 94.XX.XX.94
work without asking any password (when using dbadmin account)

Does someone know what can be the problem ?

I am using centos 7 and the only thing I changed in sshd.config is PermitRootLogin (set to without-password), but i'ml pretty sure that's not what caused that problem.

Maibe it is a permission issue ? :
on each host :
ls -la /home/dbamin
gives
drwx------ 3 dbadmin verticadba 4096 17 août 19:36 .
drwxr-xr-x 27 root root 4096 17 août 19:35 ..
-rw------- 1 dbadmin verticadba 91 18 août 17:51 .bash_history
-rw-r--r-- 1 dbadmin verticadba 18 7 déc. 2016 .bash_logout
-rw-r--r-- 1 dbadmin verticadba 193 7 déc. 2016 .bash_profile
-rw-r--r-- 1 dbadmin verticadba 290 17 août 19:35 .bashrc
drwx------ 2 dbadmin verticadba 4096 17 août 19:35 .ssh

and
ls -la .ssh
drwx------ 2 dbadmin verticadba 4096 17 août 19:35 .
drwx------ 3 dbadmin verticadba 4096 17 août 19:36 ..
-rw-r--r-- 1 dbadmin verticadba 381 17 août 19:35 authorized_keys2
-rw------- 1 dbadmin verticadba 1675 17 août 19:35 id_rsa
-rw------- 1 dbadmin verticadba 381 17 août 19:35 id_rsa.pub

Thanks in advance
Félicien

Answers

  • Jim_KnicelyJim_Knicely - Select Field - Administrator

    Hi,

    All nodes need to communicate with each other...

    Make sure:

    node1 can SSH to node2
    node1 can SSH to node3

    node2 can SSH to node1
    node2 can SSH to node3

    node3 can SSH to node1
    node3 can SSH to node2

    One of 'em might be missing the required SSH fingerprint?

  • Hi Jim,
    Thanks for your answer but yes, all node can communicate with each other..
    I also tried to stop the firewall and nothing changes.

    Manually, the ssh works perfectly between dbadmin users in all hosts.

  • Jim_KnicelyJim_Knicely - Select Field - Administrator
    edited August 2017
  • The ports seems available, (none is used by another app), and the firewall don't bllock them.

    I just saw that in the logs :

    2017-08-22 16:42:03.260 admintools/521026:0x66f9bb1b0700 [SSHConnection._login_if_needed] : login error: Error! pty.fork() failed: out of pty devices
    2017-08-22 16:42:03.260 admintools/521026:0x66f9c4797740 [AdapterConnectionPool_3.connect2_using_stored_auth] Unable to SSH as dbadmin to 188.XX.XX.95: Error! pty.fork() failed: out of pty devices
    2017-08-22 16:42:03.261 admintools/521026:0x66f9ba92a700 [SSHConnection._do_login] : pexpect.spawn failed too many times: Error! pty.fork() failed: out of pty devices
    Traceback (most recent call last):
    File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/network/adapters/pexpect_adapter.py", line 251, in _do_login
    connectStr
    File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/utils/pexpect.py", line 429, in init
    self._spawn (command, args)
    File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/utils/pexpect.py", line 537, in _spawn
    raise ExceptionPexpect('Error! pty.fork() failed: ' + str(e))
    ExceptionPexpect: Error! pty.fork() failed: out of pty devices

    I don't understand why I get this error because my number of PTy is fixed at 4096 (/proc/sys/kernel/pty/max) and the current number of used pty (/proc/sys/kernel/pty/nr) is less than 3 on all hosts.

    Maibe is this a hint ?

    Thanks

  • Jim_KnicelyJim_Knicely - Select Field - Administrator
    edited August 2017

    Make sure that the "User Process Limit" is set correctly on each node. 4096 is probably not high enough. See:

    https://my.vertica.com/docs/8.1.x/HTML/index.htm#Authoring/InstallationGuide/BeforeYouInstall/UserProcessLimit.htm

  • Hi,
    I'm trying again to install vertica but face the exact same problem :
    I downloaded vertica9 and also tried with verticaMC. I also verified that the passwordless ssh works between all hosts.
    I'm still getting the same error (could not reach node) and in the logs I can see :

    : login start: ssh -oConnectTimeout=30 -o TCPKeepAlive=no -o ServerAliveInterval=15 -o ServerAliveCountMax=2 -o StrictHostKeyChecking=no -o BatchMode=yes -l dbadmin 37.XX.XX.53
    : pexpect.spawn failed: Error! pty.fork() failed: out of pty devices, trying again.
    : pexpect.spawn failed too many times: Error! pty.fork() failed: out of pty devices

    But when I copy the exact same command (ssh -oConnectTimeout=30 -o TCPKeepAlive=no -o ServerAliveInterval=15 -o ServerAliveCountMax=2 -o StrictHostKeyChecking=no -o BatchMode=yes -l dbadmin 37.XX.XX.53) in my terminal, the ssh works perfectly...

    I finally checked the code :
    File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/network/adapters/pexpect_adapter.py", line 344, in _do_login

    In the comment of this file, I saw that my problem seems to be known (line 332) :
    " # Workaround for VER-2830:
    # Sometimes os.forkpty() raises OSError: "out of pty devices". Did not
    # find good explanation why it's happening. There are plenty
    # devices(/proc/sys/kernel/pty/max=4096) available on the system level
    # and not much of them are used(/proc/sys/kernel/pty/nr is not grater
    # than 4) the error happens and goes away for unclear reasons. Trying
    # to login 5 times in a row.
    "
    It seems that this is my exact problem but trying 5 fives or more does not work for me.

    Do you have an idea on how I could fix it ?

    Thanks
    Félicien

  • Jim_KnicelyJim_Knicely - Select Field - Administrator

    Wow, that "Workaround" was implemented way back on 09/18/2007!

    Do you have /dev/ptys correctly mounted?

    Here's what I have:

    [dbadmin@s18384357 ~]$ cat /etc/fstab | grep devpts
    none    /dev/pts        devpts  rw,gid=5,mode=620       0       0
    
    [dbadmin@s18384357 ~]$ mount | grep devpts
    none on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
    
  • Hi Jim ! Thanks for your answer, sorry I haven't check before today

    /dev/pty doesn't seem wrong, is it ? :

    dbadmin@ns321212:~$ cat /etc/fstab | grep devpts
    devpts          /dev/pts        devpts  defaults        0       0
    dbadmin@ns321212:~$ mount | grep devpts
    devpts on /dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000)
    

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file