create DB failed due to node connectivity issues

felicien

Hi,
I installed vertica on 3 hosts and the installation worked well, but when I want to create a database, I get this error :

*** Creating database: test ***
Press RETURN to continue
create DB failed due to node connectivity issues
All nodes must be able to connect to each other via passwordless ssh
Could not reach node 37.XX.XX.53
Could not reach node 94.XX.XX.94
Press RETURN to continue

I am using the user dbadmin and the passwordless connection works perfectly fine between all dbamin account on each host

ssh 37.XX.XX.53
and
ssh 94.XX.XX.94
work without asking any password (when using dbadmin account)

Does someone know what can be the problem ?

I am using centos 7 and the only thing I changed in sshd.config is PermitRootLogin (set to without-password), but i'ml pretty sure that's not what caused that problem.

Maibe it is a permission issue ? :
on each host :
ls -la /home/dbamin
gives
drwx------ 3 dbadmin verticadba 4096 17 août 19:36 .
drwxr-xr-x 27 root root 4096 17 août 19:35 ..
-rw------- 1 dbadmin verticadba 91 18 août 17:51 .bash_history
-rw-r--r-- 1 dbadmin verticadba 18 7 déc. 2016 .bash_logout
-rw-r--r-- 1 dbadmin verticadba 193 7 déc. 2016 .bash_profile
-rw-r--r-- 1 dbadmin verticadba 290 17 août 19:35 .bashrc
drwx------ 2 dbadmin verticadba 4096 17 août 19:35 .ssh

and
ls -la .ssh
drwx------ 2 dbadmin verticadba 4096 17 août 19:35 .
drwx------ 3 dbadmin verticadba 4096 17 août 19:36 ..
-rw-r--r-- 1 dbadmin verticadba 381 17 août 19:35 authorized_keys2
-rw------- 1 dbadmin verticadba 1675 17 août 19:35 id_rsa
-rw------- 1 dbadmin verticadba 381 17 août 19:35 id_rsa.pub

Thanks in advance
Félicien

Jim_Knicely

Hi,

All nodes need to communicate with each other...

Make sure:

node1 can SSH to node2
node1 can SSH to node3

node2 can SSH to node1
node2 can SSH to node3

node3 can SSH to node1
node3 can SSH to node2

One of 'em might be missing the required SSH fingerprint?

felicien

Hi Jim,
Thanks for your answer but yes, all node can communicate with each other..
I also tried to stop the firewall and nothing changes.

Manually, the ssh works perfectly between dbadmin users in all hosts.

Jim_Knicely

Hmm. What about the firewall? Did you open the appropriate ports?

https://my.vertica.com/docs/8.1.x/HTML/index.htm#Authoring/InstallationGuide/BeforeYouInstall/EnsurePortsAreAvailable.htm

felicien

The ports seems available, (none is used by another app), and the firewall don't bllock them.

I just saw that in the logs :

2017-08-22 16:42:03.260 admintools/521026:0x66f9bb1b0700 [SSHConnection._login_if_needed] : login error: Error! pty.fork() failed: out of pty devices
2017-08-22 16:42:03.260 admintools/521026:0x66f9c4797740 [AdapterConnectionPool_3.connect2_using_stored_auth] Unable to SSH as dbadmin to 188.XX.XX.95: Error! pty.fork() failed: out of pty devices
2017-08-22 16:42:03.261 admintools/521026:0x66f9ba92a700 [SSHConnection._do_login] : pexpect.spawn failed too many times: Error! pty.fork() failed: out of pty devices
Traceback (most recent call last):
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/network/adapters/pexpect_adapter.py", line 251, in _do_login
connectStr
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/utils/pexpect.py", line 429, in init
self._spawn (command, args)
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/utils/pexpect.py", line 537, in _spawn
raise ExceptionPexpect('Error! pty.fork() failed: ' + str(e))
ExceptionPexpect: Error! pty.fork() failed: out of pty devices

I don't understand why I get this error because my number of PTy is fixed at 4096 (/proc/sys/kernel/pty/max) and the current number of used pty (/proc/sys/kernel/pty/nr) is less than 3 on all hosts.

Maibe is this a hint ?

Thanks

Jim_Knicely

Make sure that the "User Process Limit" is set correctly on each node. 4096 is probably not high enough. See:

https://my.vertica.com/docs/8.1.x/HTML/index.htm#Authoring/InstallationGuide/BeforeYouInstall/UserProcessLimit.htm

felicien

Hi,
I'm trying again to install vertica but face the exact same problem :
I downloaded vertica9 and also tried with verticaMC. I also verified that the passwordless ssh works between all hosts.
I'm still getting the same error (could not reach node) and in the logs I can see :

: login start: ssh -oConnectTimeout=30 -o TCPKeepAlive=no -o ServerAliveInterval=15 -o ServerAliveCountMax=2 -o StrictHostKeyChecking=no -o BatchMode=yes -l dbadmin 37.XX.XX.53
: pexpect.spawn failed: Error! pty.fork() failed: out of pty devices, trying again.
: pexpect.spawn failed too many times: Error! pty.fork() failed: out of pty devices

But when I copy the exact same command (ssh -oConnectTimeout=30 -o TCPKeepAlive=no -o ServerAliveInterval=15 -o ServerAliveCountMax=2 -o StrictHostKeyChecking=no -o BatchMode=yes -l dbadmin 37.XX.XX.53) in my terminal, the ssh works perfectly...

I finally checked the code :
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/network/adapters/pexpect_adapter.py", line 344, in _do_login

In the comment of this file, I saw that my problem seems to be known (line 332) :
" # Workaround for VER-2830:
# Sometimes os.forkpty() raises OSError: "out of pty devices". Did not
# find good explanation why it's happening. There are plenty
# devices(/proc/sys/kernel/pty/max=4096) available on the system level
# and not much of them are used(/proc/sys/kernel/pty/nr is not grater
# than 4) the error happens and goes away for unclear reasons. Trying
# to login 5 times in a row.
"
It seems that this is my exact problem but trying 5 fives or more does not work for me.

Do you have an idea on how I could fix it ?

Thanks
Félicien

Jim_Knicely

Wow, that "Workaround" was implemented way back on 09/18/2007!

Do you have /dev/ptys correctly mounted?

Here's what I have:

[dbadmin@s18384357 ~]$ cat /etc/fstab | grep devpts
none    /dev/pts        devpts  rw,gid=5,mode=620       0       0

[dbadmin@s18384357 ~]$ mount | grep devpts
none on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)

felicien

Hi Jim ! Thanks for your answer, sorry I haven't check before today

/dev/pty doesn't seem wrong, is it ? :

dbadmin@ns321212:~$ cat /etc/fstab | grep devpts
devpts          /dev/pts        devpts  defaults        0       0
dbadmin@ns321212:~$ mount | grep devpts
devpts on /dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000)