the package 'pstack' and No JSON object could be decoded

Minh_Phan

I got this message when install vertica script with a 2-node installation: Info: the package 'pstack' is useful during troubleshooting. Vertica recommends this package is installed. Checking/fixing OS parameters..... Error parsing validation output: No JSON object could be decoded Validation output: -bash: /opt/vertica/bin/validators.sh: No such file or directory Installation completed with warnings. Installation completed with errors. Can you help me out. Thanks

[Deleted User]

Hi Minh, What version of Vertica are you installing? And what Linux distribution are you using? The 'pstack' bit is likely not relevant. (Though you're welcome to install it anyway, if you'd like.) The bit about the JSON object may be a known issue, but I'd have to know more about your environment. Adam

Minh_Phan

HI Adam I am installing the community version: vertica-6.1.2-0.x86_64.RHEL5.rpm into a two-node on a Red Hat linux system Before we installed enterprise version of vertica on a single node on , but later the Vertica team recommended us to install the community version with two-node, So we uninstalled the enterprise version from and reinstall it with community version. After install the RPM package on , We installed the run the script: /opt/vertica/sbin/install_vertica -s , -r vertica-6.1.2-0.x86_64.RHEL5.rpm I have feeling that the user "dbadmin" is already exist in by previous installation, but it is not yet created on so that the script: /opt/vertica/bin/validators.sh has no JSON output return. Do we need to either manually drop the dbadmin from or useradd dbadmin from to make it same as in 2 hosts, or it doesn't matter ? Here is the complete output from the install_vertica script: # /opt/vertica/sbin/install_vertica -s , -r vertica-6.1.2-0.x86_64.RHEL5.rpm Vertica Analytic Database 6.1.2-0 Installation Tool Starting installation tasks... Getting system information for cluster (this may take a while).... Enter password for root@ (2 attempts left): backing up admintools.conf on Default shell on nodes: /bin/bash /bin/bash Installing rpm on 1 hosts.... installing node.... Info: the package 'pstack' is useful during troubleshooting. Vertica recommends this package is installed. Checking/fixing OS parameters..... Error parsing validation output: No JSON object could be decoded Validation output: -bash: /opt/vertica/bin/validators.sh: No such file or directory Installation completed with warnings. Installation completed with errors. Installation failed.

Minh_Phan

Sorry, the previous post stripped out some of my words, so I correct it below ---------------------------------------------------------------------------------------- I am installing the community version: vertica-6.1.2-0.x86_64.RHEL5.rpm into a two-node on a Red Hat linux system Before we installed enterprise version of vertica on a single node on "host1", but later the Vertica team recommended us to install the community version with two-node, So we uninstalled the enterprise version from "host1" and reinstall it with community version. After installed the RPM package on "host1", We ran the install_vertica script: /opt/vertica/sbin/install_vertica -s "host1","host2" -r vertica-6.1.2-0.x86_64.RHEL5.rpm I have feeling that the user "dbadmin" is already exist in "host1" by previous installation, but it is not yet created on "host2" so that the script: /opt/vertica/bin/validators.sh has no JSON output return. Do we need to either manually drop the dbadmin from "host1" or useradd dbadmin from "host1" to make it same as in 2 hosts, or it doesn't matter ? Here is the complete output from the install_vertica script: # /opt/vertica/sbin/install_vertica -s "host1","host2" -r vertica-6.1.2-0.x86_64.RHEL5.rpm Vertica Analytic Database 6.1.2-0 Installation Tool Starting installation tasks... Getting system information for cluster (this may take a while).... Enter password for root@"host2" (2 attempts left): backing up admintools.conf on "host1" Default shell on nodes: "host1" /bin/bash "host2" /bin/bash Installing rpm on 1 hosts.... installing node...."host2" Info: the package 'pstack' is useful during troubleshooting. Vertica recommends this package is installed. Checking/fixing OS parameters..... Error parsing validation output: No JSON object could be decoded Validation output: -bash: /opt/vertica/bin/validators.sh: No such file or directory Installation completed with warnings. Installation completed with errors. Installation failed.

[Deleted User]

Hi Minh, Hm... Unfortunately, doesn't sound like the known issue that I was thinking of. That error looks to me like it's unable to find the "/opt/vertica/bin/validators.sh" script. Is it present on both your systems? Are you able to run it manually? (If run with no arguments, it should print out a bunch of JSON.) If you're already working with other folks at Vertica, I would imagine that they can help you more directly with this as well. Though we're glad to try to help here too. Adam

Minh_Phan

No, the /opt/vertica/bin/validators.sh only exist in "Host1" (main system) because I only download and install the HP Vertica RPM package on this host only. Hmm, do we have to install the RPM on both system?

[Deleted User]

Hm, you shouldn't generally have to install on both nodes... install_vertica should install for you on the second node. But if you try to install manually on the second node, is there an error of some sort? Maybe some old package still around that needs cleaning up? I wonder if the automatic install there is failing silently for some reason. Then the verify script would not be there as expected.

Minh_Phan

Adam, Just confirmed that on second node, I verified that is no "dbadmin" user added, and there is no /opt/vertica directory in the second node, too Do you think I should try to manually install the RPM package on second node to see what happen? Is that could cause any conflict when we run the install_script ? One thing, when I uninstall the old enterprise version on the first node, I already remove the /opt/vertica, is there anywhere the old package can be left around ? Thanks Minh

[Deleted User]

Hi Minh, Sure, I'd suggest that you try to manually install the RPM. Not permanently -- just install it briefly, to verify that the RPM installation process works; then uninstall it again. (You could try leaving it installed too. The installer should install the package on its own; but something's not working properly with the installation, and the installer should at least error out with an informative message if it can't deal with the package being installed in your situation.) I'm not sure exactly what's wrong here. But the errors here indicate that the installer thinks that it has installed the RPM on the second node (which would mean that the verify script must be there), and also that the verify script is not there. So in general, I'd suggest that you try a few things to figure out what's going on. In the worst case, just completely uninstall the RPM from both systems and (after uninstalling it!) remove /opt/vertica/; this should clean the systems and let you start over from scratch. (Or at least, start over from where you are now.) Adam

Minh_Phan

YES !!! After install the RPM package from second node (in both nodes, actually) Then from the first node, I ran the install_script, and it go through .. I am very happy so far, a lot of output to the screen, however the install still failed at last when trying to connect from host to host with passwordless SSH access to other hosts: Here is the error output: Test of host "host1" (FAILED) ======================================== Passwordless SSH access to other hosts (FAILED) --------------------------------------------------- Cannot connect from "host1" to host "host1" without password Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). Vertica requires that SSH be configured for passwordless authentication. Verify that /etc/ssh/sshd_config is configured properly on "host1". See the Vertica Installation Guide for more information. Test of host "host2" (FAILED) ======================================== Passwordless SSH access to other hosts (FAILED) --------------------------------------------------- Cannot connect from "host2" to host "host1" without password Vertica requires that SSH be configured for passwordless authentication. Verify that /etc/ssh/sshd_config is configured properly on "host1". See the Vertica Installation Guide for more information. Enough RAM per CPUs (FAILED) -------------------------------- Vertica requires at least 1 GB per CPU (you have 0.73 GB/CPU) See the Vertica Installation Guide for more information. Consistency Test (ok) ========================= Info: The $TZ environment variable is not set on "host1" Info: The $TZ environment variable is not set on "host2" Network Test (FAILED) ========================= Network communication (FAILED) ---------------------------------- Echo server failed on host "host1":48073 : ['', ''] Failed to send/recieve messages from all hosts, Check your netverify log for details and make sure your firewall and SELinux are disabled for the cluster interfaces. Verification failed. Correct the above issues to proceed Installation completed with warnings. Installation completed with errors. Installation failed. I think I need to read the manual again, but if you can help me out, that will save me a lot of time. Thank you so much, Adam

Minh_Phan

Hi again, Adam I manually set the network configuration for the Passwordless SSH access and RAM per CPUs However the setting RAM is still getting an error: The last output is as below: # /opt/vertica/sbin/install_vertica -s "host1","host2" -r vertica-6.1.2-0.x86_64.RHEL5.rpm Vertica Analytic Database 6.1.2-0 Installation Tool Starting installation tasks... Getting system information for cluster (this may take a while).... Enter password for root@"host2" (2 attempts left): backing up admintools.conf on "host1" Default shell on nodes: "host1" /bin/bash "host2" /bin/bash Info: the package 'pstack' is useful during troubleshooting. Vertica recommends this package is installed. Checking/fixing OS parameters..... Setting vm.min_free_kbytes to 90112 ... Detected cpufreq module loaded on "host1" Detected cpufreq module loaded on "host2" CPU frequency scaling is enabled. This may adversely affect the performance of your database. Vertica recommends that cpu frequency scaling be turned off or set to 'performance' Creating/Checking Vertica DBA group Creating/Checking Vertica DBA user Installing/Repairing SSH keys for dbadmin Modifying existing Vertica DBA user Creating Vertica Data Directory... Testing N-way network test. (this may take a while) All hosts are available ... Verifying system requirements on cluster. IP configuration ... IP configuration ... Testing hosts (1 of 2).... Running Consistency Tests LANG and TZ environment variables ... Running Network Connectivity and Throughput Tests... Waiting for 1 of 2 sites... ... Test of host "host2" (ok) ==================================== Enough RAM per CPUs (FAILED) -------------------------------- Vertica requires at least 1 GB per CPU (you have 0.73 GB/CPU) See the Vertica Installation Guide for more information. Consistency Test (ok) ========================= Info: The $TZ environment variable is not set on "host1" Info: The $TZ environment variable is not set on "host2" Network Test (FAILED) ========================= Network communication (FAILED) ---------------------------------- Echo server failed on host "host1":48073 : ['', ''] Failed to send/recieve messages from all hosts, Check your netverify log for details and make sure your firewall and SELinux are disabled for the cluster interfaces. Verification failed. Correct the above issues to proceed Installation completed with warnings. Installation completed with errors. Do you have any idea can help me out. Thank you. Minh

[Deleted User]

Hi Minh, I don't think this is a problem with the script. Take another look -- the script is telling you that your hardware is not able to run Vertica. "RAM" isn't a setting; it's how much physical RAM your hardware has. Vertica requires a minimum of 2gb RAM for each CPU core in your system; 4gb RAM is recommended. (So a quad-core system without hyperthreading would need 8gb to 16gb RAM, for example.) Do you meet that requirement? There's also a failure with network communication. Most likely a firewall somewhere on your computers or your network is blocking one of Vertica's communications ports. If you're able to resolve the memory issue, you should take a look through the Installation Guide and make sure you've either turned off any relevant firewalls or configured them to allow traffic as specified in the guide. Adam

Minh_Phan

Hi again, Adam Can you please help, I am out of brain for this password less issue, Recall from last post, I resolved the memory/cpu issue But for this passwordless between each server, I can't get out of this. I try many times but always get the same error, I did try to manually setup the passwordless between each server, and I can connect each of them passwordless successfully. However when I run the install script, the install will try to backup my working setup and replace it with the none- working setup. Can you get me out of this: Here is the complete output log ############################################# # /opt/vertica/sbin/install_vertica -s host1,host2,host2 -r /home/mnphan/vertica-6.1.2-0.x86_64.RHEL5.rpm Vertica Analytic Database 6.1.2-0 Installation Tool Starting installation tasks... Getting system information for cluster (this may take a while).... Enter password for root@xxx.xxx.xx.138 (2 attempts left): Enter password for root@xxx.xxx.xx.104 (1 attempts left): backing up admintools.conf on xxx.xxx.xx.139 Default shell on nodes: xxx.xxx.xx.139 /bin/bash xxx.xxx.xx.138 /bin/bash xxx.xxx.xx.104 /bin/bash NTP service is not running on the hosts: ['xxx.xxx.xx.104'] Vertica recommends that you keep the system clock synchronized using NTP or some other time synchronization mechanism to keep all hosts synchronized. Time variances can cause (inconsistent) query results when using Date/Time Functions. For instructions, see: * http://kbase.redhat.com/faq/FAQ_43_755.shtm * http://kbase.redhat.com/faq/FAQ_43_2790.shtm Info: the package 'pstack' is useful during troubleshooting. Vertica recommends this package is installed. Checking/fixing OS parameters..... Setting vm.min_free_kbytes to 45056 ... Detected cpufreq module loaded on xxx.xxx.xx.139 Detected cpufreq module loaded on xxx.xxx.xx.138 Detected cpufreq module loaded on xxx.xxx.xx.104 CPU frequency scaling is enabled. This may adversely affect the performance of your database. Vertica recommends that cpu frequency scaling be turned off or set to 'performance' Creating/Checking Vertica DBA group Creating/Checking Vertica DBA user Password for dbadmin: Installing/Repairing SSH keys for dbadmin Modifying existing Vertica DBA user Creating Vertica Data Directory... Testing N-way network test. (this may take a while) All hosts are available ... Host availability ... IP configuration ... IP configuration ... IP configuration ... Testing hosts (2 of 3).... Running Consistency Tests LANG and TZ environment variables ... Running Network Connectivity and Throughput Tests... Waiting for 1 of 3 sites... ... Test of host xxx.xxx.xx.139 (FAILED) ======================================== Passwordless SSH access to other hosts (FAILED) --------------------------------------------------- Cannot connect from xxx.xxx.xx.139 to host xxx.xxx.xx.138 without password Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). Vertica requires that SSH be configured for passwordless authentication. Verify that /etc/ssh/sshd_config is configured properly on xxx.xxx.xx.138. See the Vertica Installation Guide for more information. Test of host xxx.xxx.xx.138 (FAILED) ======================================== Passwordless SSH access to other hosts (FAILED) --------------------------------------------------- Cannot connect from xxx.xxx.xx.138 to host xxx.xxx.xx.139 without password Vertica requires that SSH be configured for passwordless authentication. Verify that /etc/ssh/sshd_config is configured properly on xxx.xxx.xx.139. See the Vertica Installation Guide for more information. Test of host xxx.xxx.xx.104 (FAILED) ======================================== Passwordless SSH access to other hosts (FAILED) --------------------------------------------------- Cannot connect from xxx.xxx.xx.104 to host xxx.xxx.xx.138 without password Vertica requires that SSH be configured for passwordless authentication. Verify that /etc/ssh/sshd_config is configured properly on xxx.xxx.xx.138. See the Vertica Installation Guide for more information. Consistency Test (ok) ========================= Info: The $TZ environment variable is not set on xxx.xxx.xx.139 Info: The $TZ environment variable is not set on xxx.xxx.xx.138 Info: The $TZ environment variable is not set on xxx.xxx.xx.104 Network Test (FAILED) ========================= Network communication (FAILED) ---------------------------------- Echo server failed on host xxx.xxx.xx.139:48073 : ['', ''] Echo server failed on host xxx.xxx.xx.139:48073 : ['', ''] Echo server failed on host xxx.xxx.xx.138:48073 : ['', ''] Echo server failed on host xxx.xxx.xx.104:48073 : ['', ''] Failed to send/recieve messages from all hosts, Check your netverify log for details and make sure your firewall and SELinux are disabled for the cluster interfaces. Verification failed. Correct the above issues to proceed Installation completed with warnings. Installation completed with errors. Installation failed.

[Deleted User]

Hi Minh, Hm... I'm afraid I'm really not quite sure what's going on here... You clearly had it working at one point; you previously posted the output from an install that succeeded at logging in via SSH. So something must have changed since then. Any idea what? For passwordless SSH, don't try to set it up yourself; Vertica needs a very particular setup and will not use your setup. It might help to clean out all of your ssh config for this user and start from scratch. Or you could take the "non-working" version as set up by Vertica, and try to use it; use ssh's debugging tools ("ssh -v", for starters) to track down what's blocking the connection. Adam

Minh_Phan

Adam, thanks for your reply I am lucky to have my co-worker to help me out of this. - First just confirm that, I have never success to pass the logging via SSH, I always failed in this step. - Then, it's weird that after delete all dbadmin user accounts from all host, so I let Vertica create account & setup the ssh by itself, it is still failed I here the steps that I can replicate and make it works: - Manually setup the ssh as show in the vertica install document, after this step, in each host there are 4 files: authorized_keys2, id_rsa, id_rsa_pub, known_hosts - My coworker rename the authorized_keys2 to authorized_keys, I think this is the point - Run the sertica install script again. Now it work !!!!! I saw that, the install script in each host - Backup my .ssh folder to .ssh.vtbackup - Create new authorized_keys2 file with root access from its own server - Add a line with root access from each own server to authorized_keys So I believe the key here is before in the ssh folders I already have the file authorized_keys2, that make the script failed to setup the ssh. Hope you understand what I am done. Thanks so much Minh

[Deleted User]

Hi Minh, Glad you were able to figure this out! And interesting that this caused the failure. We'll keep it in mind. Adam

Amine

Hi Adam,

So I was having the same issue (regarding the validators.sh) trying to install vertica-6.1.3-0 on 3 nodes CentOS.
Then I manually installed vertica (the rpm) on the other 2 nodes, and this error disappeared, except I am running into a different issue that is related to the vspread.conf, and I can see this error:

Dec 26 21:36:50  [SSH.scpFrom] Command Failed: Failed to run scp on 192.168.3.6: scp -r  -oConnectTimeout=30 -o TCPKeepAlive=no -o ServerAliveInterval=15 -o ServerAliveCountMax=2 -o StrictHostKeyChecking=no root@X.X.X.X:/opt/vertica/config/vspread.conf /opt/vertica/config/vspread.conf

followed by (which makes sense, since the above copy failed):

Dec 26 21:36:50  root@X.X.X.X: ['1', ["chown: cannot access '/opt/vertica/config/vspread.conf': No such file or directory"]]

My question is this seems that it's copying the vspread file from node1 (this command is executed from node2 and node3) using root, but at no point in the installation I was asked for the root pwd for node1, so I can't see how is the installation expecting to copy using root@node1 without having the root pwd.
Will manually copying that file over to node2 and node2 (since according to the doc, that file must be the same in all nodes) fix this issue?

I would appreciate any help as I am stuck for over a week trying to setup a cluster.

Thank you,
Amine