After a cluster import Managment Console discovers my database but not monitoring it

AsheAshe Vertica Customer
edited January 2020 in General Discussion

hello,
i have searched this issue all over the vertica forums and it looks like no one is experiencing this issue as i have.i have vertica 9.3 installed on a 3 node Red hat linux 7.X cluster. Mc 9.3 is also installed on one of the nodes and during import of the existing cluster it imports it with no issues and discovers the underlying database as well. But when i import the discovered db it imports it with another cluster with 0 nodes in it and database shows as null. i have tried a bunch of things and still no hope. please help if u have any ideas.
error message:
Database xxxxx seems to be running an older version of vertica software ....upgrade latest version...
Thanks

Tagged:

Answers

  • LenoyJLenoyJ - Select Field - Employee

    Try looking at /opt/vertica/config/admintools.conf and see if there's anything extraneous there.

  • AsheAshe Vertica Customer

    Thanks again Lenoy, admintools.conf seems to have the right settings.
    To put it out there , the database was created using 8.0x version of vertica but its been upgraded to 9.2 .
    Is there a database compatibility setting after the upgrade or would vertica upgrades the database automatically?
    Thanks

  • Jim_KnicelyJim_Knicely - Select Field - Administrator

    @Ashe - In the first post you said that you are using Vertica 9.3, but in the last post you said you upgraded to 9.2. Which versionis correct?

  • SruthiASruthiA Administrator

    @Ashe Did you try re importing the cluster into MC after upgrade has been done? If not please try it.

  • AsheAshe Vertica Customer
    @SruthiA yes, many times I did a clean install and of Mc and re-imported it many times. Nothing has changed.
  • AsheAshe Vertica Customer
    @Jim_Knicely . It’s because I have tried all Mc versions 9.0x plus . The only thing that’s not changed is vertica server . It’s still 9.2.
  • LenoyJLenoyJ - Select Field - Employee
    edited January 2020

    @Ashe,

    • What does the MC log /opt/vconsole/log/mc/mconsole.log say when you import the database?
    • Is there more than one database on the cluster?

    • What do these output? Anything unusual in terms of databases, hosts, version number?

      • curl -H "VerticaApiKey: <APIKEY>" https://<NODE_IP>:5444/databases
      • curl -H "VerticaApiKey: <APIKEY>" https://<NODE_IP>:5444/databases/<DBNAME>
      • curl -H "VerticaApiKey: <APIKEY>" https://<NODE_IP>:5444/hosts

    • See here for how get an API key.

  • SruthiASruthiA Administrator

    Is the management console agent running on all the nodes?

  • SruthiASruthiA Administrator

    Along with that please check if any network interface is down on the machine where MC is installed. if so, please enable all network interfaces even though MC doesn't use it.

  • AsheAshe Vertica Customer
    edited February 2020

    @LenoyJ
    here is the portion of the log as of today and and the result of curl as well.
    21 Jan 2020 10:16:55,178 [generalScheduler_Worker-7] AgentHostCommands WARN - hostname is null for ip "xx.xx.xx.x"
    21 Jan 2020 10:16:55,178 [generalScheduler_Worker-7] AgentHostCommands WARN - hostname is null for ip "xx.xx.xx.x"
    21 Jan 2020 10:16:55,178 [generalScheduler_Worker-7] AgentHostCommands WARN - hostname is null for ip "xx.xx.xx.x"
    21 Jan 2020 10:16:55,187 [generalScheduler_Worker-7] AgentCommands INFO - agentapi=GET, https: //xx.xx.xx.x:5444/databases
    21 Jan 2020 10:16:55,253 [generalScheduler_Worker-7] AgentCommands INFO - executeCommand>> statusCode: 200, Raw agentResponse: https: //xx.xx.xx.x:5444/databases
    21 Jan 2020 10:16:55,263 [generalScheduler_Worker-7] AgentCommands INFO - agentapi=GET, https: //xx.xx.xx.x:5444/hosts?only=hostname,host_id,nics.name,nics.ipaddr,total_memory,cpu_info.number_of_cpus,cpu_info.cpu_type,vertica
    21 Jan 2020 10:16:55,915 [generalScheduler_Worker-7] AgentCommands INFO - executeCommand>> statusCode: 200, Raw agentResponse: https ://xx.xx.xx.x:5444/hosts?only=hostname,host_id,nics.name,nics.ipaddr,total_memory,cpu_info.number_of_cpus,cpu_info.cpu_type,vertica
    21 Jan 2020 10:16:55,915 [generalScheduler_Worker-7] AgentHostCommands WARN - hostname is null for ip "xx.xx.xx.x"
    21 Jan 2020 10:16:55,915 [generalScheduler_Worker-7] AgentHostCommands WARN - hostname is null for ip "xx.xx.xx.x"
    21 Jan 2020 10:16:55,915 [generalScheduler_Worker-7] AgentHostCommands WARN - hostname is null for ip "xx.xx.xx.x"
    21 Jan 2020 10:16:55,919 [generalScheduler_Worker-7] AgentCommands INFO - agentapi=GET, https: //1xx.xx.xx.x:5444/databases/My_DB_Name/hosts
    21 Jan 2020 10:16:55,956 [generalScheduler_Worker-7] AgentCommands INFO - executeCommand>> statusCode: 200, Raw agentResponse: https: //xx.xx.xx.x:5444/databases/My_DB_Name/hosts
    21 Jan 2020 10:16:55,963 [generalScheduler_Worker-7] VuiUtils INFO - Target and hosts match NOT found, return NULL matchHost
    21 Jan 2020 10:16:55,966 [generalScheduler_Worker-7] VuiUtils INFO - Target and hosts match NOT found, return NULL matchHost
    21 Jan 2020 10:16:55,969 [generalScheduler_Worker-7] VuiUtils INFO - Target and hosts match NOT found, return NULL matchHost
    21 Jan 2020 10:16:55,970 [generalScheduler_Worker-7] RepositoryAccessServiceImpl INFO - There are no currently running databases in the cluster
    21 Jan 2020 10:16:55,977 [generalScheduler_Worker-7] VerticaAgentPingingJob INFO - VerticaAgentPingingJob took 2501 milliseconds.
    21 Jan 2020 10:17:53,582 [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] ThresholdJob WARN - Warnning when checking Resource Pool Available. Message: null

    --> when i run curl -H "VerticaApiKey: " https: //:5444/databases on one of the hosts

    curl performs SSL certificate verification by default, using a "bundle"
    of Certificate Authority (CA) public keys (CA certs). If the default
    bundle file isn't adequate, you can specify an alternate file
    using the --cacert option.
    If this HTTPS server uses a certificate signed by a CA represented in
    the bundle, the certificate verification probably failed due to a
    problem with the certificate (it might be expired, or the name might
    not match the domain name in the URL).
    If you'd like to turn off curl's verification of the certificate, use
    the -k (or --insecure) option.

    Thanks

  • AsheAshe Vertica Customer

    @SruthiA
    Yes, its running on all nodes. also all network interfaces are enabled.
    Thank you.

  • AsheAshe Vertica Customer

    @LenoyJ
    when i run the same curl on the MC node ( not sure if it should be run there) i get this diffrent error msg
    curl: (7) Failed connect to xx.xx.xx.x:5444; Connection refused

  • LenoyJLenoyJ - Select Field - Employee

    @Ashe , the curl commands are to be run on the Vertica nodes. Try running the commands again with the -k option or provide a valid key. Probably a dumb question, but is the database running when you're importing it?

  • AsheAshe Vertica Customer
    edited February 2020

    @ Lenoy, yes the Database was Running. Thank you all for the Help but Vertica support seem to grab the root cause of the issue and had me modify /opt/vertica/bin/validators.sh and added a line of code as below and that does it. I am not a network engineer but the issue seemed to be a network/communication issue.

    if [ "$NIC_CONFIG" == "" ]; then
        NIC_CONFIG="\"speed\":\"unknown\""
     fi;

    ** if [[ "$NIC_CONFIG" != "speed" ]]; then
    NIC_CONFIG="\"speed\":\"unknown\""
    fi **

         if [ "$IP_INFO" == "" ]; then
           IP_INFO="ipaddr: broadcast: netmask:"
     fi;

    the middle part was the code i had to add and that fixed it. hope this will help others in the future.
    Thanks

  • Jim_KnicelyJim_Knicely - Select Field - Administrator

    @Ashe - Thanks for sharing!

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Format
SpoilerCodeQuote
Heading 2Heading 1
Emoji
:):D:(;):/:o:s:p:'(:|B):#:*<3o:)>:)
Url
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file
Home • General DiscussionComment As ...