Unable to Import Existing Vertica Cluster via Management Console

verbanverban Vertica Customer ✭
in General Discussion

This issue is same as i found in this questions:

https://forum.vertica.com/discussion/239465/cannot-import-existing-vertica-cluster-via-management-console

But both posts doen't help me.
I did all commands that @Jim_Knicely said in above link.
Also i get same error that got here:

https://forum.vertica.com/discussion/comment/240610/#Comment_240610

curl -H "VerticaApiKey: ***********" https://********:5444/licenses --insecure
{ "body": [{"status": true, "comment": "Vertica license is valid", "end": "Perpetual", "start": > "2011-11-22", "vendor": "Vertica Community Edition", "grace": "0", "size": "1TB CE Nodes 3"}],
"href": "/license",
"mime-type": "application/vertica.license.json-v2",
"links": []
}

curl -H "VerticaApiKey: ***********" https://********:5444/databases/**/users/null?user_id=dbadmin&passwd=***" --insecure

{ "http_code":405,
  "name":"Method Not Allowed",
  "url":"https://********:5444/databases/**/users/null?user_id=dbadmin&passwd=***",
  "data":"Method not allowed."
}

MC version is 9.2.1 and vertica version is 9.0.1

«1

Comments

  • verbanverban Vertica Customer ✭

    This is what a have in MC log file after schecking password on MC web gui (/opt/vconsole/log/mc/mconsole.log):

    [MCThread-1558333690403] AgentCommands INFO - agentapi=POST, https://...:5444/databases//users/null?user_id=dbadmin&passwd=******
    [MCThread-1558333690403] AgentCommands INFO - executeCommand>> statusCode: 500, Raw agentResponse: https://    ...:5444/databases/    /users/null?user_id=dbadmin&passwd=******
    [MCThread-1558333690403] AgentCommands WARN - Agent Response Not OK: 500
    [MCThread-1558333690403] AgentCommands WARN - Exception while parsing agent response: Expected a ',' or '}' at character 182 of { "http_code":500,
    "name":"Internal Server Error",
    "url":"https://...:5444/databases/***/users/null",
    "data":" ERROR: Error get user details: Column "user_name" does not exist"
    }
    [MCThread-1558333690403] ClusterAdminController ERROR - Password Validation failed with error, code: 500 , Error message: Error parsing the response from database server

    It seems this is a bug....

  • verbanverban Vertica Customer ✭

    any comment?

  • verbanverban Vertica Customer ✭

    Finally i upgrade my vertica to 9.2.0
    And also the version of my MC is 9.2.1
    But i cannot connect MC to my db again.
    same error in mconsole log....
    :'(:'(:'(
    Also i created a new user but not changed...

  • verbanverban Vertica Customer ✭

    any comment?
    @Jim_Knicely

  • Jim_KnicelyJim_Knicely - Select Field - Administrator
    edited June 2019

    Hmm. I have not seen this. Do you have any settings in MC you need to retain? If not, I'd completlty uninstall it, remove the /opt/vertica/mc directory and reinstall. Seems like a password issue. Don't include any weird characters in your MC password. Just a guess If you have weird characters, let me know, might be a bug we can fix.

  • verbanverban Vertica Customer ✭

    Thanks @Jim_Knicely,
    I installed it again but same error!
    the /opt/vertica/mc/ wasn't valid and doesn't exists:

    Installation:

    Configuration (with simple password: 123456)

    Same error again

    Log file /opt/vconsole/log/mc/mconsole.log
    [Attached]

    I hope it may help you to find the root cause (or maybe bug)

    Log.txt 41.5K
  • verbanverban Vertica Customer ✭

    any update?
    @Jim_Knicely

  • verbanverban Vertica Customer ✭

    any update?
    Is there any bug fix?

    @Jim_Knicely

  • Bryan_HBryan_H Vertica Employee Administrator

    Hi, I looked at the log you posted and I see a few issues:
    When installing, set group ID to "verticadba" to match dbadmin's primary group, otherwise chown/chmod will fail on invalid group.
    The dbadmin password used for MC is not sent to Vertica. For the second screen "Import Vertica", use the same password for dbadmin that is used in vsql.
    Have you upgraded Vertica recently? There seems to be some issue with agent communication. It might help to restart all services include agent, e.g. "sudo systemctl restart vertica_agent"

  • Jim_KnicelyJim_Knicely - Select Field - Administrator

    @Bryan_H is on to something. Can you check the file permissions?

  • verbanverban Vertica Customer ✭

    @Jim_Knicely which file you mean?

  • Jim_KnicelyJim_Knicely - Select Field - Administrator
    edited July 2019

    Check the file and directory permissions.

    Below I am the Linux user that runs MC (vconsole in my case):

    [vconsole@SE-Sandbox-26-node1 ~]$ whoami
vconsole

[vconsole@SE-Sandbox-26-node1 ~]$ ls -lrt /opt/vconsole/
total 20
drwxr-x---. 2 vconsole verticadba    19 Apr 22 17:42 sbin
drwxr-x---. 5 vconsole verticadba   104 Apr 22 17:42 bin
drwxr-x---. 2 vconsole verticadba     6 Apr 22 17:50 pluggable
drwxr-x---. 3 vconsole verticadba 12288 Jul  9 08:43 reports
drwxr-x---. 2 vconsole verticadba    30 Jul  9 08:43 shared
drwxr-x---. 4 vconsole verticadba    52 Jul  9 08:43 vendor
drwxrwx---. 3 vconsole verticadba    19 Jul  9 08:43 mcdb
drwxr-x---. 3 vconsole verticadba    58 Jul  9 08:43 lib
drwxr-x---. 2 vconsole verticadba     6 Jul  9 08:43 tools
drwxr-x---. 3 vconsole verticadba    16 Jul  9 08:43 log
drwxr-x---. 4 vconsole verticadba    31 Jul  9 08:43 temp
drwxr-x---. 5 vconsole verticadba  4096 Jul  9 08:51 config

[vconsole@SE-Sandbox-26-node1 ~]$ ls -lrt /opt/vconsole/config/
total 48
-rwxr-----. 1 vconsole verticadba   155 Apr 22 17:50 version.properties
-rwxr-----. 1 vconsole verticadba   999 Apr 22 17:50 openssl.cnf
-rwxr-----. 1 vconsole verticadba   731 Apr 22 17:50 keystore_README.txt
-rwxr-----. 1 vconsole verticadba  3514 Apr 22 17:50 keystore.key
-rwxr-----. 1 vconsole verticadba  2253 Apr 22 17:50 keystore.jks
-rwxr-----. 1 vconsole verticadba   894 Apr 22 17:50 console.properties.before.9.2.1
-rwxr-----. 1 vconsole verticadba   349 Apr 22 17:50 console.properties.before.7.0.0
-rwxr-----. 1 vconsole verticadba 10592 Apr 22 17:50 aws-instances.json
drwxr-x---. 3 vconsole verticadba    17 Jul  9 08:43 ansible
drwxr-x---. 2 vconsole verticadba    75 Jul  9 08:43 logrotate
drwx------. 4 vconsole verticadba    42 Jul  9 08:43 security
-rwxr-----. 1 vconsole verticadba  2864 Jul  9 08:51 console.properties.9.2.1.nonUpgradeBackup
-rw-rw-rw-. 1 vconsole verticadba  2979 Jul  9 09:21 console.properties
  • verbanverban Vertica Customer ✭
    edited July 2019

    Thanks Jim_Knicely,

    This is for old installation:

    But i removed it completely (rm /opt/vconsole/* -rf)
    and installed it again,
    with verticadba group ID and simple password for dbadmin,

    and also logfile:

    and file permission for new installation:

  • Jim_KnicelyJim_Knicely - Select Field - Administrator
    edited July 2019

    When I installed MC it asked for an MC user. I used vconsole. Looks like you used dbadmin as the user, wich is fine.

    But looks like there is something wrong with the group.

    Can you list the groups for your dbadmin user? Here's mine for my dbadmin and vconsole users:

    [dbadmin@SE-Sandbox-26-node1 ~]$ groups dbadmin
dbadmin : verticadba

[dbadmin@SE-Sandbox-26-node1 ~]$ groups vconsole
vconsole : verticadba
  • verbanverban Vertica Customer ✭

    @Bryan_H ,
    Thanks your help,
    you mentioned to dbadmin password, but i think this is not dealing with wrong password, because as you can see in log i received 401 error code in case of wrong password (intentionally) and also i received 500 Error code in case of correct password (same as password used in vsql)!!
    but i think it's a little complicated, it's talking about "Internal Server Error" in logfile!

    AgentCommands WARN - Exception while parsing agent response: Expected a ',' or '}' at character 178 of { "http_code":500,
    "name":"Internal Server Error",
    "url":"https://127.0.0.1:5444/databases/ce/users/null",
    "data":" ERROR: Error get user details: Column "user_name" does not exist"
    }
    ClusterAdminController ERROR - Password Validation failed with error, code: 500 , Error message: Error parsing the response from database server

  • Jim_KnicelyJim_Knicely - Select Field - Administrator

    Curious. Does your dbadmin password contain any weird characters?

  • verbanverban Vertica Customer ✭

    No, the password is one letter!! (bcoz it's test environment)
    let me change it to longer....

  • verbanverban Vertica Customer ✭

    I changed it to 6 letter (ASCII char), but same result :neutral:

  • verbanverban Vertica Customer ✭

    @Jim_Knicely Helppppp!

  • Bryan_HBryan_H Vertica Employee Administrator

    The request is going to port 5444, which is vertica_agent - can you check under /opt/vertica/logs and see if there is any output with more detail in agent*.log - there may be several files for the agent, some ending in log and some ending in err, and some possibly with a username.

  • Jim_KnicelyJim_Knicely - Select Field - Administrator
    edited July 2019

    @verban - Just noticed that you checked the "Use TLS Connection" on one import attempt, and you unchecked the "Use TLS Connection" on another import attempt.

    Which is it?

    Also, can you try to import again, and if you get the invalid PW error, check the LOGIN_FAILURES system table to see if MC is even hitting the DB?

    SELECT login_timestamp, client_hostname, client_os_user_name, reason FROM login_failures ORDER BY 1 DESC LIMIT 5;

    Next, did you set up any client authentication rules?

    SELECT * FROM client_auth;

  • verbanverban Vertica Customer ✭

    @Bryan_H
    This is the output for me on:
    ls -a /opt/vertica/log/agent* |grep -v gz|xargs tail -f

    first line for incorrect password and 2nd one for correct password!

  • verbanverban Vertica Customer ✭

    @Jim_Knicely

    About "Use TLS Connection" checking, i tried both and send output for both test (checked and unchecked)

    I checked LOGIN_FAILURES (but without new import, i continued last import by "Import Discovered" button), and i have new record on LOGIN_FAILURES when get incorrect pass (and no record for correct pass)

    And also no rules for client authentication :

  • Bryan_HBryan_H Vertica Employee Administrator

    Can you try using the adapter IP address rather than localhost/127.0.0.1 when importing? Sometimes localhost is configured differently (bind rules, for example) and real IP address works, as shown on mine:
    agent.log:INFO:wsgi:192.168.1.206 - - [16/Jul/2019:10:57:01 -0400] "POST /databases/docker/users/null HTTP/1.1" 200 618 "-" "Apache-HttpClient/4.5 (Java/1.7.0_191)"

  • verbanverban Vertica Customer ✭

    I don't know why it logs 127.0.0.1, because i opened web gui from interface ip address (192.168.x.x)
    But i guess this is due to vertica_db and vertica_console are installed on same machine

  • Bryan_HBryan_H Vertica Employee Administrator

    My MC and Vertica server are on same box - when you import DB, do you enter the adapter IP address first in the four boxes, then API key, then user/password?
    The next step is to increase logging on agent. This requires an edit to /opt/vertica/agent/simply_fast.py - change the line rest_handler.setLevel(logging.INFO) to rest_handler.setLevel(logging.DEBUG) then restart vertica_agent. This will greatly increase logging in all components but hopefully agent.log or vertica.log will then show the root cause.

  • verbanverban Vertica Customer ✭

    I found the root cause B)
    it seems vconsole query user for authentication like this:
    select * from users where user_name = 'dbadmin';'
    In this form:

    And i got this error on vertica.log:

    But it should be:
    select * from v_catalog.users where user_name = 'dbadmin';'

    Because** i have a 'users' table in my database **

    I changed my table name temporarily and successfully imported!

    Finally Fixed :)))

  • Bryan_HBryan_H Vertica Employee Administrator

    Thank you for your work in finding the root cause! I'll file a ticket to have this fixed in MC. Glad to hear it's finally working!

  • Jim_KnicelyJim_Knicely - Select Field - Administrator

    @verban - Wow! We should have thought of that as a possibility earier! Thanks for hanging in there and helping us find this bug.

  • Bryan_HBryan_H Vertica Employee Administrator

    @marry009 The link in your comment doesn't work. Could you post a screenshot oand also the output of the error from /opt/vconsole/log/mc/mconsole.log on the MC instance? Thanks!

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Format
SpoilerCodeQuote
Heading 2Heading 1
Emoji
:):D:(;):/:o:s:p:'(:|B):#:*<3o:)>:)
Url
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file
Home • General DiscussionComment As ...