Unable to start DB

LakshmiNarayanLakshmiNarayan Community Edition User

Hi,
I have setup 2 node cluster on community version. installation had no issues, post starting DB,

DB file can be seen on nodes whereas unable to connect.

ERROR: Not all nodes came up, but not all down. Run scrutinize.
Unable to establish vsql script connection: Unable to connect to 'test3'
Unable to establish client-server connection: Unable to connect to 'test3'
Unable to create depot storage locations (if Eon) without a client-server connection.
Unable to rebalance shards (if Eon) without a client-server connection.
Unable to install default extension packages without a vsql script connection
Unable to sync database catalog (if Eon) without a client-server connection.
Database creation SQL tasks included one or more failures (see above).

Tried with different techniques found in forums. nothing helped. Any help would be appreciable.

Answers

  • Bryan_HBryan_H Vertica Employee Administrator

    Please check the following files for additional errors and possible root cause:
    If starting the node using admintools, check /opt/vertica/log for adminTools.errors and adminTools.log
    On the node where admintools was run to start the database, check the catalog folder for bootstrap-catalog.log and startup.log

  • LakshmiNarayanLakshmiNarayan Community Edition User

    Hi Bryan,
    Thanks for response. Admintool.errors got no entries, whereas admintools.log got entries as,
    raw_socket = self.establish_connection()\n File \"/opt/vertica/oss/python3/lib/python3.7/site-packages/vertica_python/vertica/connection.py\", line 480, in establish_connection\n raise errors.ConnectionError(err_msg)\nvertica_python.errors.ConnectionError: Failed to establish a connection to the primary server or any backup address.\n", "type": "", "message": "Failed to establish a connection to the primary server or any backup address."}, "runner_ack": true} error_message=None

    on every DB start attempt,
    admitools.conf misses node entries, need to update manually to proceed with other options.
    DB creates but no node turns up.

  • LakshmiNarayanLakshmiNarayan Community Edition User
    • startup log says,
      "stage" : "Waiting for Cluster Invite",
      "text" : "Ready to be invited",
  • Bryan_HBryan_H Vertica Employee Administrator

    Vertica uses peer-to-peer protocol driven by spread to manage cluster. Are there any firewall restrictions in place? The quickest fix is to allow all UDP and TCP traffic between all nodes. Otherwise, please review networking requirements at https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/InstallationGuide/BeforeYouInstall/EnsurePortsAreAvailable.htm

  • LakshmiNarayanLakshmiNarayan Community Edition User

    Hi Bryan,
    yes i tried adding all ports. nothing helped.
    Thinking as space issue, created fresh vms in azure with 512gb RAM. which are inside some network. there is no control between their communication. still getting same issue, when i try to start DB local DB starts and stops, then fails to start both nodes.

    as ,

    Error in /opt/vertica/config/admintools.conf?: No option 'v_testdb_node0001' in section: 'Nodes'
    /opt/vertica/oss/python3/lib/python3.7/site-packages/vertica/tools/ATMain.py:524: DeprecationWarn ing: The 'warn' method is deprecated, use 'warning' instead
    _logger.warn(msg)
    /opt/vertica/oss/python3/lib/python3.7/site-packages/vertica/tools/ATMain.py:526: DeprecationWarn ing: The 'warn' method is deprecated, use 'warning' instead
    % (str(userName), str(nonInteractive), str(makeUniquePorts)))
    /opt/vertica/oss/python3/lib/python3.7/site-packages/vertica/tools/ATMain.py:527: DeprecationWarn ing: The 'warn' method is deprecated, use 'warning' instead

    _logger.warn(traceback.format_exc())

    Vertica installation was successful. Vertica is unable to communicate correctly what issue it is facing . NOt getting how to deal with it.

  • LakshmiNarayanLakshmiNarayan Community Edition User
    edited May 2020

    'v_testdb_node0001' node issue i can correct by adding it manually in admin.conf . If i try to start db again, i end up in same state.

    When i try to run scrutinize, it says
    Error: Failed to configure Scrutinize.
    Error: Directory /tmp does not have 5 GB of space available to the current user

    [dbadmin@vnode1 ~]$ df -h
    Filesystem Size Used Avail Use% Mounted on
    devtmpfs 252G 0 252G 0% /dev
    tmpfs 252G 0 252G 0% /dev/shm
    tmpfs 252G 9.2M 252G 1% /run
    tmpfs 252G 0 252G 0% /sys/fs/cgroup
    /dev/mapper/rootvg-rootlv 2.0G 69M 2.0G 4% /
    /dev/mapper/rootvg-usrlv 10G 1.2G 8.9G 12% /usr
    /dev/sda2 494M 119M 376M 25% /boot
    /dev/sda1 500M 9.7M 491M 2% /boot/efi
    /dev/mapper/rootvg-homelv 1014M 455M 560M 45% /home
    /dev/mapper/rootvg-varlv 8.0G 1.1G 7.0G 13% /var
    /dev/mapper/rootvg-tmplv 2.0G 33M 2.0G 2% /tmp
    /dev/mapper/rootvg-optlv 2.0G 1.7G 360M 83% /opt
    /dev/sdb1 630G 2.1G 596G 1% /mnt/resource
    tmpfs 51G 0 51G 0% /run/user/1000

    May above data helps to suggest further?

  • LakshmiNarayanLakshmiNarayan Community Edition User
    edited May 2020

    Hi Bryan,
    I could trace below issue in dbLog
    SP_connect: unable to connect via UNIX socket to /opt/vertica/spread/tmp/4803 (pid=7165): Error: No such file or directory,

    i tried vm restart and port overlap checks, though they have not given any resolutions.
    Could you please help ?

  • LakshmiNarayanLakshmiNarayan Community Edition User

    Hi Bryan,
    Thanks for response. Admintool.errors got no entries, whereas admintools.log got entries as,
    raw_socket = self.establish_connection()\n File \"/opt/vertica/oss/python3/lib/python3.7/site-packages/vertica_python/vertica/connection.py\", line 480, in establish_connection\n raise errors.ConnectionError(err_msg)\nvertica_python.errors.ConnectionError: Failed to establish a connection to the primary server or any backup address.\n", "type": "", "message": "Failed to establish a connection to the primary server or any backup address."}, "runner_ack": true} error_message=None

    on every DB start attempt,
    admitools.conf misses node entries, need to update manually to proceed with other options.
    DB creates but no node turns up.

    startup.log contains
    "stage" : "Waiting for Cluster Invite",
    "text" : "Ready to be invited",

  • LakshmiNarayanLakshmiNarayan Community Edition User

    Hi Bryan,
    Thanks for response. Admintool.errors got no entries, whereas admintools.log got entries as,
    raw_socket = self.establish_connection()\n File \"/opt/vertica/oss/python3/lib/python3.7/site-packages/vertica_python/vertica/connection.py\", line 480, in establish_connection\n raise errors.ConnectionError(err_msg)\nvertica_python.errors.ConnectionError: Failed to establish a connection to the primary server or any backup address.\n", "type": "", "message": "Failed to establish a connection to the primary server or any backup address."}, "runner_ack": true} error_message=None

    on every DB start attempt,
    admitools.conf misses node entries, need to update manually to proceed with other options.
    DB creates but no node turns up.
    startup log says,
    "stage" : "Waiting for Cluster Invite",
    "text" : "Ready to be invited",

  • LakshmiNarayanLakshmiNarayan Community Edition User

    Hi Bryan,
    Thanks for response. Admintool.errors got no entries, whereas admintools.log got entries as,
    raw_socket = self.establish_connection()\n File \"/opt/vertica/oss/python3/lib/python3.7/site-packages/vertica_python/vertica/connection.py\", line 480, in establish_connection\n raise errors.ConnectionError(err_msg)\nvertica_python.errors.ConnectionError: Failed to establish a connection to the primary server or any backup address.\n", "type": "", "message": "Failed to establish a connection to the primary server or any backup address."}, "runner_ack": true} error_message=None

    on every DB start attempt,
    admitools.conf misses node entries, need to update manually to proceed with other options.
    DB creates but no node turns up.
    startup log says,
    "stage" : "Waiting for Cluster Invite",
    "text" : "Ready to be invited",

  • Bryan_HBryan_H Vertica Employee Administrator

    Hi, can you check permissions on folders in /opt/vertica? Folders must be owned by dbadmin user, verticadba group. Sometimes installer will miss some folders. Try "chown -R dbadmin:verticadba /opt/vertica/*" to correct permissions then restart using dbadmin. Also did install_vertica report any issues with system config? sometimes even INFO or WARN level problems will stop Vertica from running if certain combinations prevent allocating resources.

  • LakshmiNarayanLakshmiNarayan Community Edition User

    Permissions are correct. for cluster creation can we use private IPs?, As my VMs are in azure availability set, they can communicate with each other.

  • LenoyJLenoyJ - Select Field - Employee

    @LakshmiNarayan , you need to use private IPs. You should not expect inter-node communication over the public IPs to work without any performance issues and nodes dropping. Also, since this is Azure, did you set point-to-point mode when creating the cluster?

  • LakshmiNarayanLakshmiNarayan Community Edition User
    edited June 2020

    yes, PTP was not set.Tried again with mode is set to PTP,
    AWS,MachineLearning,ParquetExport,VFunctions,approximate,flextable,kafka packages installed successfully.
    logsearch,place,txtindex,voltagesecure fails as,
    Failure: unable to install logsearch package: Failed due to vsql return code 2
    need to check what else is missing :neutral:

    Secondly, Tried recording current setup got below output
    failure_threshold = WARN
    ipv4 = False
    ipv6 = False
    record_to = /home/vmuser/config.txt
    vertica_dba_group = verticadba
    vertica_dba_user = dbadmin

  • LenoyJLenoyJ - Select Field - Employee
  • LakshmiNarayanLakshmiNarayan Community Edition User
    edited June 2020

    Hi LenoyJ and Bryan, Thanks for your supports.
    Main issue was w.r.t disk mounting and DB installation path was pointing to root path.
    later data disc attached ,mounted and that path used for installation. issue resolved

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file