Unable to start DB
LakshmiNarayan
Community Edition User ✭
Hi,
I have setup 2 node cluster on community version. installation had no issues, post starting DB,
DB file can be seen on nodes whereas unable to connect.
ERROR: Not all nodes came up, but not all down. Run scrutinize.
Unable to establish vsql script connection: Unable to connect to 'test3'
Unable to establish client-server connection: Unable to connect to 'test3'
Unable to create depot storage locations (if Eon) without a client-server connection.
Unable to rebalance shards (if Eon) without a client-server connection.
Unable to install default extension packages without a vsql script connection
Unable to sync database catalog (if Eon) without a client-server connection.
Database creation SQL tasks included one or more failures (see above).
Tried with different techniques found in forums. nothing helped. Any help would be appreciable.
Answers
Please check the following files for additional errors and possible root cause:
If starting the node using admintools, check /opt/vertica/log for adminTools.errors and adminTools.log
On the node where admintools was run to start the database, check the catalog folder for bootstrap-catalog.log and startup.log
Hi Bryan,
Thanks for response. Admintool.errors got no entries, whereas admintools.log got entries as,
raw_socket = self.establish_connection()\n File \"/opt/vertica/oss/python3/lib/python3.7/site-packages/vertica_python/vertica/connection.py\", line 480, in establish_connection\n raise errors.ConnectionError(err_msg)\nvertica_python.errors.ConnectionError: Failed to establish a connection to the primary server or any backup address.\n", "type": "", "message": "Failed to establish a connection to the primary server or any backup address."}, "runner_ack": true} error_message=None
on every DB start attempt,
admitools.conf misses node entries, need to update manually to proceed with other options.
DB creates but no node turns up.
"stage" : "Waiting for Cluster Invite",
"text" : "Ready to be invited",
Vertica uses peer-to-peer protocol driven by spread to manage cluster. Are there any firewall restrictions in place? The quickest fix is to allow all UDP and TCP traffic between all nodes. Otherwise, please review networking requirements at https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/InstallationGuide/BeforeYouInstall/EnsurePortsAreAvailable.htm
Hi Bryan,
yes i tried adding all ports. nothing helped.
Thinking as space issue, created fresh vms in azure with 512gb RAM. which are inside some network. there is no control between their communication. still getting same issue, when i try to start DB local DB starts and stops, then fails to start both nodes.
as ,
Error in /opt/vertica/config/admintools.conf?: No option 'v_testdb_node0001' in section: 'Nodes'
/opt/vertica/oss/python3/lib/python3.7/site-packages/vertica/tools/ATMain.py:524: DeprecationWarn ing: The 'warn' method is deprecated, use 'warning' instead
_logger.warn(msg)
/opt/vertica/oss/python3/lib/python3.7/site-packages/vertica/tools/ATMain.py:526: DeprecationWarn ing: The 'warn' method is deprecated, use 'warning' instead
% (str(userName), str(nonInteractive), str(makeUniquePorts)))
/opt/vertica/oss/python3/lib/python3.7/site-packages/vertica/tools/ATMain.py:527: DeprecationWarn ing: The 'warn' method is deprecated, use 'warning' instead
_logger.warn(traceback.format_exc())
Vertica installation was successful. Vertica is unable to communicate correctly what issue it is facing . NOt getting how to deal with it.
'v_testdb_node0001' node issue i can correct by adding it manually in admin.conf . If i try to start db again, i end up in same state.
When i try to run scrutinize, it says
Error: Failed to configure Scrutinize.
Error: Directory /tmp does not have 5 GB of space available to the current user
[dbadmin@vnode1 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 252G 0 252G 0% /dev
tmpfs 252G 0 252G 0% /dev/shm
tmpfs 252G 9.2M 252G 1% /run
tmpfs 252G 0 252G 0% /sys/fs/cgroup
/dev/mapper/rootvg-rootlv 2.0G 69M 2.0G 4% /
/dev/mapper/rootvg-usrlv 10G 1.2G 8.9G 12% /usr
/dev/sda2 494M 119M 376M 25% /boot
/dev/sda1 500M 9.7M 491M 2% /boot/efi
/dev/mapper/rootvg-homelv 1014M 455M 560M 45% /home
/dev/mapper/rootvg-varlv 8.0G 1.1G 7.0G 13% /var
/dev/mapper/rootvg-tmplv 2.0G 33M 2.0G 2% /tmp
/dev/mapper/rootvg-optlv 2.0G 1.7G 360M 83% /opt
/dev/sdb1 630G 2.1G 596G 1% /mnt/resource
tmpfs 51G 0 51G 0% /run/user/1000
May above data helps to suggest further?
Hi Bryan,
I could trace below issue in dbLog
SP_connect: unable to connect via UNIX socket to /opt/vertica/spread/tmp/4803 (pid=7165): Error: No such file or directory,
i tried vm restart and port overlap checks, though they have not given any resolutions.
Could you please help ?
Hi Bryan,
Thanks for response. Admintool.errors got no entries, whereas admintools.log got entries as,
raw_socket = self.establish_connection()\n File \"/opt/vertica/oss/python3/lib/python3.7/site-packages/vertica_python/vertica/connection.py\", line 480, in establish_connection\n raise errors.ConnectionError(err_msg)\nvertica_python.errors.ConnectionError: Failed to establish a connection to the primary server or any backup address.\n", "type": "", "message": "Failed to establish a connection to the primary server or any backup address."}, "runner_ack": true} error_message=None
on every DB start attempt,
admitools.conf misses node entries, need to update manually to proceed with other options.
DB creates but no node turns up.
startup.log contains
"stage" : "Waiting for Cluster Invite",
"text" : "Ready to be invited",
Hi Bryan,
Thanks for response. Admintool.errors got no entries, whereas admintools.log got entries as,
raw_socket = self.establish_connection()\n File \"/opt/vertica/oss/python3/lib/python3.7/site-packages/vertica_python/vertica/connection.py\", line 480, in establish_connection\n raise errors.ConnectionError(err_msg)\nvertica_python.errors.ConnectionError: Failed to establish a connection to the primary server or any backup address.\n", "type": "", "message": "Failed to establish a connection to the primary server or any backup address."}, "runner_ack": true} error_message=None
on every DB start attempt,
admitools.conf misses node entries, need to update manually to proceed with other options.
DB creates but no node turns up.
startup log says,
"stage" : "Waiting for Cluster Invite",
"text" : "Ready to be invited",
Hi Bryan,
Thanks for response. Admintool.errors got no entries, whereas admintools.log got entries as,
raw_socket = self.establish_connection()\n File \"/opt/vertica/oss/python3/lib/python3.7/site-packages/vertica_python/vertica/connection.py\", line 480, in establish_connection\n raise errors.ConnectionError(err_msg)\nvertica_python.errors.ConnectionError: Failed to establish a connection to the primary server or any backup address.\n", "type": "", "message": "Failed to establish a connection to the primary server or any backup address."}, "runner_ack": true} error_message=None
on every DB start attempt,
admitools.conf misses node entries, need to update manually to proceed with other options.
DB creates but no node turns up.
startup log says,
"stage" : "Waiting for Cluster Invite",
"text" : "Ready to be invited",
Hi, can you check permissions on folders in /opt/vertica? Folders must be owned by dbadmin user, verticadba group. Sometimes installer will miss some folders. Try "chown -R dbadmin:verticadba /opt/vertica/*" to correct permissions then restart using dbadmin. Also did install_vertica report any issues with system config? sometimes even INFO or WARN level problems will stop Vertica from running if certain combinations prevent allocating resources.
Permissions are correct. for cluster creation can we use private IPs?, As my VMs are in azure availability set, they can communicate with each other.
@LakshmiNarayan , you need to use private IPs. You should not expect inter-node communication over the public IPs to work without any performance issues and nodes dropping. Also, since this is Azure, did you set point-to-point mode when creating the cluster?
yes, PTP was not set.Tried again with mode is set to PTP,
AWS,MachineLearning,ParquetExport,VFunctions,approximate,flextable,kafka packages installed successfully.
logsearch,place,txtindex,voltagesecure fails as,
Failure: unable to install logsearch package: Failed due to vsql return code 2
need to check what else is missing
Secondly, Tried recording current setup got below output
failure_threshold = WARN
ipv4 = False
ipv6 = False
record_to = /home/vmuser/config.txt
vertica_dba_group = verticadba
vertica_dba_user = dbadmin
A little unusual. But you can reinstall packages after the fact. See this page: https://www.vertica.com/docs/10.0.x/HTML/Content/Authoring/InstallationGuide/Upgrade/PostUpgrade/UpgradingandReinstallingPackages.htm
Hi LenoyJ and Bryan, Thanks for your supports.
Main issue was w.r.t disk mounting and DB installation path was pointing to root path.
later data disc attached ,mounted and that path used for installation. issue resolved