MC import cluster process brings in TWO clusters and a database on a single 3 node cluster 9.2

Sudhakar_B

We have a three node cluster Vertica 9.2.0.0. The database and cluster was upgraded a month ago. The database is functioning properly ( seems so ). I removed all previous installs/versions of MC and did a fresh install of MC (9.2.0.0) which succeeded, however the MC is seeing TWO clusters and imports 2 clusters and 1 database.
MC imports TWO clusters.
1. Cluster 1 that has the three nodes and with proper IP addresses
2. Cluster 2, which has a database associated but no nodes and no database related information is available.
The nodes / IP of first cluster can be seen in the MC, however no details like CPU type storage etc. are available and has no database is associated with it.
Second cluster has no nodes (zero nodes), but has an associated database which is named correctly.
The clusters and database are not usable in MC.
As FYI,
/opt/vertica/config/admintools.conf had a wrong IP address for controlsubnet parameter.
Not sure how or when this plays a role, but I corrected it to default prior to installing MC.
However the Vertica DB SW has not been reinstalled/restarted nor the processes like agent.sh and simply_fast.py restarted.
Any help in resolving this is greatly appreciated. If you need additional information please let me know.
TIA

LenoyJ

After you modified admintools.conf, did you redistribute it to all the nodes?

admintools -t distribute_config_files

Sudhakar_B

Thanks Lenoy,
No I did not redistribute the the config files to all nodes. I modified /opt/vertica/config/admintools.conf in each of the three nodes independently. As I wasn't sure of the impact of this field. I'll try this suggestion.
Also, do I have restart the agent.sh and simply_fast.py process after redistribution?
Thanks again. I'll post the updates.

Sudhakar_B

Thanks Lenoy,
Per your suggestion, I distributed the admintools.conf and restarted the agent. Still no lock :-(
MC is still seeing two clusters.
Also Every 10 seconds I see the following messages in /opt/vconsole/log/mc.

27 Aug 2019 12:32:25,720 [Thread-4] WorkQueue INFO  - kickOffPollingTask>> ----Kicking off adhoc polling for this cluster: Cluster [id=10, name=1566922452678_cluster,  agentConnection=null, create_time=null, last_updated_time=2019-08-27 12:32:16.676 nodes=[]]-------
27 Aug 2019 12:32:26,916 [qtp270792089-7258 - /webui/admin/databases.json?1566923546776&showDbClusters=true] DbAdminController INFO  - getAllDatabases>> size: 1
27 Aug 2019 12:32:26,916 [qtp270792089-7263 - /webui/admin/databases.json?1566923546787&showDbClusters=true] DbAdminController INFO  - getAllDatabases>> size: 1
27 Aug 2019 12:32:26,918 [qtp270792089-7258 - /webui/admin/databases.json?1566923546776&showDbClusters=true] DbAdminController WARN  - Could not find AgentConnection, so attempting to get again
27 Aug 2019 12:32:26,918 [qtp270792089-7263 - /webui/admin/databases.json?1566923546787&showDbClusters=true] DbAdminController WARN  - Could not find AgentConnection, so attempting to get again
27 Aug 2019 12:32:26,919 [qtp270792089-7258 - /webui/admin/databases.json?1566923546776&showDbClusters=true] DbAdminController ERROR - Failed to create webhook after trying with all hosts of the cluster
27 Aug 2019 12:32:26,919 [qtp270792089-7263 - /webui/admin/databases.json?1566923546787&showDbClusters=true] DbAdminController ERROR - Failed to create webhook after trying with all hosts of the cluster

Should I stop and restart the DB? At this point I am kind of at a loss.
Any help / pointers are greatly appreciated.
Thx.

Bryan_H

After redistributing the config files, you should restart the database and agent to pick up the update. I don't think it polls for changes.

Sudhakar_B

Thanks Bryan for quick response.
I'll restart the DB and MC on the cluster. Keep you posted of the progress.
Thanks.

LenoyJ

Also - I don't think MC would revalidate an imported cluster. So you may want to get rid of the imported cluster and reimport it. Or Factory Reset it (bit extreme, but still works if needed).
Also worth going through the admintools.conf again to see if there are any hosts listed there that are not being used.

Sudhakar_B

Thanks Bryan, Lenoy,
Your help is much appreciated.
Just restarted the DB using admintools. Then restarted the MC as well (I did not reinstall MC). No Luck though
Lenoy, I do get rid and re-import the cluster and DB, every time.
Tomorrow, I'll reinstall MC and see if that helps.
Customer has initiated a support ticket. I'll be able to share more details there, as the cluster was recently upgraded from 7.1.x through 9.2.0.0. May be that is playing a role.
Thank much!