Upgrading OS in 3 Node Cluster

avi120

Hi Experts,

I plan to upgrade my Vertica server's OS and would like to better understand the "Operating System" limitation * (listed below) when replacing one host in a cluster.

I need to upgrade my Vertica server's operating system from CentOS 6.5 to CentOS 7.3. I planned to take out one of the 3 nodes and do a kickstart with the same Vertica version, but with new OS version. Then add it back to the cluster and wait for it to sync with the other 2 nodes. Once the sync is completed do the same for the second and third nodes. This way I upgrade the OS without having to do backup and restore.

1) Can a cluster work properly when for a short time some of the servers are CentOS 6.5 and some are 7.3?
2) When upgrading the OS one node at a time is it be much better to bring down my application server so no one will access the database at that time or it will only have a small performance affect?
3) Can backup taken on the same Vertica version but on CentOS 6.5 be restored on the same Vertica version on CentOS 7.3?
4) Is this the recommended way of upgrading the OS? I know there is an option for inline OS upgrade, but I was told by Linux experts that this is not recommended. Avoiding inline OS upgrade is also the recommendation of this forum?

The following taken from Vertica site *
You can replace the host with a new host that has the following same characteristics as the old host:
• Name
• IP address
• Operating system
• The OS administrator user
• Directory location

https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/AdministratorsGuide/ManageNodes/ReplacingANodeUsingTheSameNameAndIPAddress.htm?TocPath=Administrator's Guide|Managing the Database|Managing Nodes|Replacing Nodes|_____1

Vertica_Curtis

I would recommend you consult with support on this topic. I've anecdotally heard of clients successfully upgrading in this manner, but I'm sure there are specific scenarios which make this difficult.

In general, you should be OK to have mixed operating systems within a cluster, but we certainly don't recommend that long-term. It also matters which flavors they are, but if you're sticking with the same flavor, you should be OK. Again - in general.

You can have one node down in a 3 node cluster and still be operational. You may notice a slight performance degrade, and there are certainly other factors at play internally which could cause this to be worse, depending on the physical layout of your data, and some network considerations. But you shouldn't have to take an application outage to do this.

I don't believe a backup cares what Vertica version you are using. But again, consult with support. I'm not qualified to answer #4, as I'm not a Linux admin. Bottom line, you should reach out to support for guidance on this process. They have a lot more experience and use-cases to draw upon for this kind of situation.

Bryan_H

Hi, I've worked on a similar situation where a customer needed to upgrade from RHEL 6.x to 7.x in a larger cluster. Two comments:
You are correct that as long as Vertica version is the same, cluster should stay up.
Other customer found that by preserving Vertica files and directories (e.g., /opt/vertica, init scripts in /etc, along with catalog, data folders) that they were able to restart Vertica and sync a bit faster.

Vertica_Curtis

Yes, that is what I meant. I don't believe the Vertica backup cares too much about the OS version. Thanks.

avi120

Thanks for the reply.
You wrote "I don't believe a backup cares what Vertica version you are using" Did you mean "I don't believe a backup cares what CentOS version you are using"?