Upgrading Linux from RedHat/CentOS 6.x to 7.3 for Vertica 9.0 upgrade on live database
Upgrading Linux from RedHat/CentOS 6.x to 7.3 for Vertica 9.0 upgrade on live database
Recent release of Vertica 9.0 mark end of support for RedHat/CentOS 6.x.
With release of Vertica 9.0, all previous Vertica versions have been desupported.
Out of my production Vertica databases 3 are v8.x running on CentOS 6.2.
Vertica 9.0 is linked against version of glibc from 7.x, and will not be able to start on 6.x OS. Upgrade from 8.x to 9.0 on CentOS 6.x is not possible (as well as not supported).
And you do not want to run production in unsupported configuration.
I am facing OS upgrade on live databases with many PB audited size and many nodes.
Vertica recommendation for OS upgrade - create backup before and restore after - is ridiculous for many PB database size.
Fortunately I already have experience with upgrading OS (from CentOS 5.3 to 7.2) on large Vertica clusters with large data size, keeping data intact.
Upgrade works, but Vertica support is not going to support it. You are pretty much on your own.
Do not do it at home! We can do it because we are professionals!
Find big bag of money and seek for qualified help.
Or, if your bag of money looks more like a small one, do following on your own risk.
How to do OS upgrade to 7.3 while keeping database in place.
Approach below works if database catalog and data are on separate disk volume(s) that can be kept intact while re-imaging system disk with new version of OS. If you have catalog/data on system disk, things became very tricky.
Test OS upgrade on non-prod development cluster first, to be confident. Cluster can be smaller (at least 3 nodes) and have less data (I used test database with one table with one row), but have same mount points for Vertica data and catalog as your production. Try upgrade on one node first.
1) Save dbadmin ssh keys .ssh/* and Vertica config /opt/vertica/config/* somewhere where it will not be loss during re-image. Preserve permissions and ownership.
2) Create OS image for CentOS7.3 that do not touch catalog/data volume.
3) Image should create dbadmin/verticadba user/group with same uid/gid as you have across cluster.
4) Do clean shutdown of database. To check it's clean, you can start it again, and shutdown again, with no sessions allowed to connect.
5) Re-image OS to 7.3 on one node in cluster. After re-image you catalog/data volume should still be mounted in same place, with data intact.
6) Install same version of Vertica rpm current version of your database.
sudo yum -y localinstall vertica-8.xxxx.rpm
7) Run install vertica script for SINGLE-NODE cluster for re-imaged node
sudo /opt/vertica/sbin/install_vertica -r vertica-8.xxx.rpm -s
8) Restore dbadmin ssh keys in ~dbadmin/.ssh and Vertica config /opt/vertica/config/*. Ownership/permissions should be preserved.
9) Check that passwordless access from other nodes in cluster into re-imaged node works.
10) Go to working node in cluster and in admintools do node start.
Node should come up cleanly on new CentOS 7.3.
I recommend breaking cluster into 3 group of nodes (every third node) and do sequential OS re-image for each group of nodes. In this case database will stay up. If something goes wrong (and it will go wrong even it should not) you can use normal procedure to recover node).
After re-imaging all nodes to CentOS 7.3, you will be able to upgrade Vertica to 9.0.
Notes.
Dbadmin ssh keys and Vertica config are identical on all nodes in cluster. You can save them once and re-use during OS re-image.
If you have non-identical dbadmin/verticadba uid/gid (bad practice!!!), it is better to switch to identical one's. Do chown -r dbadmin:verticadba on your catalog and data directories after re-image.
OS re-image can be done without complete database shutdown.
Good luck!