Error VX001/3381 when migrating Vertica 23.4 from RHEL7.5 to RHEL8
I'm trying to migrate Vertica from 23.4 on RHEL7.5 to RHEL8.1 on EC2 but encountering an error.
The database starts and runs just fine on RHEL7.5 but fails to start on RHEL8.1.
/opt/vertica/bin/admintools -t start_db -d mydatabase -p ******** fails with host ['127.0.0.1'] report: @v_protobi_db_node0001: VX001/3381: Failed to install new catalog
Going with traditional slower startup
Starting nodes:
v_protobi_db_node0001 (127.0.0.1)
Starting Vertica on all nodes. Please wait, databases with a large catalog may take a while to initialize.
Node Status: v_protobi_db_node0001: (DOWN)
Node Status: v_protobi_db_node0001: (DOWN)
Node Status: v_protobi_db_node0001: (DOWN)
Node Status: v_protobi_db_node0001: (DOWN)
Node Status: v_protobi_db_node0001: (DOWN)
Node Status: v_protobi_db_node0001: (DOWN)
Node Status: v_protobi_db_node0001: (DOWN)
Node Status: v_protobi_db_node0001: (DOWN)
Node Status: v_protobi_db_node0001: (DOWN)
Node Status: v_protobi_db_node0001: (DOWN)
Found these errors in startup.logs on hosts:
host ['127.0.0.1'] report: @v_protobi_db_node0001: VX001/3381: Failed to install new catalog
LOCATION: doInstallAndJoin, /data/jenkins/workspace/RE-ReleaseBuilds/RE-Laserbeam/server/vertica/Transaction/TransAPI.cpp:6520
Database protobi_db did not start successfully: Something Failed.
Hint: if you use Eon mode, then you may need to start all primary nodes.
Both systems are running the same version of Vertica:
vertica(v23.4.0-17) built by @re-docker3-server from
releases/VER_23_4_RELEASE_BUILD_0_17_20250523@5746f827c5a375bd2c4ad04d5d4a5c770fb82bfd on 'Fri May 23 21:50:07 2025'
Both systems have the same configuration (copied /opt/vertica/config/admintools.conf from old to new).
Both systems have exactly the same information on the EBS volumns, same directories mounted, same permissions, etc.
The only difference I think is the RHEL7.5 was launched from a Vertica AMI with root access disabled and no RedHat license, whereas the new RHEL8.1 is a new instance with a RedHat license.
Answers
RESOLVED
It turns out the issue was making sure that
/var/verticaexists and that userdbadminhas read+write permissions. I had assumed that all Vertica files are contained under/opt/vertica/and/vertica/data/The clue was in
/vertica/data/protobi_db/v_$DATABASENAME_node0001_catalog/vertica.logand/vertica/data/protobi_db/v_$DATABASENAME_node0001_catalog/startup.log