Creating vertica database on HDFS + NFS failed
Setup -
HDFS + NFS :
Take a 3 node Hadoop cluster.
Export the HDFS filesystem using the Hadoop NFS gateway.
The HDFS filesystem is now accesible as a NFS export.
Vertica :
Install vertica on 2 nodes. ( Community License )
Mount the HDFS + NFS exported filesystem on one vertica node.
mount -t nfs -o nolock hserver:/ /mnt/vert1
Create a vertica database with 1 node only, and specify catalog and data directories in mounted path -
/mnt/vert1/vertica/c3, /mnt/vert1/vertica/d3
Database creation fails with an error -
Unable to bootstrap database: License installed for Vertica Community Edition
I tried the same experiment with a normal NFS exported filesystem, and there were no problems while creating the database.
The vertica documentation says -
Community Edition licenses cannot be installed co-located in a Hadoop infrastructure and used to query data stored in Hadoop formats.
But is there really anyway for vertica to figure out the actual filesystem behind NFS ?
The HDFS filesystem has terabytes of free space.
What could be the problem ?
Log -
Feb 11 18:55:06 [31299] <BashAdapter: dbadmin@10.1.10.29>: ["echo '/mnt/vert1/vertica/c3/sapdb/v_sapdb_node0001_catalog/vertica.log /mnt/vert1/vertica/c3/sapdb/v_sapdb_node0001_catalog/UDxLogs/UDxFencedProcesses.log {\n # rotate weekly\n weekly\n # and keep for 52 weeks\n rotate 52\n # no complaining if vertica did not start yet\n missingok\n # compress log after rotation\n compress\n # no creating a new empty log, vertica will do that\n nocreate\n # if set, only rotates when log size is greater than X\n size 10M\n # delete files after 90 days (not all logrotate pkgs support this keyword)\n# maxage 90\n # we have 2 log files to rotate but only signal vertica once\n sharedscripts\n # signal vertica to reopen and create the log\n postrotate\n kill -USR1 'head -1 /mnt/vert1/vertica/c3/sapdb/v_sapdb_node0001_catalog/vertica.pid 2> /dev/null' 2> /dev/null || true\n endscript\n}\n\n/mnt/vert1/vertica/c3/sapdb/dbLog {\n # rotate weekly\n weekly\n # and keep for 52 weeks\n rotate 52\n # no complaining if vertica did not start yet\n missingok\n # compress log after rotation\n compress\n # this log is stdout, so rotate by copying it aside and truncating\n copytruncate\n}\n' > /opt/vertica/config/logrotate/sapdb"]
Feb 11 18:55:06 [31299] <BashAdapter: dbadmin@10.1.10.29>: (rc=0) []
Feb 11 18:55:06 [31299] <BashAdapter: dbadmin@10.1.10.29>: ['touch /opt/vertica/config/admintools.conf']
Feb 11 18:55:06 [31299] <BashAdapter: dbadmin@10.1.10.29>: (rc=0) []
Feb 11 18:55:06 [31299] Starting DB in multi-node mode
Feb 11 18:55:06 [31299] {"type":"NODEHOST", "state":"VALIDATING", "host":"10.1.10.29", "node":"v_sapdb_node0001", "dbname":"sapdb", "msg":"Creating and Starting Database", "maxSteps":"6", "currStep":"5"}
Feb 11 18:55:06 [31299] [SSH.createDBMultiNodes] dbName 'sapdb', logFile '/mnt/vert1/vertica/c3/sapdb/dbLog', catalogPath '/mnt/vert1/vertica/c3/sapdb', dataPath '/mnt/vert1/vertica/d3/sapdb', nodes '['v_sapdb_node0001']', nodeHosts '['10.1.10.29']', nodeCatalogs '['v_sapdb_node0001_catalog']', nodeDatas '['v_sapdb_node0001_data']', userName 'dbadmin', host '10.1.10.29', spreadInfo '{'largecluster': None, 'logging': 'False', 'controlsubnet': 'default', 'mode': 'broadcast'}', portNo = '5433')
Feb 11 18:55:06 [31299] <BashAdapter: dbadmin@10.1.10.29>: ['LANG=C /sbin/ip -oneline -family inet addr']
Feb 11 18:55:06 [31299] <BashAdapter: dbadmin@10.1.10.29>: (rc=0) ['1: lo inet 127.0.0.1/8 scope host lo', '2: eth0 inet 10.1.10.29/24 brd 10.1.10.255 scope global eth0']
Feb 11 18:55:06 [31299] NetworkProfiles: {'10.1.10.29': [u'10.1.10.29', u'10.1.10.255']}
Feb 11 18:55:06 [31299] Creating database
Feb 11 18:55:06 [31299] [SSH.createDBMultiNodes] bootstrapping with idx 0 node v_sapdb_node0001 on host 10.1.10.29
Feb 11 18:55:06 [31299] [SSH.createDBMultiNodes] bootstrap cmd '/opt/vertica/bin/bootstrap-catalog -C 'sapdb' -H '10.1.10.29' -s 'v_sapdb_node0001' -D '/mnt/vert1/vertica/c3/sapdb/v_sapdb_node0001_catalog' -S '/mnt/vert1/vertica/d3/sapdb/v_sapdb_node0001_data' -L '/opt/vertica/config/share/license.key' -p '5433' -c '10.1.10.29' -B '10.1.10.255' '
Feb 11 18:55:06 [31299] <BashAdapter: dbadmin@10.1.10.29> (Bootstrapping database sapdb): <command hidden>
Feb 11 18:55:13 [31299] Unable to bootstrap database: License installed for Vertica Community Edition
Feb 11 18:55:13 [31299] Multi-node start returns: 0
Feb 11 18:55:13 [31299] [killVerticaProcess] [ -e /mnt/vert1/vertica/c3/sapdb/v_sapdb_node0001_catalog/vertica.pid ]
Feb 11 18:55:13 [31299] <BashAdapter: dbadmin@10.1.10.29>: ['[ -e /mnt/vert1/vertica/c3/sapdb/v_sapdb_node0001_catalog/vertica.pid ]']
Feb 11 18:55:13 [31299] <BashAdapter: dbadmin@10.1.10.29>: (rc=1) []
Feb 11 18:55:13 [31299] [removeCatDataDirs] cat /mnt/vert1/vertica/c3/sapdb/dbLog >> /opt/vertica/log/adminTools-dbadmin.log
Feb 11 18:55:13 [31299] <BashAdapter: dbadmin@10.1.10.29>: ['cat /mnt/vert1/vertica/c3/sapdb/dbLog >> /opt/vertica/log/adminTools-dbadmin.log']
Feb 11 18:55:13 [31299] <BashAdapter: dbadmin@10.1.10.29>: (rc=1) ['cat: /mnt/vert1/vertica/c3/sapdb/dbLog: No such file or directory']
Feb 11 18:55:13 [31299] [removeCatDataDirs] cat /mnt/vert1/vertica/c3/sapdb/v_sapdb_node0001_catalog/vertica.log >> /opt/vertica/log/adminTools-dbadmin.log
Feb 11 18:55:13 [31299] <BashAdapter: dbadmin@10.1.10.29>: ['cat /mnt/vert1/vertica/c3/sapdb/v_sapdb_node0001_catalog/vertica.log >> /opt/vertica/log/adminTools-dbadmin.log']
Feb 11 18:55:13 [31299] <BashAdapter: dbadmin@10.1.10.29>: (rc=1) ['cat: /mnt/vert1/vertica/c3/sapdb/v_sapdb_node0001_catalog/vertica.log: No such file or directory']
Feb 11 18:55:13 [31299] <BashAdapter: dbadmin@10.1.10.29>: ['rm -rf /mnt/vert1/vertica/c3/sapdb']
Feb 11 18:55:13 [31299] <BashAdapter: dbadmin@10.1.10.29>: (rc=0) []
Feb 11 18:55:13 [31299] <BashAdapter: dbadmin@10.1.10.29>: ['rm -rf /mnt/vert1/vertica/d3/sapdb']
Feb 11 18:55:13 [31299] <BashAdapter: dbadmin@10.1.10.29>: (rc=0) []
Feb 11 18:55:13 [31299] Multi-node DB creation unsuccessful
Feb 11 18:55:13 [31299] Create DB complete, signaling FAIL
Feb 11 18:55:13 [31299] Saved Configurator(clusterSize=2, dbs=()) -> /opt/vertica/config/admintools.conf
Feb 11 18:55:13 [31299] <PexpectAdapter: dbadmin@10.1.10.28>: install --owner dbadmin --mode 700 -d /tmp/vstage-dd130859-9c6a-46e0-8752-79a3a9300299
Feb 11 18:55:13 [31299] <PexpectAdapter: dbadmin@10.1.10.28>: (rc=0) []
Feb 11 18:55:13 [31299] Scp outbound: local_file: /opt/vertica/config/admintools.conf, remote_file: dbadmin@10.1.10.28:/tmp/vstage-dd130859-9c6a-46e0-8752-79a3a9300299/file
Feb 11 18:55:14 [31299] <PexpectAdapter: dbadmin@10.1.10.28>: mv -f /tmp/vstage-dd130859-9c6a-46e0-8752-79a3a9300299/file /opt/vertica/config/admintools.conf
HDFS + NFS :
Take a 3 node Hadoop cluster.
Export the HDFS filesystem using the Hadoop NFS gateway.
The HDFS filesystem is now accesible as a NFS export.
Vertica :
Install vertica on 2 nodes. ( Community License )
Mount the HDFS + NFS exported filesystem on one vertica node.
mount -t nfs -o nolock hserver:/ /mnt/vert1
Create a vertica database with 1 node only, and specify catalog and data directories in mounted path -
/mnt/vert1/vertica/c3, /mnt/vert1/vertica/d3
Database creation fails with an error -
Unable to bootstrap database: License installed for Vertica Community Edition
I tried the same experiment with a normal NFS exported filesystem, and there were no problems while creating the database.
The vertica documentation says -
Community Edition licenses cannot be installed co-located in a Hadoop infrastructure and used to query data stored in Hadoop formats.
But is there really anyway for vertica to figure out the actual filesystem behind NFS ?
The HDFS filesystem has terabytes of free space.
What could be the problem ?
Log -
Feb 11 18:55:06 [31299] <BashAdapter: dbadmin@10.1.10.29>: ["echo '/mnt/vert1/vertica/c3/sapdb/v_sapdb_node0001_catalog/vertica.log /mnt/vert1/vertica/c3/sapdb/v_sapdb_node0001_catalog/UDxLogs/UDxFencedProcesses.log {\n # rotate weekly\n weekly\n # and keep for 52 weeks\n rotate 52\n # no complaining if vertica did not start yet\n missingok\n # compress log after rotation\n compress\n # no creating a new empty log, vertica will do that\n nocreate\n # if set, only rotates when log size is greater than X\n size 10M\n # delete files after 90 days (not all logrotate pkgs support this keyword)\n# maxage 90\n # we have 2 log files to rotate but only signal vertica once\n sharedscripts\n # signal vertica to reopen and create the log\n postrotate\n kill -USR1 'head -1 /mnt/vert1/vertica/c3/sapdb/v_sapdb_node0001_catalog/vertica.pid 2> /dev/null' 2> /dev/null || true\n endscript\n}\n\n/mnt/vert1/vertica/c3/sapdb/dbLog {\n # rotate weekly\n weekly\n # and keep for 52 weeks\n rotate 52\n # no complaining if vertica did not start yet\n missingok\n # compress log after rotation\n compress\n # this log is stdout, so rotate by copying it aside and truncating\n copytruncate\n}\n' > /opt/vertica/config/logrotate/sapdb"]
Feb 11 18:55:06 [31299] <BashAdapter: dbadmin@10.1.10.29>: (rc=0) []
Feb 11 18:55:06 [31299] <BashAdapter: dbadmin@10.1.10.29>: ['touch /opt/vertica/config/admintools.conf']
Feb 11 18:55:06 [31299] <BashAdapter: dbadmin@10.1.10.29>: (rc=0) []
Feb 11 18:55:06 [31299] Starting DB in multi-node mode
Feb 11 18:55:06 [31299] {"type":"NODEHOST", "state":"VALIDATING", "host":"10.1.10.29", "node":"v_sapdb_node0001", "dbname":"sapdb", "msg":"Creating and Starting Database", "maxSteps":"6", "currStep":"5"}
Feb 11 18:55:06 [31299] [SSH.createDBMultiNodes] dbName 'sapdb', logFile '/mnt/vert1/vertica/c3/sapdb/dbLog', catalogPath '/mnt/vert1/vertica/c3/sapdb', dataPath '/mnt/vert1/vertica/d3/sapdb', nodes '['v_sapdb_node0001']', nodeHosts '['10.1.10.29']', nodeCatalogs '['v_sapdb_node0001_catalog']', nodeDatas '['v_sapdb_node0001_data']', userName 'dbadmin', host '10.1.10.29', spreadInfo '{'largecluster': None, 'logging': 'False', 'controlsubnet': 'default', 'mode': 'broadcast'}', portNo = '5433')
Feb 11 18:55:06 [31299] <BashAdapter: dbadmin@10.1.10.29>: ['LANG=C /sbin/ip -oneline -family inet addr']
Feb 11 18:55:06 [31299] <BashAdapter: dbadmin@10.1.10.29>: (rc=0) ['1: lo inet 127.0.0.1/8 scope host lo', '2: eth0 inet 10.1.10.29/24 brd 10.1.10.255 scope global eth0']
Feb 11 18:55:06 [31299] NetworkProfiles: {'10.1.10.29': [u'10.1.10.29', u'10.1.10.255']}
Feb 11 18:55:06 [31299] Creating database
Feb 11 18:55:06 [31299] [SSH.createDBMultiNodes] bootstrapping with idx 0 node v_sapdb_node0001 on host 10.1.10.29
Feb 11 18:55:06 [31299] [SSH.createDBMultiNodes] bootstrap cmd '/opt/vertica/bin/bootstrap-catalog -C 'sapdb' -H '10.1.10.29' -s 'v_sapdb_node0001' -D '/mnt/vert1/vertica/c3/sapdb/v_sapdb_node0001_catalog' -S '/mnt/vert1/vertica/d3/sapdb/v_sapdb_node0001_data' -L '/opt/vertica/config/share/license.key' -p '5433' -c '10.1.10.29' -B '10.1.10.255' '
Feb 11 18:55:06 [31299] <BashAdapter: dbadmin@10.1.10.29> (Bootstrapping database sapdb): <command hidden>
Feb 11 18:55:13 [31299] Unable to bootstrap database: License installed for Vertica Community Edition
Feb 11 18:55:13 [31299] Multi-node start returns: 0
Feb 11 18:55:13 [31299] [killVerticaProcess] [ -e /mnt/vert1/vertica/c3/sapdb/v_sapdb_node0001_catalog/vertica.pid ]
Feb 11 18:55:13 [31299] <BashAdapter: dbadmin@10.1.10.29>: ['[ -e /mnt/vert1/vertica/c3/sapdb/v_sapdb_node0001_catalog/vertica.pid ]']
Feb 11 18:55:13 [31299] <BashAdapter: dbadmin@10.1.10.29>: (rc=1) []
Feb 11 18:55:13 [31299] [removeCatDataDirs] cat /mnt/vert1/vertica/c3/sapdb/dbLog >> /opt/vertica/log/adminTools-dbadmin.log
Feb 11 18:55:13 [31299] <BashAdapter: dbadmin@10.1.10.29>: ['cat /mnt/vert1/vertica/c3/sapdb/dbLog >> /opt/vertica/log/adminTools-dbadmin.log']
Feb 11 18:55:13 [31299] <BashAdapter: dbadmin@10.1.10.29>: (rc=1) ['cat: /mnt/vert1/vertica/c3/sapdb/dbLog: No such file or directory']
Feb 11 18:55:13 [31299] [removeCatDataDirs] cat /mnt/vert1/vertica/c3/sapdb/v_sapdb_node0001_catalog/vertica.log >> /opt/vertica/log/adminTools-dbadmin.log
Feb 11 18:55:13 [31299] <BashAdapter: dbadmin@10.1.10.29>: ['cat /mnt/vert1/vertica/c3/sapdb/v_sapdb_node0001_catalog/vertica.log >> /opt/vertica/log/adminTools-dbadmin.log']
Feb 11 18:55:13 [31299] <BashAdapter: dbadmin@10.1.10.29>: (rc=1) ['cat: /mnt/vert1/vertica/c3/sapdb/v_sapdb_node0001_catalog/vertica.log: No such file or directory']
Feb 11 18:55:13 [31299] <BashAdapter: dbadmin@10.1.10.29>: ['rm -rf /mnt/vert1/vertica/c3/sapdb']
Feb 11 18:55:13 [31299] <BashAdapter: dbadmin@10.1.10.29>: (rc=0) []
Feb 11 18:55:13 [31299] <BashAdapter: dbadmin@10.1.10.29>: ['rm -rf /mnt/vert1/vertica/d3/sapdb']
Feb 11 18:55:13 [31299] <BashAdapter: dbadmin@10.1.10.29>: (rc=0) []
Feb 11 18:55:13 [31299] Multi-node DB creation unsuccessful
Feb 11 18:55:13 [31299] Create DB complete, signaling FAIL
Feb 11 18:55:13 [31299] Saved Configurator(clusterSize=2, dbs=()) -> /opt/vertica/config/admintools.conf
Feb 11 18:55:13 [31299] <PexpectAdapter: dbadmin@10.1.10.28>: install --owner dbadmin --mode 700 -d /tmp/vstage-dd130859-9c6a-46e0-8752-79a3a9300299
Feb 11 18:55:13 [31299] <PexpectAdapter: dbadmin@10.1.10.28>: (rc=0) []
Feb 11 18:55:13 [31299] Scp outbound: local_file: /opt/vertica/config/admintools.conf, remote_file: dbadmin@10.1.10.28:/tmp/vstage-dd130859-9c6a-46e0-8752-79a3a9300299/file
Feb 11 18:55:14 [31299] <PexpectAdapter: dbadmin@10.1.10.28>: mv -f /tmp/vstage-dd130859-9c6a-46e0-8752-79a3a9300299/file /opt/vertica/config/admintools.conf
0