Node went down due to insufficient catalog space and not coming up.
Hi,
2 of the nodes in my 13 node cluster has went down due to insufficient catalog space.
I saw there was lot of space occupied by vertica.log file so cleared those and restarted the database. One of the node came up but other is still stuck.
Anybody has any idea about below errors or any suggestions on how to make this node up again.
I am getting following error.
****startup.log****
{
"node" : "v_dbname_node0001",
"stage" : "Database Halted",
"text" : "@v_vertexdb_node0001: VX001/3394: Failure during commit of transaction 0xfff0000000000bca, cannot proceed\n\tHINT: Restart this node\n\tLOCATION: catalogTransactionPrecommitHook, /scratch_a/release/svrtar3059/vbuild/vertica/Catalog/CatalogHooks.cpp:269",
"timestamp" : "2019-01-22 12:35:28.344"
}
Vertica.log
2019-01-22 12:35:13.568 Spread Client:0x93670f0-fff0000000000bca [Txn] Begin Txn: fff0000000000bca 'Installing New Catalog'
2019-01-22 12:35:13.569 Spread Client:0x93670f0-fff0000000000bca [Catalog] installNewCatalog: Received new catalog, replacing current TRANSACTION-fff0000000000bca catalog (old version=0x78e1e5, new version 0x797c95)
2019-01-22 12:35:13.703 Spread Client:0x93670f0-fff0000000000bca [Catalog] replaceCatalog: dropping 640 objects
2019-01-22 12:35:20.011 DiskSpaceRefresher:0x7f835012f0a0 [Util] Task 'DiskSpaceRefresher' enabled
2019-01-22 12:35:27.626 Spread Client:0x93670f0-fff0000000000bca [Catalog] Catalog OID generator updated based on SYSTEM tier catalog
2019-01-22 12:35:28.344 Spread Client:0x93670f0-fff0000000000bca @v_dbname_node0001: VX001/5445: VIAssert(svb.encompasses(mrSVB)) failed
DETAIL: /scratch_a/release/svrtar3059/vbuild/vertica/Catalog/MiniRosQueries.cpp: 145
HINT: Please report this error to Vertica; try restating your query
LOCATION: ??, /scratch_a/release/svrtar3059/vbuild/vertica/Basics/VAssert.cpp:22
2019-01-22 12:35:28.344 Spread Client:0x93670f0-fff0000000000bca @v_dbname_node0001: {catalogTransactionPrecommitHook} VX001/5445: VIAssert(svb.encompasses(mrSVB)) failed
DETAIL: /scratch_a/release/svrtar3059/vbuild/vertica/Catalog/MiniRosQueries.cpp: 145
HINT: Please report this error to Vertica; try restating your query
LOCATION: ??, /scratch_a/release/svrtar3059/vbuild/vertica/Basics/VAssert.cpp:22
2019-01-22 12:35:28.344 Spread Client:0x93670f0-fff0000000000bca @v_dbname_node0001: VX001/3394: Failure during commit of transaction 0xfff0000000000bca, cannot proceed
HINT: Restart this node
LOCATION: catalogTransactionPrecommitHook, /scratch_a/release/svrtar3059/vbuild/vertica/Catalog/CatalogHooks.cpp:269
2019-01-22 12:35:28.837 Spread Client:0x93670f0-fff0000000000bca [Main] Wrote backtrace to ErrorReport.txt
Thanks
Comments
If it is still down, please open a support case since these kind of issues generally need webex session.
Is that issue resolved?
Increase homelv space allocation ,post try restarting node.