Node went down due to insufficient catalog space and not coming up.

Hi,

2 of the nodes in my 13 node cluster has went down due to insufficient catalog space.
I saw there was lot of space occupied by vertica.log file so cleared those and restarted the database. One of the node came up but other is still stuck.
Anybody has any idea about below errors or any suggestions on how to make this node up again.
I am getting following error.

****startup.log****
{
"node" : "v_dbname_node0001",
"stage" : "Database Halted",
"text" : "@v_vertexdb_node0001: VX001/3394: Failure during commit of transaction 0xfff0000000000bca, cannot proceed\n\tHINT: Restart this node\n\tLOCATION: catalogTransactionPrecommitHook, /scratch_a/release/svrtar3059/vbuild/vertica/Catalog/CatalogHooks.cpp:269",
"timestamp" : "2019-01-22 12:35:28.344"
}

Vertica.log

2019-01-22 12:35:13.568 Spread Client:0x93670f0-fff0000000000bca [Txn] Begin Txn: fff0000000000bca 'Installing New Catalog'
2019-01-22 12:35:13.569 Spread Client:0x93670f0-fff0000000000bca [Catalog] installNewCatalog: Received new catalog, replacing current TRANSACTION-fff0000000000bca catalog (old version=0x78e1e5, new version 0x797c95)
2019-01-22 12:35:13.703 Spread Client:0x93670f0-fff0000000000bca [Catalog] replaceCatalog: dropping 640 objects
2019-01-22 12:35:20.011 DiskSpaceRefresher:0x7f835012f0a0 [Util] Task 'DiskSpaceRefresher' enabled
2019-01-22 12:35:27.626 Spread Client:0x93670f0-fff0000000000bca [Catalog] Catalog OID generator updated based on SYSTEM tier catalog
2019-01-22 12:35:28.344 Spread Client:0x93670f0-fff0000000000bca @v_dbname_node0001: VX001/5445: VIAssert(svb.encompasses(mrSVB)) failed
DETAIL: /scratch_a/release/svrtar3059/vbuild/vertica/Catalog/MiniRosQueries.cpp: 145
HINT: Please report this error to Vertica; try restating your query
LOCATION: ??, /scratch_a/release/svrtar3059/vbuild/vertica/Basics/VAssert.cpp:22
2019-01-22 12:35:28.344 Spread Client:0x93670f0-fff0000000000bca @v_dbname_node0001: {catalogTransactionPrecommitHook} VX001/5445: VIAssert(svb.encompasses(mrSVB)) failed
DETAIL: /scratch_a/release/svrtar3059/vbuild/vertica/Catalog/MiniRosQueries.cpp: 145
HINT: Please report this error to Vertica; try restating your query
LOCATION: ??, /scratch_a/release/svrtar3059/vbuild/vertica/Basics/VAssert.cpp:22
2019-01-22 12:35:28.344 Spread Client:0x93670f0-fff0000000000bca @v_dbname_node0001: VX001/3394: Failure during commit of transaction 0xfff0000000000bca, cannot proceed
HINT: Restart this node
LOCATION: catalogTransactionPrecommitHook, /scratch_a/release/svrtar3059/vbuild/vertica/Catalog/CatalogHooks.cpp:269
2019-01-22 12:35:28.837 Spread Client:0x93670f0-fff0000000000bca [Main] Wrote backtrace to ErrorReport.txt

Thanks

Comments

  • SruthiASruthiA Vertica Employee Administrator
    edited January 2019

    If it is still down, please open a support case since these kind of issues generally need webex session.

  • LakshmiNarayanLakshmiNarayan Community Edition User

    Is that issue resolved?

    Increase homelv space allocation ,post try restarting node.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file