storage_location + copy cluster + node down - dependencies?
Hi,
Recently we had added few storage locations with the type USER to have some non-superuser accounts use the COPY command. We run copy cluster every night to copy data to our DR location.
The night these new storage locations (type=USER) was added, COPY cluster failed (atleast did not report that it was completed like it usually does). But it appeared that the data had been copied, the new storage location metadata had been copied (when i did a select * from storage_locations i found same entries in both the live and DR environments). BUT for some reason Vertica could not come up in couple of nodes. The logs pointed that the underlying directory behind the new storage location that was added was not present in these nodes.
Snippet from vertica.log:-
2014-10-27 09:45:57.492 Spread Client:0x843fb70-fff0000000000bca [Catalog] <INFO> Full License Hash Check Passed.2014-10-27 09:45:57.512 Spread Client:0x843fb70-fff0000000000bca [Txn] <INFO> Starting Commit: Txn: fff0000000000bca 'Installing New Catalog'
2014-10-27 09:45:57.513 CatalogCheckpointer:0x7ff8240103d0 [Txn] <INFO> Considering Checkpoint
2014-10-27 09:45:57.513 CatalogCheckpointer:0x7ff8240103d0 [Catalog] <INFO> Checkpointing - Versions GLOBAL: 60283202 LOCAL: 60283199, COMMIT 5
2014-10-27 09:45:57.513 Spread Client:0x843fb70 [Txn] <INFO> Commit Complete: Txn: fff0000000000bca at epoch 0xce9a68
2014-10-27 09:45:58.624 Spread Client:0x843fb70-fff0000000000bca [Txn] <INFO> Begin Txn: fff0000000000bca 'verify Storage Locations & Export Addresses'
2014-10-27 09:45:58.624 Spread Client:0x843fb70-fff0000000000bca [SAL] <INFO> Storage Location /data2/vertica/data/blah/v_blah_node0012_data Exists
2014-10-27 09:45:58.624 Spread Client:0x843fb70-fff0000000000bca [Init] <INFO> Validated Storage Location /data2/vertica/data/blah/v_blah_node0012_data
2014-10-27 09:45:58.624 Spread Client:0x843fb70-fff0000000000bca <ERROR> @v_node0012: {doInstallAndJoin} 42501/2833: Could not create directory [/ora_mnt]: Permission denied
LOCATION: mkdir, /scratch_a/release/vbuild/vertica/Util/IO/File.cpp:374
2014-10-27 09:45:58.624 Spread Client:0x843fb70-fff0000000000bca <PANIC> @v_node0012: VX001/3381: Failed to install new catalog
LOCATION: doInstallAndJoin, /scratch_a/release/vbuild/vertica/Transaction/TransAPI.cpp:2559
2014-10-27 09:45:58.625 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location1
2014-10-27 09:45:58.625 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location2
2014-10-27 09:45:58.625 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location3
2014-10-27 09:45:59.006 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location1
2014-10-27 09:45:59.006 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location2
2014-10-27 09:45:59.006 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location3
2014-10-27 09:46:00.037 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location1
2014-10-27 09:46:00.037 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location2
2014-10-27 09:46:00.037 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location3
2014-10-27 09:46:01.005 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location1
2014-10-27 09:46:01.005 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location2
2014-10-27 09:46:01.005 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location3
2014-10-27 09:46:02.006 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location1
2014-10-27 09:46:02.006 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location2
2014-10-27 09:46:02.006 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location3
2014-10-27 09:46:02.571 Spread Client:0x843fb70-fff0000000000bca [Main] <PANIC> Wrote backtrace to ErrorReport.txt
What does this mean? Vertica tried to create some underlying directories for a USER storage location and it failed (permission denied) and hence the instance couldnt be brought up? Can someone please shed some light on this?
Recently we had added few storage locations with the type USER to have some non-superuser accounts use the COPY command. We run copy cluster every night to copy data to our DR location.
The night these new storage locations (type=USER) was added, COPY cluster failed (atleast did not report that it was completed like it usually does). But it appeared that the data had been copied, the new storage location metadata had been copied (when i did a select * from storage_locations i found same entries in both the live and DR environments). BUT for some reason Vertica could not come up in couple of nodes. The logs pointed that the underlying directory behind the new storage location that was added was not present in these nodes.
Snippet from vertica.log:-
2014-10-27 09:45:57.492 Spread Client:0x843fb70-fff0000000000bca [Catalog] <INFO> Full License Hash Check Passed.2014-10-27 09:45:57.512 Spread Client:0x843fb70-fff0000000000bca [Txn] <INFO> Starting Commit: Txn: fff0000000000bca 'Installing New Catalog'
2014-10-27 09:45:57.513 CatalogCheckpointer:0x7ff8240103d0 [Txn] <INFO> Considering Checkpoint
2014-10-27 09:45:57.513 CatalogCheckpointer:0x7ff8240103d0 [Catalog] <INFO> Checkpointing - Versions GLOBAL: 60283202 LOCAL: 60283199, COMMIT 5
2014-10-27 09:45:57.513 Spread Client:0x843fb70 [Txn] <INFO> Commit Complete: Txn: fff0000000000bca at epoch 0xce9a68
2014-10-27 09:45:58.624 Spread Client:0x843fb70-fff0000000000bca [Txn] <INFO> Begin Txn: fff0000000000bca 'verify Storage Locations & Export Addresses'
2014-10-27 09:45:58.624 Spread Client:0x843fb70-fff0000000000bca [SAL] <INFO> Storage Location /data2/vertica/data/blah/v_blah_node0012_data Exists
2014-10-27 09:45:58.624 Spread Client:0x843fb70-fff0000000000bca [Init] <INFO> Validated Storage Location /data2/vertica/data/blah/v_blah_node0012_data
2014-10-27 09:45:58.624 Spread Client:0x843fb70-fff0000000000bca <ERROR> @v_node0012: {doInstallAndJoin} 42501/2833: Could not create directory [/ora_mnt]: Permission denied
LOCATION: mkdir, /scratch_a/release/vbuild/vertica/Util/IO/File.cpp:374
2014-10-27 09:45:58.624 Spread Client:0x843fb70-fff0000000000bca <PANIC> @v_node0012: VX001/3381: Failed to install new catalog
LOCATION: doInstallAndJoin, /scratch_a/release/vbuild/vertica/Transaction/TransAPI.cpp:2559
2014-10-27 09:45:58.625 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location1
2014-10-27 09:45:58.625 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location2
2014-10-27 09:45:58.625 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location3
2014-10-27 09:45:59.006 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location1
2014-10-27 09:45:59.006 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location2
2014-10-27 09:45:59.006 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location3
2014-10-27 09:46:00.037 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location1
2014-10-27 09:46:00.037 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location2
2014-10-27 09:46:00.037 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location3
2014-10-27 09:46:01.005 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location1
2014-10-27 09:46:01.005 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location2
2014-10-27 09:46:01.005 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location3
2014-10-27 09:46:02.006 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location1
2014-10-27 09:46:02.006 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location2
2014-10-27 09:46:02.006 nameless:0x67b2ac0 <WARNING> @v_node0012: 01000/4859: SysInfo::getMegaBytesFree(): Could not call statvfs on /my/location3
2014-10-27 09:46:02.571 Spread Client:0x843fb70-fff0000000000bca [Main] <PANIC> Wrote backtrace to ErrorReport.txt
What does this mean? Vertica tried to create some underlying directories for a USER storage location and it failed (permission denied) and hence the instance couldnt be brought up? Can someone please shed some light on this?
0