Vertica Cluster Shutting Down repeatedly

udgupta95

Hi Team,

I am facing an issue where the Vertica database shutting down with the below log:

2018-11-21 12:11:34.020 AnalyzeRowCount:0x @v_product_extract_node0001: VX001/3231: Error on opening file for write [/home/dbadmin/product_extract/v_product_extract_node0001_catalog/Catalog/sequencegenerator.cat]: Operation not permitted

Please help.

Jim_Knicely

Did the file permissions/ownership get messed up? Here's what I see:

[dbadmin@s18384357 Catalog]$ ls -lrt sequencegenerator.cat
-rw------- 1 dbadmin verticadba 150 Nov 21 10:46 sequencegenerator.cat

udgupta95

I see the exact same file ownership for me.
The situation is, I start the vertica database and it works perfectly for few hours and suddenly the database stops with the mentioned log at the end of the vertica.log file.

I have checked the file ownership multiple times and it is the exactly as mentioned in your comment but still the issue happens.

Not sure what goes wrong.

RaghavA

Hi,

Even we have the same file rights in our system.

Did you checked your Filesystem? may be your /catalog is reaching 100% while you run this cluster for sometime.

[root@Vertest1 /catalog/vdb/v_vdb_node0001_catalog]$ ls -lrth /catalog/vdb/v_vdb_node0001_catalog/Catalog/sequencegenerator.cat
-rw------- 1 dbadmin verticadba 152 Nov 22 04:45 /catalog/vdb/v_vdb_node0001_catalog/Catalog/sequencegenerator.cat
[root@Vertest1 /catalog/vdb/v_vdb_node0001_catalog]$

Regards,
Raghav

udgupta95

Hi Raghav,
Thanks for the reply.

When you say that my /catalog is reaching 100%, do you mean my disk space is reaching 100% occupied ?

The file permissions in my system look exact same as yours and I have about (100GB)20% of free space on disk.

Also, I see that the catalog folder is occupying just 4KB of space.

Jim_Knicely

Do you have an anti-virus program running? Vertica requires that no AV scanner process should be running on/scanning Vertica related files. This is because the way Vertica operates on some files, may lead to AV scanner believing that the file is infected/malicious.