Vertica service not starting after server reboot

We have vertica setup in one of our server with one node. Last night the server went down and it was restarted. From then on, vertica service is not starting and so we are unable to start the database as well. What could be the issue here? Please help.

Comments

  • Hi Hari

     

    Can you kindly provide us what version of vertica you have been running on?

     

    $ rpm -qa | grep vertica

     

    Also provide us the output of

     

    $admintools -t list_allnodes

     

    Along with the above output please pass on the vertica.log file of the same node to have a look into it.

     

     

    Thanks

    Rahul

  • Hi Rahul, below are the details you asked: -

     

    $ rpm -qa | grep vertica

     

    vertica-7.2.1-0.x86_64

     

     

    $admintools -t list_allnodes

    attached the output as screenshot

     

     

    vertica.log

    attached herewith


  • Hi Hari

     

    On quick analysis it can be clearly seen from the below snippet of vertica.log file that the vertica got a signal from another process, an OS kill which indicates that another process intentionally sent the Vertica process a shutdown command.

    These were the last messages reported in vertica.log file.So just wanted to know is it the same time when you rebooted your cluster because it looks like to be the case here?

    If yes did you try restarting the cluster after that since we can't see any further messages in vertica.log getting reported.

    Please try to restart the node/cluster from admintools and let me know what error message you receive? Also provide us the latest vertica.log file along with admintools-dbadmin.log in order to help you out further.

    2016-06-08 01:13:23.373 Main:0x7c87ab0 [Basics] <INFO>
    Process 30326 received SIGTERM from 14290
    ------------------- ps -ef --forest -----------------
    UID PID PPID C STIME TTY TIME CMD
    root 2 0 0 May18 ? 00:00:00 [kthreadd]
    root 3 2 0 May18 ? 00:00:34 \_ [migration/0]
    root 4 2 0 May18 ? 00:00:32 \_ [ksoftirqd/0]
    root 5 2 0 May18 ? 00:00:00 \_ [stopper/0]
    root 6 2 0 May18 ? 00:00:01 \_ [watchdog/0]
    root 7 2 0 May18 ? 00:00:28 \_ [migration/1]
    root 8 2 0 May18 ? 00:00:00 \_ [stopper/1]
    root 9 2 0 May18 ? 00:00:21 \_ [ksoftirqd/1]
    root 10 2 0 May18 ? 00:00:01 \_ [watchdog/1]
    root 11 2 0 May18 ? 00:00:28 \_ [migration/2]
    root 12 2 0 May18 ? 00:00:00 \_ [stopper/2]
    root 13 2 0 May18 ? 00:00:16 \_ [ksoftirqd/2]
    root 14 2 0 May18 ? 00:00:01 \_ [watchdog/2]
    root 15 2 0 May18 ? 00:00:24 \_ [migration/3]
    root 16 2 0 May18 ? 00:00:00 \_ [stopper/3]
    root 17 2 0 May18 ? 00:00:17 \_ [ksoftirqd/3]
    root 18 2 0 May18 ? 00:00:01 \_ [watchdog/3]
    root 19 2 0 May18 ? 00:00:27 \_ [migration/4]
    root 20 2 0 May18 ? 00:00:00 \_ [stopper/4]
    root 21 2 0 May18 ? 00:02:55 \_ [ksoftirqd/4]
    root 22 2 0 May18 ? 00:00:01 \_ [watchdog/4]
    root 23 2 0 May18 ? 00:00:25 \_ [migration/5]
    root 24 2 0 May18 ? 00:00:00 \_ [stopper/5]
    root 25 2 0 May18 ? 00:02:34 \_ [ksoftirqd/5]
    root 26 2 0 May18 ? 00:00:01 \_ [watchdog/5]
    root 27 2 0 May18 ? 00:00:32 \_ [migration/6]
    root 28 2 0 May18 ? 00:00:00 \_ [stopper/6]
    root 29 2 0 May18 ? 00:02:48 \_ [ksoftirqd/6]
    root 30 2 0 May18 ? 00:00:01 \_ [watchdog/6]
    root 31 2 0 May18 ? 00:00:25 \_ [migration/7]
    root 32 2 0 May18 ? 00:00:00 \_ [stopper/7]
    root 33 2 0 May18 ? 00:03:51 \_ [ksoftirqd/7]
    root 34 2 0 May18 ? 00:00:01 \_ [watchdog/7]
    root 35 2 0 May18 ? 00:01:49 \_ [events/0]
    root 36 2 0 May18 ? 00:00:47 \_ [events/1]
    root 37 2 0 May18 ? 00:00:45 \_ [events/2]
    root 38 2 0 May18 ? 00:00:38 \_ [events/3]
    root 39 2 0 May18 ? 00:01:16 \_ [events/4]
    root 40 2 0 May18 ? 00:00:54 \_ [events/5]
    root 41 2 0 May18 ? 00:00:41 \_ [events/6]
    root 42 2 0 May18 ? 00:01:05 \_ [events/7]
    root 43 2 0 May18 ? 00:00:00 \_ [events/0]
    root 44 2 0 May18 ? 00:00:00 \_ [events/1]
    root 45 2 0 May18 ? 00:00:00 \_ [events/2]
    root 46 2 0 May18 ? 00:00:00 \_ [events/3]
    root 47 2 0 May18 ? 00:00:00 \_ [events/4]
    root 48 2 0 May18 ? 00:00:00 \_ [events/5]
    root 49 2 0 May18 ? 00:00:00 \_ [events/6]
    root 50 2 0 May18 ? 00:00:00 \_ [events/7]
    root 51 2 0 May18 ? 00:00:00 \_ [events_long/0]
    root 52 2 0 May18 ? 00:00:00 \_ [events_long/1]
    root 53 2 0 May18 ? 00:00:00 \_ [events_long/2]
    root 54 2 0 May18 ? 00:00:00 \_ [events_long/3]
    root 55 2 0 May18 ? 00:00:00 \_ [events_long/4]
    root 56 2 0 May18 ? 00:00:00 \_ [events_long/5]
    root 57 2 0 May18 ? 00:00:00 \_ [events_long/6]
    root 58 2 0 May18 ? 00:00:00 \_ [events_long/7]
    root 59 2 0 May18 ? 00:00:00 \_ [events_power_ef]
    root 60 2 0 May18 ? 00:00:00 \_ [events_power_ef]
    root 61 2 0 May18 ? 00:00:00 \_ [events_power_ef]
    root 62 2 0 May18 ? 00:00:00 \_ [events_power_ef]
    root 63 2 0 May18 ? 00:00:00 \_ [events_power_ef]
    root 64 2 0 May18 ? 00:00:00 \_ [events_power_ef]
    root 65 2 0 May18 ? 00:00:00 \_ [events_power_ef]
    root 66 2 0 May18 ? 00:00:00 \_ [events_power_ef]
    root 67 2 0 May18 ? 00:00:00 \_ [cgroup]
    root 68 2 0 May18 ? 00:00:00 \_ [khelper]
    root 69 2 0 May18 ? 00:00:00 \_ [netns]
    root 70 2 0 May18 ? 00:00:00 \_ [async/mgr]
    root 71 2 0 May18 ? 00:00:00 \_ [pm]
    root 72 2 0 May18 ? 00:00:06 \_ [sync_supers]
    root 73 2 0 May18 ? 00:00:05 \_ [bdi-default]
    root 74 2 0 May18 ? 00:00:00 \_ [kintegrityd/0]
    root 75 2 0 May18 ? 00:00:00 \_ [kintegrityd/1]
    root 76 2 0 May18 ? 00:00:00 \_ [kintegrityd/2]
    root 77 2 0 May18 ? 00:00:00 \_ [kintegrityd/3]
    root 78 2 0 May18 ? 00:00:00 \_ [kintegrityd/4]
    root 79 2 0 May18 ? 00:00:00 \_ [kintegrityd/5]
    root 80 2 0 May18 ? 00:00:00 \_ [kintegrityd/6]
    root 81 2 0 May18 ? 00:00:00 \_ [kintegrityd/7]
    root 82 2 0 May18 ? 00:00:01 \_ [kblockd/0]
    root 83 2 0 May18 ? 00:00:00 \_ [kblockd/1]
    root 84 2 0 May18 ? 00:00:01 \_ [kblockd/2]
    root 85 2 0 May18 ? 00:00:00 \_ [kblockd/3]
    root 86 2 0 May18 ? 00:00:00 \_ [kblockd/4]
    root 87 2 0 May18 ? 00:00:00 \_ [kblockd/5]
    root 88 2 0 May18 ? 00:00:03 \_ [kblockd/6]
    root 89 2 0 May18 ? 00:00:00 \_ [kblockd/7]
    root 90 2 0 May18 ? 00:00:00 \_ [kacpid]
    root 91 2 0 May18 ? 00:00:00 \_ [kacpi_notify]
    root 92 2 0 May18 ? 00:00:00 \_ [kacpi_hotplug]
    root 93 2 0 May18 ? 00:00:00 \_ [ata_aux]
    root 94 2 0 May18 ? 00:00:00 \_ [ata_sff/0]
    root 95 2 0 May18 ? 00:00:00 \_ [ata_sff/1]
    root 96 2 0 May18 ? 00:00:00 \_ [ata_sff/2]
    root 97 2 0 May18 ? 00:00:00 \_ [ata_sff/3]
    root 98 2 0 May18 ? 00:00:00 \_ [ata_sff/4]
    root 99 2 0 May18 ? 00:00:00 \_ [ata_sff/5]
    root 100 2 0 May18 ? 00:00:00 \_ [ata_sff/6]
    root 101 2 0 May18 ? 00:00:00 \_ [ata_sff/7]
    root 102 2 0 May18 ? 00:00:00 \_ [ksuspend_usbd]
    root 103 2 0 May18 ? 00:00:00 \_ [khubd]
    root 104 2 0 May18 ? 00:00:00 \_ [kseriod]
    root 105 2 0 May18 ? 00:00:00 \_ [md/0]
    root 106 2 0 May18 ? 00:00:00 \_ [md/1]
    root 107 2 0 May18 ? 00:00:00 \_ [md/2]
    root 108 2 0 May18 ? 00:00:00 \_ [md/3]
    root 109 2 0 May18 ? 00:00:00 \_ [md/4]
    root 110 2 0 May18 ? 00:00:00 \_ [md/5]
    root 111 2 0 May18 ? 00:00:00 \_ [md/6]
    root 112 2 0 May18 ? 00:00:00 \_ [md/7]
    root 113 2 0 May18 ? 00:00:00 \_ [md_misc/0]
    root 114 2 0 May18 ? 00:00:00 \_ [md_misc/1]
    root 115 2 0 May18 ? 00:00:00 \_ [md_misc/2]
    root 116 2 0 May18 ? 00:00:00 \_ [md_misc/3]
    root 117 2 0 May18 ? 00:00:00 \_ [md_misc/4]
    root 118 2 0 May18 ? 00:00:00 \_ [md_misc/5]
    root 119 2 0 May18 ? 00:00:00 \_ [md_misc/6]
    root 120 2 0 May18 ? 00:00:00 \_ [md_misc/7]
    root 121 2 0 May18 ? 00:00:00 \_ [linkwatch]
    root 123 2 0 May18 ? 00:00:00 \_ [khungtaskd]
    root 124 2 0 May18 ? 00:04:58 \_ [kswapd0]
    root 125 2 0 May18 ? 00:00:00 \_ [ksmd]
    root 126 2 0 May18 ? 00:01:21 \_ [khugepaged]
    root 127 2 0 May18 ? 00:00:00 \_ [aio/0]
    root 128 2 0 May18 ? 00:00:00 \_ [aio/1]
    root 129 2 0 May18 ? 00:00:00 \_ [aio/2]
    root 130 2 0 May18 ? 00:00:00 \_ [aio/3]
    root 131 2 0 May18 ? 00:00:00 \_ [aio/4]
    root 132 2 0 May18 ? 00:00:00 \_ [aio/5]
    root 133 2 0 May18 ? 00:00:00 \_ [aio/6]
    root 134 2 0 May18 ? 00:00:00 \_ [aio/7]
    root 135 2 0 May18 ? 00:00:00 \_ [crypto/0]
    root 136 2 0 May18 ? 00:00:00 \_ [crypto/1]
    root 137 2 0 May18 ? 00:00:00 \_ [crypto/2]
    root 138 2 0 May18 ? 00:00:00 \_ [crypto/3]
    root 139 2 0 May18 ? 00:00:00 \_ [crypto/4]
    root 140 2 0 May18 ? 00:00:00 \_ [crypto/5]
    root 141 2 0 May18 ? 00:00:00 \_ [crypto/6]
    root 142 2 0 May18 ? 00:00:00 \_ [crypto/7]
    root 149 2 0 May18 ? 00:00:00 \_ [kthrotld/0]
    root 150 2 0 May18 ? 00:00:00 \_ [kthrotld/1]
    root 151 2 0 May18 ? 00:00:00 \_ [kthrotld/2]
    root 152 2 0 May18 ? 00:00:00 \_ [kthrotld/3]
    root 153 2 0 May18 ? 00:00:00 \_ [kthrotld/4]
    root 154 2 0 May18 ? 00:00:00 \_ [kthrotld/5]
    root 155 2 0 May18 ? 00:00:00 \_ [kthrotld/6]
    root 156 2 0 May18 ? 00:00:00 \_ [kthrotld/7]
    root 158 2 0 May18 ? 00:00:00 \_ [kpsmoused]
    root 159 2 0 May18 ? 00:00:00 \_ [usbhid_resumer]
    root 160 2 0 May18 ? 00:00:00 \_ [deferwq]
    root 192 2 0 May18 ? 00:00:00 \_ [kdmremove]
    root 193 2 0 May18 ? 00:00:00 \_ [kstriped]
    root 338 2 0 May18 ? 00:00:00 \_ [scsi_eh_0]
    root 339 2 0 May18 ? 00:00:00 \_ [scsi_eh_1]
    root 340 2 0 May18 ? 00:00:00 \_ [scsi_eh_2]
    root 341 2 0 May18 ? 00:00:00 \_ [scsi_eh_3]
    root 342 2 0 May18 ? 00:00:00 \_ [scsi_eh_4]
    root 343 2 0 May18 ? 00:00:00 \_ [scsi_eh_5]
    root 454 2 0 May18 ? 00:07:35 \_ [jbd2/sda5-8]
    root 455 2 0 May18 ? 00:00:00 \_ [ext4-dio-unwrit]
    root 748 2 0 May18 ? 00:13:32 \_ [flush-8:0]
    root 1027 2 0 May18 ? 00:00:00 \_ [jbd2/sda1-8]
    root 1028 2 0 May18 ? 00:00:00 \_ [ext4-dio-unwrit]
    root 1029 2 0 May18 ? 00:00:12 \_ [jbd2/sda2-8]
    root 1030 2 0 May18 ? 00:00:00 \_ [ext4-dio-unwrit]
    root 1031 2 0 May18 ? 00:00:15 \_ [jbd2/sda3-8]
    root 1032 2 0 May18 ? 00:00:00 \_ [ext4-dio-unwrit]
    root 1112 2 0 May18 ? 00:00:12 \_ [kauditd]
    root 6449 2 0 May19 ? 00:00:00 \_ [bluetooth]
    root 1 0 0 May18 ? 00:00:06 /sbin/init
    rpcuser 1573 1 0 May18 ? 00:00:00 rpc.statd
    dbadmin 4182 1 0 May18 ? 00:02:19 /opt/vertica/spread/sbin/spread -c /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/spread.conf -D /opt/vertica/spread/tmp
    dbadmin 4184 1 0 May18 ? 04:13:12 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5445 -P 5447 -Y ipv4
    dbadmin 4384 4184 0 May18 ? 00:02:31 \_ /opt/vertica/bin/vertica-udx-zygote 15 3 4184 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    dbadmin 14327 4184 0 01:13 ? 00:00:00 \_ [ps] <defunct>
    dbadmin 4263 1 0 May18 ? 00:02:23 /opt/vertica/spread/sbin/spread -c /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/spread.conf -D /opt/vertica/spread/tmp
    dbadmin 4265 1 0 May18 ? 04:12:38 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5457 -P 5459 -Y ipv4
    dbadmin 4404 4265 0 May18 ? 00:02:32 \_ /opt/vertica/bin/vertica-udx-zygote 15 3 4265 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    dbadmin 4341 1 0 May18 ? 00:02:25 /opt/vertica/spread/sbin/spread -c /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/spread.conf -D /opt/vertica/spread/tmp
    dbadmin 4343 1 0 May18 ? 04:44:19 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5437 -P 5439 -Y ipv4
    dbadmin 5157 4343 0 May18 ? 00:02:31 \_ /opt/vertica/bin/vertica-udx-zygote 15 3 4343 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    dbadmin 30324 1 0 May18 ? 00:02:21 /opt/vertica/spread/sbin/spread -c /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/spread.conf -D /opt/vertica/spread/tmp
    dbadmin 30326 1 0 May18 ? 04:36:11 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5441 -P 5443 -Y ipv4
    dbadmin 32026 30326 0 May18 ? 00:02:31 \_ /opt/vertica/bin/vertica-udx-zygote 15 3 30326 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    dbadmin 14326 30326 0 01:13 ? 00:00:00 \_ ps -ef --forest
    dbadmin 12199 1 0 May18 ? 00:02:21 /opt/vertica/spread/sbin/spread -c /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/spread.conf -D /opt/vertica/spread/tmp
    dbadmin 12201 1 0 May18 ? 04:22:35 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5433 -P 4803 -Y ipv4
    dbadmin 14365 12201 0 May18 ? 00:02:33 \_ /opt/vertica/bin/vertica-udx-zygote 15 3 12201 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    dbadmin 14328 12201 0 01:13 ? 00:00:00 \_ ps -fLA
    dbadmin 18484 1 0 May18 ? 00:02:18 /opt/vertica/spread/sbin/spread -c /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/spread.conf -D /opt/vertica/spread/tmp
    dbadmin 18486 1 0 May18 ? 04:11:46 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5449 -P 5451 -Y ipv4
    dbadmin 21050 18486 0 May18 ? 00:02:31 \_ /opt/vertica/bin/vertica-udx-zygote 15 3 18486 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    root 6015 1 0 May22 ? 01:15:16 usr/sbin/httpd
    root 13508 1 0 01:13 ? 00:00:00 /sbin/plymouthd --mode=shutdown
    root 13538 1 0 01:13 ? 00:00:00 /bin/bash /etc/rc6.d/S01reboot start
    root 14291 13538 0 01:13 ? 00:00:00 \_ sleep 2
    ------------------------------------

    ------------------- ps -fLA -----------------
    UID PID PPID LWP C NLWP STIME TTY TIME CMD
    root 1 0 1 0 1 May18 ? 00:00:06 /sbin/init
    root 2 0 2 0 1 May18 ? 00:00:00 [kthreadd]
    root 3 2 3 0 1 May18 ? 00:00:34 [migration/0]
    root 4 2 4 0 1 May18 ? 00:00:32 [ksoftirqd/0]
    root 5 2 5 0 1 May18 ? 00:00:00 [stopper/0]
    root 6 2 6 0 1 May18 ? 00:00:01 [watchdog/0]
    root 7 2 7 0 1 May18 ? 00:00:28 [migration/1]
    root 8 2 8 0 1 May18 ? 00:00:00 [stopper/1]
    root 9 2 9 0 1 May18 ? 00:00:21 [ksoftirqd/1]
    root 10 2 10 0 1 May18 ? 00:00:01 [watchdog/1]
    root 11 2 11 0 1 May18 ? 00:00:28 [migration/2]
    root 12 2 12 0 1 May18 ? 00:00:00 [stopper/2]
    root 13 2 13 0 1 May18 ? 00:00:16 [ksoftirqd/2]
    root 14 2 14 0 1 May18 ? 00:00:01 [watchdog/2]
    root 15 2 15 0 1 May18 ? 00:00:24 [migration/3]
    root 16 2 16 0 1 May18 ? 00:00:00 [stopper/3]
    root 17 2 17 0 1 May18 ? 00:00:17 [ksoftirqd/3]
    root 18 2 18 0 1 May18 ? 00:00:01 [watchdog/3]
    root 19 2 19 0 1 May18 ? 00:00:27 [migration/4]
    root 20 2 20 0 1 May18 ? 00:00:00 [stopper/4]
    root 21 2 21 0 1 May18 ? 00:02:55 [ksoftirqd/4]
    root 22 2 22 0 1 May18 ? 00:00:01 [watchdog/4]
    root 23 2 23 0 1 May18 ? 00:00:25 [migration/5]
    root 24 2 24 0 1 May18 ? 00:00:00 [stopper/5]
    root 25 2 25 0 1 May18 ? 00:02:34 [ksoftirqd/5]
    root 26 2 26 0 1 May18 ? 00:00:01 [watchdog/5]
    root 27 2 27 0 1 May18 ? 00:00:32 [migration/6]
    root 28 2 28 0 1 May18 ? 00:00:00 [stopper/6]
    root 29 2 29 0 1 May18 ? 00:02:48 [ksoftirqd/6]
    root 30 2 30 0 1 May18 ? 00:00:01 [watchdog/6]
    root 31 2 31 0 1 May18 ? 00:00:25 [migration/7]
    root 32 2 32 0 1 May18 ? 00:00:00 [stopper/7]
    root 33 2 33 0 1 May18 ? 00:03:51 [ksoftirqd/7]
    root 34 2 34 0 1 May18 ? 00:00:01 [watchdog/7]
    root 35 2 35 0 1 May18 ? 00:01:49 [events/0]
    root 36 2 36 0 1 May18 ? 00:00:47 [events/1]
    root 37 2 37 0 1 May18 ? 00:00:45 [events/2]
    root 38 2 38 0 1 May18 ? 00:00:38 [events/3]
    root 39 2 39 0 1 May18 ? 00:01:16 [events/4]
    root 40 2 40 0 1 May18 ? 00:00:54 [events/5]
    root 41 2 41 0 1 May18 ? 00:00:41 [events/6]
    root 42 2 42 0 1 May18 ? 00:01:05 [events/7]
    root 43 2 43 0 1 May18 ? 00:00:00 [events/0]
    root 44 2 44 0 1 May18 ? 00:00:00 [events/1]
    root 45 2 45 0 1 May18 ? 00:00:00 [events/2]
    root 46 2 46 0 1 May18 ? 00:00:00 [events/3]
    root 47 2 47 0 1 May18 ? 00:00:00 [events/4]
    root 48 2 48 0 1 May18 ? 00:00:00 [events/5]
    root 49 2 49 0 1 May18 ? 00:00:00 [events/6]
    root 50 2 50 0 1 May18 ? 00:00:00 [events/7]
    root 51 2 51 0 1 May18 ? 00:00:00 [events_long/0]
    root 52 2 52 0 1 May18 ? 00:00:00 [events_long/1]
    root 53 2 53 0 1 May18 ? 00:00:00 [events_long/2]
    root 54 2 54 0 1 May18 ? 00:00:00 [events_long/3]
    root 55 2 55 0 1 May18 ? 00:00:00 [events_long/4]
    root 56 2 56 0 1 May18 ? 00:00:00 [events_long/5]
    root 57 2 57 0 1 May18 ? 00:00:00 [events_long/6]
    root 58 2 58 0 1 May18 ? 00:00:00 [events_long/7]
    root 59 2 59 0 1 May18 ? 00:00:00 [events_power_ef]
    root 60 2 60 0 1 May18 ? 00:00:00 [events_power_ef]
    root 61 2 61 0 1 May18 ? 00:00:00 [events_power_ef]
    root 62 2 62 0 1 May18 ? 00:00:00 [events_power_ef]
    root 63 2 63 0 1 May18 ? 00:00:00 [events_power_ef]
    root 64 2 64 0 1 May18 ? 00:00:00 [events_power_ef]
    root 65 2 65 0 1 May18 ? 00:00:00 [events_power_ef]
    root 66 2 66 0 1 May18 ? 00:00:00 [events_power_ef]
    root 67 2 67 0 1 May18 ? 00:00:00 [cgroup]
    root 68 2 68 0 1 May18 ? 00:00:00 [khelper]
    root 69 2 69 0 1 May18 ? 00:00:00 [netns]
    root 70 2 70 0 1 May18 ? 00:00:00 [async/mgr]
    root 71 2 71 0 1 May18 ? 00:00:00 [pm]
    root 72 2 72 0 1 May18 ? 00:00:06 [sync_supers]
    root 73 2 73 0 1 May18 ? 00:00:05 [bdi-default]
    root 74 2 74 0 1 May18 ? 00:00:00 [kintegrityd/0]
    root 75 2 75 0 1 May18 ? 00:00:00 [kintegrityd/1]
    root 76 2 76 0 1 May18 ? 00:00:00 [kintegrityd/2]
    root 77 2 77 0 1 May18 ? 00:00:00 [kintegrityd/3]
    root 78 2 78 0 1 May18 ? 00:00:00 [kintegrityd/4]
    root 79 2 79 0 1 May18 ? 00:00:00 [kintegrityd/5]
    root 80 2 80 0 1 May18 ? 00:00:00 [kintegrityd/6]
    root 81 2 81 0 1 May18 ? 00:00:00 [kintegrityd/7]
    root 82 2 82 0 1 May18 ? 00:00:01 [kblockd/0]
    root 83 2 83 0 1 May18 ? 00:00:00 [kblockd/1]
    root 84 2 84 0 1 May18 ? 00:00:01 [kblockd/2]
    root 85 2 85 0 1 May18 ? 00:00:00 [kblockd/3]
    root 86 2 86 0 1 May18 ? 00:00:00 [kblockd/4]
    root 87 2 87 0 1 May18 ? 00:00:00 [kblockd/5]
    root 88 2 88 0 1 May18 ? 00:00:03 [kblockd/6]
    root 89 2 89 0 1 May18 ? 00:00:00 [kblockd/7]
    root 90 2 90 0 1 May18 ? 00:00:00 [kacpid]
    root 91 2 91 0 1 May18 ? 00:00:00 [kacpi_notify]
    root 92 2 92 0 1 May18 ? 00:00:00 [kacpi_hotplug]
    root 93 2 93 0 1 May18 ? 00:00:00 [ata_aux]
    root 94 2 94 0 1 May18 ? 00:00:00 [ata_sff/0]
    root 95 2 95 0 1 May18 ? 00:00:00 [ata_sff/1]
    root 96 2 96 0 1 May18 ? 00:00:00 [ata_sff/2]
    root 97 2 97 0 1 May18 ? 00:00:00 [ata_sff/3]
    root 98 2 98 0 1 May18 ? 00:00:00 [ata_sff/4]
    root 99 2 99 0 1 May18 ? 00:00:00 [ata_sff/5]
    root 100 2 100 0 1 May18 ? 00:00:00 [ata_sff/6]
    root 101 2 101 0 1 May18 ? 00:00:00 [ata_sff/7]
    root 102 2 102 0 1 May18 ? 00:00:00 [ksuspend_usbd]
    root 103 2 103 0 1 May18 ? 00:00:00 [khubd]
    root 104 2 104 0 1 May18 ? 00:00:00 [kseriod]
    root 105 2 105 0 1 May18 ? 00:00:00 [md/0]
    root 106 2 106 0 1 May18 ? 00:00:00 [md/1]
    root 107 2 107 0 1 May18 ? 00:00:00 [md/2]
    root 108 2 108 0 1 May18 ? 00:00:00 [md/3]
    root 109 2 109 0 1 May18 ? 00:00:00 [md/4]
    root 110 2 110 0 1 May18 ? 00:00:00 [md/5]
    root 111 2 111 0 1 May18 ? 00:00:00 [md/6]
    root 112 2 112 0 1 May18 ? 00:00:00 [md/7]
    root 113 2 113 0 1 May18 ? 00:00:00 [md_misc/0]
    root 114 2 114 0 1 May18 ? 00:00:00 [md_misc/1]
    root 115 2 115 0 1 May18 ? 00:00:00 [md_misc/2]
    root 116 2 116 0 1 May18 ? 00:00:00 [md_misc/3]
    root 117 2 117 0 1 May18 ? 00:00:00 [md_misc/4]
    root 118 2 118 0 1 May18 ? 00:00:00 [md_misc/5]
    root 119 2 119 0 1 May18 ? 00:00:00 [md_misc/6]
    root 120 2 120 0 1 May18 ? 00:00:00 [md_misc/7]
    root 121 2 121 0 1 May18 ? 00:00:00 [linkwatch]
    root 123 2 123 0 1 May18 ? 00:00:00 [khungtaskd]
    root 124 2 124 0 1 May18 ? 00:04:58 [kswapd0]
    root 125 2 125 0 1 May18 ? 00:00:00 [ksmd]
    root 126 2 126 0 1 May18 ? 00:01:21 [khugepaged]
    root 127 2 127 0 1 May18 ? 00:00:00 [aio/0]
    root 128 2 128 0 1 May18 ? 00:00:00 [aio/1]
    root 129 2 129 0 1 May18 ? 00:00:00 [aio/2]
    root 130 2 130 0 1 May18 ? 00:00:00 [aio/3]
    root 131 2 131 0 1 May18 ? 00:00:00 [aio/4]
    root 132 2 132 0 1 May18 ? 00:00:00 [aio/5]
    root 133 2 133 0 1 May18 ? 00:00:00 [aio/6]
    root 134 2 134 0 1 May18 ? 00:00:00 [aio/7]
    root 135 2 135 0 1 May18 ? 00:00:00 [crypto/0]
    root 136 2 136 0 1 May18 ? 00:00:00 [crypto/1]
    root 137 2 137 0 1 May18 ? 00:00:00 [crypto/2]
    root 138 2 138 0 1 May18 ? 00:00:00 [crypto/3]
    root 139 2 139 0 1 May18 ? 00:00:00 [crypto/4]
    root 140 2 140 0 1 May18 ? 00:00:00 [crypto/5]
    root 141 2 141 0 1 May18 ? 00:00:00 [crypto/6]
    root 142 2 142 0 1 May18 ? 00:00:00 [crypto/7]
    root 149 2 149 0 1 May18 ? 00:00:00 [kthrotld/0]
    root 150 2 150 0 1 May18 ? 00:00:00 [kthrotld/1]
    root 151 2 151 0 1 May18 ? 00:00:00 [kthrotld/2]
    root 152 2 152 0 1 May18 ? 00:00:00 [kthrotld/3]
    root 153 2 153 0 1 May18 ? 00:00:00 [kthrotld/4]
    root 154 2 154 0 1 May18 ? 00:00:00 [kthrotld/5]
    root 155 2 155 0 1 May18 ? 00:00:00 [kthrotld/6]
    root 156 2 156 0 1 May18 ? 00:00:00 [kthrotld/7]
    root 158 2 158 0 1 May18 ? 00:00:00 [kpsmoused]
    root 159 2 159 0 1 May18 ? 00:00:00 [usbhid_resumer]
    root 160 2 160 0 1 May18 ? 00:00:00 [deferwq]
    root 192 2 192 0 1 May18 ? 00:00:00 [kdmremove]
    root 193 2 193 0 1 May18 ? 00:00:00 [kstriped]
    root 338 2 338 0 1 May18 ? 00:00:00 [scsi_eh_0]
    root 339 2 339 0 1 May18 ? 00:00:00 [scsi_eh_1]
    root 340 2 340 0 1 May18 ? 00:00:00 [scsi_eh_2]
    root 341 2 341 0 1 May18 ? 00:00:00 [scsi_eh_3]
    root 342 2 342 0 1 May18 ? 00:00:00 [scsi_eh_4]
    root 343 2 343 0 1 May18 ? 00:00:00 [scsi_eh_5]
    root 454 2 454 0 1 May18 ? 00:07:35 [jbd2/sda5-8]
    root 455 2 455 0 1 May18 ? 00:00:00 [ext4-dio-unwrit]
    root 748 2 748 0 1 May18 ? 00:13:32 [flush-8:0]
    root 1027 2 1027 0 1 May18 ? 00:00:00 [jbd2/sda1-8]
    root 1028 2 1028 0 1 May18 ? 00:00:00 [ext4-dio-unwrit]
    root 1029 2 1029 0 1 May18 ? 00:00:12 [jbd2/sda2-8]
    root 1030 2 1030 0 1 May18 ? 00:00:00 [ext4-dio-unwrit]
    root 1031 2 1031 0 1 May18 ? 00:00:15 [jbd2/sda3-8]
    root 1032 2 1032 0 1 May18 ? 00:00:00 [ext4-dio-unwrit]
    root 1112 2 1112 0 1 May18 ? 00:00:12 [kauditd]
    dbadmin 4182 1 4182 0 1 May18 ? 00:02:19 /opt/vertica/spread/sbin/spread -c /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/spread.conf -D /opt/vertica/spread/tmp
    dbadmin 4184 1 4184 0 10 May18 ? 00:00:09 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5445 -P 5447 -Y ipv4
    dbadmin 4184 1 4382 0 10 May18 ? 01:18:34 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5445 -P 5447 -Y ipv4
    dbadmin 4184 1 4383 0 10 May18 ? 02:22:24 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5445 -P 5447 -Y ipv4
    dbadmin 4184 1 4387 0 10 May18 ? 00:00:00 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5445 -P 5447 -Y ipv4
    dbadmin 4184 1 4388 0 10 May18 ? 00:00:28 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5445 -P 5447 -Y ipv4
    dbadmin 4184 1 4390 0 10 May18 ? 00:00:11 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5445 -P 5447 -Y ipv4
    dbadmin 4184 1 4409 0 10 May18 ? 00:06:09 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5445 -P 5447 -Y ipv4
    dbadmin 4184 1 4410 0 10 May18 ? 00:00:02 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5445 -P 5447 -Y ipv4
    dbadmin 4184 1 4411 0 10 May18 ? 00:01:36 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5445 -P 5447 -Y ipv4
    dbadmin 4184 1 4432 0 10 May18 ? 00:00:24 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5445 -P 5447 -Y ipv4
    dbadmin 4263 1 4263 0 1 May18 ? 00:02:23 /opt/vertica/spread/sbin/spread -c /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/spread.conf -D /opt/vertica/spread/tmp
    dbadmin 4265 1 4265 0 10 May18 ? 00:00:08 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5457 -P 5459 -Y ipv4
    dbadmin 4265 1 4402 0 10 May18 ? 01:18:39 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5457 -P 5459 -Y ipv4
    dbadmin 4265 1 4403 0 10 May18 ? 02:22:33 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5457 -P 5459 -Y ipv4
    dbadmin 4265 1 4407 0 10 May18 ? 00:00:00 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5457 -P 5459 -Y ipv4
    dbadmin 4265 1 4408 0 10 May18 ? 00:00:28 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5457 -P 5459 -Y ipv4
    dbadmin 4265 1 4413 0 10 May18 ? 00:00:11 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5457 -P 5459 -Y ipv4
    dbadmin 4265 1 4468 0 10 May18 ? 00:04:38 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5457 -P 5459 -Y ipv4
    dbadmin 4265 1 4469 0 10 May18 ? 00:00:02 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5457 -P 5459 -Y ipv4
    dbadmin 4265 1 4470 0 10 May18 ? 00:01:38 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5457 -P 5459 -Y ipv4
    dbadmin 4265 1 4502 0 10 May18 ? 00:00:24 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5457 -P 5459 -Y ipv4
    dbadmin 4341 1 4341 0 1 May18 ? 00:02:25 /opt/vertica/spread/sbin/spread -c /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/spread.conf -D /opt/vertica/spread/tmp
    dbadmin 4343 1 4343 0 14 May18 ? 00:00:11 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5437 -P 5439 -Y ipv4
    dbadmin 4343 1 5155 0 14 May18 ? 01:21:54 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5437 -P 5439 -Y ipv4
    dbadmin 4343 1 5156 0 14 May18 ? 02:22:47 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5437 -P 5439 -Y ipv4
    dbadmin 4343 1 5160 0 14 May18 ? 00:00:00 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5437 -P 5439 -Y ipv4
    dbadmin 4343 1 5161 0 14 May18 ? 00:00:27 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5437 -P 5439 -Y ipv4
    dbadmin 4343 1 5162 0 14 May18 ? 00:00:12 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5437 -P 5439 -Y ipv4
    dbadmin 4343 1 5241 0 14 May18 ? 00:04:16 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5437 -P 5439 -Y ipv4
    dbadmin 4343 1 5242 0 14 May18 ? 00:00:05 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5437 -P 5439 -Y ipv4
    dbadmin 4343 1 5243 0 14 May18 ? 00:01:45 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5437 -P 5439 -Y ipv4
    dbadmin 4343 1 5281 0 14 May18 ? 00:00:24 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5437 -P 5439 -Y ipv4
    dbadmin 4343 1 32281 0 14 Jun07 ? 00:00:03 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5437 -P 5439 -Y ipv4
    dbadmin 4343 1 8176 0 14 Jun07 ? 00:00:00 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5437 -P 5439 -Y ipv4
    dbadmin 4343 1 8363 0 14 Jun07 ? 00:00:06 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5437 -P 5439 -Y ipv4
    dbadmin 4343 1 15719 0 14 Jun07 ? 00:00:02 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5437 -P 5439 -Y ipv4
    dbadmin 4384 4184 4384 0 3 May18 ? 00:00:00 /opt/vertica/bin/vertica-udx-zygote 15 3 4184 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    dbadmin 4384 4184 4385 0 3 May18 ? 00:02:24 /opt/vertica/bin/vertica-udx-zygote 15 3 4184 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    dbadmin 4384 4184 4386 0 3 May18 ? 00:00:07 /opt/vertica/bin/vertica-udx-zygote 15 3 4184 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    dbadmin 4404 4265 4404 0 3 May18 ? 00:00:00 /opt/vertica/bin/vertica-udx-zygote 15 3 4265 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    dbadmin 4404 4265 4405 0 3 May18 ? 00:02:24 /opt/vertica/bin/vertica-udx-zygote 15 3 4265 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    dbadmin 4404 4265 4406 0 3 May18 ? 00:00:07 /opt/vertica/bin/vertica-udx-zygote 15 3 4265 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    dbadmin 5157 4343 5157 0 3 May18 ? 00:00:00 /opt/vertica/bin/vertica-udx-zygote 15 3 4343 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    dbadmin 5157 4343 5158 0 3 May18 ? 00:02:23 /opt/vertica/bin/vertica-udx-zygote 15 3 4343 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    dbadmin 5157 4343 5159 0 3 May18 ? 00:00:07 /opt/vertica/bin/vertica-udx-zygote 15 3 4343 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    root 6015 1 6015 0 1 May22 ? 01:15:16 usr/sbin/httpd
    root 6449 2 6449 0 1 May19 ? 00:00:00 [bluetooth]
    dbadmin 12199 1 12199 0 1 May18 ? 00:02:21 /opt/vertica/spread/sbin/spread -c /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/spread.conf -D /opt/vertica/spread/tmp
    dbadmin 12201 1 12201 0 10 May18 ? 00:00:10 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5433 -P 4803 -Y ipv4
    dbadmin 12201 1 14363 0 10 May18 ? 01:18:18 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5433 -P 4803 -Y ipv4
    dbadmin 12201 1 14364 0 10 May18 ? 02:21:53 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5433 -P 4803 -Y ipv4
    dbadmin 12201 1 14368 0 10 May18 ? 00:00:00 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5433 -P 4803 -Y ipv4
    dbadmin 12201 1 14369 0 10 May18 ? 00:00:27 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5433 -P 4803 -Y ipv4
    dbadmin 12201 1 14370 0 10 May18 ? 00:00:11 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5433 -P 4803 -Y ipv4
    dbadmin 12201 1 14441 0 10 May18 ? 00:08:31 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5433 -P 4803 -Y ipv4
    dbadmin 12201 1 14442 0 10 May18 ? 00:00:03 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5433 -P 4803 -Y ipv4
    dbadmin 12201 1 14443 0 10 May18 ? 00:01:39 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5433 -P 4803 -Y ipv4
    dbadmin 12201 1 14491 0 10 May18 ? 00:00:24 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5433 -P 4803 -Y ipv4
    root 13508 1 13508 0 1 01:13 ? 00:00:00 /sbin/plymouthd --mode=shutdown
    root 13538 1 13538 0 1 01:13 ? 00:00:00 /bin/bash /etc/rc6.d/S01reboot start
    root 14291 13538 14291 0 1 01:13 ? 00:00:00 sleep 2
    dbadmin 14330 18486 14330 3 1 01:13 ? 00:00:00 [ps] <defunct>
    dbadmin 14333 30326 14333 6 1 01:13 ? 00:00:00 ps -fLA
    dbadmin 14365 12201 14365 0 3 May18 ? 00:00:00 /opt/vertica/bin/vertica-udx-zygote 15 3 12201 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    dbadmin 14365 12201 14366 0 3 May18 ? 00:02:25 /opt/vertica/bin/vertica-udx-zygote 15 3 12201 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    dbadmin 14365 12201 14367 0 3 May18 ? 00:00:07 /opt/vertica/bin/vertica-udx-zygote 15 3 12201 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    dbadmin 18484 1 18484 0 1 May18 ? 00:02:18 /opt/vertica/spread/sbin/spread -c /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/spread.conf -D /opt/vertica/spread/tmp
    dbadmin 18486 1 18486 0 10 May18 ? 00:00:08 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5449 -P 5451 -Y ipv4
    dbadmin 18486 1 21048 0 10 May18 ? 01:18:20 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5449 -P 5451 -Y ipv4
    dbadmin 18486 1 21049 0 10 May18 ? 02:21:46 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5449 -P 5451 -Y ipv4
    dbadmin 18486 1 21053 0 10 May18 ? 00:00:00 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5449 -P 5451 -Y ipv4
    dbadmin 18486 1 21054 0 10 May18 ? 00:00:28 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5449 -P 5451 -Y ipv4
    dbadmin 18486 1 21055 0 10 May18 ? 00:00:11 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5449 -P 5451 -Y ipv4
    dbadmin 18486 1 21130 0 10 May18 ? 00:05:52 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5449 -P 5451 -Y ipv4
    dbadmin 18486 1 21131 0 10 May18 ? 00:00:02 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5449 -P 5451 -Y ipv4
    dbadmin 18486 1 21132 0 10 May18 ? 00:01:35 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5449 -P 5451 -Y ipv4
    dbadmin 18486 1 21191 0 10 May18 ? 00:00:24 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5449 -P 5451 -Y ipv4
    dbadmin 21050 18486 21050 0 3 May18 ? 00:00:00 /opt/vertica/bin/vertica-udx-zygote 15 3 18486 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    dbadmin 21050 18486 21051 0 3 May18 ? 00:02:23 /opt/vertica/bin/vertica-udx-zygote 15 3 18486 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    dbadmin 21050 18486 21052 0 3 May18 ? 00:00:07 /opt/vertica/bin/vertica-udx-zygote 15 3 18486 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    dbadmin 30324 1 30324 0 1 May18 ? 00:02:21 /opt/vertica/spread/sbin/spread -c /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/spread.conf -D /opt/vertica/spread/tmp
    dbadmin 30326 1 30326 0 10 May18 ? 00:00:11 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5441 -P 5443 -Y ipv4
    dbadmin 30326 1 32024 0 10 May18 ? 01:19:30 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5441 -P 5443 -Y ipv4
    dbadmin 30326 1 32025 0 10 May18 ? 02:21:49 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5441 -P 5443 -Y ipv4
    dbadmin 30326 1 32029 0 10 May18 ? 00:00:00 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5441 -P 5443 -Y ipv4
    dbadmin 30326 1 32030 0 10 May18 ? 00:00:27 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5441 -P 5443 -Y ipv4
    dbadmin 30326 1 32031 0 10 May18 ? 00:00:12 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5441 -P 5443 -Y ipv4
    dbadmin 30326 1 32203 0 10 May18 ? 00:06:03 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5441 -P 5443 -Y ipv4
    dbadmin 30326 1 32204 0 10 May18 ? 00:00:03 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5441 -P 5443 -Y ipv4
    dbadmin 30326 1 32205 0 10 May18 ? 00:01:39 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5441 -P 5443 -Y ipv4
    dbadmin 30326 1 32254 0 10 May18 ? 00:00:24 /opt/vertica/bin/vertica -D /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog -C DatabaseName -n v_DatabaseName_node0001 -h 127.0.0.1 -p 5441 -P 5443 -Y ipv4
    dbadmin 32026 30326 32026 0 3 May18 ? 00:00:00 /opt/vertica/bin/vertica-udx-zygote 15 3 30326 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    dbadmin 32026 30326 32027 0 3 May18 ? 00:02:23 /opt/vertica/bin/vertica-udx-zygote 15 3 30326 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    dbadmin 32026 30326 32028 0 3 May18 ? 00:00:07 /opt/vertica/bin/vertica-udx-zygote 15 3 30326 debug-log-off /home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog/UDxLogs 60 16
    ------------------------------------


    2016-06-08 01:13:23.374 Main:0x7c87ab0 [Main] <INFO> Handling signal: 15

     

    Thanks

    Rahul

  • Hi Rahul,

     

    The server had been going down for quite a few times in May. But the databases were coming up when started after server reboot. But Yesterday again the server went down and it was started manually, since then we are unable to up the databases. When we try to start the database from adminTools, after selecting database and entering the password for the database we get the

    *** Starting database : database name ***

    display and the process comes back the adminTools in a flash.

    I have attached the adminTools.log and

    latest vertica.log file I have shared in the below link as the size is more: -

     

    https://www.sendspace.com/file/b34uge

  • Hi Hari

     

    Its a tricky situation where you catalog is not getting initialized and since you don't have k-safety feature set (with no extra nodes) up which can help in performing in such scenerios.

     

    I am afraid you can bring this db up without that.

     

    I will suggest you kindly raise a vertica support case if you have a valid account for the same.Else try restoring from backup or create a new db.

     

    snippet of admintools-dbadmin.log

    =============================

    2016-06-08 13:39:09.972 admintools/2431:0x7fd50697f700 [adminExec.display] <INFO> Unable to read database catalogs - cannot start database.
    2016-06-08 13:39:13.132 admintools/2431:0x7fd50697f700 [adminExec.removeRunningDbs] <INFO> items '[('DatabaseName', 'DatabaseName', 0), ('DatabaseName', 'DatabaseName', 0), ('DatabaseName', 'DatabaseName', 0), ('DatabaseName', 'DatabaseName', 0), ('DatabaseName', 'DatabaseName', 0), ('DatabaseName', 'DatabaseName', 0)]' runningDBs '[]'
    2016-06-08 13:39:13.133 admintools/2431:0x7fd50697f700 [adminController.dbStart] <INFO> candidates to start (no active) '[('DatabaseName', 'DatabaseName', 0), ('DatabaseName', 'DatabaseName', 0), ('DatabaseName', 'DatabaseName', 0), ('DatabaseName', 'DatabaseName', 0), ('DatabaseName', 'DatabaseName', 0), ('DatabaseName', 'DatabaseName', 0)]'
    2016-06-08 13:39:18.336 admintools/2431:0x7fd50697f700 [Passwords.getpass] <INFO> get password '['DatabaseName']'
    2016-06-08 13:39:23.262 admintools/2431:0x7fd4fea2a700 [BashAdapter._exec_module_impl] <INFO> Running command ['/opt/vertica/oss/python/bin/python', '-m', 'vertica.engine.api.at_runner', '--no-prompt', '--module', 'vertica.engine.api.compute_vdatabase']#{"catalogpath": "/home/dbadmin/DatabaseName/v_DatabaseName_node0001_catalog"}
    2016-06-08 13:39:23.814 at_exec/2510:0x7fc83568b700 [root.setup_custom_logging] <INFO> New log for 'at_exec'
    2016-06-08 13:39:23.815 at_exec/2510:0x7fc83568b700 [root.setup_custom_logging] <INFO> sys.argv: '/opt/vertica/share/eggs/vertica/engine/api/at_runner.py' --no-prompt --module 'vertica.engine.api.compute_vdatabase'
    2016-06-08 13:39:23.880 admintools/2431:0x7fd50697f700 [adminExec.display] <INFO> Unable to read database catalogs - cannot start database.
    2016-06-08 13:39:47.310 admintools/2431:0x7fd50697f700 [adminExec.checkLicenseNodeCount] <WARNING> License file /opt/vertica/config/share/license.key was not valid, returncode -11, message:
    None


    2016-06-08 14:35:54.346 admintools/4763:0x7fa8f0fb4700 [adminExec.getLastGoodEpoch] <ERROR> failed to find majority of nodes (1) reporting the same ASR epoch (largest was epoch 0, count 0).
    2016-06-08 14:35:54.347 admintools/4763:0x7fa8f0fb4700 [adminController.restartDBFromEpoch] <WARNING> Too few nodes are available to restart cluster
    2016-06-08 14:35:55.351 admintools/4763:0x7fa8f0fb4700 [adminController.restartDBFromEpoch] <WARNING> no good epoch found -- warning user
    2016-06-08 14:35:56.138 admintools/4763:0x7fa8f0fb4700 [adminController.restartDBFromEpoch] <INFO> found last epoch -1 corresponding to timestamp ''
    2016-06-08 14:36:45.232 admintools/5021:0x7f66496df700 [root.setup_custom_logging] <INFO> New log for 'admintools'
    2016-06-08 14:36:45.233 admintools/5021:0x7f66496df700 [root.setup_custom_logging] <INFO> sys.argv: /opt/vertica/bin/adminTools -t command_host -cstatus
    2016-06-08 14:36:45.751 admintools/5021:0x7f66496df700 [SshAuth.setup_from_admintools_conf_if_needed] <INFO> Setup complete, default ssh user is dbadmin, default ssh options are -oConnectTimeout=30 -o TCPKeepAlive=no -o ServerAliveInterval=15 -o ServerAliveCountMax=2 -o StrictHostKeyChecking=no -o BatchMode=yes.
    2016-06-08 14:36:45.752 admintools/5021:0x7f66496df700 [adminExec.__init__] <INFO> Unique Ports: False
    2016-06-08 14:36:45.752 admintools/5021:0x7f66496df700 [adminExec._is_current_user_conf_owner] <INFO> Owner of admintools.conf (dbauser) is dbadmin.
    2016-06-08 14:36:45.754 admintools/5021:0x7f66496df700 [adminExec.isOkToRun] <INFO>
    2016-06-08 14:36:45.754 admintools/5021:0x7f66496df700 [__main__.<module>] <INFO> Use unique ports = False
    2016-06-08 14:36:45.754 admintools/5021:0x7f66496df700 [adminExec._is_current_user_conf_owner] <INFO> Owner of admintools.conf (dbauser) is dbadmin.
    2016-06-08 14:36:45.755 admintools/5021:0x7f66496df700 [adminExec.isOkToRun] <INFO>
    2016-06-08 14:36:46.208 admintools/5041:0x7fd92a18e700 [root.setup_custom_logging] <INFO> New log for 'admintools'
    2016-06-08 14:36:46.209 admintools/5041:0x7fd92a18e700 [root.setup_custom_logging] <INFO> sys.argv: '/opt/vertica/bin/TXTlcheck.py' /opt/vertica/config /opt/vertica/bin
    2016-06-08 14:36:46.213 admintools/5021:0x7f66496df700 [commandLineCtrl.commandHost] <INFO> processing status
    2016-06-08 14:36:46.311 admintools/5021:0x7f66496df700 [commandLineCtrl.commandHost] <INFO> this host's DBs: {'DatabaseName': {'host': '127.0.0.1', 'state': 'DOWN', 'db': 'DatabaseName', 'name': 'v_DatabaseName_node0001'}, 'DatabaseName': {'host': '127.0.0.1', 'state': 'DOWN', 'db': 'DatabaseName', 'name': 'v_DatabaseName_node0001'}, 'DatabaseName': {'host': '127.0.0.1', 'state': 'DOWN', 'db': 'DatabaseName', 'name': 'v_DatabaseName_node0001'}, 'DatabaseName': {'host': '127.0.0.1', 'state': 'DOWN', 'db': 'DatabaseName', 'name': 'v_DatabaseName_node0001'}, 'DatabaseName': {'host': '127.0.0.1', 'state': 'DOWN', 'db': 'DatabaseName', 'name': 'v_DatabaseName_node0001'}, 'DatabaseName': {'host': '127.0.0.1', 'state': 'DOWN', 'db': 'DatabaseName', 'name': 'v_DatabaseName_node0001'}}

     

     

    Thanks

    Rahul

  • Oh ok. So there is no other way by which we can bring up the database. Is there a certain reason why this has happened? it was working fine until the server went down and came up.

     

  • Even if we have backup files, we will have to create new databases to dump the backup files, since now the old databases are like corrupted, right?

  • You can straight away restore the database using the restore process (task restore used with vbr.py) if you have a valid backup.

     

    Please refer documentation for the same.

     

    Thanks

    Rahul

  • Hi ,

     did you change any of the OS values ? 

    Like memory increse or other stuff like that in between reboots ? 

     

  • Hi Rahul,

     

    I have a valid backup but the we are unable to start the vertica service at all. When we check the 

     

    service verticad status

     

    we get 

     

    Status not OK

     

    and even if we try to start the service, we get

     

    start not OK                [FAILED]

     

    Also, we are unable to create any database. When I try to create a new database in adminTools, I get 

     

    *** Creating database: testing ***

    Press RETURN to continue

     

    then I get an error:- 

     

    Installation error:

    Host=127.0.0.1 version=NO VERSION FOUND.

     

  • Check the file system ownership on the server !! make sure dbadmin and verticadba group have full access.For catalog path and data paths 

  • Hi Adrian,

     

    I'm not aware of such changes in the OS yet. This server is situated at a remote location and the server management guys say they had just restarted the server. Let me check with the server guys and come back.

  • Ok. Let me check those too. 

    I got a reply saying they just rebooted without any changes. 

     

    Now can we re-install vertica on top the existing vertica setup?

  • We have installed vertica after complete uninstallation. Installation happened without any error but when we give the license in adminTools which we have to during the first time, we get and error

     

    "Error: License key /opt/vertica/config/licensing/vertica_comminuty_edition.license.key not installed. Invalid license status " .

     

    What could be the problem here?

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file