installer fails with this error: vertica.local_coerce returncode = 255

Sunny_1

in the process of upgrade from 6.0.2 to vertica-7.0.1-2 and install fails 

/opt/vertica/sbin/update_vertica --failure-threshold NONE --rpm /opt/CA/IMDataRepository_vertica7/resources/vertica-7.0.1-2.x86_64.RHEL5.rpm

>> Validating node and cluster prerequisites...
Error: Command failed on 'ipofserver':
        VERT_DBA_USR=dbadmin VERT_DBA_HOME=/home/dbadmin VERT_DBA_GRP=verticadba /opt/vertica/oss/python/bin/python -m vertica.local_coerce
        returncode = 255
Error: Command failed on 'ipofserver':
        VERT_DBA_USR=dbadmin VERT_DBA_HOME=/home/dbadmin VERT_DBA_GRP=verticadba /opt/vertica/oss/python/bin/python -m vertica.local_coerce
        returncode = 255
Error: Command failed on 'ipofserver':
        VERT_DBA_USR=dbadmin VERT_DBA_HOME=/home/dbadmin VERT_DBA_GRP=verticadba /opt/vertica/oss/python/bin/python -m vertica.local_coerce
        returncode = 255

Adrian_Oprea_1

Remove the --failure-threshold NONE  and run again ! this might give you an idea of what might be .
Also it looks like the SSH was not done correctly between hosts take a look into that. 

Sunny_1

removing --failure-threshold NONE
>> Validating node and cluster prerequisites...
Error: Command failed on 'ipofserver':
        VERT_DBA_USR=dbadmin VERT_DBA_HOME=/home/dbadmin VERT_DBA_GRP=verticadba /opt/vertica/oss/python/bin/python -m vertica.local_coerce
        returncode = 255
Error: Command failed on 'ipofserver1':
        VERT_DBA_USR=dbadmin VERT_DBA_HOME=/home/dbadmin VERT_DBA_GRP=verticadba /opt/vertica/oss/python/bin/python -m vertica.local_coerce
        returncode = 255
Error: Command failed on 'ipofserver':
        VERT_DBA_USR=dbadmin VERT_DBA_HOME=/home/dbadmin VERT_DBA_GRP=verticadba /opt/vertica/oss/python/bin/python -m vertica.local_coerce
        returncode = 255
System prerequisites failed.  Threshold = WARN
        Hint: Fix above failures or use --failure-threshold

Installation FAILED with errors.

****
AdminTools and your existing Vertica databases may be unavailable.
Investigate the above warnings/errors and re-run installation.

Adrian_Oprea_1

any vertica log data ? 

Sunny_1

strace of the error:
clone(child_stack=0x2ab5ebf96250, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x2ab5ebf969d0, tls=0x2ab5ebf96940, child_tidptr=0x2ab5ebf969d0) = 3953futex(0x19d59ec0, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x1994dab0, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x1994dab0, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
clone(child_stack=0x2ab5ec997250, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x2ab5ec9979d0, tls=0x2ab5ec997940, child_tidptr=0x2ab5ec9979d0) = 3954
futex(0x19dc4db0, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x1994dab0, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
clone(child_stack=0x2ab5eb595250, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x2ab5eb5959d0, tls=0x2ab5eb595940, child_tidptr=0x2ab5eb5959d0) = 3956
futex(0x19de04b0, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x1994dab0, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x1994dab0, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x1994dab0, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1994dab0, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1994dab0, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1994dab0, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1994dab0, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1994dab0, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1a14e330, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
write(1, "Error: Command failed on '167.14"..., 44Error: Command failed on '1.1.1.125':
) = 44
write(1, "\tVERT_DBA_USR=dbadmin VERT_DBA_H"..., 133    VERT_DBA_USR=dbadmin VERT_DBA_HOME=/home/dbadmin VERT_DBA_GRP=verticadba /opt/vertica/oss/python/bin/python -m vertica.local_coerce
) = 133
write(1, "\treturncode = 255\n", 18     returncode = 255
)    = 18
write(1, "Error: Command failed on '167.14"..., 44Error: Command failed on '1.1.1.191':
) = 44
write(1, "\tVERT_DBA_USR=dbadmin VERT_DBA_H"..., 133    VERT_DBA_USR=dbadmin VERT_DBA_HOME=/home/dbadmin VERT_DBA_GRP=verticadba /opt/vertica/oss/python/bin/python -m vertica.local_coerce
) = 133
write(1, "\treturncode = 255\n", 18     returncode = 255
)    = 18
write(1, "Error: Command failed on '167.14"..., 44Error: Command failed on '1.1.1198':
) = 44
write(1, "\tVERT_DBA_USR=dbadmin VERT_DBA_H"..., 133    VERT_DBA_USR=dbadmin VERT_DBA_HOME=/home/dbadmin VERT_DBA_GRP=verticadba /opt/vertica/oss/python/bin/python -m vertica.local_coerce
) = 133
write(1, "\treturncode = 255\n", 18     returncode = 255
)    = 18
write(1, "System prerequisites failed.  Th"..., 47System prerequisites failed.  Threshold = WARN
) = 47
write(1, "\tHint: Fix above failures or use"..., 53     Hint: Fix above failures or use --failure-threshold
) = 53
write(1, "\n", 1
)                       = 1
write(1, "Installation FAILED with errors."..., 33Installation FAILED with errors.
) = 33
write(1, "\n", 1
)                       = 1
write(1, "****\n", 5****
)                   = 5
write(1, "AdminTools and your existing Ver"..., 67AdminTools and your existing Vertica databases may be unavailable.
) = 67
write(1, "Investigate the above warnings/e"..., 63Investigate the above warnings/errors and re-run installation.
) = 63
write(1, "****\n", 5****

Sunny_1

20140604 093142:DEBUG:root:Logging started.  Logging to: ['/opt/vertica/log/install.log']20140604 093142:INFO:root:------------------------------------------------------------
20140604 093142:INFO:root:Begin create_dba
20140604 093142:INFO:root:------------------------------------------------------------
20140604 093142:INFO:vertica.platform.node.dba.DbaCreate:DbaCreate initialized: <UserGroup user='dbadmin' group='verticadba' home='/home/dbadmin'>
20140604 093142:INFO:vertica.platform.node.dba.DbaCreate:step: Provided DB Admin account details:
20140604 093142:INFO:vertica.platform.node.dba.DbaCreate:progress: user = dbadmin
20140604 093142:INFO:vertica.platform.node.dba.DbaCreate:progress: group = verticadba
20140604 093142:INFO:vertica.platform.node.dba.DbaCreate:progress: home = /home/dbadmin
20140604 093142:INFO:vertica.platform.node.dba.DbaCreate:step: Creating group...
20140604 093142:INFO:vertica.platform.node.dba.DbaCreate:progress: Group already exists
20140604 093142:INFO:vertica.platform.node.dba.DbaCreate:step: Validating group...
20140604 093142:INFO:vertica.platform.node.dba.DbaCreate:progress: Okay
20140604 093142:INFO:vertica.platform.node.dba.DbaCreate:step: Creating user...
20140604 093142:INFO:vertica.platform.node.dba.DbaCreate:progress: User already exists
20140604 093142:INFO:vertica.platform.node.dba.DbaCreate:step: Validating user...
20140604 093142:DEBUG:vertica.platform.node.dba.DbaCreate:Hidden from user, running /opt/vertica/share/binlib/test/file-access -u dbadmin /home/dbadmin
20140604 093142:DEBUG:vertica.platform.node.dba.DbaCreate:Output of test/file-access: '\x1b[H\x1b[2J'
20140604 093142:INFO:vertica.platform.node.dba.DbaCreate:progress: Okay
20140604 093142:DEBUG:root:Logging started.  Logging to: ['/opt/vertica/log/install.log']
20140604 093142:DEBUGlithium_main__.LocalCoerce:Preparing output directory /opt/vertica/log/local-coerce-20140604_093142
20140604 093142:DEBUGlithium_main__.LocalCoerce:Parent of output directory exists. Creating dir.
20140604 093142:DEBUGlithium_main__.LocalCoerce:Output directory created
20140604 093142:DEBUG:vertica.platform.node.t.cpu.CheckNodeCpu:ondemand governor nice policy unknown
20140604 093142:DEBUG:vertica.platform.node.t.cpu.CheckNodeCpu:cpufreq scaling not available for cpu0
20140604 093142:DEBUG:vertica.platform.node.t.cpu.CheckNodeCpu:cpufreq scaling not available for cpu1
20140604 093142:DEBUG:vertica.platform.node.t.cpu.CheckNodeCpu:Found 2 processors and 3912644 kb memory
20140604 093142:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:Disks:
20140604 093142:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/sda1', disk='sda', readahead='256', schedule='cfq', fs='ext3', blocksz='4096')
20140604 093142:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-home', disk='dm-1', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 093142:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-root', disk='dm-0', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 093142:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:Disks:
20140604 093142:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/sda1', disk='sda', readahead='256', schedule='cfq', fs='ext3', blocksz='4096')
20140604 093142:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-home', disk='dm-1', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 093142:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-root', disk='dm-0', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 093143:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:lvscan says: "  ACTIVE            '/dev/system/root' [33.22 GB] inherit\n  ACTIVE            '/dev/system/home' [256.00 MB] inherit\n  ACTIVE            '/dev/system/swap' [6.00 GB] inherit\n"
20140604 093143:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:Disks:
20140604 093143:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/sda1', disk='sda', readahead='256', schedule='cfq', fs='ext3', blocksz='4096')
20140604 093143:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-home', disk='dm-1', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 093143:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-root', disk='dm-0', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 093143:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:Disks:
20140604 093143:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/sda1', disk='sda', readahead='256', schedule='cfq', fs='ext3', blocksz='4096')
20140604 093143:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-home', disk='dm-1', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 093143:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-root', disk='dm-0', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 093143:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:Disks:
20140604 093143:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/sda1', disk='sda', readahead='256', schedule='cfq', fs='ext3', blocksz='4096')
20140604 093143:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-home', disk='dm-1', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 093143:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-root', disk='dm-0', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 093143:INFO:vertica.platform.node.t.disk.CheckNodeDisk:Found meminfo line 12: 'SwapTotal:     6291448 kB'
20140604 093143:DEBUG:vertica.platform.node.t.dbadmin.CheckDbAdmin:dbadmin home owned by 'dbadmin'
20140604 093143:DEBUG:vertica.platform.node.t.dbadmin.CheckDbAdmin:dbadmin home owned by group 'dbadmin'
20140604 093143:INFO:vertica.platform.node.t.dbadmin.CheckDbAdmin:LANG = 'en_US.UTF-8\n\x1b[H\x1b[2J'
20140604 093143:INFO:vertica.platform.node.t.dbadmin.CheckDbAdmin:TZ = '\x1b[H\x1b[2J'
20140604 093143:DEBUG:vertica.platform.node.t.limits.CheckNodeLimits:Found line 14: 'session required pam_limits.so'
20140604 093143:INFO:vertica.platform.node.t.misc.CheckNodeMisc:NTPD is running as pid 3251
20140604 093143:INFO:vertica.platform.node.t.misc.CheckNodeMisc:Found /etc/redhat-release
20140604 093143:INFO:vertica.platform.node.t.misc.CheckNodeMisc:release string = 'Red Hat Enterprise Linux Server release 5.10 (Tikanga)'
20140604 093143:INFO:vertica.platform.node.t.misc.CheckNodeMisc:Found sestatus: /usr/sbin/sestatus
20140604 093144:INFO:vertica.platform.node.t.misc.CheckNodeMisc:sestatus output: 'SELinux status:                 disabled\n'
20140604 093144:INFO:vertica.platform.node.t.misc.CheckNodeMisc:transparent_hugepage feature unavailable.
20140604 093144:INFO:vertica.platform.node.t.network.CheckNodeNetwork:Found iptables: /sbin/iptables
20140604 093144:INFO:vertica.platform.node.t.network.CheckNodeNetwork:Did not find ip_tables_names. No tables defined.
20140604 093144:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No tcp6 usage on this machine
20140604 093144:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No udp6 usage on this machine
20140604 093144:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No tcp6 usage on this machine
20140604 093144:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No tcp6 usage on this machine
20140604 093144:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No tcp6 usage on this machine
20140604 093144:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No tcp6 usage on this machine
20140604 093144:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No udp6 usage on this machine
20140604 093144:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No udp6 usage on this machine
20140604 093144:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No udp6 usage on this machine
20140604 093144:INFO:vertica.platform.node.t.network.CheckNodeNetwork:Did not find SuSEfirewall2
20140604 093144:INFO:vertica.platform.node.t.sysctl.CheckNodeSysctl:Read from fs.file-max: '390819'
20140604 093144:INFO:vertica.platform.node.t.sysctl.CheckNodeSysctl:Read from vm.max_map_count: '244540'
20140604 093144:INFO:vertica.platform.node.t.sysctl.CheckNodeSysctl:Read from vm.min_free_kbytes: '8115'
20140604 093144:INFO:vertica.platform.node.t.tool.CheckNodeTool:Found mcelog as /usr/sbin/mcelog

20140604 093144:INFO:vertica.platform.node.t.tool.CheckNodeTool:Found pstack or gstack as /usr/bin/gstack

20140604 093144:INFO:vertica.platform.node.t.tool.CheckNodeTool:Found sysstat as /usr/bin/mpstat

20140604 093144:INFO:vertica.platform.node.t.tool.CheckNodeTool:Found sysstat as /usr/bin/iostat

20140604 093144:INFO:vertica.local_verify.LocalVerify:Creating symlink from /opt/vertica/log/verify-latest.xml to local-coerce-20140604_093142/iter01/verify.xml.
20140604 093144:INFO:vertica.local_verify.LocalVerify:Creating symlink from /opt/vertica/log/local-coerce-20140604_093142/verify-latest.xml to iter01/verify.xml.
20140604 093144:DEBUG:vertica.platform.fix.context.FixContext:Preparing staging directory /opt/vertica/log/local-coerce-20140604_093142/iter01/configure
20140604 093144:DEBUG:vertica.platform.fix.context.FixContext:Real location of staging directory: /opt/vertica/log/local-coerce-20140604_093142/iter01/configure
20140604 093144:DEBUG:vertica.platform.fix.context.FixContext:Staging directory does not exist. Creating.
20140604 093144:DEBUG:vertica.platform.fix.context.FixContext:Creating backup, new, and temporary directories
20140604 102333:DEBUG:root:Logging started.  Logging to: ['/opt/vertica/log/install.log']
20140604 102333:INFO:root:------------------------------------------------------------
20140604 102333:INFO:root:Begin create_dba
20140604 102333:INFO:root:------------------------------------------------------------
20140604 102333:INFO:vertica.platform.node.dba.DbaCreate:DbaCreate initialized: <UserGroup user='dbadmin' group='verticadba' home='/home/dbadmin'>
20140604 102333:INFO:vertica.platform.node.dba.DbaCreate:step: Provided DB Admin account details:
20140604 102333:INFO:vertica.platform.node.dba.DbaCreate:progress: user = dbadmin
20140604 102333:INFO:vertica.platform.node.dba.DbaCreate:progress: group = verticadba
20140604 102333:INFO:vertica.platform.node.dba.DbaCreate:progress: home = /home/dbadmin
20140604 102333:INFO:vertica.platform.node.dba.DbaCreate:step: Creating group...
20140604 102333:INFO:vertica.platform.node.dba.DbaCreate:progress: Group already exists
20140604 102333:INFO:vertica.platform.node.dba.DbaCreate:step: Validating group...
20140604 102333:INFO:vertica.platform.node.dba.DbaCreate:progress: Okay
20140604 102333:INFO:vertica.platform.node.dba.DbaCreate:step: Creating user...
20140604 102333:INFO:vertica.platform.node.dba.DbaCreate:progress: User already exists
20140604 102333:INFO:vertica.platform.node.dba.DbaCreate:step: Validating user...
20140604 102333:DEBUG:vertica.platform.node.dba.DbaCreate:Hidden from user, running /opt/vertica/share/binlib/test/file-access -u dbadmin /home/dbadmin
20140604 102333:DEBUG:vertica.platform.node.dba.DbaCreate:Output of test/file-access: '\x1b[H\x1b[2J'
20140604 102333:INFO:vertica.platform.node.dba.DbaCreate:progress: Okay
20140604 102333:INFO:vertica.platform.node.dba.DbaCreate:Vertica .bashrc already in place
20140604 102334:DEBUG:root:Logging started.  Logging to: ['/opt/vertica/log/install.log']
20140604 102334:DEBUGlithium_main__.LocalCoerce:Preparing output directory /opt/vertica/log/local-coerce-20140604_102334
20140604 102334:DEBUGlithium_main__.LocalCoerce:Parent of output directory exists. Creating dir.
20140604 102334:DEBUGlithium_main__.LocalCoerce:Output directory created
20140604 102334:DEBUG:vertica.platform.node.t.cpu.CheckNodeCpu:ondemand governor nice policy unknown
20140604 102334:DEBUG:vertica.platform.node.t.cpu.CheckNodeCpu:cpufreq scaling not available for cpu0
20140604 102334:DEBUG:vertica.platform.node.t.cpu.CheckNodeCpu:cpufreq scaling not available for cpu1
20140604 102334:DEBUG:vertica.platform.node.t.cpu.CheckNodeCpu:Found 2 processors and 3912644 kb memory
20140604 102334:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:Disks:
20140604 102334:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/sda1', disk='sda', readahead='256', schedule='cfq', fs='ext3', blocksz='4096')
20140604 102334:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-home', disk='dm-1', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 102334:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-root', disk='dm-0', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 102334:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:Disks:
20140604 102334:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/sda1', disk='sda', readahead='256', schedule='cfq', fs='ext3', blocksz='4096')
20140604 102334:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-home', disk='dm-1', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 102334:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-root', disk='dm-0', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 102334:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:lvscan says: "  ACTIVE            '/dev/system/root' [33.22 GB] inherit\n  ACTIVE            '/dev/system/home' [256.00 MB] inherit\n  ACTIVE            '/dev/system/swap' [6.00 GB] inherit\n"
20140604 102334:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:Disks:
20140604 102334:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/sda1', disk='sda', readahead='256', schedule='cfq', fs='ext3', blocksz='4096')
20140604 102334:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-home', disk='dm-1', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 102334:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-root', disk='dm-0', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 102334:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:Disks:
20140604 102334:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/sda1', disk='sda', readahead='256', schedule='cfq', fs='ext3', blocksz='4096')
20140604 102334:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-home', disk='dm-1', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 102334:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-root', disk='dm-0', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 102334:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:Disks:
20140604 102334:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/sda1', disk='sda', readahead='256', schedule='cfq', fs='ext3', blocksz='4096')
20140604 102334:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-home', disk='dm-1', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 102334:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-root', disk='dm-0', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 102334:INFO:vertica.platform.node.t.disk.CheckNodeDisk:Found meminfo line 12: 'SwapTotal:     6291448 kB'
20140604 102334:DEBUG:vertica.platform.node.t.dbadmin.CheckDbAdmin:dbadmin home owned by 'dbadmin'
20140604 102334:DEBUG:vertica.platform.node.t.dbadmin.CheckDbAdmin:dbadmin home owned by group 'dbadmin'
20140604 102334:INFO:vertica.platform.node.t.dbadmin.CheckDbAdmin:LANG = 'en_US.UTF-8\n\x1b[H\x1b[2J'
20140604 102334:INFO:vertica.platform.node.t.dbadmin.CheckDbAdmin:TZ = '\x1b[H\x1b[2J'
20140604 102335:DEBUG:vertica.platform.node.t.limits.CheckNodeLimits:Found line 14: 'session required pam_limits.so'
20140604 102335:INFO:vertica.platform.node.t.misc.CheckNodeMisc:NTPD is running as pid 3251
20140604 102335:INFO:vertica.platform.node.t.misc.CheckNodeMisc:Found /etc/redhat-release
20140604 102335:INFO:vertica.platform.node.t.misc.CheckNodeMisc:release string = 'Red Hat Enterprise Linux Server release 5.10 (Tikanga)'
20140604 102335:INFO:vertica.platform.node.t.misc.CheckNodeMisc:Found sestatus: /usr/sbin/sestatus
20140604 102335:INFO:vertica.platform.node.t.misc.CheckNodeMisc:sestatus output: 'SELinux status:                 disabled\n'
20140604 102335:INFO:vertica.platform.node.t.misc.CheckNodeMisc:transparent_hugepage feature unavailable.
20140604 102335:INFO:vertica.platform.node.t.network.CheckNodeNetwork:Found iptables: /sbin/iptables
20140604 102335:INFO:vertica.platform.node.t.network.CheckNodeNetwork:Did not find ip_tables_names. No tables defined.
20140604 102335:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No tcp6 usage on this machine
20140604 102335:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No udp6 usage on this machine
20140604 102335:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No tcp6 usage on this machine
20140604 102335:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No tcp6 usage on this machine
20140604 102335:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No tcp6 usage on this machine
20140604 102335:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No tcp6 usage on this machine
20140604 102335:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No udp6 usage on this machine
20140604 102335:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No udp6 usage on this machine
20140604 102335:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No udp6 usage on this machine
20140604 102335:INFO:vertica.platform.node.t.network.CheckNodeNetwork:Did not find SuSEfirewall2
20140604 102335:INFO:vertica.platform.node.t.sysctl.CheckNodeSysctl:Read from fs.file-max: '390819'
20140604 102335:INFO:vertica.platform.node.t.sysctl.CheckNodeSysctl:Read from vm.max_map_count: '244540'
20140604 102335:INFO:vertica.platform.node.t.sysctl.CheckNodeSysctl:Read from vm.min_free_kbytes: '8115'
20140604 102335:INFO:vertica.platform.node.t.tool.CheckNodeTool:Found mcelog as /usr/sbin/mcelog

20140604 102335:INFO:vertica.platform.node.t.tool.CheckNodeTool:Found pstack or gstack as /usr/bin/gstack

20140604 102335:INFO:vertica.platform.node.t.tool.CheckNodeTool:Found sysstat as /usr/bin/mpstat

20140604 102335:INFO:vertica.platform.node.t.tool.CheckNodeTool:Found sysstat as /usr/bin/iostat

20140604 102335:DEBUG:vertica.local_verify.LocalVerify:Removed pre-existing symlink at /opt/vertica/log/verify-latest.xml.
20140604 102335:INFO:vertica.local_verify.LocalVerify:Creating symlink from /opt/vertica/log/verify-latest.xml to local-coerce-20140604_102334/iter01/verify.xml.
20140604 102335:INFO:vertica.local_verify.LocalVerify:Creating symlink from /opt/vertica/log/local-coerce-20140604_102334/verify-latest.xml to iter01/verify.xml.
20140604 102335:DEBUG:vertica.platform.fix.context.FixContext:Preparing staging directory /opt/vertica/log/local-coerce-20140604_102334/iter01/configure
20140604 102335:DEBUG:vertica.platform.fix.context.FixContext:Real location of staging directory: /opt/vertica/log/local-coerce-20140604_102334/iter01/configure
20140604 102335:DEBUG:vertica.platform.fix.context.FixContext:Staging directory does not exist. Creating.
20140604 102335:DEBUG:vertica.platform.fix.context.FixContext:Creating backup, new, and temporary directories
20140604 102737:DEBUG:root:Logging started.  Logging to: ['/opt/vertica/log/install.log']
20140604 102737:INFO:root:------------------------------------------------------------
20140604 102737:INFO:root:Begin create_dba
20140604 102737:INFO:root:------------------------------------------------------------
20140604 102737:INFO:vertica.platform.node.dba.DbaCreate:DbaCreate initialized: <UserGroup user='dbadmin' group='verticadba' home='/home/dbadmin'>
20140604 102737:INFO:vertica.platform.node.dba.DbaCreate:step: Provided DB Admin account details:
20140604 102737:INFO:vertica.platform.node.dba.DbaCreate:progress: user = dbadmin
20140604 102737:INFO:vertica.platform.node.dba.DbaCreate:progress: group = verticadba
20140604 102737:INFO:vertica.platform.node.dba.DbaCreate:progress: home = /home/dbadmin
20140604 102737:INFO:vertica.platform.node.dba.DbaCreate:step: Creating group...
20140604 102737:INFO:vertica.platform.node.dba.DbaCreate:progress: Group already exists
20140604 102737:INFO:vertica.platform.node.dba.DbaCreate:step: Validating group...
20140604 102737:INFO:vertica.platform.node.dba.DbaCreate:progress: Okay
20140604 102737:INFO:vertica.platform.node.dba.DbaCreate:step: Creating user...
20140604 102737:INFO:vertica.platform.node.dba.DbaCreate:progress: User already exists
20140604 102737:INFO:vertica.platform.node.dba.DbaCreate:step: Validating user...
20140604 102737:DEBUG:vertica.platform.node.dba.DbaCreate:Hidden from user, running /opt/vertica/share/binlib/test/file-access -u dbadmin /home/dbadmin
20140604 102737:DEBUG:vertica.platform.node.dba.DbaCreate:Output of test/file-access: '\x1b[H\x1b[2J'
20140604 102737:INFO:vertica.platform.node.dba.DbaCreate:progress: Okay
20140604 102737:INFO:vertica.platform.node.dba.DbaCreate:Vertica .bashrc already in place
20140604 102738:DEBUG:root:Logging started.  Logging to: ['/opt/vertica/log/install.log']
20140604 102738:DEBUGlithium_main__.LocalCoerce:Preparing output directory /opt/vertica/log/local-coerce-20140604_102738
20140604 102738:DEBUGlithium_main__.LocalCoerce:Parent of output directory exists. Creating dir.
20140604 102738:DEBUGlithium_main__.LocalCoerce:Output directory created
20140604 102738:DEBUG:vertica.platform.node.t.cpu.CheckNodeCpu:ondemand governor nice policy unknown
20140604 102738:DEBUG:vertica.platform.node.t.cpu.CheckNodeCpu:cpufreq scaling not available for cpu0
20140604 102738:DEBUG:vertica.platform.node.t.cpu.CheckNodeCpu:cpufreq scaling not available for cpu1
20140604 102738:DEBUG:vertica.platform.node.t.cpu.CheckNodeCpu:Found 2 processors and 3912644 kb memory
20140604 102738:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:Disks:
20140604 102738:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/sda1', disk='sda', readahead='256', schedule='cfq', fs='ext3', blocksz='4096')
20140604 102738:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-home', disk='dm-1', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 102738:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-root', disk='dm-0', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 102738:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:Disks:
20140604 102738:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/sda1', disk='sda', readahead='256', schedule='cfq', fs='ext3', blocksz='4096')
20140604 102738:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-home', disk='dm-1', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 102738:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-root', disk='dm-0', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 102738:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:lvscan says: "  ACTIVE            '/dev/system/root' [33.22 GB] inherit\n  ACTIVE            '/dev/system/home' [256.00 MB] inherit\n  ACTIVE            '/dev/system/swap' [6.00 GB] inherit\n"
20140604 102738:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:Disks:
20140604 102738:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/sda1', disk='sda', readahead='256', schedule='cfq', fs='ext3', blocksz='4096')
20140604 102738:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-home', disk='dm-1', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 102738:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-root', disk='dm-0', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 102738:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:Disks:
20140604 102738:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/sda1', disk='sda', readahead='256', schedule='cfq', fs='ext3', blocksz='4096')
20140604 102738:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-home', disk='dm-1', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 102738:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-root', disk='dm-0', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 102738:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:Disks:
20140604 102738:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/sda1', disk='sda', readahead='256', schedule='cfq', fs='ext3', blocksz='4096')
20140604 102738:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-home', disk='dm-1', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 102738:DEBUG:vertica.platform.node.t.disk.CheckNodeDisk:DiskInfo(device='/dev/mapper/system-root', disk='dm-0', readahead='256', schedule='', fs='ext3', blocksz='4096')
20140604 102738:INFO:vertica.platform.node.t.disk.CheckNodeDisk:Found meminfo line 12: 'SwapTotal:     6291448 kB'
20140604 102738:DEBUG:vertica.platform.node.t.dbadmin.CheckDbAdmin:dbadmin home owned by 'dbadmin'
20140604 102738:DEBUG:vertica.platform.node.t.dbadmin.CheckDbAdmin:dbadmin home owned by group 'dbadmin'
20140604 102738:INFO:vertica.platform.node.t.dbadmin.CheckDbAdmin:LANG = 'en_US.UTF-8\n\x1b[H\x1b[2J'
20140604 102738:INFO:vertica.platform.node.t.dbadmin.CheckDbAdmin:TZ = '\x1b[H\x1b[2J'
20140604 102738:DEBUG:vertica.platform.node.t.limits.CheckNodeLimits:Found line 14: 'session required pam_limits.so'
20140604 102739:INFO:vertica.platform.node.t.misc.CheckNodeMisc:NTPD is running as pid 3251
20140604 102739:INFO:vertica.platform.node.t.misc.CheckNodeMisc:Found /etc/redhat-release
20140604 102739:INFO:vertica.platform.node.t.misc.CheckNodeMisc:release string = 'Red Hat Enterprise Linux Server release 5.10 (Tikanga)'
20140604 102739:INFO:vertica.platform.node.t.misc.CheckNodeMisc:Found sestatus: /usr/sbin/sestatus
20140604 102739:INFO:vertica.platform.node.t.misc.CheckNodeMisc:sestatus output: 'SELinux status:                 disabled\n'
20140604 102739:INFO:vertica.platform.node.t.misc.CheckNodeMisc:transparent_hugepage feature unavailable.
20140604 102739:INFO:vertica.platform.node.t.network.CheckNodeNetwork:Found iptables: /sbin/iptables
20140604 102739:INFO:vertica.platform.node.t.network.CheckNodeNetwork:Did not find ip_tables_names. No tables defined.
20140604 102739:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No tcp6 usage on this machine
20140604 102739:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No udp6 usage on this machine
20140604 102739:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No tcp6 usage on this machine
20140604 102739:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No tcp6 usage on this machine
20140604 102739:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No tcp6 usage on this machine
20140604 102739:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No tcp6 usage on this machine
20140604 102739:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No udp6 usage on this machine
20140604 102739:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No udp6 usage on this machine
20140604 102739:INFO:vertica.platform.node.t.network.CheckNodeNetwork:No udp6 usage on this machine
20140604 102739:INFO:vertica.platform.node.t.network.CheckNodeNetwork:Did not find SuSEfirewall2
20140604 102739:INFO:vertica.platform.node.t.sysctl.CheckNodeSysctl:Read from fs.file-max: '390819'
20140604 102739:INFO:vertica.platform.node.t.sysctl.CheckNodeSysctl:Read from vm.max_map_count: '244540'
20140604 102739:INFO:vertica.platform.node.t.sysctl.CheckNodeSysctl:Read from vm.min_free_kbytes: '8115'
20140604 102739:INFO:vertica.platform.node.t.tool.CheckNodeTool:Found mcelog as /usr/sbin/mcelog

20140604 102739:INFO:vertica.platform.node.t.tool.CheckNodeTool:Found pstack or gstack as /usr/bin/gstack

20140604 102739:INFO:vertica.platform.node.t.tool.CheckNodeTool:Found sysstat as /usr/bin/mpstat

20140604 102739:INFO:vertica.platform.node.t.tool.CheckNodeTool:Found sysstat as /usr/bin/iostat

20140604 102739:DEBUG:vertica.local_verify.LocalVerify:Removed pre-existing symlink at /opt/vertica/log/verify-latest.xml.
20140604 102739:INFO:vertica.local_verify.LocalVerify:Creating symlink from /opt/vertica/log/verify-latest.xml to local-coerce-20140604_102738/iter01/verify.xml.
20140604 102739:INFO:vertica.local_verify.LocalVerify:Creating symlink from /opt/vertica/log/local-coerce-20140604_102738/verify-latest.xml to iter01/verify.xml.
20140604 102739:DEBUG:vertica.platform.fix.context.FixContext:Preparing staging directory /opt/vertica/log/local-coerce-20140604_102738/iter01/configure
20140604 102739:DEBUG:vertica.platform.fix.context.FixContext:Real location of staging directory: /opt/vertica/log/local-coerce-20140604_102738/iter01/configure
20140604 102739:DEBUG:vertica.platform.fix.context.FixContext:Staging directory does not exist. Creating.
20140604 102739:DEBUG:vertica.platform.fix.context.FixContext:Creating backup, new, and temporary directories
20140604 102817:DEBUG:root:Logging started.  Logging to: ['/opt/vertica/log/install.log']
20140604 102817:INFO:root:------------------------------------------------------------
20140604 102817:INFO:root:Begin create_dba
20140604 102817:INFO:root:------------------------------------------------------------
20140604 102817:INFO:vertica.platform.node.dba.DbaCreate:DbaCreate initialized: <UserGroup user='dbadmin' group='verticadba' home='/home/dbadmin'>
20140604 102817:INFO:vertica.platform.node.dba.DbaCreate:step: Provided DB Admin account details:
20140604 102817:INFO:vertica.platform.node.dba.DbaCreate:progress: user = dbadmin
20140604 102817:INFO:vertica.platform.node.dba.DbaCreate:progress: group = verticadba
20140604 102817:INFO:vertica.platform.node.dba.DbaCreate:progress: home = /home/dbadmin
20140604 102817:INFO:vertica.platform.node.dba.DbaCreate:step: Creating group...
20140604 102817:INFO:vertica.platform.node.dba.DbaCreate:progress: Group already exists
20140604 102817:INFO:vertica.platform.node.dba.DbaCreate:step: Validating group...
20140604 102817:INFO:vertica.platform.node.dba.DbaCreate:progress: Okay
20140604 102817:INFO:vertica.platform.node.dba.DbaCreate:step: Creating user...
20140604 102817:INFO:vertica.platform.node.dba.DbaCreate:progress: User already exists
20140604 102817:INFO:vertica.platform.node.dba.DbaCreate:step: Validating user...
20140604 102817:DEBUG:vertica.platform.node.dba.DbaCreate:Hidden from user, running /opt/vertica/share/binlib/test/file-access -u dbadmin /home/dbadmin
20140604 102817:DEBUG:vertica.platform.node.dba.DbaCreate:Output of test/file-access: '\x1b[H\x1b[2J'
20140604 102817

Sunny_1

admintools.log output: 

un  4 11:12:50  [2764] <PexpectAdapter: root@1.1.1.198>: closeConnection Jun  4 11:12:51  [2764] Hosts to add: [] 
Jun  4 11:12:51  [2764] Hosts to remove: [] 
Jun  4 11:12:51  [2764] Hosts to update: ['1.1.1.125', '1.1.1.191', '1.1.1.198'] 
Jun  4 11:12:51  [2764] Resulting cluster: ['1.1.1.125', '1.1.1.191', '1.1.1.198'] 
Jun  4 11:12:51  [2764] <PexpectAdapter: root@1.1.1.198>: login start: ssh  -oConnectTimeout=30 -o TCPKeepAlive=no -o ServerAliveInterval=15 -o ServerAliveCountMax=2  -l root 1.1.1.198 
Jun  4 11:12:51  [2764] <PexpectAdapter: root@1.1.1.125>: login start: ssh  -oConnectTimeout=30 -o TCPKeepAlive=no -o ServerAliveInterval=15 -o ServerAliveCountMax=2  -l root 1.1.1.125 
Jun  4 11:12:51  [2764] <PexpectAdapter: root@1.1.1.125>: <command hidden> 
Jun  4 11:12:51  [2764] <BashAdapter: root@1.1.1.191>: <command hidden> 
Jun  4 11:12:51  [2764] <PexpectAdapter: root@1.1.1.198>: <command hidden> 
Jun  4 11:12:52  [2764] <PexpectAdapter: root@1.1.1.125>: <command hidden> 
Jun  4 11:12:52  [2764] <BashAdapter: root@1.1.1.191>: <command hidden> 
Jun  4 11:12:52  [2764] <PexpectAdapter: root@1.1.1.198>: <command hidden> 
Jun  4 11:12:52  [2764] <PexpectAdapter: root@1.1.1.125>: echo 'if [ -e "/opt/vertica" ]; then cd /tmp; find /opt/vertica -perm -755 -type d | grep "/opt/vertica\$"; else echo /opt/vertica; fi' 
Jun  4 11:12:52  [2764] <BashAdapter: root@1.1.1.191>: ['echo 'if [ -e "/opt/vertica" ]; then cd /tmp; find /opt/vertica -perm -755 -type d | grep "/opt/vertica\\$"; else echo /opt/vertica; fi''] 
Jun  4 11:12:52  [2764] <PexpectAdapter: root@1.1.1.198>: echo 'if [ -e "/opt/vertica" ]; then cd /tmp; find /opt/vertica -perm -755 -type d | grep "/opt/vertica\$"; else echo /opt/vertica; fi' 
Jun  4 11:12:52  [2764] <BashAdapter: root@1.1.1.191>: (rc=0) ['/opt/vertica'] 
Jun  4 11:12:52  [2764] <PexpectAdapter: root@1.1.1.125>: (rc=0) ['/opt/vertica'] 
Jun  4 11:12:52  [2764] <PexpectAdapter: root@1.1.1.198>: (rc=0) ['/opt/vertica'] 
Jun  4 11:12:52  [2764] RPM Check,  The same version is already installed 
Jun  4 11:12:52  [2764] RPM Check,  The same version is already installed 
Jun  4 11:12:52  [2764] RPM Check,  The same version is already installed 
Jun  4 11:12:52  [2764] <PexpectAdapter: root@1.1.1.125>: <command hidden> 
Jun  4 11:12:52  [2764] <BashAdapter: root@1.1.1.191>: <command hidden> 
Jun  4 11:12:52  [2764] <PexpectAdapter: root@1.1.1.198>: <command hidden> 
Jun  4 11:12:52  [2764] backing up admintools.conf on 1.1.1.125  
Jun  4 11:12:52  [2764] backing up admintools.conf on 1.1.1.191  
Jun  4 11:12:52  [2764] backing up admintools.conf on 1.1.1.198  
Jun  4 11:12:52  [2764] <PexpectAdapter: root@1.1.1.125>: id dbadmin 
Jun  4 11:12:52  [2764] <BashAdapter: root@1.1.1.191>: ['id dbadmin'] 
Jun  4 11:12:52  [2764] <PexpectAdapter: root@1.1.1.198>: id dbadmin 
Jun  4 11:12:52  [2764] <BashAdapter: root@1.1.1.191>: (rc=0) ['uid=59806(dbadmin) gid=59807(dbadmin) groups=59807(dbadmin),59805(verticadba)'] 
Jun  4 11:12:53  [2764] <PexpectAdapter: root@1.1.1.125>: (rc=0) ['uid=59806(dbadmin) gid=59807(dbadmin) groups=59807(dbadmin),59806(verticadba)'] 
Jun  4 11:12:53  [2764] <PexpectAdapter: root@1.1.1.198>: (rc=0) ['uid=59806(dbadmin) gid=59807(dbadmin) groups=59807(dbadmin),59806(verticadba)'] 
Jun  4 11:12:53  [2764] <PexpectAdapter: root@1.1.1.125>: <command hidden> 
Jun  4 11:12:53  [2764] <BashAdapter: root@1.1.1.191>: <command hidden> 
Jun  4 11:12:53  [2764] <PexpectAdapter: root@1.1.1.198>: <command hidden> 
Jun  4 11:12:53  [2764] <PexpectAdapter: root@1.1.1.125>: VERT_DBA_USR=dbadmin VERT_DBA_HOME=/home/dbadmin VERT_DBA_GRP=verticadba /opt/vertica/oss/python/bin/python -m vertica.local_coerce 
Jun  4 11:12:53  [2764] <BashAdapter: root@1.1.1.191>: ['VERT_DBA_USR=dbadmin VERT_DBA_HOME=/home/dbadmin VERT_DBA_GRP=verticadba /opt/vertica/oss/python/bin/python -m vertica.local_coerce'] 
Jun  4 11:12:53  [2764] <PexpectAdapter: root@1.1.1.198>: VERT_DBA_USR=dbadmin VERT_DBA_HOME=/home/dbadmin VERT_DBA_GRP=verticadba /opt/vertica/oss/python/bin/python -m vertica.local_coerce 
Jun  4 11:12:54  [2764] <BashAdapter: root@1.1.1.191>: (rc=255) ['Traceback (most recent call last):', '  File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 162, in _run_module_as_main', '    "__main__", fname, loader, pkg_name)', '  File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 72, in _run_code', '    exec code in run_globals', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 204, in <module>', '    _main()', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 196, in _main', '    coerce.run()', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 76, in run', '    halted = self._run_to_halt()', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 81, in _run_to_halt', '    if self._iterate(x):', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 113, in _iterate', '    configure.load()', '  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/platform/fix/runner.py", line 49, in load', '    self._report.load_results(self._infile)', '  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/platform/check/report.py", line 59, in load_results', '    tree = xml.etree.ElementTree.parse(xmlfile)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 1176, in parse', '    tree.parse(source, parser)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 653, in parse', '    parser.feed(data)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 1623, in feed', '    self._raiseerror(v)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 1487, in _raiseerror', '    raise err', 'xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 122, column 34'] 
Jun  4 11:12:54  [2764] <PexpectAdapter: root@1.1.1.198>: (rc=255) ['Traceback (most recent call last):', '  File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 162, in _run_module_as_main', '    "__main__", fname, loader, pkg_name)', '  File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 72, in _run_code', '    exec code in run_globals', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 204, in <module>', '    _main()', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 196, in _main', '    coerce.run()', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 76, in run', '    halted = self._run_to_halt()', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 81, in _run_to_halt', '    if self._iterate(x):', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 113, in _iterate', '    configure.load()', '  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/platform/fix/runner.py", line 49, in load', '    self._report.load_results(self._infile)', '  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/platform/check/report.py", line 59, in load_results', '    tree = xml.etree.ElementTree.parse(xmlfile)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 1176, in parse', '    tree.parse(source, parser)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 653, in parse', '    parser.feed(data)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 1623, in feed', '    self._raiseerror(v)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 1487, in _raiseerror', '    raise err', 'xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 122, column 34'] 
Jun  4 11:12:54  [2764] <PexpectAdapter: root@1.1.1.125>: (rc=255) ['Traceback (most recent call last):', '  File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 162, in _run_module_as_main', '    "__main__", fname, loader, pkg_name)', '  File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 72, in _run_code', '    exec code in run_globals', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 204, in <module>', '    _main()', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 196, in _main', '    coerce.run()', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 76, in run', '    halted = self._run_to_halt()', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 81, in _run_to_halt', '    if self._iterate(x):', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 113, in _iterate', '    configure.load()', '  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/platform/fix/runner.py", line 49, in load', '    self._report.load_results(self._infile)', '  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/platform/check/report.py", line 59, in load_results', '    tree = xml.etree.ElementTree.parse(xmlfile)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 1176, in parse', '    tree.parse(source, parser)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 653, in parse', '    parser.feed(data)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 1623, in feed', '    self._raiseerror(v)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 1487, in _raiseerror', '    raise err', 'xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 122, column 34'] 
Jun  4 11:12:55  [2764] <PexpectAdapter: root@1.1.1.125>: closeConnection 
Jun  4 11:12:55  [2764] <PexpectAdapter: root@1.1.1.198>: closeConnection 
Jun  4 11:20:33  [3799] NEW LOG: '/opt/vertica/share/eggs/vertica/install/__main__.py' --update --rpm '/opt/CA/IMDataRepository_vertica7/resources/vertica-7.0.1-2.x86_64.RHEL5.rpm' 
Jun  4 11:20:33  [3799] [adminExec.__init__] Unique Ports: False 
Jun  4 11:20:33  [3799] Saved Configurator(clusterSize=3, dbs=(im)) -> /opt/vertica/config/admintools.conf 
Jun  4 11:20:33  [3799] <PexpectAdapter: root@1.1.1.125>: login start: ssh  -oConnectTimeout=30 -o TCPKeepAlive=no -o ServerAliveInterval=15 -o ServerAliveCountMax=2  -l root 1.1.1.125 
Jun  4 11:20:34  [3799] <PexpectAdapter: root@1.1.1.198>: login start: ssh  -oConnectTimeout=30 -o TCPKeepAlive=no -o ServerAliveInterval=15 -o ServerAliveCountMax=2  -l root 1.1.1.198 
Jun  4 11:20:35  [3799] <PexpectAdapter: root@1.1.1.125>: [ -f /etc/fedora-release ] 
Jun  4 11:20:35  [3799] <BashAdapter: root@1.1.1.191>: ['[ -f /etc/fedora-release ]'] 
Jun  4 11:20:35  [3799] <PexpectAdapter: root@1.1.1.198>: [ -f /etc/fedora-release ] 
Jun  4 11:20:35  [3799] <BashAdapter: root@1.1.1.191>: (rc=1) [] 
Jun  4 11:20:35  [3799] <BashAdapter: root@1.1.1.191>: ['[ -f /etc/redhat-release ]'] 
Jun  4 11:20:35  [3799] <BashAdapter: root@1.1.1.191>: (rc=0) [] 
Jun  4 11:20:35  [3799] <BashAdapter: root@1.1.1.191>: ['egrep "release (5|6)" /etc/redhat-release'] 
Jun  4 11:20:35  [3799] <BashAdapter: root@1.1.1.191>: (rc=0) ['Red Hat Enterprise Linux Server release 5.10 (Tikanga)'] 
Jun  4 11:20:35  [3799] <BashAdapter: root@1.1.1.191>: ['LANG=C /sbin/ip -oneline -family inet addr'] 
Jun  4 11:20:35  [3799] <BashAdapter: root@1.1.1.191>: (rc=0) ['1: lo    inet 127.0.0.1/8 scope host lo', '2: eth0    inet 1.1.1.191/24 brd 167.143.132.255 scope global eth0'] 
Jun  4 11:20:35  [3799] <BashAdapter: root@1.1.1.191>: ['rpm -q vertica --qf "%{NAME} %{VERSION} %{RELEASE} %{ARCH}\\n"'] 
Jun  4 11:20:35  [3799] <BashAdapter: root@1.1.1.191>: (rc=0) ['vertica 7.0.1 2 x86_64'] 
Jun  4 11:20:35  [3799] [Profile.VerticaRelease] query results 'vertica 7.0.1 2 x86_64' 
Jun  4 11:20:35  [3799] <PexpectAdapter: root@1.1.1.125>: (rc=1) [] 
Jun  4 11:20:35  [3799] <PexpectAdapter: root@1.1.1.125>: [ -f /etc/redhat-release ] 
Jun  4 11:20:35  [3799] <PexpectAdapter: root@1.1.1.198>: (rc=1) [] 
Jun  4 11:20:35  [3799] <PexpectAdapter: root@1.1.1.198>: [ -f /etc/redhat-release ] 
Jun  4 11:20:35  [3799] <PexpectAdapter: root@1.1.1.125>: (rc=0) [] 
Jun  4 11:20:35  [3799] <PexpectAdapter: root@1.1.1.125>: egrep "release (5|6)" /etc/redhat-release 
Jun  4 11:20:35  [3799] <PexpectAdapter: root@1.1.1.198>: (rc=0) [] 
Jun  4 11:20:35  [3799] <PexpectAdapter: root@1.1.1.198>: egrep "release (5|6)" /etc/redhat-release 
Jun  4 11:20:36  [3799] <PexpectAdapter: root@1.1.1.125>: (rc=0) ['Red Hat Enterprise Linux Server release 5.10 (Tikanga)'] 
Jun  4 11:20:36  [3799] <PexpectAdapter: root@1.1.1.125>: LANG=C /sbin/ip -oneline -family inet addr 
Jun  4 11:20:36  [3799] <PexpectAdapter: root@1.1.1.198>: (rc=0) ['Red Hat Enterprise Linux Server release 5.10 (Tikanga)'] 
Jun  4 11:20:36  [3799] <PexpectAdapter: root@1.1.1.198>: LANG=C /sbin/ip -oneline -family inet addr 
Jun  4 11:20:36  [3799] <PexpectAdapter: root@1.1.1.125>: (rc=0) ['1: lo    inet 127.0.0.1/8 scope host lo', '2: eth0    inet 1.1.1.125/24 brd 167.143.132.255 scope global eth0'] 
Jun  4 11:20:36  [3799] <PexpectAdapter: root@1.1.1.125>: rpm -q vertica --qf "%{NAME} %{VERSION} %{RELEASE} %{ARCH}\n" 
Jun  4 11:20:36  [3799] <PexpectAdapter: root@1.1.1.198>: (rc=0) ['1: lo    inet 127.0.0.1/8 scope host lo', '2: eth0    inet 1.1.1.198/24 brd 167.143.132.255 scope global eth0'] 
Jun  4 11:20:36  [3799] <PexpectAdapter: root@1.1.1.198>: rpm -q vertica --qf "%{NAME} %{VERSION} %{RELEASE} %{ARCH}\n" 
Jun  4 11:20:36  [3799] <PexpectAdapter: root@1.1.1.125>: (rc=0) ['vertica 7.0.1 2 x86_64'] 
Jun  4 11:20:36  [3799] [Profile.VerticaRelease] query results 'vertica 7.0.1 2 x86_64' 
Jun  4 11:20:36  [3799] <PexpectAdapter: root@1.1.1.198>: (rc=0) ['vertica 7.0.1 2 x86_64'] 
Jun  4 11:20:36  [3799] [Profile.VerticaRelease] query results 'vertica 7.0.1 2 x86_64' 
Jun  4 11:20:36  [3799] <PexpectAdapter: root@1.1.1.125>: closeConnection 
Jun  4 11:20:36  [3799] <PexpectAdapter: root@1.1.1.198>: closeConnection 
Jun  4 11:20:37  [3799] Hosts to add: [] 
Jun  4 11:20:37  [3799] Hosts to remove: [] 
Jun  4 11:20:37  [3799] Hosts to update: ['1.1.1.125', '1.1.1.191', '1.1.1.198'] 
Jun  4 11:20:37  [3799] Resulting cluster: ['1.1.1.125', '1.1.1.191', '1.1.1.198'] 
Jun  4 11:20:37  [3799] <PexpectAdapter: root@1.1.1.125>: login start: ssh  -oConnectTimeout=30 -o TCPKeepAlive=no -o ServerAliveInterval=15 -o ServerAliveCountMax=2  -l root 1.1.1.125 
Jun  4 11:20:37  [3799] <PexpectAdapter: root@1.1.1.198>: login start: ssh  -oConnectTimeout=30 -o TCPKeepAlive=no -o ServerAliveInterval=15 -o ServerAliveCountMax=2  -l root 1.1.1.198 
Jun  4 11:20:38  [3799] <PexpectAdapter: root@1.1.1.125>: <command hidden> 
Jun  4 11:20:38  [3799] <BashAdapter: root@1.1.1.191>: <command hidden> 
Jun  4 11:20:38  [3799] <PexpectAdapter: root@1.1.1.198>: <command hidden> 
Jun  4 11:20:38  [3799] <BashAdapter: root@1.1.1.191>: <command hidden> 
Jun  4 11:20:38  [3799] <PexpectAdapter: root@1.1.1.125>: <command hidden> 
Jun  4 11:20:38  [3799] <PexpectAdapter: root@1.1.1.198>: <command hidden> 
Jun  4 11:20:38  [3799] <PexpectAdapter: root@1.1.1.125>: echo 'if [ -e "/opt/vertica" ]; then cd /tmp; find /opt/vertica -perm -755 -type d | grep "/opt/vertica\$"; else echo /opt/vertica; fi' 
Jun  4 11:20:38  [3799] <BashAdapter: root@1.1.1.191>: ['echo 'if [ -e "/opt/vertica" ]; then cd /tmp; find /opt/vertica -perm -755 -type d | grep "/opt/vertica\\$"; else echo /opt/vertica; fi''] 
Jun  4 11:20:38  [3799] <PexpectAdapter: root@1.1.1.198>: echo 'if [ -e "/opt/vertica" ]; then cd /tmp; find /opt/vertica -perm -755 -type d | grep "/opt/vertica\$"; else echo /opt/vertica; fi' 
Jun  4 11:20:38  [3799] <BashAdapter: root@1.1.1.191>: (rc=0) ['/opt/vertica'] 
Jun  4 11:20:38  [3799] <PexpectAdapter: root@1.1.1.125>: (rc=0) ['/opt/vertica'] 
Jun  4 11:20:38  [3799] <PexpectAdapter: root@1.1.1.198>: (rc=0) ['/opt/vertica'] 
Jun  4 11:20:38  [3799] RPM Check,  The same version is already installed 
Jun  4 11:20:38  [3799] RPM Check,  The same version is already installed 
Jun  4 11:20:38  [3799] RPM Check,  The same version is already installed 
Jun  4 11:20:38  [3799] <PexpectAdapter: root@1.1.1.125>: <command hidden> 
Jun  4 11:20:38  [3799] <BashAdapter: root@1.1.1.191>: <command hidden> 
Jun  4 11:20:38  [3799] <PexpectAdapter: root@1.1.1.198>: <command hidden> 
Jun  4 11:20:38  [3799] backing up admintools.conf on 1.1.1.125  
Jun  4 11:20:38  [3799] backing up admintools.conf on 1.1.1.191  
Jun  4 11:20:38  [3799] backing up admintools.conf on 1.1.1.198  
Jun  4 11:20:38  [3799] <PexpectAdapter: root@1.1.1.125>: id dbadmin 
Jun  4 11:20:38  [3799] <BashAdapter: root@1.1.1.191>: ['id dbadmin'] 
Jun  4 11:20:38  [3799] <PexpectAdapter: root@1.1.1.198>: id dbadmin 
Jun  4 11:20:38  [3799] <BashAdapter: root@1.1.1.191>: (rc=0) ['uid=59806(dbadmin) gid=59807(dbadmin) groups=59807(dbadmin),59805(verticadba)'] 
Jun  4 11:20:39  [3799] <PexpectAdapter: root@1.1.1.125>: (rc=0) ['uid=59806(dbadmin) gid=59807(dbadmin) groups=59807(dbadmin),59806(verticadba)'] 
Jun  4 11:20:39  [3799] <PexpectAdapter: root@1.1.1.198>: (rc=0) ['uid=59806(dbadmin) gid=59807(dbadmin) groups=59807(dbadmin),59806(verticadba)'] 
Jun  4 11:20:39  [3799] <PexpectAdapter: root@1.1.1.125>: <command hidden> 
Jun  4 11:20:39  [3799] <BashAdapter: root@1.1.1.191>: <command hidden> 
Jun  4 11:20:39  [3799] <PexpectAdapter: root@1.1.1.198>: <command hidden> 
Jun  4 11:20:39  [3799] <PexpectAdapter: root@1.1.1.125>: VERT_DBA_USR=dbadmin VERT_DBA_HOME=/home/dbadmin VERT_DBA_GRP=verticadba /opt/vertica/oss/python/bin/python -m vertica.local_coerce 
Jun  4 11:20:39  [3799] <BashAdapter: root@1.1.1.191>: ['VERT_DBA_USR=dbadmin VERT_DBA_HOME=/home/dbadmin VERT_DBA_GRP=verticadba /opt/vertica/oss/python/bin/python -m vertica.local_coerce'] 
Jun  4 11:20:39  [3799] <PexpectAdapter: root@1.1.1.198>: VERT_DBA_USR=dbadmin VERT_DBA_HOME=/home/dbadmin VERT_DBA_GRP=verticadba /opt/vertica/oss/python/bin/python -m vertica.local_coerce 
Jun  4 11:20:40  [3799] <BashAdapter: root@1.1.1.191>: (rc=255) ['Traceback (most recent call last):', '  File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 162, in _run_module_as_main', '    "__main__", fname, loader, pkg_name)', '  File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 72, in _run_code', '    exec code in run_globals', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 204, in <module>', '    _main()', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 196, in _main', '    coerce.run()', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 76, in run', '    halted = self._run_to_halt()', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 81, in _run_to_halt', '    if self._iterate(x):', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 113, in _iterate', '    configure.load()', '  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/platform/fix/runner.py", line 49, in load', '    self._report.load_results(self._infile)', '  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/platform/check/report.py", line 59, in load_results', '    tree = xml.etree.ElementTree.parse(xmlfile)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 1176, in parse', '    tree.parse(source, parser)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 653, in parse', '    parser.feed(data)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 1623, in feed', '    self._raiseerror(v)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 1487, in _raiseerror', '    raise err', 'xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 122, column 34'] 
Jun  4 11:20:41  [3799] <PexpectAdapter: root@1.1.1.198>: (rc=255) ['Traceback (most recent call last):', '  File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 162, in _run_module_as_main', '    "__main__", fname, loader, pkg_name)', '  File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 72, in _run_code', '    exec code in run_globals', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 204, in <module>', '    _main()', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 196, in _main', '    coerce.run()', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 76, in run', '    halted = self._run_to_halt()', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 81, in _run_to_halt', '    if self._iterate(x):', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 113, in _iterate', '    configure.load()', '  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/platform/fix/runner.py", line 49, in load', '    self._report.load_results(self._infile)', '  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/platform/check/report.py", line 59, in load_results', '    tree = xml.etree.ElementTree.parse(xmlfile)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 1176, in parse', '    tree.parse(source, parser)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 653, in parse', '    parser.feed(data)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 1623, in feed', '    self._raiseerror(v)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 1487, in _raiseerror', '    raise err', 'xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 122, column 34'] 
Jun  4 11:20:41  [3799] <PexpectAdapter: root@1.1.1.125>: (rc=255) ['Traceback (most recent call last):', '  File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 162, in _run_module_as_main', '    "__main__", fname, loader, pkg_name)', '  File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 72, in _run_code', '    exec code in run_globals', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 204, in <module>', '    _main()', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 196, in _main', '    coerce.run()', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 76, in run', '    halted = self._run_to_halt()', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 81, in _run_to_halt', '    if self._iterate(x):', '  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 113, in _iterate', '    configure.load()', '  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/platform/fix/runner.py", line 49, in load', '    self._report.load_results(self._infile)', '  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/platform/check/report.py", line 59, in load_results', '    tree = xml.etree.ElementTree.parse(xmlfile)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 1176, in parse', '    tree.parse(source, parser)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 653, in parse', '    parser.feed(data)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 1623, in feed', '    self._raiseerror(v)', '  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 1487, in _raiseerror', '    raise err', 'xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 122, column 34'] 
Jun  4 11:20:41  [3799] <PexpectAdapter: root@1.1.1.125>: closeConnection 
Jun  4 11:20:41  [3799] <PexpectAdapter: root@1.1.1.198>: closeConnection 

Daniel_Leybovic

))

>> Validating node and cluster prerequisites...
Error: Command failed on 'ipofserver':

write(1, "Error: Command failed on '167.14"..., 44Error: Command failed on '1.1.1.198':

needless to say 'ipofserver' = IP of server
Network configuration (public/private network) and/or may be wrong user.

Sunny_1

hi i have 3 node cluster with 6.0.2 vertica running, but i only get this error when i upgrade or install a fresh 7.0. 
I had removed the real ip address from the post and add the "ipofserver" and 1.1.1.198 stuff on the post. Not allowed to post real ip address

Sunny_1

communication between all of the servers is working, root and dbadmin has passwordless ssh between them. i have verfied that. 

Daniel_Leybovic

So probably its wrong user

I got same problem (identical by your logs ) and reason was - user. I installed with sudo as USER_A after it tried to add node/update_vertica but as root with no success. Recreated and reinstalled all keys, ssh password-less  for all world - nothing helped

Check  owner of EULA file  and try to update/upgrade as this user with sudo.

For example

$ ls -l /opt/vertica/config/d5415f948449e9d4c421b568f2411140.dat 
-rw-rw-r-- 1 daniel verticadba 67 2013-12-17 21:48 /opt/vertica/config/d5415f948449e9d4c421b568f2411140.dat

I hope it will help.

Daniel_Leybovic

>> I had removed the real ip address from the post
Actually not, look for bcast - 167.14X.XXX.255

Also Im afraid you caused to some problem that require a technical support help:

Jun  4 11:12:52  [2764] RPM Check,  The same version is already installed 

All your next tries Im afraid will fail.

Sunny_1

Looks like LVM's are not supported with 7.0 vertica, 6.0 is running fine on VM's with LVM but 7.0 requirments doc says LVM is not supported. So i guess that is the root of my problem. 

Sunny_1

-rw-r--r-- 1 root root 9896 Apr 30 12:32 d237f83d0a61c3594829a574c63530b.dat ..root is owner of the file

Sunny_1

found this under /opt/vertica/log/local-coerce-20140610_101306/iter01

<failure eid="S0020" help="https://my.vertica.com/docs/7.0.x/HTML/index.htm#cshid=S0020"; message="Readahead size of dm-1 (/dev/mapper/system-home) is too low for typical systems: 256 &lt; 2048 Readahead size of dm-0 (/dev/mapper/system-root) is too low for typical systems: 256 &lt; 2048" severity="FAIL" severitystr="FAIL" type="VerticaAssertionError.vertica.platform.check.assertions">Traceback (most recent call last):  File &quot;/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/shared/classrunner.py&quot;, line 150, in _inner_run
    out = method(*args, **kwargs)
  File &quot;/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/platform/node/t/disk.py&quot;, line 192, in check_readahead
    eid=&quot;S0020&quot;, sev = Severity.FAIL)
  File &quot;/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/platform/check/assertions.py&quot;, line 88, in vassert
    raise VerticaAssertionError(msg, **kwargs)
VerticaAssertionError: FAIL(eS0020): Readahead size of dm-1 (/dev/mapper/system-home) is too low for typical systems: 256 &lt; 2048 Readahead size of dm-0 (/dev/mapper/system-root) is too low for typical systems: 256 &lt; 2048
</failure>

Daniel_Leybovic

Hi!

What is output of:

sudo /sbin/blockdev --report

?

Sunny_1

[root@iphvste001g134 IMDataRepository_vertica7]# sudo /sbin/blockdev --report RO RA SSZ BSZ StartSec Size Device rw 2048 512 4096 0 83886080 /dev/sda rw 2048 512 1024 63 1044162 /dev/sda1 rw 2048 512 2048 1044225 82831140 /dev/sda2 [root@iphvste001g134 IMDataRepository_vertica7]#

Daniel_Leybovic

Hm ...

Can you try same as here? - https://community.vertica.com/vertica/topics/error_when_creating_database_after_clean_install_of_ver...
i.e define "--control-network" and apply a patch.

---
Vertica 6 does not supports in LVM too.
https://community.vertica.com/vertica/topics/why_does_vertica_6_not_support_lvm
https://community.vertica.com/vertica/topics/why_does_vertica_6_not_support_lvm-afas2
so I hardly recommend to reconfigure your cluster.
---
>>root is owner of the file
so do everything as root, not as sudoer user.

Sajan_Sangraula

Hi Sunny,
What happens when you run the following as root user?
/opt/vertica/oss/python/bin/python -m vertica.local_coerce

Your initial post seems to indicate this step fails in 3 nodes of your cluster. The execution log of just this command should be easier to debug than the complete stacktrace.

/Sajan

Daniel_Leybovic

Hi Sajan!

>> Your initial post seems to indicate this step fails in 3 nodes of your cluster.
Exactly, it fails on other nodes. I thought its a permissions issue, but now I'm pretty sure that its problem in script. Take a look on output of Topic Starter - he got a lot of garbage in output = \x1b[H\x1b[2J. Actually its ANSI  escape characters: "[2J" - a last one means clear screen. So problem that script reads not properly from tty/pty.

20140604 102817:INFO:vertica.platform.node.t.dbadmin.CheckDbAdmin:LANG = 'en_US.UTF-8\n\x1b[H\x1b[2J'

20140604 093143:INFO:vertica.platform.node.t.dbadmin.CheckDbAdmin:LANG = 'en_US.UTF-8\n\x1b[H\x1b[2J'

20140604 093143:INFO:vertica.platform.node.t.dbadmin.CheckDbAdmin:TZ = '\x1b[H\x1b[2J'

http://ascii-table.com/ansi-escape-sequences.php

Sunny_1

 /opt/vertica/oss/python/bin/python -m vertica.local_coerce
Traceback (most recent call last):
  File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/opt/vertica/oss/python/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 204, in <module>
    _main()
  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 196, in _main
    coerce.run()
  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 76, in run
    halted = self._run_to_halt()
  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 81, in _run_to_halt
    if self._iterate(x):
  File "/opt/vertica/share/eggs/vertica/local_coerce.py", line 113, in _iterate
    configure.load()
  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/platform/fix/runner.py", line 49, in load
    self._report.load_results(self._infile)
  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/platform/check/report.py", line 59, in load_results
    tree = xml.etree.ElementTree.parse(xmlfile)
  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 1176, in parse
    tree.parse(source, parser)
  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 653, in parse
    parser.feed(data)
  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 1623, in feed
    self._raiseerror(v)
  File "/opt/vertica/oss/python/lib/python2.7/xml/etree/ElementTree.py", line 1487, in _raiseerror
    raise err
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 113, column 34

Jeffrey_Pinard

Seeing same stack trace at a customer installing 7.0.1 fresh and seeing similar garbage.  Customer running RedHat Enterprise 5 U10.

20140630 132419:INFO:vertica.platform.node.t.dbadmin.CheckDbAdmin:LANG = 'en_US.UTF-8\n\x1b[H\x1b[2J'20140630 132419:INFO:vertica.platform.node.t.dbadmin.CheckDbAdmin:TZ = '\x1b[H\x1b[2J'