The Vertica Forum recently got a makeover! Let us know what you think by filling out this short, anonymous survey.
Please take this survey to help us learn more about how you use third party tools. Your input is greatly appreciated!
Cannot start database after upgrade vertica from 9.0.1-7 to 9.1.1
We've tried to upgrade from 9.0.1-7 to 9.1.1. The upgrade task was very smooth. But we can't start DB after the upgrade. We got the error below in /opt/vertica/log/adminTools.log same error on all the nodes we have. And it doesn't have any things print to startup.log and vertica.log. BTW During the upgrade, we've opened a case to Vertica Support. They've tried to do some troubleshooting. Finally we decided to rollback to 9.0.1-7. And the database can start UP as usual. Any suggestions are welcome?
Regards,
BoMBaY
Error in /opt/vertica/log/adminTools.log about "Vertica Catalog Editor: broken pipe detected"
2018-08-16 18:47:54.827 at_exec/45050:0x7f0cbc6a9740 [root.setup_custom_logging] <INFO> New log for 'at_exec' 2018-08-16 18:47:54.827 at_exec/45050:0x7f0cbc6a9740 [root.setup_custom_logging] <INFO> sys.argv: '/opt/vertica/share/eggs/vertica/engine/api/at_runner.py' --no-prompt --module 'vertica.engine.api.compute_vdatabase' 2018-08-16 18:47:54.828 at_exec/45050:0x7f0cbc6a9740 [ATRunner._parse_command] <INFO> Reading a line from stdin... 2018-08-16 18:47:54.828 at_exec/45050:0x7f0cbc6a9740 [ATRunner._parse_command] <INFO> Reading complete: [#{u'__rollback__': '********', u'__protocol__': '********', u'catalogpath': u'/xxxxx/xxxxx/xxxxxx_nodexxxx_catalog'}] 2018-08-16 18:47:54.829 at_exec/45050:0x7f0cbc6a9740 [compute_vdatabase.__init__] <INFO> Initialized new instance of compute_vdatabase 2018-08-16 18:47:54.829 at_exec/45050:0x7f0cbc6a9740 [ATRunner.exec_module] <INFO> ATRunner exec_module: command: <ATCommand> module=None version=1.0 args={u'catalogpath': u'/xxxxx/xxxxx/xxxxxx_nodexxxx_catalog'} 2018-08-16 18:47:54.829 at_exec/45050:0x7f0cbc6a9740 [CatalogEditor.__init__] <INFO> Running catalog editor command: ['/opt/vertica/bin/vertica', '-D', u'/xxxxx/xxxxx/xxxxxx_nodexxxx_catalog', '-E', '-z'] 2018-08-16 18:47:54.831 at_exec/45050:0x7f0cbc6a9740 [CatalogEditor._parse_header] <INFO> Parsing CE header 2018-08-16 18:47:54.831 at_exec/45050:0x7f0cbc6a9740 [CatalogEditor._recorded_readline] <INFO> About to call readline on catalog editor output 2018-08-16 18:47:54.857 at_exec/45050:0x7f0cbc6a9740 [CatalogEditor._recorded_readline] <INFO> Next line of response was [''] 2018-08-16 18:47:54.857 at_exec/45050:0x7f0cbc6a9740 [CatalogEditor._censor_then_log] <INFO> Sending to catalog editor: ['get singleton Database name\n'] 2018-08-16 18:47:54.857 at_exec/45050:0x7f0cbc6a9740 [CatalogEditor.sendCmd] <ERROR> Exception encountered while running catalog editor Traceback (most recent call last): File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/tools/CatalogEditor.py", line 120, in sendCmd self.ceproc.stdin.write(cmd_string) IOError: [Errno 32] Broken pipe 2018-08-16 18:47:54.857 at_exec/45050:0x7f0cbc6a9740 [CatalogEditor.sendCmd] <ERROR> Vertica Catalog Editor: broken pipe detected Added stdout messages to error. BEGIN CE output END CE output 2018-08-16 18:47:54.857 at_exec/45050:0x7f0cbc6a9740 [ATRunner.exec_module] <ERROR> command got exception: Could not load from Catalog Editor. Catalog Editor state CatalogEditor instance Closed? False CE subprocess = <subprocess.Popen object at 0x7f0cb476ef50> CE proc.poll = -11 CE proc.stdin = <open file '<fdopen>', mode 'wb' at 0x7f0cb4801c00> CE proc.stdout = <open file '<fdopen>', mode 'rb' at 0x7f0cb4801b70> Exception was Broken pipe Vertica Catalog Editor: broken pipe detected Added stdout messages to error. BEGIN CE output END CE output
0
Comments
Webex with Vertica Support summary:
Webex Vertica Support Summary:
Can you post editor.log from the _catalog directory?
@Ben_Vandiver There is not things write to editor.log.
Today, we tried to upgrade from 9.0.1-7 to 9.1.0-5 and got the same issue. But we upgraded from 9.0.1-7 to 9.0.1-13 successfully. And then, we tried to upgrade from 9.0.1-13 to 9.1.0-5 and got the same issue. Now we rolled back to 9.0.1-13.
@Ben_Vandiver, If you can access Service Request, This Service Request ID is SD02266679.
@Itipong_Chewinp - Did you check that all projection buddies in the current database comply with the new requirements of 9.1?
See:
https://my.vertica.com/docs/9.1.x/HTML/index.htm#Authoring/NewFeatures/9.1/9.1.0/UpgradeandInstall.htm
Did you run the pre-upgrade script?
@Jim_Knicely, Yes, we've run the pre-upgrade script. The pre-upgrade script output is below.
@Itipong_Chewinp - Hmm. After the upgrade you can try and start the DB manually (i.e. without admintools). I attached a zip file that includes a script that should start the nodes in your cluster manually. Do you have a test environment where you can test it?
It is a bash shell script and you run it like the following example where "test_db" is the name of the DB I am trying to start:
@Jim_Knicely Thanks. I'll try.
@Jim_Knicely It doesn't work. It went to 'echo ...Test #$c' loop for 40 times. and during that i tried to grep "vertica -D" on all nodes but didn't found the process. one thing that different from start via the admintools is i found startup.log with 0 byte.
One more thing is we got a feedback from Vertica support. They've asked to try to running manually the catalog editor command "/opt/vertica/bin/vertica -D /xxxxx/xxxxx/xxxxx/v_xxxxx_node0001_catalog -E -z" but got an erorr "Segmentation fault".
Interesting. What OS are you running? Can you try to run the catalog editor on another node?
@Jim_Knicely, we use CentOS Linux release 7.4.1708 (Core). I got the same error "Segmentation fault" when run the catalog editor on another nodes.
Are you willing to send support a core file? Also 'ldd /opt/vertica/bin/vertica' and the content of editor.log if it exists.
@Itipong_Chewinp - Per Ben's request, here is what I see: