Seeing Broken pipe errors after upgrading the Vertica nodes from Linux 6 to 7.
gjabraj
Vertica Customer
`Caused by: java.sql.SQLRecoverableException: [Vertica]VJDBC IOException while communicating with server: java.net.SocketException: Broken pipe (Write failed).
at com.vertica.io.ProtocolStream.logAndConvertToNetworkException(Unknown Source)
at com.vertica.io.ProtocolStream.flush(Unknown Source)
at com.vertica.core.VConnection.handleError(Unknown Source)
at com.vertica.dataengine.VDataEngine.prepareImpl(Unknown Source)
at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
at com.vertica.jdbc.common.SStatement.executeNoParams(Unknown Source)
at com.vertica.jdbc.common.SStatement.executeUpdate(Unknown Source)
at com.xx.management.bigdata.common.vertica.VerticaTableSQL.finalizeTable(VerticaTableSQL.java:430)
at com.xx.management.bigdata.common.vertica.VerticaTableSQL.finalizeTable(VerticaTableSQL.java:379)
at com.xx.management.bigdata.common.vertica.VerticaTableSQL.finalizeTables(VerticaTableSQL.java:328)
at com.xx.management.wolfe.hadoop.crunch.job.CrunchJob.finalize(CrunchJob.java:436)
at com.xx.management.wolfe.hadoop.crunch.job.CrunchJob.execute(CrunchJob.java:390)
at com.xx.management.wolfe.hadoop.crunch.job.CrunchJob.executeJob(CrunchJob.java:220)
at com.xx.management.vertica.job.ESMJobManager.executeJob(ESMJobManager.java:88)
... 9 more
Caused by: com.vertica.support.exceptions.RecoverableException: [Vertica]VJDBC IOException while communicating with server: java.net.SocketException: Broken pipe (Write failed).
... 24 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at com.vertica.io.VStream.flush(Unknown Source)
... 23 more
Build step 'Execute shell' marked build as failure
Finished: FAILURE`
Answers
What specifically causes the exception? That is, was it on connect, or a query, or something else like a prepared statement?
Do other clients (JDBC, vsql, etc.) still work from the same environment?
Was the JDK upgraded also during the OS upgrade? Did any OS settings change, such as firewall or iptables?
We have hadoop jobs Inserting data to Vertica. If the job runs over 1 hour or more it fails with above error.
No JDK is installed on the Vertica nodes.
No firewalls/ACL's present on the Vertica cluster.
is it copy from HDFS which is failing?
Yes. HDFS data is inserted to Vertica.
how are you connecting to vertica and issuing copy from HDFS queries? is it via JDBC? If so, could you please run one long running COPY statement via vsql to see if completes successfully?
JDBC being used is vertica-jdbc-8.1.1-18.jar.
@SruthiA : Will upgrading the vertica-jdbc to higher version help??
@gjabraj issue looks like connection time out based on the errors. please run one long running COPY statement via vsql to see if completes successfully. it will confirm that issue is with long running jobs via JDBC. do you see any errors in vertica log when it fails with error?
@SruthiA : Looks like vsql does not have an issue being idle over an hour. Our next plan is to upgrade vertica-jdbc jar to latest. Please let us know otherwise.
@gjabraj Thank you for confirming. Please try using the latest JDBC driver and see if it helps. if there is any load balancer between vertica and your client using JDBC driver try check if that is disconnecting the idle sessions
JDBC being used is vertica-jdbc-8.1.1-18.jar
@SruthiA : After updating the jdbc driver to latest version we are still seeing the same issue.
@gjabraj : It looks like we need to review it on the webEx. Could you please open a support case? what is the vertica version.
Current Vertica version is 10.1.x. Upgrade to 11.x scheduled next week.
Support case will be logged soon.
When you upgrade to 11.x and if you have changed keepalive settings at OS level, make sure you change it in vertica DB.. we introduced new config parameters for keepalive settings in 11.x
https://www.vertica.com/docs/11.1.x/HTML/Content/Authoring/AdministratorsGuide/ManagingClientConnections/OverviewClientConnections.htm
@SruthiA : Logged this case 02428942
@gjabraj : Thank you. Assigned engineer will get back to you soon.