Seeing Broken pipe errors after upgrading the Vertica nodes from Linux 6 to 7.

`Caused by: java.sql.SQLRecoverableException: [Vertica]VJDBC IOException while communicating with server: java.net.SocketException: Broken pipe (Write failed).
at com.vertica.io.ProtocolStream.logAndConvertToNetworkException(Unknown Source)
at com.vertica.io.ProtocolStream.flush(Unknown Source)
at com.vertica.core.VConnection.handleError(Unknown Source)
at com.vertica.dataengine.VDataEngine.prepareImpl(Unknown Source)
at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
at com.vertica.jdbc.common.SStatement.executeNoParams(Unknown Source)
at com.vertica.jdbc.common.SStatement.executeUpdate(Unknown Source)
at com.xx.management.bigdata.common.vertica.VerticaTableSQL.finalizeTable(VerticaTableSQL.java:430)
at com.xx.management.bigdata.common.vertica.VerticaTableSQL.finalizeTable(VerticaTableSQL.java:379)
at com.xx.management.bigdata.common.vertica.VerticaTableSQL.finalizeTables(VerticaTableSQL.java:328)
at com.xx.management.wolfe.hadoop.crunch.job.CrunchJob.finalize(CrunchJob.java:436)
at com.xx.management.wolfe.hadoop.crunch.job.CrunchJob.execute(CrunchJob.java:390)
at com.xx.management.wolfe.hadoop.crunch.job.CrunchJob.executeJob(CrunchJob.java:220)
at com.xx.management.vertica.job.ESMJobManager.executeJob(ESMJobManager.java:88)
... 9 more
Caused by: com.vertica.support.exceptions.RecoverableException: [Vertica]VJDBC IOException while communicating with server: java.net.SocketException: Broken pipe (Write failed).
... 24 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at com.vertica.io.VStream.flush(Unknown Source)
... 23 more

Build step 'Execute shell' marked build as failure
Finished: FAILURE`

Tagged:

Answers

  • Bryan_HBryan_H Vertica Employee Administrator

    What specifically causes the exception? That is, was it on connect, or a query, or something else like a prepared statement?
    Do other clients (JDBC, vsql, etc.) still work from the same environment?
    Was the JDK upgraded also during the OS upgrade? Did any OS settings change, such as firewall or iptables?

  • gjabrajgjabraj Vertica Customer

    We have hadoop jobs Inserting data to Vertica. If the job runs over 1 hour or more it fails with above error.

    No JDK is installed on the Vertica nodes.

    No firewalls/ACL's present on the Vertica cluster.

  • SruthiASruthiA Vertica Employee Administrator

    is it copy from HDFS which is failing?

  • gjabrajgjabraj Vertica Customer

    @SruthiA said:
    is it copy from HDFS which is failing?

    Yes. HDFS data is inserted to Vertica.

  • SruthiASruthiA Vertica Employee Administrator

    how are you connecting to vertica and issuing copy from HDFS queries? is it via JDBC? If so, could you please run one long running COPY statement via vsql to see if completes successfully?

  • gjabrajgjabraj Vertica Customer

    @SruthiA said:
    how are you connecting to vertica and issuing copy from HDFS queries? is it via JDBC? If so, could you please run one long running COPY statement via vsql to see if completes successfully?

    JDBC being used is vertica-jdbc-8.1.1-18.jar.

  • gjabrajgjabraj Vertica Customer

    @SruthiA : Will upgrading the vertica-jdbc to higher version help??

  • SruthiASruthiA Vertica Employee Administrator

    @gjabraj issue looks like connection time out based on the errors. please run one long running COPY statement via vsql to see if completes successfully. it will confirm that issue is with long running jobs via JDBC. do you see any errors in vertica log when it fails with error?

  • gjabrajgjabraj Vertica Customer

    @SruthiA : Looks like vsql does not have an issue being idle over an hour. Our next plan is to upgrade vertica-jdbc jar to latest. Please let us know otherwise.

  • SruthiASruthiA Vertica Employee Administrator

    @gjabraj Thank you for confirming. Please try using the latest JDBC driver and see if it helps. if there is any load balancer between vertica and your client using JDBC driver try check if that is disconnecting the idle sessions

  • KennethAmanKennethAman Community Edition User
    edited October 2022

    @SruthiA said:
    how are you connecting to vertica and issuing copy from HDFS queries? is it via JDBC? If so, could you please run one long running COPY statement via vsql to see if completes successfully?

    JDBC being used is vertica-jdbc-8.1.1-18.jar

  • gjabrajgjabraj Vertica Customer

    @SruthiA : After updating the jdbc driver to latest version we are still seeing the same issue.

  • SruthiASruthiA Vertica Employee Administrator

    @gjabraj : It looks like we need to review it on the webEx. Could you please open a support case? what is the vertica version.

  • gjabrajgjabraj Vertica Customer

    Current Vertica version is 10.1.x. Upgrade to 11.x scheduled next week.

    Support case will be logged soon.

  • SruthiASruthiA Vertica Employee Administrator

    When you upgrade to 11.x and if you have changed keepalive settings at OS level, make sure you change it in vertica DB.. we introduced new config parameters for keepalive settings in 11.x

    https://www.vertica.com/docs/11.1.x/HTML/Content/Authoring/AdministratorsGuide/ManagingClientConnections/OverviewClientConnections.htm

  • gjabrajgjabraj Vertica Customer

    @SruthiA : Logged this case 02428942

  • SruthiASruthiA Vertica Employee Administrator

    @gjabraj : Thank you. Assigned engineer will get back to you soon.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file