The Vertica Forum recently got a makeover! Let us know what you think by filling out this short, anonymous survey.
Please take this survey to help us learn more about how you use third party tools. Your input is greatly appreciated!

Spark DF to Vertica : "Driver not capable"

I am trying to write an Spark DataFrame of Rows to Vertica via JDBC Function
in the following manner:

dataframe.write().mode(SaveMode.Append).jdbc(url, table, properties);

This works when there are no NULL values in any of the rows. When
there are NULL values in rows, I am getting the following error:

17/06/27 14:51:12 INFO orc.RecordReaderFactory: Schema is not specified on read. Using file schema.
org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 1.0 failed 1 times, most recent failure: Lost task 2.0 in stage 1.0 (TID 6, localhost): java.sql.SQLFeatureNotSupportedException: [Vertica]JDBC Driver not capable.
at com.vertica.exceptions.ExceptionConverter.toSQLException(Unknown Source)
at com.vertica.jdbc.common.SPreparedStatement.checkTypeSupported(Unknown Source)
at com.vertica.jdbc.common.SPreparedStatement.setNull(Unknown Source)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:181)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:277)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:276)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$35.apply(RDD.scala:927)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$35.apply(RDD.scala:927)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1881)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1881)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.executor.Executor$
at java.util.concurrent.ThreadPoolExecutor.runWorker(
at java.util.concurrent.ThreadPoolExecutor$

My environment: Spark 1.6.1, Vertica Vertica Analytic Database v8.1.0-3, Driver vertica-8.1.0_spark1.6_scala2.10.jar, vertica-jdbc-8.1.0-3.jar

Any suggestion how to solve this issue? Thanks


  • Vinoth78Vinoth78 Vertica Customer
    I am also getting the same error, but in my case null at character columns is causing the problem. If i have null at columns with numeric datatype, i am able to append them to vertica table.
  • Vinoth78Vinoth78 Vertica Customer
    Then i used"",Array(col1,col2,col3)).write().mode(SaveMode.Append).jdbc(url, table, properties)
    It worked.
  • LenoyJLenoyJ - Select Field - Employee
    edited June 2020

    Also, this thread and this comment in particular for creating a custom dialect that handles TEXT nulls better.

  • Bryan_HBryan_H Vertica Employee Administrator

    Hi stellaa9x, what version of Spark and Vertica are you running - same as above or hopefully newer? The solution options are a bit different depending on versions.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file