The Vertica Forum recently got a makeover! Let us know what you think by filling out this short, anonymous survey.
Please take this survey to help us learn more about how you use third party tools. Your input is greatly appreciated!

Verica Write Operation from Spark not working

Hi Team,

I want to write a data frame in vertica table through spark. I have the following options:

1: Using Spark Vertica connector I get the error :
py4j.protocol.Py4JJavaError: An error occurred while calling
java.lang.NoSuchMethodError: java.lang.String.isBlank()Z
This can be due to the version im using , my versions are :
Spark : Spark 3.3.1
Vertica : Vertica Analytic Database v9.2.1-28
Java : openjdk version "1.8.0_372"

2: Using JDBC : I tried with many JDBC drivers Im always able to read the table but when I perform the write operation it gives me error :
java.sql.SQLFeatureNotSupportedException: [Vertica]JDBC Driver does not support this optional feature.
Code : df.write \
.jdbc(jdbc_url, table_name, mode="append", properties=properties)

3: vertica_python Im able to write the data into table using INSERT into command but I don't want to do this as I have huge dataframe and want to load the data in batch

Let me know if anyone has idea how to connect vertica with spark


  • VValdarVValdar Vertica Employee Employee
    edited September 15

    Hi haidernaveed_100,

    Vertica 9.2 is a five year old version.
    The spark connector asks for at least Vertica 10.1.1 (which is already a two and half year old version):

    Getting Started
    To get started with using the connector, we'll need to make sure all the prerequisites are in place. These are:

    • Vertica (10.1.1-0 or higher)
    • Spark (3.0.0 or higher)
    • An HDFS cluster or HDFS-compatible filesystem (S3, Google Cloud Storage, etc), for use as an intermediary between Spark and Vertica
    • A Spark application, either running locally for quick testing, or running on a Spark cluster. If using S3, Spark must be using hadoop 3.3

    I would start there first!

  • Bryan_HBryan_H Vertica Employee Administrator
    edited September 15

    java.lang.String.isBlank() method was added in Java 11. You'll need a newer runtime. (edit) This is fixed in source (issue#534( but not in a release build.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file