Verica Write Operation from Spark not working

haidernaveed_100

Hi Team,

I want to write a data frame in vertica table through spark. I have the following options:

• Spark Vertica connector ( https://github.com/vertica/spark-connector/releases )
• Using JDBC ( https://www.vertica.com/download/vertica/client-drivers/ )
• vertica_python module ( sequential )

1: Using Spark Vertica connector I get the error :

py4j.protocol.Py4JJavaError: An error occurred while calling o132.save.

java.lang.NoSuchMethodError: java.lang.String.isBlank()Z
This can be due to the version im using , my versions are :
Spark : Spark 3.3.1
Vertica : Vertica Analytic Database v9.2.1-28
Java : openjdk version "1.8.0_372"

2: Using JDBC : I tried with many JDBC drivers Im always able to read the table but when I perform the write operation it gives me error :
java.sql.SQLFeatureNotSupportedException: [Vertica]JDBC Driver does not support this optional feature.
Code : df.write \
.jdbc(jdbc_url, table_name, mode="append", properties=properties)

3: vertica_python Im able to write the data into table using INSERT into command but I don't want to do this as I have huge dataframe and want to load the data in batch

Let me know if anyone has idea how to connect vertica with spark

VValdar

Hi haidernaveed_100,

Vertica 9.2 is a five year old version.
The spark connector asks for at least Vertica 10.1.1 (which is already a two and half year old version):

Getting Started
To get started with using the connector, we'll need to make sure all the prerequisites are in place. These are:

• Vertica (10.1.1-0 or higher)
• Spark (3.0.0 or higher)
• An HDFS cluster or HDFS-compatible filesystem (S3, Google Cloud Storage, etc), for use as an intermediary between Spark and Vertica
• A Spark application, either running locally for quick testing, or running on a Spark cluster. If using S3, Spark must be using hadoop 3.3

I would start there first!

Bryan_H

java.lang.String.isBlank() method was added in Java 11. You'll need a newer runtime. (edit) This is fixed in source (issue#534( but not in a release build.