Verica Write Operation from Spark not working
I want to write a data frame in vertica table through spark. I have the following options:
- Spark Vertica connector ( https://github.com/vertica/spark-connector/releases )
- Using JDBC ( https://www.vertica.com/download/vertica/client-drivers/ )
- vertica_python module ( sequential )
- 1: Using Spark Vertica connector I get the error :
- py4j.protocol.Py4JJavaError: An error occurred while calling o132.save.
- java.lang.NoSuchMethodError: java.lang.String.isBlank()Z
This can be due to the version im using , my versions are :
Spark : Spark 3.3.1
Vertica : Vertica Analytic Database v9.2.1-28
Java : openjdk version "1.8.0_372"
2: Using JDBC : I tried with many JDBC drivers Im always able to read the table but when I perform the write operation it gives me error :
java.sql.SQLFeatureNotSupportedException: [Vertica]JDBC Driver does not support this optional feature.
Code : df.write \
.jdbc(jdbc_url, table_name, mode="append", properties=properties)
3: vertica_python Im able to write the data into table using INSERT into command but I don't want to do this as I have huge dataframe and want to load the data in batch
Let me know if anyone has idea how to connect vertica with spark