Vertica connector for Spark

[Deleted User]

Hey,

I want to integrate a Spark job with a Vertica DB and used the hpe-spark-connector.

It worked when I used spark 1.6, but now we want to upgrade to 2.1 and I've noticed that the connector doesn't support that version of Spark yet.

 

I wanted to know if there is planned release to upgrade the connector to work with spark 2.0+?

And I've also noticed it's pretty hard to even find the connector jar, is the project discontinued for now?

 

Thanks in advance for anyone helping me

jlu

Hello,
have you tried to write your final spark dataframe into parquet or orc files and read them with vertica ?

https://my.vertica.com/docs/8.0.x/HTML/index.htm#Authoring/HadoopIntegrationGuide/NativeFormats/ReadingNativeFormats.htm

jlu

It seems that the best case scenario is that the connector will be shipped with vertica 8.1 in the coming couple of months

Connyrt-emp1

Thanks for posting your question and for your feedback. We hope to support Spark 2.0 with Scala 2.11 on the next Vertica release, a new jar would be available on my.vertica.com. Please make sure to download the correct jar for your environment, the name of the connector specifies the versions of vertica, spark and scala that should be used.