vertica spark connector

ira

Hello,

 

I am currently playing arround with vertica 7.2 and I wanted to access some data via Spark.

There where some announcements that vertica should work with spark and I've seen several posts saying that there's a vertica-spark connector available here:
https://saas.hpe.com/marketplace/big-data/hpe-vertica-connector-apache-spark

 

but apperently the connector has moved. I' ve only found the hadoop connector so far.

Any pointers to the spark connector would be great!

 

Thanks in advance! 

Ira

kaurora

Hi Ira, 

 

From what I have heard, the spark connector isn't available right now since it's undergoing some bug fixes/performance improvement. 

ira

Hi Kaurora,

 

thank you a lot for the reply! I see, but isn't there the old version available somewhere?

For now I didn't have any luck googling for it. 

 

Thanks a lot!

 

Sam_M

Hi Ira and others, are there any updates on the expected re-release of the connector, or previous versions?  I see some recent activity on https://community.dev.hpe.com/t5/Vertica-Forum/Loading-spark-dataframe-into-vertica/td-p/233482.

 

Best,

Sam

Schneider315

Any update on this? Even if there is unstable version?

ira

Hi Sam and Schneider315,

 

For now I have used the SparkSql module and jdbc to read and write to Vertica.

As you've probably already seen here https://community.dev.hpe.com/t5/Vertica-Forum/Loading-spark-dataframe-into-vertica/td-p/233482 .

No news on the connector on my side - I'd be glad to get an update on that too.

 

Thanks,

Ira

jsonencode

Hello Ira,

 

I tried your solution it worked but only for dataframe that doesnt contain NULL in any of the rows. It errors out when there is a null present. Did you face the same issue? Do you hve any solutions in mind?

 

Regards,

JSON

xyin

Hi ira,

 

How are you connecting sparksql and Vertica with jdbc? What kind of jdbc driver are you using?

 

Thanks!

[Deleted User]

Hi,

 

For 8.0 there is a connector available for download on my.vertica.com downloads.

https://my.vertica.com/download/vertica/8-0-x/

Under the "Connector section" towards the bottom of the page.

 

And the documentation is here: 

https://my.vertica.com/docs/8.0.x/HTML/index.htm#Authoring/SparkConnector/ApacheSparkGuide.htm%3FTocPath%3DIntegrating%2520with%2520Apache%2520Spark%7C_____0

Thanks,
Mark

 

HPE Product Management

ira

Hi,

 

you can get the jdbc driver from the vertica download site (https://my.vertica.com/download/vertica/client-drivers/) and then you can acces your data like this:

String theQueryToGetTheData = "Select column1, column2 from MyTable";
String myconnectionurl = "jdbc:vertica://127.0.0.1:5433/DBNAME?user=myusr&password=mypass";

Dataset<Row> df = session.read().format("jdbc").option("driver", "com.vertica.jdbc.Driver").option("urll", myconnectionurl)
				.option("dbtable", theQueryToGetTheData).load();

You have to make sure that the jdbc jar is available to spark, either package it in your application jar or provide it via the --jars option.

 

Concerning the null issue during the writing of dataframes I started playing with a custom sql dialect but I think the connector mentioned by Mark should help.

Haven't tried it yet, but it seems that it supports only Spark 1.6.

 

Cheers,

Ira