We're Moving!

The Vertica Forum is moving to a new OpenText Analytics Database (Vertica) Community.

Join us there to post discussion topics, learn about

product releases, share tips, access the blog, and much more.

Create My New Community Account Now


vertica spark connector issue — Vertica Forum

vertica spark connector issue

I'm trying to write spark DataSet to vertica with the latest spark connector. But can't create the BaseRelation with below error.

 

Exception in thread "streaming-job-executor-1" java.lang.AbstractMethodError: com.vertica.spark.datasource.DefaultSource.createRelation(Lorg/apache/spark/sql/SQLContext;Lorg/apache/spark/sql/SaveMode;Lscala/collection/immutable/Map;Lorg/apache/spark/sql/Dataset;)Lorg/apache/spark/sql/sources/BaseRelation;
at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:442)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:211)

 

This is the connection code I used:

 

Dataset<Row> employeeDataFrame = spark.createDataFrame(employeeRecordJavaRDD, Employee.class);
employeeDataFrame.schema();
employeeDataFrame.show();

employeeDataFrame.toDF().write()
.format("com.vertica.spark.datasource.DefaultSource")
.option("table", "employee")
.option("db", "hc")
.option("user", "dbadmin")
.option("password", "xx")
.option("host", "10.3.x.1x")
.option("hdfs_url", "hdfs://hadoop:9000/user/hadoop/hc")
.option("web_hdfs_url", "webhdfs://hadoop:50070/user/hadoop/hc")
.option("dbschema", "hce")
.option("port", "5433")
.mode(SaveMode.Append).save();

I also tried

 

employeeDataFrame.write()

 Got the same error.

 

Could anyone help to point out what I did wrong? Thanks.

Comments

  • Solved. Current version of connector only support spark 1.6. I was using spark 2.0

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file