The Vertica Forum recently got a makeover! Let us know what you think by filling out this short, anonymous survey.
Please take this survey to help us learn more about how you use third party tools. Your input is greatly appreciated!

Partial data load from Vertica to Spark

One of my customers wants to know if they can load partial data into Spark from a vertica table using select.

Is there anyway we can load partial data from the table using a SQL Query? We don’t want to load the whole table to our Spark app.

Dataset ds = spark.read().format("com.vertica.spark.datasource.DefaultSource")
.option("user", "...")
.option("password", "...")
.option("driver", "com.vertica.jdbc.Driver")
.option("dbschema", "PDW_ERIEUTRAN_VIEWS")
.option("table", "RBS_EUTRANCELLFDD1") // DON’T WANT to load the whole table!!!!
.option("db", "PDW")
//.option("url", "jdbc:vertica://***:5433/SRVVERTICA")
.option("host", "verticapride-brhmal.it.att.com")
.load();

They have restricted access to views and their views are very large as well.

Any pointers?

thanks

Comments

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file