Please take this survey to help us learn more about how you use third party tools. Your input is greatly appreciated!

Load data from spark dataframe to vertica from AWS Glue

We are using Vertica version 9.2.1. AWS Glue as ETL tool. Trying to load the data from pyspark data frame to Vertica. Getting below error

An error occurred while calling
java.lang.Exception: ERROR: did not pass the Vertica requirements pre-check. The following problems were encountered: hdfs_url scheme should be 'hdfs', but user provided:null. hdfs_url path is not valid, user provided:. java.lang.IllegalArgumentException: Can not create a Path from an empty string

--data loading step"com.vertica.spark.datasource.DefaultSource", mode="append", **opts)

opts['dbschema'] = 'staging'
opts['table'] = 'fact_rating_aggregate_stage'

Glue service is serverless and brings servers on the fly and processes data. I have not given the hdfs_url as this is keep changing.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file

Can't find what you're looking for? Search the Vertica Documentation, Knowledge Base, or Blog for more information.