Error while writing to vertica from spark using Vertica spark connector

Exception in thread "main" java.lang.Exception: S2V: FATAL ERROR for job S2V_job1251056748245412653. Job status information is available in the Vertica table wcc.S2V_JOB_STATUS_USER_SAMPLE. Unable to create/insert into target table wcc.T_SAMPLE_TABLE with SaveMode: Append. ERROR MESSAGE: ERROR: java.sql.SQLException: [Vertica]VJDBC INTERNAL: VIAssert((size_t)pio[i].attnum <= loadStep->queryCols.size()) failed
[Vertica][VJDBC]Detail: /scratch_a/release/svrtar14870/vbuild/vertica/Optimizer/Path/BulkLoad.cpp: 1710
at com.vertica.spark.s2v.S2V.do2Stage(S2V.scala:339)
at com.vertica.spark.s2v.S2V.save(S2V.scala:389)
at com.vertica.spark.datasource.DefaultSource.createRelation(VerticaSource.scala:88)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:668)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:276)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:270)
at actualLoad.gspTranLoad$.main(gspTranLoad.scala:37)
at actualLoad.gspTranLoad.main(gspTranLoad.scala)

Comments

  • Can some one please help me? I tried to search this issue, but no single answer

  • SruthiASruthiA Vertica Employee Administrator

    It looks like a bug. please open a support case and share the logs.

  • That is a very odd error. Can you provide the Spark code you are running to get this error? what is the structure of the DF? what is the structure of your target table in Vertica? what parameters are you using? What version of the connector and Spark and Vertica and JDBC driver you are using?

  • Hi, Thank you very much for the response. Please see the details below.

    This is the command I am using for the load,
    dataFrame.write.format("com.vertica.spark.datasource.DefaultSource").options(opts).mode("Append").save()

    val opts: Map[String, String] = Map(
    "table" -> "T_SAMPLE_TABLE",
    "dbschema" -> "wcc",
    "db" -> "dbname",
    "user" -> "user",
    "password" -> "password",
    "host" -> "vertica-sample-dev.bnymellon.net",
    "hdfs_url" -> "hdfs://hiveServer.domain.net:9000/user/hive/warehouse",
    "web_hdfs_url" -> "webhdfs://hdfs://hiveServer.domain.net:50070/user/hive/warehouse"

  • Structure of a DataFrame
    Column1 String
    Column2 String
    Column3 String
    Column4 String
    Column5 String
    Column6 Integer
    Column7 String
    Column8 Integer
    Column9 Integer
    Column10 String
    Column11 Date
    Column12 Integer
    Column13 Integer
    Column14 String
    Column15 String
    Column16 Integer
    Column17 Integer
    Column18 Integer
    Column19 String
    Column20 String
    Column21 String
    Column22 String
    Column23 String
    Column24 String
    Column25 String
    Column26 String
    Column27 String
    Column28 Integer
    Column29 String
    Column30 Integer
    Column31 String
    Column32 String
    Column33 Integer
    Column34 String
    Column35 Timestamp
    Column36 String
    Column37 String
    Column38 String
    Column39 String
    Column40 String
    Column41 String
    Column42 String
    Column43 String
    Column44 Date
    Column45 String
    Column46 String
    Column47 String
    Column48 String
    Column49 Integer
    Column50 String
    Column51 String
    Column52 String
    Column53 String
    Column54 String
    Column55 String
    Column56 String
    Column57 String
    Column58 Timestamp
    Column59 Timestamp
    Column60 Timestamp
    Column61 String
    Column62 Integer
    Column63 Date
    Column64 String
    Column65 Date
    Column66 String
    Column67 String
    Column68 String
    Column69 String
    Column70 String
    Column71 String
    Column72 String
    Column73 Integer
    Column74 Date
    Column75 String
    Column76 String
    Column77 String
    Column78 String
    Column79 String
    Column80 String
    Column81 String
    Column82 String
    Column83 String
    Column84 String
    Column85 String
    Column86 String
    Column87 Integer
    Column88 Integer
    Column89 String
    Column90 Date
    Column91 String
    Column92 Integer
    Column93 String
    Column94 String
    Column95 String
    Column96 String
    Column97 Integer
    Column98 String
    Column99 Integer
    Column100 Integer
    Column101 String
    Column102 String
    Column103 String
    Column104 String
    Column105 Integer
    Column106 Integer
    Column107 Date
    Column108 String
    Column109 Integer
    Column110 Integer
    Column111 String
    Column112 String
    Column113 String
    Column114 Date
    Column115 String
    Column116 String
    Column117 String
    Column118 String
    Column119 String
    Column120 String
    Column121 String
    Column122 String
    Column123 Integer
    Column124 String
    Column125 String
    Column126 String
    Column127 String
    Column128 String
    Column129 String
    Column130 String
    Column131 Integer
    Column132 Integer
    Column133 String
    Column134 String
    Column135 String
    Column136 String
    Column137 String
    Column138 String
    Column139 String
    Column140 String
    Column141 Integer
    Column142 Timestamp
    Column143 String
    Column144 Date
    Column145 String
    Column146 String
    Column147 Integer
    Column148 String
    Column149 String
    Column150 String
    Column151 String
    Column152 Integer
    Column153 String
    Column154 String
    Column155 String
    Column156 String
    Column157 String
    Column158 String
    Column159 String
    Column160 Integer
    Column161 String
    Column162 String
    Column163 String
    Column164 String
    Column165 String
    Column166 String
    Column167 String
    Column168 String
    Column169 String
    Column170 Date
    Column171 String
    Column172 Date
    Column173 String
    Column174 String
    Column175 Timestamp
    Column176 Date
    Column177 String
    Column178 String
    Column179 String
    Column180 String
    Column181 String
    Column182 Integer
    Column183 String
    Column184 String
    Column185 String
    Column186 String
    Column187 Integer
    Column188 Integer
    Column189 String
    Column190 String
    Column191 String
    Column192 Integer
    Column193 String
    Column194 String
    Column195 Timestamp
    Column196 String
    Column197 Integer
    Column198 Integer
    Column199 Date
    Column200 String
    Column201 String
    Column202 Integer
    Column203 String
    Column204 String
    Column205 String
    Column206 String
    Column207 String
    Column208 String
    Column209 String
    Column210 String
    Column211 String
    Column212 String

  • Vertica Structure
    Column Name Data Type
    Column1 Varchar
    Column2 Varchar
    Column3 Varchar
    Column4 Varchar
    Column5 Varchar
    Column6 Integer
    Column7 Varchar
    Column8 Numeric
    Column9 Numeric
    Column10 Varchar
    Column11 Date
    Column12 Integer
    Column13 Integer
    Column14 Varchar
    Column15 Varchar
    Column16 Numeric
    Column17 Numeric
    Column18 Numeric
    Column19 Varchar
    Column20 Varchar
    Column21 Varchar
    Column22 Varchar
    Column23 Varchar
    Column24 Varchar
    Column25 Varchar
    Column26 Varchar
    Column27 Varchar
    Column28 Numeric
    Column29 Varchar
    Column30 Numeric
    Column31 Varchar
    Column32 Varchar
    Column33 Numeric
    Column34 Varchar
    Column35 Timestamp
    Column36 Varchar
    Column37 Varchar
    Column38 Varchar
    Column39 Varchar
    Column40 Varchar
    Column41 Varchar
    Column42 Varchar
    Column43 Varchar
    Column44 Date
    Column45 Varchar
    Column46 Varchar
    Column47 Varchar
    Column48 Varchar
    Column49 Integer
    Column50 Varchar
    Column51 Varchar
    Column52 Varchar
    Column53 Varchar
    Column54 Varchar
    Column55 Varchar
    Column56 Varchar
    Column57 Varchar
    Column58 Timestamp
    Column59 Timestamp
    Column60 Timestamp
    Column61 Varchar
    Column62 Numeric
    Column63 Date
    Column64 Varchar
    Column65 Date
    Column66 Varchar
    Column67 Varchar
    Column68 Varchar
    Column69 Varchar
    Column70 Varchar
    Column71 Varchar
    Column72 Varchar
    Column73 Integer
    Column74 Date
    Column75 Varchar
    Column76 Varchar
    Column77 Varchar
    Column78 Varchar
    Column79 Varchar
    Column80 Varchar
    Column81 Varchar
    Column82 Varchar
    Column83 Varchar
    Column84 Varchar
    Column85 Varchar
    Column86 Varchar
    Column87 Numeric
    Column88 Numeric
    Column89 Varchar
    Column90 Date
    Column91 Varchar
    Column92 Numeric
    Column93 Varchar
    Column94 Varchar
    Column95 Varchar
    Column96 Varchar
    Column97 Integer
    Column98 Varchar
    Column99 Numeric
    Column100 Numeric
    Column101 Varchar
    Column102 Varchar
    Column103 Varchar
    Column104 Varchar
    Column105 Numeric
    Column106 Numeric
    Column107 Date
    Column108 Varchar
    Column109 Numeric
    Column110 Numeric
    Column111 Varchar
    Column112 Varchar
    Column113 Varchar
    Column114 Date
    Column115 Varchar
    Column116 Varchar
    Column117 Varchar
    Column118 Varchar
    Column119 Varchar
    Column120 Varchar
    Column121 Varchar
    Column122 Varchar
    Column123 Varchar
    Column124 Varchar
    Column125 Numeric
    Column126 Varchar
    Column127 Varchar
    Column128 Varchar
    Column129 Varchar
    Column130 Varchar
    Column131 Varchar
    Column132 Varchar
    Column133 Integer
    Column134 Integer
    Column135 Varchar
    Column136 Varchar
    Column137 Varchar
    Column138 Varchar
    Column139 Varchar
    Column140 Varchar
    Column141 Varchar
    Column142 Varchar
    Column143 Varchar
    Column144 Integer
    Column145 Timestamp
    Column146 Varchar
    Column147 Date
    Column148 Varchar
    Column149 Varchar
    Column150 Numeric
    Column151 Varchar
    Column152 Varchar
    Column153 Varchar
    Column154 Varchar
    Column155 Numeric
    Column156 Varchar
    Column157 Varchar
    Column158 Varchar
    Column159 Varchar
    Column160 Varchar
    Column161 Varchar
    Column162 Varchar
    Column163 Varchar
    Column164 Varchar
    Column165 Varchar
    Column166 Numeric
    Column167 Varchar
    Column168 Varchar
    Column169 Varchar
    Column170 Varchar
    Column171 Varchar
    Column172 Varchar
    Column173 Varchar
    Column174 Varchar
    Column175 Varchar
    Column176 Date
    Column177 Varchar
    Column178 Date
    Column179 Varchar
    Column180 Varchar
    Column181 Timestamp
    Column182 Date
    Column183 Varchar
    Column184 Varchar
    Column185 Varchar
    Column186 Varchar
    Column187 Varchar
    Column188 Numeric
    Column189 Varchar
    Column190 Varchar
    Column191 Varchar
    Column192 Varchar
    Column193 Numeric
    Column194 Numeric
    Column195 Varchar
    Column196 Varchar
    Column197 Varchar
    Column198 Numeric
    Column199 Varchar
    Column200 Varchar
    Column201 Timestamp
    Column202 Varchar
    Column203 Numeric
    Column204 Numeric
    Column205 Date
    Column206 Varchar
    Column207 Varchar
    Column208 Numeric
    Column209 Varchar
    Column210 Varchar
    Column211 Varchar
    Column212 Varchar

  • Kindly note that, I am selecting the data from HIVE table in the above mentioned DF structure

  • JAR Version is - vertica-9.0.1_spark2.1_scala2.11.jar

  • SruthiASruthiA Vertica Employee Administrator

    what is your vertica version?

  • vertica version is 9.1.1-5

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file