Error while writing to vertica from spark using Vertica spark connector
Exception in thread "main" java.lang.Exception: S2V: FATAL ERROR for job S2V_job1251056748245412653. Job status information is available in the Vertica table wcc.S2V_JOB_STATUS_USER_SAMPLE. Unable to create/insert into target table wcc.T_SAMPLE_TABLE with SaveMode: Append. ERROR MESSAGE: ERROR: java.sql.SQLException: [Vertica]VJDBC INTERNAL: VIAssert((size_t)pio[i].attnum <= loadStep->queryCols.size()) failed
[Vertica][VJDBC]Detail: /scratch_a/release/svrtar14870/vbuild/vertica/Optimizer/Path/BulkLoad.cpp: 1710
at com.vertica.spark.s2v.S2V.do2Stage(S2V.scala:339)
at com.vertica.spark.s2v.S2V.save(S2V.scala:389)
at com.vertica.spark.datasource.DefaultSource.createRelation(VerticaSource.scala:88)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:668)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:276)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:270)
at actualLoad.gspTranLoad$.main(gspTranLoad.scala:37)
at actualLoad.gspTranLoad.main(gspTranLoad.scala)
Comments
Can some one please help me? I tried to search this issue, but no single answer
It looks like a bug. please open a support case and share the logs.
That is a very odd error. Can you provide the Spark code you are running to get this error? what is the structure of the DF? what is the structure of your target table in Vertica? what parameters are you using? What version of the connector and Spark and Vertica and JDBC driver you are using?
Hi, Thank you very much for the response. Please see the details below.
This is the command I am using for the load,
dataFrame.write.format("com.vertica.spark.datasource.DefaultSource").options(opts).mode("Append").save()
val opts: Map[String, String] = Map(
"table" -> "T_SAMPLE_TABLE",
"dbschema" -> "wcc",
"db" -> "dbname",
"user" -> "user",
"password" -> "password",
"host" -> "vertica-sample-dev.bnymellon.net",
"hdfs_url" -> "hdfs://hiveServer.domain.net:9000/user/hive/warehouse",
"web_hdfs_url" -> "webhdfs://hdfs://hiveServer.domain.net:50070/user/hive/warehouse"
Structure of a DataFrame
Column1 String
Column2 String
Column3 String
Column4 String
Column5 String
Column6 Integer
Column7 String
Column8 Integer
Column9 Integer
Column10 String
Column11 Date
Column12 Integer
Column13 Integer
Column14 String
Column15 String
Column16 Integer
Column17 Integer
Column18 Integer
Column19 String
Column20 String
Column21 String
Column22 String
Column23 String
Column24 String
Column25 String
Column26 String
Column27 String
Column28 Integer
Column29 String
Column30 Integer
Column31 String
Column32 String
Column33 Integer
Column34 String
Column35 Timestamp
Column36 String
Column37 String
Column38 String
Column39 String
Column40 String
Column41 String
Column42 String
Column43 String
Column44 Date
Column45 String
Column46 String
Column47 String
Column48 String
Column49 Integer
Column50 String
Column51 String
Column52 String
Column53 String
Column54 String
Column55 String
Column56 String
Column57 String
Column58 Timestamp
Column59 Timestamp
Column60 Timestamp
Column61 String
Column62 Integer
Column63 Date
Column64 String
Column65 Date
Column66 String
Column67 String
Column68 String
Column69 String
Column70 String
Column71 String
Column72 String
Column73 Integer
Column74 Date
Column75 String
Column76 String
Column77 String
Column78 String
Column79 String
Column80 String
Column81 String
Column82 String
Column83 String
Column84 String
Column85 String
Column86 String
Column87 Integer
Column88 Integer
Column89 String
Column90 Date
Column91 String
Column92 Integer
Column93 String
Column94 String
Column95 String
Column96 String
Column97 Integer
Column98 String
Column99 Integer
Column100 Integer
Column101 String
Column102 String
Column103 String
Column104 String
Column105 Integer
Column106 Integer
Column107 Date
Column108 String
Column109 Integer
Column110 Integer
Column111 String
Column112 String
Column113 String
Column114 Date
Column115 String
Column116 String
Column117 String
Column118 String
Column119 String
Column120 String
Column121 String
Column122 String
Column123 Integer
Column124 String
Column125 String
Column126 String
Column127 String
Column128 String
Column129 String
Column130 String
Column131 Integer
Column132 Integer
Column133 String
Column134 String
Column135 String
Column136 String
Column137 String
Column138 String
Column139 String
Column140 String
Column141 Integer
Column142 Timestamp
Column143 String
Column144 Date
Column145 String
Column146 String
Column147 Integer
Column148 String
Column149 String
Column150 String
Column151 String
Column152 Integer
Column153 String
Column154 String
Column155 String
Column156 String
Column157 String
Column158 String
Column159 String
Column160 Integer
Column161 String
Column162 String
Column163 String
Column164 String
Column165 String
Column166 String
Column167 String
Column168 String
Column169 String
Column170 Date
Column171 String
Column172 Date
Column173 String
Column174 String
Column175 Timestamp
Column176 Date
Column177 String
Column178 String
Column179 String
Column180 String
Column181 String
Column182 Integer
Column183 String
Column184 String
Column185 String
Column186 String
Column187 Integer
Column188 Integer
Column189 String
Column190 String
Column191 String
Column192 Integer
Column193 String
Column194 String
Column195 Timestamp
Column196 String
Column197 Integer
Column198 Integer
Column199 Date
Column200 String
Column201 String
Column202 Integer
Column203 String
Column204 String
Column205 String
Column206 String
Column207 String
Column208 String
Column209 String
Column210 String
Column211 String
Column212 String
Vertica Structure
Column Name Data Type
Column1 Varchar
Column2 Varchar
Column3 Varchar
Column4 Varchar
Column5 Varchar
Column6 Integer
Column7 Varchar
Column8 Numeric
Column9 Numeric
Column10 Varchar
Column11 Date
Column12 Integer
Column13 Integer
Column14 Varchar
Column15 Varchar
Column16 Numeric
Column17 Numeric
Column18 Numeric
Column19 Varchar
Column20 Varchar
Column21 Varchar
Column22 Varchar
Column23 Varchar
Column24 Varchar
Column25 Varchar
Column26 Varchar
Column27 Varchar
Column28 Numeric
Column29 Varchar
Column30 Numeric
Column31 Varchar
Column32 Varchar
Column33 Numeric
Column34 Varchar
Column35 Timestamp
Column36 Varchar
Column37 Varchar
Column38 Varchar
Column39 Varchar
Column40 Varchar
Column41 Varchar
Column42 Varchar
Column43 Varchar
Column44 Date
Column45 Varchar
Column46 Varchar
Column47 Varchar
Column48 Varchar
Column49 Integer
Column50 Varchar
Column51 Varchar
Column52 Varchar
Column53 Varchar
Column54 Varchar
Column55 Varchar
Column56 Varchar
Column57 Varchar
Column58 Timestamp
Column59 Timestamp
Column60 Timestamp
Column61 Varchar
Column62 Numeric
Column63 Date
Column64 Varchar
Column65 Date
Column66 Varchar
Column67 Varchar
Column68 Varchar
Column69 Varchar
Column70 Varchar
Column71 Varchar
Column72 Varchar
Column73 Integer
Column74 Date
Column75 Varchar
Column76 Varchar
Column77 Varchar
Column78 Varchar
Column79 Varchar
Column80 Varchar
Column81 Varchar
Column82 Varchar
Column83 Varchar
Column84 Varchar
Column85 Varchar
Column86 Varchar
Column87 Numeric
Column88 Numeric
Column89 Varchar
Column90 Date
Column91 Varchar
Column92 Numeric
Column93 Varchar
Column94 Varchar
Column95 Varchar
Column96 Varchar
Column97 Integer
Column98 Varchar
Column99 Numeric
Column100 Numeric
Column101 Varchar
Column102 Varchar
Column103 Varchar
Column104 Varchar
Column105 Numeric
Column106 Numeric
Column107 Date
Column108 Varchar
Column109 Numeric
Column110 Numeric
Column111 Varchar
Column112 Varchar
Column113 Varchar
Column114 Date
Column115 Varchar
Column116 Varchar
Column117 Varchar
Column118 Varchar
Column119 Varchar
Column120 Varchar
Column121 Varchar
Column122 Varchar
Column123 Varchar
Column124 Varchar
Column125 Numeric
Column126 Varchar
Column127 Varchar
Column128 Varchar
Column129 Varchar
Column130 Varchar
Column131 Varchar
Column132 Varchar
Column133 Integer
Column134 Integer
Column135 Varchar
Column136 Varchar
Column137 Varchar
Column138 Varchar
Column139 Varchar
Column140 Varchar
Column141 Varchar
Column142 Varchar
Column143 Varchar
Column144 Integer
Column145 Timestamp
Column146 Varchar
Column147 Date
Column148 Varchar
Column149 Varchar
Column150 Numeric
Column151 Varchar
Column152 Varchar
Column153 Varchar
Column154 Varchar
Column155 Numeric
Column156 Varchar
Column157 Varchar
Column158 Varchar
Column159 Varchar
Column160 Varchar
Column161 Varchar
Column162 Varchar
Column163 Varchar
Column164 Varchar
Column165 Varchar
Column166 Numeric
Column167 Varchar
Column168 Varchar
Column169 Varchar
Column170 Varchar
Column171 Varchar
Column172 Varchar
Column173 Varchar
Column174 Varchar
Column175 Varchar
Column176 Date
Column177 Varchar
Column178 Date
Column179 Varchar
Column180 Varchar
Column181 Timestamp
Column182 Date
Column183 Varchar
Column184 Varchar
Column185 Varchar
Column186 Varchar
Column187 Varchar
Column188 Numeric
Column189 Varchar
Column190 Varchar
Column191 Varchar
Column192 Varchar
Column193 Numeric
Column194 Numeric
Column195 Varchar
Column196 Varchar
Column197 Varchar
Column198 Numeric
Column199 Varchar
Column200 Varchar
Column201 Timestamp
Column202 Varchar
Column203 Numeric
Column204 Numeric
Column205 Date
Column206 Varchar
Column207 Varchar
Column208 Numeric
Column209 Varchar
Column210 Varchar
Column211 Varchar
Column212 Varchar
Kindly note that, I am selecting the data from HIVE table in the above mentioned DF structure
JAR Version is - vertica-9.0.1_spark2.1_scala2.11.jar
what is your vertica version?
vertica version is 9.1.1-5