vertica jdbc not working for kerberos enabled hdfs copy
i have kerberos enabled hdfs cluster and i also have enabled kerberos on my vertica cluster. If I try to copy some files from hdfs to vertica using vsql. Eg.,
vsql -k <servicename> -K <principalname> -h <hostname> -U <username> -t -A -c "copy <tablename> source hdfs(url='<webhdfsurl>', user.name='<username>')"
it works. But when I try to do the same thing using jdbc connection, it gives 401 error while connecting to webhdfs. Is jdbc not capable enough or am I doing anything wrong.
PS: am passing kerberosservicename and principalname in jdbc and am able to connect to vertica. Am also able to execute select command and view results. The problem happens only when I try to copy from hdfs.
Any help will be greatly appreciated.
vsql -k <servicename> -K <principalname> -h <hostname> -U <username> -t -A -c "copy <tablename> source hdfs(url='<webhdfsurl>', user.name='<username>')"
it works. But when I try to do the same thing using jdbc connection, it gives 401 error while connecting to webhdfs. Is jdbc not capable enough or am I doing anything wrong.
PS: am passing kerberosservicename and principalname in jdbc and am able to connect to vertica. Am also able to execute select command and view results. The problem happens only when I try to copy from hdfs.
Any help will be greatly appreciated.
0
Comments
Could you please share a portion of logs with us.
Regards
Bhawana
here is whats happening :
1 : Success while doing vsql
>$ vsql -k $SERVICE_NAME -K $PRINCIPAL_NAME -h $HOST_NAME -U $USER_NAME -t -A -c "copy $TABLE_NAME source Hdfs(url='$WEB_HDFS_URL', username='$USER_NAME')" -x
Rows Loaded|$NUM_ROWS
2 : Failure while doing jdbc call
>$ java -cp ../lib/vertica-jdbc.jar:. TestCopy
Exception in thread "main" java.sql.SQLNonTransientException: [Vertica][VJDBC](3399) ERROR: Failure in UDx RPC call InvokePlanUDL(): Error calling planUDL() in User Defined Object [Hdfs] at [src/Hdfs.cpp:307], error code: 0, message: [The requested URL returned error: 401. URL: $WEB_HDFS_URL]
at com.vertica.util.ServerErrorData.buildException(Unknown Source)
at com.vertica.dataengine.VQueryExecutor.executeSimpleProtocol(Unknown Source)
at com.vertica.dataengine.VQueryExecutor.execute(Unknown Source)
at com.vertica.jdbc.common.SStatement.executeNoParams(Unknown Source)
at com.vertica.jdbc.common.SStatement.execute(Unknown Source)
at TestCopy.main(TestCopy.java:28)
Caused by: com.vertica.support.exceptions.NonTransientException: [Vertica][VJDBC](3399) ERROR: Failure in UDx RPC call InvokePlanUDL(): Error calling planUDL() in User Defined Object [Hdfs] at [src/Hdfs.cpp:307], error code: 0, message: [The requested URL returned error: 401. URL: $WEB_HDFS_URL]
... 6 more
Java Code
public class TestCopy {
public static void main(String[] args) throws SQLException, ClassNotFoundException {
Class.forName("com.vertica.jdbc.Driver");
Properties props = new Properties();
props.setProperty("user", "$USER_NAME");
props.setProperty("KerberosServiceName", "$SERVICE_NAME");
props.setProperty("KerberosHostName", "$PRINCIPAL_NAME");
props.setProperty("JAASConfigName", "verticajdbc");
VerticaConnection conn = (VerticaConnection) DriverManager.getConnection(
"$JDBC_STRING", props);
Statement stmt = conn.createStatement();
boolean res = stmt.execute("copy $TABLE_NAME source Hdfs(url='$WEB_HDFS_URL', username='$USER_NAME')");
System.out.println(res);
stmt.close();
conn.close();
}
}
Do let me know if you need more info. Whatever error messages I provided are obtained as console output.
string USER_NAME = "dbadmin";
All variables have been properly defined in the code. Has it worked for anybody? I mean, has anyone tried and succeeded in copying data from kerberos enabled hdfs to vertica using jdbc??
Hi All,
Can you please let me know if this work for someone? I tested the same code it is not working so i am working if the HDFS Copy can work in a kerberized environment over JDBC.
Thanks for your help,
Abdoul