vertica jdbc not working for kerberos enabled hdfs copy

ashwini_kumar

i have kerberos enabled hdfs cluster and i also have enabled kerberos on my vertica cluster. If I try to copy some files from hdfs to vertica using vsql. Eg., 
vsql -k <servicename> -K <principalname> -h <hostname> -U <username> -t -A -c "copy <tablename> source hdfs(url='<webhdfsurl>', user.name='<username>')"
 it works. But when I try to do the same thing using jdbc connection, it gives 401 error while connecting to webhdfs. Is jdbc not capable enough or am I doing anything wrong.
PS: am passing kerberosservicename and principalname in jdbc and am able to connect to vertica. Am also able to execute select command and view results. The problem happens only when I try to copy from hdfs.
Any help will be greatly appreciated.

Bhawana

Hi,

Could you please share a portion of logs with us.


Regards

Bhawana

ashwini_kumar

Hi,
here is whats happening : 
1 : Success while doing vsql
>$ vsql -k $SERVICE_NAME -K $PRINCIPAL_NAME -h $HOST_NAME -U $USER_NAME -t -A -c "copy $TABLE_NAME source Hdfs(url='$WEB_HDFS_URL', username='$USER_NAME')" -x

Rows Loaded|$NUM_ROWS

2 : Failure while doing jdbc call

>$ java -cp ../lib/vertica-jdbc.jar:. TestCopy

Exception in thread "main" java.sql.SQLNonTransientException: [Vertica][VJDBC](3399) ERROR: Failure in UDx RPC call InvokePlanUDL(): Error calling planUDL() in User Defined Object [Hdfs] at [src/Hdfs.cpp:307], error code: 0, message: [The requested URL returned error: 401. URL: $WEB_HDFS_URL]

at com.vertica.util.ServerErrorData.buildException(Unknown Source)

at com.vertica.dataengine.VQueryExecutor.executeSimpleProtocol(Unknown Source)

at com.vertica.dataengine.VQueryExecutor.execute(Unknown Source)

at com.vertica.jdbc.common.SStatement.executeNoParams(Unknown Source)

at com.vertica.jdbc.common.SStatement.execute(Unknown Source)

at TestCopy.main(TestCopy.java:28)

Caused by: com.vertica.support.exceptions.NonTransientException: [Vertica][VJDBC](3399) ERROR: Failure in UDx RPC call InvokePlanUDL(): Error calling planUDL() in User Defined Object [Hdfs] at [src/Hdfs.cpp:307], error code: 0, message: [The requested URL returned error: 401. URL: $WEB_HDFS_URL]

... 6 more

Java Code

public class TestCopy {

  public static void main(String[] args) throws SQLException, ClassNotFoundException {

    Class.forName("com.vertica.jdbc.Driver");

    Properties props = new Properties();

    props.setProperty("user", "$USER_NAME");

    props.setProperty("KerberosServiceName", "$SERVICE_NAME");

    props.setProperty("KerberosHostName", "$PRINCIPAL_NAME");

    props.setProperty("JAASConfigName", "verticajdbc");

    VerticaConnection conn = (VerticaConnection) DriverManager.getConnection(

        "$JDBC_STRING", props);

    Statement stmt = conn.createStatement();

    boolean res = stmt.execute("copy $TABLE_NAME source Hdfs(url='$WEB_HDFS_URL', username='$USER_NAME')");

    System.out.println(res);

    stmt.close();

    conn.close();

  }

}

Do let me know if you need more info. Whatever error messages I provided are obtained as console output.

~Ashwini.

Prasanta_Pal

define all $<variable_name> in java code, could be like,

string USER_NAME = "dbadmin";

ashwini_kumar

Hi,
All variables have been properly defined in the code. Has it worked for anybody? I mean, has anyone tried and succeeded in copying data from kerberos enabled  hdfs to vertica using jdbc??

adiallo

Hi All,

 

Can you please let me know if this work for someone? I tested the same code it is not working so i am working if the HDFS Copy can work in a kerberized environment over JDBC.

 

Thanks for your help,

Abdoul