AnalyticalUDx Errors: Java SDK

Hi Experts ,

We are trying to implement the Mode function using Vertica Java SDK. We are able to compile the code successfully. It runs okay on the smaller data set less than 10,000 records successfully but it start giving the error as we run it against large dataset. The error we get is mentioned below:

2017-07-17 08:37:46.753 [Java-8499] 0x20 Vanilla exception
java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkIndex(Buffer.java:546)
at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:416)
at com.vertica.sdk.BlockReader.getLong(BlockReader.java:73)
at com.vertica.sdk.BlockReader.isLongNull(BlockReader.java:82)
at com.vertica.JavaLibs.Mode.processPartition(Mode.java:59)
at com.vertica.udxfence.UDxExecContext.processPartitionForUDT(UDxExecContext.java:1952)
at com.vertica.udxfence.UDxExecContext.processPartition(UDxExecContext.java:1875)
at com.vertica.udxfence.UDxExecContext.run(UDxExecContext.java:295)
at java.lang.Thread.run(Thread.java:748)

2017-07-17 08:37:46.753 [Java-8499] 0x20 java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkIndex(Buffer.java:546)
at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:416)
at com.vertica.sdk.BlockReader.getLong(BlockReader.java:73)
at com.vertica.sdk.BlockReader.isLongNull(BlockReader.java:82)
at com.vertica.JavaLibs.Mode.processPartition(Mode.java:59)
at com.vertica.udxfence.UDxExecContext.processPartitionForUDT(UDxExecContext.java:1952)
at com.vertica.udxfence.UDxExecContext.processPartition(UDxExecContext.java:1875)
at com.vertica.udxfence.UDxExecContext.run(UDxExecContext.java:295)
at java.lang.Thread.run(Thread.java:748)

I have also attached the Mode function Java source code. Any help on this will be greatly appreciated.

Thanks

Comments

  • Hi!

    It runs okay on the smaller data set less than 10,000 records successfully but it start giving the error as we run it against large dataset.

    Can you try to increase heap size? Im not sure that problem in heap, but as you said it works for small data set and falls on large.

  • We have tried to increase the MAXMEMORYSIZE of the JVM resource pool but that didn't made any difference. Is there any other query to modify the heap size as we used the one mentioned below:

    ALTER RESOURCE POOL jvm MAXMEMORYSIZE '50%' PLANNEDCONCURRENCY AUTO;

  • edited July 2017

    Hi!

    ALTER RESOURCE POOL jvm MAXMEMORYSIZE '50%'

    No, its not what I mean.
    Can you modify your code something like below?:

    try {
    
       // here is your code do-while
       //...
    
    } catch (Exception e) {
    
        // Get current size of heap in bytes
        long heapSize = Runtime.getRuntime().totalMemory();
    
        // Get maximum size of heap in bytes.
        // The heap cannot grow beyond this size.
        // Any attempt will result in an OutOfMemoryException.
        long heapMaxSize = Runtime.getRuntime().maxMemory();
    
        // Get amount of free memory within the heap in bytes. 
        // This size will increase after garbage collection and 
        // decrease as new objects are created.
        long heapFreeSize = Runtime.getRuntime().freeMemory();
    
        srvInterface.log("HEAP SIZE: %d [B]", heapSize);
        srvInterface.log("MAX SIZE : %d [B]", heapMaxSize);
        srvInterface.log("FREE SIZE: %d [B]", heapFreeSize);
    
        throw e;
    }
    

    it will rethrow exception but also will give information for memory usage in log file.


    FYI:

  • Hi, I executed the code with try and catch block. Here is the error:

    2017-07-18 14:01:03.046 [Java-101151] 0x11 [UserMessage] Mode - HEAP SIZE: 46661632 [B]
    2017-07-18 14:01:03.047 [Java-101151] 0x11 [UserMessage] Mode - MAX SIZE : 58916864 [B]
    2017-07-18 14:01:03.048 [Java-101151] 0x11 [UserMessage] Mode - FREE SIZE: 38034232 [B]
    2017-07-18 14:01:03.052 [Java-101151] 0x11 Vanilla exception
    java.lang.IndexOutOfBoundsException
    at java.nio.Buffer.checkIndex(Buffer.java:546)
    at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:416)
    at com.vertica.sdk.BlockReader.getLong(BlockReader.java:73)
    at com.vertica.sdk.BlockReader.isLongNull(BlockReader.java:82)
    at com.vertica.JavaLibs.Mode.processPartition(Mode.java:61)
    at com.vertica.udxfence.UDxExecContext.processPartitionForUDT(UDxExecContext.java:1952)
    at com.vertica.udxfence.UDxExecContext.processPartition(UDxExecContext.java:1875)
    at com.vertica.udxfence.UDxExecContext.run(UDxExecContext.java:295)
    at java.lang.Thread.run(Thread.java:748)

    2017-07-18 14:01:03.053 [Java-101151] 0x11 java.lang.IndexOutOfBoundsException
    at java.nio.Buffer.checkIndex(Buffer.java:546)
    at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:416)
    at com.vertica.sdk.BlockReader.getLong(BlockReader.java:73)
    at com.vertica.sdk.BlockReader.isLongNull(BlockReader.java:82)
    at com.vertica.JavaLibs.Mode.processPartition(Mode.java:61)
    at com.vertica.udxfence.UDxExecContext.processPartitionForUDT(UDxExecContext.java:1952)
    at com.vertica.udxfence.UDxExecContext.processPartition(UDxExecContext.java:1875)
    at com.vertica.udxfence.UDxExecContext.run(UDxExecContext.java:295)
    at java.lang.Thread.run(Thread.java:748)

    Also, could you please let me know if you have the link for documentation for enabling (IGNORE NULLS) in the UDAnF ?

    Thanks

  • We were able to sort this out, it seems like there is an issue with Vertica Java SDK (it was trying to access the rows which doesn't exist and gives IndexOutOfBoundsException error). We are now using for loop instead of while loop to iterate through the rows in the partitions and it works fine:

    for (int i = 0; i < inputReader.getNumRows(); i++)

    Thanks for your help.

  • edited July 2017

    HI!

    link for documentation for enabling (IGNORE NULLS) in the UDAnF ?

    You can't, you have to deal with it, i.e. return some value.

    Developing a User Defined Analytic Function

    UDAnFs must output a single value for each row of data read.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file