AnalyticalUDx Errors: Java SDK

guptamayank

Hi Experts ,

We are trying to implement the Mode function using Vertica Java SDK. We are able to compile the code successfully. It runs okay on the smaller data set less than 10,000 records successfully but it start giving the error as we run it against large dataset. The error we get is mentioned below:

2017-07-17 08:37:46.753 [Java-8499] 0x20 Vanilla exception
java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkIndex(Buffer.java:546)
at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:416)
at com.vertica.sdk.BlockReader.getLong(BlockReader.java:73)
at com.vertica.sdk.BlockReader.isLongNull(BlockReader.java:82)
at com.vertica.JavaLibs.Mode.processPartition(Mode.java:59)
at com.vertica.udxfence.UDxExecContext.processPartitionForUDT(UDxExecContext.java:1952)
at com.vertica.udxfence.UDxExecContext.processPartition(UDxExecContext.java:1875)
at com.vertica.udxfence.UDxExecContext.run(UDxExecContext.java:295)
at java.lang.Thread.run(Thread.java:748)

2017-07-17 08:37:46.753 [Java-8499] 0x20 java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkIndex(Buffer.java:546)
at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:416)
at com.vertica.sdk.BlockReader.getLong(BlockReader.java:73)
at com.vertica.sdk.BlockReader.isLongNull(BlockReader.java:82)
at com.vertica.JavaLibs.Mode.processPartition(Mode.java:59)
at com.vertica.udxfence.UDxExecContext.processPartitionForUDT(UDxExecContext.java:1952)
at com.vertica.udxfence.UDxExecContext.processPartition(UDxExecContext.java:1875)
at com.vertica.udxfence.UDxExecContext.run(UDxExecContext.java:295)
at java.lang.Thread.run(Thread.java:748)

I have also attached the Mode function Java source code. Any help on this will be greatly appreciated.

Thanks

sKwa

Hi!

It runs okay on the smaller data set less than 10,000 records successfully but it start giving the error as we run it against large dataset.

Can you try to increase heap size? Im not sure that problem in heap, but as you said it works for small data set and falls on large.

guptamayank

We have tried to increase the MAXMEMORYSIZE of the JVM resource pool but that didn't made any difference. Is there any other query to modify the heap size as we used the one mentioned below:

ALTER RESOURCE POOL jvm MAXMEMORYSIZE '50%' PLANNEDCONCURRENCY AUTO;

sKwa

Hi!

ALTER RESOURCE POOL jvm MAXMEMORYSIZE '50%'

No, its not what I mean.
Can you modify your code something like below?:

try {

   // here is your code do-while
   //...

} catch (Exception e) {

    // Get current size of heap in bytes
    long heapSize = Runtime.getRuntime().totalMemory();

    // Get maximum size of heap in bytes.
    // The heap cannot grow beyond this size.
    // Any attempt will result in an OutOfMemoryException.
    long heapMaxSize = Runtime.getRuntime().maxMemory();

    // Get amount of free memory within the heap in bytes. 
    // This size will increase after garbage collection and 
    // decrease as new objects are created.
    long heapFreeSize = Runtime.getRuntime().freeMemory();

    srvInterface.log("HEAP SIZE: %d [B]", heapSize);
    srvInterface.log("MAX SIZE : %d [B]", heapMaxSize);
    srvInterface.log("FREE SIZE: %d [B]", heapFreeSize);

    throw e;
}

it will rethrow exception but also will give information for memory usage in log file.

---

FYI:

• How to control Java heap size (memory) allocation (xmx, xms)
• Getting JVM heap size, used memory, total memory using Java Runtime

guptamayank

Hi, I executed the code with try and catch block. Here is the error:

2017-07-18 14:01:03.046 [Java-101151] 0x11 [UserMessage] Mode - HEAP SIZE: 46661632 [B]
2017-07-18 14:01:03.047 [Java-101151] 0x11 [UserMessage] Mode - MAX SIZE : 58916864 [B]
2017-07-18 14:01:03.048 [Java-101151] 0x11 [UserMessage] Mode - FREE SIZE: 38034232 [B]
2017-07-18 14:01:03.052 [Java-101151] 0x11 Vanilla exception
java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkIndex(Buffer.java:546)
at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:416)
at com.vertica.sdk.BlockReader.getLong(BlockReader.java:73)
at com.vertica.sdk.BlockReader.isLongNull(BlockReader.java:82)
at com.vertica.JavaLibs.Mode.processPartition(Mode.java:61)
at com.vertica.udxfence.UDxExecContext.processPartitionForUDT(UDxExecContext.java:1952)
at com.vertica.udxfence.UDxExecContext.processPartition(UDxExecContext.java:1875)
at com.vertica.udxfence.UDxExecContext.run(UDxExecContext.java:295)
at java.lang.Thread.run(Thread.java:748)

2017-07-18 14:01:03.053 [Java-101151] 0x11 java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkIndex(Buffer.java:546)
at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:416)
at com.vertica.sdk.BlockReader.getLong(BlockReader.java:73)
at com.vertica.sdk.BlockReader.isLongNull(BlockReader.java:82)
at com.vertica.JavaLibs.Mode.processPartition(Mode.java:61)
at com.vertica.udxfence.UDxExecContext.processPartitionForUDT(UDxExecContext.java:1952)
at com.vertica.udxfence.UDxExecContext.processPartition(UDxExecContext.java:1875)
at com.vertica.udxfence.UDxExecContext.run(UDxExecContext.java:295)
at java.lang.Thread.run(Thread.java:748)

Also, could you please let me know if you have the link for documentation for enabling (IGNORE NULLS) in the UDAnF ?

Thanks

guptamayank

We were able to sort this out, it seems like there is an issue with Vertica Java SDK (it was trying to access the rows which doesn't exist and gives IndexOutOfBoundsException error). We are now using for loop instead of while loop to iterate through the rows in the partitions and it works fine:

for (int i = 0; i < inputReader.getNumRows(); i++)

Thanks for your help.

sKwa

HI!

link for documentation for enabling (IGNORE NULLS) in the UDAnF ?

You can't, you have to deal with it, i.e. return some value.

Developing a User Defined Analytic Function

UDAnFs must output a single value for each row of data read.