UDFilter with KafkaSource

dochernyshov

Hello everyone!
Are there any best practices of filtering json messages from kafka?
I implemented UD Filter in C ++ (like SearchAndReplace - https://github.com/vertica/UDx-Examples/blob/master/Java-and-C++/FilterFunctions/SearchAndReplaceFilter.cpp) - works very slowly.
SearchAndReplace (AS IS) with option reject_on_empty_keys - not suitable, because a lot of rejects.
Maybe someone has already solved a similar problem?

Bryan_H

Hi, are you able to share more details? For example, what does the COPY command look like, are there other filters or parsers inline? What type of filtering is needed - check for JSON key/values, edit JSON key/values, etc.? Can the algorithm be parallelized, or maybe written in Java instead since Java has JSON libraries like Gson that may already be optimized for your project?