UDFilter with KafkaSource

Hello everyone!
Are there any best practices of filtering json messages from kafka?
I implemented UD Filter in C ++ (like SearchAndReplace - https://github.com/vertica/UDx-Examples/blob/master/Java-and-C++/FilterFunctions/SearchAndReplaceFilter.cpp) - works very slowly.
SearchAndReplace (AS IS) with option reject_on_empty_keys - not suitable, because a lot of rejects.
Maybe someone has already solved a similar problem?

Tagged:

Answers

  • Bryan_HBryan_H Vertica Employee Administrator

    Hi, are you able to share more details? For example, what does the COPY command look like, are there other filters or parsers inline? What type of filtering is needed - check for JSON key/values, edit JSON key/values, etc.? Can the algorithm be parallelized, or maybe written in Java instead since Java has JSON libraries like Gson that may already be optimized for your project?

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file