The Vertica Forum recently got a makeover! Let us know what you think by filling out this short, anonymous survey.

UDFilter with KafkaSource

Hello everyone!
Are there any best practices of filtering json messages from kafka?
I implemented UD Filter in C ++ (like SearchAndReplace - https://github.com/vertica/UDx-Examples/blob/master/Java-and-C++/FilterFunctions/SearchAndReplaceFilter.cpp) - works very slowly.
SearchAndReplace (AS IS) with option reject_on_empty_keys - not suitable, because a lot of rejects.
Maybe someone has already solved a similar problem?

Tagged:

Answers

  • Hi, are you able to share more details? For example, what does the COPY command look like, are there other filters or parsers inline? What type of filtering is needed - check for JSON key/values, edit JSON key/values, etc.? Can the algorithm be parallelized, or maybe written in Java instead since Java has JSON libraries like Gson that may already be optimized for your project?

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file

Can't find what you're looking for? Search the Vertica Documentation, Knowledge Base, or Blog for more information.