Please take this survey to help us learn more about how you use third party tools. Your input is greatly appreciated!

How to fix Kafka load NETWORK_ISSUE end reason

We have a Kafka Cluster with 3 brokers (No SSL with SASL PlainText) and a Vertica Cluster with 3 Nodes.
I've made a bash script file which creates and configures a scheduler with Frame Duration = 00:01:00 which it works almost fine and loads the new data every minute.

But, when the connection breaks (even for a couple of seconds), I see network issue (The cluster was unavailable) in Load tab of MC and it tries to load new data every minute but it stuck on NETWORK_ISSUE and never recovers the connection.

The only way which I've found to recover the connection is creating a bash script file and put it in CronTab and check these things every 5 minutes:

  1. Query Scheduler State (Running or stopped) and launch scheduler if it's stopped

  2. Query stream_microbatch_history and check the end_reason and if it's NETWORK_ISSUE then:
    2.1. Shutdown the Instance
    2.2. Delete and recreate the Kafka Source
    2.3. Delete and recreate the Microbatch
    2.4. Launch the instance again

So,
1. Is there any other way to recover the connection ?!
2. How should I keep the scheduler up when failures happens? (network, db failures and etc...)

Best Answers

  • lop2loplop2lop
    edited June 17 Accepted Answer

    @SergeB said:
    You should really get to the bottom of your network issue because but can you provide details on how scheduler is stuck? There are regular checks on Kafka's health and if the health check fails too often (look at these options for scheduler CLI : pushback-policy , pushback-max-count and auto-sync) . If health check fails, the affected microbatch gets disabled (look at stream_microbatches table).

    After tons of tests and researches, I found the problem and that was absolutely my fault. But there's another little problem which I think it should be Vertica's fault :smiley:

    So, my fault:

    I used 'Data Collector Table Queries' to see the exact COPY statement which scheduler executes:

    select time,request, * from dc_requests_issued where time >= timestampadd(hour,-1,current_timestamp) and request ilike 'copy%' order by time desc;

    And I realized when I restart the node or the connection breaks, the kafka_conf parameter inside the copy statement changes after a while because my checker bash script file (Which was already created by the Create and Configure Scheduler bash script file) tries to export VERTICA_RDKAFKA_CONF to the Environment Variables again, using this command:

    export VERTICA_RDKAFKA_CONF =sasl.username=consumer;sasl.password=consumer_password;sasl.mechanism=PLAIN;security.protocol=SASL_PLAINTEXT;

    And it fails when it reaches to the semicolon character because it assumes that after the semicolons, there is another command !

    export VERTICA_RDKAFKA_CONF=sasl.username=consumer;sasl.password=consumer_password;sasl.mechanism=PLAIN;security.protocol=SASL_PLAINTEXT;
    sasl.password=consumer_password: command not found
    sasl.mechanism=PLAIN: command not found
    security.protocol=SASL_PLAINTEXT: command not found

    and It must be changed to this:

    export VERTICA_RDKAFKA_CONF ='sasl.username=consumer;sasl.password=consumer_password;sasl.mechanism=PLAIN;security.protocol=SASL_PLAINTEXT;'

    I realized that the single quotes at the beginning and end of kafka_conf have been removed during the creation of the file !

    The new problem is that, When I completely delete the scheduler and recreate the scheduler again, It consumes whole messages inside the Kafka topic from the beginning and it doesn't use consumer group's saved offset, neither the offsets in KafkaOffsets() over() .

    According to this:
    https://www.vertica.com/docs/10.0.x/HTML/Content/Authoring/KafkaIntegrationGuide/UtilityOptions/MicroBatchUtilityOptions.htm?tocpath=Integrating with Apache Kafka|vkconfig Script Options|_____7#Special

    I tried to set offset for the microbatch to -3 using --offset -3|-3|-3|-3|-3|-3|-3-3|-3|-3|-3|-3|-3|-3|-3 Parameter and it throws an exception:

    Exception in thread "main" com.vertica.solutions.kafka.exception.ConfigurationException: Invalid Offset number detected: -3. Offset out of range. Offset should be positive integer, or -2 which indicates head of the stream.
    at com.vertica.solutions.kafka.model.StreamMicrobatch.validateConfiguration(StreamMicrobatch.java:455)
    at com.vertica.solutions.kafka.model.StreamMicrobatch.setFromMapAndValidate(StreamMicrobatch.java:362)
    at com.vertica.solutions.kafka.cli.CLI.run(CLI.java:72)
    at com.vertica.solutions.kafka.cli.MicrobatchCLI.run(MicrobatchCLI.java:77)
    at com.vertica.solutions.kafka.cli.CLI._main(CLI.java:141)
    at com.vertica.solutions.kafka.cli.MicrobatchCLI.main(MicrobatchCLI.java:72)

    It seems that it only accept -2 and integer values >= 0 !

    How can I force it to use consumer group's saved offset ?

  • SergeBSergeB Employee
    Accepted Answer

    As you found out, currently -3 won't work. This will be fixed in the next major release

    In the meantime, you could read last consumed offsets by running /kafka-consumer-groups.sh (or equivalent) and using these offsets when you reset the scheduler.

    After the first microbatch, the scheduler will no longer rely on starting offsets nor on consumer group but will maintain last consumed offsets in one of its tables (microbatch_history).

Answers

  • SergeBSergeB Employee

    You should really get to the bottom of your network issue because but can you provide details on how scheduler is stuck? There are regular checks on Kafka's health and if the health check fails too often (look at these options for scheduler CLI : pushback-policy , pushback-max-count and auto-sync) . If health check fails, the affected microbatch gets disabled (look at stream_microbatches table).

  • Ok, thank you so much 🌹

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file

Can't find what you're looking for? Search the Vertica Documentation, Knowledge Base, or Blog for more information.