How to fix Kafka load NETWORK_ISSUE end reason
We have a Kafka Cluster with 3 brokers (No SSL with SASL PlainText) and a Vertica Cluster with 3 Nodes.
I've made a bash script file which creates and configures a scheduler with Frame Duration = 00:01:00 which it works almost fine and loads the new data every minute.
But, when the connection breaks (even for a couple of seconds), I see network issue (The cluster was unavailable) in Load tab of MC and it tries to load new data every minute but it stuck on NETWORK_ISSUE and never recovers the connection.
The only way which I've found to recover the connection is creating a bash script file and put it in CronTab and check these things every 5 minutes:
Query Scheduler State (Running or stopped) and launch scheduler if it's stopped
Query stream_microbatch_history and check the end_reason and if it's NETWORK_ISSUE then:
2.1. Shutdown the Instance
2.2. Delete and recreate the Kafka Source
2.3. Delete and recreate the Microbatch
2.4. Launch the instance again
1. Is there any other way to recover the connection ?!
2. How should I keep the scheduler up when failures happens? (network, db failures and etc...)