[Kafka Streaming] Multiple Scheduler Instance Behavior

[Deleted User]

When I was trying to launch multiple instances of a scheduler, I can tell from scheduler.log that it is happening(Lane Worker 1 waiting for batch...), but when I checked scheduler and scheduler _history table, I can’t tell that multiple instances of same scheduler launched,  only one entry for both tables, so am I missing something here or is this the way how it works by design as of now? I’d expect to see entry for each instance and its status, for example.

 

A question from Wang, Wei (HPSW Big Data Platform Presales)

[Deleted User]

Only 1 scheduler is in charge per schema at any given time. The others are stand-by backups (if the owner goes down, a backup will pick up immediately and cause a new entry in the history table). This is the current design. In the future there may be a row per instance as the schema evolves. Also, a Lane Worker refers to the concurrency on a per instance level, so a singular scheduler will have n Lane Workers (equal to planned_concurrency value of the specified resource pool).

 

Answer credit to Mark Fay