We're Moving!

The Vertica Forum is moving to a new OpenText Analytics Database (Vertica) Community.

Join us there to post discussion topics, learn about

product releases, share tips, access the blog, and much more.

Create My New Community Account Now


[Kafka Streaming] Recover scheduler and start loading from newest offset first — Vertica Forum

[Kafka Streaming] Recover scheduler and start loading from newest offset first

[Deleted User][Deleted User] Administrator
in General Discussion

If the scheduler goes down for 2 hrs, for example,  when it comes back, is there a way to get the latest data first ? They’d  like to prioritize the latest data to be processed first, then go back to read older data, any recommendations?  Or  if it is possible to have 2 schedulers which read from different beginning and ending offsets, it seems this is NOT an option but please confirm.

 

Question credit to Wang, Wei (HPSW Big Data Platform Presales)

Comments

  • [Deleted User][Deleted User] Administrator

    At this time this type of configuration is not possible. It is possible to start from specific offsets which can be configured manually, however I would not recommend this approach as setting the offset back after collecting the recent data would likely cause data duplication.

     

    Answer credit to Mark Fay

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Format
SpoilerCodeQuote
Heading 2Heading 1
Emoji
:):D:(;):/:o:s:p:'(:|B):#:*<3o:)>:)
Url
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file
Home • General DiscussionComment As ...