Please take this survey to help us learn more about how you use third party tools. Your input is greatly appreciated!

Question on not loading older partitions into the depot in Eon mode

We can pin tables and partitions to the database depot. Do you plan to also support the opposite scenario, e.g. do not load partitions older than 6 months into the depot at anytime?

@skeswani , @skamat
Sizing and Configuring Vertica in Eon Mode for Different Use Cases

Answers

  • skeswaniskeswani Employee

    One of the properties of a LRU cache is that what is not/never used is not pinned and/or eventually evicted. the act of now using some data causes it to be evicted. So there is no plan to support the opposite. if you don't use something it wont be cached. if you do use it, and there is space we will cache it (why not).
    pinning a table to the cache does not actively download it to the cache. it merely ensures that if/when its downloaded its not evicted.

  • Hi,
    the problem is when a customer decides to reprocess very old data e.g. once per year.
    It may cause recent data to be completely evicted.
    I raise this question because I was thinking about if EON could process the data without storing them into depot,
    something like load the file from S3 directly into memory.
    There are many ways how to do it, for example:
    https://stackoverflow.com/questions/44043036/how-to-read-image-file-from-s3-bucket-directly-into-memory

    Another option could be to mark old partitions as to be evicted as soon as possible.
    Once a batch reprocessing old data would finish a transaction, related files would be aggressively evicted.

  • skeswaniskeswani Employee
    edited April 2020

    There is a direct read from S3 option hint, so you can control the granuallity
    select /+DEPOT_FETCH(All) count() from bar;
    select /+DEPOT_FETCH(FETCHES) count() from bar;
    select /+DEPOT_FETCH(NONE) count() from bar;
    https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/SQLReferenceManual/LanguageElements/Hints/DepotFetch.htm

  • Hi Skeswani,
    this is awesome, thanks!
    Jan

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file

Can't find what you're looking for? Search the Vertica Documentation, Knowledge Base, or Blog for more information.