Logic behind load_streams monitoring table

I am trying to monitor our bulk loading using the load_streams table but I do not completely understand how it works. Is it supposed to be cleared/trimmed after it hits a certain point (internal variable). Whats the logic behind clearing it? Whats the max size? Is it possible to modify it? If we start loading data if its close to its storage threshold, does it clear the entire table? 

Comments

  • The Load stream table is a view of the data collector tables, most of the v_monitor.* tables are views of the data_collector tables so to increase the size you need to increase the size of the underline data_collector table. 

    The data_collector tables retention is by size and start to purge the data when it reach the size, it removes older data, it does not truncate.  You can look at the documentation for all the functions in how to get, set or flush the data_collector data. Here is a link but you can also search for data_collector keyword and you will get the related links,

    https://my.vertica.com/docs/7.0.x/HTML/index.htm#Attic/Troubleshooting/WorkingWithDataCollectionLogs...


    In the load stream case is a view of the dc_execution_engine_profiles tables so you need to increase the retention policy of that table to see more data in the load_stream table. 

    I hope it make sense, 

    Eugenia


  • Yup it did. Thanks for your response. I have a followup questions. So following the view definitions and the table they reference I believe the following dc_* tables need to be given more space for our environment. 

    dc_errors
    dc_execution_engine_profiles
    dc_projections_used
    dc_requests_completed
    dc_requests_issued

    On looking up the view definition of execution_engine_profiles it references v_internal.vs_execution_engine_profiles. I am assuming this is an actual table and not a view. Can you please confirm?

    Thanks! 

  • I am seeing data in the load streams table with the following stats
    accepted_row_count        | 0
    rejected_row_count          | 0
    read_bytes                       | 0
    input_file_size_bytes         | 0
    parse_complete_percent |
    unsorted_row_count         | 13328
    sorted_row_count             | 13328
    sort_complete_percent    | 100
    I need a little explanation about how this is valid. Does this mean the copy failed but then the rejected_row_count is 0. Please explain. 

    Thanks! 


Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file