Answers

• SruthiA Administrator

  October 2020

  Why are you passing start_offset as -3? Any specific reason? offset should be a positive number of -2.

  0
• msk_2016 ✭

  October 2020

  It is mentioned in the vertica documentation of 9.2 version at following location "Special Starting Offset Values" at this hyperlink : https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/KafkaIntegrationGuide/UtilityOptions/MicroBatchUtilityOptions.htm?tocpath=Integrating with Apache Kafka|vkconfig Script Options|_____7#Special

  Actually if i specify -2, it always start to read the kafka log from begining which is not what i need. I want that vkconfig should read the kafka incrementally.

  0
• SruthiA Administrator

  October 2020

  Do you use consumer groups in kafka then? Can you share me the output of below query

  SELECT * FROM stream_config.stream_microbatch_history order by batch_start desc limit 15;

  0
• msk_2016 ✭

  October 2020

  Since my kafka cluster is a producer that is connected to my stream log and vertica is a consumer. The consumer group has to be setup at vertica end and there is no option to set consumer group except the group-id in the load_spec. That is not working.
  There is nothing in the stream_microbatch_history table. Here is output of all my stream tables. When i run the launch, it loads perfectly for the first time but next time again it starts from begin rather than incremental
  /opt/vertica/packages/kafka/bin/vkconfig launch --config-schema stca_kschema --microbatch stca_kmb_lsbstatus --perform-load --cluster stca_kcluster --source lsbstatus_job --username xxxx --password xxxx
  Starting load
  COPY "pauser"."tal_lsb_status" SOURCE KafkaSource(stream='lsbstatus_job|0|-2', brokers='xxxxxxxxxx:9092', message_max_bytes=1048576, stop_on_eof=true) PARSER KafkaJSONParser(flatten_maps=FALSE) DIRECT STREAM NAME 'vkstream_one_time_load_' NO COMMIT

  OUTPUT OF STREAM TABLES :
  padb=> select * from stca_kschema.stream_clusters;
  id | cluster | hosts
  ---------+---------------+--------------------------
  2000001 | stca_kcluster | xxxxxxxxxxxxxxx:9092
  (1 row)

  padb=> select * from stca_kschema.stream_load_specs;;
  id | load_spec | filters | parser | parser_parameters | load_method | message_max_bytes | uds_kv_parameters
  --------+----------------+---------+-----------------+--------------------+-------------+-------------------+-------------------
  500001 | stca_kloadspec | | KafkaJSONParser | flatten_maps=FALSE | DIRECT | 1048576 |
  (1 row)

  padb=> select * from stca_kschema.stream_microbatches;
  id | microbatch | target | load_spec | target_columns | rejection_schema | rejection_table | enabled
  --------+--------------------+--------+-----------+----------------+------------------+-----------------+---------
  500001 | stca_kmb_lsbstatus | 500001 | 500001 | | | | t
  (1 row)

  padb=> select * from stca_kschema.stream_microbatch_history;
  source_name | source_cluster | source_partition | start_offset | end_offset | end_reason | end_reason_message | partition_bytes | partition_messages | microbatch_id | microbatch | target_schema | target_table | timeslice | batch_start | batch_end | last_batch_duration | consecutive_error_count | transaction_id | frame_start | frame_end
  -------------+----------------+------------------+--------------+------------+------------+--------------------+-----------------+--------------------+---------------+------------+---------------+--------------+-----------+-------------+-----------+---------------------+-------------------------+----------------+-------------+-----------
  (0 rows)

  padb=> select * from stca_kschema.stream_microbatch_source_map;
  microbatch | source
  ------------+--------
  500001 | 500001
  (1 row)

  padb=> select * from stca_kschema.stream_scheduler;
  version | frame_duration | resource_pool | config_refresh | new_source_policy | eof_timeout_ms | pushback_policy | pushback_max_count | auto_sync
  ---------+----------------+--------------------+----------------+-------------------+----------------+-----------------+--------------------+-----------
  v8.1.1 | 00:00:10 | kafka_default_pool | 00:05 | FAIR | 1000 | LINEAR | 5 | t
  (1 row)

  padb=> select * from stca_kschema.stream_sources;
  id | source | cluster | partitions | enabled
  --------+---------------+---------+------------+---------
  500001 | lsbstatus_job | 2000001 | 1 | t
  (1 row)

  padb=> select * from stca_kschema.stream_targets;
  id | target_schema | target_table
  --------+---------------+----------------
  500001 | pauser | tal_lsb_status
  (1 row)

  padb=>

  0
• SruthiA Administrator

  October 2020

  it looks like this issue requires webEx. please open a support case and share logs for review.

  0