Kafka not loading data for json input
I am trying to load json data using vkconfig but not able to do so . It just doesn't load the data into the target table. Don't see any error as well. Here is what i did.
-- create scheduler
dbadmin> /opt/vertica/packages/kafka/bin/vkconfig scheduler --add --brokers 10.90.84.122:9092 --conf config.properties
Warning: no operator was given. Kafka Scheduler can only be used by dbadmin or the operator. Use "--edit" to add an operator to a created scheduler in the future.
## Since i am using dbadmin so i can ignore it. My config.properties file includes jdbc parameter and config-schema "test_kafka". There is no error regarding jdbc connection
--create topic , i have topic named vert-test already created in kakfa broker
dbadmin> /opt/vertica/packages/kafka/bin/vkconfig topic --add --target public.tbl_kafka_test --topic vert-kafka --rejection-table public.tbl_rejection --parser KafkaJSONParser --conf config.properties
-- kafka-topic "vert-kafka" exists in kafka-broker
root>bin/kafka-topics.sh --list --topic vert-kafka --zookeeper 10.90.84.122:2181
vert-kafka
--The json input format I am trying to load
{"name":"zum","description":"Zum.com","level1":"Web"}
{"name":"zynga","description":"Zynga.com","level1":"Web"}
-- Here is the kafka tables created inside schema "test_kafka"
dbadmin=> select * from test_kafka.kafka_scheduler;
scheduler_version | frame_duration | brokers | config_refresh_interval | resource_pool | new_topic_policy | eof_timeout_interval
-------------------+----------------+-------------------+-------------------------+--------------------+------------------+----------------------
v7.2.1 | 00:00:10 | 10.90.84.122:9092 | 00:05 | kafka_default_pool | FAIR |
(1 row)
dbadmin=> select * from test_kafka.kafka_targets;
target_schema | target_table | enabled | parser | rejection_table | parser_parameters | load_method | target_columns
---------------+----------------+---------+-----------------+----------------------+-------------------+-------------+----------------
public | tbl_kafka_test | t | KafkaJSONParser | public.tbl_rejection | | TRICKLE |
(1 row)
dbadmin=> select * from test_kafka.kafka_events;
event_time | log_level | location | batch_create | target_schema | target_table | message | exception
-------------------------+-----------+---------------------------------------------------------+--------------+---------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------
2016-04-07 09:59:12.641 | INFO | com.vertica.solutions.kafka.scheduler.LeaderSelector | | | | New leader registered for schema test_kafka. New ID: 0, new Host: 10.90.84.124 |
2016-04-07 09:59:12.723 | INFO | com.vertica.solutions.kafka.scheduler.StreamCoordinator | | | | Received configuration details; frame duration: 10000, refresh interval 300000, eof timeout interval: 0, brokers (10.90.84.122:9092), resource pool: kafka_default_pool, new topic policy: FAIR |
2016-04-07 09:59:12.755 | INFO | com.vertica.solutions.kafka.scheduler.StreamCoordinator | | | | Received configuration details; planned concurrency: 2, max concurrency: 0, max execution parallelism: 32. Setting lane count: 2 |
transaction_id | batch_create | batch_start | batch_end | timeslice | target_schema | target_table | node_name | ktopic | kpartition | start_offset | end_offset | num_messages | la
st_duration | total_bytes | reason
-------------------+-------------------------+----------------------------+----------------------------+--------------+---------------+----------------+-----------+------------+------------+--------------+------------+--------------+----
-------------+-------------+---------------
54043195537683002 | 2016-04-07 09:59:32.985 | 2016-04-07 09:59:33.00887 | 2016-04-07 09:59:38.090145 | 00:00:09.792 | public | tbl_kafka_test | | vert-kafka | 0 | | -2 | 0 | 00:
00:05.033224 | 0 | END_OF_STREAM
54043195537683029 | 2016-04-07 09:59:52.983 | 2016-04-07 09:59:53.00676 | 2016-04-07 09:59:58.10941 | 00:00:09.796 | public | tbl_kafka_test | | vert-kafka | 0 | | -2 | 0 | 00:
00:05.053202 | 0 | END_OF_STREAM
54043195537683053 | 2016-04-07 10:00:12.992 | 2016-04-07 10:00:13.015495 | 2016-04-07 10:00:18.106753 | 00:00:09.787 | public | tbl_kafka_test | | vert-kafka | 0 | | -2 | 0 | 00:
00:05.039507 | 0 | END_OF_STREAM
54043195537683066 | 2016-04-07 10:00:22.995 | 2016-04-07 10:00:23.018065 | 2016-04-07 10:00:28.099939 | 00:00:09.784 | public | tbl_kafka_test | | vert-kafka | 0 | | -2 | 0 | 00:
00:05.036515 | 0 | END_OF_STREAM
The output table is flex table . There are no records inserted :
dbadmin=> select * from public.tbl_kafka_test;
__identity__ | __raw__
--------------+---------
(0 rows)
What am i missing here ?
Comments
Changing the start offset value from -2 to 0 worked. Really worth testing the copy command from vertica log first.
Hi!
Hmmm, it's very strange, in my setup the -2 is perfectly work. Could you give more context, what's kafka and vertica version?