We're Moving!

The Vertica Forum is moving to a new OpenText Analytics Database (Vertica) Community.

Join us there to post discussion topics, learn about

product releases, share tips, access the blog, and much more.

Create My New Community Account Now


Kafka not loading data for json input — Vertica Forum

Kafka not loading data for json input

I am trying to load json data using vkconfig but not able to do so . It just doesn't load the data into the target table. Don't see any error as well.  Here is what i did. 

 

-- create scheduler 

 

dbadmin> /opt/vertica/packages/kafka/bin/vkconfig scheduler --add --brokers 10.90.84.122:9092 --conf config.properties

 

Warning: no operator was given. Kafka Scheduler can only be used by dbadmin or the operator. Use "--edit" to add an operator to a created scheduler in the future.

 

 

## Since i am using dbadmin so i can ignore it. My config.properties file includes jdbc parameter and config-schema "test_kafka". There is no error regarding jdbc connection 

 

--create topic , i have topic named vert-test already created in kakfa broker

 

dbadmin> /opt/vertica/packages/kafka/bin/vkconfig topic --add --target public.tbl_kafka_test --topic vert-kafka --rejection-table public.tbl_rejection --parser KafkaJSONParser --conf config.properties

 

--  kafka-topic "vert-kafka" exists in kafka-broker

 

root>bin/kafka-topics.sh --list --topic vert-kafka --zookeeper 10.90.84.122:2181
vert-kafka

 

--The json input format I am trying to load 

 

{"name":"zum","description":"Zum.com","level1":"Web"}
{"name":"zynga","description":"Zynga.com","level1":"Web"}

 

-- Here is the kafka tables created inside schema "test_kafka"

 

dbadmin=> select * from test_kafka.kafka_scheduler;
scheduler_version | frame_duration | brokers | config_refresh_interval | resource_pool | new_topic_policy | eof_timeout_interval
-------------------+----------------+-------------------+-------------------------+--------------------+------------------+----------------------
v7.2.1 | 00:00:10 | 10.90.84.122:9092 | 00:05 | kafka_default_pool | FAIR |
(1 row)

 

dbadmin=> select * from test_kafka.kafka_targets;
target_schema | target_table | enabled | parser | rejection_table | parser_parameters | load_method | target_columns
---------------+----------------+---------+-----------------+----------------------+-------------------+-------------+----------------
public | tbl_kafka_test | t | KafkaJSONParser | public.tbl_rejection | | TRICKLE |
(1 row)

 

dbadmin=> select * from test_kafka.kafka_events;
event_time | log_level | location | batch_create | target_schema | target_table | message | exception
-------------------------+-----------+---------------------------------------------------------+--------------+---------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------
2016-04-07 09:59:12.641 | INFO | com.vertica.solutions.kafka.scheduler.LeaderSelector | | | | New leader registered for schema test_kafka. New ID: 0, new Host: 10.90.84.124 |
2016-04-07 09:59:12.723 | INFO | com.vertica.solutions.kafka.scheduler.StreamCoordinator | | | | Received configuration details; frame duration: 10000, refresh interval 300000, eof timeout interval: 0, brokers (10.90.84.122:9092), resource pool: kafka_default_pool, new topic policy: FAIR |
2016-04-07 09:59:12.755 | INFO | com.vertica.solutions.kafka.scheduler.StreamCoordinator | | | | Received configuration details; planned concurrency: 2, max concurrency: 0, max execution parallelism: 32. Setting lane count: 2 |

 

transaction_id | batch_create | batch_start | batch_end | timeslice | target_schema | target_table | node_name | ktopic | kpartition | start_offset | end_offset | num_messages | la
st_duration | total_bytes | reason
-------------------+-------------------------+----------------------------+----------------------------+--------------+---------------+----------------+-----------+------------+------------+--------------+------------+--------------+----
-------------+-------------+---------------
54043195537683002 | 2016-04-07 09:59:32.985 | 2016-04-07 09:59:33.00887 | 2016-04-07 09:59:38.090145 | 00:00:09.792 | public | tbl_kafka_test | | vert-kafka | 0 | | -2 | 0 | 00:
00:05.033224 | 0 | END_OF_STREAM
54043195537683029 | 2016-04-07 09:59:52.983 | 2016-04-07 09:59:53.00676 | 2016-04-07 09:59:58.10941 | 00:00:09.796 | public | tbl_kafka_test | | vert-kafka | 0 | | -2 | 0 | 00:
00:05.053202 | 0 | END_OF_STREAM
54043195537683053 | 2016-04-07 10:00:12.992 | 2016-04-07 10:00:13.015495 | 2016-04-07 10:00:18.106753 | 00:00:09.787 | public | tbl_kafka_test | | vert-kafka | 0 | | -2 | 0 | 00:
00:05.039507 | 0 | END_OF_STREAM
54043195537683066 | 2016-04-07 10:00:22.995 | 2016-04-07 10:00:23.018065 | 2016-04-07 10:00:28.099939 | 00:00:09.784 | public | tbl_kafka_test | | vert-kafka | 0 | | -2 | 0 | 00:
00:05.036515 | 0 | END_OF_STREAM

 

 

The output table is flex table . There are no records inserted :

 

dbadmin=> select * from public.tbl_kafka_test;
__identity__ | __raw__
--------------+---------
(0 rows)

 

What am i missing here ?

 

 

 

 

 

 

 

 

 

Comments

  • Changing the start offset value from -2 to 0 worked. Really worth testing the copy command from vertica log first. 

  • Hi!

     

    Hmmm, it's very strange, in my setup the -2 is perfectly work. Could you give more context, what's kafka and vertica version?

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file