kafka stream - target table default value issue

nrkmaheshnrkmahesh Registered User

we have configured(streams) automatically Copying Data From Kafka. continues data insertion happens on the table but default (system date) value inserted as '2000-01-01 00:00:00' not system date[basically to inserted timestamp of the record]. is it a issue in vertica or I have to do something else.

kindly note that the default value is properly set on the column. when I run 'update set

=default', it is properly updating current timestamp for al the records...

Comments

  • Bryan_HBryan_H Employee, Registered User ✭✭

    Could you post more detail on the scheduler, such as target table definition, load-spec, and parser definitions? That will help identify where the wrong date is coming from.

  • nrkmaheshnrkmahesh Registered User

    Thanks Brayan.

    vkconfig script:

    /opt/vertica/packages/kafka/bin/vkconfig scheduler --create --config-schema ktm --operator dbadmin --conf myscheduler.config

    /opt/vertica/packages/kafka/bin/vkconfig cluster --create --cluster test_clust \
    --hosts xxx.xx.xx.x:9092 \
    --conf myscheduler.config \
    --kafka_conf 'api.version.request=false;check.crcs=true'

    /opt/vertica/packages/kafka/bin/vkconfig source --create --cluster test_clust --source XXX --partitions 1 --conf myscheduler.config

    /opt/vertica/packages/kafka/bin/vkconfig target --create --target-schema ts --target-table test --conf myscheduler.config

    /opt/vertica/packages/kafka/bin/vkconfig load-spec --create --parser KafkaParser --load-spec ktm_clust --conf myscheduler.config

    /opt/vertica/packages/kafka/bin/vkconfig microbatch --create --microbatch myscheduler --target-schema ts --config-schema ktm --target-table test --add-source XXX --add-source-cluster test_clust --load-spec ktm_clust --conf myscheduler.config


    table definition:

    CREATE TABLE ts.test (id auto_increment,jsval varchar(65000), dt timestamp default now()) ;

    --> I tried creating the function (getdate/sysdate ) for default as well.

    The scheduler is runningfine and records are as expectted except default value in the column. for all the records, only below value exists on dt column.

    dbadmin=> select distinct dt from ts.test;

    dt

    2000-01-01 00:00:00

    I hope this helps.

    Thanks

  • Bryan_HBryan_H Employee, Registered User ✭✭

    Hi, thanks for posting. I can reproduce the issue so I will forward to engineering for further advice.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file