Rreading data inti Vertica using StreamSets
joergschaber
Vertica Customer ✭
Hi,
I am trying to INSERT Json files into Vertica using Streamsets. I created a table in Vertica having column names like the field names in the json file. They can be NULL. The pipeline runs and shows the correct number of records processed, however, the records are not written to Vertica and the pipeline never stops, but no errors occur. I basically the same setting as in https://www.vertica.com/kb/StreamSetsCG/Content/Partner/StreamSetsCG.htm
How can I see/test what the problem is? Any idea?
Tagged:
0
Best Answer
-
siddyash
Vertica Employee Employee
Make sure that mapping is correct then your input data is loaded successfully. Otherwise your pipeline is executed successfully without error but data is not loaded. Please see the attached pdf for reference.
Please let us know if it worked. Thanks.1
Answers
There's many different ways to get JSON data into Vertica. Sometimes Vertica will doing the processing of JSON directly, other times it will happen upstream from Vertica in a third-party tool like StreamSets.
The first step to understanding what is going on is to understand which of the several approaches is being utilized here. You probably want to start by reading the vertica.log for the node that received the JDBC connections from StreamSets. You can also check the v_monitor.query_requests table to see the recent commands that Vertica has processed and v_monitor.error_messages for recent errors.
You will hopefully see some evidence of StreamSets running commands somewhere across those three places. Once you know specifically how StreamSets is interfacing with Vertica you can then troubleshoot further by focusing on the particular approach it is using. If you can share some of the queries or errors you see I can try to provide additional advice depending on what you find.