Can you provide guidance on how to accomplish this?
mackpatrick
Community Edition User
Hi, I have a significant amount of data stored in Arcsight Logger, which I haven't yet integrated with Vertica. I am interested in automatically ingesting the data into Vertica using a script. Can you provide guidance on how to accomplish this? Additionally, if scripts are required for this process, could you please advise me on how to create a script for any type of data? Thank you.
0
Answers
I think you can use kafka integration to automatically ingest data into vertica.
https://docs.vertica.com/12.0.x/en/kafka-integration/
In addition, lacat is old and simple utility that exports CEF records from a Logger archive file.
It prints them to stdout by design, allowing the user to redirect them to a file or pipe them into csv or redirect it all to a Vertica COPY statement.
It is written in python (targetting 2.7 and 3.5) using only the standard python library.
See: https://github.com/hpsec/lacat
As mentioned, Arcsight has a built-in integration with Vertica via Kafka. Vertica also provides a built-in CEF flex parser that might match your needs. https://docs.vertica.com/12.0.x/en/sql-reference/statements/copy/parsers/fcefparser/
Determine the format and location of the data in ArcSight Logger. You'll need to know the file format (e.g., CSV, JSON) and where the files are stored on disk or in the ArcSight Logger database. Write a script to extract the data from ArcSight Logger and transform it into a format that can be ingested into Vertica. This script can be written in any scripting language that supports connectivity to both ArcSight Logger and Vertica, such as Python or Bash.