KafkaAvroParser and SSL

LauriPessiLauriPessi Vertica Customer
edited February 21 in General Discussion

I would appreciate some guidance on debugging the mTLS authentication with Confluent Schema Registry.

SSL keys work fine with curl and KafkaSource broker connections, but KafkaAvroParser with schema registry raises "[SSL certificate problem: self signed certificate in certificate chain]

These are the parameters given (copy without registry works fine, albeit the raw payload is not usable):

PARSER KafkaAvroParser(
  , schema_registry_ssl_ca_path='/xxx/caroot.pem'
  , schema_registry_ssl_cert_path='/xxx/cert.pem'
  , schema_registry_ssl_key_path='/xxx/vertica.pem'

Referred key locations are available on all nodes.

Another question is that whether it's possible to to pass keys to KafkaAvroParser using UDParameters similarly as with KafkaSource?

Vertica 11.1. Enterprise Mode


Best Answer

  • SergeBSergeB - Select Field - Employee
    edited February 27 Answer ✓

    For everyone's benefit, wanted to share the solution.

    schema_registry_ssl_ca_path has to be a path to a directory containing hashed CA certificates (hashed as in openssl rehash)

    Documentation will be clarified accordingly.


  • SruthiASruthiA Vertica Employee Employee

    you need to set KafkaSSL* UDParameters before running copy from kafkasource with kafkaavroparser.. since kafkasource works fine with default parser, it looks like the issue is with your schema registry TLS certs. please try to run copy with non TLS schema registry to see if everything works fine when using kafkaavroparser.

  • LauriPessiLauriPessi Vertica Customer

    Hi, to clarify more - the brokers and schema-registry authenticate using the same keys, which I've set into following UDparameters:

    • kafka_SSL_CA
    • kafka_SSL_Certificate
    • kafka_SSL_PrivateKey_secret
    • kafka_Enable_SSL

    And after setting those, the broker authentication works fine (checkbrokers, listtopics, copy from KafkaSource etc), but they don't seem to get passed to KafkaAvroParser.

    I've also verified that the keys work against schema-registry by passing them to curl from command line:

    $ curl --cacert /xxx/caroot.pem --cert /xxx/cert.pem --key /xxx/vertica.pem https://x.x.x.x:8081/config

    So the keys and authentication work by themselves, but not when used with KafkaAvroParser - and vertica.log gives no additional info in addition to message which is printed to the client:

    ERROR 5861:  Error calling setup() in User Function UDParser at [/data/jenkins/workspace/RE-ReleaseBuilds/RE-Jackhammer/server/udx/supported/StreamHelpers/StreamAvroParser.cpp:150], error code: 0, message: Error: unable to validate schema registry [https://x.x.x.x:8081/config] : [SSL certificate problem: self signed certificate in certificate chain]; provide a valid schema_registry_url and make sure the schema registry is running
  • SruthiASruthiA Vertica Employee Employee

    @LauriPessi : Thank you for confirming. Based on the doc, it looks like we may not support self signed certs. to debug more on this issue, please open a support case and share vertica logs for the same.


  • SruthiASruthiA Vertica Employee Employee

    Could you please try curl test just with cacert?

    curl --cacert /xxx/caroot.pem https://x.x.x.x:8081/config

  • LauriPessiLauriPessi Vertica Customer

    "it looks like we may not support self signed certs" --> But also the documentation you linked describes the parameter "schema_registry_ssl_ca_path" for verifying these?

    Curl with plain cacert just raises an "bad certificate" error.

    I did realize that when passing just the cert and key without CA, the error message matches the one from vertica.log.
    So could it be that KafkaAvroParser fails to use the given "schema_registry_ssl_ca_path", and falls back to whatever is defined on system level?

    Is it possible for me to increase the logging of UDParser in order to see what actually happens in the background, or do you have other suggestions?

  • VValdarVValdar Vertica Employee Employee

    Tagging @SergeB !

  • SruthiASruthiA Vertica Employee Employee

    "it looks like we may not support self signed certs" --> But also the documentation you linked describes the parameter "schema_registry_ssl_ca_path" for verifying these? --- Yes.. that is correct.

    it is possible to increase verbosity of the parser... Could you please open a support case so that we can share the same.

  • SergeBSergeB - Select Field - Employee

    Could you try passing the directory containing the CA vs. the CA file name ?

    i.e. schema_registry_ssl_ca_path='/xxx' or schema_registry_ssl_ca_path='/xxx/'

    instead of schema_registry_ssl_ca_path='/xxx/caroot.pem'

  • LauriPessiLauriPessi Vertica Customer

    Removing explicit filename from schema_registry_ssl_ca_path (and/or from the ssl_cert_path) made no difference.
    Only when removing filename also from ssl_key_path, the error was changed into:

    ERROR 8468:  Error calling prepare() in User Function KafkaAvroParserFactory, message: basic_filebuf::underflow error reading the file

    We'll raise a support case to look further into this.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file