KafkaAvroParser and SSL

LauriPessi

I would appreciate some guidance on debugging the mTLS authentication with Confluent Schema Registry.

SSL keys work fine with curl and KafkaSource broker connections, but KafkaAvroParser with schema registry raises "[SSL certificate problem: self signed certificate in certificate chain]

These are the parameters given (copy without registry works fine, albeit the raw payload is not usable):

PARSER KafkaAvroParser(
  schema_registry_url='https://x.x.x.x:8081'
  , schema_registry_ssl_ca_path='/xxx/caroot.pem'
  , schema_registry_ssl_cert_path='/xxx/cert.pem'
  , schema_registry_ssl_key_path='/xxx/vertica.pem'
 );

Referred key locations are available on all nodes.

Another question is that whether it's possible to to pass keys to KafkaAvroParser using UDParameters similarly as with KafkaSource?

Vertica 11.1. Enterprise Mode

SergeB

For everyone's benefit, wanted to share the solution.

schema_registry_ssl_ca_path has to be a path to a directory containing hashed CA certificates (hashed as in openssl rehash)

Documentation will be clarified accordingly.

SruthiA

you need to set KafkaSSL* UDParameters before running copy from kafkasource with kafkaavroparser.. since kafkasource works fine with default parser, it looks like the issue is with your schema registry TLS certs. please try to run copy with non TLS schema registry to see if everything works fine when using kafkaavroparser.

LauriPessi

Hi, to clarify more - the brokers and schema-registry authenticate using the same keys, which I've set into following UDparameters:

• kafka_SSL_CA
• kafka_SSL_Certificate
• kafka_SSL_PrivateKey_secret
• kafka_Enable_SSL

And after setting those, the broker authentication works fine (checkbrokers, listtopics, copy from KafkaSource etc), but they don't seem to get passed to KafkaAvroParser.

I've also verified that the keys work against schema-registry by passing them to curl from command line:

$ curl --cacert /xxx/caroot.pem --cert /xxx/cert.pem --key /xxx/vertica.pem https://x.x.x.x:8081/config
{"compatibilityLevel":"BACKWARD"}

So the keys and authentication work by themselves, but not when used with KafkaAvroParser - and vertica.log gives no additional info in addition to message which is printed to the client:

ERROR 5861:  Error calling setup() in User Function UDParser at [/data/jenkins/workspace/RE-ReleaseBuilds/RE-Jackhammer/server/udx/supported/StreamHelpers/StreamAvroParser.cpp:150], error code: 0, message: Error: unable to validate schema registry [https://x.x.x.x:8081/config] : [SSL certificate problem: self signed certificate in certificate chain]; provide a valid schema_registry_url and make sure the schema registry is running

SruthiA

@LauriPessi : Thank you for confirming. Based on the doc, it looks like we may not support self signed certs. to debug more on this issue, please open a support case and share vertica logs for the same.

https://www.vertica.com/docs/11.1.x/HTML/Content/Authoring/KafkaIntegrationGuide/KafkaFunctions/KafkaAvroParser.htm

SruthiA

Could you please try curl test just with cacert?

curl --cacert /xxx/caroot.pem https://x.x.x.x:8081/config

LauriPessi

"it looks like we may not support self signed certs" --> But also the documentation you linked describes the parameter "schema_registry_ssl_ca_path" for verifying these?

Curl with plain cacert just raises an "bad certificate" error.

I did realize that when passing just the cert and key without CA, the error message matches the one from vertica.log.
So could it be that KafkaAvroParser fails to use the given "schema_registry_ssl_ca_path", and falls back to whatever is defined on system level?

Is it possible for me to increase the logging of UDParser in order to see what actually happens in the background, or do you have other suggestions?

VValdar

Tagging @SergeB !

SruthiA

"it looks like we may not support self signed certs" --> But also the documentation you linked describes the parameter "schema_registry_ssl_ca_path" for verifying these? --- Yes.. that is correct.

it is possible to increase verbosity of the parser... Could you please open a support case so that we can share the same.

SergeB

Could you try passing the directory containing the CA vs. the CA file name ?

i.e. schema_registry_ssl_ca_path='/xxx' or schema_registry_ssl_ca_path='/xxx/'

instead of schema_registry_ssl_ca_path='/xxx/caroot.pem'

LauriPessi

Removing explicit filename from schema_registry_ssl_ca_path (and/or from the ssl_cert_path) made no difference.
Only when removing filename also from ssl_key_path, the error was changed into:

ERROR 8468:  Error calling prepare() in User Function KafkaAvroParserFactory, message: basic_filebuf::underflow error reading the file

We'll raise a support case to look further into this.