Configuring Scheduler for SSL

benoit_fr

Hi,
I try to integrate data from Kafka into Vertica. Kafka topic is secured so we need to configure SSL on our scheduler.
I follow steps on documentation (https://my.vertica.com/docs/8.1.x/HTML/index.htm#Authoring/KafkaIntegrationGuide/ConfiguringSchedulerforSSL.htm?TocPath=Integrating%20with%20Apache%20Kafka|Using%20SSL%20with%20Kafka|_____3) to create truststore and keystore but when I when to launch my scheduler I have the following :

java.sql.SQLNonTransientException: com.vertica.solutions.kafka.exception.ConfigurationException: No keystore system property found: null
at com.vertica.solutions.kafka.util.SQLUtilities.getConnection(SQLUtilities.java:173)
at com.vertica.solutions.kafka.cli.CLIUtil.assertDBConnectionWorks(CLIUtil.java:40)
at com.vertica.solutions.kafka.Launcher.run(Launcher.java:128)
at com.vertica.solutions.kafka.Launcher.main(Launcher.java:258)
Caused by: com.vertica.solutions.kafka.exception.ConfigurationException: No keystore system property found: null
at com.vertica.solutions.kafka.security.KeyStoreUtil.loadStore(KeyStoreUtil.java:77)
at com.vertica.solutions.kafka.security.KeyStoreUtil.(KeyStoreUtil.java:42)
at com.vertica.solutions.kafka.util.SQLUtilities.getConnection(SQLUtilities.java:171)
... 3 more

Did I miss something ?
Regards,
Benoît

SruthiA

Can you share me the exact commands that you used to create a keystore on your scheduler? Can you share me the command you used to launch scheduler?

benoit_fr

Hi SruthiA,

• Create a Trust Store and and Key Store on Your Scheduler *

/usr/java/jdk1.8.0_66/jre/bin/keytool -keystore cq_config_aa.truststore.jks -alias cacert -import -file /tmp/certif/myServer.crt
(#)/usr/java/jdk1.8.0_66/jre/bin/keytool -keystore cq_config_aa.truststore.jks -alias root -import -file "/tmp/certif/root.crt"
(#)/usr/java/jdk1.8.0_66/jre/bin/keytool -keystore cq_config_aa.truststore.jks -alias intermed -import -file "/tmp/certif/servers.crt"

/usr/java/jdk1.8.0_66/jre/bin/keytool -keystore cq_config_aa.keystore.jks -alias ctlvs011 -validity 365 -keyalg RSA -genkey
/usr/java/jdk1.8.0_66/jre/bin/keytool -keystore cq_config_aa.keystore.jks -alias ctlvs011 -certreq -file cert-file.tmp
openssl x509 -req -CA /tmp/certif/myServer.crt -CAkey /tmp/certif/myServer.key -in /data/vertica/cert-file.tmp -out cert-signed.tmp -days 365 -CAcreateserial
/usr/java/jdk1.8.0_66/jre/bin/keytool -keystore cq_config_aa.keystore.jks -alias cacert -import -file /tmp/certif/myServer.crt
/usr/java/jdk1.8.0_66/jre/bin/keytool -keystore cq_config_aa.keystore.jks -alias root -trustcacerts -import -file "/tmp/certif/root.crt"
/usr/java/jdk1.8.0_66/jre/bin/keytool -keystore cq_config_aa.keystore.jks -alias intermed -trustcacerts -import -file "/tmp/certif/servers.crt"
/usr/java/jdk1.8.0_66/jre/bin/keytool -keystore cq_config_aa.keystore.jks -alias ctlvs011 -import -file /data/vertica/cert-signed.tmp

• Launch Your Scheduler with SSL Enabled *
  /opt/vertica/packages/kafka/bin/vkconfig launch --enable-SSL true --ssl-ca-alias cacert --ssl-key-alias ctlvs011 --ssl-key-password XXXXX

I try with and without the 2 lignes with (#)
Keystore files are created on /data/vertica folder.

Regards,
Benoît

benoit_fr

Hi SruthiA,

• Create a Trust Store and and Key Store on Your Scheduler *

/usr/java/jdk1.8.0_66/jre/bin/keytool -keystore cq_config_aa.truststore.jks -alias cacert -import -file /tmp/certif/myServer.crt
(#)/usr/java/jdk1.8.0_66/jre/bin/keytool -keystore cq_config_aa.truststore.jks -alias root -import -file "/tmp/certif/root.crt"
(#)/usr/java/jdk1.8.0_66/jre/bin/keytool -keystore cq_config_aa.truststore.jks -alias intermed -import -file "/tmp/certif/servers.crt"

/usr/java/jdk1.8.0_66/jre/bin/keytool -keystore cq_config_aa.keystore.jks -alias ctlvs011 -validity 365 -keyalg RSA -genkey
/usr/java/jdk1.8.0_66/jre/bin/keytool -keystore cq_config_aa.keystore.jks -alias ctlvs011 -certreq -file cert-file.tmp
openssl x509 -req -CA /tmp/certif/myServer.crt -CAkey /tmp/certif/myServer.key -in /data/vertica/cert-file.tmp -out cert-signed.tmp -days 365 -CAcreateserial
/usr/java/jdk1.8.0_66/jre/bin/keytool -keystore cq_config_aa.keystore.jks -alias cacert -import -file /tmp/certif/myServer.crt
/usr/java/jdk1.8.0_66/jre/bin/keytool -keystore cq_config_aa.keystore.jks -alias root -trustcacerts -import -file "/tmp/certif/root.crt"
/usr/java/jdk1.8.0_66/jre/bin/keytool -keystore cq_config_aa.keystore.jks -alias intermed -trustcacerts -import -file "/tmp/certif/servers.crt"
/usr/java/jdk1.8.0_66/jre/bin/keytool -keystore cq_config_aa.keystore.jks -alias ctlvs011 -import -file /data/vertica/cert-signed.tmp

• Launch Your Scheduler with SSL Enabled *
  /opt/vertica/packages/kafka/bin/vkconfig launch --enable-SSL true --ssl-ca-alias cacert --ssl-key-alias ctlvs011 --ssl-key-password XXXXX

I tried with and without the two lignes with (#).
Keystore files are created on /data/vertica folder.

Regards,
Benoît

benoit_fr

In addition I set JAVA_TOOL_OPTIONS with -DJVMjavax.net.ssl.trustStore, -DJVMjavax.net.ssl.keyStore and -DJVMjavax.net.ssl.keyStorePassword but still having the same error

benoit_fr

Corrects args are -Djavax.net.ssl.trustStore, -Djavax.net.ssl.keyStore and -Djavax.net.ssl.keyStorePassword

SruthiA

Can you share me kafka scheduler logs in private message. I will take a look.

benoit_fr

Hi SruthiA,

I managed to configure my scheduler for SSL.
First issue was JVM args that are wrong on Vertica documentation (-Djavax.net.ssl.trustStore, -Djavax.net.ssl.keyStore and -Djavax.net.ssl.keyStorePassword expected).
Second, my launch command should specified config-schema and password options.

But we still have an issue, my scheduler was created on cq_config_aa config-schema and this error appears in vkafka-sched.log

2017-05-02 09:17:48.904 com.vertica.solutions.kafka.security.KeyStoreUtil::VDBLogger Thread [INFO] Loading KeyStores as store type: jks
2017-05-02 09:17:48.941 com.vertica.solutions.kafka.logger.VDBLogger::VDBLogger Thread [ERROR] Caught exception from VDBLogger, not writing logs to database. Closing connection, will try to recreate connection next time.
java.sql.SQLException: [Vertica]VJDBC ERROR: Schema "stream_config" does not exist
at com.vertica.util.ServerErrorData.buildException(Unknown Source)
at com.vertica.io.ProtocolStream.readExpectedMessage(Unknown Source)
at com.vertica.dataengine.VDataEngine.prepareImpl(Unknown Source)
at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
at com.vertica.jdbc.common.SPreparedStatement.(Unknown Source)
at com.vertica.jdbc.jdbc4.S4PreparedStatement.(Unknown Source)
at com.vertica.jdbc.VerticaJdbc4PreparedStatementImpl.(Unknown Source)
at com.vertica.jdbc.VJDBCObjectFactory.createPreparedStatement(Unknown Source)
at com.vertica.jdbc.common.SConnection.prepareStatement(Unknown Source)
at com.vertica.solutions.kafka.util.CountedConnection.prepareStatement(CountedConnection.java:69)
at com.vertica.solutions.kafka.logger.VDBLogger$VDBLogRunner.run(VDBLogger.java:121)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.vertica.support.exceptions.ErrorException: [Vertica]VJDBC ERROR: Schema "stream_config" does not exist
... 13 more

To avoid this, I create my scheduler in stream_config schema and it works, I receive my data in my target table.
Is there a real issue using SSL with scheduler or is there a parameter that I miss about schema for keyStore/VDBLogger ?

Regards,
Benoît