S3UDFS Curl returned error code 56

ksandeep

I am utilizing Vertica Copy command to copy data from an S3 Source (MinIO) into Vertica.
My Copy command looks as below:
copy test.metrics from 's3://bucket1/sampledata.csv' enclosed by '"' delimiter ',';
I invoke this Copy command from a JDBC client.
I have setup necessary certificates in Vertica for TLS communication with my S3 Source.
When invoking the Copy command from VSQL client, copy works well. However, when I run my Java application where I run the Copy command through JDBC client, Copy fails with an 'Unable to connect to endpoint' message. My Java application is being run from a Docker container.

I turned on SAL debug. I am attaching detailed debug log of this failure.

When I run the same Java application from my IDE, Copy runs fine. I am not understanding how when running the same JDBC application from a container is causing this problem.

SruthiA

Did you set AWSRegion appropriately? Can you ping AWS Endpoint from your docker container to check if there is any network issue?

ksandeep

Hi Sruthi,

Application running in my docker container does two things. First, it contacts S3 end point (MinIO in this case) and lists all objects. Then it sets AWSAuth, AWSEndpoint session parameters and then invokes the Copy command on Vertica.

Vertica is pre-configured with database wide parameters such as AWSEnableHttps, AWSCAPath, AWSCAFile and AWSRegion.

This is where it gets confusing. When I set AWSAuth, AWSEnpoint session parameters from VSQL session, the Copy command works well. However, I am seeing connection issues as I noted above when running the code from Docker container.

Sandeep.

poojan

If you have proxy enabled in your docker container can you please disable it and try the java program again?

ksandeep

Hi Poojan,
I do not have a proxy setting on my Docker container. Even if I have one, do you think that these settings some how effect JDBC connection in such a way that they change the behavior of Vertica? Reason I ask is, if they do in fact have such effect then I can start looking at other environmental aspects of my Docker container that is running the Java program.
Thanks,
Sandeep.

SruthiA

@ksandeep Could you please run sample curl commands from your docker container and connect to MINIO. Please try both GET and PUT. If those fail, then it has something to do with network settings or client connection timeout. Below link has a sample script.

https://gist.github.com/CAJan93/b5f484af15f8f161431fbf3c154bc0e5

ksandeep

Hi Sruthi,
I do not have a problem connecting to MinIO from my Docker container. As I explained previously, Java application running within Docker container first connects with MinIO and lists all the objects to be copied into Vertica. Then the Java application goes in loop invoking Vertica Copy command to copy data from MinIO objects into Vertica table. It is the later part that is failing.

Vertica uses AWS CPP SDK for Copy operation. AWS CPP SDK utilizes CURL libraries for connecting with Vertica. It is these CURL operations that are failing. They fail with the error message that looks like CurlHttpClient [140483850385152] Curl returned error code 56 as you will notice from the log I have attached in my original post.

Please note these observations:
1. If I turn off TLS on MinIO and set 'AWSEnableHttps' parameter to 0 on Vertica, Vertica copy operation succeeds.
2. If I turn on TLS on MinIO and set 'AWSEnableHttps' parameter to 1 on Vertica, Vertica copy operation succeeds - if I invoke the Copy commands through VSQL client or my Java application running in my IDE.
3. If I turn on TLS on MinIO and set 'AWSEnableHttps' parameter to 1 on Vertica, Vertica copy operation fails - if I invoke my Java application from a Docker container.

Something within my Java application that is running in Docker container is influencing how Vertica connects with MinIO over TLS and in a bad way. Java application in Docker container has some class path loading etc. going on for the way we need to get this working.

What bothers me is the fact that something in the application is changing the way Vertica behaves. And I have no explanation to what could be causing this.

Thanks,
Sandeep.

ksandeep

Hi Sruthi,
I am able to resolve this problem. As embarrassing as it is, the problem was happening all along because of my Java code. Java code was setting AWSEndpoint as 'https://MinIoHost:MinioPort' rather than setting it as 'MinioHost:MinioPort'. I caught this by looking at the Vertica logs.

I was in a middle of a complete development environment upgrade and somehow ended up thinking that my environment was responsible for the issue.

Thanks for the help.
Sandeep.

SruthiA

@ksandeep Thank you for updating on the solution.. Good to know.