I have a copy command like this

COPY SDNSLOGS_stg FROM 
's3://raw/converted-logs/test/4cff0890-f597-4374-9671-edaaa0ce8bef.gz'
,'s3://raw/converted-logs/test/4ecdebb7-f401-47d7-a3d1-6bcce1329aab.gz'
,'s3://raw/converted-logs/test/6110cfe7-b383-4166-a0a1-8acb2de9258f.gz'

GZIP
DELIMITER '^'
RECORD TERMINATOR '▼'
REJECTED DATA AS TABLE loader_rejects100;   

When I run the COPY like above with 3 file names I am seeing that only data from the last file 6110*.gz is getting loaded while the others are rejected with errors saying **Invalid input syntax for date

There is actually no error in the first and the second file too !

I confirmed that by running COPY with just one of the file name

COPY SDNSLOGS_stg FROM 
's3://raw/converted-logs/test/4ecdebb7-f401-47d7-a3d1-6bcce1329aab.gz'
GZIP
DELIMITER '^'
RECORD TERMINATOR '▼'
REJECTED DATA AS TABLE loader_rejects101;

why am I seeing this problem when trying to copy multiple files

I'm looking for something like this.

        CREATE LOCAL TEMPORARY TABLE "iris" ("Sepal.Length" DOUBLE PRECISION, "Sepal.Width" DOUBLE PRECISION, "Petal.Length" DOUBLE PRECISION, "Petal.Width" DOUBLE PRECISION, "Species" VARCHAR) 

        INSERT INTO "iris" VALUES (5.1, 3.5, 1.4, 0.2, 'setosa'),
        (4.9, 3.0, 1.4, 0.2, 'setosa'), 
        (4.7, 3.2, 1.3, 0.2, 'setosa)

...
ect.

This produces an error. The documentation states "You must insert one complete tuple at a time." Does this mean that I need one INSERT INTO SQL statement for each row of data?

Thanks!

The json file referenced seems present and accessible. Ideas?

[dbadmin@01 ~]$ aws s3 cp s3://vertica-test-dev-data/test/metadata/TEST/cluster_config.json /tmp/

download: s3://vertica-test-dev-data/test/metadata/TEST/cluster_config.json to ../../tmp/cluster_config.json

[dbadmin@01 ~]$ /opt/vertica/bin/admintools -t revive_db -s 10.168.36.58,10.168.38.174,10.168.39.38 --communal-storage-location=s3://vertica-test-dev-data/test/ -d TEST --force
Attempting to retrieve file: [s3://vertica-test-dev-data/test/metadata/TEST/cluster_config.json]
Attempting to retrieve file: [s3://vertica-test-dev-data/test/metadata/TEST/cluster_config_prev.json]

Database could not be revived.
Error:
Failed to run vertica-download-file on host(s). See /opt/vertica/log/adminTools.log for details.
10.168.36.58: Initialization thread logged exception:
Could not copy file [s3://vertica-test-dev-data/test/metadata/TEST/cluster_config.json] to [/tmp/desc.json]: Access Denied
Exiting process ( exit(1) ).

Note: I didn't set aws auth key's , assuming AWS IAM Role will provide necessary permissions when reviving DB.

I installed vertica-console_9.0.1-0_amd64.deb, I can reach and can enter. But when I want to import existing vertica cluster, I am getting invalid username or password error. I am pretty sure that username and password is true because I am reaching to database with the same username and password. I even don't try to write it, I do copy-paste extreme carefully (no space etc.) but still getting the same error.

I restarted vertica-agent, vertica-consoled several times but still no luck.

I am sure that getting the right API Key as following:

cat /opt/vertica/config/apikeys.dat

Any help would be appreciated. Thank you!

At this time I can COPY files from our own Vertica provided S3 bucket into an AWS EON V instance without issues:

, SoftwareModule
)
FROM 's3://p-vertica-poc/Data/01-iXR_cdf_events/dump-iXR_cdf_events-2016-01-01-12h-13h.csv.gz' gzip
DELIMITER ',' ENCLOSED BY '"' null E'\002NULL\003'
rejectmax 10
;

Unfortuately I get Vertica error 7160 (access denied) when trying this with files from the customer provided S3 bucket. awslib is available, we have run

ALTER SESSION SET UDPARAMETER FOR awslib aws_id=...
ALTER SESSION SET UDPARAMETER FOR awslib aws_secret=...
-- AWS Region: EU-West region: eu-west-1
select set_config_parameter('AWSRegion', 'eu-west-1');

Reaching the customer Bucket using the S3 Browser or the aws S3 command works.

Can you help me to better understand what the V doc says

Using COPY
Before you can read data from S3, you must create an IAM role for your EC2 instances to use, and grant that role permission to access your S3 resources.

Would you have experience in this area and have hints (or even better statements) for me what to do?

Thank you
Dieter

select count(1) over() recordCount ,a.column1 ,b.column2 from table_1 a left join table_2 b on a.column_1 = b.column_1

With this clause place in a query the execution time is ~31 secs
But when i remove this clause, the execution time is ~5 secs.

Is there any alternative way to get the count of number of records returned by a query.
Something like the ROWCOUNT function in Oracle.

Thanks,
pujesh

1) Will Terrace routing work same as its in EON also.
2) since EON separated compute and storage, how terrace routing to going to help us in performance
3) If we impletented Terrace Routing , the same architecture we will be carrying while implementing to EON?
4) if we implemented EON do we need to additional implement Terrace routing?
Please respond as soon as possible

Amit

14 Jan 2018 11:09:51,627 [Thread-8555] StitchCluster INFO  - **** Done Processing stitching output ****
14 Jan 2018 11:09:51,628 [Thread-8555] StitchCluster INFO  - Step 4 FINISHED Stitch status = 
14 Jan 2018 11:09:51,630 [Thread-8555] GenericTaskBuilder INFO  - GenericTaskBuilder POST_INSTALL_CLEANUP
14 Jan 2018 11:09:51,641 [Thread-8555] PostInstallCleanupTask INFO  - ------START---------Executing Post Validation Script, host: 172.31.20.7-----------------------

14 Jan 2018 11:09:53,495 [Thread-8555] PostInstallCleanupTask INFO  - processStreams>> exitCode: 127
14 Jan 2018 11:09:53,495 [Thread-8555] PostInstallCleanupTask ERROR - Failure condition
14 Jan 2018 11:09:53,495 [Thread-8555] PostInstallCleanupTask INFO  - ------END---------Executing Post Validation Script, host: host: 172.31.20.7-----------------------

14 Jan 2018 11:09:53,506 [Thread-8555] PostInstallCleanupTask INFO  - ------START---------Cleaning up Remote Temp Folder, host: host: 172.31.20.7-----------------------

14 Jan 2018 11:09:55,354 [Thread-8555] PostInstallCleanupTask INFO  - processStreams>> exitCode: 0
14 Jan 2018 11:09:55,354 [Thread-8555] PostInstallCleanupTask INFO  - **** PASSED cleanup ****
14 Jan 2018 11:09:55,355 [Thread-8555] PostInstallCleanupTask INFO  - Cleaning up remote temp folder

The instances are rolled back before I press the rollback button on the MC .

We're currently timing the queries for xxx on a 12-node cluster, using 12 shards.

We would like to repeat the tests with 6 and/or 4 nodes, and would like to shrink the cluster not one node at a time, but removing for example 6 in one go.

We were thinking of calling admintools, Advanced Menu -> Cluster Management -> Remove Host(s), followed by terminating 6 nodes from the AWS EON management console.

What is your take on this? is there a more efficient, safer, more elegant or better for any other reason solution?

Thanks for any hints & cheers -
Dieter and Marco

"Each node can run 3 concurrent queries"
Can you please explain this in detail?
Thank you

Dieter

I'm trying to create eon db on our cluster using admintools
It success creating the DB but I see it fails for installing the AWS package.
when I'm trying to install AWS package I receive the error:

"Could not stat file [s3://poc_eon/metadata/poc_eon/nodes/v_poc_eon_node0001/UDxLogs]: Unable to parse ExceptionName: PermanentRedirect Message:
The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.")

Please advise.

Thanks.
Emanuel Frish

we have been provided a new great V AMI.

I have tried to provision a new V EON cluster: I did not find a way in MC to specify the AMI I want to use for the new cluster.
Can you please point me to a way to select the new AMI for a V EON database?

Is this possible at all?
Or would I need to go another route for installation (e.g. new rpm and run upgrade_vertica)?

Please advise.
Thank you

dbadmin=> ALTER SESSION SET UDPARAMETER FOR awslib aws_id='AKIAIYP7J44TPC3LUFGQ';
ROLLBACK 6600: [awslib] does not exist
dbadmin=>

Thank you
Dieter

How do i troubleshoot this ?
Anything wrong with the bucket?
02823bcda288a8811479d012e64c241d00c0000000000e10_0.gt]: Could not stat file [s3://BUCKET_NAME/communal/70c/02823bcda288a8811479d012e64c241d00c0000000000e10_0.gt]: Unable to parse ExceptionName: PermanentRedirect Message: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
2017-10-30 19:15:07.964 Init Session:7f4e218c0700-c000000000055c [Dist] Cleaning up Uploader on fail
2017-10-30 19:15:07.965 Init Session:7f4e218c0700-c000000000055c [Dist] Cleaning up Uploader on fail
2017-10-30 19:15:07.966 Init Session:7f4e218c0700-c000000000055c @v_verticadb_node0002: 58030/8162: Something went wrong while copying the files to persistent location.

Applies to: Vertica Versions 7.1.x and 7.2.x

Read the full story here

vsql:export.sql:7: ERROR 2927: [verticadb] Could not write to [/vertica/data/VerticaDB/v_verticadb_node0003_data]: Volume [/vertica/data/VerticaDB/v_verticadb_node0003_data] has insufficient space.
vsql:export.sql:8: ERROR 2927: [verticadb] Could not write to [/vertica/data/VerticaDB/v_verticadb_node0003_data]: Volume [/vertica/data/VerticaDB/v_verticadb_node0003_data] has insufficient space.

Here is the vertica storage on the source on 1 node
/dev/md127 590G 356G 205G 64% /scratch_b
/dev/xvde 50G 1.3G 46G 3% /vertica/data

and on the destination
/dev/xvde 50G 260M 47G 1% /vertica/data -> filled up and then failed
/dev/md127 590G 142M 560G 1% /scratch_b

/vertica/data is the catalog path so why is that getting filled ? I checked and could not see files in either catalog / data , I suppose they are hidden or reference counts are held.

Anyhow, this should be workping a bit different and the depot should be used instead + S3 storage when the depot runs out.

06 Nov 2017 23:19:59,221 [Thread-11160] CloudManagementThread ERROR - Cannot get updated new host information for 172.31.xx.xx
06 Nov 2017 23:19:59,230 [Thread-11160] CloudManagementThread INFO - Add Instances Success...

The host is created and provisioned, but the node is not added to the DB.
Any hints?

Also pressing the close button doesnot cancel the process..

aws s3 cp s3://eonpoc2/communal/metadata/VerticaDB/cluster_config.json -

[dbadmin@ip-172-31-46-62 ~]$ admintools -t revive_db --communal-storage-location=s3://eonpoc2/communal/ -s 172.31.46.62 --force -d VerticaDB

            Vertica Eon Mode Beta
    The operation you are performing is part of
    beta functionality. The terms listed in the beta
    agreement apply.

    Vertica Eon Mode Beta forum:
    https://forum.vertica.com/categories/verticabeta

Attempting to read database information from: [s3://eonpoc2/communal/metadata/VerticaDB/cluster_config.json]

Database could not be revived.
Error: Error while downloading file communal/metadata/VerticaDB/cluster_config.json from communal location: An error occurred (400) when calling the HeadObject operation: Bad Request

Created iamRole: verticaeon1_1509581033274_203
Info Failed to provision AWS Cluster: [fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Instance creation failed => Unsupported: EBS-optimized instances are not supported for your requested configuration. Please check the documentation for supported configurations."}]
Failed Provision step failed, please check log for details.
Info Rollback AWS resources.

I picked c3.4xlarge in us-east-2

While running the admintools CLI to create Db in EON mode, i am getting the following logs :

2017-10-31 15:40:37.716 Spread Client:7f7409df3700 [SAL] [S3UDFS] [RETRY] [Attempt 14/50] [Canceled? N] [Should retry? Y] [Operation virtual void SAL::S3FileSystem::mkdirs(const char*, mode_t) const] (99) : 'Unable to connect to endpoint'

Could you please advise.

Creating a cluster in EON mode fails with "Unable to parse ExceptionName: PermanentRedirect Message: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint."

We got the same error when trying to query a parquet file after upgrading to vertica 9 enterprise mode.

Case 00071999 has more details. The instances and the s3 bucket are in the same region.

-Bharath

Welcome to the first part of our tutorial series on developing User Defined Extensions (UDxs) with the Java programming language. In this part, you’ll learn what UDxs are and why you would want to develop one. We’ll also show you how to set up a development environment to make developing your UDxs easier.

Read the full story here

 Click here for a PDF version of this document.

Read the full story here

below is the sample data from
v_catalog.PROJECTION_COLUMNS ,

v_monitor.column_storage

data_type encoding_type encodings compressions
varchar(12) AUTO String lzo
varchar(9) AUTO String lzo
varchar(9) AUTO String lzo
date AUTO Uncompressed int delta
date AUTO Uncompressed int delta
varchar(7) AUTO String lzo