When working with terabytes of data, storage and transfer become major time and cost sinks. Vertica can help minimize storage cost and transfer time with column compression and encoding.

How can we identify Vertica tables that might benefit from compression? Information about column size and current compression is stored across column_storage and projection_columns table. The following query will locate tables that have default (AUTO) compression, which could be sub-optimal or even no compression at all: (The WHERE clause eliminates flex tables[1] and system schemas that can't be edited)

select distinct anchor_table_schema,anchor_table_name,MAX(encoding_type),SUM(ros_used_bytes)
from column_storage cs inner join projection_columns pc using (column_id)
where column_name NOT IN ('__raw__','__identity__') AND anchor_table_schema NOT LIKE '%dbd%'
group by anchor_table_schema,anchor_table_name having max(encoding_type) = 'AUTO'
order by anchor_table_schema,anchor_table_name;

Let's work with an example:

anchor_table_schema | anchor_table_name | MAX | SUM
---------------------+---------------------------+------+------------
public | big_fact_table | AUTO | 6841357747

This table has no projections with compression. We can have Vertica determine optimized compression using the meta-function DESIGNER_DESIGN_PROJECTION_ENCODINGS, which will output a projection definition with compression:

SELECT DESIGNER_DESIGN_PROJECTION_ENCODINGS('public.big_fact_table','');

This will output a script that will create a new projection and drop the existing one (likely a super-projection). Let's first see how much space we could save though. I'll run the script right up to the MAKE_AHM_NOW()
so there's data in the projection and compare the projection sizes:

=> select anchor_table_name, projection_name, used_bytes from projection_storage where anchor_table_name = 'big_fact_table' limit 5;
anchor_table_name | projection_name | used_bytes
-------------------+-----------------------------------------+------------
big_fact_table | big_fact_table_DBD_1_seg_EncodingDesign | 2331490442
big_fact_table | big_fact_table_super | 6843970195

That's a 65% size reduction - your mileage may vary, of course! But at terabyte scale, even a smaller savings will still help with storage usage and processing speed because you're now moving about that much less data over network and I/O channels too.

Have fun and get saving! Contact us on the forum if you'd like to talk about projection design and compression.

For more information:

COLUMN_STORAGE system table: https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/SQLReferenceManual/SystemTables/MONITOR/COLUMN_STORAGE.htm
PROJECTION_COLUMNS system table: https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/SQLReferenceManual/SystemTables/CATALOG/PROJECTION_COLUMNS.htm
PROJECTION_STORAGE system table: https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/SQLReferenceManual/SystemTables/MONITOR/PROJECTION_STORAGE.htm
DataBase Designer DESIGNER_DESIGN_PROJECTION_ENCODINGS meta-function: https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/SQLReferenceManual/Functions/VerticaFunctions/DatabaseDesigner/DESIGNER_DESIGN_PROJECTION_ENCODINGS.htm

[1] It is possible to run DESIGNER_DESIGN_PROJECTION_ENCODINGS on materialized columns in flex tables, but you need to run on the materialized projection, not the flex table itself. Here's an abridged example:

dbadmin=> CREATE FLEX TABLE foo();
dbadmin=> INSERT /*+ DIRECT */ INTO foo SELECT c FROM some_big_table;
dbadmin=> SELECT compute_flextable_keys('foo');
dbadmin=> SELECT materialize_flextable_columns('foo', 1);
dbadmin=> SELECT projection_name, encoding_type FROM projection_columns WHERE projection_column_name = 'c' AND projection_name ILIKE 'foo%';
projection_name | encoding_type
-----------------+---------------
foo_b1 | AUTO
foo_b0 | AUTO
(2 rows)
dbadmin=> SELECT DISTINCT projection_basename FROM projections WHERE anchor_table_name = 'foo';

projection_basename

foo
(1 row)
dbadmin=> SELECT designer_design_projection_encodings('foo', '/home/dbadmin/foo.sql', TRUE, TRUE);
dbadmin=> SELECT projection_name, encoding_type FROM projection_columns WHERE projection_column_name = 'c' AND projection_name ILIKE 'foo%';
projection_name | encoding_type
-----------------+------------------
foo_b1 | COMMONDELTA_COMP
foo_b0 | COMMONDELTA_COMP
(2 rows)

The built-in VERSION function returns a VARCHAR that contains your Vertica node’s version information.

Example:

dbadmin=> SELECT version();
              version
------------------------------------
Vertica Analytic Database v9.1.0-2
(1 row)

The Vertica version is formatted as X.Y.Z-R, where…

• X.Y is the two-digit Vertica major release number, e.g., 8.1, 9.0 and 9.1
• Z is the service pack number, e.g., 7.2.3, 8.1.1 and 9.0.1
• R is the hotfix number, e.g., 9.0.1-9 and 9.1.0-2

Have Fun!

To partition a table by more than one column, use the HASH function!

Example:

dbadmin=> CREATE TABLE some_fact_table (c1 INT NOT NULL, c2 INT NOT NULL, c3 VARCHAR(1) NOT NULL, c4 INT, c5 VARCHAR(100)) PARTITION BY (c1, c2);
ERROR 4331:  PARTITION BY expression cannot return a tuple

dbadmin=> CREATE TABLE some_fact_table (c1 INT NOT NULL, c2 INT NOT NULL, c3 VARCHAR(1) NOT NULL, c4 INT, c5 VARCHAR(100)) PARTITION BY HASH(c1, c2);
CREATE TABLE

Helpful Link:
https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/SQLReferenceManual/Statements/partition-clause.htm

Have fun!

Any sugestions help.
Thanks
Navaneet

Keywords are words that have a specific meaning in the SQL language. Every SQL statement contains one or more keywords. Many keywords are also reserved words. Vertica recommends that you not use reserved words as names for objects, or as identifiers. Including reserved words can make your SQL statements confusing.

You can query the V_CATALOG.STANDARD_KEYWORDS system table to find the list of Vertica SQL reserved words.

Example:

dbadmin=> SELECT *FROM v_catalog.standard_keywords WHERE reserved = 'R' ORDER BY standard_version DESC LIMIT 1;
 keyword | standard_version | reserved
---------+------------------+----------
ABS     |             2011 | R
(1 row)

Notice how the number of reserved words increases by ANSI SQL version:

dbadmin=> SELECT standard_version, COUNT(*) reserved_word_cnt FROM v_catalog.standard_keywords WHERE reserved = 'R' GROUP BY standard_version ORDER BY 1;
 standard_version | reserved_word_cnt
------------------+-------------------
              1992 |               227
              2003 |               288
              2008 |               298
              2011 |               357
(4 rows)

Have Fun!

To convert it, use:

SELECT to_timestamp_tz((<Value in histogram data> / 1000000) + 946684800);

curl -L --negotiate -u : "http://namenode1:9870/webhdfs/v1/tmp/encr/test.txt?op=OPEN"
{"RemoteException":{"exception":"AuthorizationException","javaClassName":"org.apache.hadoop.security.authorize.AuthorizationException","message":"User:hdfs not allowed to do 'DECRYPT_EEK' on 'hdp_cluster_key'"}

Solution is to remove hdfs from the hadoop.kms.blacklist.DECRYPT_EEK property from hdfs-site.xml

curl -L --negotiate -u : "http://namenode1:9870/webhdfs/v1/tmp/encr/test.txt?op=OPEN&user.name=test_user"
a|1
b|2

Vertica's webhdfs uses curl lib underneath the hood to access the webhdfs rest API so removing hdfs from blacklist will also enable Vertica to read from encrypted hdfs zone.

Refer
HWX: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_hdfs_admin_tools/content/webhdfs-hdfs-encr.html
Cloudera: https://www.cloudera.com/documentation/enterprise/5-5-x/topics/cdh_sg_kms_security.html

Suppose I have hourly data files in the pattern hourlyXX.csv where XX is the hour from 00-23. I want to load them into a single staging table.

Let's define the staging table:

dbadmin=> create table hourly (filename VARCHAR, symbol VARCHAR, tm TIME, q1 int, q2 int, q3 int);

COPY data with "filename" set to the source file name or CURRENT_LOAD_SOURCE() and use a wildcard to match all "hourly" files:

dbadmin=> COPY hourly (filename AS CURRENT_LOAD_SOURCE(), symbol, tm, q1, q2, q3) FROM '/tmp/hourly*.csv';

Now let's see what we loaded:

dbadmin=> select * from hourly ;
filename | symbol | tm | q1 | q2 | q3
-------------------+--------+----------+----+----+----
/tmp/hourly00.csv | AAA | 00:01:02 | 1 | 2 | 3
/tmp/hourly00.csv | BBB | 00:02:04 | 1 | 2 | 3
/tmp/hourly01.csv | AAA | 01:01:02 | 1 | 2 | 3
/tmp/hourly01.csv | BBB | 01:02:04 | 1 | 2 | 3
/tmp/hourly02.csv | AAA | 02:01:02 | 1 | 2 | 3
/tmp/hourly02.csv | BBB | 02:02:04 | 1 | 2 | 3

This is another way Vertica can help process data on load and avoid multiple staging steps. Let us know how it works for you!

Reference:
https://www.vertica.com/docs/10.1.x/HTML/Content/Authoring/SQLReferenceManual/Functions/CurrentLoad/CURRENT_LOAD_SOURCE.htm

We have a table named cpu in with the following records in 2 different clusters.

dbadmin=> select * from cpu;
cpu_util | timestamp_utc
----------+---------------
1.86 | 1539251662
1.83 | 1539251662
1.8 | 1539251700
2.06 | 1539251700
1.2 | 1539252100
0.91 | 1539252102
0.63 | 1539252200
1.35 | 1539252400
1.13 | 1539252700
0.97 | 1539252800
1.53 | 1539252800
1.6 | 1539252800
1.02 | 1539252900
1.11 | 1539252900
1.28 | 1539253100
0.87 | 1539253500
0.96 | 1539253500
1.03 | 1539336600
1.13 | 1539336900
1.31 | 1539337200
(20 rows)

dbadmin=>

Results in cluster 1(3 node cluster): It is consistent for every run and it resulted same output as follows.

dbadmin=> select regr_slope (cpu_util,timestamp_utc/86400) AS cpu_util_slope from cpu;

cpu_util_slope

-0.158649414072066
(1 row)

Results in cluster 2(2 node cluster): Results varied for every run. Please find the sample outputs below.

dbadmin=> select regr_slope (cpu_util,timestamp_utc/86400) AS cpu_util_slope from cpu;
cpu_util_slope

-0.158649476255395
(1 row)

dbadmin=> select regr_slope (cpu_util,timestamp_utc/86400) AS cpu_util_slope from cpu;
cpu_util_slope

-0.158649414024238
(1 row)

dbadmin=> select regr_slope (cpu_util,timestamp_utc/86400) AS cpu_util_slope from cpu;
cpu_util_slope

-0.158649476255395
(1 row)

Why is it so?

64-bit float is accurate to 15-16 digits. Summations are going to lose a digit or two from the most significant digit in any of the summed items. Multiplications are also present in regr_slope, which basically cuts the number of digits in half. So you'd expect about 7 digits of precision here, and we have same result till 7 digit precision. Hence all the results are accurate.

Example:

[dbadmin@vertica01 ~]$ admintools -t list_allnodes
Node                    | Host          | State | Version         | DB
-------------------------+---------------+-------+-----------------+--------------
v_port_5432_node0001    | 192.168.2.200 | DOWN  | vertica-9.1.1.1 | port_5432
v_port_5435_node0001    | 192.168.2.200 | DOWN  | vertica-9.1.1.1 | port_5435
v_rename_me_node0001    | 192.168.2.200 | DOWN  | vertica-9.1.1.1 | rename_me
v_rename_me_node0002    | 192.168.2.201 | DOWN  | vertica-9.1.1.1 | rename_me
v_rename_me_node0003    | 192.168.2.202 | DOWN  | vertica-9.1.1.1 | rename_me
v_sfdc_node0001         | 192.168.2.200 | DOWN  | vertica-9.1.1.1 | sfdc
v_test_4_nodes_node0001 | 192.168.2.200 | DOWN  | vertica-9.1.1.1 | test_4_nodes
v_test_4_nodes_node0002 | 192.168.2.201 | DOWN  | vertica-9.1.1.1 | test_4_nodes
v_test_4_nodes_node0003 | 192.168.2.202 | DOWN  | vertica-9.1.1.1 | test_4_nodes
v_test_4_nodes_node0004 | 192.168.2.203 | DOWN  | unavailable     | test_4_nodes
v_test_db_node0001      | 192.168.2.200 | UP    | vertica-9.1.1.1 | test_db
v_test_db_node0002      | 192.168.2.201 | UP    | vertica-9.1.1.1 | test_db
v_test_db_node0003      | 192.168.2.202 | UP    | vertica-9.1.1.1 | test_db

If I only want to see the IP addresses and states of the nodes from a particular database, I could combine the Linux grep and awk commands to limit the output of admintools like so:

[dbadmin@vertica01 ~]$ admintools -t list_allnodes | grep test_db | awk '{print $1,$3,$5}'
v_test_db_node0001 192.168.2.200 UP
v_test_db_node0002 192.168.2.201 UP
v_test_db_node0003 192.168.2.202 UP

Helpful link:
https://my.vertica.com/docs/9.1.x/HTML/index.htm#Authoring/ConceptsGuide/Components/TheAdministrationTools.htm

Have fun!

One of my favorite functions in Vertica is named SPLIT_PART. It splits up a string into parts by a given delimiter.

Example:

dbadmin=> SELECT split_part(my_text, ',', 1) the_first_part,
dbadmin->        split_part(my_text, ',', 2) the_second_part,
dbadmin->        split_part(my_text, ',', 3) the_third_part,
dbadmin->        split_part(my_text, ',', 4) the_fourth_part
dbadmin->   FROM (SELECT 'ONE,TWO,THREE,FOUR' my_text) foo;
the_first_part | the_second_part | the_third_part | the_fourth_part
----------------+-----------------+----------------+-----------------
ONE            | TWO             | THREE          | FOUR
(1 row)

But what if I don’t know how many parts there are in my text? As the following example shows, I can use the ROW NUMBER analytic function to dynamically split my text up!

dbadmin=> SELECT * FROM split_this_up ORDER BY 1;
id |               some_text
----+----------------------------------------
  1 | Please split this sentence up by word
  2 | And do the same with this sentence too
(2 rows)

dbadmin=> SELECT id, word
dbadmin->   FROM (SELECT id, split_part(some_text, ' ', row_number() over (PARTITION BY id)) word
dbadmin(>            FROM split_this_up
dbadmin(>            CROSS JOIN columns) foo
dbadmin->  WHERE word <> '';
id |   word
----+----------
  1 | Please
  1 | split
  1 | this
  1 | sentence
  1 | up
  1 | by
  1 | word
  2 | And
  2 | do
  2 | the
  2 | same
  2 | with
  2 | this
  2 | sentence
  2 | too
(15 rows)

Example:

[dbadmin@SE-Sandbox-26-node1 ~]$ cat beep.sql
SELECT version();
SELECT sysdate;
SELECT user;
SELECT 'SOME_DATA';
SELECT 'SOME MORE DATA';
-- Alot more stuff...
SELECT sysdate;

[dbadmin@SE-Sandbox-26-node1 ~]$ vsql -bf beep.sql -o beep.log

<< I heard a “Beep” >>

[dbadmin@SE-Sandbox-26-node1 ~]$ cat beep.log
              version
------------------------------------
Vertica Analytic Database v9.2.1-5
(1 row)

          sysdate
----------------------------
2019-09-03 15:54:41.510405
(1 row)

current_user
--------------
dbadmin
(1 row)

?column?
-----------
SOME_DATA
(1 row)

   ?column?
----------------
SOME MORE DATA
(1 row)

          sysdate
---------------------------
2019-09-03 15:54:41.53582
(1 row)

Helpful Link:
https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/ConnectingToVertica/vsql/CommandLineOptions.htm

Have fun!

[dbadmin@vdb2 ~]$ admintools -t start_db -d opsadb -U
Info: no password specified, using none
    Starting nodes: 
        v_opsadb_node0001 (192.168.252.31)
        v_opsadb_node0002 (192.168.252.216)
        v_opsadb_node0003 (192.168.252.217)
    Starting Vertica on all nodes. Please wait, databases with a large catalog may take a while to initialize.
    Node Status: v_opsadb_node0001: (DOWN) v_opsadb_node0002: (DOWN) v_opsadb_node0003: (DOWN) 
    Node Status: v_opsadb_node0001: (DOWN) v_opsadb_node0002: (DOWN) v_opsadb_node0003: (DOWN) 
    Node Status: v_opsadb_node0001: (DOWN) v_opsadb_node0002: (DOWN) v_opsadb_node0003: (DOWN) 
    Node Status: v_opsadb_node0001: (DOWN) v_opsadb_node0002: (DOWN) v_opsadb_node0003: (DOWN) 
    Node Status: v_opsadb_node0001: (DOWN) v_opsadb_node0002: (DOWN) v_opsadb_node0003: (DOWN) 
    Node Status: v_opsadb_node0001: (DOWN) v_opsadb_node0002: (DOWN) v_opsadb_node0003: (DOWN) 
    Node Status: v_opsadb_node0001: (UP) v_opsadb_node0002: (UP) v_opsadb_node0003: (UP) 
Database opsadb: Startup Succeeded.  All Nodes are UP
[dbadmin@vdb2 ~]$ vsql 
...
dbadmin=> SELECT get_ahm_epoch();
 get_ahm_epoch 
---------------
     109626381
(1 row)
dbadmin=> SELECT get_expected_recovery_epoch();
INFO 4544:  Recovery Epoch Computation:
Node Dependencies:
011 - cnt: 847
101 - cnt: 847
110 - cnt: 847
111 - cnt: 158

001 - name: v_opsadb_node0001
010 - name: v_opsadb_node0002
100 - name: v_opsadb_node0003
Nodes certainly in the cluster:
    Node 2(v_opsadb_node0003), epoch 109610889
    Node 1(v_opsadb_node0002), epoch 104348833
Filling more nodes to satisfy node dependencies:
Data dependencies fulfilled, remaining nodes LGEs don't matter:
    Node 0(v_opsadb_node0001), epoch 104348128
--
 get_expected_recovery_epoch 
-----------------------------
                   104348833
(1 row)

So far so good, we should be able to recover to 104348833. But:

[dbadmin@vdb2 ~]$ admintools -t restart_db -d opsadb -e '104348833' -p xxx
Invalid value for last good epoch: '104348833'
Epoch number must be 'last' or between 109626381 and 104348833 inclusive

I have tried various values for the epoch, including the 2 mentioned values - always the same error message. Now my questions:
1. The range is backwards, AHM > LGE. Is that the reason for the (poor!) error message? Is it that my recovery epoch must be >=AHM and <=LGE, and therefore I don't have a chance of (normal) recovery here?
2. Can someone come up with a hypothesis about what must have gone wrong to end up like that? As mentioned, the power loss affected the 3 nodes and the storage, but no disks were damaged. I thought the DB can always be recovered if the files that made it to the disk are unharmed ...

I know (after reading https://softwaresupport.softwaregrp.com/doc/KM03449287 ) that I can try to salvage table data, but more than 2000 projections are affected. As it is a test env I will just revert to an older snapshot - this post is only for my curiosity.
Thank you for any insights!

When using the Re-IP tool, you need to perform some configuration steps to change any associated IP address. To learn more, see https://www.vertica.com/kb/Re-IP/Content/FAQs/Re-IP.htm

You can use access policies in Vertica to show only limited data to users. We took an example of Credit Card numbers and hid few characters to protect sensitive data.

Let us create a table with two database users and provide necessary permissions to those users.

$ vsql -U dbadmin -w password 
dbadmin=> DROP TABLE customers_table CASCADE;
dbadmin=> CREATE TABLE customers_table  ( 
            ID      INT, 
            Name    VARCHAR(20), 
            CARDNUM VARCHAR(20)
          );
INSERT INTO customers_table VALUES ( 1, 'Tom','1234567809094321');
INSERT INTO customers_table VALUES ( 2, 'Jerry','4321567809096789');
COMMIT;

-- create users 
dbadmin=> CREATE user mgr identified by 'password';
dbadmin=> CREATE user opt identified by 'password';
dbadmin=> GRANT USAGE ON SCHEMA public to mgr,opt;
dbadmin=> GRANT SELECT on public.customers_table to mgr,opt;
-- create Roles and grant to users
dbadmin=> CREATE ROLE manager;
dbadmin=> CREATE ROLE operator;
dbadmin=> grant manager to mgr;
dbadmin=> grant operator to opt;
-- give default role operator to opt user only -- 
dbadmin=> alter user opt default role operator;

-- create Colum level Access Policy 

dbadmin=> CREATE ACCESS POLICY ON customers_table FOR COLUMN CARDNUM
    CASE
      WHEN ENABLED_ROLE('manager') THEN CARDNUM
      WHEN ENABLED_ROLE('operator') THEN 
          ( SUBSTR(CARDNUM, 1, 4)||'-XXXX-XXXX-'||SUBSTR(CARDNUM, 13, 4) )
      ELSE NULL
END ENABLE;

dbadmin=> \q

-- Login again and see what dbadmin user can see. 
$ vsql -U dbadmin -w password 
dbadmin=> SELECT * FROM  public.customers_table;
 ID | Name  | CARDNUM
----+-------+---------
  1 | Tom   |
  2 | Jerry |
(2 rows)
dbadmin=> \q

Even the admin user dbadmin cannot see the full credit card number. Now, let us see what values are visible to user mgr:

$ vsql -U mgr -w password
mgr=>
mgr=> SELECT * FROM  public.customers_table;
 ID | Name  | CARDNUM
----+-------+---------
  1 | Tom   |
  2 | Jerry |
(2 rows)
mgr=> set role manager;
SET
mgr=> SELECT * FROM  public.customers_table;
 ID | Name  |     CARDNUM
----+-------+------------------
  1 | Tom   | 1234567809094321
  2 | Jerry | 4321567809096789
(2 rows)

mgr=> \q

Let us see what values are visible to user opt:

$ Vsql -U opt -w password
opt=>
opt=> SELECT * FROM  public.customers_table;
 ID | Name  |       CARDNUM
----+-------+---------------------
  1 | Tom   | 1234-XXXX-XXXX-4321
  2 | Jerry | 4321-XXXX-XXXX-6789
(2 rows)

opt=> \q

Depending on the roles, users can either see the full credit card number or masked data.

At the bottom of the report there is a “total” line that shows the database raw and compressed size!

Example:

[dbadmin@vertica8 ~]$ /opt/vertica/scripts/collect_diag_dump.sh -c -s false
Database Password:
Exporting profiling data........ Done
Exporting catalog and design.. Done
Computing compression  Done
Forming archive file  Done
Diagnostics dump available at ./diag_dump_20181008102602.tar.gz

[dbadmin@vertica8 ~]$ tar -xvzf ./diag_dump_20181008102602.tar.gz
tmp/Out_20181008102602/
tmp/Out_20181008102602/execution_engine_profiling_plans
tmp/Out_20181008102602/out_execution_engine_profiles
tmp/Out_20181008102602/out_dc_plans
tmp/Out_20181008102602/out_dc_plan_steps
tmp/Out_20181008102602/out_dc_plan_step_properties
tmp/Out_20181008102602/out_dc_plan_step_output_schema
tmp/Out_20181008102602/out_dc_plan_parents
tmp/Out_20181008102602/design
tmp/Out_20181008102602/planning_objects
tmp/Out_20181008102602/version
tmp/Out_20181008102602/compression

[dbadmin@vertica8 ~]$ cat tmp/Out_20181008102602/compression
                                TableName     PjCnt    PjType            RowCount         RawSize(MB)         Vertica(MB)         Compr.Ratio
=============================================================================================================================================
                              public.test         1         S           268435456              986.18              257.74                3.83
                   some_schema.some_table         1         S             4194304                8.00                0.02              400.00
_____________________________________________________________________________________________________________________________________________
                                    TOTAL                                                      994.18              257.76                3.86 

Helpful links:
https://www.vertica.com/docs/9.1.x/HTML/index.htm#Authoring/AdministratorsGuide/Diagnostics/ExportingProfilingData.htm

Have fun!

To be able to copy a cluster in Vertica, you must provide the same node name on the target. However, you might have nonsequential node names in the source database as in the following:

$ admintools -t list_allnodes
Node              | Host           | State | Version          | DB
-------------------+----------------+-------+------------------+--------
v_prodee_node0004 | 192.168.51.201 | UP    | vertica-9.3.1.10 | prodee
v_prodee_node0005 | 192.168.51.202 | UP    | vertica-9.3.1.10 | prodee
v_prodee_node0006 | 192.168.51.203 | UP    | vertica-9.3.1.10 | prodee

In this case, you would need to create a database with the desired node names to copy a cluster.

To do this,

1. Modify the following [nodes] section in admintools.conf with the desired node names:

[nodes]
node0004 = 192.168.51.204,/home/dbadmin,/home/dbadmin
node0005 = 192.168.51.205,/home/dbadmin,/home/dbadmin
node0006 = 192.168.51.206,/home/dbadmin,/home/dbadmin
v_prodee_node0004 = 192.168.51.204,/home/dbadmin,/home/dbadmin
v_prodee_node0005 = 192.168.51.205,/home/dbadmin,/home/dbadmin
v_prodee_node0006 = 192.168.51.206,/home/dbadmin,/home/dbadmin

2. Run the following create db command by providing node names in the hosts list:

$ admintools -t create_db -d prodee --compat21 -s v_prodee_node0004,v_prodee_node0005,v_prodee_node0006

Info: no password specified, using none
WARNING: --compat21 is deprecated as of Vertica. Support may be removed in future versions.
        Creating database prodee

3. Check the database status.

$ admintools -t list_allnodes
Node              | Host           | State | Version          | DB
-------------------+----------------+-------+------------------+--------
v_prodee_node0004 | 192.168.51.204 | UP    | vertica-9.3.1.10 | prodee
v_prodee_node0005 | 192.168.51.205 | UP    | vertica-9.3.1.10 | prodee
v_prodee_node0006 | 192.168.51.206 | UP    | vertica-9.3.1.10 | prodee

The vsql \timing meta-command reports, in milliseconds, the length of time it takes each SQL statement to execute.

Example:

dbadmin=> \timing
Timing is on.

dbadmin=> SELECT COUNT(*) FROM big_varchar_table WHERE pk BETWEEN 1000000 AND 2000000;
  COUNT
---------
1000001
(1 row)
Time: First fetch (1 row): 179.609 ms. All rows formatted: 179.691 ms

You can also enable \timing from the command line using the vsql -i command.

Example:

[dbadmin@s18384357 ~]$ vsql -ic "SELECT COUNT(*) FROM big_varchar_table WHERE pk BETWEEN 1000000 AND 2000000;"
  COUNT
---------
1000001
(1 row)
Time: First fetch (1 row): 52.728 ms. All rows formatted: 52.881 ms

Have Fun!

Vertica handles the SIGSEGV signal (segmentation fault) well by logging the information to vertica.log, ErrorReport.txt, and the core dump files. If Vertica is killed with the SIGKILL signal such as kill -9, there is no information in any of the log files (in Vertica or system side). Additionally, there is no information even if Vertica receives SIGSEGV signal. In this case, you need to capture the signal information on system side.

Following are two ways to capture this information, auditd and SystemTap.

auditd (Linux Auditing System)
The audit daemon writes audit records to the disk. It captures all system audit events including system calls and watches the file system objects. To capture the specific system calls, you need to configure auditd. auditd records any process that ends abnormally without additional configuration as the event is one of the system audit events.

To enable auditd to capture kill system call, run the following commands after starting up the auditd daemon:

$ auditctl -a exit,always -F arch=b64 -S kill -k audit_kill
$ auditctl -a exit,always -F arch=b64 -S tkill -k audit_kill
$ auditctl -a exit,always -F arch=b64 -S tgkill -k audit_kill

These configurations are temporary and exist only until restarting auditd or rebooting the system. To configure permanently, add the following lines to /etc/audit/audit.rules and restart auditd:

-a exit,always -F arch=b64 -S kill -k audit_kill
-a exit,always -F arch=b64 -S tkill -k audit_kill
-a exit,always -F arch=b64 -S tgkill -k audit_kill

Run the following commands to list the current configuration:

$ auditctl -l
-a always,exit -F arch=b64 -S kill -F key=audit_kill
-a always,exit -F arch=b64 -S tkill -F key=audit_kill
-a always,exit -F arch=b64 -S tgkill -F key=audit_kill

In the this example, 'audit_kill' is specified as the value of -k option. This is the filter key that can be used to search the audit records later.

Try to kill the Vertica process after enabling auditd to capture the kill system call:

$ killall -9 vertica

$ tail -f /var/log/audit/audit.log
type=SYSCALL msg=audit(1592901047.912:529): arch=c000003e syscall=62 success=yes exit=0 a0=f66 a1=9 a2=0 a3=22d items=0 ppid=4172 pid=4173 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="killall" exe="/usr/bin/killall" key="audit_kill"
type=OBJ_PID msg=audit(1592901047.912:529): opid=3942 oauid=1000 ouid=1001 oses=1 ocomm="vertica"
type=PROCTITLE msg=audit(1592901047.912:529): proctitle=6B696C6C616C6C002D390076657274696361

Search the audit records by using the filter key:

$ ausearch -i -k audit_kill
----
type=PROCTITLE msg=audit(06/23/2020 17:30:47.912:529) : proctitle=killall -9 vertica
type=OBJ_PID msg=audit(06/23/2020 17:30:47.912:529) : opid=3942 oauid=duser ouid=dbadmin oses=1 ocomm=vertica
type=SYSCALL msg=audit(06/23/2020 17:30:47.912:529) : arch=x86_64 syscall=kill success=yes exit=0 a0=0xf66 a1=SIGKILL a2=0x0 a3=0x22d items=0 ppid=4172 pid=4173 auid=duser uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 ses=1 comm=killall exe=/usr/bin/killall key=audit_kill

This outputenables you to figure out that /usr/bin/killall sent SIGKILL signal successfully to the Vertica process with PID 3942.

With the default configuration for auditd, the log file is rotated. If you want to search the records in the rotated files, use -if/--input option as in the following:

$ ausearch -i -k audit_kill -if /var/log/audit/audit.log.1

To search audit records for the process that ended abnormally, run the following command:

$ ausearch -i -m ANOM_ABEND
----
type=ANOM_ABEND msg=audit(06/22/2020 18:26:21.915:8331836) : auid=dbadmin uid=dbadmin gid=verticadba ses=458 pid=32595 comm=rdk:main exe=/opt/vertica/bin/vertica sig=SIGSEGV res=yes

This output enables you to figure out that the Vertica process was ended by the SIGSEGV signal and the segmentation fault is associated with rdk:main.

In most cases, auditd is installed by default. If not, install the audit package for RHEL/CentOS/SUSE or the auditd package for Debian/Ubuntu.

SystemTap

SystemTap is a tracing and probing tool that allows you to monitor the activities of the operating system by simply running user-written SystemTap scripts.

Install the following packages:

• systemtap
• systemtap-runtime

Additionally, you need to install the following packages to get information about the kernel. These packages must be for your kernel version.

• kernel-debuginfo
• kernel-debuginfo-common
• kernel-devel

To determine what kernel version your system is currently using, run the following command:

$ uname -r
3.10.0-1062.12.1.el7.x86_64

For example, if your kernel version is 3.10.0-1062.12.1.el7.x86_64, then you need to install the following RPMs:

kernel-debuginfo-3.10.0-1062.12.1.el7.x86_64.rpm
kernel-debuginfo-common-x86_64-3.10.0-1062.12.1.el7.x86_64.rpm
kernel-devel-3.10.0-1062.12.1.el7.x86_64.rpm

In case the kernel version is not the latest, these RPMs may not exist in the repository and you may need to install them manually. The following RPMs are for CentOS 7.7:

http://debuginfo.centos.org/7/x86_64/kernel-debuginfo-3.10.0-1062.12.1.el7.x86_64.rpm
http://debuginfo.centos.org/7/x86_64/kernel-debuginfo-common-x86_64-3.10.0-1062.12.1.el7.x86_64.rpm
http://vault.centos.org/7.7.1908/updates/x86_64/Packages/kernel-devel-3.10.0-1062.12.1.el7.x86_64.rpm

For Ubuntu, refer to https://wiki.ubuntu.com/Kernel/Systemtap.
For Debian, refer to https://wiki.debian.org/DebugPackage.

Create the SystemTap script as in the following to capture the signal sent to the specific process:

 #! /usr/bin/env stap

#
# sigmon.stp for capturing the signals.
#

probe begin
{
  printf("%-8s %-16s %-5s %-16s %6s %-16s\n",
         "SPID", "SNAME", "RPID", "RNAME", "SIGNUM", "SIGNAME")
}

probe signal.send 
{
  if (sig_pid == target())
    printf("%-8d %-16s %-5d %-16s %-6d %-16s\n", 
           pid(), execname(), sig_pid, pid_name, sig, sig_name)
}

Run SystemTap with the above script in the background by providing Vertica process ID.

$ nohup stap -x `pgrep -o vertica` /tmp/sigmon.stp > /tmp/sigmon.log 2>&1 &

Try to kill the Vertica process after running SystemTap script:

$ killall -9 vertica

$ tail -f /tmp/sigmon.log
SPID     SNAME            RPID  RNAME            SIGNUM SIGNAME
9707     killall          6555  vertica          9      SIGKILL

This output enables you to figure that killall sent SIGKILL signal to the Vertica process with PID 6555.

CREATE TABLE public.my_table
(
    order_no int,
    order_date timestamp NOT NULL,
    shipper varchar(20),
    ship_date date
)
PARTITION BY DATE(order_date);

\set DEMO_ROWS 10

INSERT INTO public.my_table
WITH myrows AS (SELECT
row_number() over()    AS order_no,
NOW() - RANDOMINT(30) AS order_date
FROM ( SELECT 1 FROM ( SELECT NOW() as se union all
SELECT NOW() + :DEMO_ROWS - 1 AS se) a timeseries ts AS '1 day' OVER (ORDER BY se)) b)
SELECT order_no, order_date, 'something', order_date AS ship_date
FROM myrows
ORDER BY order_no;
COMMIT;

WITH my_list AS
(SELECT max(order_date) as oldest from public.my_table)
SELECT date(order_date),
       CASE WHEN order_date = oldest THEN 'Will be droped' ELSE 'Will keep' END as Status
FROM public.my_table, my_list ORDER BY 1;

-- Option A:
\set OLDEST `vsql -XAtc "SELECT max(order_date)::date from public.my_table;"`
\set OLDEST '''':OLDEST''''
SELECT DROP_PARTITIONS ('my_table', :OLDEST, :OLDEST);

SELECT DATE(order_date) FROM public.my_table ORDER BY 1;

-- Option B:
WITH my_list AS
(SELECT max(order_date) as oldest from public.my_table)
SELECT date(order_date),
       CASE WHEN order_date = oldest THEN 'Will be droped' ELSE 'Will keep' END as Status
FROM public.my_table, my_list ORDER BY 1;

CREATE OR REPLACE FUNCTION my_drop(d2d DATE) RETURN VARCHAR(100)
   AS BEGIN
     RETURN 'SELECT DROP_PARTITIONS (''my_table'', ''' || d2d || ''' , ''' || d2d || ''');';
   END;

\t
\o | vsql
SELECT my_drop((SELECT max(order_date)::DATE from public.my_table));
\o
\t

SELECT DATE(order_date) FROM public.my_table ORDER BY 1;

Here is the run time output:

CREATE TABLE public.my_table
(
    order_no int,
    order_date timestamp NOT NULL,
    shipper varchar(20),
    ship_date date
)
PARTITION BY DATE(order_date);
CREATE TABLE
INSERT INTO public.my_table
WITH myrows AS (SELECT
row_number() over()    AS order_no,
NOW() - RANDOMINT(30) AS order_date
FROM ( SELECT 1 FROM ( SELECT NOW() as se union all
SELECT NOW() + 10 - 1 AS se) a timeseries ts AS '1 day' OVER (ORDER BY se)) b)
SELECT order_no, order_date, 'something', order_date AS ship_date
FROM myrows
ORDER BY order_no;
 OUTPUT
--------
     10
(1 row)

COMMIT;
COMMIT

WITH my_list AS
(SELECT max(order_date) as oldest from public.my_table)
SELECT date(order_date),
       CASE WHEN order_date = oldest THEN 'Will be droped' ELSE 'Will keep' END as Status
FROM public.my_table, my_list ORDER BY 1;
    date    |     Status
------------+----------------
 2020-05-01 | Will keep
 2020-05-02 | Will keep
 2020-05-06 | Will keep
 2020-05-07 | Will keep
 2020-05-07 | Will keep
 2020-05-10 | Will keep
 2020-05-13 | Will keep
 2020-05-20 | Will keep
 2020-05-22 | Will keep
 2020-05-24 | Will be droped
(10 rows)

SELECT DROP_PARTITIONS ('my_table', '2020-05-24', '2020-05-24');
  DROP_PARTITIONS
-------------------
 Partition dropped
(1 row)

SELECT DATE(order_date) FROM public.my_table ORDER BY 1;
    DATE
------------
 2020-05-01
 2020-05-02
 2020-05-06
 2020-05-07
 2020-05-07
 2020-05-10
 2020-05-13
 2020-05-20
 2020-05-22
(9 rows)

WITH my_list AS
(SELECT max(order_date) as oldest from public.my_table)
SELECT date(order_date),
       CASE WHEN order_date = oldest THEN 'Will be droped' ELSE 'Will keep' END as Status
FROM public.my_table, my_list ORDER BY 1;
    date    |     Status
------------+----------------
 2020-05-01 | Will keep
 2020-05-02 | Will keep
 2020-05-06 | Will keep
 2020-05-07 | Will keep
 2020-05-07 | Will keep
 2020-05-10 | Will keep
 2020-05-13 | Will keep
 2020-05-20 | Will keep
 2020-05-22 | Will be droped
(9 rows)

CREATE OR REPLACE FUNCTION my_drop(d2d DATE) RETURN VARCHAR(100)
   AS BEGIN
     RETURN 'SELECT DROP_PARTITIONS (''my_table'', ''' || d2d || ''' , ''' || d2d || ''');';
   END;
CREATE FUNCTION

Showing only tuples.
SELECT my_drop((SELECT max(order_date)::DATE from public.my_table));
  DROP_PARTITIONS
-------------------
 Partition dropped
(1 row)

Tuples only is off.
SELECT DATE(order_date) FROM public.my_table ORDER BY 1;
    DATE
------------
 2020-05-01
 2020-05-02
 2020-05-06
 2020-05-07
 2020-05-07
 2020-05-10
 2020-05-13
 2020-05-20
(8 rows)

https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/AdministratorsGuide/ManageNodes/ReplacingANodeUsingTheSameNameAndIPAddress.htm?tocpath=Administrator%27s%20Guide%7CManaging%20the%20Database%7CManaging%20Nodes%7CReplacing%20Nodes%7C_____1

If you are using a version prior to 9.3 and the node with the old IP address is not reachable, follow these steps to replace the host with the new IP address. In these steps we have used dbadmin user as the database administrator. If you have a different user for database administrator, edit the script accordingly and pass –u to install_vertica command in step 6.

1. Login to the database and run the following command:
   • ALTER NODE <failed Node name> hostname '<NEW_IPADDRESS>' control hostname '<NEW_IPADDRESS>';
   • Select reload_spread(true);
2. Replace the old_ipaddress with new_ipaddress in /opt/vertica/config/admintools.conf file on a good node. Use configuration options to distribute admintools.conf file to all nodes in the cluster.
3. Login to the new host as root.
4. Run “id dbadmin” to verify dbadmin user exists. If not, create the user with verticadba as a primary group.
5. Install the vertica rpm usingrpm -Uvh verticaxxx.rpm. (preferably place the rpm in /tmp location)
6. Execute install_vertica -s locahost -r /tmp/verticaxx.rpm. This step fixes the file permission on folders in /opt/vertica directory.
7. Switch to dbadmin user by running “su – dbadmin”.
8. SSH to any other host in the cluster as dbadmin. If passwordless SSH is not setup, copy the following three files from any other node to the ~/.ssh/ folder and ensure permissions are identical to other nodes:
   a. ~/.ssh/id_rsa
   b. ~/.ssh/id_rsa.pub
   c. ~/.ssh/ authorized_keys
9. Once the passwordless SSH is fixed, ensure you are able to connect to and from other hosts in the cluster.
10. On a good host open admintools and use configuration options to distribute admintools.conf file and database configuration file to all nodes in the cluster.
11. Get a list of data, catalog, and other storage locations if any exist by running the following query:
    select storage_path, storage_usage from disk_storage where node_name ilike '<down node name>';
12. Connect to the new host as dbadmin and create directories using the output from the above query “mkdir -p storage_path” .
    Note: Ensure permissions and ownership of these directories match other nodes.
13. From a good host, use admintools command line to restart the node that is down with force flag:
    /opt/vertica/bin/admintools -t restart_node -d dbname -s down_node_ipaddress -F

1. Login to the new host as root.
2. Run “id dbadmin” to verify dbadmin user exists. If not, create the user with verticadba as a primary group.
3. Install the vertica rpm using rpm -Uvh verticaxxx.rpm. (preferably place the rpm in /tmp location)
4. Executeinstall_vertica -s locahost -r /tmp/verticaxx.rpm. This step fixes the file permission on folders in /opt/vertica directory.
5. Switch to dbadmin user by running “su – dbadmin”.
6. SSH to any other host in the cluster as dbadmin. If passwordless SSH is not setup, copy the following three files from any other node to the ~/.ssh/ folder and ensure permissions are identical to other nodes:
   a. ~/.ssh/id_rsa
   b. ~/.ssh/id_rsa.pub
   c. ~/.ssh/ authorized_keys
7. Once the passwordless SSH is fixed, ensure you are able to connect to and from other hosts in the cluster.
8. On a good host open admintools and use configuration options to distribute admintools.conf file and database configuration file to all nodes in the cluster.
9. Get a list of data, catalog, and other storage locations if any exist by running the following query:
   select storage_path, storage_usage from disk_storage where node_name ilike '<down node name>';
10. Connect to the new host as dbadmin and create directories using the output from the above query “mkdir -p storage_path” .
    Note: Ensure permissions and ownership of these directories match other nodes.
11. From a good host, use admintools command line to restart the node that is down with force flag:
    /opt/vertica/bin/admintools -t restart_node -d dbname -s down_node_ipaddress -F

Ever wonder if it’s accurate? A simple test shows that it is 100 percent accurate if you specify an error tolerance of 0 and a confidence level of 100.

Example:

First let’s audit the table BIG_TABLE.

[dbadmin@s18384357 ~]$ /opt/vertica/bin/vsql -c "SELECT audit('public.big_table', 0, 100);"
  audit
----------
10696205
(1 row)

Next extract the data from the table to a file, eliminating all field separators and record terminators as Vertica does not include those bytes in the audit.

[dbadmin@s18384357 ~]$ /opt/vertica/bin/vsql -o /home/dbadmin/big_table.out -F '' -R '' -At -c "SELECT * FROM public.big_table;" -q

Here is the exported file size:

[dbadmin@s18384357 ~]$ stat -c %s /home/dbadmin/big_table.out
10696206

Wait a second! Why is the output file size 1 byte bigger than the result of the Vertica AUDIT function? It’s because there is a training new line character in the file. Let’s get rid of it!

[dbadmin@s18384357 ~]$ printf %s "$(< /home/dbadmin/big_table.out)" > /home/dbadmin/big_table_no_trailing_nl.out

[dbadmin@s18384357 ~]$ stat -c %s /home/dbadmin/big_table_no_trailing_nl.out
10696205

Now we are 100% accurate!

Helpful Links:
https://www.vertica.com/docs/9.1.x/HTML/index.htm#Authoring/SQLReferenceManual/Functions/VerticaFunctions/LicenseManagement/AUDIT.htm
https://www.vertica.com/docs/9.1.x/HTML/index.htm#Authoring/AdministratorsGuide/Licensing/CalculatingTheDatabaseSize.htm

Have fun!

When you have millions to billions of rows, data type becomes a bit more important: even an extra 10 bytes per row across a huge data set will impact storage or performance (or both!).

When I created the big_fact_table, I included some VARCHAR fields, but failed to set a width. Vertica defaults to VARCHAR(80) in this case. However, all of my strings are 8-character alpahnumerics, so why have VARCHAR(80) when VARCHAR(10) will suffice? I created a new table named "varchar10" with the same fields but altered the VARCHAR fields to VARCHAR(10) with ALTER TABLE...ALTER COLUMN...SET DATA TYPE VARCHAR(10).

This had almost no impact on table size thanks to compression and encoding:

dbadmin=> select anchor_table_name, projection_name, used_bytes from projection_storage where anchor_table_name like 'big_fact_table%' limit 5;
anchor_table_name | projection_name | used_bytes
-----------------------------+-----------------------------------------+------------
big_fact_table | big_fact_table_super | 6843970195
big_fact_table_varchar10 | big_fact_table_varchar10_super | 6843968398

BUT, when I load data with INSERT...SELECT, I see a significant performance improvement with the smaller VARCHAR:

dbadmin=> insert into public.big_fact_table select * from public.big1090;

OUTPUT

445580312
(1 row)
Time: First fetch (1 row): 380333.748 ms. All rows formatted: 380333.823 ms
dbadmin=> insert into public.big_fact_table_varchar10 select * from public.big1090;

OUTPUT

445580312
(1 row)
Time: First fetch (1 row): 269333.788 ms. All rows formatted: 269333.864 ms

That is around 25% faster! Why? During load and query, Vertica must allocate enough resources assuming the fields will actually contain the full data size. If you allocate too much memory, this will slow down sorts, segmentation, and so on as Vertica needs to move around data types that have lots of empty space. So optimizing data type and length helps Vertica optimize memory usage and run faster!

You can check whether you've allocated too much space in a VARCHAR or VARBINARY with SELECT MAX(LENGTH(<column_name>)) FROM <table_name>; and compare to the column DDL.

For more info and limitations: https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/AdministratorsGuide/Tables/ColumnManagement/ReduceColumnWidth.htm

Vertica executes DEFAULT queries only on new rows when they are added to the flattened table, through load operations such as INSERT and COPY. Thereafter, changes in the original data sources have no effect on the flattened table.

Vertica executes SET USING queries only when you invoke the function REFRESH_COLUMNS. Load operations set SET USING columns in new rows to NULL. After the load, you must call REFRESH_COLUMNS to populate these columns from the queried tables. This can be useful in two ways: you can defer the overhead of updating the flattened table to any time that is convenient; and you can repeatedly query source tables for new data.

A flattened table column is not limited to having either a DEFAULT or SET USING constraint. It can have both constriants!

Example:

dbadmin=> CREATE TABLE customer_dim (customer_id INT, customer_name VARCHAR(100));
CREATE TABLE

dbadmin=> INSERT INTO customer_dim SELECT 1, 'LUKE';
 OUTPUT
--------
      1
(1 row)

dbadmin=> INSERT  NTO customer_dim SELECT 2, 'HAN';
 OUTPUT
--------
      1
(1 row)

dbadmin=> INSERT INTO customer_dim SELECT 3, 'LANDO';
 OUTPUT
--------
      1
(1 row)

dbadmin=> CREATE TABLE customer_fact (pk INT, customer_id INT, customer_name VARCHAR(100) DEFAULT USING (SELECT customer_name FROM customer_dim WHERE customer_id = customer_fact.customer_id));
CREATE TABLE

dbadmin=> INSERT INTO customer_fact (pk, customer_id) SELECT 1, 2;
 OUTPUT
--------
      1
(1 row)

dbadmin=> INSERT INTO customer_fact (pk, customer_id) SELECT 2, 1;
 OUTPUT
--------
      1
(1 row)

dbadmin=> SELECT * FROM customer_fact;
 pk | customer_id | customer_name
----+-------------+---------------
  1 |           2 | HAN
  2 |           1 | LUKE
(2 rows)


dbadmin=> UPDATE customer_dim SET customer_name = 'BEN' WHERE customer_id = 1;
 OUTPUT
--------
      1
(1 row)

dbadmin=> SELECT refresh_columns('customer_fact', 'customer_name', 'UPDATE'); COMMIT;
      refresh_columns
---------------------------
 refresh_columns completed
(1 row)

COMMIT

dbadmin=> SELECT * FROM customer_fact;
 pk | customer_id | customer_name
----+-------------+---------------
  1 |           2 | HAN
  2 |           1 | BEN
(2 rows)

Useful Links:
https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/AnalyzingData/FlattenedTables/FlattenedTables.htm
https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/AnalyzingData/FlattenedTables/SetUsingVersusDefault.htm

Have Fun!

Here is the shapefile source: https://www.census.gov/geographies/mapping-files/time-series/geo/carto-boundary-file.html

Obtain the file for ZIP Code Tabulation Areas (ZCTAs): cb_2018_us_zcta510_500k.zip [59 MB] (2018 file shown; you can use a newer year, if available)

I unpacked the shapefile download to /data1/bryan/geo/

Listing:

-rw-rw---- 1 bryan bryan 165 Apr 15 2019 cb_2018_us_zcta510_500k.prj
-rw-rw---- 1 bryan bryan 5 Apr 15 2019 cb_2018_us_zcta510_500k.cpg
-rw-rw---- 1 bryan bryan 265252 Apr 15 2019 cb_2018_us_zcta510_500k.shx
-rw-rw---- 1 bryan bryan 91072224 Apr 15 2019 cb_2018_us_zcta510_500k.shp
-rw-rw---- 1 bryan bryan 1756826 Apr 15 2019 cb_2018_us_zcta510_500k.dbf
-rwxrwxrwx 1 bryan bryan 34290 Apr 15 2019 cb_2018_us_zcta510_500k.shp.iso.xml
-rwxrwxrwx 1 bryan bryan 9052 Apr 15 2019 cb_2018_us_zcta510_500k.shp.ea.iso.xml

Be sure to chmod or chown files so they are readable by Vertica!

Follow the documentation: https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/AnalyzingData/Geospatial/GeospatialAnalytics/LoadingSpatialDataFromShapefiles.htm

Step 1: Use STV_ShpCreateTable to generate a CREATE TABLE statement.

dbadmin=> SELECT STV_ShpCreateTable ( USING PARAMETERS file = '/data1/bryan/geo/cb_2018_us_zcta510_500k.shp') OVER() AS zcta;

zcta

CREATE TABLE cb_2018_us_zcta510_500k(
gid IDENTITY(64) PRIMARY KEY,
ZCTA5CE10 VARCHAR(5),
AFFGEOID10 VARCHAR(14),
GEOID10 VARCHAR(5),
ALAND10 INT8,
AWATER10 INT8,
geom GEOMETRY(99582)
);
(9 rows)

Step 2: Create the table.

dbadmin=> CREATE TABLE cb_2018_us_zcta510_500k(
dbadmin(> gid IDENTITY(64) PRIMARY KEY,
dbadmin(> ZCTA5CE10 VARCHAR(5),
dbadmin(> AFFGEOID10 VARCHAR(14),
dbadmin(> GEOID10 VARCHAR(5),
dbadmin(> ALAND10 INT8,
dbadmin(> AWATER10 INT8,
dbadmin(> geom GEOMETRY(99582)
dbadmin(> );
CREATE TABLE

Step 3: Load the shapefile.

dbadmin=> COPY cb_2018_us_zcta510_500k WITH SOURCE STV_ShpSource(file='/data1/bryan/geo/cb_2018_us_zcta510_500k.shp') PARSER STV_ShpParser();

Rows Loaded

   33144

(1 row)

That was quick and easy! Let's look at a record:

dbadmin=> select gid,ZCTA5CE10,AFFGEOID10,GEOID10,ALAND10,AWATER10,ST_AsText(geom) from cb_2018_us_zcta510_500k where zcta5ce10 = '11554' limit 5;
gid | ZCTA5CE10 | AFFGEOID10 | GEOID10 | ALAND10 | AWATER10 | ST_AsText (* geom field, truncated for publication) ST_AsText
------+-----------+----------------+---------+----------+----------+-----------------------------------------------------------------------------
1712 | 11554 | 8600000US11554 | 11554 | 16186098 | 61309 | POLYGON ((-73.587642 40.747566, -73.579641 40.749524, -73.578799 40.746505,
(1 row)

The key fields we'll look at are the ZIP code, "ZCTA5CE10", and the geometry of the ZIP code tract, "geom".

Now let's test the data by verifying the ZIP code of my ADS-B receiver, which is at longitude (X) -73.54, latitude (Y) 40.71:

dbadmin=> select gid,ZCTA5CE10,AFFGEOID10,GEOID10,ALAND10,AWATER10 from cb_2018_us_zcta510_500k where ST_Within(STV_GeometryPoint(-73.54,40.71),geom) = 1 limit 5;
gid | ZCTA5CE10 | AFFGEOID10 | GEOID10 | ALAND10 | AWATER10
------+-----------+----------------+---------+----------+----------
1712 | 11554 | 8600000US11554 | 11554 | 16186098 | 61309
(1 row)

ZIP 11554 is East Meadow, NY, which is correct! You could also JOIN a table mapping ZIP codes to place names such as this one: https://simplemaps.com/data/us-zips

Vertica offers full support for standard WGS84 geometry and geography types, geospatial functions, and import and export to WKB, WKT, and shapefiles as shown here.

All of these features are included in the free Vertica Community Edition. Try it out on your data!

References:
Data load: https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/AnalyzingData/Geospatial/GeospatialAnalytics/LoadingSpatialDataFromShapefiles.htm
ST_Within: https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/SQLReferenceManual/Functions/Geospatial/ST_Within.htm
Defining a point to use in ST_Within:
https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/SQLReferenceManual/Functions/Geospatial/STV_GeometryPoint.htm

CREATE LIBRARY pylib as '/path_to/vb64.py' LANGUAGE 'Python';

To register the base64 decode and encode USER functions:

CREATE FUNCTION vb64encode AS LANGUAGE 'Python' NAME 'vb64encode_factory' LIBRARY pylib fenced;
CREATE FUNCTION vb64decode AS LANGUAGE 'Python' NAME 'vb64decode_factory' LIBRARY pylib fenced;

To use these two functions:

dbuser=> select vb64encode('Vertica');
vb64encode
--------------
VmVydGljYQ==
(1 row)
dbuser=> select vb64decode('VmVydGljYQ==');

vb64decode

Vertica
(1 row)

Good news! As of Vertica 9.3.1, the V_MONITOR.UDX_EVENTS system table now also records the events logged during the export!

Example:

dbadmin=> SELECT * FROM test;
 c1
----
  1
  2
  3
(3 rows)

dbadmin=> EXPORT TO PARQUET(DIRECTORY='/home/dbadmin/parq/') AS SELECT * FROM test;
 Rows Exported
---------------
             3
(1 row)

dbadmin=> SELECT created, file, rows FROM v_monitor.udx_events WHERE udx_name = 'ParquetExport';
            created            |                                       file                                       | rows
-------------------------------+----------------------------------------------------------------------------------+------
 2020-01-27 10:05:30.166202-05 | /home/dbadmin/parqhUGvipCI/ad3a94ae-v_test_db_node0002-139833020229376-0.parquet | 3
(1 row)

Helpful Links:
https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/SQLReferenceManual/Statements/EXPORTTOPARQUET.htm
https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/HadoopIntegrationGuide/NativeFormats/ExportMonitor.htm
https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/SQLReferenceManual/SystemTables/MONITOR/UDX_EVENTS.htm

Have Fun!

If you are working with a new database that you didn’t create, you can can determine which Mode it is using by querying the V_CATALOG.STORAGE_LOCATIONS or the V_CATALOG.SHARDS system table.

Examples:

dbadmin=> SELECT CASE COUNT(*)
dbadmin->          WHEN 0 THEN 'Enterprise'
dbadmin->          ELSE 'Eon'
dbadmin->        END AS database_mode
dbadmin->   FROM v_catalog.storage_locations
dbadmin->  WHERE sharing_type = 'COMMUNAL';
 database_mode
---------------
 Eon
(1 row)

SELECT CASE COUNT(*)
         WHEN 0 THEN 'Enterprise'
         ELSE 'Eon'
       END AS database_mode
  FROM v_catalog.shards;
 database_mode
---------------
 Eon
(1 row)

Helpful Links:
https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/Eon/VerticaArchitectureEonVsEnterpriseMode.htm
https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/Eon/ComparingEonandEnterpriseModes.htm
https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/SQLReferenceManual/SystemTables/CATALOG/STORAGE_LOCATIONS.htm
https://www.vertica.com/docs/9.3.x/HTML/Content/Authoring/SQLReferenceManual/SystemTables/CATALOG/SHARDS.htm

Have Fun!