Hive Integration - missing jar files
Hello,
I have installed hadoop and hive
[centos@ip-10-103-2-173 2.3.2.0-2950]$ hadoop version
Hadoop 2.7.1.2.3.2.0-2950
Subversion git@github.com:hortonworks/hadoop.git -r 5cc60e0003e33aa98205f18bccaeaf36cb193c1c
Compiled by jenkins on 2015-09-30T18:08Z
Compiled with protoc 2.5.0
From source with checksum 69a3bf8c667267c2c252a54fbbf23d
This command was run using /usr/hdp/2.3.2.0-2950/hadoop/lib/hadoop-common-2.7.1.2.3.2.0-2950.jar
[centos@ip-10-103-2-173 2.3.2.0-2950]$ hive --version
WARNING: Use "yarn jar" to launch YARN applications.
Hive 1.2.1.2.3.2.0-2950
Subversion git://c66-slave-20176e25-6/grid/0/jenkins/workspace/HDP-2.3-maint-centos6/bigtop/build/hive/rpm/BUILD/hive-1.2.1.2.3.2.0 -r c67988138ca472655a6978f50c7423525b71dc27
Compiled by jenkins on Wed Sep 30 19:07:31 UTC 2015
From source with checksum 6c57f9c021f6f833a9ef075f629c8b03
When i try to run the following command to copy the jars some of the packages are missing .
Should i install all the jars? Please Help!
[centos@ip-10-103-2-173 2.3.2.0-2950]$ sudo /opt/vertica/packages/hcat/tools/hcatUtil --copyJars \
--hadoopHiveHome="$HADOOP_HOME/lib;$HIVE_HOME/lib;/usr/hdp/2.3.2.0-2950/hive-hcatalog/share/hcatalog" \
--hadoopHiveConfPath="$HADOOP_CONF_DIR;$HIVE_CONF_DIR;$WEBHCAT_CONF_DIR" \
--hcatLibPath="/usr/hdp/2.3.2.0-2950"
Looking at [/lib] for Jars
Looking at [/usr/hdp/2.3.2.0-2950/hive/lib] for Jars
Looking at [/usr/hdp/2.3.2.0-2950/hive-hcatalog/share/hcatalog] for Jars
ERROR:: Following required Jars were not found:
[NOTE: Some jars not found below may only be needed for some versions of Hadoop. If your HP Vertica HCatalog connector works correctly without these missing jars, they are not actually required for your Hadoop distribution]
hadoop-hdfs-.jar
hadoop-mapreduce-client-core-.jar
slf4j-log4j12.jar
hadoop-common-.jar
hadoop-archives-.jar
hadoop-mapreduce-client-common-.jar
commons-configuration-.jar
protobuf-java-2..jar
hive-webhcat-java-client-.jar
hive-shims-common-secure-.jar
slf4j-api.jar
hadoop-annotations-.jar
hadoop-yarn-api-.jar
hadoop-auth-.jar
webhcat-.jar
WARNING:: Following optional Serde and Codec Jars were not found:
hadoop-lzo.jar
WARNING:: Following HDP required Jars were not found:
commons-lang3-.jar
kryo-.jar
WARNING:: Following MapR required Jars were not found:
hadoop-yarn-common-.jar
maprfs-.jar
WARNING: Following native libraries are not found:
libgplcompression.so
libsnappy.so
libhadoop*.so
Summary
Found 21 out of 36 required JARs
Found 4 out of 5 optional Serde and Codec JARs
Found 0 out of 2 HDP required JARs
Found 1 out of 3 MapR required JARs
Found 0 out of 3 optional native libraries, 0 files in total
Start copying JARs and native libraries to hcatLibPath [/usr/hdp/2.3.2.0-2950]..........................
Done
Looking at [] for configuration files
WARNING: hadoopHiveConfPath [] does not exist
Looking at [] for configuration files
WARNING: hadoopHiveConfPath [] does not exist
Looking at [] for configuration files
WARNING: hadoopHiveConfPath [] does not exist
ERROR: Could not find required Hive configuration file [hive-site.xml]
WARNING: Could not find Hadoop configuration file [core-site.xml]. If you are using Kerberos authentication, please make sure this file is copied
WARNING: Could not find Yarn configuration file [yarn-site.xml]. If you are using Kerberos authentication, please make sure this file is copied
WARNING: Could not find WebHCat configration file [webhcat-site.xml]. You must specify WEBSERVICE_HOSTNAME and WEBSERVICE_PORT when CREATE HCATALOG SCHEMA
WARNING: Could not find HDFS configration file [hdfs-site.xml]. You must specify WEBHDFS_PORT when CREATE HCATALOG SCHEMA
Start copying configuration files to hcatLibPath [/usr/hdp/2.3.2.0-2950]
Done
Answers
Not sure if you got an answer for this yet. But yes, these are required jars. My guess based on your output is that $HADOOP_HOME and $HIVE_HOME (as well as *_CONFIG_DIR) are not correctly configured in your environment. I would suggest using "/usr/hdp/2.3.2.0-2950/hadoop/lib" and "/usr/hdp/2.3.2.0-2950/hive/lib" in --hadoopHiveHome instead.