Hive将数据导入到ElasticSearch Hive将数据导入到ElasticSearch0.环境准备ElasticSearchHive1.安装插件1.下载对应版本https://www.elastic.co/cn/downloads/hadoophive支持jsonhttp://www.congiu.net/hive-json-serde/1.3.8/hdp23/json-serde-1.3.8-jar-with-dependencies.jarhttp://www.congiu.net/hive-json-serde/1.3.8/hdp23/json-udf-1.3.8-jar-with-dependencies.jarwget -r https://artifacts.elastic.co/downloads/elasticsearch-hadoop/elasticsearch-hadoop-7.17.1.zip2.安装unzip工具yum install -y unzip3.解压文件unzip elasticsearch-hadoop-7.17.1.zip4.找到jar文件cd elasticsearch-hadoop-7.17.1/dist/ ll #只能添加这个，hadoop.jar不用添加 elasticsearch-hadoop-hive-7.17.1.jar #/root/elasticsearch/elasticsearch-hadoop-7.17.1/dist/elasticsearch-hadoop-hive-7.17.1.jar5.进入hive添加jar(add jar 仅对当前窗口有效)hive #add jar 包 仅对当前窗口有效，下次使用需重新添加 hive> add jar /root/elasticsearch/elasticsearch-hadoop-7.17.1/dist/elasticsearch-hadoop-hive-7.17.1.jar; #Added [/root/elasticsearch/elasticsearch-hadoop-7.17.1/dist/elasticsearch-hadoop-hive-7.17.1.jar] to class path #Added resources: [/root/elasticsearch/elasticsearch-hadoop-7.17.1/dist/elasticsearch-hadoop-hive-7.17.1.jar]2.hive表映射1.创建Hive辅助表hive> create database 5ewb; hive> use 5ewb; hive> create table `inwb` ( `phone` bigint, `uid` bigint ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'; OK Time taken: 0.69 seconds2.导入数据hive> use 5ewb; OK hive> load data local inpath '/root/shegongku/wb5e.txt' into table `inwb`; Loading data to table 5ewb.user OK Time taken: 97.234 seconds hive> select * from `inwb` limit 10; OK NULL NULL 15890981333 5350176154 15944850489 6057766172 17073799004 6547208199 18392710332 3754369810 18047430444 6444293239 13762520188 3866009977 18408812716 6134347857 18477461107 6031338428 13647595899 6796854079 #删除数据中的空行 hive> insert overwrite table `inwb` select * from `inwb` where phone is not null;3.创建hive映射表hive hive> use 5ewb; OK hive> CREATE TABLE `outwb` ( `phone` bigint, `uid` bigint ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' TBLPROPERTIES('es.resource' = 'wb/_doc', 'es.index.auto.create' = 'true', 'es.nodes' = 'http://10.107.116.11', 'es.port'='9200', 'es.http.timeout'='120m', 'es.nodes.wan.only'='true'); OK Time taken: 0.252 seconds3.导入数据hive> use 5ewb; OK Time taken: 0.176 seconds hive> insert overwrite table `outwb` select * from `inwb`;

Hadoop配置Hive Hadoop配置Hive{cloud title="" type="default" url="http://pan.000081.xyz/%E5%8D%9A%E5%AE%A2/%E5%A4%A7%E6%95%B0%E6%8D%AE/Hive" password=""/}0.环境前置Hadoop分布式部署完毕准备文件mysql-5.7.26-1.el7.x86_64.rpm-bundle.tarapache-hive-2.3.4-bin.tar.gzmysql-connector-java-5.1.46.jar1.安装Mysql1.检查是否存在与mysql冲突的包rpm -qa | grep mariadb #输出 mariadb-libs-5.5.56-2.el7.x86_642.卸载冲突包rpm -ev --nodeps mariadb-libs-5.5.56-2.el7.x86_643.解压Mysql安装包mkdir /mysql tar -xf mysql-5.7.26-1.el7.x86_64.rpm-bundle.tar -C /mysql/ cd /mysql/4.rpm按顺序安装Mysqlrpm -ivh mysql-community-common-5.7.26-1.el7.x86_64.rpm rpm -ivh mysql-community-libs-5.7.26-1.el7.x86_64.rpm rpm -ivh mysql-community-client-5.7.26-1.el7.x86_64.rpm rpm -ivh mysql-community-server-5.7.26-1.el7.x86_64.rpm5.启动mysqlservice mysqld start6.查看初始密码grep "password" /var/log/mysqld.log #输出 2022-03-24T12:04:39.603157Z 1 [Note] A temporary password is generated for root@localhost: 2yEFsa!sd2S77.进入Mysql并且修改密码mysql -uroot -p2yEFsa!sd2S7 #修改密码，因为我修改的是简单密码，所以修改相关参数 #修改validate_password_policy参数的值 mysql> set global validate_password_policy=0; Query OK, 0 rows affected (0.00 sec) #validate_password_length (密码长度) 参数默认为8，修改为1 mysql> set global validate_password_length=1; Query OK, 0 rows affected (0.00 sec) #修改密码 mysql> alter user 'root'@'localhost' identified by '000000'; Query OK, 0 rows affected (0.00 sec)8.开放远程登录mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '000000'; Query OK, 0 rows affected, 1 warning (0.00 sec) mysql> flush privileges; Query OK, 0 rows affected (0.00 sec)2.安装Hive1.解压Hivetar -xzf apache-hive-2.3.4-bin.tar.gz -C /2.添加环境变量vim /etc/profile #追加 export HIVE_HOME=/apache-hive-2.3.4-bin export PATH=$HIVE_HOME/bin:$PATH #刷新环境变量 source /etc/profile3.配置1.新建mysql数据库mysql -uroot -p000000 -e "create database hive_db";2.配置hive-site.xmlcd /apache-hive-2.3.4-bin/conf cp -r hive-default.xml.template hive-site.xml vim hive-site.xml #配置文件 <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://master:3306/hive_db?createDatabaseIfNotExist=true</value> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>000000</value> </property> <!--mysql 驱动--> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> ！如果改成mysql-connector-java8一定要把 com.mysql.jdbc.Driver改成com.mysql.cj.jdbc.Driver </property> <property> <name>hive.downloaded.resources.dir</name> <value>/apache-hive-2.3.4-bin/tmp</value> </property> <property> <name>hive.exec.local.scratchdir</name> <value>/apache-hive-2.3.4-bin/tmp/${hive.session.id}_resources</value> </property> <property> <name>hive.server2.enable.doAs</name> <value>false</value> </property> <property> <name>hive.metastore.schema.verification</name> <value>false</value> </property>3.将mysql驱动包放入hive的lib下pwd /apache-hive-2.3.4-bin/lib ls | grep mysql-connector-java-5.1.46.jar mysql-connector-java-5.1.46.jar4.启动Hadoopstart-all.sh5.配置hive-env.shcd /apache-hive-2.3.4-bin/conf cp -r hive-env.sh.template hive-env.sh vim hive-env.sh #追加 ---------------------------- export JAVA_HOME=/jdk1.8.0_191 export HADOOP_HOME=/hadoop-2.7.7 export HIVE_CONF_DIR=/apache-hive-2.3.4-bin/conf export HIVE_AUX_JARS_PATH=/apache-hive-2.3.4-bin/lib4.初始化并启动1.删除文件rm -rf /hadoop-2.7.7/share/hadoop/yarn/lib/jline-0.9.94.jar2.初始化Hiveschematool -initSchema -dbType mysql #出现以下即为成功 schemaTool completed