AI摘要 本文介绍了将Hive数据导入ElasticSearch的流程。首先需下载并安装elasticsearch-hadoop插件，并在Hive中添加对应的jar包。接着创建Hive辅助表导入原始数据，最后通过创建映射表（指定EsStorageHandler及相关配置参数），利用INSERT语句将数据高效同步至ElasticSearch中。 此内容根据文章生成，仅用于文章内容的解释与总结 Hive将数据导入到ElasticSearch0.环境准备ElasticSearchHive1.安装插件1.下载对应版本https://www.elastic.co/cn/downloads/hadoophive支持jsonhttp://www.congiu.net/hive-json-serde/1.3.8/hdp23/json-serde-1.3.8-jar-with-dependencies.jarhttp://www.congiu.net/hive-json-serde/1.3.8/hdp23/json-udf-1.3.8-jar-with-dependencies.jarwget -r https://artifacts.elastic.co/downloads/elasticsearch-hadoop/elasticsearch-hadoop-7.17.1.zip2.安装unzip工具yum install -y unzip3.解压文件unzip elasticsearch-hadoop-7.17.1.zip4.找到jar文件cd elasticsearch-hadoop-7.17.1/dist/ ll #只能添加这个，hadoop.jar不用添加 elasticsearch-hadoop-hive-7.17.1.jar #/root/elasticsearch/elasticsearch-hadoop-7.17.1/dist/elasticsearch-hadoop-hive-7.17.1.jar5.进入hive添加jar(add jar 仅对当前窗口有效)hive #add jar 包 仅对当前窗口有效，下次使用需重新添加 hive> add jar /root/elasticsearch/elasticsearch-hadoop-7.17.1/dist/elasticsearch-hadoop-hive-7.17.1.jar; #Added [/root/elasticsearch/elasticsearch-hadoop-7.17.1/dist/elasticsearch-hadoop-hive-7.17.1.jar] to class path #Added resources: [/root/elasticsearch/elasticsearch-hadoop-7.17.1/dist/elasticsearch-hadoop-hive-7.17.1.jar]2.hive表映射1.创建Hive辅助表hive> create database 5ewb; hive> use 5ewb; hive> create table `inwb` ( `phone` bigint, `uid` bigint ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n'; OK Time taken: 0.69 seconds2.导入数据hive> use 5ewb; OK hive> load data local inpath '/root/shegongku/wb5e.txt' into table `inwb`; Loading data to table 5ewb.user OK Time taken: 97.234 seconds hive> select * from `inwb` limit 10; OK NULL NULL 15890981333 5350176154 15944850489 6057766172 17073799004 6547208199 18392710332 3754369810 18047430444 6444293239 13762520188 3866009977 18408812716 6134347857 18477461107 6031338428 13647595899 6796854079 #删除数据中的空行 hive> insert overwrite table `inwb` select * from `inwb` where phone is not null;3.创建hive映射表hive hive> use 5ewb; OK hive> CREATE TABLE `outwb` ( `phone` bigint, `uid` bigint ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' TBLPROPERTIES('es.resource' = 'wb/_doc', 'es.index.auto.create' = 'true', 'es.nodes' = 'http://10.107.116.11', 'es.port'='9200', 'es.http.timeout'='120m', 'es.nodes.wan.only'='true'); OK Time taken: 0.252 seconds3.导入数据hive> use 5ewb; OK Time taken: 0.176 seconds hive> insert overwrite table `outwb` select * from `inwb`;
AI摘要 本文介绍了在CentOS 7环境下为ElasticSearch 7安装IK分词器插件的步骤。主要包括安装解压工具、解压插件包、将其移动至插件目录并同步至集群各节点，最后通过重启ElasticSearch服务使插件生效。 此内容根据文章生成，仅用于文章内容的解释与总结 ElasticSearch安装插件（ik分词器）0.环境准备centos7ElasticSearch71.安装插件1.安装zip解压工具yum install -y unzip2.解压文件mkdir ik unzip elasticsearch-analysis-ik-7.17.1.zip -d ik/3.移动到ElasticSearch插件目录下mv ik /elasticsearch-7.17.1/plugins/4.scp到集群的每台节点插件目录下cd /elasticsearch-7.17.1/ scp -r plugins/ik slave1:/elasticsearch-7.17.1/plugins/ scp -r plugins/ik slave2:/elasticsearch-7.17.1/plugins/2.每台机重启ElasticSearch#切换es用户 su es #查看ElasticSearch进程 jps 1688 Elasticsearch #强杀进程 kill -9 1688 #执行命令重启 elasticsearch -d
AI摘要 本文介绍了在已安装ElasticSearch的环境中部署head插件的步骤：首先安装并配置Node.js环境，接着通过Git拉取head项目源码，最后通过npm安装依赖并启动服务，实现通过浏览器访问9100端口对ElasticSearch进行管理。 此内容根据文章生成，仅用于文章内容的解释与总结 ElasticSearch安装head0.环境准备ElasticSearch安装完毕准备文件：node.js安装包（node-v16.14.2-linux-x64.tar.xz）head项目文件1.安装node.js1.解压压缩包tar -xvf node-v16.14.2-linux-x64.tar.xz -C /2.配置环境变量cd /node-v16.14.2-linux-x64/ pwd vim /etc/profile #####追加内容##### export NODE_HOME=/node-v16.14.2-linux-x64 export PATH=$PATH:$NODE_HOME/bin ################# #刷新环境变量 source /etc/profile3.验证安装成功node -v2.安装git并拉取head项目文件1.yum安装gityum install -y git2.拉取项目文件cd /root/ mkdir head cd head/ git clone https://github.com/mobz/elasticsearch-head.git #查看项目是否克隆完成 ls elasticsearch-head3.安装并且启动head1.更换node.js镜像源npm config set registry https://registry.npm.taobao.org #检测是否成功更换 npm config get registry2.安装cd /root/head/elasticsearch-head/ npm install3.启动cd /root/head/elasticsearch-head/ nohup npm run start &4.打开地址并连接http://本机ip:9100连接地址：ElasticSearch连接地址ip:9200
AI摘要 本文介绍了在CentOS 7环境下部署ELK（Elasticsearch、Logstash、Kibana）集群的详细步骤。内容涵盖了Elasticsearch集群的节点配置、系统参数优化、用户权限设置，以及Kibana和Logstash的安装、配置文件编写及后台启动方法，并提供了常见报错的解决方案。 此内容根据文章生成，仅用于文章内容的解释与总结 Centos7安装ELK集群0.准备环境系统：centos7Hadoop分布式部署完毕hive部署完毕准备文件：elasticsearch-7.17.1-linux-x86_64.tar.gzkibana-7.17.1-linux-x86_64.tar.gzlogstash-7.17.1-linux-x86_64.tar.gz1.所有节点安装ElasticSearch1.解压压缩包tar -xzf elasticsearch-7.17.1-linux-x86_64.tar.gz -C /2.添加ElasticSearch的环境变量（每台机都执行）vim /etc/profile #末尾加入环境变量 export ELASTICSEARCH_HOME=/elasticsearch-7.17.1 export PATH=$PATH:$ELASTICSEARCH_HOME/bin #刷新环境变量 source /etc/profile3.修改ElasticSearch环境使用自己的jdkcd /elasticsearch-7.17.1 vim bin/elasticsearch-env #第二行插入java环境变量 JAVA_HOME="/elasticsearch-7.17.1/jdk"4.修改垃圾回收器配置参数vim config/jvm.options #在大约52行 #####修改前###### -XX:+UseConcMarkSweepGC ################ #####修改后###### -XX:+UseG1GC ################5.修改主配置文件设置集群vim config/elasticsearch.yml #####yaml文件，注意格式##### cluster.name: es node.name: node-x #节点名，各台机器不同 (1-3) network.host: 0.0.0.0 http.port: 9200 discovery.seed_hosts: ["master", "slave1", "slave2"] cluster.initial_master_nodes: ["node-1", "node-2", "node-3"] #添加跨域第三方插件可以请求es http.cors.enabled: true http.cors.allow-origin: "*"6.修改普通用户可创建的最大线程数(每台机都执行)vim /etc/security/limits.conf #末尾追加 es soft nofile 65535 es hard nofile 65535 es soft nproc 4096 es hard nproc 4096 # End of file7.设置最大虚拟内存区域(每台机都执行)vim /etc/sysctl.conf #末尾追加 vm.max_map_count = 262144 ulimit -n 65536 #手动执行重新加载虚拟内存 sysctl -p8.批量拷贝cd / scp -r elasticsearch-7.17.1 root@slave1:/ scp -r elasticsearch-7.17.1 root@slave2:/ source /etc/profile9.修改另外两台机的主配置文件######slave1###### cd /elasticsearch-7.17.1 vim config/elasticsearch.yml node.name: node-2 #节点名，各台机器不同 (1-3) ################## ######slave2###### cd /elasticsearch-7.17.1 vim config/elasticsearch.yml node.name: node-3 #节点名，各台机器不同 (1-3) ##################2.创建新用户设置密码修改属组因为ElasticSearch不支持root用户启动，所以创建一个新用户三台机同时执行#新建用户 useradd es #设置密码000000 passwd es #修改文件属组 cd / chown -Rf es:es /elasticsearch-7.17.1/3.切换用户并启动三台机同时执行#切换用户 su es #启动ElasticSearch elasticsearch4.验证启动状态http://任意一台机IP:9200/_cluster/health?pretty{ "cluster_name" : "es", "status" : "green", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 3, "active_primary_shards" : 3, "active_shards" : 6, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 }status:red：集群失败yellow：基本分片可用，备份不可用green：集群健康，所有分片和备份都可用5.前台停止es，通过后台启动ctrl+c强制终止后台启动命令elasticsearch -d6.安装kibana单节点：1.解压文件tar -xzf kibana-7.17.1-linux-x86_64.tar.gz -C /2.修改配置文件cd /kibana-7.17.1-linux-x86_64 vim config/kibana.yml ######修改内容###### server.host: "0.0.0.0" server.name: "master" #主机名 elasticsearch.hosts: ["http://10.107.116.10:9200"] #es地址 kibana.index: ".kibana" i18n.locale: "zh-CN" #中文 #elasticsearch.username: "admin" #账号 #elasticsearch.password: "000000" #密码 ###################3.创建kibana用户并且修改属组创建用户useradd kibana passwd kibana #设置密码000000修改属组cd / chown -Rf kibana:kibana /kibana-7.17.1-linux-x86_64/4.后台启动kibanasu kibana cd /kibana-7.17.1-linux-x86_64/ nohup bin/kibana >> /dev/null 2>&1 & exit5.验证kibana启动访问地址http://10.107.116.10:5601/7.安装logstash单节点：1.解压文件tar -xzf logstash-7.17.1-linux-x86_64.tar.gz -C /2.准备patterns新建patterns文件夹cd /logstash-7.17.1/ mkdir patterns创建java文件vim patterns/java ############### # user-center MYAPPNAME ([0-9a-zA-Z_-]*) # RMI TCP Connection(2)-127.0.0.1 MYTHREADNAME ([0-9a-zA-Z._-]|\(|\)|\s)* ###############3.修改配置文件创建配置文件 logstash.conf在使用如下配置时，需修改内容项：filter中的 两个 patterns_dirElasticSearch有账号密码就添加账号密码vim config/logstash.conf ########################### input { beats { port => 5044 } } filter { if [type] == "syslog" { grok { match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" } add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] } syslog_pri { } date { match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] } } if [fields][docType] == "sys-log" { grok { patterns_dir => ["/logstash-7.17.1/patterns"] match => { "message" => "\[%{NOTSPACE:appName}:%{IP:serverIp}:%{NOTSPACE:serverPort}\] %{TIMESTAMP_ISO8601:logTime} %{LOGLEVEL:logLevel} %{WORD:pid} \[%{MYAPPNAME:traceId}\] \[%{MYTHREADNAME:threadName}\] %{NOTSPACE:classname} %{GREEDYDATA:message}" } overwrite => ["message"] } date { match => ["logTime","yyyy-MM-dd HH:mm:ss.SSS Z"] } date { match => ["logTime","yyyy-MM-dd HH:mm:ss.SSS"] target => "timestamp" locale => "en" timezone => "+08:00" } mutate { remove_field => "logTime" remove_field => "@version" remove_field => "host" remove_field => "offset" } } if [fields][docType] == "point-log" { grok { patterns_dir => ["/logstash-7.17.1/patterns"] match => { "message" => "%{TIMESTAMP_ISO8601:logTime}\|%{MYAPPNAME:appName}\|%{WORD:resouceid}\|%{MYAPPNAME:type}\|%{GREEDYDATA:object}" } } kv { source => "object" field_split => "&" value_split => "=" } date { match => ["logTime","yyyy-MM-dd HH:mm:ss.SSS Z"] } date { match => ["logTime","yyyy-MM-dd HH:mm:ss.SSS"] target => "timestamp" locale => "en" timezone => "+08:00" } mutate { remove_field => "message" remove_field => "logTime" remove_field => "@version" remove_field => "host" remove_field => "offset" } } if [fields][docType] == "mysqlslowlogs" { grok { match => [ "message", "^#\s+User@Host:\s+%{USER:user}\[[^\]]+\]\s+@\s+(?:(?<clienthost>\S*) )?\[(?:%{IP:clientip})?\]\s+Id:\s+%{NUMBER:id}\n# Query_time: %{NUMBER:query_time}\s+Lock_time: %{NUMBER:lock_time}\s+Rows_sent: %{NUMBER:rows_sent}\s+Rows_examined: %{NUMBER:rows_examined}\nuse\s(?<dbname>\w+);\nSET\s+timestamp=%{NUMBER:timestamp_mysql};\n(?<query_str>[\s\S]*)", "message", "^#\s+User@Host:\s+%{USER:user}\[[^\]]+\]\s+@\s+(?:(?<clienthost>\S*) )?\[(?:%{IP:clientip})?\]\s+Id:\s+%{NUMBER:id}\n# Query_time: %{NUMBER:query_time}\s+Lock_time: %{NUMBER:lock_time}\s+Rows_sent: %{NUMBER:rows_sent}\s+Rows_examined: %{NUMBER:rows_examined}\nSET\s+timestamp=%{NUMBER:timestamp_mysql};\n(?<query_str>[\s\S]*)", "message", "^#\s+User@Host:\s+%{USER:user}\[[^\]]+\]\s+@\s+(?:(?<clienthost>\S*) )?\[(?:%{IP:clientip})?\]\n# Query_time: %{NUMBER:query_time}\s+Lock_time: %{NUMBER:lock_time}\s+Rows_sent: %{NUMBER:rows_sent}\s+Rows_examined: %{NUMBER:rows_examined}\nuse\s(?<dbname>\w+);\nSET\s+timestamp=%{NUMBER:timestamp_mysql};\n(?<query_str>[\s\S]*)", "message", "^#\s+User@Host:\s+%{USER:user}\[[^\]]+\]\s+@\s+(?:(?<clienthost>\S*) )?\[(?:%{IP:clientip})?\]\n# Query_time: %{NUMBER:query_time}\s+Lock_time: %{NUMBER:lock_time}\s+Rows_sent: %{NUMBER:rows_sent}\s+Rows_examined: %{NUMBER:rows_examined}\nSET\s+timestamp=%{NUMBER:timestamp_mysql};\n(?<query_str>[\s\S]*)" ] } date { match => ["timestamp_mysql","yyyy-MM-dd HH:mm:ss.SSS","UNIX"] } date { match => ["timestamp_mysql","yyyy-MM-dd HH:mm:ss.SSS","UNIX"] target => "timestamp" } mutate { convert => ["query_time", "float"] convert => ["lock_time", "float"] convert => ["rows_sent", "integer"] convert => ["rows_examined", "integer"] remove_field => "message" remove_field => "timestamp_mysql" remove_field => "@version" } } } output { if [fields][docType] == "sys-log" { elasticsearch { hosts => ["http://10.107.116.10:9200"] index => "sys-log-%{+YYYY.MM.dd}" #user => "elastic" #password => "000000" } } if [fields][docType] == "point-log" { elasticsearch { hosts => ["http://10.107.116.11:9200"] index => "point-log-%{+YYYY.MM.dd}" routing => "%{type}" #user => "elastic" #password => "000000" } } if [fields][docType] == "mysqlslowlogs" { elasticsearch { hosts => ["http://10.107.116.12:9200"] index => "mysql-slowlog-%{+YYYY.MM.dd}" #user => "elastic" #password => "000000" } } } ###########################修改logstash配置vim config/logstash.yml ################ api.http.host: 0.0.0.0 ################4.后台启动logstashcd /logstash-7.17.1/ nohup bin/logstash -f config/logstash.conf &5.验证启动ps -ef | grep logstash #查看相关进程 cat config/logstash.conf6.报错解决如果启动出现下面错误[2022-03-31T19:22:29,834][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit org.jruby.exceptions.SystemExit: (SystemExit) exit at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?] at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?] at logstash_minus_7_dot_17_dot_1.lib.bootstrap.environment. (/logstash-7.17.1/lib/bootstrap/environment.rb:94) ~[?:?]解决办法rm -rf /logstash-7.17.1/data/.lock重新启动