Centos7安装ELK集群 Centos7安装ELK集群0.准备环境系统：centos7Hadoop分布式部署完毕hive部署完毕准备文件：elasticsearch-7.17.1-linux-x86_64.tar.gzkibana-7.17.1-linux-x86_64.tar.gzlogstash-7.17.1-linux-x86_64.tar.gz1.所有节点安装ElasticSearch1.解压压缩包tar -xzf elasticsearch-7.17.1-linux-x86_64.tar.gz -C /2.添加ElasticSearch的环境变量（每台机都执行）vim /etc/profile #末尾加入环境变量 export ELASTICSEARCH_HOME=/elasticsearch-7.17.1 export PATH=$PATH:$ELASTICSEARCH_HOME/bin #刷新环境变量 source /etc/profile3.修改ElasticSearch环境使用自己的jdkcd /elasticsearch-7.17.1 vim bin/elasticsearch-env #第二行插入java环境变量 JAVA_HOME="/elasticsearch-7.17.1/jdk"4.修改垃圾回收器配置参数vim config/jvm.options #在大约52行 #####修改前###### -XX:+UseConcMarkSweepGC ################ #####修改后###### -XX:+UseG1GC ################5.修改主配置文件设置集群vim config/elasticsearch.yml #####yaml文件，注意格式##### cluster.name: es node.name: node-x #节点名，各台机器不同 (1-3) network.host: 0.0.0.0 http.port: 9200 discovery.seed_hosts: ["master", "slave1", "slave2"] cluster.initial_master_nodes: ["node-1", "node-2", "node-3"] #添加跨域第三方插件可以请求es http.cors.enabled: true http.cors.allow-origin: "*"6.修改普通用户可创建的最大线程数(每台机都执行)vim /etc/security/limits.conf #末尾追加 es soft nofile 65535 es hard nofile 65535 es soft nproc 4096 es hard nproc 4096 # End of file7.设置最大虚拟内存区域(每台机都执行)vim /etc/sysctl.conf #末尾追加 vm.max_map_count = 262144 ulimit -n 65536 #手动执行重新加载虚拟内存 sysctl -p8.批量拷贝cd / scp -r elasticsearch-7.17.1 root@slave1:/ scp -r elasticsearch-7.17.1 root@slave2:/ source /etc/profile9.修改另外两台机的主配置文件######slave1###### cd /elasticsearch-7.17.1 vim config/elasticsearch.yml node.name: node-2 #节点名，各台机器不同 (1-3) ################## ######slave2###### cd /elasticsearch-7.17.1 vim config/elasticsearch.yml node.name: node-3 #节点名，各台机器不同 (1-3) ##################2.创建新用户设置密码修改属组因为ElasticSearch不支持root用户启动，所以创建一个新用户三台机同时执行#新建用户 useradd es #设置密码000000 passwd es #修改文件属组 cd / chown -Rf es:es /elasticsearch-7.17.1/3.切换用户并启动三台机同时执行#切换用户 su es #启动ElasticSearch elasticsearch4.验证启动状态http://任意一台机IP:9200/_cluster/health?pretty{ "cluster_name" : "es", "status" : "green", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 3, "active_primary_shards" : 3, "active_shards" : 6, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 }status:red：集群失败yellow：基本分片可用，备份不可用green：集群健康，所有分片和备份都可用5.前台停止es，通过后台启动ctrl+c强制终止后台启动命令elasticsearch -d6.安装kibana单节点：1.解压文件tar -xzf kibana-7.17.1-linux-x86_64.tar.gz -C /2.修改配置文件cd /kibana-7.17.1-linux-x86_64 vim config/kibana.yml ######修改内容###### server.host: "0.0.0.0" server.name: "master" #主机名 elasticsearch.hosts: ["http://10.107.116.10:9200"] #es地址 kibana.index: ".kibana" i18n.locale: "zh-CN" #中文 #elasticsearch.username: "admin" #账号 #elasticsearch.password: "000000" #密码 ###################3.创建kibana用户并且修改属组创建用户useradd kibana passwd kibana #设置密码000000修改属组cd / chown -Rf kibana:kibana /kibana-7.17.1-linux-x86_64/4.后台启动kibanasu kibana cd /kibana-7.17.1-linux-x86_64/ nohup bin/kibana >> /dev/null 2>&1 & exit5.验证kibana启动访问地址http://10.107.116.10:5601/7.安装logstash单节点：1.解压文件tar -xzf logstash-7.17.1-linux-x86_64.tar.gz -C /2.准备patterns新建patterns文件夹cd /logstash-7.17.1/ mkdir patterns创建java文件vim patterns/java ############### # user-center MYAPPNAME ([0-9a-zA-Z_-]*) # RMI TCP Connection(2)-127.0.0.1 MYTHREADNAME ([0-9a-zA-Z._-]|\(|\)|\s)* ###############3.修改配置文件创建配置文件 logstash.conf在使用如下配置时，需修改内容项：filter中的 两个 patterns_dirElasticSearch有账号密码就添加账号密码vim config/logstash.conf ########################### input { beats { port => 5044 } } filter { if [type] == "syslog" { grok { match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" } add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] } syslog_pri { } date { match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] } } if [fields][docType] == "sys-log" { grok { patterns_dir => ["/logstash-7.17.1/patterns"] match => { "message" => "\[%{NOTSPACE:appName}:%{IP:serverIp}:%{NOTSPACE:serverPort}\] %{TIMESTAMP_ISO8601:logTime} %{LOGLEVEL:logLevel} %{WORD:pid} \[%{MYAPPNAME:traceId}\] \[%{MYTHREADNAME:threadName}\] %{NOTSPACE:classname} %{GREEDYDATA:message}" } overwrite => ["message"] } date { match => ["logTime","yyyy-MM-dd HH:mm:ss.SSS Z"] } date { match => ["logTime","yyyy-MM-dd HH:mm:ss.SSS"] target => "timestamp" locale => "en" timezone => "+08:00" } mutate { remove_field => "logTime" remove_field => "@version" remove_field => "host" remove_field => "offset" } } if [fields][docType] == "point-log" { grok { patterns_dir => ["/logstash-7.17.1/patterns"] match => { "message" => "%{TIMESTAMP_ISO8601:logTime}\|%{MYAPPNAME:appName}\|%{WORD:resouceid}\|%{MYAPPNAME:type}\|%{GREEDYDATA:object}" } } kv { source => "object" field_split => "&" value_split => "=" } date { match => ["logTime","yyyy-MM-dd HH:mm:ss.SSS Z"] } date { match => ["logTime","yyyy-MM-dd HH:mm:ss.SSS"] target => "timestamp" locale => "en" timezone => "+08:00" } mutate { remove_field => "message" remove_field => "logTime" remove_field => "@version" remove_field => "host" remove_field => "offset" } } if [fields][docType] == "mysqlslowlogs" { grok { match => [ "message", "^#\s+User@Host:\s+%{USER:user}\[[^\]]+\]\s+@\s+(?:(?<clienthost>\S*) )?\[(?:%{IP:clientip})?\]\s+Id:\s+%{NUMBER:id}\n# Query_time: %{NUMBER:query_time}\s+Lock_time: %{NUMBER:lock_time}\s+Rows_sent: %{NUMBER:rows_sent}\s+Rows_examined: %{NUMBER:rows_examined}\nuse\s(?<dbname>\w+);\nSET\s+timestamp=%{NUMBER:timestamp_mysql};\n(?<query_str>[\s\S]*)", "message", "^#\s+User@Host:\s+%{USER:user}\[[^\]]+\]\s+@\s+(?:(?<clienthost>\S*) )?\[(?:%{IP:clientip})?\]\s+Id:\s+%{NUMBER:id}\n# Query_time: %{NUMBER:query_time}\s+Lock_time: %{NUMBER:lock_time}\s+Rows_sent: %{NUMBER:rows_sent}\s+Rows_examined: %{NUMBER:rows_examined}\nSET\s+timestamp=%{NUMBER:timestamp_mysql};\n(?<query_str>[\s\S]*)", "message", "^#\s+User@Host:\s+%{USER:user}\[[^\]]+\]\s+@\s+(?:(?<clienthost>\S*) )?\[(?:%{IP:clientip})?\]\n# Query_time: %{NUMBER:query_time}\s+Lock_time: %{NUMBER:lock_time}\s+Rows_sent: %{NUMBER:rows_sent}\s+Rows_examined: %{NUMBER:rows_examined}\nuse\s(?<dbname>\w+);\nSET\s+timestamp=%{NUMBER:timestamp_mysql};\n(?<query_str>[\s\S]*)", "message", "^#\s+User@Host:\s+%{USER:user}\[[^\]]+\]\s+@\s+(?:(?<clienthost>\S*) )?\[(?:%{IP:clientip})?\]\n# Query_time: %{NUMBER:query_time}\s+Lock_time: %{NUMBER:lock_time}\s+Rows_sent: %{NUMBER:rows_sent}\s+Rows_examined: %{NUMBER:rows_examined}\nSET\s+timestamp=%{NUMBER:timestamp_mysql};\n(?<query_str>[\s\S]*)" ] } date { match => ["timestamp_mysql","yyyy-MM-dd HH:mm:ss.SSS","UNIX"] } date { match => ["timestamp_mysql","yyyy-MM-dd HH:mm:ss.SSS","UNIX"] target => "timestamp" } mutate { convert => ["query_time", "float"] convert => ["lock_time", "float"] convert => ["rows_sent", "integer"] convert => ["rows_examined", "integer"] remove_field => "message" remove_field => "timestamp_mysql" remove_field => "@version" } } } output { if [fields][docType] == "sys-log" { elasticsearch { hosts => ["http://10.107.116.10:9200"] index => "sys-log-%{+YYYY.MM.dd}" #user => "elastic" #password => "000000" } } if [fields][docType] == "point-log" { elasticsearch { hosts => ["http://10.107.116.11:9200"] index => "point-log-%{+YYYY.MM.dd}" routing => "%{type}" #user => "elastic" #password => "000000" } } if [fields][docType] == "mysqlslowlogs" { elasticsearch { hosts => ["http://10.107.116.12:9200"] index => "mysql-slowlog-%{+YYYY.MM.dd}" #user => "elastic" #password => "000000" } } } ###########################修改logstash配置vim config/logstash.yml ################ api.http.host: 0.0.0.0 ################4.后台启动logstashcd /logstash-7.17.1/ nohup bin/logstash -f config/logstash.conf &5.验证启动ps -ef | grep logstash #查看相关进程 cat config/logstash.conf6.报错解决如果启动出现下面错误[2022-03-31T19:22:29,834][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit org.jruby.exceptions.SystemExit: (SystemExit) exit at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?] at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?] at logstash_minus_7_dot_17_dot_1.lib.bootstrap.environment.<main>(/logstash-7.17.1/lib/bootstrap/environment.rb:94) ~[?:?]解决办法rm -rf /logstash-7.17.1/data/.lock重新启动