Hadoop分布式搭建

0.环境

• 系统：centos7 64位
• 处理器：8核
• 内存：8G
• 硬盘：200G

数量：3台

1.基础环境配置

1.配置静态ip

vi /etc/sysconfig/network-scripts/ifcfg-xxx

BOOTPROTO=static
ONBOOT="yes
IPADDR=10.107.116.x
NETMASK=255.255.255.0
GATEWAY=10.107.116.254
DNS1=114.114.114.114

2.关闭防火墙和selinux

systemctl stop firewalld
systemctl disable firewalld
setenforce 0
vi /etc/selinux/config
#修改为
SELINUX=disabled

3.修改主机名

hostnamectl set-hostname 主机名
bash

4.配置hosts文件

vi /etc/hosts
加入
10.107.116.10 master
10.107.116.11 slave1
10.107.116.12 slave2

5.配置ssh免密登入

#生成密钥
ssh-keygen

#三台机互相执行配置免密
ssh-copy-id -i 主机

6.配置时间同步

yum install -y chrony
vi /etc/chrony.conf

master:
------------------------------------
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
server master iburst
allow 0.0.0.0/24
local stratum 10
------------------------------------

slave(1和2)
------------------------------------
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
server master iburst
------------------------------------

systemctl restart chronyd &&  systemctl enable chronyd

#查看时间同步源
chronyc sources -v
#查看时间同步源状态
chronyc sourcestats -v
#校准时间同步服务
chronyc tracking

8.安装JDK环境

#准备文件 jdk-8u191-linux-x64.tar.gz
tar -xzf jdk-8u191-linux-x64.tar.gz -C /

#添加环境变量
vi /etc/profile
#追加
export JAVA_HOME=/jdk1.8.0_191
export PATH=$PATH:$JAVA_HOME/bin

#刷新环境变量
source /etc/profile

#验证安装
java -version
#成功安装
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)

2.Hadoop安装

所有节点

1.解压Hadoop（hadoop-2.7.7.tar.gz）

tar -xzf hadoop-2.7.7.tar.gz -C /

2.添加环境变量

vi /etc/profile
#末尾追加
export HADOOP_HOME=/hadoop-2.7.7
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

#刷新环境变量
source /etc/profile

3.验证Hadoop环境变量

hadoop version

Hadoop 2.7.7
Subversion Unknown -r c1aad84bd27cd79c3d1a7dd58202a8c3ee1ed3ac
Compiled by stevel on 2018-07-18T22:47Z
Compiled with protoc 2.5.0
From source with checksum 792e15d20b12c74bd6f19a1fb886490
This command was run using /hadoop-2.7.7/share/hadoop/common/hadoop-common-2.7.7.jar

3.Hadoop配置文件

0.文件路径

cd /hadoop-2.7.7/etc/hadoop/

1.配置core-site.xml

vim core-site.xml
------------------------------
<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://master:9000</value>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/root/hadoop/tmp</value>
        </property>
</configuration>

2.配置hdfs-site.xml

vim hdfs-site.xml
------------------------------
<configuration>
        <property>
                <name>dfs.replication</name>
                <value>3</value>
        </property>
</configuration>

3.配置mapred-site.xml

cp -r mapred-site.xml.template mapred-site.xml
vim mapred-site.xml
------------------------------
<configuration>
      <property>
              <name>mapreduce.framework.name</name>
              <value>yarn</value>
      </property>
</configuration>

4.配置yarn-site.xml

vim yarn-site.xml
-------------------------------
<configuration>
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
</configuration>

5.配置slaves

vim slaves
#清空改为
--------------------------------
master
slave1
slave2

4.初始化Hadoop

master

hadoop namenode -format

## 5.启动Hadoop

master

start-all.sh

如果JAVA_HOME报错，则所有节点hadoop-env.sh文件改为绝对路径，再次启动

• 报错内容

  0.0.0.0: Error: JAVA_HOME is not set and could not be found.

• 修改路径为绝对路径

  vim /hadoop-2.7.7/etc/hadoop/hadoop-env.sh
  
  export JAVA_HOME=${JAVA_HOME}
  改为
  export JAVA_HOME=/jdk1.8.0_191

6.验证是否启动成功

jps
#出现
11858 NameNode
11495 ResourceManager
12167 SecondaryNameNode
12535 Jps
11993 DataNode
12411 NodeManager