CentOS7安装hadoop(分布式)

环境

192.168.10.150 xiangys0134-haddop01 master

192.168.10.151 xiangys0134-haddop02 slave

192.168.10.152 xiangys0134-haddop03 slave

一、配置hosts

192.168.10.150 xiangys0134-haddop01
192.168.10.151 xiangys0134-haddop02
192.168.10.152 xiangys0134-haddop03

二、安装jdk

[root@xiangys0134-haddop01 tmp]# yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel

三、配置ssh

[opadm@xiangys0134-haddop02 ~]ssh-keygen -t rsa
[opadm@xiangys0134-haddop02 ~] cd ~/
[opadm@xiangys0134-haddop02 ~]tar -czvf /tmp/ssh-key.tar.gz .ssh

#将压缩包拷贝至另外两台机器
[opadm@xiangys0134-haddop02 ~] scp -P 31235 /tmp/ssh-key.tar.gz opadm@192.168.10.150:/tmp
[opadm@xiangys0134-haddop02 ~]scp -P 31235 /tmp/ssh-key.tar.gz opadm@192.168.10.152:/tmp

[opadm@xiangys0134-haddop01 tmp] tar -zxvf ssh-key.tar.gz -C ~/    #分别在其他两台机器解压
[opadm@xiangys0134-haddop03 tmp]tar -zxvf ssh-key.tar.gz -C ~/

[opadm@xiangys0134-haddop01 .ssh] vi authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGwt4dcJx7MHi9w6X2na4EcpHOtmXhFGTSQs2rD3tYrUw0cqG/Tz6jbc1lpAGK+/8pAzcQTACpweFnURrma9um/5Xj9LGXoPGzrHPJ+X197rTFrWA8pZJ93HHTAR59cEUfTGrjoOn+c41ccozMH4BiBFZNgC7dIIUTVDWnTsxQzy5R9VgHzaCNvCaf5uTgDTlWiz1IHZ6YPUPwqarZRibBYSDqowyppIzAKo6+6UHU8dRxQ4h5EZ58ueYWwLW8VKtziVtkFIjDWPkPkLbJ1J3evwYhs9QVISs4j2VSa8INrkcXGeunwEhDW9yBiZCbZYLvv0PyGwQRbEzO6DDf27ex opadm@xiangys0134-haddop02

测试：
[opadm@xiangys0134-haddop01 ~]$ ssh -p 31235 opadm@192.168.10.150

三、配置hadoop环境变量

[opadm@xiangys0134-haddop01 tmp]scp -P 31235 hadoop-2.7.7.tar.gz opadm@192.168.10.151:/tmp/
[opadm@xiangys0134-haddop01 tmp] scp -P 31235 hadoop-2.7.7.tar.gz opadm@192.168.10.152:/tmp/

[opadm@xiangys0134-haddop01 tmp]sudo tar -zxvf hadoop-2.7.7.tar.gz -C /usr/local/

添加JAVA_HOME
[opadm@xiangys0134-haddop01 tmp] vi ~/.bashrc
export HADOOP_HOME=/usr/local/hadoop-2.7.7
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64
export PATH=JAVA_HOME/bin:HADOOP_HOME/bin:HADOOP_HOME/sbin:PATH

[opadm@xiangys0134-haddop01 tmp]$ source ~/.bashrc

四、创建数据与日志目录

[opadm@xiangys0134-haddop01 tmp]sudo mkdir /data/hadoop/hdfs/{nn,snn,dn} -p
[opadm@xiangys0134-haddop01 tmp] sudo mkdir /var/log/hadoop/yarn -p
[opadm@xiangys0134-haddop01 tmp]sudo chown -R opadm. /data/hadoop/
[opadm@xiangys0134-haddop01 tmp] sudo chown -R opadm. /var/log/hadoop/

五、修改配置文件

应用配置文件：
core-default.xml
hdfs-default.xml
yarn-default.xml
mapred-default.xml

集群必要配置文件：

etc/hadoop/core-site.xml
etc/hadoop/hdfs-site.xml
etc/hadoop/yarn-site.xml
etc/hadoop/mapred-site.xml

[opadm@xiangys0134-haddop01 tmp]cd /usr/local/hadoop-2.7.7/etc/hadoop
[opadm@xiangys0134-haddop01 hadoop] vim core-site.xml
添加如下内容：
<configuration>
    <property>
        <name>io.file.buffer.size</name>
        <value>131072</value>
    </property>

    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://xiangys0134-haddop01:8020</value>
    </property>
</configuration>

备注：
fs.defaultFS    NameNode URI
io.file.buffer.size 131072 

[opadm@xiangys0134-haddop01 hadoop]vim hdfs-site.xml
<configuration>
        <property>
            <name>dfs.namenode.name.dir</name>
            <value>file:///data/hadoop/hdfs/nn</value>
        </property>
        <property>
            <name>dfs.datanode.data.dir</name>
            <value>file:///data/hadoop/hdfs/dn</value>
        </property>
        <property>
                <name>dfs.replication</name>
                <value>2</value>
        </property>
        <property>
                <name>dfs.permissions</name>
                <value>false</value>
        </property>
        <property>
                <name>dfs.namenode.secondary.http-address</name>
                <value>xiangys0134-haddop02:50090</value>
        </property>
</configuration>

备注：在其他slave节点则不需要配置dfs.namenode.name.dir

[root@xiangys0134-haddop01 hadoop]# vim yarn-site.xml
<configuration>
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
        <property>
                <name>yarn.resourcemanager.webapp.address</name>
                <value>xiangys0134-haddop01:8088</value>
        </property>
        <property>
                <name>yarn.resourcemanager.address</name>
                <value>xiangys0134-haddop01:8032</value>
        </property>
        <property>
                <name>yarn.resourcemanager.scheduler.address</name>
                <value>xiangys0134-haddop01:8030</value>
        </property>
        <property>
                <name>yarn.resourcemanager.resource-tracker.address</name>
                <value>xiangys0134-haddop01:8031</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.address</name>
                <value>xiangys0134-haddop01:10020</value>
        </property>
        <property>
                <name>yarn.nodemanager.vmem-check-enabled</name>
                <value>false</value>
        </property>
</configuration>

[opadm@xiangys0134-haddop01 hadoop] cp mapred-site.xml.template mapred-site.xml
[opadm@xiangys0134-haddop01 hadoop]$ vim mapred-site.xml
<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.address</name>
                <value>xiangys0134-haddop03:10020</value>
        </property>
</configuration>

六、宣告slaves

[opadm@xiangys0134-haddop01 hadoop]$ vim slaves
xiangys0134-haddop01
xiangys0134-haddop02
xiangys0134-haddop03

七、配置集群

7.1如果集群是第一次启动，需要格式化namenode
[opadm@xiangys0134-haddop01 hadoop]hadoop namenode -format
...
20/04/18 13:44:42 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
20/04/18 13:44:42 INFO util.ExitUtil: Exiting with status 0
20/04/18 13:44:42 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at xiangys0134-haddop01/192.168.10.150
************************************************************/

验证：
[opadm@xiangys0134-haddop01 hadoop] ls /data/hadoop/hdfs/nn/
current

八、启动服务

8.1配置ssh的远程连接端口

[opadm@xiangys0134-haddop01 hadoop]$ sudo vim /etc/services
ssh 22/tcp    22端口修改为31235
ssh 22/udp   22端口修改为31235

8.2启动hdfs

[opadm@xiangys0134-haddop01 hadoop]start-dfs.sh

验证：
[opadm@xiangys0134-haddop01 hadoop] start-dfs.sh 
Starting namenodes on [xiangys0134-haddop01]
xiangys0134-haddop01: starting namenode, logging to /usr/local/hadoop-2.7.7/logs/hadoop-opadm-namenode-xiangys0134-haddop01.out
xiangys0134-haddop01: starting datanode, logging to /usr/local/hadoop-2.7.7/logs/hadoop-opadm-datanode-xiangys0134-haddop01.out
xiangys0134-haddop02: starting datanode, logging to /usr/local/hadoop-2.7.7/logs/hadoop-opadm-datanode-xiangys0134-haddop02.out
xiangys0134-haddop03: starting datanode, logging to /usr/local/hadoop-2.7.7/logs/hadoop-opadm-datanode-xiangys0134-haddop03.out
Starting secondary namenodes [xiangys0134-haddop02]
xiangys0134-haddop02: starting secondarynamenode, logging to /usr/local/hadoop-2.7.7/logs/hadoop-opadm-secondarynamenode-xiangys0134-haddop02.out

查看NN与DN
[opadm@xiangys0134-haddop01 hadoop]$ jps
22098 DataNode
22402 Jps
21959 NameNode

8.3启动yarn

[opadm@xiangys0134-haddop01 hadoop]$ start-yarn.sh

九、测试上传

[opadm@xiangys0134-haddop02 hadoop]hdfs dfs -mkdir /test
[opadm@xiangys0134-haddop02 hadoop] hdfs dfs -put /etc/fstab /test/fstab

[opadm@xiangys0134-haddop02 hadoop]$ hdfs dfs -ls /test
Found 1 items
-rw-r--r-- 2 opadm supergroup 465 2020-04-18 14:16 /test/fstab

十、WEB UI

10.1hdfs的ui

http://192.168.10.150:50070

10.2yarn的ui

http://192.168.10.150:8088

10.3namesecondary的ui

http://192.168.10.151:50090