CentOS7安装hadoop(分布式)

环境

192.168.10.150 xiangys0134-haddop01 master

192.168.10.151 xiangys0134-haddop02 slave

192.168.10.152 xiangys0134-haddop03 slave

一、配置hosts

192.168.10.150 xiangys0134-haddop01
192.168.10.151 xiangys0134-haddop02
192.168.10.152 xiangys0134-haddop03

二、安装jdk

[root@xiangys0134-haddop01 tmp]# yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel

三、配置ssh

[opadm@xiangys0134-haddop02 ~]$ ssh-keygen -t rsa
[opadm@xiangys0134-haddop02 ~]$ cd ~/
[opadm@xiangys0134-haddop02 ~]$ tar -czvf /tmp/ssh-key.tar.gz .ssh

#将压缩包拷贝至另外两台机器
[opadm@xiangys0134-haddop02 ~]$ scp -P 31235 /tmp/ssh-key.tar.gz opadm@192.168.10.150:/tmp
[opadm@xiangys0134-haddop02 ~]$ scp -P 31235 /tmp/ssh-key.tar.gz opadm@192.168.10.152:/tmp

[opadm@xiangys0134-haddop01 tmp]$ tar -zxvf ssh-key.tar.gz -C ~/    #分别在其他两台机器解压
[opadm@xiangys0134-haddop03 tmp]$ tar -zxvf ssh-key.tar.gz -C ~/

[opadm@xiangys0134-haddop01 .ssh]$ vi authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGwt4dcJx7MHi9w6X2na4EcpHOtmXhFGTSQs2rD3tYrUw0cqG/Tz6jbc1lpAGK+/8pAzcQTACpweFnURrma9um/5Xj9LGXoPGzrHPJ+X197rTFrWA8pZJ93HHTAR59cEUfTGrjoOn+c41ccozMH4BiBFZNgC7dIIUTVDWnTsxQzy5R9VgHzaCNvCaf5uTgDTlWiz1IHZ6YPUPwqarZRibBYSDqowyppIzAKo6+6UHU8dRxQ4h5EZ58ueYWwLW8VKtziVtkFIjDWPkPkLbJ1J3evwYhs9QVISs4j2VSa8INrkcXGeunwEhDW9yBiZCbZYLvv0PyGwQRbEzO6DDf27ex opadm@xiangys0134-haddop02

测试：
[opadm@xiangys0134-haddop01 ~]$ ssh -p 31235 opadm@192.168.10.150

三、配置hadoop环境变量

[opadm@xiangys0134-haddop01 tmp]$ scp -P 31235 hadoop-2.7.7.tar.gz opadm@192.168.10.151:/tmp/
[opadm@xiangys0134-haddop01 tmp]$ scp -P 31235 hadoop-2.7.7.tar.gz opadm@192.168.10.152:/tmp/

[opadm@xiangys0134-haddop01 tmp]$ sudo tar -zxvf hadoop-2.7.7.tar.gz -C /usr/local/

添加JAVA_HOME
[opadm@xiangys0134-haddop01 tmp]$ vi ~/.bashrc
export HADOOP_HOME=/usr/local/hadoop-2.7.7
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64
export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

[opadm@xiangys0134-haddop01 tmp]$ source ~/.bashrc

四、创建数据与日志目录

[opadm@xiangys0134-haddop01 tmp]$ sudo mkdir /data/hadoop/hdfs/{nn,snn,dn} -p
[opadm@xiangys0134-haddop01 tmp]$ sudo mkdir /var/log/hadoop/yarn -p
[opadm@xiangys0134-haddop01 tmp]$ sudo chown -R opadm. /data/hadoop/
[opadm@xiangys0134-haddop01 tmp]$ sudo chown -R opadm. /var/log/hadoop/

五、修改配置文件

应用配置文件：
core-default.xml
hdfs-default.xml
yarn-default.xml
mapred-default.xml

集群必要配置文件：

etc/hadoop/core-site.xml
etc/hadoop/hdfs-site.xml
etc/hadoop/yarn-site.xml
etc/hadoop/mapred-site.xml

[opadm@xiangys0134-haddop01 tmp]$ cd /usr/local/hadoop-2.7.7/etc/hadoop
[opadm@xiangys0134-haddop01 hadoop]$ vim core-site.xml
添加如下内容：

    
        io.file.buffer.size
        131072
    

    
        fs.defaultFS
        hdfs://xiangys0134-haddop01:8020
    


备注：
fs.defaultFS    NameNode URI
io.file.buffer.size 131072 

[opadm@xiangys0134-haddop01 hadoop]$ vim hdfs-site.xml

        
            dfs.namenode.name.dir
            file:///data/hadoop/hdfs/nn
        
        
            dfs.datanode.data.dir
            file:///data/hadoop/hdfs/dn
        
        
                dfs.replication
                2
        
        
                dfs.permissions
                false
        
        
                dfs.namenode.secondary.http-address
                xiangys0134-haddop02:50090
        


备注：在其他slave节点则不需要配置dfs.namenode.name.dir

[root@xiangys0134-haddop01 hadoop]# vim yarn-site.xml

        
                yarn.nodemanager.aux-services
                mapreduce_shuffle
        
        
                yarn.resourcemanager.webapp.address
                xiangys0134-haddop01:8088
        
        
                yarn.resourcemanager.address
                xiangys0134-haddop01:8032
        
        
                yarn.resourcemanager.scheduler.address
                xiangys0134-haddop01:8030
        
        
                yarn.resourcemanager.resource-tracker.address
                xiangys0134-haddop01:8031
        
        
                mapreduce.jobhistory.address
                xiangys0134-haddop01:10020
        
        
                yarn.nodemanager.vmem-check-enabled
                false
        


[opadm@xiangys0134-haddop01 hadoop]$ cp mapred-site.xml.template mapred-site.xml
[opadm@xiangys0134-haddop01 hadoop]$ vim mapred-site.xml

        
                mapreduce.framework.name
                yarn
        
        
                mapreduce.jobhistory.address
                xiangys0134-haddop03:10020
        

六、宣告slaves

[opadm@xiangys0134-haddop01 hadoop]$ vim slaves
xiangys0134-haddop01
xiangys0134-haddop02
xiangys0134-haddop03

七、配置集群

7.1如果集群是第一次启动，需要格式化namenode
[opadm@xiangys0134-haddop01 hadoop]$ hadoop namenode -format
...
20/04/18 13:44:42 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
20/04/18 13:44:42 INFO util.ExitUtil: Exiting with status 0
20/04/18 13:44:42 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at xiangys0134-haddop01/192.168.10.150
************************************************************/

验证：
[opadm@xiangys0134-haddop01 hadoop]$ ls /data/hadoop/hdfs/nn/
current

八、启动服务

8.1配置ssh的远程连接端口

[opadm@xiangys0134-haddop01 hadoop]$ sudo vim /etc/services
ssh 22/tcp    22端口修改为31235
ssh 22/udp   22端口修改为31235

8.2启动hdfs

[opadm@xiangys0134-haddop01 hadoop]$ start-dfs.sh

验证：
[opadm@xiangys0134-haddop01 hadoop]$ start-dfs.sh 
Starting namenodes on [xiangys0134-haddop01]
xiangys0134-haddop01: starting namenode, logging to /usr/local/hadoop-2.7.7/logs/hadoop-opadm-namenode-xiangys0134-haddop01.out
xiangys0134-haddop01: starting datanode, logging to /usr/local/hadoop-2.7.7/logs/hadoop-opadm-datanode-xiangys0134-haddop01.out
xiangys0134-haddop02: starting datanode, logging to /usr/local/hadoop-2.7.7/logs/hadoop-opadm-datanode-xiangys0134-haddop02.out
xiangys0134-haddop03: starting datanode, logging to /usr/local/hadoop-2.7.7/logs/hadoop-opadm-datanode-xiangys0134-haddop03.out
Starting secondary namenodes [xiangys0134-haddop02]
xiangys0134-haddop02: starting secondarynamenode, logging to /usr/local/hadoop-2.7.7/logs/hadoop-opadm-secondarynamenode-xiangys0134-haddop02.out

查看NN与DN
[opadm@xiangys0134-haddop01 hadoop]$ jps
22098 DataNode
22402 Jps
21959 NameNode

8.3启动yarn

[opadm@xiangys0134-haddop01 hadoop]$ start-yarn.sh

九、测试上传

[opadm@xiangys0134-haddop02 hadoop]$ hdfs dfs -mkdir /test
[opadm@xiangys0134-haddop02 hadoop]$ hdfs dfs -put /etc/fstab /test/fstab

[opadm@xiangys0134-haddop02 hadoop]$ hdfs dfs -ls /test
Found 1 items
-rw-r--r-- 2 opadm supergroup 465 2020-04-18 14:16 /test/fstab

十、WEB UI

10.1hdfs的ui

http://192.168.10.150:50070

10.2yarn的ui

http://192.168.10.150:8088

10.3namesecondary的ui

http://192.168.10.151:50090