CentOS7部署CDH-6.2.0

一、安装前准备

1.1修改主机名并配置hosts解析

192.168.0.126 poc1.xuncetehch.com poc1

192.168.0.127 poc2.xuncetehch.com poc2

192.168.0.128 poc3.xuncetehch.com poc3

1.2配置ssh的远程连接端口

[root@cdh-master tmp]# vim /etc/services

ssh 22/tcp 22端口修改为31235

ssh 22/udp 22端口修改为31235

1.3启用firewalld信任(CDH集群内部机器均需要配置)

查看firewall拥有的zones

# firewall-cmd –get-zones

查看当前使用zones(激活状态)

# firewall-cmd –get-active-zones

启用trusted

# vim /etc/firewalld/zones/trusted.xml

<?xml version=”1.0″ encoding=”utf-8″?>

<short>Trusted</short>

<description>All network connections are accepted.</description>

</zone>

备注：这里网卡口及允许ip访问根据实际情况填写

# firewall-cmd –reload

二、安装依赖包

yum -y install ntp

yum -y install libxml2-python

yum -y install libxslt.x86_64

yum -y install psmisc

yum -y install wget

yum -y install redhat-lsb

yum -y install openssl-devel

yum -y install python-psycopg2

yum -y install mod_ssl

yum -y install httpd

yum -y install MySQL-python

yum -y install cyrus-sasl-gssapi

#ntp服务安装后不启用

三、配置Cloudera源

参考文档：https://www.cloudera.com/documentation/enterprise/upgrade/topics/cm_ig_create_local_package_repo.html

3.1启动httpd服务

# sudo systemctl start httpd

3.2修改httpd配置文件

# vim /etc/httpd/conf/httpd.conf

开启：

AddType application/x-gzip .gz .tgz .parcel

# sudo systemctl restart httpd

3.3配置站点

[root@poc1 ~]# sudo mkdir -p /var/www/html/cloudera-repos

[root@poc1 ~]# sudo wget –recursive –no-parent –no-host-directories https://archive.cloudera.com/cm6/6.2.0/redhat7/ -P /var/www/html/cloudera-repos

[root@poc1 ~]# sudo wget https://archive.cloudera.com/cm6/6.2.0/allkeys.asc -P /var/www/html/cloudera-repos/cm6/6.2.0/

[root@poc1 ~]# sudo chmod -R ugo+rX /var/www/html/cloudera-repos/cm6

[root@poc1 ~]# sudo mkdir -p /var/www/html/cloudera-repos

[root@poc1 ~]# sudo wget –recursive –no-parent –no-host-directories https://archive.cloudera.com/cdh6/6.2.0/parcels/ -P /var/www/html/cloudera-repos

[root@poc1 ~]# sudo wget –recursive –no-parent –no-host-directories https://archive.cloudera.com/gplextras6/6.2.0/redhat7/ -P /var/www/html/cloudera-repos

# sudo chmod -R ugo+rX /var/www/html/cloudera-repos/cdh6

# sudo chmod -R ugo+rX /var/www/html/cloudera-repos/gplextras6

[root@poc1 html]# cd /var/www/html

四、安装mysql

4.1安装版本mysql-community-server-5.7

参考文档：https://raw.githubusercontent.com/xiangys0134/deploy/v1.0.12/software_install/mysql/mysql_xunce_rpm-5-7-25.sh

4.2配置mysql用户

create database metastore default character set utf8;

CREATE USER ‘hive’@’%’ IDENTIFIED BY ‘MySQL@123’;

GRANT ALL PRIVILEGES ON metastore. * TO ‘hive’@’%’;

FLUSH PRIVILEGES;

create database cm default character set utf8;

CREATE USER ‘cm’@’%’ IDENTIFIED BY ‘MySQL@123’;

GRANT ALL PRIVILEGES ON cm. * TO ‘cm’@’%’;

FLUSH PRIVILEGES;

create database hue default character set utf8;

CREATE USER ‘hue’@’%’ IDENTIFIED BY ‘MySQL@123’;

GRANT ALL PRIVILEGES ON hue. * TO ‘hue’@’%’;

FLUSH PRIVILEGES;

create database oozie default character set utf8;

CREATE USER ‘oozie’@’%’ IDENTIFIED BY ‘MySQL@123’;

GRANT ALL PRIVILEGES ON oozie. * TO ‘oozie’@’%’;

FLUSH PRIVILEGES;

#for Sentry

create database sentry default character set utf8;

CREATE USER ‘sentry’@’%’ IDENTIFIED BY ‘MySQL@123’;

GRANT ALL PRIVILEGES ON sentry. * TO ‘sentry’@’%’;

FLUSH PRIVILEGES;

#for Activity Monitor

create database am default character set utf8;

CREATE USER ‘am’@’%’ IDENTIFIED BY ‘MySQL@123’;

GRANT ALL PRIVILEGES ON am.* TO ‘am’@’%0’;

FLUSH PRIVILEGES;

#for Reports Manager

create database rm default character set utf8;

CREATE USER ‘rm’@’%’ IDENTIFIED BY ‘MySQL@123’;

GRANT ALL PRIVILEGES ON rm.* TO ‘rm’@’%’;

FLUSH PRIVILEGES;

五、安装CDH

5.1配置repo源(CDH所有主机均配置源)

[root@poc1 ~]# vim /etc/yum.repos.d/cloudera-manager.repo

[cloudera-manager]

name=Cloudera Manager 6.2.0

baseurl=http://192.168.0.225/cloudera-repos/cm6/6.2.0/redhat7/yum/

gpgkey=http://192.168.0.225/cloudera-repos/cm6/6.2.0/redhat7/yum/RPM-GPG-KEY-cloudera

gpgcheck=1

enabled=1

autorefresh=0

type=rpm-md

导入hash

# rpm -import http://192.168.0.126/cloudera-repos/cm6/6.2.0/redhat7/yum/RPM-GPG-KEY-cloudera

5.2安装jdk(所有cdh集群机器都需要安装)

[root@poc1 tmp]# yum install java-1.8.0-openjdk-devel

5.3安装cloudera-manager-server

[root@poc1 tmp]# sudo yum install cloudera-manager-server

[root@poc1 tmp]# systemctl enable cloudera-scm-server.service

[root@poc1 tmp]# systemctl start cloudera-scm-server.service

5.4开放firewalld端口

firewall-cmd –zone=public –add-port=7180/tcp –permanent

firewall-cmd –zone=public –add-port=9092/tcp –permanent

firewall-cmd –zone=public –add-port=8042/tcp –permanent

firewall-cmd –zone=public –add-port=8888/tcp –permanent

firewall-cmd –reload

六、配置mysql驱动

[root@poc1 tmp]# wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz

[root@poc1 tmp]# tar -zxvf mysql-connector-java-5.1.46.tar.gz

[root@poc1 tmp]# cp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

七、初始化数据库

[root@poc1 tmp]# sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql cm cm

八、安装agent

8.1配置License(使用免费版本)

http://192.168.0.126:7180/cmf/login admin/admin

8.2配置agent的ssh端口

Specify Hosts 配置SSH端口为31235

8.2配置/opt/cloudera/parcel-repo/本地源

下载地址： https://archive.cloudera.com/cdh6/6.2.0/parcels/

[root@poc1 parcel-repo]# pwd

/opt/cloudera/parcel-repo

[root@poc1 parcel-repo]# ll

总用量 2038776

-rw-r–r– 1 root root 2087665645 8月 13 13:43 CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel

-rw-r–r– 1 root root 64 8月 13 13:43 CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha256

-rw-r–r– 1 root root 33726 8月 13 13:43 manifest.json

8.3配置url源

8.4配置sshagent凭据

填写对应ssh连接账号密码及相应端口

九、安装组件

9.1类型如下进行添加服务：

9.2组件

十、应用管理

10.1关闭yarn内存检查

yarn-site.xml 的 NodeManager 高级配置代码段

yarn.nodemanager.pmem-check-enabled=false

yarn.nodemanager.vmem-check-enabled=false

10.2yarn其他基础参数优化

yarn.scheduler.minimum-allocation-mb=1G

yarn.scheduler.maximum-allocation-mb=10G

yarn.app.mapreduce.am.resource.cpu-vcores=3

mapreduce.map.cpu.vcores=3

mapreduce.reduce.cpu.vcores=3

mapreduce.map.memory.mb=5G

mapreduce.reduce.memory.mb=5G

yarn.nodemanager.resource.memory-mb=32G

作者

留言

撰写回覆或留言取消回复

CentOS7部署CDH-6.2.0

作者

留言

撰写回覆或留言 取消回复

撰写回覆或留言取消回复