CentOS 7.4下Hadoop 2.7.6安装搭建

1. 主机规划
主机名称
 
外网IP
 
内网IP
 
操作系统
 
备注
 
安装软件
 
mini01
 
10.0.0.11
 
172.16.1.11
 
CentOS 7.4
 
ssh port:22
 
Hadoop 【NameNode  SecondaryNameNode】
 
mini02
 
10.0.0.12
 
172.16.1.12
 
CentOS 7.4
 
ssh port:22
 
Hadoop 【ResourceManager】
 
mini03
 
10.0.0.13
 
172.16.1.13
 
CentOS 7.4
 
ssh port:22
 
Hadoop 【DataNode  NodeManager】
 
mini04
 
10.0.0.14
 
172.16.1.14
 
CentOS 7.4
 
ssh port:22
 
Hadoop 【DataNode  NodeManager】
 
mini05
 
10.0.0.15
 
172.16.1.15
 
CentOS 7.4
 
ssh port:22
 
Hadoop 【DataNode  NodeManager】
 
 
 
添加hosts信息，保证每台都可以相互ping通
 
[root@mini01 ~]# cat /etc/hosts
127.0.0.1  localhost localhost.localdomain localhost4 localhost4.localdomain4
::1        localhost localhost.localdomain localhost6 localhost6.localdomain6
 
10.0.0.11    mini01
10.0.0.12    mini02
10.0.0.13    mini03
10.0.0.14    mini04
10.0.0.15    mini05
 
2. ASP站长网添加用户账号
 
# 使用一个专门的用户，避免直接使用root用户
# 添加用户、指定家目录并指定用户密码
useradd -d /app yun && echo '123456' | /usr/bin/passwd --stdin yun
# sudo提权
echo "yun  ALL=(ALL)      NOPASSWD: ALL" >>  /etc/sudoers
# 让其它普通用户可以进入该目录查看信息
chmod 755 /app/
 
3. 实现yun用户免秘钥登录
 
要求：根据规划实现 mini01 到 mini01、mini02、mini03、mini04、mini05 免秘钥登录
              实现 mini02 到 mini01、mini02、mini03、mini04、mini05 免秘钥登录
# 可以使用ip也可以是hostname  但是由于我们计划使用的是 hostname 方式交互，所以使用hostname
# 同时hostname方式分发，可以通过hostname远程登录，也可以IP远程登录
 
3.1. 创建密钥
 
# 实现 mini01 到 mini02、mini03、mini04、mini05 免秘钥登录
[yun@mini01 ~]$ ssh-keygen -t rsa  # 一路回车即可
Generating public/private rsa key pair.
Enter file in which to save the key (/app/.ssh/id_rsa):
Created directory '/app/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /app/.ssh/id_rsa.
Your public key has been saved in /app/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:rAFSIyG6Ft6qgGdVl/7v79DJmD7kIDSTcbiLtdKyTQk yun@mini01
The key's randomart image is:
+---[RSA 2048]----+
|. o.o    .      |
|.. o .  o..      |
|... . . o=      |
|..o. oE+B        |
|.o .. .*S*      |
|o ..  +oB.. .= . |
|o.o  .* ..++ +  |
|oo    . .  oo.  |
|.          .++o  |
+----[SHA256]-----+
 
# 生成之后会在用户的根目录生成一个 “.ssh”的文件夹
[yun@mini01 ~]$ ll -d .ssh/
drwx------ 2 yun yun 38 Jun  9 19:17 .ssh/
[yun@mini01 ~]$ ll .ssh/
total 8
-rw------- 1 yun yun 1679 Jun  9 19:17 id_rsa
-rw-r--r-- 1 yun yun  392 Jun  9 19:17 id_rsa.pub
 
3.2. 分发密钥
 
# 可以使用ip也可以是hostname  但是由于我们使用的是 hostname 方式通信，所以使用hostname
[yun@mini01 ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub 172.16.1.11  # IP方式【这里不用】
# 分发
[yun@mini01 ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mini03  # 主机名方式【所有的都这样 从mini01到mini05】
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/app/.ssh/id_rsa.pub"
The authenticity of host '[mini03]:22 ([10.0.0.13]:22)' can't be established.
ECDSA key fingerprint is SHA256:pN2NUkgCTt+b9P5TfQZcTh4PF4h7iUxAs6+V7Slp1YI.
ECDSA key fingerprint is MD5:8c:f0:c7:d6:7c:b1:a8:59:1c:c1:5e:d7:52:cb:5f:51.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
yun@mini03's password:
 
Number of key(s) added: 1
 
Now try logging into the machine, with:  "ssh -p '22' 'mini03'"
and check to make sure that only the key(s) you wanted were added.
 
mini01分发密钥
 
[yun@mini01 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mini01
[yun@mini01 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mini02
[yun@mini01 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mini03
[yun@mini01 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mini04
[yun@mini01 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mini05
 
mini02分发密钥
 
[yun@mini02 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mini01
[yun@mini02 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mini02
[yun@mini02 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mini03
[yun@mini02 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mini04
[yun@mini02 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mini05
 
远程登录测试【最好都测试一下】
 
 [yun@mini02 ~]$ ssh mini05
Last login: Sat Jun  9 19:47:43 2018 from 10.0.0.11
 
Welcome You Login
 
[yun@mini05 ~]$            # 表示远程登录成功
 
3.3. 远程免密登录原理图
 
 
 
 
3.4. .ssh目录中的文件说明
 [yun@mini01 .ssh]$ pwd
/app/.ssh
[yun@mini01 .ssh]$ ll
total 16
-rw------- 1 yun yun  784 Jun  9 19:43 authorized_keys
-rw------- 1 yun yun 1679 Jun  9 19:17 id_rsa
-rw-r--r-- 1 yun yun  392 Jun  9 19:17 id_rsa.pub
-rw-r--r-- 1 yun yun 1332 Jun  9 19:41 known_hosts
########################################################################################
　　authorized_keys:存放远程免密登录的公钥,主要通过这个文件记录多台机器的公钥
　　id_rsa : 生成的私钥文件
　　id_rsa.pub ： 生成的公钥文件
know_hosts : 已知的主机公钥清单
 
4. Jdk【java8】
4.1. 软件安装
 [yun@mini01 software]# pwd
/app/software
[yun@mini01 software]# tar xf jdk1.8.0_112.tar.gz
[yun@mini01 software]# ll
total 201392
drwxr-xr-x 8  10  143      4096 Dec 20 13:27 jdk1.8.0_112
-rw-r--r-- 1 root root 189815615 Mar 12 16:47 jdk1.8.0_112.tar.gz
[yun@mini01 software]# mv jdk1.8.0_112/ /app/
[yun@mini01 software]# cd /app/
[yun@mini01 app]# ll
total 8
drwxr-xr-x  8  10  143 4096 Dec 20 13:27 jdk1.8.0_112
[yun@mini01 app]# ln -s jdk1.8.0_112/ jdk
[yun@mini01 app]# ll
total 8
lrwxrwxrwx  1 root root    13 May 16 23:19 jdk -> jdk1.8.0_112/
drwxr-xr-x  8  10  143 4096 Dec 20 13:27 jdk1.8.0_112