八、ranger权限开启
8.1 ranger登录
访问图中链接会跳转到ranger登录页面，登录失败可能需要将主机名换成IP。

登录ranger web页面，用户名：admin ，密码：winnerxxxx

登录后的页面如下



我们需要添加winner_spark的服务组件操作权限。

8.2 HDFS权限控制
点击进入默认的service设置页面

点击进入Policy ID为1 的策略进入编辑

添加winner_spark用户





添加完成后 save保存。

Policy ID为2的策略也是编辑添加winner_spark 后 保存。winner_spark 用户添加HDFS操作权限完成后如下图



我们需要Policy ID 为1 策略中添加hive 用户对HDFS操作权限，完成后如下图



8.3 HBase权限控制
点击进入默认的service设置页面

点击进入Policy ID为4 的策略进入编辑



添加winner_spark用户



添加完成后 save保存。Policy ID为5的策略也是编辑添加winner_spark 后 保存。

winner_spark 用户添加HBase操作权限完成后如下图



8.4 Hive权限控制
点击进入默认的service设置页面

点击进入Policy ID为8 的策略进入编辑

添加winner_spark用户权限完成后 save保存。



剩下的Policy 策略也是编辑添加winner_spark 后 保存。

winner_spark 用户添加Hive操作权限完成后如下图



8.5 Yarn权限控制
点击进入默认的service设置页面



点击进入Policy ID为6 的策略进入编辑



添加winner_spark用户



剩下的Policy 策略也是编辑添加winner_spark 后 保存。

winner_spark 用户添加Yarn操作权限完成后如下图



九、Ansible自动化安装脚本
说明：脚本可以在博客资源中可以下载，如下是部署目录结构。

脚本入口 installDeployAmbari.sh

#!/bin/bash
#  
# 脚本功能: 配置初始化，完成ambari-server安装启动  
# 作 者: kangll  
# 创建时间: 2024-03-29  
# 修改时间: 2024-03-29
# 当前版本: 1.0v  
# 调度周期: 一次性任务  
# 脚本参数: 无  
#  
#
. /etc/profile > /dev/null 2>&1
set -x
#set -e
# 请确保已经按照部署文档中事先配置好了, ./autoDeployFiles/scripts/hostlist.txt中的内容以及temphosts.txt
# ./config/global.sh


BASEDIR=$(cd "$(dirname "$0")"; pwd)
scriptsDir=$BASEDIR/autoDeployFiles/scripts
cd ${scriptsDir}

if [ `rpm -qa |grep expect > /dev/null 2>&1;echo $?` -ne 0 ]; then
sudo yum install -y expect
fi

if [ `rpm -qa |grep ansible > /dev/null 2>&1;echo $?` -ne 0 ]; then
sudo yum install epel-release -y
sudo yum install ansible -y
fi

cat ${scriptsDir}/temphosts.txt |awk '{print $1,$2}' | while read vIP vHost
do
if [ `grep ${vHost} /etc/hosts > /dev/null 2>&1;echo $?` -ne 0 ]; then
echo "${vIP} ${vHost}" >> /etc/hosts
fi
done


# linux 免密
sh $scriptsDir/batchSendKey.sh

# linux hosts
for i in `cat ${scriptsDir}/temphosts.txt |awk '{print $2}' |grep -v \`hostname\`|xargs `
do
scp /etc/hosts $i:/etc/
done


# linux hosts
for i in `cat ${scriptsDir}/temphosts.txt |awk '{print $2}' |xargs `
do
ssh $i "sudo hostnamectl set-hostname $i"
done

source /etc/profile

ambari_server_source=`hostname`
sudo sed -i 's/windp-aio/'"${ambari_server_source}"'/g' $BASEDIR/ambari.yml

# ansible hosts
echo [all_node] > /etc/ansible/hosts
cat ${scriptsDir}/temphosts.txt |awk '{print $2}' >> /etc/ansible/hosts
echo "" >> /etc/ansible/hosts

# 默认安装ambariserver的服务器为控制端,也就是master节点
echo [master] >> /etc/ansible/hosts
echo `hostname` >> /etc/ansible/hosts
echo "" >> /etc/ansible/hosts

# 除当前服务器的其他服务器为agent
echo [slave] >> /etc/ansible/hosts
cat ${scriptsDir}/temphosts.txt |awk '{print $2}' |grep -v `hostname` >> /etc/ansible/hosts

cd $BASEDIR
ansible-playbook ambari.yml

ambari.yml

---

# author: kangll
# date: 2024-04-03
# funtion: 离线自动化部署ambari-server
# 版本要求：
# OS: Redhat7.2-CentOS7.9(仅支持该操作系统版本)
# ambari-2.7.4
# MySQL 5.7+/Python3+
# 

- hosts: all_node
gather_facts: F
vars:
paths:
scriptsDir: /opt/windp-deploy/autoDeployFiles/scripts
JDKDir: ./autoDeployFiles/JDK
JAVA_HOME_PATH: /usr/java
packages:
jdk: jdk-8u162-linux-x64.tar.gz
tasks:
- include: ./component/setup_base_env.yml # 设置基础环境
- include: ./component/setup_java_env.yml # 设置JAVA_HOME
remote_user: root
tags: jdk_base_env

- hosts: master
gather_facts: F
vars:
local_ambari_os_yum_repo: hdp-node1
paths:
MySQLDir: ./autoDeployFiles/MySQL
installScriptDir: /opt/windp-deploy
MySQLConfDir: ./autoDeployFiles/configFiles
driverDir: /usr/share/java
confDir: ./autoDeployFiles/configFiles
password:
mysql: Winner001
ambari: Winner001
hive: Winner001
packages:
mysql_driver: mysql-connector-java.jar
tasks:
- include: ./component/setup_mysql_server.yml # 创建用户解压MySQL包于/usr/local/mysql、MySQL初始化、修改root密码
- include: ./component/setup_kdc_server.yml # 安装配置kerberos
- include: ./component/setup_ambari_server.yml # 安装并启动Ambari-server
remote_user: root
tags: mysql

- hosts: slave
gather_facts: F
vars:
paths:
repoTmpDir: ./config/repo
repoDir: /etc/yum.repos.d
krb5File: /etc/
keytabFile: /etc/security/keytabs
tasks:
- include: ./component/setup_hdp_repo.yml # repo,kerberos配置同步
remote_user: root
tags: repo

install_base_kdc.sh

#! /bin/bash
#
# Author: kangll
# CreateTime: 2024-03-10
# Desc: kerberos配置
#

#set -x
BASEDIR=$(cd "$(dirname "$0")"; pwd)
# 加载配置
source $BASEDIR/config/global.sh

# global.sh 配置文件中获取
ssh_passwd="winner@001"
kerberos_user=winner_spark

hostName=`hostname`
########################
# 配置kerberos,安装启动
########################
config_krb5() {
# kerberos server and client
sudo yum install krb5-server krb5-libs krb5-workstation -y

# config file
sudo cat $BASEDIR/config/krb5.conf > /etc/krb5.conf
# 修改为 kdc serve hostname
sudo sed -i 's/windp-aio/'"${hostName}"'/g' /etc/krb5.conf
sudo cat $BASEDIR/config/kdc.conf > /var/kerberos/krb5kdc/kdc.conf
sudo cat $BASEDIR/config/kadm5.acl > /var/kerberos/krb5kdc/kadm5.acl

echo "******* 创建kdc数据库 *********"
/usr/bin/expect << eof
# 设置捕获字符串后，期待回复的超时时间
set timeout 30

spawn kdb5_util create -s -r WINNER.COM $1@$2

## 开始进连续捕获
expect {
"Enter KDC database master key:" { send "${ssh_passwd}\n"; exp_continue }
"master key to verify:" { send "${ssh_passwd}\n"; exp_continue }
}
eof
echo "******** 创建admin实例 *********"
/usr/bin/expect << eof
# 设置捕获字符串后，期待回复的超时时间
set timeout 30

spawn kadmin.local 

## 开始进连续捕获
expect {
"kadmin.local:" { send "addprinc admin/admin\n"; exp_continue }
"Enter password for principal" { send "${ssh_passwd}\n"; exp_continue }
"Re-enter password for principal" { send "${ssh_passwd}\n"; }
}
expect "kadmin.local:" { send "quit\r"; }
eof

# start kdc and kadmin 
sudo systemctl restart krb5kdc
sudo systemctl enable krb5kdc
sudo systemctl restart kadmin
sudo systemctl enable kadmin

# add linux user 
sudo useradd winner_spark
# keytabs file path
sudo mkdir -p /etc/security/keytabs/

echo "********** kerberos installation completed **********"
}

##################################
# 配置kerberos用户: winner_spark 
# 生成keytab 文件
##################################
config_kerberos_user() {
echo "******** 创建winner_spark用户实例 ********"
/usr/bin/expect << eof
# 设置捕获字符串后，期待回复的超时时间
set timeout 30

spawn kadmin.local 

## 开始进连续捕获
expect {
"kadmin.local:" { send "addprinc ${kerberos_user}\n"; exp_continue }
"Enter password for principal" { send "${ssh_passwd}\n"; exp_continue }
"Re-enter password for principal" { send "${ssh_passwd}\n"; }
}
expect "kadmin.local:" { send "quit\r"; }
eof

echo "******** winner_spark用户生成keytab文件 ********"
/usr/bin/expect << eof
# 设置捕获字符串后，期待回复的超时时间
set timeout 30

spawn kadmin.local 

## 开始进连续捕获
expect {
"kadmin.local:" { send "xst -k /etc/security/keytabs/${kerberos_user}.keytab ${kerberos_user}@WINNER.COM\n"; }
}
expect "kadmin.local:" { send "quit\r"; }
eof
sleep 2s
# modify keytab file privilege  
sudo chown ${kerberos_user}:${kerberos_user} /etc/security/keytabs/${kerberos_user}.keytab
echo "********** kerberos user winner_spark add completed **********"
}


# 配置kerberos,并启动
config_krb5

# 配置kerberos用户: winner_spark, 生成keytab 文件
config_kerberos_user

install_mysql.sh

#! /bin/bash
#
# Author: kangll
# CreateTime: 2023-11-10
# Desc: install mysql5.7
#
set -x
echo "******** INSTALL MYSQL *********"
####################################
BASEDIR=$(cd "$(dirname "$0")"; pwd)
# 加载数据库默认连接信息
source $BASEDIR/config/global.sh
install_path=$mysql_install_path
hostname=`"hostname"`

#####################################

# 卸载原有的mariadb
OLD_MYSQL=`rpm -qa|grep mariadb`
profile=/etc/profile
for mariadb in $OLD_MYSQL
do
rpm -e --nodeps $mariadb
done

# 删除原有的my.cnf
sudo rm -rf /etc/my.cnf

#添加用户组 用户
sudo groupadd mysql
sudo useradd -g mysql mysql

# 解压mysql包并修改名称
tar -zxvf $BASEDIR/autoDeployFiles/MySQL/mysql-5.7.44-el7-x86_64.tar.gz -C $install_path
sudo mv $install_path/mysql-5.7.44-el7-x86_64 $install_path/mysql

# 更改所属的组和用户
sudo chown -R mysql $install_path/mysql
sudo chgrp -R mysql $install_path/mysql

sudo mkdir -p $install_path/mysql/data
sudo mkdir -p $install_path/mysql/log
sudo chown -R mysql:mysql $install_path/mysql/data

# 粘贴配置文件my.cnf 内容见八 中的 my.cnf
cp -f $BASEDIR/config/my.cnf $install_path/mysql/

# 安装mysql
$install_path/mysql/bin/mysql_install_db --user=mysql --basedir=$install_path/mysql/ --datadir=$install_path/mysql/data/

# 设置文件及目录权限：
cp $install_path/mysql/support-files/mysql.server /etc/init.d/mysqld
sudo chown 777 $install_path/mysql/my.cnf
sudo chmod +x /etc/init.d/mysqld

sudo mkdir /var/lib/mysql
sudo chmod 777 /var/lib/mysql

# 启动mysql
/etc/init.d/mysqld start

# 设置开机启动
chkconfig --level 35 mysqld on
chmod +x /etc/rc.d/init.d/mysqld
chkconfig --add mysqld

# 修改环境变量 
ln -s $install_path/mysql/bin/mysql /usr/bin
ln -s /var/lib/mysql/mysql.sock /tmp/
cat > /etc/profile.d/mysql.sh<<EOF
export PATH=$PATH:$install_path/mysql/bin
EOF

mysqlPw=`sed -n 2p /root/.mysql_secret`
mysqlPwTMP=`sed -n 2p /root/.mysql_secret`1

mysql -u$myuser -p$mysqlPw --connect-expired-password -e "SET PASSWORD = PASSWORD('${mypwd}');"

echo "******** MYSQL installation completed ********"

install_repo.sh

#! /bin/bash
#
# Author: kangll
# CreateTime: 2024-03-10
# Desc: 配置HDP repo
#

set -x
BASEDIR=$(cd "$(dirname "$0")"; pwd)
#
source $BASEDIR/config/global.sh
# HDP tar install path
config_path=$install_path
tar_name=$hdp_tar_name

source /etc/profile > /dev/null 2>&1

#ambari server源地址

if [ $# -eq 1 ] ;then
ambari_server_source=$1
else
ambari_server_source=`hostname`
fi

###########################
# 配置 HDP repo
###########################
config_repo() {
mkdir -p $config_path
if [ ! -d $config_path/hdp ];then
sudo mv $BASEDIR/autoDeployFiles/HDP/hdp $config_path
fi

sudo ln -s $config_path/hdp/ambari /var/www/html/ambari
sudo ln -s $config_path/hdp/HDP /var/www/html/HDP
sudo ln -s $config_path/hdp/HDP-GPL /var/www/html/HDP-GPL
sudo ln -s $config_path/hdp/HDP-UTILS /var/www/html/HDP-UTILS
sudo cp -f $BASEDIR/config/repo/*.repo /etc/yum.repos.d/
sudo sed -i 's/windp-aio/'"${ambari_server_source}"'/g' /etc/yum.repos.d/*.repo
sudo yum clean all
sudo yum makecache
sudo yum repolist
echo "********** repo installation completed **********"
}

###########################
# 初始化db
###########################
config_db() {

mysql -h${myurl} -u${myuser} -p${mypwd} < $BASEDIR/config/init_db.sql 
mysql -h${myurl} -u${myuser} -p${mypwd} ambari < $BASEDIR/config/Ambari-DDL-MySQL-CREATE.sql
}

###########################
# install ambari
###########################
install_ambari() {
sudo yum install ambari-server -y
sudo mkdir -p /usr/share/java/
sudo cp -f $BASEDIR/config/mysql-connector-java.jar /usr/share/java/
sudo cat $BASEDIR/config/ambari.properties > /etc/ambari-server/conf/ambari.properties
sudo sed -i 's/localhost/'"${ambari_server_source}"'/g' /etc/ambari-server/conf/ambari.properties
sudo cp -f $BASEDIR/config/password.dat /etc/ambari-server/conf/
ambari-server restart
ambari-server status
}

######################################################
# 修改服务 ambari,在安装页面隐藏 无需安装的服务组件
######################################################
config_metainfo_modify(){

stack_path=/var/lib/ambari-server/resources/stacks/HDP
cat $BASEDIR/config/repo/services/ACCUMULO/metainfo.xml > $stack_path/3.0/services/ACCUMULO/metainfo.xml
cat $BASEDIR/config/repo/services/KAFKA/metainfo.xml > $stack_path/3.1/services/KAFKA/metainfo.xml
cat $BASEDIR/config/repo/services/PIG/metainfo.xml > $stack_path/3.1/services/PIG/metainfo.xml
cat $BASEDIR/config/repo/services/DRUID/metainfo.xml > $stack_path/3.0/services/DRUID/metainfo.xml
cat $BASEDIR/config/repo/services/LOGSEARCH/metainfo.xml > $stack_path/3.0/services/LOGSEARCH/metainfo.xml
cat $BASEDIR/config/repo/services/SUPERSET/metainfo.xml > $stack_path/3.0/services/SUPERSET/metainfo.xml
cat $BASEDIR/config/repo/services/ATLAS/metainfo.xml > $stack_path/3.1/services/ATLAS/metainfo.xml
cat $BASEDIR/config/repo/services/ZEPPELIN/metainfo.xml > $stack_path/3.0/services/ZEPPELIN/metainfo.xml  
cat $BASEDIR/config/repo/services/STORM/metainfo.xml > $stack_path/3.0/services/STORM/metainfo.xml
cat $BASEDIR/config/repo/services/RANGER_KMS/metainfo.xml > $stack_path/3.1/services/RANGER_KMS/metainfo.xml
cat $BASEDIR/config/repo/services/OOZIE/metainfo.xml > $stack_path/3.0/services/OOZIE/metainfo.xml
cat $BASEDIR/config/repo/services/KNOX/metainfo.xml > $stack_path/3.1/services/KNOX/metainfo.xml
cat $BASEDIR/config/repo/services/SQOOP/metainfo.xml > $stack_path/3.0/services/SQOOP/metainfo.xml
cat $BASEDIR/config/repo/services/SMARTSENSE/metainfo.xml > $stack_path/3.0/services/SMARTSENSE/metainfo.xml
ambari-server restart
}
config_repo
config_db
install_ambari
config_metainfo_modify

《行业指标体系白皮书》下载地址：https://www.dtstack.com/resources/1057/?src=bbs

《数据治理行业实践白皮书》下载地址：https://www.dtstack.com/resources/1001/?src=bbs

《数栈V6.0产品白皮书》下载地址：https://www.dtstack.com/resources/1004/?src=bbs

想了解或咨询更多有关袋鼠云大数据产品、行业解决方案、客户案例的朋友，浏览袋鼠云官网：https://www.dtstack.com/?src=bbs

同时，欢迎对大数据开源项目有兴趣的同学加入「袋鼠云开源框架钉钉技术群」，交流最新开源技术信息，群号码：30537511，项目地址：https://github.com/DTStack