hadoop+spark+scala环境--单实例版-创新互联

hadoop+spark环境--单实例版
1、修改主机名及关系映射
2、关闭防火墙并创建文件夹
mkdir /hadoop/tmp
mkdir /hadoop/dfs/name
mkdir /hadoop/dfs/data
mkdir /hadoop/var
3、配置Scala环境
[root@hadoop conf]#vim /etc/profile
export SCALA_HOME=/opt/scala2.11.12
export PATH=.:${JAVA_HOME}/bin:${SCALA_HOME}/bin:$PATH
[root@hadoop conf]#scala -version //查看是否安装成功
4、配置Spark环境/usr/java/jdk1.8.0_201-amd64
[root@hadoop conf]# vim /etc/profile
export SPARK_HOME=/opt/spark2.2.3
export PATH=.:${JAVA_HOME}/bin:${SCALA_HOME}/bin:${SPARK_HOME}/bin:$PATH
[root@hadoop conf]#vim /opt/spark2.2.3/conf/spark-env.sh
export SCALA_HOME=/opt/scala2.11.12
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/
export HADOOP_HOME=/opt/hadoop2.7.6
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SPARK_HOME=/opt/spark2.2.3
export SPARK_MASTER_IP=hadoop.master //主机名称
export SPARK_EXECUTOR_MEMORY=1G //设置运行内存
5、配置Hadoop环境
[root@hadoop hadoop2.7.6]# vim /etc/profile
export HADOOP_HOME=/opt/hadoop2.7.6
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
export PATH=${HADOOP_HOME}/bin
5.1 修改cor-site.xml文件
[root@hadoop hadoop]# vim core-site.xml

hadoop.tmp.dir
/hadoop/tmpAbase for other temporary directories.


fs.default.name
hdfs://hadoop.master:9000

5.2 修改hadoop-env.sh文件/usr/java/jdk1.8.0_201-amd64
[root@hadoop hadoop]# vim hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/
5.3 修改hdfs-site.xml文件
[root@hadoop hadoop]# vim hdfs-site.xml

dfs.name.dir
/hadoop/dfs/name
Path on the local filesystem where theNameNode stores the namespace and transactions logs persistently.


dfs.data.dir
/hadoop/dfs/data
Comma separated list of paths on the localfilesystem of a DataNode where it should store its blocks.


dfs.replication
2


dfs.permissions
false
need not permissions

5.4 修改mapred-site.xml文件
[root@hadoop hadoop]# vim mapred-site.xml

mapred.job.tracker
hadoop.master:9001


mapred.local.dir
/hadoop/var


mapreduce.framework.name
yarn

6、 启动Hadoop
cd /opt/hadoop2.7.6/bin
./hadoop namenode -format
cd /opt/hadoop2.7.6/sbin
启动hdfs和yarn
start-dfs.sh
start-yarn.sh
查看验证正常
在浏览器输入192.168.47.45:8088 和192.168.47.45:50070 界面查看是否能访问
7、 启动Spark
cd /opt/spark2.2.3/sbin
start-all.sh
http://192.168.47.45:8080/

另外有需要云服务器可以了解下创新互联scvps.cn，海内外云服务器15元起步，三天无理由+7*72小时售后在线，公司持有idc许可证，提供“云服务器、裸金属服务器、高防服务器、香港服务器、美国服务器、虚拟主机、免备案服务器”等云主机租用服务以及企业上云的综合解决方案，具有“安全稳定、简单易用、服务可用性高、性价比高”等特点与优势，专为企业上云打造定制，能够满足用户丰富、多元化的应用场景需求。