Hadoop 설치 및 설정

Question

CentOS에서 자체적으로 hadoop 클러스터 환경을 설정하고 설치하려고합니다. TaskTracker가 실행되고 있지 않습니다. start-all.sh를 실행 한 후에 사라집니다.

누군가가 단일 노드/다중 노드 클러스터를 설치하는 방법을 도울 수 있습니까?

Answer 1

Can you try this and let me know if you face any issues while installing the hadooop. 


Pre-requiste 
1. Ensure Hosts file – changes (master, slave) – no loopback , no local host 
2. Ensure Sshd runs 
3. Ensure hadoop Sudo ers – No passwd authentication 
4. Ensure selinux is disabled (not permissive) for user, policies and contexts 
5. Ensure Reboot - Pings masters and slaves. 
6. Ensure JDK 7 is installed in master and slaves. 
7. Ensure firewall is off or iptables allows ports mentioned by hadoop 
8. Ensure IPV6 is disabled or export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true in hadoop_env.sh 

In Master for seamless cut to slaves 

Generate SHH - password less access 
ssh-keygen -t rsa 
$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@master 
$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@slave 
mkdir /opt/hadoop 
# cd /opt/hadoop/ 
gunzip and untar the hadoop ball under /opt/hadoop/hadoop 

Ensure hadoop owns the /opt directory 
Changes in xml files under /opt/hadoop/hadoop/conf 
core-site.xml 
#Add the following inside the configuration tag 
<property> 
<name>fs.default.name</name> 
<value>hdfs://master:9000/</value> 
</property> 
<property> 
<name>dfs.permissions</name> 
<value>false</value> 
</property> 







hdfs-site.xml 
# Add the following inside the configuration tag 
<property> 
<name>dfs.data.dir</name> 
<value>/opt/hadoop/hadoop/dfs/name/data</value> 
<final>true</final> 
</property> 
<property> 
<name>dfs.name.dir</name> 
<value>/opt/hadoop/hadoop/dfs/name</value> 
<final>true</final> 
</property> 
<property> 
<name>dfs.replication</name> 
<value>1</value> 
</property> 
mapred-site.xml 
<property> 
<name>mapred.job.tracker</name> 
<value>hadoop-master:9001</value> 
</property> 
conf/hadoop-env.sh 
export JAVA_HOME=/opt/jdk1.7.0_45 
export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true 
export HADOOP_CONF_DIR=/opt/hadoop/hadoop/conf 
Ship this cut to slaves ------------------------------> 

In Master cut only (under conf) 
$ cat masters 
hadoop-master 
$ cat slaves`enter code here` 
Slave 
$ bin/hadoop namenode –format 
$ bin/start-dfs.sh`enter code here` 
$ bin/start-mapred.sh