hadoop 설치 (hdfs)

 

개발환경 구축을 위해서 hadoop 1.0.3 설치

<hadoop 설치>
# http://www.apache.org/dyn/closer.cgi/hadoop/common/ 에 접속해서 curl이나 wget으로 hadoop 1.0.3을 다운로드

 

<hdfs 지정>

$ mkdir -p /home/www/hadoop/fs/name /home/www/hadoop/fs/data /home/www/hadoop/mapred/system
$ chmod 755 /home/www/hadoop/fs/data

 

<설정 파일 >

$ vi conf/core-site.xml

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>

$ vi conf/hdfs-site.xml

<configuration>

<property>
<name>dfs.name.dir</name>
<value>/home/www/hadoop/fs/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/home/www/hadoop/fs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>

</configuration>

$ vi conf/mapred-site.xml

<configuration>

<property>
<name>mapred.job.tracker</name>
<value>localhost:9001</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>/home/www/hadoop/mapred/system</value>
</property>

</configuration>

$ vi conf/hadoop-env.sh

export JAVA_HOME=/usr/j2se/

 

<마스터로 root 로 접속>
$ vi  /etc/ssh/sshd_config
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile      .ssh/authorized_keys

sshd 데몬을 재시작

$ service sshd restart

 

<www로 접속>

암호 없이 연결하기

$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/www/.ssh/id_rsa):
/home/www/.ssh/id_rsa already exists.
Overwrite (y/n)? y
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/www/.ssh/id_rsa.
Your public key has been saved in /home/www/.ssh/id_rsa.pub.
The key fingerprint is:
bd:cb:86:99:00:6f:b3:66:06:76:17:0e:6e:21:93:2b www@VNMSGBE1508

 

$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

 

 

<hadoop 실행>
$ bin/hadoop namenode -format

$ bin/start-all.sh

또는

$ bin/start-dfs.sh