Monday, July 22, 2013

Apache Hadoop with single Fedora node

Today when I tried to start Jobtracker for Apache Hadoop 1.2.1, I kept getting the following ENOENT error
FATAL org.apache.hadoop.mapred.JobTracker: ENOENT: No such file or directory
        at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
        at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
        at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
        at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
        at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
        at org.apache.hadoop.mapred.CompletedJobStatusStore.(CompletedJobStatusStore.java:81)
        at org.apache.hadoop.mapred.JobTracker.initialize(JobTracker.java:2051)
        at org.apache.hadoop.mapred.JobTracker.offerService(JobTracker.java:2328)
        at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792)

I realized the most important error is the bold one, which is related to logging as specified in mapred-site.xml
mapred.job.tracker.persist.jobstatus.dir
file:////var/log/hadoop/mr/jobstatus


After the the correct folder permissions are set, it should look like following
$ ll /var/log/hadoop/
drwxr-xr-x. 2 hdfs   hadoop 4096 Jul 22 22:45 hdfs
drwxr-xr-x. 4 mapred hadoop 4096 Jul 22 23:10 mapred
drwxr-xr-x. 2 mapred hadoop 4096 Jul 22 20:14 mr
After all the services are running, I can list them by
#jps
22896 DataNode
26189 TaskTracker
22225 NameNode
26502 Jps
26042 JobTracker
And I can list the listening ports of the services by
# lsof -i -P | egrep -v "ESTABLISHED" | grep java
java      22225   hdfs   66u  IPv4 573305      0t0  TCP localhost:8020 (LISTEN)
java      22225   hdfs   79u  IPv4 573311      0t0  TCP localhost:50070 (LISTEN)
java      22896   hdfs   63u  IPv4 575844      0t0  TCP *:50010 (LISTEN)
java      22896   hdfs   65u  IPv4 576654      0t0  TCP *:50075 (LISTEN)
java      22896   hdfs   73u  IPv4 575856      0t0  TCP *:50020 (LISTEN)
java      26042 mapred   64u  IPv4 589571      0t0  TCP localhost:9000 (LISTEN)
java      26042 mapred   75u  IPv4 589575      0t0  TCP localhost:50030 (LISTEN)
java      26189 mapred   61u  IPv4 590005      0t0  TCP localhost:40660 (LISTEN)
java      26189 mapred   79u  IPv4 590177      0t0  TCP *:50060 (LISTEN)



No comments: