Apache Hadoop with single Fedora node

Today when I tried to start Jobtracker for Apache Hadoop 1.2.1, I kept getting the following ENOENT error
FATAL org.apache.hadoop.mapred.JobTracker: ENOENT: No such file or directory
        at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
        at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
        at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
        at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
        at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
        at org.apache.hadoop.mapred.CompletedJobStatusStore.(CompletedJobStatusStore.java:81)
        at org.apache.hadoop.mapred.JobTracker.initialize(JobTracker.java:2051)
        at org.apache.hadoop.mapred.JobTracker.offerService(JobTracker.java:2328)
        at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792)

I realized the most important error is the bold one, which is related to logging as specified in mapred-site.xml

mapred.job.tracker.persist.jobstatus.dir
file:////var/log/hadoop/mr/jobstatus

After the the correct folder permissions are set, it should look like following
$ ll /var/log/hadoop/
drwxr-xr-x. 2 hdfs   hadoop 4096 Jul 22 22:45 hdfs
drwxr-xr-x. 4 mapred hadoop 4096 Jul 22 23:10 mapred
drwxr-xr-x. 2 mapred hadoop 4096 Jul 22 20:14 mr
After all the services are running, I can list them by
#jps
22896 DataNode
26189 TaskTracker
22225 NameNode
26502 Jps
26042 JobTracker
And I can list the listening ports of the services by
# lsof -i -P | egrep -v "ESTABLISHED" | grep java
java      22225   hdfs   66u  IPv4 573305      0t0  TCP localhost:8020 (LISTEN)
java      22225   hdfs   79u  IPv4 573311      0t0  TCP localhost:50070 (LISTEN)
java      22896   hdfs   63u  IPv4 575844      0t0  TCP *:50010 (LISTEN)
java      22896   hdfs   65u  IPv4 576654      0t0  TCP *:50075 (LISTEN)
java      22896   hdfs   73u  IPv4 575856      0t0  TCP *:50020 (LISTEN)
java      26042 mapred   64u  IPv4 589571      0t0  TCP localhost:9000 (LISTEN)
java      26042 mapred   75u  IPv4 589575      0t0  TCP localhost:50030 (LISTEN)
java      26189 mapred   61u  IPv4 590005      0t0  TCP localhost:40660 (LISTEN)
java      26189 mapred   79u  IPv4 590177      0t0  TCP *:50060 (LISTEN)

Fedora dracut in chroot environment

The following is the procedure to create new initframfs when the root filesystem has structural changes such as between LVM, MD or plain. 

• Chroot to the desired folder (.e.g. /mnt/temp)
  chroot /mnt/temp
• mount the system partitions /proc, /run, /dev, /sys
  mount -t proc proc /proc
mount -t tmpfs tmpfs /run
mount -t devtmpfs devtmpfs /dev
mount -t sysfs sysfs /sys
• mount the /boot and / partition
  mount -t ext4 /dev/sdb1 /boot
mount -t ext4 /dev/md0 /
• Create the udev/data folder to emulate the udev database
  cd /run
mkdir udev/data
• To include mdadm in Fedora dracut: add module mdraid found in /usr/lib/dracut/modules.d/
  dracut -v -f -a mdraid --mdadmconf /boot/initramfs-$(uname -r).img $(uname -r)

How to make Samba 4 shares writable with Fedora 19 and Selinux

Today I found I couldn't create new folders in my samba 4 share under Fedora 19. And this was due to the settings of Selinux since there wasn't any problem without enabling Selinux.

The status of the Selinux boolean values that are relevant to Samba can be queried via the following command:

#getsebool -a | grep samba
#getsebool -a | grep smbd
Since we need to allow samba to modify shared files, the files/directories must be labeled public_content_rw_t and we must turn on the smbd_anon_write boolean.

• Labeling all files under the share folder (e.g. /mnt/media) to public_content_rw_t so that they can be modified by multiple domains (Apache, FTP, rsync, Samba).


#semanage fcontext -a -t public_content_rw_t "/mnt/media(/.*)?"
#restorecon -F -R /mnt/media
Note: File context can be temporarily modified with the chcon command. However, that won't survive a relabeling. We need to permanently change the file context by using the semanage fcontext command. This will modify the SELinux labeling database. It is necessary to use restorecon to apply the labels.

• Turn on the smbd_anon_write boolean:


#setsebool -P smbd_anon_write 1

• It's NOT necessary to turn on samba_export_all_rw if the shared files/directories are properly labeled and the smbd_anon_write boolean is turned on.
• The security context of the files can be verified via
  # ls -Z /mnt/media
drwxrwxr-x. video media system_u:object_r:public_content_rw_t:s0 video