Saturday, October 20, 2012

Increase/grow the capacity/space of a linux Raid 1 partition without losing data

Today I found I needed to increase the capacity of my data partition (on a Raid 1 array) to fit a hard drive upgrade. Below are a couple of good references on how that can be done and some arguments to mdadm need to be adjusted to work for me
http://www.flyonthenet.it/blog/?p=71
http://johnlange.wordpress.com/tech-tips/linux/expanding-linux-software-raid1-with-an-ext3-file-system/
The safe and lengthy procedure is as follows (assuming the old array consists of drive A and B and the new array with increased capacity consists of drive C and D):
  1. Back up all the data on the Raid 1 array (e.g. /dev/md2) and umount the array from the system
  2. Remove the partition on one old drive (A) from the Raid 1 array that needs to grow capacity
  3. Partition a new hard drive (C) as Linux raid autodetect/fd type with the wanted capacity and add that partition to the Raid 1 array. Mdadm will start recovery/resync of the array.
  4. Let mdadm finish resyncing the array and the array is now clean again (very important, otherwise data loss). Make sure the array status has [UU]
    cat /proc/mdstat
    md0 : active raid1 sdb1[1] sda1[0]
    76798592 blocks [2/2] [UU]
  5. Remove the partition on the remaining old drive (B) from the array.
  6. Partition the second new hard drive (D) as Linux raid autodetect/fd type with the wanted capacity and add that partition to the Raid 1 array. Mdadm will start recovery/resync of the array.
  7. After resyncing is complete, the underlying block device of the array is now of an appropriate size to hold the file system with increased capacity. Grow the capacity of the array and let the resyncing finishmdadm --grow /dev/md2 --size=max
  8. Resize the ext file system
    e2fsck -f /dev/md2
    resize2fs /dev/md2
    e2fsck -f /dev/md2
  9. Mount the array


    Alternatively, a quick and dirty solution to grow the capacity of the array (metadata version 0.90) on the same pair of drives (A&B) is shown below
  • Back up all the data on the Raid 1 array (e.g. /dev/md2) and umount the array from the system
  • Increase the size of the partitions of the raid devices on the existing hard drives (A&B)
  • Create new raid array with mdadm (note the metadata argument to specify the same metadata as in the previous array is critical. Otherwise e2fsck will not find the superblock since latest mdadm uses metadata version 1.2 as default)
    mdadm --create /dev/md2 --level=raid1 --raid-devices=2 /dev/sdc1 /dev/sdb1 --metadata 0.90
    If mdadm prompts that the partitions already contain an ext2fs file system, hit "y" to continue.
    Continue creating array? y
    mdadm: array /dev/md2 started.
  • Check the file system on the array and fix the errors. Then resize the file system.
    e2fsck -f /dev/md2
    resize2fs /dev/md2
    e2fsck -f /dev/md2
  • Mount the array

Saturday, August 4, 2012

Disk benchmarks


  1. Ramdisk (FAT32) created by DATARAM RAMDisk v3.5.130R24 (freeware) under windows 7 64-bit
  2. SSD Crucial M4-CT064M4SSD2 Firmware 0309 with SATA II 3Gb/s connection
  3. SSD

Tuesday, June 19, 2012

Use ImageMagick to create GIF file

convert -resize 50% -delay 50 -loop 0 IMG*.JPG animation.gif
convert animation.gif -crop 1152x768+0+150 crop.gif
Reduce image size: convert animation.gif -resize 100x50 small.gif

Sunday, June 3, 2012

Use FFMPEG to process video files


  • Rotate video files
  1. Download latest FFMPEG windows build from here
  2. Move all video files to be rotated (e.g. MOV files) into a subfolder. 
  3. Execute the command below from within a CMD script file (it must be executed from within a CMD script file since it will fail if copy and paste on the command line)
    FOR /F "tokens=*" %%G IN ('dir /b *.MOV') DO ffmpeg -i %%G -vf "transpose=1" -qscale 0 -s 608x1080 -y %%G.rotated.MOV
    All videos will rotate 90 degree clockwise (transpose=1). Change it to transpose=2 for counterclockwise. Height of the videos is reduced to 1080 since some media players have problem playing back files whose height is larger than 1080. Output files will append .rotated.MOV to the name of input files.
    • Set the dimension and frame rate of output file
      Command option -r set frame rate in frames/second, -s set dimension to be widthxheight
      ffmpeg -i input.mp4 -qscale 0 -r 30 -s 720x1080 -y output.mp4
    • Slow down a video
      • Copy the video to a raw bitstream format
        ffmpeg -i input.mp4 -map 0:v -c:v copy -bsf:v h264_mp4toannexb raw.h264
      • Generate new timestamps while muxing to a container
        ffmpeg -fflags +genpts -r 19.25 -i raw.h264 -c:v copy output.mp4
    • Trim video file
    1. Extract the video stream and discard the audio stream: -c copy -an option
      ffmpeg  -i input.mp4 -c copy -an noaudio.mp4
    2. Trim time: Today I have a need to trim the first 8 seconds off a mp4 video file taken by my cell phone. I found I could use -ss option to specify the start time and -t option to specify the duration of the trimmed video (below 12 seconds) and re-encode the video: 
      • ffmpeg -i input.mp4 -ss 00:00:08 -t 00:00:12 output.mp4
      • If the encoding is slow, -c copy option can skip the encoding but could generate an output file than can not seek properly
    3. Trim size: can be done with the crop filter
      • ffmpeg -i input.mp4 -filter:v "crop=out_w:out_h:x:y" output.mp4
        the arguments are as follows:
        • out_w is the width of the output rectangle
        • out_h is the height of the output rectangle
        • x and y specify the top left corner of the output rectangle relative to the top left corner of the input video
    • Convert video to GIF file
      • Generate individual frames from the video file
        mkdir frames
        ffmpeg -i out.mp4 -r 10 frames/ffout%03d.png
      • Remove unwanted frames
      • Convert frames to GIF file using ImageMagick
        convert -delay 0.25 -loop 0 frames/ffout*.png output.gif
    • Concatenate multiple video files into one
      Utilize the concat filter to concatenate multiple video files into one
      ffmpeg -f concat -i inputs.txt -c copy -y output.mp4
      The file inputs.txt should list all the video files to be concatenated (one file per line) and its contents look like the following:
      file 1.mp4
      file 2.mp4
      file 3.mp4
      file 4.mp4
    • Mix video and audio files
      I found a quick way to mix an audio mp3 file and a video/audio mp4 file by using the -map argument with ffmpeg:
      ffmpeg -i input0.mp4 -i input1.mp3 -map 0:v:0 -map 1:a:0 -c copy -shortest out output.mp4
      -map 0:v:0 for (0) input file zero, (v) video streams of the input file, (0) stream zero (first) of the video streams
      -map 1:a:0 for (0) input file one, (a) audio streams of the input file,  (0) stream zero (first) of the audio streams
      -c copy copy the desired streams to the output file instead of encoding them to maintain the same quality
      -shortest designate the duration of the output file to be the same as the shortest one among input streams

    Monday, March 26, 2012

    Using multiple Google Voice (GV) numbers in one android phone

    I found it practical to use at least two GV numbers in one android phone with reasonable usability. I had purchased the Groove IP app for my primary GV number since it was proven to be the best solution for using GV with android phones. Then I installed SIPdroid and created an PBxes.org account for using a second GV account with PBxes.org. I then switched to csipsimple for this GV number since it can be integrated into the native dialer. In theory, I could run SIPdroid to register a third GV account then I can have three GV numbers to work in one cell phone but I haven't tested that yet.

    I did found one issue initially: csipsimple no outgoing calls
    I updated csipsimple to the latest version but that didn't help: it can receive incoming calls without any problems but can not make outgoing calls with its dialer. It typically drops calls in one second after the outgoing calls are dialed. I thought Groove IP and csipsimple could share the native dialer but apparently that is not the case. I needed to uncheck integration csipimple with android in its settings to make outgoing calls working again. Otherwise it would try to use the native dialer for outgoing calls and that was default to use Groove IP in my native dialer and that's why it failed.