http://serverfault.com/questions/219739/i-o-performance-benchmarking-linux
I/O Performance Benchmarking Linux
Fri, 7 Jan 2011 08:18:00 -0500

For filesystem:
  Read test: dd if=/dev/zero of=/mount/test.dat bs=1M count=1024 oflag=direct
  Write test: dd if=/mount/test.dat of=/dev/null bs=1M count=1024 iflag=direct

  Note that even accessing a file, we are not using the buffer-cache.

For the network:
  Just test raw TCP sockets on both directions between servers.  Beware of
  the firewall blocking TCP port 5001.

  server1# dd if=/dev/zero bs=1M count=1024 | netcat 5001
  server2# netcat -l -p 5001 | dd of=/dev/null

Block size of 1M is OK to test bulk transfer speed for most modern devices.
For transaction/sec tests, use small sizes like 4k.  Change count to make a
more realistic test (I suggest long test to test sustained rate against
transitory interferences).  "odirect" flag avoids using buffer-cache, so
the test results should be repeatable.

--------------------------------------------------------------------------
dd if=/dev/zero of=test.dat bs=1M count=1024 oflag=direct
dd if=test.dat of=/dev/null bs=1M iflag=direct
dd if=/dev/zero of=test.dat bs=4K count=256K oflag=direct
dd if=test.dat of=/dev/null bs=4K iflag=direct

--------------------------------------------------------------------------
ST31500341AS drives
svr1
    % dd if=/dev/zero of=test.dat bs=1M count=1024 oflag=direct
    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB) copied, 21.1405 seconds, 50.8 MB/s

    % dd if=test.dat of=/dev/null bs=1M iflag=direct
    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB) copied, 12.2983 seconds, 87.3 MB/s

    % dd if=/dev/zero of=test.dat bs=4K count=256K oflag=direct
    262144+0 records in
    262144+0 records out
    1073741824 bytes (1.1 GB) copied, 48.0881 seconds, 22.3 MB/s

    % dd if=test.dat of=/dev/null bs=4K iflag=direct
    262144+0 records in
    262144+0 records out
    1073741824 bytes (1.1 GB) copied, 28.036 seconds, 38.3 MB/s

--------------------------------------------------------------------------
ST31500341AS drives
svr2  (rw,nodev,noatime,commit=60,regular inode setup,big partition)
    % dd if=/dev/zero of=test.dat bs=1M count=1024 oflag=direct
    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB) copied, 178.726 s, 6.0 MB/s

    % dd if=test.dat of=/dev/null bs=1M iflag=direct
    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB) copied, 13.184 s, 81.4 MB/s

    % dd if=/dev/zero of=test.dat bs=4K count=256K oflag=direct
    262144+0 records in
    262144+0 records out
    1073741824 bytes (1.1 GB) copied, 2230.14 s, 481 kB/s

    % dd if=test.dat of=/dev/null bs=4K iflag=direct
    262144+0 records in
    262144+0 records out
    1073741824 bytes (1.1 GB) copied, 35.1459 s, 30.6 MB/s

--------------------------------------------------------------------------
ST31500341AS drives
svr3  (rw,nodev,noatime,commit=60,1-inode/64k setup,smaller partition)
    % dd if=/dev/zero of=test.dat bs=1M count=1024 oflag=direct
    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB) copied, 173.661 s, 6.2 MB/s

    % dd if=test.dat of=/dev/null bs=1M iflag=direct
    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB) copied, 9.43734 s, 114 MB/s

    % dd if=/dev/zero of=test.dat bs=4K count=256K oflag=direct
    262144+0 records in
    262144+0 records out
    1073741824 bytes (1.1 GB) copied, 2209.6 s, 486 kB/s

    % dd if=test.dat of=/dev/null bs=4K iflag=direct
    262144+0 records in
    262144+0 records out
    1073741824 bytes (1.1 GB) copied, 33.5308 s, 32.0 MB/s

--------------------------------------------------------------------------
svr4  (146-Gb SCSI drive; dd does not grok "direct" option)
    % dd if=/dev/zero of=test.dat bs=1M count=1024
    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB) copied, 17.1498 s, 62.6 MB/s

    % dd if=test.dat of=/dev/null bs=1M
    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB) copied, 3.82837 s, 280 MB/s

    % dd if=/dev/zero of=test.dat bs=4K count=256K
    262144+0 records in
    262144+0 records out
    1073741824 bytes (1.1 GB) copied, 16.6598 s, 64.5 MB/s

    % dd if=test.dat of=/dev/null bs=4K
    262144+0 records in
    262144+0 records out
    1073741824 bytes (1.1 GB) copied, 4.8582 s, 221 MB/s

--------------------------------------------------------------------------
svr5  (FreeBSD, 250-Gb WD3200JB-00KFA0)
    % gdd if=/dev/zero of=test.dat bs=1M count=1024 oflag=direct
    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB) copied, 28.1326 s, 38.2 MB/s

    % gdd if=test.dat of=/dev/null bs=1M iflag=direct
    1024+0 records in
    1024+0 records out
    1073741824 bytes (1.1 GB) copied, 27.5702 s, 38.9 MB/s

    % rm test.dat
    % gdd if=/dev/zero of=test.dat bs=4K count=256K oflag=direct
    262144+0 records in
    262144+0 records out
    1073741824 bytes (1.1 GB) copied, 164.953 s, 6.5 MB/s

    % gdd if=test.dat of=/dev/null bs=4K iflag=direct
    262144+0 records in
    262144+0 records out
    1073741824 bytes (1.1 GB) copied, 92.0367 s, 11.7 MB/s