So a while ago I bought a new motherboard. It took me a while to make a choice because I wanted it to be cheap, while still having at least 6 sata connections, pata too and then be able to use ECC type memory. Having ZFS is great to keep your data correctly on your disks but if the data gets corrupted in memory you're still lost, hence the ECC.
I finally choose the asus M4A88TDM-EVO because it had all the specs I wanted AND included usb3. Great I thought, just what I need... this is future proof. Too bad it turned out to be just a separate pci-e 1x card. But it's there and after compiling a kernel module for it (usb3 requires the xhci module, or xhci_hcd on gentoo) it worked.
Kinda.
It's very spammy in dmesg. Lots of debug info it seems. Or at least info I don't need. This is kernel "Linux server 2.6.36-gentoo-r8 #2 SMP Tue Apr 5 08:18:52 CEST 2011 i686 AMD Athlon(tm) II X2 250 Processor AuthenticAMD GNU/Linux" mind you, so newer kernels may no have this.
Time to test this puppy out. Since the theoretical limit of usb2 is 60MByte/sec there may be a change that my disks are sometimes hitting the limit of the bus, instead of the speed limit of the platters. Since usb3 is 10x as fast, the bus will never be the bottleneck.
So would this really matter, in real life?
I used a Hitachi desktstar 7k1000 HDT721075SLA380 to test with. It's supposed to be able to read at speeds between 86MB/sec (outer disk section) and 46MB/sec (inner sections). With the 60MB max speed of usb2 there is something to gain here. The disk is connected using a LaCie usb-sata external enclosure.
The usb3 controller is this one:
02:00.0 USB Controller: NEC Corporation Device 0194 (rev 03)
I performed some tests, first on usb2. Please don't mind the changing drive letters, every time I unplug and re-plug the device it may take up a new drive letter. It remains the same drive.
So theoretically this disk should be able to transfer 1.3GB/sec from its cache to the cpu. When reading from the platter this drops to 30MB/sec. It's an old disk ;-) Hdparm is a bit of a synthetic test, so what does a file transfer do for us:
write:
read :
read while taking the filesystem out of the loop:
then on usb3:
that's a small improvement to read from the platters. Interestingly the cache read is slightly slower
write:
write zero's
server maxx # dd count=9000000 if=/dev/zero of=/mnt/usb-backup-disk-750g-20081001/bigfile
9000000+0 records in
9000000+0 records out
4608000000 bytes (4.6 GB) copied, 122.208 s, 37.7 MB/s
read:
read from disk without fs:
server maxx # dd count=9000000 if=/dev/sdk of=/dev/null
9000000+0 records in
9000000+0 records out
4608000000 bytes (4.6 GB) copied, 122.362 s, 37.7 MB/s
Next up was bonnie. When running the usb3 test my machine catched a kernel panic! Odd.. I reset the machine and run bonnie again. No crash this time. This was a trigger for me to implement
`bonnie++ -d /mnt/usb-backup-disk-750g-20081001/tmp -s 6576 -c 4 -m usb220110925 -x 3 -u 1000:100`
the result:
I finally choose the asus M4A88TDM-EVO because it had all the specs I wanted AND included usb3. Great I thought, just what I need... this is future proof. Too bad it turned out to be just a separate pci-e 1x card. But it's there and after compiling a kernel module for it (usb3 requires the xhci module, or xhci_hcd on gentoo) it worked.
Kinda.
It's very spammy in dmesg. Lots of debug info it seems. Or at least info I don't need. This is kernel "Linux server 2.6.36-gentoo-r8 #2 SMP Tue Apr 5 08:18:52 CEST 2011 i686 AMD Athlon(tm) II X2 250 Processor AuthenticAMD GNU/Linux" mind you, so newer kernels may no have this.
Time to test this puppy out. Since the theoretical limit of usb2 is 60MByte/sec there may be a change that my disks are sometimes hitting the limit of the bus, instead of the speed limit of the platters. Since usb3 is 10x as fast, the bus will never be the bottleneck.
So would this really matter, in real life?
I used a Hitachi desktstar 7k1000 HDT721075SLA380 to test with. It's supposed to be able to read at speeds between 86MB/sec (outer disk section) and 46MB/sec (inner sections). With the 60MB max speed of usb2 there is something to gain here. The disk is connected using a LaCie usb-sata external enclosure.
The usb3 controller is this one:
02:00.0 USB Controller: NEC Corporation Device 0194 (rev 03)
I performed some tests, first on usb2. Please don't mind the changing drive letters, every time I unplug and re-plug the device it may take up a new drive letter. It remains the same drive.
server ~ # hdparm -tT /dev/sdl
/dev/sdl:
Timing cached reads: 2644 MB in 2.00 seconds = 1322.00 MB/sec
Timing buffered disk reads: 92 MB in 3.06 seconds = 30.03 MB/sec
server ~ #
So theoretically this disk should be able to transfer 1.3GB/sec from its cache to the cpu. When reading from the platter this drops to 30MB/sec. It's an old disk ;-) Hdparm is a bit of a synthetic test, so what does a file transfer do for us:
write:
server maxx # dd if=bigfile of=/mnt/usb-backup-disk-750g-20081001/bigfile
9170564+1 records in
9170564+1 records out
4695328969 bytes (4.7 GB) copied, 211.23 s, 22.2 MB/s
server linux # dd count=9000000 if=/dev/zero of=/mnt/usb-backup-disk-750g-20081001/bigfile
9000000+0 records in
9000000+0 records out
4608000000 bytes (4.6 GB) copied, 133.498 s, 34.5 MB/s
server linux #
read :
server linux # dd if=/mnt/usb-backup-disk-750g-20081001/bigfile of=/dev/null
9170564+1 records in
9170564+1 records out
4695328969 bytes (4.7 GB) copied, 138.333 s, 33.9 MB/s
read while taking the filesystem out of the loop:
server linux # dd count=10000000 if=/dev/sdk of=/dev/null
10000000+0 records in
10000000+0 records out
5120000000 bytes (5.1 GB) copied, 155.335 s, 33.0 MB/s
[odd, that's actually slower then when reading from the fs... ]then on usb3:
server ~ # hdparm -tT /dev/sdn
/dev/sdn:
Timing cached reads: 2434 MB in 2.00 seconds = 1216.78 MB/sec
Timing buffered disk reads: 104 MB in 3.00 seconds = 34.63 MB/sec
server ~ #
that's a small improvement to read from the platters. Interestingly the cache read is slightly slower
write:
server maxx # dd if=bigfile of=/mnt/usb-backup-disk-750g-20081001/bigfile
mount
9170564+1 records in
9170564+1 records out
4695328969 bytes (4.7 GB) copied, 123.346 s, 38.1 MB/s
write zero's
server maxx # dd count=9000000 if=/dev/zero of=/mnt/usb-backup-disk-750g-20081001/bigfile
9000000+0 records in
9000000+0 records out
4608000000 bytes (4.6 GB) copied, 122.208 s, 37.7 MB/s
read:
server maxx # dd if=/mnt/usb-backup-disk-750g-20081001/bigfile of=/dev/null
9170564+1 records in
9170564+1 records out
4695328969 bytes (4.7 GB) copied, 119.702 s, 39.2 MB/s
read from disk without fs:
server maxx # dd count=9000000 if=/dev/sdk of=/dev/null
9000000+0 records in
9000000+0 records out
4608000000 bytes (4.6 GB) copied, 122.362 s, 37.7 MB/s
Next up was bonnie. When running the usb3 test my machine catched a kernel panic! Odd.. I reset the machine and run bonnie again. No crash this time. This was a trigger for me to implement
`bonnie++ -d /mnt/usb-backup-disk-750g-20081001/tmp -s 6576 -c 4 -m usb220110925 -x 3 -u 1000:100`
the result:
| Version 1.96 | Sequential Output | Sequential Input | Random Seeks | Sequential Create | Random Create | ||||||||||||||||||||||
| Concurrency | Size | Per Char | Block | Rewrite | Per Char | Block | Num Files | Create | Read | Delete | Create | Read | Delete | ||||||||||||||
| K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | ||||
| usb2 | 4 | 6576M | 622 | 93 | 31604 | 7 | 14249 | 5 | 1215 | 90 | 34417 | 6 | 172.1 | 8 | 16 | 10578 | 46 | +++++ | +++ | 31734 | 50 | 31396 | 34 | +++++ | +++ | +++++ | +++ |
| Latency | 54816us | 599ms | 633ms | 34401us | 47854us | 646ms | Latency | 3091us | 2151us | 2125us | 17656us | 150us | 6314us | ||||||||||||||
| usb3 | 4 | 6576M | 615 | 95 | 32326 | 7 | 14434 | 5 | 1579 | 87 | 34827 | 5 | 172.8 | 8 | 16 | 19709 | 42 | +++++ | +++ | +++++ | +++ | 32034 | 36 | +++++ | +++ | +++++ | +++ |
| Latency | 44231us | 673ms | 643ms | 41676us | 64893us | 655ms | Latency | 6150us | 530us | 806us | 4056us | 283us | 2863us | ||||||||||||||
| usb2 | 4 | 6576M | 679 | 96 | 32972 | 7 | 14547 | 5 | 1490 | 94 | 34936 | 6 | 172.9 | 8 | 16 | 10681 | 46 | +++++ | +++ | 20249 | 61 | 15184 | 65 | +++++ | +++ | 20964 | 62 |
| Latency | 29018us | 438ms | 649ms | 31497us | 30470us | 587ms | Latency | 2858us | 2295us | 3218us | 3825us | 5738us | 2266us | ||||||||||||||
| usb3 | 4 | 6576M | 583 | 94 | 36709 | 8 | 17449 | 6 | 1025 | 99 | 43703 | 8 | 195.4 | 10 | 16 | 13722 | 56 | +++++ | +++ | +++++ | +++ | 18304 | 71 | +++++ | +++ | +++++ | +++ |
| Latency | 38191us | 519ms | 538ms | 27632us | 28568us | 539ms | Latency | 547us | 509us | 706us | 211us | 15us | 36us | ||||||||||||||
| usb3 | 4 | 6576M | 645 | 98 | 36868 | 8 | 17472 | 6 | 1971 | 98 | 43596 | 8 | 198.3 | 10 | 16 | 14829 | 60 | +++++ | +++ | +++++ | +++ | +++++ | +++ | +++++ | +++ | +++++ | +++ |
| Latency | 40016us | 520ms | 584ms | 24271us | 27662us | 474ms | Latency | 558us | 509us | 555us | 84us | 8us | 32us | ||||||||||||||
| usb3 | 4 | 6576M | 782 | 97 | 36838 | 8 | 17439 | 6 | 1490 | 98 | 43479 | 8 | 196.4 | 9 | 16 | 22927 | 33 | +++++ | +++ | +++++ | +++ | +++++ | +++ | +++++ | +++ | +++++ | +++ |
| Latency | 10200us | 583ms | 562ms | 28055us | 37441us | 538ms | Latency | 434us | 514us | 559us | 87us | 8us | 35us | ||||||||||||||
The above results are rather clear. The usb3 controller is resulting in better performance according to bonnie too. Just to be sure I re-ran the test and the results were slightly less pronounced.
In conclusion:
Too bad this usb controller only has 2 connections. I wonder how much performance remains when I start daisy-chaining multiple devices on those connectors...
Geen opmerkingen:
Een reactie posten