real world usb3 disk performance

So a while ago I bought a new motherboard. It took me a while to make a choice because I wanted it to be cheap, while still having at least 6 sata connections, pata too and then be able to use ECC type memory. Having ZFS is great to keep your data correctly on your disks but if the data gets corrupted in memory you're still lost, hence the ECC.
I finally choose the asus M4A88TDM-EVO because it had all the specs I wanted AND included usb3. Great I thought, just what I need... this is future proof. Too bad it turned out to be just a separate pci-e 1x card. But it's there and after compiling a kernel module for it (usb3 requires the xhci module, or xhci_hcd on gentoo) it worked.

Kinda.

It's very spammy in dmesg. Lots of debug info it seems. Or at least info I don't need. This is kernel "Linux server 2.6.36-gentoo-r8 #2 SMP Tue Apr 5 08:18:52 CEST 2011 i686 AMD Athlon(tm) II X2 250 Processor AuthenticAMD GNU/Linux" mind you, so newer kernels may no have this.

Time to test this puppy out. Since the theoretical limit of usb2 is 60MByte/sec there may be a change that my disks are sometimes hitting the limit of the bus, instead of the speed limit of the platters. Since usb3 is 10x as fast, the bus will never be the bottleneck.
So would this really matter, in real life?

I used a Hitachi desktstar 7k1000 HDT721075SLA380 to test with. It's supposed to be able to read at speeds between 86MB/sec (outer disk section) and 46MB/sec (inner sections). With the 60MB max speed of usb2 there is something to gain here. The disk is connected using a LaCie usb-sata external enclosure.
The usb3 controller is this one:
02:00.0 USB Controller: NEC Corporation Device 0194 (rev 03)

I performed some tests, first on usb2. Please don't mind the changing drive letters, every time I unplug and re-plug the device it may take up a new drive letter. It remains the same drive.

server ~ # hdparm -tT /dev/sdl

/dev/sdl:

 Timing cached reads:   2644 MB in  2.00 seconds = 1322.00 MB/sec

 Timing buffered disk reads:  92 MB in  3.06 seconds =  30.03 MB/sec

server ~ #

So theoretically this disk should be able to transfer 1.3GB/sec from its cache to the cpu. When reading from the platter this drops to 30MB/sec. It's an old disk ;-) Hdparm is a bit of a synthetic test, so what does a file transfer do for us:

write:

server maxx # dd if=bigfile of=/mnt/usb-backup-disk-750g-20081001/bigfile

9170564+1 records in

9170564+1 records out

4695328969 bytes (4.7 GB) copied, 211.23 s, 22.2 MB/s

server linux # dd count=9000000 if=/dev/zero of=/mnt/usb-backup-disk-750g-20081001/bigfile

9000000+0 records in

9000000+0 records out

4608000000 bytes (4.6 GB) copied, 133.498 s, 34.5 MB/s

server linux #

read :

server linux # dd if=/mnt/usb-backup-disk-750g-20081001/bigfile of=/dev/null

9170564+1 records in

9170564+1 records out

4695328969 bytes (4.7 GB) copied, 138.333 s, 33.9 MB/s

read while taking the filesystem out of the loop:

server linux # dd count=10000000 if=/dev/sdk of=/dev/null

10000000+0 records in

10000000+0 records out

5120000000 bytes (5.1 GB) copied, 155.335 s, 33.0 MB/s

[odd, that's actually slower then when reading from the fs... ]


then on usb3:

server ~ # hdparm -tT /dev/sdn

/dev/sdn:

 Timing cached reads:   2434 MB in  2.00 seconds = 1216.78 MB/sec

 Timing buffered disk reads: 104 MB in  3.00 seconds =  34.63 MB/sec

server ~ #

that's a small improvement to read from the platters. Interestingly the cache read is slightly slower

write:

server maxx # dd if=bigfile of=/mnt/usb-backup-disk-750g-20081001/bigfile

mount

9170564+1 records in

9170564+1 records out

4695328969 bytes (4.7 GB) copied, 123.346 s, 38.1 MB/s

write zero's
server maxx # dd count=9000000 if=/dev/zero of=/mnt/usb-backup-disk-750g-20081001/bigfile
9000000+0 records in
9000000+0 records out
4608000000 bytes (4.6 GB) copied, 122.208 s, 37.7 MB/s

read:

server maxx # dd if=/mnt/usb-backup-disk-750g-20081001/bigfile of=/dev/null

9170564+1 records in

9170564+1 records out

4695328969 bytes (4.7 GB) copied, 119.702 s, 39.2 MB/s

read from disk without fs:
server maxx # dd count=9000000 if=/dev/sdk of=/dev/null
9000000+0 records in
9000000+0 records out
4608000000 bytes (4.6 GB) copied, 122.362 s, 37.7 MB/s

Next up was bonnie. When running the usb3 test my machine catched a kernel panic! Odd.. I reset the machine and run bonnie again. No crash this time. This was a trigger for me to implement
`bonnie++ -d /mnt/usb-backup-disk-750g-20081001/tmp -s 6576 -c 4 -m usb220110925 -x 3 -u 1000:100`

the result:

Version 1.96

Sequential Output

Sequential Input

Random Seeks

Sequential Create

Random Create

Concurrency

Size

Per Char

Block

Rewrite

Per Char

Block

Num Files

Create

Read

Delete

Create

Read

Delete

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

usb2

4

6576M

622

93

31604

7

14249

5

1215

90

34417

6

172.1

8

16

10578

46

+++++

+++

31734

50

31396

34

+++++

+++

+++++

+++

Latency

54816us

599ms

633ms

34401us

47854us

646ms

Latency

3091us

2151us

2125us

17656us

150us

6314us

usb3

4

6576M

615

95

32326

7

14434

5

1579

87

34827

5

172.8

8

16

19709

42

+++++

+++

+++++

+++

32034

36

+++++

+++

+++++

+++

Latency

44231us

673ms

643ms

41676us

64893us

655ms

Latency

6150us

530us

806us

4056us

283us

2863us

usb2

4

6576M

679

96

32972

7

14547

5

1490

94

34936

6

172.9

8

16

10681

46

+++++

+++

20249

61

15184

65

+++++

+++

20964

62

Latency

29018us

438ms

649ms

31497us

30470us

587ms

Latency

2858us

2295us

3218us

3825us

5738us

2266us

usb3

4

6576M

583

94

36709

8

17449

6

1025

99

43703

8

195.4

10

16

13722

56

+++++

+++

+++++

+++

18304

71

+++++

+++

+++++

+++

Latency

38191us

519ms

538ms

27632us

28568us

539ms

Latency

547us

509us

706us

211us

15us

36us

usb3

4

6576M

645

98

36868

8

17472

6

1971

98

43596

8

198.3

10

16

14829

60

+++++

+++

+++++

+++

+++++

+++

+++++

+++

+++++

+++

Latency

40016us

520ms

584ms

24271us

27662us

474ms

Latency

558us

509us

555us

84us

8us

32us

usb3

4

6576M

782

97

36838

8

17439

6

1490

98

43479

8

196.4

9

16

22927

33

+++++

+++

+++++

+++

+++++

+++

+++++

+++

+++++

+++

Latency

10200us

583ms

562ms

28055us

37441us

538ms

Latency

434us

514us

559us

87us

8us

35us

The above results are rather clear. The usb3 controller is resulting in better performance according to bonnie too. Just to be sure I re-ran the test and the results were slightly less pronounced.

In conclusion:

Using a usb3 controller resulted in a 10%-20% increase in performance on the same disk while using a usb2 connection. At times the difference is smaller, rarely it is bigger.

Too bad this usb controller only has 2 connections. I wonder how much performance remains when I start daisy-chaining multiple devices on those connectors...

jumbo frames at home

I have a home network that runs partly on gigabit. With partly I mean that a section of the network is connected by a gigabit switch, and a section is connecting with a 'home router' that services my fiberoptic internet connection (yes, I have fiber to the home aka FttH).
When transferring a file between 2 machines on the gigabit section of the network I noticed it was taking a loooong time. Actually the transferspeed was  not or not much more than 100mbit.
Because it was one large file (10GB) I suspected that turning on jumboframes would help the situation.
So, I went into the windows 7 networking hardware settings and turned jumbo frames on with a 7K max frame size and did the same for my linux machine. The transfer died. I tried to re-start it, but it wouldn't work. I could still ping the linux box from the windows box, so there was still something working properly.
SSH worked too, but I couldn't get large outputs to be displayed all at once. Only small commands with small outputs would give a result. Odd! I suspected that large frames would not be transmitted/received correctly and proceeded to lower the MTU back in steps until everything would work again. I ended up at an MTU of 3000 bytes. That's only twice the size of the standard MTU of about 1500.
It didn't help the transferspeed at all.
I decided to look a bit deeper and found that linux warns of changing the MTU on that specific network card :
[1201852.913302] r8169: WARNING! Changing of MTU on this NIC may lead to frame reception errors!

Great. This was actually happening. That's not a 'may', it's a 'will'.

For reference, this is an onboard NIC for a motherboard that I bought a month or 2 ago.
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)

On the windows machine the onboard NIC is a RTL8167. Different NIC, no frame corruption? I hope so.

There are many other possible reasons for these 2 machines not being able to transfer files at high speeds. The gigabit switch I have is a cheap no-brand (actually it's an SMC switch), the NICs themselves may not be up to the task, the machines may be too busy doing other things and the windows filesharing protocol is notoriously inefficient. There may be further optimalizations to be done with offloading settings on these NICs.

I've set the MTU back to the default (1500) and later figured out the 100mbit was because the Linux machine was plugged into the wrong network segment (the 100mbit switch). *DUH*
Regardless it's still interesting to see the NIC having problems with a bigger MTU.