Dell lifecycle controller

Dell has this great feature in their modern servers, the Unified Server Configurator (what's in a name), USC. Or it's the Lifecycle controller.. never know which is which. You can get to it through the bios and it allows you to update the system firmware. That's a neat trick, because it saves you from needing to install any OS dependent packages. Especially on linux that can be a lifesave because Dell only supports RedHat and Suse. They make life intentionally hard for CentOS users by trying to put as many 'redhat' checks in their code, previously only on /etc/redhadt-release but recently also on /etc/issue. So those need to be modified on CentOS systems in order for Dell linux firmware updates to work. Not so with the lifecycle controller.
Just press F10 to enter the 'sytem services' (another name for the USC?) in the bios screen, and select 'usc settings','network settings' to set up the network connection. Then through 'platform update', 'launch platform update', 'ftp server' you end up at a screen where you can configure the ftp server (ftp.dell.com or ftp.euro.dell.com [why don't they use geo-dns?]). And then you can 'test network connection' or just press next.

That's where the fun stopped for me. This server, an r510, has nic1 connected to the san and a pci intel 10gig nic going to the routable network and internet. The update apparantly is only possible over NIC1, even though you are able to configure the USC for a different nic. Such a bummer..

I tried this instead, but that only works for the BIOS. It put me on the right track though. I needed to update the H700 perc controller, so I ended up doing this:
./sasdupie -u -s /usr/share/firmware/dell/dup/pci_firmware_ven_0x1000_dev_0x0079_subven_0x1028_subdev_0x1f17_version_a10/ -o doit.txt
and in the doit.txt it said 'The operation was successful. '
`omreport storage controller` confirmed it:
Firmware Version                              : 12.10.2-0004

done.

ocz vertex 3 linux performance: promise vs onboard ATI SB700/SB800

I finally bought myself an SSD. I've been using spinning run for ever and recently we started using SSDs for the servers at work. I just had to go and buy one for myself. The windows boot time was starting to really annoy me.
At €99,- for 60GB it's not cheap, but it should be fast. The box says up to 535MB/sec read and 490MB/sec write. I doubt that'll even be achieven in real world situations. Ofcourse I do want to try :)

So first I hooked it up to one of the ports of my promise sata 300tx4 controller. This was the hdparm result:

/dev/sdl:
 Timing cached reads:   2514 MB in  2.00 seconds = 1257.10 MB/sec
 Timing buffered disk reads: 274 MB in  3.00 seconds =  91.33 MB/sec

That's... not any faster than my harddisks... :(


I couldn't believe it. So I moved the SSD over to my onboard ati sata controller, and hdparm did something different:
/dev/sdm:
 Timing cached reads:   2830 MB in  2.00 seconds = 1414.96 MB/sec
 Timing buffered disk reads: 1074 MB in  3.00 seconds = 357.86 MB/sec

That's better! Not quite there yet, but atleast now I get the idea I spent my money wisely.

Especially compared to my OS disk, and a newer 2TB disk I bought:

/dev/hda: (pata disk)
 Timing cached reads:   2516 MB in  2.00 seconds = 1257.80 MB/sec
 Timing buffered disk reads: 166 MB in  3.02 seconds =  54.90 MB/sec

/dev/disk/by-id/ata-Hitachi_HDS5C3020ALA632_ML0220F30TG22D:
 Timing cached reads:   2222 MB in  2.00 seconds = 1111.31 MB/sec
 Timing buffered disk reads: 300 MB in  3.02 seconds =  99.50 MB/sec

I find this huge performance difference between my 2 sata controllers a bit disconcernting. Especially since the promise controller is a sata-II capable device (300MB/sec), but not even achieving sata-I speeds (150MB/sec). At the same time my onboard sata controller is sata-III capable (600MB/sec) and not achieving that, although it does get a little bit above the sata-II spec. What could cause this?
One limiting factor is the PCI bus that the promise card is on. That limits it to 266MB/sec so that's not the bottleneck. The sata cables according to the sata wikipedia page should all go up to sata-III. Maybe the rest of the system is too slow to keep up? Let's do an easier test:

server ~ # mount /dev/disk/by-id/ata-OCZ-VERTEX3_OCZ-797F5Y0407KD83KC /mnt/ssd
server ~ # dd if=/dev/zero of=/mnt/ssd/zero-write count=20000000
20000000+0 records in
20000000+0 records out
10240000000 bytes (10 GB) copied, 41.3909 s, 247 MB/s
server ~ # dd if=/mnt/ssd/zero-write of=/dev/null count=20000000
20000000+0 records in
20000000+0 records out
10240000000 bytes (10 GB) copied, 30.1224 s, 340 MB/s
server ~ #

While doing this test the io-utilisation was at 40-60% ('iostat -x 1 sdm) when writing and 75% when reading. So the device could handle more, but it was just waiting for the system. In fact, taking that 247MB/sec and correcting for the IO utilisation, the device indeed seems to be able to handle about 500MB/sec. Wow.
So why isn't the device being 100% utilised? When looking at 'top' while doing these tests it shows one of the 2 cores to be 100% busy when writing to the device, but only 50% busy when reading.
read: 40% waiting and 60% system time. 0% idle.
increasing the blocksize to 1MB increases read speed to 360MB/sec
switching to direct io and nonblocking io the speed increases further:
server ~ # dd if=/mnt/ssd/zero-write of=/dev/null count=20000000 bs=1048576  iflag=direct,nonblock
9765+1 records in
9765+1 records out
10240000000 bytes (10 GB) copied, 24.5059 s, 418 MB/s

and writing:
server ~ # dd if=/dev/zero of=/mnt/ssd/zero-write count=5000 bs=1048576 oflag=direct,nonblock
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 14.8777 s, 352 MB/s

75% io wait, 23% system time. 75% utilisation

So using a larger blocksize with nonblocking io and direct read/write the io increases slightly. But we're still not where whe should be.

Let's cut ext4 out of the loop:
server ~ # dd if=/dev/disk/by-id/ata-OCZ-VERTEX3_OCZ-797F5Y0407KD83KC of=/dev/null count=20000000 bs=1048576 iflag=direct,nonblock
57241+1 records in
57241+1 records out
60022480896 bytes (60 GB) copied, 136.918 s, 438 MB/s
server ~ #
 this results in 90% utilisation and 87% cpu waiting on io, with 12% cpu system time.

write:
server ~ # dd if=/dev/zero of=/dev/disk/by-id/ata-OCZ-VERTEX3_OCZ-797F5Y0407KD83KC count=20000 bs=1048576 oflag=direct,nonblock skip=2
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB) copied, 58.2396 s, 360 MB/s
server ~ #
80% util, 75% wait, 25% system

Based on all if this I think the performance of this SSD is CPU bound.
The difference between the promise and the onboard sata controller may be related to AHCI: the stock kernel driver for promise doesn't support AHCI.

Now that I'm done, I'm cleaning up the SSD before handing it over to my windows pc:
i=0; while [ $i -lt 117231408 ]; do echo $i:64000; i=$(((i+64000))); done  | hdparm --trim-sector-ranges-stdin --please-destroy-my-drive /dev/disk/by-id/ata-OCZ-VERTEX3_OCZ-797F5Y0407KD83KC
This makes sure that all data is 'trimmed' aka removed without writing to the device. This keeps the SSD fast, because a write to a block of data that already has data in it is much slower that a write to a block that is empty, aka trimmed.

This is what bonnie has to say:

Version 1.96			Sequential Output						Sequential Input				Random Seeks			Sequential Create						Random Create
Concurrency		Size	Per Char		Block		Rewrite		Per Char		Block				Num Files	Create		Read		Delete		Create		Read		Delete
			K/sec	% CPU	K/sec	% CPU	K/sec	% CPU	K/sec	% CPU	K/sec	% CPU	/sec	% CPU		/sec	% CPU	/sec	% CPU	/sec	% CPU	/sec	% CPU	/sec	% CPU	/sec	% CPU
hda-pata	4	6576M	640	94	26899	11	14683	4	1828	98	45783	6	145.1	7	16	8091	9	+++++	+++	23202	22	30607	40	+++++	+++	+++++	+++
	Latency		36690us		2432ms		4584ms		17325us		229ms		2529ms		Latency	12317us		849us		828us		107us		67us		73us
hda-pata	4	6576M	681	93	26794	10	14465	4	1453	98	43385	6	135.2	7	16	6404	25	+++++	+++	23371	60	18094	66	+++++	+++	24027	60
	Latency		13723us		2087ms		3965ms		20442us		267ms		2308ms		Latency	13195us		2236us		2291us		252us		136us		67us
hda-pata	4	6576M	492	96	26432	10	14338	4	1921	94	39876	4	120.6	6	16	4261	4	+++++	+++	17685	13	21240	21	+++++	+++	23140	17
	Latency		38599us		2195ms		2586ms		20359us		404ms		2291ms		Latency	12161us		633us		895us		1212us		257us		3930us
zfs-2	4	6576M	20	22	18509	11	15498	10	1598	98	64493	8	142.7	4	16	1362	21	+++++	+++	6838	23	2714	21	+++++	+++	8875	28
	Latency		940ms		2100ms		2053ms		27966us		268ms		1879ms		Latency	134ms		6559us		4142us		66330us		105us		10113us
zfs-2	4	6576M	30	19	19664	12	15545	10	998	98	65135	7	164.6	5	16	2227	19	30995	8	6253	24	3056	18	+++++	+++	9731	22
	Latency		432ms		1860ms		2054ms		25254us		218ms		1825ms		Latency	97645us		2929us		10128us		76339us		82us		18389us
zfs-2	4	6576M	31	19	18794	12	15585	10	1655	98	59008	9	165.1	5	16	1337	22	27141	17	2681	26	1413	20	+++++	+++	8789	27
	Latency		384ms		1890ms		1993ms		21819us		236ms		1747ms		Latency	61531us		3114us		7322us		339ms		105us		42091us
ssd-promise	4	6576M	604	99	97059	24	47298	15	1700	99	119279	19	5231	220	16	15393	17	+++++	+++	+++++	+++	+++++	+++	+++++	+++	+++++	+++
	Latency		38396us		721ms		804ms		10871us		9170us		81333us		Latency	83us		531us		549us		97us		8us		41us
ssd-promise	4	6576M	766	98	97783	24	47870	15	1810	99	120092	20	4713	200	16	13357	15	+++++	+++	+++++	+++	+++++	+++	+++++	+++	+++++	+++
	Latency		11147us		661ms		787ms		11471us		3283us		97766us		Latency	210us		531us		547us		85us		10us		46us
ssd-promise	4	6576M	774	98	97767	24	47757	15	1464	99	120094	20	4672	191	16	10697	19	+++++	+++	+++++	+++	+++++	+++	+++++	+++	+++++	+++
	Latency		11036us		226ms		775ms		12233us		2866us		91878us		Latency	439us		531us		549us		90us		9us		64us
ssd-ati	4	6576M	648	99	454837	43	201613	29	1316	99	482448	35	12924	381	16	12573	14	+++++	+++	+++++	+++	+++++	+++	+++++	+++	+++++	+++
	Latency		38218us		175ms		109ms		12974us		124ms		3048us		Latency	90us		524us		556us		87us		7us		30us
ssd-ati	4	6576M	669	99	466455	47	202751	28	2659	99	475295	38	11540	333	16	15038	17	+++++	+++	+++++	+++	+++++	+++	+++++	+++	+++++	+++
	Latency		38867us		127ms		111ms		4088us		121ms		2753us		Latency	256us		524us		558us		91us		63us		63us
ssd-ati	4	6576M	625	99	467713	50	176020	57	1345	99	484839	47	11420	342	16	21595	24	+++++	+++	+++++	+++	+++++	+++	+++++	+++	+++++	+++
	Latency		39029us		113ms		54983us		11317us		74753us		4429us		Latency	207us		534us		573us		86us		65us		66us

As you can see, the SSD is much, much faster than any disk. ZFS is reaaaalllly slow, but that's because I'm running it through fuse so all IO needs to go through userspace. But that's ok, I don't need speed on those disks. Only safety. And once zfsonlinux gets around to finishing their code it should speed up tremendously.

If I get to buy a faster PC I'll update this doc with new measurements.

scrubbing bits under linux; making sure your data is safe

Let's say you've got a couple of photo's you'd like to keep. Actually as the years have gone by, it's more like a few hundred, perhaps a few thousand photos you've collected and would really hate to lose. And let's assume you've lost data before due to a harddisk crash or otherwise. You swore never to be troubled by that problem again, so you set up a mirror, being raid-1, on your computer.
And let's assume you use Linux to manage that. Windows could too, but in a minute I'll show you why windows is no good for a mirror.

It all has to do with bit rot. You see, data on your disks isn't safe. It's not just because disks can die and you won't be able to access them. It's that data on your disks slowly rots and 1's can turn into 0's. 0's can turn into 1's. That makes your data corrupt. This happens because these days the bits on harddisks are so tiny that small influences can change them. Possibly cosmic rays. Or a magnet slightly too close to the disk. Perhaps the magnetic field of one of the computer fans. Or more likely a small patch of magnetic substrate that was just a bit off at production and isn't able to hold its contents for more than a month. What ever the reason, the fact is that data slowly rots.

Now you have a mirror. A raid set. Raid-1 even. You're not safe. But you're a bit safer. What you could do is have your computer run a check on the 2 disks, to make sure that the contents of the disks is still exactly the same. You could have it run every night. This is called a 'data scrub'.

 And then one night, maybe a year from now, your computer will beep and state it has found a different. Perhaps it's only one bit. But what now? Which of the disks is right? Disk A says it's a 0. Disk B says it's a 1. If you're lucky you have backups and you can restore the file. If not... it's anyone's guess. Oh did I mention that backups can suffer from the same problem ? This is not just limited to harddisks. CD's rot too. So do tapes, floppy's, zip's, jazz's, etc.

Now, please, amuse me. Go a google for 'windows data scrub' and do one for 'linux data scrub'. Notice any difference? There are no data scrubbing tools for windows. But there are for Linux. The reason is that when Windows finds a difference, it will use the 'master' disk as the source to write to the 'slave' disk. Even when the master is wrong.

So under windows having a mirror is not much help. You won't even notice that there was a problem. Under linux you can have the scrub tell you there was a problem, and you can decide to do a file restore from backup.

If you are using raid-5 instead of raid-1 you have the added benefit of being able to repair the bad data. Read here on how to set up scrubbing for md-raid software raid setups under Linux.

And if you're really smart you're using a checksumming filesystem, such as ZFS or BTRFS. These calculate a checksum of the data and by that allow the OS to figure out which of the two disks has the correct data, and overwrite the bad data with the good data. Automatically. Problem fixed.

If you are not scrubbing yet, you don't value your photo's enough.

ZFS data corruption and CRC checksum errors

It's been a while since I took some time to analyze the following problem:

server ~ # zpool status -v
  pool: mir-2tb
 state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
 scrub: scrub completed after 9h26m with 1 errors on Mon Sep 26 09:01:29 2011
config:

    NAME                                                       STATE     READ WRITE CKSUM
    mir-2tb                                                    ONLINE       0     0     1
      mirror-0                                                 ONLINE       0     0     2
        disk/by-id/ata-Hitachi_HDS5C3020ALA632_ML0220F30SHWWD  ONLINE       0     0     2
        disk/by-id/ata-SAMSUNG_HD204UI_S2H7J9BZC05894          ONLINE       0     0     2
        disk/by-id/ata-Hitachi_HDS5C3020ALA632_ML0220F30U4RKD  ONLINE       0     0     2

errors: Permanent errors have been detected in the following files:

        /mnt/origin/mir-2tb/diskfailure/sdb1.image.dd


So there is a problem here. What this is saying is that the same block of data is not readable from 3 different disks. And the file sdb1.image.dd experienced data loss because of this.

Every week on monday, the counters for the amount of checksum errors will go up. That's because I'm running a scrub every monday morning. And once the scrub is done, the counter is 2 checksum errors higher for each disk, and one checksum error higher for the pool. So it's not like there are more and more sections of data that can't be read.


How did this happen? How small is that chance, that 3 drives are unreadable for the same specific block of data? It's infinitesimally small. And that's not what's going on here. There are 2 possible reasons for this:
 - What's going on here is that the ZFS pool was created initially with one source disk, with apparantly 2 bad sectors or bad blocks. Data was copied to that pool before there was a 2nd disk to mirror the data. Once the data was copied and the source disk was empty, the source disk was added to the pool as a mirror. But because of the bad blocks/sectors, 2 blocks could not be copied to the other disk. Those 2 sectors therefore also contain invalid data. Later a 3rd disk was added, but is has the same problem as the 2nd disk. Because the bad sectors can not be read, this will not resolve itself. The data is lost.
- What's going on here is that there is a very specific type of data that triggers a bug in ZFS where the data cannot be stored or the checksum cannot correctly be calculated. The data is there but the checksum fails because of this bug. There is no actual data lost.


I can't remember whether I started this pool with the scenario of having 1 disk and adding more later. I no longer have the source file so I cannot overwrite the data with it. The only option I have is to try and find the block that is broken and force a write to it. This will force the disk to overwrite the data, making it readable again or in case of not being able to write will force the disk to re-allocate the sector.

However, the disks are reporting no problems at all regarding pending sector re-allocations, the only thing I notice is:


SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   100   051    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0026   055   055   000    Old_age   Always       -       19396
  3 Spin_Up_Time            0x0023   068   066   025    Pre-fail  Always       -       9994
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       35
  5 Reallocated_Sector_Ct   0x0033   252   252   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   252   252   051    Old_age   Always       -       0
  8 Seek_Time_Performance   0x0024   252   252   015    Old_age   Offline      -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       5924
 10 Spin_Retry_Count        0x0032   252   252   051    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   252   252   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       70
181 Program_Fail_Cnt_Total  0x0022   098   098   000    Old_age   Always       -       48738283
191 G-Sense_Error_Rate      0x0022   100   100   000    Old_age   Always       -       37
192 Power-Off_Retract_Count 0x0022   252   252   000    Old_age   Always       -       0
194 Temperature_Celsius     0x0002   064   052   000    Old_age   Always       -       31 (Min/Max 14/48)
195 Hardware_ECC_Recovered  0x003a   100   100   000    Old_age   Always       -       0
196 Reallocated_Event_Count 0x0032   252   252   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   252   252   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   252   252   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0036   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x002a   100   100   000    Old_age   Always       -       86
223 Load_Retry_Count        0x0032   252   252   000    Old_age   Always       -       0
225 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       70


And that's not very helpfull either. There is one counter (Program_Fail_Cnt_Total) that's huge but some googling turns up that many users of this disk report that counter and it's not a problem for any of them. That doesn't worry me.

I wish that zfs had some more tools to make this analysis easier. Why does the CRC occur? What block exactly is having the problem.

So I'm at a loss here... there is a checksum error, but the disks don't report a problem with data on the disks. Is the problem therefore with ZFS, or with one of the disks that has (via ZFS) propagated to the other disks ? I would be very interested to hear anyones feedback on this.

choosing disks and controllers for ZFS

I looked at my ZFS setup and realised that I have 2 raid-1 setups, both with the same type of disk:

server ~ # zpool status
  pool: mir-2tb

 state: ONLINE
 scrub: none requested
config:

        NAME                                                       STATE     READ WRITE CKSUM
        mir-2tb                                                    ONLINE       0     0     0
          mirror-0                                                 ONLINE       0     0     0
            disk/by-id/ata-Hitachi_HDS5C3020ALA632_ML0220F30SHWWD  ONLINE       0     0     0
            disk/by-id/ata-Hitachi_HDS5C3020ALA632_ML0220F30U4RKD  ONLINE       0     0     0

errors: No known data errors

  pool: mir-2tb2
 state: ONLINE
 scrub: none requested
config:

        NAME                                                       STATE     READ WRITE CKSUM
        mir-2tb2                                                   ONLINE       0     0     0
          mirror-0                                                 ONLINE       0     0     0

            disk/by-id/ata-SAMSUNG_HD204UI_S2H7J9BZC05894          ONLINE       0     0     0
            disk/by-id/ata-SAMSUNG_HD204UI_S2H7J9BZC05884          ONLINE       0     0     0

errors: No known data errors
server ~ #

So that's one time ZFS with two hitachi disks and one time ZFS mirror with two samsung disks. That's bad because if there is a problem with a certain type of disk (such as a firmware problem with samsung HD204UI's, see here) there is a high chance the mirror will fail. ZFS can then detect the problem, but possibly not recover.I quickly bought another hitachi disk and added it to the samsung mirror. That allowed me to swap a samsung disk to the hitachi mirror pool. So now I have:

server ~ # zpool status  pool: mir-2tb
 state: ONLINE
 scrub: none requested
config:

        NAME                                                       STATE     READ WRITE CKSUM
        mir-2tb                                                    ONLINE       0     0     0
          mirror-0                                                 ONLINE       0     0     0
            disk/by-id/ata-Hitachi_HDS5C3020ALA632_ML0220F30SHWWD  ONLINE       0     0     0
            disk/by-id/ata-SAMSUNG_HD204UI_S2H7J9BZC05894          ONLINE       0     0     0
            disk/by-id/ata-Hitachi_HDS5C3020ALA632_ML0220F30U4RKD  ONLINE       0     0     0errors: No known data errors

  pool: mir-2tb2
 state: ONLINE
 scrub: none requested
config:

        NAME                                                       STATE     READ WRITE CKSUM
        mir-2tb2                                                   ONLINE       0     0     0
          mirror-0                                                 ONLINE       0     0     0
            disk/by-id/ata-Hitachi_HDS5C3020ALA632_ML0220F30TG22D  ONLINE       0     0     0
            disk/by-id/ata-SAMSUNG_HD204UI_S2H7J9BZC05884          ONLINE       0     0     0

errors: No known data errors
server ~ #

So now one pool has a raid-1 of 3 disks and the other a raid-1 of 2 disks. Availability thus even improved a bit.

Great, problem fixed. Not quite! Just making sure the disks in a pool are of mixed species is not enough.
It's not visible in the zpool status but the disks of the same pool were all on the same sata controller. I have an onboard sata controller and a promise sata controller. Here is a selective lspci:

00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode] (rev 40)
05:05.0 Mass storage controller: Promise Technology, Inc. PDC40718 (SATA 300 TX4) (rev 02)

So if a controller were to either die, corrupt data or have intermittant problems that could again result in the pool being lost because ZFS wouldn't be able to fix the problem using a source that still has correct data. Easy fix, just make sure that the disks in a zfs pool are spread out over different controllers. That way the chance of problems again diminishes.

Too paranoid? Not really. Before I replaced my server a couple of months ago I was using an onboard sata controller that was corrupting the data transferred to the disks. I never knew until I switched to ZFS.

Don't blow your bits. Spread them around.