ZFS data corruption and CRC checksum errors

It's been a while since I took some time to analyze the following problem:

server ~ # zpool status -v
  pool: mir-2tb
 state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
 scrub: scrub completed after 9h26m with 1 errors on Mon Sep 26 09:01:29 2011
config:

    NAME                                                       STATE     READ WRITE CKSUM
    mir-2tb                                                    ONLINE       0     0     1
      mirror-0                                                 ONLINE       0     0     2
        disk/by-id/ata-Hitachi_HDS5C3020ALA632_ML0220F30SHWWD  ONLINE       0     0     2
        disk/by-id/ata-SAMSUNG_HD204UI_S2H7J9BZC05894          ONLINE       0     0     2
        disk/by-id/ata-Hitachi_HDS5C3020ALA632_ML0220F30U4RKD  ONLINE       0     0     2

errors: Permanent errors have been detected in the following files:

        /mnt/origin/mir-2tb/diskfailure/sdb1.image.dd


So there is a problem here. What this is saying is that the same block of data is not readable from 3 different disks. And the file sdb1.image.dd experienced data loss because of this.

Every week on monday, the counters for the amount of checksum errors will go up. That's because I'm running a scrub every monday morning. And once the scrub is done, the counter is 2 checksum errors higher for each disk, and one checksum error higher for the pool. So it's not like there are more and more sections of data that can't be read.


How did this happen? How small is that chance, that 3 drives are unreadable for the same specific block of data? It's infinitesimally small. And that's not what's going on here. There are 2 possible reasons for this:
 - What's going on here is that the ZFS pool was created initially with one source disk, with apparantly 2 bad sectors or bad blocks. Data was copied to that pool before there was a 2nd disk to mirror the data. Once the data was copied and the source disk was empty, the source disk was added to the pool as a mirror. But because of the bad blocks/sectors, 2 blocks could not be copied to the other disk. Those 2 sectors therefore also contain invalid data. Later a 3rd disk was added, but is has the same problem as the 2nd disk. Because the bad sectors can not be read, this will not resolve itself. The data is lost.
- What's going on here is that there is a very specific type of data that triggers a bug in ZFS where the data cannot be stored or the checksum cannot correctly be calculated. The data is there but the checksum fails because of this bug. There is no actual data lost.


I can't remember whether I started this pool with the scenario of having 1 disk and adding more later. I no longer have the source file so I cannot overwrite the data with it. The only option I have is to try and find the block that is broken and force a write to it. This will force the disk to overwrite the data, making it readable again or in case of not being able to write will force the disk to re-allocate the sector.

However, the disks are reporting no problems at all regarding pending sector re-allocations, the only thing I notice is:


SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   100   051    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0026   055   055   000    Old_age   Always       -       19396
  3 Spin_Up_Time            0x0023   068   066   025    Pre-fail  Always       -       9994
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       35
  5 Reallocated_Sector_Ct   0x0033   252   252   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   252   252   051    Old_age   Always       -       0
  8 Seek_Time_Performance   0x0024   252   252   015    Old_age   Offline      -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       5924
 10 Spin_Retry_Count        0x0032   252   252   051    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   252   252   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       70
181 Program_Fail_Cnt_Total  0x0022   098   098   000    Old_age   Always       -       48738283
191 G-Sense_Error_Rate      0x0022   100   100   000    Old_age   Always       -       37
192 Power-Off_Retract_Count 0x0022   252   252   000    Old_age   Always       -       0
194 Temperature_Celsius     0x0002   064   052   000    Old_age   Always       -       31 (Min/Max 14/48)
195 Hardware_ECC_Recovered  0x003a   100   100   000    Old_age   Always       -       0
196 Reallocated_Event_Count 0x0032   252   252   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   252   252   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   252   252   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0036   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x002a   100   100   000    Old_age   Always       -       86
223 Load_Retry_Count        0x0032   252   252   000    Old_age   Always       -       0
225 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       70


And that's not very helpfull either. There is one counter (Program_Fail_Cnt_Total) that's huge but some googling turns up that many users of this disk report that counter and it's not a problem for any of them. That doesn't worry me.

I wish that zfs had some more tools to make this analysis easier. Why does the CRC occur? What block exactly is having the problem.

So I'm at a loss here... there is a checksum error, but the disks don't report a problem with data on the disks. Is the problem therefore with ZFS, or with one of the disks that has (via ZFS) propagated to the other disks ? I would be very interested to hear anyones feedback on this.

choosing disks and controllers for ZFS

I looked at my ZFS setup and realised that I have 2 raid-1 setups, both with the same type of disk:

server ~ # zpool status
  pool: mir-2tb

 state: ONLINE
 scrub: none requested
config:

        NAME                                                       STATE     READ WRITE CKSUM
        mir-2tb                                                    ONLINE       0     0     0
          mirror-0                                                 ONLINE       0     0     0
            disk/by-id/ata-Hitachi_HDS5C3020ALA632_ML0220F30SHWWD  ONLINE       0     0     0
            disk/by-id/ata-Hitachi_HDS5C3020ALA632_ML0220F30U4RKD  ONLINE       0     0     0

errors: No known data errors

  pool: mir-2tb2
 state: ONLINE
 scrub: none requested
config:

        NAME                                                       STATE     READ WRITE CKSUM
        mir-2tb2                                                   ONLINE       0     0     0
          mirror-0                                                 ONLINE       0     0     0

            disk/by-id/ata-SAMSUNG_HD204UI_S2H7J9BZC05894          ONLINE       0     0     0
            disk/by-id/ata-SAMSUNG_HD204UI_S2H7J9BZC05884          ONLINE       0     0     0

errors: No known data errors
server ~ #

So that's one time ZFS with two hitachi disks and one time ZFS mirror with two samsung disks. That's bad because if there is a problem with a certain type of disk (such as a firmware problem with samsung HD204UI's, see here) there is a high chance the mirror will fail. ZFS can then detect the problem, but possibly not recover.I quickly bought another hitachi disk and added it to the samsung mirror. That allowed me to swap a samsung disk to the hitachi mirror pool. So now I have:

server ~ # zpool status  pool: mir-2tb
 state: ONLINE
 scrub: none requested
config:

        NAME                                                       STATE     READ WRITE CKSUM
        mir-2tb                                                    ONLINE       0     0     0
          mirror-0                                                 ONLINE       0     0     0
            disk/by-id/ata-Hitachi_HDS5C3020ALA632_ML0220F30SHWWD  ONLINE       0     0     0
            disk/by-id/ata-SAMSUNG_HD204UI_S2H7J9BZC05894          ONLINE       0     0     0
            disk/by-id/ata-Hitachi_HDS5C3020ALA632_ML0220F30U4RKD  ONLINE       0     0     0errors: No known data errors

  pool: mir-2tb2
 state: ONLINE
 scrub: none requested
config:

        NAME                                                       STATE     READ WRITE CKSUM
        mir-2tb2                                                   ONLINE       0     0     0
          mirror-0                                                 ONLINE       0     0     0
            disk/by-id/ata-Hitachi_HDS5C3020ALA632_ML0220F30TG22D  ONLINE       0     0     0
            disk/by-id/ata-SAMSUNG_HD204UI_S2H7J9BZC05884          ONLINE       0     0     0

errors: No known data errors
server ~ #

So now one pool has a raid-1 of 3 disks and the other a raid-1 of 2 disks. Availability thus even improved a bit.

Great, problem fixed. Not quite! Just making sure the disks in a pool are of mixed species is not enough.
It's not visible in the zpool status but the disks of the same pool were all on the same sata controller. I have an onboard sata controller and a promise sata controller. Here is a selective lspci:

00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode] (rev 40)
05:05.0 Mass storage controller: Promise Technology, Inc. PDC40718 (SATA 300 TX4) (rev 02)

So if a controller were to either die, corrupt data or have intermittant problems that could again result in the pool being lost because ZFS wouldn't be able to fix the problem using a source that still has correct data. Easy fix, just make sure that the disks in a zfs pool are spread out over different controllers. That way the chance of problems again diminishes.

Too paranoid? Not really. Before I replaced my server a couple of months ago I was using an onboard sata controller that was corrupting the data transferred to the disks. I never knew until I switched to ZFS.

Don't blow your bits. Spread them around.

real world usb3 disk performance

So a while ago I bought a new motherboard. It took me a while to make a choice because I wanted it to be cheap, while still having at least 6 sata connections, pata too and then be able to use ECC type memory. Having ZFS is great to keep your data correctly on your disks but if the data gets corrupted in memory you're still lost, hence the ECC.
I finally choose the asus M4A88TDM-EVO because it had all the specs I wanted AND included usb3. Great I thought, just what I need... this is future proof. Too bad it turned out to be just a separate pci-e 1x card. But it's there and after compiling a kernel module for it (usb3 requires the xhci module, or xhci_hcd on gentoo) it worked.

Kinda.

It's very spammy in dmesg. Lots of debug info it seems. Or at least info I don't need. This is kernel "Linux server 2.6.36-gentoo-r8 #2 SMP Tue Apr 5 08:18:52 CEST 2011 i686 AMD Athlon(tm) II X2 250 Processor AuthenticAMD GNU/Linux" mind you, so newer kernels may no have this.

Time to test this puppy out. Since the theoretical limit of usb2 is 60MByte/sec there may be a change that my disks are sometimes hitting the limit of the bus, instead of the speed limit of the platters. Since usb3 is 10x as fast, the bus will never be the bottleneck.
So would this really matter, in real life?

I used a Hitachi desktstar 7k1000 HDT721075SLA380 to test with. It's supposed to be able to read at speeds between 86MB/sec (outer disk section) and 46MB/sec (inner sections). With the 60MB max speed of usb2 there is something to gain here. The disk is connected using a LaCie usb-sata external enclosure.
The usb3 controller is this one:
02:00.0 USB Controller: NEC Corporation Device 0194 (rev 03)

I performed some tests, first on usb2. Please don't mind the changing drive letters, every time I unplug and re-plug the device it may take up a new drive letter. It remains the same drive.

server ~ # hdparm -tT /dev/sdl

/dev/sdl:

 Timing cached reads:   2644 MB in  2.00 seconds = 1322.00 MB/sec

 Timing buffered disk reads:  92 MB in  3.06 seconds =  30.03 MB/sec

server ~ #

So theoretically this disk should be able to transfer 1.3GB/sec from its cache to the cpu. When reading from the platter this drops to 30MB/sec. It's an old disk ;-) Hdparm is a bit of a synthetic test, so what does a file transfer do for us:

write:

server maxx # dd if=bigfile of=/mnt/usb-backup-disk-750g-20081001/bigfile

9170564+1 records in

9170564+1 records out

4695328969 bytes (4.7 GB) copied, 211.23 s, 22.2 MB/s

server linux # dd count=9000000 if=/dev/zero of=/mnt/usb-backup-disk-750g-20081001/bigfile

9000000+0 records in

9000000+0 records out

4608000000 bytes (4.6 GB) copied, 133.498 s, 34.5 MB/s

server linux #

read :

server linux # dd if=/mnt/usb-backup-disk-750g-20081001/bigfile of=/dev/null

9170564+1 records in

9170564+1 records out

4695328969 bytes (4.7 GB) copied, 138.333 s, 33.9 MB/s

read while taking the filesystem out of the loop:

server linux # dd count=10000000 if=/dev/sdk of=/dev/null

10000000+0 records in

10000000+0 records out

5120000000 bytes (5.1 GB) copied, 155.335 s, 33.0 MB/s

[odd, that's actually slower then when reading from the fs... ]


then on usb3:

server ~ # hdparm -tT /dev/sdn

/dev/sdn:

 Timing cached reads:   2434 MB in  2.00 seconds = 1216.78 MB/sec

 Timing buffered disk reads: 104 MB in  3.00 seconds =  34.63 MB/sec

server ~ #

that's a small improvement to read from the platters. Interestingly the cache read is slightly slower

write:

server maxx # dd if=bigfile of=/mnt/usb-backup-disk-750g-20081001/bigfile

mount

9170564+1 records in

9170564+1 records out

4695328969 bytes (4.7 GB) copied, 123.346 s, 38.1 MB/s

write zero's
server maxx # dd count=9000000 if=/dev/zero of=/mnt/usb-backup-disk-750g-20081001/bigfile
9000000+0 records in
9000000+0 records out
4608000000 bytes (4.6 GB) copied, 122.208 s, 37.7 MB/s

read:

server maxx # dd if=/mnt/usb-backup-disk-750g-20081001/bigfile of=/dev/null

9170564+1 records in

9170564+1 records out

4695328969 bytes (4.7 GB) copied, 119.702 s, 39.2 MB/s

read from disk without fs:
server maxx # dd count=9000000 if=/dev/sdk of=/dev/null
9000000+0 records in
9000000+0 records out
4608000000 bytes (4.6 GB) copied, 122.362 s, 37.7 MB/s

Next up was bonnie. When running the usb3 test my machine catched a kernel panic! Odd.. I reset the machine and run bonnie again. No crash this time. This was a trigger for me to implement
`bonnie++ -d /mnt/usb-backup-disk-750g-20081001/tmp -s 6576 -c 4 -m usb220110925 -x 3 -u 1000:100`

the result:

Version 1.96

Sequential Output

Sequential Input

Random Seeks

Sequential Create

Random Create

Concurrency

Size

Per Char

Block

Rewrite

Per Char

Block

Num Files

Create

Read

Delete

Create

Read

Delete

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

K/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

/sec

% CPU

usb2

4

6576M

622

93

31604

7

14249

5

1215

90

34417

6

172.1

8

16

10578

46

+++++

+++

31734

50

31396

34

+++++

+++

+++++

+++

Latency

54816us

599ms

633ms

34401us

47854us

646ms

Latency

3091us

2151us

2125us

17656us

150us

6314us

usb3

4

6576M

615

95

32326

7

14434

5

1579

87

34827

5

172.8

8

16

19709

42

+++++

+++

+++++

+++

32034

36

+++++

+++

+++++

+++

Latency

44231us

673ms

643ms

41676us

64893us

655ms

Latency

6150us

530us

806us

4056us

283us

2863us

usb2

4

6576M

679

96

32972

7

14547

5

1490

94

34936

6

172.9

8

16

10681

46

+++++

+++

20249

61

15184

65

+++++

+++

20964

62

Latency

29018us

438ms

649ms

31497us

30470us

587ms

Latency

2858us

2295us

3218us

3825us

5738us

2266us

usb3

4

6576M

583

94

36709

8

17449

6

1025

99

43703

8

195.4

10

16

13722

56

+++++

+++

+++++

+++

18304

71

+++++

+++

+++++

+++

Latency

38191us

519ms

538ms

27632us

28568us

539ms

Latency

547us

509us

706us

211us

15us

36us

usb3

4

6576M

645

98

36868

8

17472

6

1971

98

43596

8

198.3

10

16

14829

60

+++++

+++

+++++

+++

+++++

+++

+++++

+++

+++++

+++

Latency

40016us

520ms

584ms

24271us

27662us

474ms

Latency

558us

509us

555us

84us

8us

32us

usb3

4

6576M

782

97

36838

8

17439

6

1490

98

43479

8

196.4

9

16

22927

33

+++++

+++

+++++

+++

+++++

+++

+++++

+++

+++++

+++

Latency

10200us

583ms

562ms

28055us

37441us

538ms

Latency

434us

514us

559us

87us

8us

35us

The above results are rather clear. The usb3 controller is resulting in better performance according to bonnie too. Just to be sure I re-ran the test and the results were slightly less pronounced.

In conclusion:

Using a usb3 controller resulted in a 10%-20% increase in performance on the same disk while using a usb2 connection. At times the difference is smaller, rarely it is bigger.

Too bad this usb controller only has 2 connections. I wonder how much performance remains when I start daisy-chaining multiple devices on those connectors...

SSD temperature problems

Some (most) SSD's don't have a temperature sensor. The result is that temperatures can be reported that are just wrong. Like 127 degrees, or -1. For servers this can be a problem because the server can't be sure the sensor is defective or whether there is a real problem. So it will spin the fans at maximum speed in order to try and cool the disk. That increases the power usage slightly and increases the noise level drastically. Within a DC that may not be a problem (if you have proper hearing protection)

I've tried 3 SSDs and 2 happened to work fine:
60GB OCZ Vertex 2 (OCZSSD2-2VTXE60G) - broken (firmware 1.33)
60GB Corsair Force (CSSD-F60GB2-BRKT) - fine
60GB Corsair Nova (CSSD-V60GB2) - fine

Some disks don't have a sensor but report a fixed temperature. That also works. Intel's 710 SSD range seems to have an actual temperature sensor.

So if you plan on using such a consumer grade SSD for a solution that bases fan speed on disk temperatures (like servers) this can help you figure out just slightly better what to buy, and what not to buy.

2011-10-18 update:
OCZ vertex 3 also does not have a (working) temperature sensor according to smart

4k sectors and aligning for LVM

I've been running an LVM volume spread out over a slowly growing number of disks for a couple of years now. Recently a disk broke and I RMA'd it. It was a western digital wd10eads. I got back a wd10ears. note the tiny differenc. Same size disk. But now using 4k sectors instead of 512byte sectors.
Interesting!

You know how new harddrives these days work with 4k sectors, right? And how many disks 'lie' about this and say that they (still) have a sector size of 512 bytes?
I have read about the 4k sector size a while ago and became aware of a number of possible problems that can come up when using those kinds of drives that 'lie'.

In general, what's the biggest problem?
If you are using ext3, then data is stored in chunks of 4096bytes (4k) by default. But the because the disk is stating it stores data in 512 byte chunks the OS will split up 4k writes into 8 writes of 512 bytes. That's sub optimal, but the real problem happens when you use partitions. By default, the 1st partition start at sector 63. That means that sectors 0-62 are blocks of 512 bytes used for the MBR and other things, and then sector 63 is the 1st data for your 1st partition. And because the ext3 filesystem stores data in 4k chunks [AFAIK!], the 1st ext3 inode fills sectors 63-71 (8x 512byte sectors). However.... sector 63 is not the start of a 4096 byte sector on the disk itself. The 4k blocks start at sectors 0, 8, 16, 24, 32, 40, 48, 56, 64
This means that when the OS wants to read the 1st inode on the 1st partition, it will send 8 requests for 512 byte blocks of data. The disk then needs to read 2 sectors, because these 8 blocks are split over 2 actual 4k sectors physically.
If however the 1st partition would have started at the 64th sector, the 4k inode would be exactly aligned with the actual sector on the disk. Then the OS would request 8 blocks and the disk would only need to read 1 4k sector. To make that 1st partition start at the right place is called 'alignment'.

This isn't too bad when you're just requesting 1 small block of data. But when doing big file operations or the server is very heavy on IO, you're in trouble.


For writes the problem is much worse. When doing a write of a 4k inode the disk will first need to read both sectors before it can write, because only part of the sectors will be updated. The 1st physical 4k sector will get 512 bytes overwritten, leaving 3584bytes untouched. The next sector will get 3584 bytes overwritten, with 512 bytes untouched. Those untouched bytes need to 1st be read by the disk, because it can't do a partial write to a sector. Disks can only handle entire sectors at once. So for a write the disk first does 2 reads. Then it does 2 writes.
In the aligned situation it only needs to do one write. Because there are no bytes left untouched in a particular sector.

That is the situation when you use partitions. Luckilly my disk is used in an LVM volume group and I just gave the entire disk to LVM. I did not create a partition. There is no need to, when you use LVM.
What does this mean for alignment and performance for this disk? It means I'm probably ok.
Probably? I couldn't find any information on this on the net. Which means I want to go and take a look at the actual data on the disk, and at the LVM source code to figure out how it stores the data on disk.
A quick performance test on the disk also showed that there is most likely no alignment problem. Otherwise the write performance would have been much lower.

More will follow....

"windows dynamic disks - mirroring" useless

I run Windows 7 on my desktop. I'm a big Linux fan but the Linux desktop experience is IMHO still not there yet. Although I must admit the last time I tried is 3 years ago and things have improved. Guess I'm going to give ubuntu a shot soon :)

Because my aversion to data loss and downtime I run Windows on a pair of disks in a mirrored setup. One of the disks is pata and the other is sata. It's what I had lying around. Why spend money when you have parts on the shelf? My on-board ata controller doesn't support setting up a mirror over pata and sata. They must be on the same type of connection. So instead I configured windows to do the mirroring for me. Most on-board hardware 'raid' controllers are just a bios call that's handled on the cpu anyway.
Therefore there shouldn't be that much of a performance difference between doing a mirror action in the OS versus doing it via the bios on the cpu. That's the way it is under Linux in my experience.
In Windows 7 (and older versions of windows) disks need to be converted to a special partition format to be able to partake in a raid set. Be it mirror or otherwise. This can be done through the  'computer management' application, in the 'storage' section. So I set it up and all was fine. It spent some time copying the data from the source to the mirror disk and all was done. Everything is fine, right?
Not so. Every reboot the sync starts over. All of it. Not just the bits that are out of sync, everything.
That doesn't happen under Linux. There it keeps track so that when you reboot the raid is still in sync.
Not the same for Windows 7.
So what? Well it means when I boot windows my disk performance goes down the drain for a couple of hours until it has re-synced everything. And that sucks.
There are advantages too... if I was using a 'fake' raid controller and the controller would die, I run the risk of losing my data if the configuration is in the raid controller and it uses some propriatary format on the disks. Windows mirror will maintain integrity. you could even move a windows disk to a new machine and it will run. no configuration needed. Nice, yet not what I'm looking for.

Time to kick Windows mirroring out of the door and find some other way to do this.

But... you can't. Not while it's syncing. Crazy right? So first I have to wait for an entire sync to be complete, only to break the mirror and remove the sync configuration.
Too bad there are no alternative filesystems for windows. It's just fat or ntfs. Linux has many filesystems. Another reason to just choose open source. Or apple ;-)

harddisk firmware warnings

A while ago I bought 2 new harddisks. They were Samsung EcoGreen HD204UI's. Great value in my opinion. Since I don't trust harddisks I put ZFS on them. That will give me a bit more faith to keep my data. Faith... no guarantees. Software bugs could still kill all data on there. But I disgress.
How big was my surprise when, a couple of weeks later, I just happened to read a forum post about these drives having some problems when using smart functionality (check here for details: http://sourceforge.net/apps/trac/smartmontools/wiki/SamsungF4EGBadBlocks). I have these drives running under Linux. Smart checks my disks periodically. If data would happen to get written to the disk just at the moment that a smart command would be issued, I'd end up with a bad sector. WTF!
I was able to find the patch that Samsung put on their websites by googling for it. There was no link to it on the samsung product page for these drives. These was no announcement. I patched the drives and they have been humming happily ever since.

But what about my other drives? Are they ok? Do they have problems that warrant a firmware update? How do I find out when drive makers don't willingly publish such information in fear of such things hurting sales? I have no idea. The most logical way to find out is to run into trouble. I won't spend the time googling the internet every month to try and find this out. I should be notified. Isn't that what 'registering your product' is for? Registrations have never done anything for me.

Hard disk manufacturers should step up and provide at least some way of getting notified. If they don't, someone may do it for them.

harddisk data safety

When I use a harddisk I know I'm trusting my data, which I care about, to a devise that is guaranteed to give errors.

The rate at which it gives errors is listed in the harddisks specifications. For a 2TB disk the non-recoverable read error rate can be 1 sector in 10^15 bits. That's a good buy. But there are also disks out there that offer a higher error rate, at 1 in 10^14. What does that mean? It means the disk has more errors than one in 10^15 but most can be corrected (by re-reading, error correction calculations, etc). Once in every 10^15 bits it can't recover. With 8 bits per byte that translates to one sector in 133,7TB. That sounds like veeeeery little. But is it? If I read the disk 60 times, I'm sure to get a read error. One sector is still usually 512 (but this will very soon change to 4k). So with 60 full disk reads you risk losing 512 bytes of data. That's nothing, right?
Think again. If this happens to be in the middle of a file you store your adminstration in, you had better have a backup. You have backups right? And not on the same disk, right?

The situation gets much worse when disks get larger. When you but a 3TB disk you only need to do 40 full reads. 4TB disks and you're down to 30. See what I'm getting at? If you have a 10^14 error rate disk of 2TB you get a read error every 6 full reads. That's data loss people. Or is it? The read error was 'unrecoverable', but does that mean that all retries will fail also? I'd love some feedback on that.

Raid is one way to solve this. But beware: If you use raid-5 and even raid-6 you're making the problem worse! If one disk in your raid-5 setup fails, you replace it and a re-build takes place. That's a full read of your disks! That means if any read error occurs then *poof* your data is gone. Cheap read controllers then stop the rebuild and you have just lost everything.
Using raid-1 or raid-10 is a much better solution if you value your data. But more expensive too.

However, most of the time harddisks are not fully used. Especially new disks are mostly empty initially. So if we have this error rate but duplicate the data on disk? If all your data is on the disk twice then the chance of your data being lost is much lower. And lower even when even more copies are made. To do this yourself (saving 6 copies of the same file on your filesystem) is a lot of extra work.

There are filesystems that do this, somewhat. For instance on ZFS you can configure the number of copies that should be made. But I'm looking for something more intelligent. I'm calling out to harddisk makers to make as many copies of the existing data as the free space will allow. Then when an error occurs it can just go look at one of the copies and repair the problem area. Problem solved. Simple idea really. Can it be done? I hope so :)