Choosing a NAS?

[jim]

So my Windows Home Server is filling up - I've got 4x2TB drives in there at the moment, leaving one spare bay for backups, and WHS can't accept 3TB drives to my knowledge, not without hacking it anyway. That gives me 4 options, as far as I can see:

1) Buy an additional 5-bay hub for my server for £300
2) Upgrade to Home Server 2011 and use a third party equivalent to DE
3) Build a FreeNAS box
4) Buy a NAS

I'm not really keen on doing the first, because it just delays the problem, and I'm not keen on the second either - don't like the idea of using third-party tools when it comes to data security.

That leaves options 3 and 4. So, two main queries.

1) Does ZFS in FreeNAS provide many benefits to a home user for a a home server of around 10TB?
2) What sort of RAID system would I be looking at? I've always been concerned that with RAID 5, a failure during rebuild is incredibly likely, and I also know that they tend to take absolutely hours? I have 5TB of raw data at the moment, so a RAID1 or RAID10 system would take six 2TB drives to run effectively which seems a bit overkill.

I'm guessing also that it starts to get incredibly expensive...

[alsenior]

i have always been a bit weary of zfs on bsd as is never seams to be all that stable. i was planning to use it on my server but when i was using it it kept eating data. that was a while ago and things have probably changed.

i use jfs ontop of linux md5raid on my fileserver managing around 8tb across 2 arrays with 3 2tb drives and 5 1tb drives. linux md gets faster the more drives you add. my 5 disk array can pull 250Mb/s with some first gen green drives which is mighty impressive . with rebuilds it works at around 60-70mb/s which for 2tb drives works out at about 480 minutes. you will only get another drive failure if you are unlucky. you also have the option of raid 6 if you are using 6 or more drive it gives you the space of 4 of the drives on 6 drives. you can also create an array in a degraded state and add the missing drives as you migrate them

[Hawker1986]

If your happy with the WHS OS then why not build a whs rig in a case which can accomodate a lot more HDDs, you wouldnt need any new drives, presumably you could use the serial from your current copy? Something in a case which can accomodate 10+ disks?

Hawker

[badass]

Unfortunately you're a bit stuck here.
http://subnetmask255x4.wordpress.com...-impacts-raid/
Read this and compare it to your setup.
Yep, you're looking at a significant chance that you suffer an unrecoverable bit error during a rebuild operation.
ZFS was designed to work around this problem and the problem of silent bit errors - that is where bit errors occur just perfectly to appear valid to the HDD's ECC systems, despite being the wrong data.
There's only one problem with ZFS.

It's cr*p and simply doesn't work for small scale setups in the real world. I have lost data as a result of putting my trust in Nexentastor and ZFS. Look at this thread
http://forums.hexus.net/shopping-ret...ver-100-a.html

I'm sure some people here will rush in to defend the almighty ZFS and point out some case studies where it's saved the world and point out it's designed from the ground up to keep your data as safe as possible, but I'll just point them to all of the posts here where people are having nothing but trouble with it including data loss.

I think the best thing you can do is look into RAID6, forget drive extender and try out the various free NAS appliances.
FreeNAS is one, Openfiler is another. I can't remember the names of others off the top of my head.

I can't help but think it would be nice to see ZFS on linux but it's not likely soon as the Open source licenses of ZFS and The linux kernel are incompatible.
I suspect that with the far larger developer base on Linux, ZFS bugs would get ironed out sooner than they currently are.

[jim]

If your happy with the WHS OS then why not build a whs rig in a case which can accomodate a lot more HDDs, you wouldnt need any new drives, presumably you could use the serial from your current copy? Something in a case which can accomodate 10+ disks?

Hawker

Primarily because I have a prebuilt Tranquil server that I'm very happy with - I'm loathe to rip it apart, hence my suggestion of 1) Buying a Tranquil extender, which would give me a total capacity of 9 disks, plus one for backup.

It's just that £300 seems like a lot of money to spend for that capability.

I guess I could theoretically build a new WHS with 10+ disk capacity and run them in tandem (licenses aren't a problem) but the 2TB disk problem is only going to become more of a pain in the neck as the years go by.

[jim]

Unfortunately you're a bit stuck here.
http://subnetmask255x4.wordpress.com...-impacts-raid/
Read this and compare it to your setup.
Yep, you're looking at a significant chance that you suffer an unrecoverable bit error during a rebuild operation.
ZFS was designed to work around this problem and the problem of silent bit errors - that is where bit errors occur just perfectly to appear valid to the HDD's ECC systems, despite being the wrong data.
There's only one problem with ZFS.

It's cr*p and simply doesn't work for small scale setups in the real world. I have lost data as a result of putting my trust in Nexentastor and ZFS. Look at this thread
http://forums.hexus.net/shopping-ret...ver-100-a.html

I'm sure some people here will rush in to defend the almighty ZFS and point out some case studies where it's saved the world and point out it's designed from the ground up to keep your data as safe as possible, but I'll just point them to all of the posts here where people are having nothing but trouble with it including data loss.

I think the best thing you can do is look into RAID6, forget drive extender and try out the various free NAS appliances.
FreeNAS is one, Openfiler is another. I can't remember the names of others off the top of my head.

I can't help but think it would be nice to see ZFS on linux but it's not likely soon as the Open source licenses of ZFS and The linux kernel are incompatible.
I suspect that with the far larger developer base on Linux, ZFS bugs would get ironed out sooner than they currently are.

Great post, thanks very much for the info.

That information in your first post is exactly what I'm concerned about - that's why I was interested in ZFS. So if that's out of the window, there seems very little point in building a 10TB RAID5 array to me.

It's probably going to have to be RAID6 then. I want, ideally, 10TB of storage, so RAID10 would require another six 2TB drives for ten in total, which is insane. RAID6 would require another three 2TB drives for seven in total, which is more reasonable.

What kind of horsepower would I need to run a speedy RAID6 array over up to ten drives? I'm guessing I would really have to build it myself rather than getting an off the shelf NAS?

Just another thought I'm throwing out there - would it be possible, theoretically, to run two NAS units on JBOD with some sort of failover system as an alternative to RAID?

[Champman99]

I don't know much about NAS devices but for my personal storage after being very dissapointed that new WHS wouldnt have DE I built a system based on Amahi.

http://wiki.amahi.org/index.php/Main_Page

Obviously it's nowhere near as slick as WHS being linux but I've found it perfect for my needs. "Greyhole" is the part of it that works like DE.

http://wiki.amahi.org/index.php/Greyhole

It has been perfect for my needs, although its always a bit scary when you want to update it or something.

[Splash]

Dependent on budget you could always build a couple of Openfiler boxes and run them as a cluster. My experience with ZFS pretty much matches that of both Badass and Alsenior though - the idea behind it is brilliant, but it's just too flaky on consumer grade kit for my liking.

As for speed - software raid (like mdraid) is always going to be slower and take longer to build than something on a hardware RAID controller, but again this will be massively influenced by your budget - a hardware card that will do RAID6 with the number of ports you want will be expensive, especially as it'll likely need a BBWC.

If it's any use to you my experience with Openfiler 2.99 installed to a USB key in my Microserver has been pretty positive once it was built (bear in mind I've spent around 3 weeks testing NexentaStor, FreeNAS, Ubuntu Server and Openfiler before coming to this conclusion). This only gives you 5 discs (if you put one in the 5.25" bay at the top) which will give you around 9Tb in RAID6 (if using 3Tb discs, I'm using 2Tb discs) so this is probably too small for you, but the alternative would be to buy a couple of units and run them as either JBOD or RAID0, then cluster the 2 units. I'd find that option a little tempting if I were in your shoes, especially given the current £100 cashback per unit at the mo.

[alsenior]

Great post, thanks very much for the info.

That information in your first post is exactly what I'm concerned about - that's why I was interested in ZFS. So if that's out of the window, there seems very little point in building a 10TB RAID5 array to me.

It's probably going to have to be RAID6 then. I want, ideally, 10TB of storage, so RAID10 would require another six 2TB drives for ten in total, which is insane. RAID6 would require another three 2TB drives for seven in total, which is more reasonable.

What kind of horsepower would I need to run a speedy RAID6 array over up to ten drives? I'm guessing I would really have to build it myself rather than getting an off the shelf NAS?

Just another thought I'm throwing out there - would it be possible, theoretically, to run two NAS units on JBOD with some sort of failover system as an alternative to RAID?

A single core on a Opteron 2214HE can put through 4231MB/s using linux-md

[    1.502511] raid6: int64x1   1805 MB/s
[    1.672512] raid6: int64x2   2367 MB/s
[    1.842508] raid6: int64x4   1605 MB/s
[    2.012505] raid6: int64x8   1613 MB/s
[    2.182512] raid6: sse2x1    3045 MB/s
[    2.352507] raid6: sse2x2    4064 MB/s
[    2.522504] raid6: sse2x4    4231 MB/s
[    2.522506] raid6: using algorithm sse2x4 (4231 MB/s)

a core2 or a Athlon x2 should be more than sufficient under linux md you're only problem is finding a motherboard with enough ports to host everything. i recommend a Dell sas 5 i/r as there good simple controllers witch support sata hotplug

[jim]

If it's any use to you my experience with Openfiler 2.99 installed to a USB key in my Microserver has been pretty positive once it was built (bear in mind I've spent around 3 weeks testing NexentaStor, FreeNAS, Ubuntu Server and Openfiler before coming to this conclusion). This only gives you 5 discs (if you put one in the 5.25" bay at the top) which will give you around 9Tb in RAID6 (if using 3Tb discs, I'm using 2Tb discs) so this is probably too small for you, but the alternative would be to buy a couple of units and run them as either JBOD or RAID0, then cluster the 2 units. I'd find that option a little tempting if I were in your shoes, especially given the current £100 cashback per unit at the mo.

Okay, that might be an option then. How would you cluster the 2 units? Is it fairly seamless?

EDIT: Actually, on second thoughts, I might just split everything down. Whilst I need 10TB of storage, there's no reason why it all has to reside on the same device.

There's no reason why I couldn't have a significant proportion of that on the WHS and just shift out the rest of the storage to a second server. And theoretically, a third server. Each one could handle a different type of data, so \\backupserver \\mediaserver etc. This could work

[Splash]

Okay, that might be an option then. How would you cluster the 2 units? Is it fairly seamless?

EDIT: Actually, on second thoughts, I might just split everything down. Whilst I need 10TB of storage, there's no reason why it all has to reside on the same device.

There's no reason why I couldn't have a significant proportion of that on the WHS and just shift out the rest of the storage to a second server. And theoretically, a third server. Each one could handle a different type of data, so \\backupserver \\mediaserver etc. This could work

That's an idea. I've not actually used clustering in Openfiler before, so I'm just knocking up a couple of VMs to test it with now. I'll report back with the results - even if you decide not to go down that route it might be of use to someone else in the future (and I'm pretty bored, and hayfever is killing me at the mo, so I can't just go out and play footy orr something)

EDIT - howto for setting up clustering at http://access.rzschernsdorf.de/downl...owtoforge.html

Would you actually need clustering, or would rsync from one to the other cover your requirements?

[b0redom]

Splash, do you have any performance stats on Openfiler vs FreeNAS? I really need to get off OpenSolaris asap, but I really do like the security blanket of ZFS.

[jim]

EDIT - howto for setting up clustering at http://access.rzschernsdorf.de/downl...owtoforge.html

Would you actually need clustering, or would rsync from one to the other cover your requirements?

If I was going to do it, it would definitely be clustered. All my backups of server data are offline for safekeeping, I don't have any live backup systems.

However, I've decided that I'm definitely not going to. To cut a long story short, all my data is built up of separate major directories, which are at most 2TB each. Since there's no major difference between \\server\media and \\servermedia\, unless a directory grows larger than the maximum capacity of a server, it doesn't seem worth investing any time or effort in creating massive file servers. If the HP server can hold 9TB of data at the most, that's probably a very long time away. And by then we'll have larger hard drives anyway.

My current plan is to get the HP server, stick 2x2TB drives in there for a 4TB JBOD or similar, shift 3TB of data from the WHS to the HP server, and remove the no longer required 2TB drive from the WHS.

That gives me 1TB of breathing space on both servers, plus a spare 2TB drive. All the data that needs to be more or less 24x7 will be duplicated on the WHS, and the directories I can manage without will be on the HP. If they go down, I'll stick the replacement 2TB drive in there and rebuild from backups, and put up with the downtime.

How hard can it be?!

[Splash]

Splash, do you have any performance stats on Openfiler vs FreeNAS? I really need to get off OpenSolaris asap, but I really do like the security blanket of ZFS.

Sadly not - I have 5 disks in a RAID5 (it's a home lab box, not storing backups so if I lose the array during a rebuild it's not the end of the world) in there on Openfiler, and I only have one more 2Tb disk as a spare in case of failure. That said... if you let me know what you need benchmarking (filesystem comparisons, protocols, transfer types) I'm off work today and tomorrow - I can always rebuild to FreeNAS8 and do a compare and contrast for you.

EDIT - just moving the couple of VMs that I have stored on Openfiler so I can do this, this is going to take a couple of hours I think

[Splash]

If I was going to do it, it would definitely be clustered. All my backups of server data are offline for safekeeping, I don't have any live backup systems.

not entirely sure what you mean here - from my understanding of what you need then replicating the data from one server to another using a scheduled rsync would seem to be pretty much exactly what you need.

However, I've decided that I'm definitely not going to. To cut a long story short, all my data is built up of separate major directories, which are at most 2TB each. Since there's no major difference between \\server\media and \\servermedia\, unless a directory grows larger than the maximum capacity of a server, it doesn't seem worth investing any time or effort in creating massive file servers. If the HP server can hold 9TB of data at the most, that's probably a very long time away. And by then we'll have larger hard drives anyway.

With the extra info that you give this seems like a very sensible move. Note that my statement of 9Tb being the maximum storage was made up based on 5 3Tb disks with a RAID6 array setup - RAID5 would give you 12Tb (before overheads), and if/when we have 4, 5 or 6 Tb disks obviously the calculation would be affected.

[jim]

not entirely sure what you mean here - from my understanding of what you need then replicating the data from one server to another using a scheduled rsync would seem to be pretty much exactly what you need.

You mean that if I had all of my data on a 6TB JBOD server, I could just sync it over to a second 6TB server and not bother with any RAID at all?

I get that, but surely that doesn't provide redundancy? If boxes are configured to look at \\servera\media for media files, how would they know to look at \\serverb\media if server a went down?

With the extra info that you give this seems like a very sensible move. Note that my statement of 9Tb being the maximum storage was made up based on 5 3Tb disks with a RAID6 array setup - RAID5 would give you 12Tb (before overheads), and if/when we have 4, 5 or 6 Tb disks obviously the calculation would be affected.

Yeah. Considering these have been growing for about the last 5 years, and still only take up that much space, I don't think they would ever outgrow a 4/5 bay server.