15k rpm RAID5 or 10k rpm RAID10 for the win?

[Vimeous]

Something of a 'what would you spec?' question this one:

I am speccing a new VMware host and as usual it's budget vs function. The two harddrive configs listed in the title cost similar money for similar formatted capacity but I need a second, third and more opinions to clarify my own thinking.

So you have one VMware host using local (on-board) storage. The intention is to put multiple mission-critical VM's on the system with varying disk performance requirements. Included will be a print, exchange, MS SQL, AD and file (head only) servers running Win2k/2k3/2k8, Ubuntu Server and likely CentOS in the future.
Total VM storage needed is 900Gb.

Ignoring cost your choices are:
4x 300Gb 3G 15k rpm 3.5" SAS in RAID5
or
4x 450Gb 3G 10k rpm 3.5" SAS in RAID10

Which is fastest?
Which would you choose?
What would the impact be of switching from 3.5" to 2.5" drives?
Would switching from 3G to 6G drives have a significant performance benefit?

Traditionally I've always plumped for RAID10 on mission-critical systems but speed and reliability also come into the equation so I'm torn here!

[oolon]

raid 10 as reading and writting will be faster. Reading there is a choice of 2 drives, writting 2 reads do not have to be made for the one write. Make sure it is raid 10 not raid 1 concat which would still be better than raid 5. This more than makes up for the extra 5000 rpm. The failer rate on 15k drives is much worse than 10k ones. A big write cache on the controller could improve the raid 5 however you NEED battery backup for that, raid 10 is safer. 2.5 inch drives fail less at high speed as the edge wobble on 15k 3.5 is a problem. 3-6 gig sas... no difference if your not using an expander none of the drive will max a 3g link.

[Moby-Dick]

RAID5 write penalty can be a real pain - depends on what your i/o profile is like though.

as you are only using direct attached storage , you can possibly be a bit more flexilble. ( why are not you using shared storage / clusting ESh hosts ? )

why not tier it ?

SSD array for OS volumes
RAid 5 for SQL / NTDS / Exchange Data
Raid 10 for Logs ?


its all about the spindle count - personally I would be concerned that you are not going to be able to drive enough IOPS out of 4 spindles to drive that sort of workload if its anything substantial.

[oolon]

its all about the spindle count

And contention! If you can separate your system so its reading from one set of drives and writing to a different set this cuts down on the number of seeks required greatly increasing through put of data, relative to the same number of drives in one super disk set. You also have the option here of 2 sets of raid 1, instead of the raid 10.

[Moby-Dick]

And contention!

yep , hence why seperation of logs/ data and backup is a good thing the log writes are sequential writes , but you are much more likly to have random reads on the data ( although that said it can really depend on the type of database you are hosting )

[Vimeous]

Very interesting.
I hadn't realised edge-wobble was an issue and overlooked the extra diameter (2.5" vs 3.5") impact on the speed at the outer edges of the disc. Apparently some 3.5" drives use smaller diameter discs to avoid wobble-issues.

We're not using shared storage because the business won't stomach the cost - yet. I'll be ploughing that furrow once again later in the year. It'll be cheaper in the long run to do it now but that's not currently an option.

As for tiering - cost again.
We run hot spares so for each different drive type we add an extra drive. This with mixed drive type/sizes leads to chassis size issues.
We are fine for rack space so a 5 or 6U chassis is fine but with extra drives you end up running 2.5" over 3.5" which increases cost. The logic of adding a separate shelf is defeated when you consider our relatively small space requirement vs the cost of an additional DAS chassis.

SSD's £/Gb is still far too daft to be feasible (Dell charging £1.4k/100Gb SSD list atm).


Looking at the expected workload I wouldn't describe any of the reads and writes as 'large'. There's little scope for more than 100 users making Exchange virtually a non-event in IOPs terms (calculations based on info from here) for example. Any other databases will be minor because the main production db won't sit on this VM host.

Looking at the available servers and drives I can see 2/3 reasonable options:
1. 2U Server; 3.5" drives; 2x300Gb RAID1 + 3x300Gb RAID5 + Hotspare. (0.9Tb usable)
2. 2U/5U Server; 2.5"/3.5" drives; 2x300Gb RAID1 + 4x300Gb RAID10 + Hotspare (0.9Tb usable)
3. 5U Server; 2.5" drives; 2x146Gb RAID1 + 4x146Gb RAID10 + 4x146Gb RAID5 + Hotspare (0.9Tb usable)

There are also connatations using 3.5" 450Gb or 600Gb SAS drives but unless they store data at a higher density (vs more platters) they won't offer a performance gain but cost significantly more.


Frankly it'd be a lot easier speccing a 2-shelf (SAS/SATA) SAN!

[Moby-Dick]

storage wise - have you though of getting somethign like this ? ( its on the vmware HCL )
http://www.drobo.com/Products/droboelite.php

surely that can't be too bad for budget ?

[oolon]

Frankly it'd be a lot easier speccing a 2-shelf (SAS/SATA) SAN!

Indeed! I have an fibre channel MSA1500 storage shelf myself. One thing you could use to help sell the san is you have to have alot less "hot spares". So many nice things you can do with shared storage and VM hosts.

Boot from san is nasty at times, I would always suggest minimal OS mirrored pairs, swapping to sans is a waste so try to avoid it.

[Vimeous]

The Drobo looks interesting. Others have suggested similar products but I've not had chance to give them a properly considered appraisal. Definitely worth looking at though.

Boot from SAN nasty? I can imagine an unstaged reboot of a VMware host with a stack of VM's might cause a headache but what have you seen in reality?
When it comes to SAN's I've looked at Equalogic who the Lefthand reseller happily bashed, NetApp whos' prices made me cry, MSA's which seem to be viewed as ok just not quite 'the biz' and I want to look at NexSAN if only because 42Tb in a single 5U box at £21k seems like pretty good value - even if it's a teensie bit large for our purposes!

[oolon]

Boot from SAN nasty?

It not nasty for any guests, I am talking about the host booting from the SAN, if you have something like a fiber channel one you need to make sure the disk block is presented to the machine, zoning on your switch is correct, the disk block uses the correct WWN for masking from the server otherwise everything on the fabic will see it, you then need to tell your card which lun to use and the WWN of that. All with NO tools on the host, because you have not done the install yet so fault finding can be a pain, as the tools are host based. Then you have to make sure your host has the right driver loaded so it can see the disk at INSTALL and BOOT. Then of course what happens if you want to dual path to your SAN, add another lair? As I said... Nasty! Infact the best way to do it for a unix system is to install to local disk then migrate it onto the san.

Guests are not a problem, you see the disk space on the host, you present it to the guest.. otherwise fault find.. job done.

Not sure about any of those brands, been using more mainstream ones.
IBM (high end)
IBM (midrange) rebadged LSI sans actually, same as SGI, SUN and Apple
Hitachi (High end) Tagma store
Hitachi (Midrange) AMS/WMS
HP (entry level) MSA

My favourettes are the Hitachi Tagma store and the IBM midrange sans, both were very flexable, but perhaps a bit over budget!

[Moby-Dick]

The Drobo looks interesting. Others have suggested similar products but I've not had chance to give them a properly considered appraisal. Definitely worth looking at though.

Its not too bad to be honest - for your scale of stuff it should be able to provide what you need.

Boot from SAN nasty? I can imagine an unstaged reboot of a VMware host with a stack of VM's might cause a headache but what have you seen in reality?

its called a boot storm - however when you have ESX datastores on a SAN to take advantage of vmotion etc, boot form SAn is unaviodable for those - I woudln't nessesarily boot the host form San though

When it comes to SAN's I've looked at Equalogic who the Lefthand reseller happily bashed, NetApp whos' prices made me cry, MSA's which seem to be viewed as ok just not quite 'the biz' and I want to look at NexSAN if only because 42Tb in a single 5U box at £21k seems like pretty good value - even if it's a teensie bit large for our purposes!

Vendor FUD is rife , particularly in the storage world.

Equalogic and Lefthand are both pretty nice - with a good installed userbase.
The older MSA's are more like the traditional SAN architecture , while the Lefthand / EQL kit expansds cache / controllers with disk. for a small environment unless you are really pushing it I doubt you'd notice the differerence.

Not played with the NexSAN stuff so can't comment - but we looked at a similar device form HP - a DAS shelf that could fit an obscene ammount of drives. ( MDS600 / 6000 I think )

Having been through a round of choosing SAN's at work I'm still sulking a bit because they wont buy me an EMC V-Max ( it was about $500k more than we wanted to spend - still that's what happens when you make strategic decisions with price as your main factor ! )