Project: Perfect vSphere 5 Whitebox!
I have been entertaining HP ProLiant MicroServer N36L for nearly a year now. Great machine for the money and with cost of around £120 (after the cashback) it was an absolute bargain at the time! Box has been upgraded to 8GB of DDR3 RAM (maximum the mobo can take) and its running ESXi 4.1 U1 absolutely fine. Disk space wise, there is only 30 GB Vertex SSD for few VMs (and .vswp files), rest of the storage needs is provided by QNAP TS-509 NAS (by means of NFS and iSCSI) This setup has been absolutely flawless so far but there is simply not enough RAM and CPU power for my needs (or rather my VMs). CPU Ready is going through the roof quite often due to AMD Athlon II Neo 1.30GHz processor which is just slightly better performer compared to Intel Atom range. 8GB of RAM is tight and ESXi was paging like mad to .vswp files hence why I put them on SSD which helped only to certain extent. At the end I kinda had enough and decided to build a custom server which would address all of the issues above.
Here is what I came up with:
Processor: Intel Xeon X3450 2.66GHz with HT/VT-x and VT-d
Processor Cooling: Corsair CWCH100 Hydro Series H100 Cooler
Processor Cooling Fans: 2 x Noctua NF-P12
Motherboard: Supermicro X8SIL-F-O Server Board
RAM: Kingston 4 x 8GB Memory [KVR1333D3Q8R9S/8G]
Case: Lian-Li PC-V350B
Case Noise Dampening: AcoustiPack LITE (APL) Multi-Layered Soundproof Material
Case Backplate: Custom backplate to incorporate moving PSU to the right and adding 120mm exhaust fan
5.25" Drive Bay Cooling: Evercool Armour ATX HDD Cool Box HD-AR
5.25" Drive Bay Cooling Fan: Noctua NF-R8-1800
Case Exhaust Fan [Back]: 1 x Noctua NF-P12
Case Exhaust Fan Guard: 120mm Standard Wire Case Fan Guard Grill [Black]
Power Supply: Be Quiet! BN180 L8 430W Modular PSU
Storage 1: Samsung 830 256GB SSD (main datastore)
Storage 2: Seagate Barracuda 2TB [ST2000DM001] (second datastore)
Storage 3: Vertex 1 30GB SSD (.vswp datastore)
Storage 4: Patriot Extreme Performance Xporter XT Rage 8GB (local storage for ESXi)
Storage Adapter Bracket: SilverStone SST-FP55B (allows 1 x 5.25" and 2 x 2.5" in one 5.25" slot!)
Network 1: Onboard Dual Intel 82574L Gigabit Ethernet Controllers
Network 2: HP NC360T PCI Express Dual Port Gigabit Server Adapter [which effectively is Intel PRO/1000 PT Dual Port NIC]
Network 3: Intel Ethernet Converged Network Adapter X520-DA2, 10GbE, Dual Port
RAID Controller: IBM ServeRAID M1015 [which kinda is OEM version of LSI 9220-8i]
ODD: Toshiba/Samsung TS-H653 20x DVD±RW DL SATA Drive
More when I get some pictures done (or rather work!)
Re: Project: Perfect vSphere 5 Whitebox!
UPDATE #1
Few additions/changes to the list above:
1) 5.25" drive bay cooling is now provided by Evercool Armour ATX HDD Cool Box HD-AR:
https://dl.dropbox.com/u/1471771/For...r03%5B1%5D.jpg
If you have/had V350 case you know that 5.25" devices can be mounted on either side but not on both (unless they're very short) I have my ODD and bracket holding 2 x SSDs and 1 x 3.5" HDD on one side but there is virtually no air circulation hence the 'cool box' above. I had to modify it slightly to fit though (was too long and sticking out a bit) - thanks to cheap and cheerful hacksaw that was at hand it was a few minute job! I will grab a picture when the light is better so you guys can see.
2) Stock fan in 'cool box' was replaced by Noctua NF-R8-1800 which should keep the noise down.
3) Andy from WCuk was generous enough to laser custom backplate to incorporate moving PSU to the right and adding 120mm exhaust fan on the left. V350 has, again, virtually no exhaust fans which is a pain. Here is what I'm on about:
Rear, stock, view of V350:
https://dl.dropbox.com/u/1471771/For.../v350_rear.jpg
Custom bracket fitted and the whole thing riveted together (bracket is part of removable motherboard tray):
https://dl.dropbox.com/u/1471771/For...t/DSCN5509.JPG
and the complete rear view:
https://dl.dropbox.com/u/1471771/For...t/DSCN5505.JPG
Much nicer, isn't it? I was never going to use the 2 x 3.5" drive bays so it was very natural thing to do.
4) Storage for installing ESXi is now provided by Patriot Extreme Performance Xporter XT Rage 8GB - motherboard above has internal USB port so we are all good.
As far as the case rear modifications are concerned it was a question of drilling out rivets and hacksawing the back to free up space for the new backplate:
Stock:
https://dl.dropbox.com/u/1471771/For.../Rear_v350.jpg
Modded:
https://dl.dropbox.com/u/1471771/For...t/DSCN5510.JPG
Not the best of cuts, I must admit, but £5 hacksaw from B&Q can only take you so far...
Applying noise dampening kit was next... (on pair with braiding wires in my book, not a fan of either to be honest!)
Case partially done, I'm still working on 5.25" drive bays:
https://dl.dropbox.com/u/1471771/For...t/DSCN5514.JPG
https://dl.dropbox.com/u/1471771/For...t/DSCN5515.JPG
https://dl.dropbox.com/u/1471771/For...t/DSCN5516.JPG
https://dl.dropbox.com/u/1471771/For...t/DSCN5517.JPG
https://dl.dropbox.com/u/1471771/For...t/DSCN5518.JPG
https://dl.dropbox.com/u/1471771/For...t/DSCN5519.JPG
Picture quality isn't great but without studio lights there is only so much you can do!
Leftovers:
https://dl.dropbox.com/u/1471771/For...t/DSCN5476.JPG
Here is rest of the parts (not a complete list though..):
https://dl.dropbox.com/u/1471771/For.../DSCN5477_.jpg
https://dl.dropbox.com/u/1471771/For...t/DSCN5478.jpg
https://dl.dropbox.com/u/1471771/For...t/DSCN5479.jpg
https://dl.dropbox.com/u/1471771/For...t/DSCN5480.jpg
https://dl.dropbox.com/u/1471771/For...t/DSCN5482.jpg
https://dl.dropbox.com/u/1471771/For...t/DSCN5484.jpg
https://dl.dropbox.com/u/1471771/For...t/DSCN5485.jpg
Re: Project: Perfect vSphere 5 Whitebox!
UPDATE #2
Small ish update. I have to work during the evenings to keep this build going so pictures won't be of any great quality. Unfortunately my usual 9 to 5 job is keeping me busy more than I would have liked. Few things have been done since last update, here is the list:
Evercool Armour has been chopped as I mentioned in my first update - here is what it looks like now:
https://dl.dropbox.com/u/1471771/For...1/DSCN5551.JPG
https://dl.dropbox.com/u/1471771/For...1/DSCN5555.JPG
[font=Open Sans', Helvetica, Arial, sans-serif]Noctua NF-R8-1800 [/font]fan fitted:
https://dl.dropbox.com/u/1471771/For...1/DSCN5557.JPG
Looking alright I think. I'm still waiting for 80mm fan guard (ordered) to keep wires out of the way.
Here is how the whole thing inside from inside the case:
https://dl.dropbox.com/u/1471771/For...1/DSCN5560.JPG
5.25" drive bay external view:
https://dl.dropbox.com/u/1471771/For...1/DSCN5562.JPG
It should be enough to extract any trapped hot air generated by my hard drives. I think.
In stock V350 there is space for two 120mm fans at the front of the case:
https://dl.dropbox.com/u/1471771/For...21/v350008.jpg
My idea of utilizing this space is to fit Corsair H100 all-in-one water cooling unit AND the 120mm fans. For those unfamiliar, Corsair H100 has 240 radiator:
https://dl.dropbox.com/u/1471771/For...100_rad_v2.png
Bracket originally holding just fans needed attention to make this happen as mounting holes on Corsair H100 only lined up on the first 120mm fan (left). Holes on the right didn't lined up at all so I had to widen them a bit:
https://dl.dropbox.com/u/1471771/For...1/DSCN5567.JPG
Bracket, two Noctua NF-P12 120mm fans and the H100 all screwed together as one unit:
https://dl.dropbox.com/u/1471771/For...1/DSCN5586.JPG
https://dl.dropbox.com/u/1471771/For...1/DSCN5589.JPG
It fits pretty well. Now, will it fit inside the V350? Answer is - yes it will!
https://dl.dropbox.com/u/1471771/For...1/DSCN5569.JPG
I have to say though, its an extremely tight fit - space left on both left and right hand side is just enough to close the case together:
https://dl.dropbox.com/u/1471771/For...1/DSCN5573.JPG
https://dl.dropbox.com/u/1471771/For...1/DSCN5577.JPG
Sweet.
Silverstone bracket to hold my two SSDs and 2TB drive was next - SSDs in place:
https://dl.dropbox.com/u/1471771/For...1/DSCN5599.JPG
2TB hard drive temporarily mounted upside down:
https://dl.dropbox.com/u/1471771/For...1/DSCN5604.JPG
And the whole bracket "correct" way around:
https://dl.dropbox.com/u/1471771/For...1/DSCN5603.JPG
I have decided to use the Lian-Li 5.25" solid bezel that came with the case and not to expose front of the Silverstone bracket as the finish didn't really match up. Optical drive got some attention as well, I had the stealth Lian-Li bezel for ages now so it was perfect time to use it! 5.25" external view once the bracket and bezels was mounted:
https://dl.dropbox.com/u/1471771/For...1/DSCN5597.JPG
That's it for now. More in a day or two...
Re: Project: Perfect vSphere 5 Whitebox!
UPDATE #3
I have order some rubber noise absorbers for the 120mm exhaust fan and the PSU:
https://dl.dropbox.com/u/1471771/For...1/DSCN5609.JPG
Had time to try only the PSU one and it fits perfectly. I'm hoping it will prevent any vibrations from being transferred onto the case.
https://dl.dropbox.com/u/1471771/For...1/DSCN5612.JPG
Newest addition to this build - Intel X520-SR2 10GbE PCIe Server Adapter:
https://dl.dropbox.com/u/1471771/For...1/DSCN5613.JPG
https://dl.dropbox.com/u/1471771/For...1/DSCN5615.JPG
https://dl.dropbox.com/u/1471771/For...1/DSCN5619.JPG
2 x 10GbE ports and 2 x short range optical transceivers. One hell of a network card. Here is the link to more specs etc:
http://www.ebuyer.co...medium=products
Current look of motherboard tray:
https://dl.dropbox.com/u/1471771/For...1/DSCN5620.JPG
https://dl.dropbox.com/u/1471771/For...1/DSCN5621.JPG
Re: Project: Perfect vSphere 5 Whitebox!
UPDATE #4
Some of the pictures above are, already, exposing my soundproofing material inside the case but I have two more!
5.25" drive bays have been nicely covered:
https://dl.dropbox.com/u/1471771/For...1/DSCN5625.JPG
Flooring is completely covered now as well:
https://dl.dropbox.com/u/1471771/For...1/DSCN5627.JPG
Lian-Li C-01 DVD Bezel needed slight modification in order to fit, stock bezel and what we need to get rid of:
https://dl.dropbox.com/u/1471771/For...1%21/C-01B.jpg
Fitted inside the case:
https://dl.dropbox.com/u/1471771/For...1/DSCN5626.JPG
Next up was back of my case, I wasn't overly impressed with, my own, cutting skills so had to even things out. Even though you can't see it (backplate covers is) it was bugging me restlessly! Here is the shoddy job in first place:
https://dl.dropbox.com/u/1471771/For...SCN5510---.JPG
Corrections in place:
https://dl.dropbox.com/u/1471771/For...1/DSCN5631.JPG
Once I have closed both side panels the case started to take its shape:
https://dl.dropbox.com/u/1471771/For...1/DSCN5633.JPG
https://dl.dropbox.com/u/1471771/For...1/DSCN5634.JPG
The idea here is to push the air out of the case using front and side vent holes. I didn't want to blow it in (dust, dust and more dust) so the space around H100 had to be almost sealed to prevent air getting back into the case (as opposed to being pushed out)
Once I will have all of my components together inside I will deffo so some testing to see if any air is making its way in.
That's it for now. I'm at home today so will have more updates a bit later on.
Re: Project: Perfect vSphere 5 Whitebox!
UPDATE #5
Postman turned up with some interesting stuff (I already had one though!):
https://dl.dropbox.com/u/1471771/For...1/DSCN5640.JPG
https://dl.dropbox.com/u/1471771/For...1/DSCN5641.JPG
https://dl.dropbox.com/u/1471771/For...1/DSCN5642.JPG
It should be nice and fast once setup! Each card is capable of 20Gbps throughput once in EtherChannel (or LACP if you don't like Cisco terminology!)
I'm still waiting for 1 metre cables for these cards, but at £70 a pop they're not cheap...
http://www.lambda-te...gin=gbaseGB11.4
So two of them and £140 is gone.
If you're interested in similar setup with SFP+ direct attached cables please make sure you buy the branded ones! SFP+ modules will not work otherwise and link light won't come up. Also the chips built into the ends are used for link speed negotiations so really important to get a good cable.
Since Cisco wires is what we used at work for the same purpose that's what I got. Exact model - SFP-H10GB-CU1M (1 metre)
DA2 (direct attached) can go up to 10 metres, there are other lengths available from Cisco as well:
SFP-H10GB-CU3M - 3 m.
SFP-H10GB-CU5M - 5 m.
SFP-H10GB-CU7M - 7 m.
SFP-H10GB-CU10M - 10m.
Going back to X520 series from Intel. What I like about them is the fact you can swap between copper and fiber if you have to.
I'm going with copper right now purely because 10GbE switch would cost me my kidney but when I decide to sell it I have the cards already! That's the beauty of SFP+.
For now there will be only 2 devices interconnected with X520 cards but you can go up to 3 (each card has two ports) without a switch - that's exactly what I'm going to do once my new NAS device is here!
Re: Project: Perfect vSphere 5 Whitebox!
UPDATE #6
I have done some braiding. Front panel power and reset switches were on the menu:
https://dl.dropbox.com/u/1471771/For...1/DSCN5649.JPG
https://dl.dropbox.com/u/1471771/For...1/DSCN5696.JPG
https://dl.dropbox.com/u/1471771/For...1/DSCN5699.JPG
Alternative power and reset switches have been done as well. I don't know if I will be using these though, depends on how my son (Alex) will get along with the server (or server buttons!) The idea is to drill one of the PCI blanking plates and mount these at the back. Nicely our of sight. I will see.
https://dl.dropbox.com/u/1471771/For...1/DSCN5695.JPG
Re: Project: Perfect vSphere 5 Whitebox!
UPDATE #7
Just a quick update to say the server is up and running!
Windows Server 2008 R2 Enterprise SP1 got installed just to see how things are working. Here are some screenshots, Xeon X3450 @ stock i.e. 2.67GHz and 32GB of DDR3 RAM installed:
https://dl.dropbox.com/u/1471771/For...Properties.png
Task manager:
https://dl.dropbox.com/u/1471771/For...%20Manager.png
CPU temps at idle are off the chain as well... H100 with those Noctua fans is doing a great job!
https://dl.dropbox.com/u/1471771/For...PU%20Temps.png
Now take all of the above into account i.e. 430W PSU, 2TB drive, 2 SSDs, Quad Core Xeon CPU with 95W of TDP, 32GB of RAM, IBM RAID card, 2 port GigE card, 2 port 10GbE card and think of power consumption... Any ideas? Here are two screenshots, both taken when the server:
1) Was running idle,
2) OS installed was Server 2008 R2,
3) OS was at CTRL+ALT+DEL screen.
https://dl.dropbox.com/u/1471771/For...1/DSCN5712.JPG
https://dl.dropbox.com/u/1471771/For...1/DSCN5708.JPG
52W @ .26A !!!
This just got to be the new record in power consumption
I will grab some final pictures once there is more light so you all see how it looks fully assembled!
Re: Project: Perfect vSphere 5 Whitebox!
Very nice indeed - Is that the supermicro board that can run its own iLO like ability? What are you planning on hooking that 10GbE up to?
Re: Project: Perfect vSphere 5 Whitebox!
Yep that's the one. Concept is exactly the same as with iDRAC or iLO, Supermicro call it IPMI and runs on dedicated NIC port which always helps!
10GbE will be using direct attach cables for now between my desktop, Synology NAS and ESXi box (each card has two ports connecting to two devices, maximum of 3 in total - you get the idea). That's as far as it goes without a switch. I have some plans to buy Catalyst 2960S with two 10GbE uplinks via SFP+ but need to put some cash away first as its well over a grand to buy...
Re: Project: Perfect vSphere 5 Whitebox!
Very nice spoon :)
Quick tip for sealing cracks/gaps, use the left over scraps of sound absorbing foam, you might need to use scissors or a sharp knife to trim down the thickness ((depending on the size of the crack/gap) but it does work well and is easy to do.
Other thing I'd consider is flipping some of the fans over, looks like all the fans are trying to pull air out of the case with no way for air to get in.
Your temps might be good but that's idel and this is winter so lower environmental starting temperatures.
Re: Project: Perfect vSphere 5 Whitebox!
Nice build - looks like you've gone from a Ford Ka to a Bugatti Veyron though if this replaces an HP server....
Re: Project: Perfect vSphere 5 Whitebox!
Some nice hardware pr0n in this thread :D Nice job.
Re: Project: Perfect vSphere 5 Whitebox!
What home vms require this kind of setup ?
Re: Project: Perfect vSphere 5 Whitebox!
Quote:
Originally Posted by
callius
What home vms require this kind of setup ?
My ones! :P
You will be pleased to know that since I've built this two ML110s have turned up - both with buckets of RAM :)