Results 1 to 16 of 16

Thread: Project: Perfect vSphere 5 Whitebox!

  1. #1
    <<== UT3 Player spoon_'s Avatar
    Join Date
    Nov 2008
    Location
    London
    Posts
    2,071
    Thanks
    113
    Thanked
    139 times in 131 posts

    Project: Perfect vSphere 5 Whitebox!

    I have been entertaining HP ProLiant MicroServer N36L for nearly a year now. Great machine for the money and with cost of around £120 (after the cashback) it was an absolute bargain at the time! Box has been upgraded to 8GB of DDR3 RAM (maximum the mobo can take) and its running ESXi 4.1 U1 absolutely fine. Disk space wise, there is only 30 GB Vertex SSD for few VMs (and .vswp files), rest of the storage needs is provided by QNAP TS-509 NAS (by means of NFS and iSCSI) This setup has been absolutely flawless so far but there is simply not enough RAM and CPU power for my needs (or rather my VMs). CPU Ready is going through the roof quite often due to AMD Athlon II Neo 1.30GHz processor which is just slightly better performer compared to Intel Atom range. 8GB of RAM is tight and ESXi was paging like mad to .vswp files hence why I put them on SSD which helped only to certain extent. At the end I kinda had enough and decided to build a custom server which would address all of the issues above.


    Here is what I came up with:


    Processor: Intel Xeon X3450 2.66GHz with HT/VT-x and VT-d
    Processor Cooling: Corsair CWCH100 Hydro Series H100 Cooler
    Processor Cooling Fans: 2 x Noctua NF-P12
    Motherboard: Supermicro X8SIL-F-O Server Board
    RAM: Kingston 4 x 8GB Memory [KVR1333D3Q8R9S/8G]
    Case: Lian-Li PC-V350B
    Case Noise Dampening: AcoustiPack LITE (APL) Multi-Layered Soundproof Material
    Case Backplate: Custom backplate to incorporate moving PSU to the right and adding 120mm exhaust fan

    5.25" Drive Bay Cooling: Evercool Armour ATX HDD Cool Box HD-AR
    5.25" Drive Bay Cooling Fan: Noctua NF-R8-1800

    Case Exhaust Fan [Back]: 1 x Noctua NF-P12
    Case Exhaust Fan Guard: 120mm Standard Wire Case Fan Guard Grill [Black]

    Power Supply: Be Quiet! BN180 L8 430W Modular PSU
    Storage 1: Samsung 830 256GB SSD (main datastore)
    Storage 2: Seagate Barracuda 2TB [ST2000DM001] (second datastore)
    Storage 3: Vertex 1 30GB SSD (.vswp datastore)
    Storage 4: Patriot Extreme Performance Xporter XT Rage 8GB (local storage for ESXi)
    Storage Adapter Bracket: SilverStone SST-FP55B (allows 1 x 5.25" and 2 x 2.5" in one 5.25" slot!)

    Network 1: Onboard Dual Intel 82574L Gigabit Ethernet Controllers
    Network 2: HP NC360T PCI Express Dual Port Gigabit Server Adapter [which effectively is Intel PRO/1000 PT Dual Port NIC]
    Network 3: Intel Ethernet Converged Network Adapter X520-DA2, 10GbE, Dual Port

    RAID Controller: IBM ServeRAID M1015 [which kinda is OEM version of LSI 9220-8i]
    ODD: Toshiba/Samsung TS-H653 20x DVD±RW DL SATA Drive



    More when I get some pictures done (or rather work!)

  2. Received thanks from:

    Pob255 (13-02-2013)

  3. #2
    <<== UT3 Player spoon_'s Avatar
    Join Date
    Nov 2008
    Location
    London
    Posts
    2,071
    Thanks
    113
    Thanked
    139 times in 131 posts

    Re: Project: Perfect vSphere 5 Whitebox!

    UPDATE #1

    Few additions/changes to the list above:

    1) 5.25" drive bay cooling is now provided by Evercool Armour ATX HDD Cool Box HD-AR:



    If you have/had V350 case you know that 5.25" devices can be mounted on either side but not on both (unless they're very short) I have my ODD and bracket holding 2 x SSDs and 1 x 3.5" HDD on one side but there is virtually no air circulation hence the 'cool box' above. I had to modify it slightly to fit though (was too long and sticking out a bit) - thanks to cheap and cheerful hacksaw that was at hand it was a few minute job! I will grab a picture when the light is better so you guys can see.

    2) Stock fan in 'cool box' was replaced by Noctua NF-R8-1800 which should keep the noise down.

    3) Andy from WCuk was generous enough to laser custom backplate to incorporate moving PSU to the right and adding 120mm exhaust fan on the left. V350 has, again, virtually no exhaust fans which is a pain. Here is what I'm on about:

    Rear, stock, view of V350:



    Custom bracket fitted and the whole thing riveted together (bracket is part of removable motherboard tray):



    and the complete rear view:



    Much nicer, isn't it? I was never going to use the 2 x 3.5" drive bays so it was very natural thing to do.

    4) Storage for installing ESXi is now provided by Patriot Extreme Performance Xporter XT Rage 8GB - motherboard above has internal USB port so we are all good.

    As far as the case rear modifications are concerned it was a question of drilling out rivets and hacksawing the back to free up space for the new backplate:

    Stock:



    Modded:



    Not the best of cuts, I must admit, but £5 hacksaw from B&Q can only take you so far...

    Applying noise dampening kit was next... (on pair with braiding wires in my book, not a fan of either to be honest!)

    Case partially done, I'm still working on 5.25" drive bays:













    Picture quality isn't great but without studio lights there is only so much you can do!

    Leftovers:



    Here is rest of the parts (not a complete list though..):








  4. #3
    <<== UT3 Player spoon_'s Avatar
    Join Date
    Nov 2008
    Location
    London
    Posts
    2,071
    Thanks
    113
    Thanked
    139 times in 131 posts

    Re: Project: Perfect vSphere 5 Whitebox!


















  5. #4
    <<== UT3 Player spoon_'s Avatar
    Join Date
    Nov 2008
    Location
    London
    Posts
    2,071
    Thanks
    113
    Thanked
    139 times in 131 posts

    Re: Project: Perfect vSphere 5 Whitebox!

    UPDATE #2

    Small ish update. I have to work during the evenings to keep this build going so pictures won't be of any great quality. Unfortunately my usual 9 to 5 job is keeping me busy more than I would have liked. Few things have been done since last update, here is the list:

    Evercool Armour has been chopped as I mentioned in my first update - here is what it looks like now:





    [font=Open Sans', Helvetica, Arial, sans-serif]Noctua NF-R8-1800 [/font]fan fitted:



    Looking alright I think. I'm still waiting for 80mm fan guard (ordered) to keep wires out of the way.


    Here is how the whole thing inside from inside the case:



    5.25" drive bay external view:



    It should be enough to extract any trapped hot air generated by my hard drives. I think.



    In stock V350 there is space for two 120mm fans at the front of the case:



    My idea of utilizing this space is to fit Corsair H100 all-in-one water cooling unit AND the 120mm fans. For those unfamiliar, Corsair H100 has 240 radiator:



    Bracket originally holding just fans needed attention to make this happen as mounting holes on Corsair H100 only lined up on the first 120mm fan (left). Holes on the right didn't lined up at all so I had to widen them a bit:



    Bracket, two Noctua NF-P12 120mm fans and the H100 all screwed together as one unit:





    It fits pretty well. Now, will it fit inside the V350? Answer is - yes it will!



    I have to say though, its an extremely tight fit - space left on both left and right hand side is just enough to close the case together:





    Sweet.


    Silverstone bracket to hold my two SSDs and 2TB drive was next - SSDs in place:



    2TB hard drive temporarily mounted upside down:



    And the whole bracket "correct" way around:



    I have decided to use the Lian-Li 5.25" solid bezel that came with the case and not to expose front of the Silverstone bracket as the finish didn't really match up. Optical drive got some attention as well, I had the stealth Lian-Li bezel for ages now so it was perfect time to use it! 5.25" external view once the bracket and bezels was mounted:



    That's it for now. More in a day or two...

  6. #5
    <<== UT3 Player spoon_'s Avatar
    Join Date
    Nov 2008
    Location
    London
    Posts
    2,071
    Thanks
    113
    Thanked
    139 times in 131 posts

    Re: Project: Perfect vSphere 5 Whitebox!

    UPDATE #3

    I have order some rubber noise absorbers for the 120mm exhaust fan and the PSU:



    Had time to try only the PSU one and it fits perfectly. I'm hoping it will prevent any vibrations from being transferred onto the case.





    Newest addition to this build - Intel X520-SR2 10GbE PCIe Server Adapter:







    2 x 10GbE ports and 2 x short range optical transceivers. One hell of a network card. Here is the link to more specs etc:

    http://www.ebuyer.co...medium=products



    Current look of motherboard tray:




  7. #6
    <<== UT3 Player spoon_'s Avatar
    Join Date
    Nov 2008
    Location
    London
    Posts
    2,071
    Thanks
    113
    Thanked
    139 times in 131 posts

    Re: Project: Perfect vSphere 5 Whitebox!

    UPDATE #4

    Some of the pictures above are, already, exposing my soundproofing material inside the case but I have two more!


    5.25" drive bays have been nicely covered:



    Flooring is completely covered now as well:



    Lian-Li C-01 DVD Bezel needed slight modification in order to fit, stock bezel and what we need to get rid of:



    Fitted inside the case:



    Next up was back of my case, I wasn't overly impressed with, my own, cutting skills so had to even things out. Even though you can't see it (backplate covers is) it was bugging me restlessly! Here is the shoddy job in first place:



    Corrections in place:



    Once I have closed both side panels the case started to take its shape:





    The idea here is to push the air out of the case using front and side vent holes. I didn't want to blow it in (dust, dust and more dust) so the space around H100 had to be almost sealed to prevent air getting back into the case (as opposed to being pushed out)

    Once I will have all of my components together inside I will deffo so some testing to see if any air is making its way in.


    That's it for now. I'm at home today so will have more updates a bit later on.

  8. #7
    <<== UT3 Player spoon_'s Avatar
    Join Date
    Nov 2008
    Location
    London
    Posts
    2,071
    Thanks
    113
    Thanked
    139 times in 131 posts

    Re: Project: Perfect vSphere 5 Whitebox!

    UPDATE #5

    Postman turned up with some interesting stuff (I already had one though!):







    It should be nice and fast once setup! Each card is capable of 20Gbps throughput once in EtherChannel (or LACP if you don't like Cisco terminology!)


    I'm still waiting for 1 metre cables for these cards, but at £70 a pop they're not cheap...

    http://www.lambda-te...gin=gbaseGB11.4

    So two of them and £140 is gone.


    If you're interested in similar setup with SFP+ direct attached cables please make sure you buy the branded ones! SFP+ modules will not work otherwise and link light won't come up. Also the chips built into the ends are used for link speed negotiations so really important to get a good cable.

    Since Cisco wires is what we used at work for the same purpose that's what I got. Exact model - SFP-H10GB-CU1M (1 metre)

    DA2 (direct attached) can go up to 10 metres, there are other lengths available from Cisco as well:

    SFP-H10GB-CU3M - 3 m.
    SFP-H10GB-CU5M - 5 m.
    SFP-H10GB-CU7M - 7 m.
    SFP-H10GB-CU10M - 10m.


    Going back to X520 series from Intel. What I like about them is the fact you can swap between copper and fiber if you have to.
    I'm going with copper right now purely because 10GbE switch would cost me my kidney but when I decide to sell it I have the cards already! That's the beauty of SFP+.

    For now there will be only 2 devices interconnected with X520 cards but you can go up to 3 (each card has two ports) without a switch - that's exactly what I'm going to do once my new NAS device is here!

  9. #8
    <<== UT3 Player spoon_'s Avatar
    Join Date
    Nov 2008
    Location
    London
    Posts
    2,071
    Thanks
    113
    Thanked
    139 times in 131 posts

    Re: Project: Perfect vSphere 5 Whitebox!

    UPDATE #6

    I have done some braiding. Front panel power and reset switches were on the menu:







    Alternative power and reset switches have been done as well. I don't know if I will be using these though, depends on how my son (Alex) will get along with the server (or server buttons!) The idea is to drill one of the PCI blanking plates and mount these at the back. Nicely our of sight. I will see.


  10. #9
    <<== UT3 Player spoon_'s Avatar
    Join Date
    Nov 2008
    Location
    London
    Posts
    2,071
    Thanks
    113
    Thanked
    139 times in 131 posts

    Re: Project: Perfect vSphere 5 Whitebox!

    UPDATE #7

    Just a quick update to say the server is up and running!

    Windows Server 2008 R2 Enterprise SP1 got installed just to see how things are working. Here are some screenshots, Xeon X3450 @ stock i.e. 2.67GHz and 32GB of DDR3 RAM installed:



    Task manager:



    CPU temps at idle are off the chain as well... H100 with those Noctua fans is doing a great job!




    Now take all of the above into account i.e. 430W PSU, 2TB drive, 2 SSDs, Quad Core Xeon CPU with 95W of TDP, 32GB of RAM, IBM RAID card, 2 port GigE card, 2 port 10GbE card and think of power consumption... Any ideas? Here are two screenshots, both taken when the server:

    1) Was running idle,
    2) OS installed was Server 2008 R2,
    3) OS was at CTRL+ALT+DEL screen.





    52W @ .26A !!!

    This just got to be the new record in power consumption


    I will grab some final pictures once there is more light so you all see how it looks fully assembled!

  11. #10
    Administrator Moby-Dick's Avatar
    Join Date
    Jul 2003
    Location
    There's no place like ::1 (IPv6 version)
    Posts
    10,665
    Thanks
    53
    Thanked
    385 times in 314 posts

    Re: Project: Perfect vSphere 5 Whitebox!

    Very nice indeed - Is that the supermicro board that can run its own iLO like ability? What are you planning on hooking that 10GbE up to?
    my Virtualisation Blog http://jfvi.co.uk Virtualisation Podcast http://vsoup.net

  12. #11
    <<== UT3 Player spoon_'s Avatar
    Join Date
    Nov 2008
    Location
    London
    Posts
    2,071
    Thanks
    113
    Thanked
    139 times in 131 posts

    Re: Project: Perfect vSphere 5 Whitebox!

    Yep that's the one. Concept is exactly the same as with iDRAC or iLO, Supermicro call it IPMI and runs on dedicated NIC port which always helps!

    10GbE will be using direct attach cables for now between my desktop, Synology NAS and ESXi box (each card has two ports connecting to two devices, maximum of 3 in total - you get the idea). That's as far as it goes without a switch. I have some plans to buy Catalyst 2960S with two 10GbE uplinks via SFP+ but need to put some cash away first as its well over a grand to buy...

  13. #12
    Senior Member Pob255's Avatar
    Join Date
    Apr 2007
    Location
    The land of Brum
    Posts
    10,128
    Thanks
    605
    Thanked
    1,220 times in 1,121 posts
    • Pob255's system
      • Motherboard:
      • Asus M5A99X EVO
      • CPU:
      • FX8350 & CM Hyper 212+
      • Memory:
      • 4 x 2gb Corsair Vengence 1600mhz cas9
      • Storage:
      • 512gb samsung SSD +1tb Samsung HDD
      • Graphics card(s):
      • EGVA GTX970
      • PSU:
      • Seasonic GX 650W
      • Case:
      • HAF 912+
      • Operating System:
      • W7 Pro
      • Monitor(s):
      • iiyama XB3270QS-B1 32" IPS 1440p

    Re: Project: Perfect vSphere 5 Whitebox!

    Very nice spoon

    Quick tip for sealing cracks/gaps, use the left over scraps of sound absorbing foam, you might need to use scissors or a sharp knife to trim down the thickness ((depending on the size of the crack/gap) but it does work well and is easy to do.

    Other thing I'd consider is flipping some of the fans over, looks like all the fans are trying to pull air out of the case with no way for air to get in.
    Your temps might be good but that's idel and this is winter so lower environmental starting temperatures.

  14. #13
    Senior Member
    Join Date
    Mar 2008
    Posts
    799
    Thanks
    66
    Thanked
    71 times in 70 posts
    • kmac's system
      • Motherboard:
      • Gigabyte GA-P55-US3L
      • CPU:
      • Intel Core i5 750
      • Memory:
      • 2GB Corsair DDR3
      • Storage:
      • 2 Samsung F3 500GB each
      • Graphics card(s):
      • ATI HD4770
      • PSU:
      • Enermax Modu 82+ 525W
      • Case:
      • Lain -Li PC-7FN Black
      • Operating System:
      • Windows 7 Prof 64 Bit
      • Monitor(s):
      • Dell U2412M

    Re: Project: Perfect vSphere 5 Whitebox!

    Nice build - looks like you've gone from a Ford Ka to a Bugatti Veyron though if this replaces an HP server....

  15. #14
    ALT0153™ Rob_B's Avatar
    Join Date
    Jul 2006
    Posts
    6,087
    Thanks
    345
    Thanked
    700 times in 484 posts

    Re: Project: Perfect vSphere 5 Whitebox!

    Some nice hardware pr0n in this thread Nice job.

  16. #15
    Registered+
    Join Date
    Aug 2004
    Posts
    45
    Thanks
    1
    Thanked
    0 times in 0 posts

    Re: Project: Perfect vSphere 5 Whitebox!

    What home vms require this kind of setup ?

  17. #16
    <<== UT3 Player spoon_'s Avatar
    Join Date
    Nov 2008
    Location
    London
    Posts
    2,071
    Thanks
    113
    Thanked
    139 times in 131 posts

    Re: Project: Perfect vSphere 5 Whitebox!

    Quote Originally Posted by callius View Post
    What home vms require this kind of setup ?
    My ones!

    You will be pleased to know that since I've built this two ML110s have turned up - both with buckets of RAM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •