Page 1 of 2 12 LastLast
Results 1 to 16 of 19

Thread: vINCEPTION (or nested virtualisation)

  1. #1
    HEXUS.social member Allen's Avatar
    Join Date
    Nov 2003
    Location
    Brighton
    Posts
    8,536
    Thanks
    363
    Thanked
    262 times in 168 posts
    • Allen's system
      • Motherboard:
      • ASUS Maximus VIII Gene
      • CPU:
      • Intel Core i5 6600K
      • Memory:
      • 2 x 8GB Kingston HyperX Predator DDR4-3000
      • Storage:
      • 256GB Samsung 950 PRO NVMe M.2 (OS) + 2 x 512GB Samsung 960 EVO in RAID 0 (Games)
      • Graphics card(s):
      • ASUS ROG Strix GeForce GTX 1080 Ti OC
      • PSU:
      • XFX P1-650X-NLG9 XXX 650W Modular
      • Case:
      • Fractal Design Node 804
      • Operating System:
      • Windows 10 Home 64-bit
      • Monitor(s):
      • 27" BenQ XL2730Z + 23" Dell U2311H
      • Internet:
      • Virgin Media 200Mbps

    vINCEPTION (or nested virtualisation)

    Does anyone here run a nested virtualisation lab at home on a single host? I am in the process of trying to set up a fully legal (trials are great!) and fully working ESXi 5.5 nested lab on my PC. So far, I have managed to get most things up and running by following a really good and in depth guide, but it focuses on 5.1 and my trials are for 5.5 (the only real difference so far has been the installation of vCenter now that it needs Single Sign On/Web Client/Inventory Service, it's totally different and I am struggling to get it working).

    The major thing with vCenter that concerns me though is that I have it running on a VM in Workstation, rather than a VM inside the Hypervisor (this is how my previous company had it set up) and it had to install with 4GB of RAM. I only have 8GB of RAM normally, however I managed to borrow another 4GB from a friend, so vCenter is using 1/3rd of my total RAM. Is there anything I can do about this or is it necessary due to it being the "centre" of all things v.

    The guide also mentioned setting up DC and SQL servers, I set them up with 2GB each to make them easier to set up and plan to drop them to 1GB each once complete. This leaves about ~4GB free for the Hypervisors (I really want two, what's the point otherwise as this is for testing and playing around with HA/DRS/FT etc).

    The real question is, with only 12GB of RAM (~10.5GB available after taking into account the host Windows 7 using a bit) what do you think the ideal set up for a nested lab should be? Should I forgo the DC and SQL server and have them all on one, along with vCenter? Should I just set up two 5GB Hypervisors with everything running inside them?

    If you run a nested lab, what is your set up and how did you install it all? Do you know of any guides for 5.5?

    Also, has anyone heard of Autolab (http://www.labguides.com/autolab/) and have you used it? Is it any good and will it help me, or will it take away valuable learning experience? Sadly, it's not for 5.5 but still worth a play about with I guess.

  2. #2
    Splash
    Guest

    Re: vINCEPTION (or nested virtualisation)

    I've used Autolab in the past (around about release date of 5.0) and it was pretty decent. As for tuning of vCenter: are you using the VCSA or the Windows based version? You can tweak the memory usage for Java (see this) which is what will be chewing the CPU. Bear in mind that ESXi5.5 requires 4Gb RAM minimum just to install, too.

    I used to have a nested lab setup in VMware Workstation (though I'm now back to 4 hosts and a shared storage box), but my desktop machine has 32gb of RAM and a pretty decent RAID array - I think squeezing into 10.5Gb is cutting it fine but should be doable - would strongly recommend having a good look around William Lam's website: some of his #Unsupported tips come in really useful for homelabs.

  3. #3
    HEXUS.social member Allen's Avatar
    Join Date
    Nov 2003
    Location
    Brighton
    Posts
    8,536
    Thanks
    363
    Thanked
    262 times in 168 posts
    • Allen's system
      • Motherboard:
      • ASUS Maximus VIII Gene
      • CPU:
      • Intel Core i5 6600K
      • Memory:
      • 2 x 8GB Kingston HyperX Predator DDR4-3000
      • Storage:
      • 256GB Samsung 950 PRO NVMe M.2 (OS) + 2 x 512GB Samsung 960 EVO in RAID 0 (Games)
      • Graphics card(s):
      • ASUS ROG Strix GeForce GTX 1080 Ti OC
      • PSU:
      • XFX P1-650X-NLG9 XXX 650W Modular
      • Case:
      • Fractal Design Node 804
      • Operating System:
      • Windows 10 Home 64-bit
      • Monitor(s):
      • 27" BenQ XL2730Z + 23" Dell U2311H
      • Internet:
      • Virgin Media 200Mbps

    Re: vINCEPTION (or nested virtualisation)

    Having been playing around with this for a while now whilst watching some training videos, it's come to my attention that 12GB still isn't enough. An AD server (1GB), and vCenter server (4GB) and my host OS (2GB) only leaves me with 5GB for the ESXi hosts. In the near future I will probably have to set up a dedicated box to run ESXi on, and then nest everything inside that, rather than in Workstation as I am currently doing.

    I have been trying to find suitable hardware for ages, because I want to do it on the cheap so thought AMD... but finding a desktop motherboard with IOMMU actually supported and ACPI IVRS tables enabled is quite difficult. I guess I will end up having to resort to Intel for the job, but that will severely limit what else I can get sadly. My CPU requirements will be quite small, I just need a lot of RAM and a reasonable amount of HDD space. What do you use/suggest?

  4. #4
    Splash
    Guest

    Re: vINCEPTION (or nested virtualisation)

    My lab consists of a HP ML115 running shared storage (FreeNAS iSCSI) and 3 Intel NUCs (16Gb RAM each). if you're looking to do it all in one unit then ServersPlus have a fairly beefy Dell server ESXi testbed that comes with 32Gb of RAM and a quadcore CPU (you'll almost never be hamstrung on CPU) for a little under £520 delivered. It's light on disks, but you should be able to source some standard disks easily enough. I'd double check what the RAID controller in it is capable of under ESXi though (it may only be able to pass through the disks, which might not be the end of the world for a single home lab setup)

  5. #5
    Senior Member
    Join Date
    Mar 2005
    Posts
    4,825
    Thanks
    161
    Thanked
    358 times in 288 posts
    • badass's system
      • Motherboard:
      • ASUS P8Z77-m pro
      • CPU:
      • Core i5 3570K
      • Memory:
      • 32GB
      • Storage:
      • 1TB Samsung 850 EVO, 2TB WD Green
      • Graphics card(s):
      • Radeon RX 580
      • PSU:
      • Corsair HX520W
      • Case:
      • Silverstone SG02-F
      • Operating System:
      • Windows 10 X64
      • Monitor(s):
      • Del U2311, LG226WTQ
      • Internet:
      • 80/20 FTTC

    Re: vINCEPTION (or nested virtualisation)

    I had a similar conundrum and just went for 32GB of RAM for my desktop. Since then I've had little to no spare time to use it.
    I've also found an SSD to be a necessity to avoid spending ages waiting for multiple VM's to do anything.
    This may sound expensive but compared to the wages you get for VMware certification plus the right attitude and experience, it's peanuts.
    Another way is to get an additional computer with 8/12/16GB RAM. I have a Microserver with 16GB (and an ML330 G6 with 36GB but that's overkill for just training)

    I will stress that for the Exchange related study, the SSD was an absolute must. On spinning disks and a 3 node DAG, it was taking 25 minutes to just get them booted!
    "In a perfect world... spammers would get caught, go to jail, and share a cell with many men who have enlarged their penises, taken Viagra and are looking for a new relationship."

  6. #6
    HEXUS.social member Allen's Avatar
    Join Date
    Nov 2003
    Location
    Brighton
    Posts
    8,536
    Thanks
    363
    Thanked
    262 times in 168 posts
    • Allen's system
      • Motherboard:
      • ASUS Maximus VIII Gene
      • CPU:
      • Intel Core i5 6600K
      • Memory:
      • 2 x 8GB Kingston HyperX Predator DDR4-3000
      • Storage:
      • 256GB Samsung 950 PRO NVMe M.2 (OS) + 2 x 512GB Samsung 960 EVO in RAID 0 (Games)
      • Graphics card(s):
      • ASUS ROG Strix GeForce GTX 1080 Ti OC
      • PSU:
      • XFX P1-650X-NLG9 XXX 650W Modular
      • Case:
      • Fractal Design Node 804
      • Operating System:
      • Windows 10 Home 64-bit
      • Monitor(s):
      • 27" BenQ XL2730Z + 23" Dell U2311H
      • Internet:
      • Virgin Media 200Mbps

    Re: vINCEPTION (or nested virtualisation)

    Quote Originally Posted by Splash View Post
    if you're looking to do it all in one unit then ServersPlus have a fairly beefy Dell server ESXi testbed that comes with 32Gb of RAM and a quadcore CPU (you'll almost never be hamstrung on CPU) for a little under £520 delivered.
    Wow, that is a pretty sweet deal, considering the price of the CPU and RAM included... However I was hoping for something a little cheaper. I am still not financially over my bout of unemployment, and despite getting paid well I am still contracting so didn't get paid for my time off over Xmas/New Year etc.

    I was hoping to get a mobo, CPU and RAM alone as I have everything else spare already. In fact, I have a Core 2 Duo sitting at home doing nothing, I wonder if that would support everything needed? Although I would imagine DDR2 RAM for the board is probably more expensive than DDR3 now?

    Conundrums... I really want to progress my career, but VMware is very expensive to study, not to mention the course fee's!

  7. #7
    Splash
    Guest

    Re: vINCEPTION (or nested virtualisation)

    I've not looked at DDR2 since before I built my current desktop, so I have no idea on price but memory prices in general seem to be creeping up across the board. There's also the question as to whether you can find a board that'll take a C2D and 32Gb of RAM (most seemed to peak out at 16Gb) without spending silly money on server grade kit.

    What's your learning environment like? Could you cope with a rack server? If so you can probably pick up something like a DL360 G5 or similar which you could fill with RAM, but it'll be hella noisy (so you might want to hide it in an attic or similar)

  8. #8
    HEXUS.social member Allen's Avatar
    Join Date
    Nov 2003
    Location
    Brighton
    Posts
    8,536
    Thanks
    363
    Thanked
    262 times in 168 posts
    • Allen's system
      • Motherboard:
      • ASUS Maximus VIII Gene
      • CPU:
      • Intel Core i5 6600K
      • Memory:
      • 2 x 8GB Kingston HyperX Predator DDR4-3000
      • Storage:
      • 256GB Samsung 950 PRO NVMe M.2 (OS) + 2 x 512GB Samsung 960 EVO in RAID 0 (Games)
      • Graphics card(s):
      • ASUS ROG Strix GeForce GTX 1080 Ti OC
      • PSU:
      • XFX P1-650X-NLG9 XXX 650W Modular
      • Case:
      • Fractal Design Node 804
      • Operating System:
      • Windows 10 Home 64-bit
      • Monitor(s):
      • 27" BenQ XL2730Z + 23" Dell U2311H
      • Internet:
      • Virgin Media 200Mbps

    Re: vINCEPTION (or nested virtualisation)

    Not possible right now, I live in a shared apartment with no storage area other than my bedroom. I could hide it in there and only power it on when needed, but I would prefer a tower with better ventilation/quieter cooling options.

    I may just have to save for a bit to get a decent tower, or whack some more RAM in my main PC. It's unfortunate really, I want to do all of this before my evaluation licenses expire, but I doubt that will happen now. I guess you can't extend them easily?

  9. #9
    Splash
    Guest

    Re: vINCEPTION (or nested virtualisation)

    depends on what you're referring to for the licenses: ESXi trial licenses will revert to free ESXi when they expire, vCenter not so much. You're probably looking at rebuilding anew anyway when you get your new environment.

    Out of interest how much RAM will your PC take? I've a Microserver here with 16Gb RAM and an ML115 with 8Gb RAM that aren't doing anything vital (so I may be able to be persuaded to part with them), so you could potentially offload (as an example) vCenter to another box to free up some resource on your main PC to put some other bits and bobs together if that's an option? It'll not be anywhere near the kind of performance of one of those Dell boxes, but it might be enough as a stopgap.

    Also: have you looked at the VCA training? Free, online, in your own time and a cheap exam to get a cert at the end of it (normally £80, £40 while the half price codes are on the go) - I've done VCA-IV and VCA-Cloud as a warm up for my VCP next month.

    EDIT2: Just checked your system specs: you can get 4*8Gb DIMMS in that board: I think that's where I'd be going in the first instance. You don't need the fastest RAM, and you certainly don't need to be overclocking.
    Last edited by Splash; 17-01-2014 at 06:14 PM.

  10. #10
    Senior Member
    Join Date
    Jan 2004
    Location
    Leicestershire
    Posts
    1,212
    Thanks
    7
    Thanked
    31 times in 30 posts
    • madman045's system
      • Motherboard:
      • P9X79 Pro
      • CPU:
      • I7-3820
      • Memory:
      • 32GB
      • Storage:
      • Not enough!
      • Graphics card(s):
      • HD7970
      • PSU:
      • 850w Corsair
      • Case:
      • Corsair Carbide 300R
      • Operating System:
      • Win 7 Ultimate X64
      • Monitor(s):
      • Dell U2713HM & 2007WFP
      • Internet:
      • Plusnet FTTC - 30mbit/7mbit

    Re: vINCEPTION (or nested virtualisation)

    I am in the process of setting up a nested lab, in two minds between VCP and MCSA training at the moment

    I've got an i7 with 32gb ram, as the others have said, SSD really helps, I have a clone template of 2008 R2 on my SSD and then create linked clones on my 1TB drive, next is ESXI hosts and vcentre and perhaps later down the line, VSAN

    I've found a blog where by they used three low powered shuttles (95w) with i7's, 16gb ram, msata and sata ssd storage and a sata disk, its about £800 a host and then they bought a NAS, total was about £3200 to replicate that...

    I looked at the microservers but you will get the best deals now from someone else, that serversplus server is tempting though

  11. #11
    Splash
    Guest

    Re: vINCEPTION (or nested virtualisation)

    Quote Originally Posted by madman045 View Post
    I am in the process of setting up a nested lab, in two minds between VCP and MCSA training at the moment
    One doesn't have to preclude the other

  12. #12
    Senior Member
    Join Date
    Jan 2004
    Location
    Leicestershire
    Posts
    1,212
    Thanks
    7
    Thanked
    31 times in 30 posts
    • madman045's system
      • Motherboard:
      • P9X79 Pro
      • CPU:
      • I7-3820
      • Memory:
      • 32GB
      • Storage:
      • Not enough!
      • Graphics card(s):
      • HD7970
      • PSU:
      • 850w Corsair
      • Case:
      • Corsair Carbide 300R
      • Operating System:
      • Win 7 Ultimate X64
      • Monitor(s):
      • Dell U2713HM & 2007WFP
      • Internet:
      • Plusnet FTTC - 30mbit/7mbit

    Re: vINCEPTION (or nested virtualisation)

    Quote Originally Posted by Splash View Post
    One doesn't have to preclude the other
    I agree however I am terrible for getting distracted, creating servers, AD, DHCP, DNS etc.. not a problem, I am a hands on learner though, so i fall asleep reading the study books, be it a real book or e-book

    also do i get my MCSA 2008 or just go straight for 2012, just have to install 2012 R2 as we don't use it at work, also might buy a book on powershell

    too many decisions.....

  13. #13
    Splash
    Guest

    Re: vINCEPTION (or nested virtualisation)

    Quote Originally Posted by madman045 View Post
    I agree however I am terrible for getting distracted, creating servers, AD, DHCP, DNS etc.. not a problem, I am a hands on learner though, so i fall asleep reading the study books, be it a real book or e-book

    also do i get my MCSA 2008 or just go straight for 2012, just have to install 2012 R2 as we don't use it at work, also might buy a book on powershell

    too many decisions.....
    2008 or 2008R2? 2008 goes into extended support in just under a year. 2012 is something of a step-change, and I'd rather be on the bleeding edge if investing my own cash.

  14. #14
    Senior Member
    Join Date
    Jan 2004
    Location
    Leicestershire
    Posts
    1,212
    Thanks
    7
    Thanked
    31 times in 30 posts
    • madman045's system
      • Motherboard:
      • P9X79 Pro
      • CPU:
      • I7-3820
      • Memory:
      • 32GB
      • Storage:
      • Not enough!
      • Graphics card(s):
      • HD7970
      • PSU:
      • 850w Corsair
      • Case:
      • Corsair Carbide 300R
      • Operating System:
      • Win 7 Ultimate X64
      • Monitor(s):
      • Dell U2713HM & 2007WFP
      • Internet:
      • Plusnet FTTC - 30mbit/7mbit

    Re: vINCEPTION (or nested virtualisation)

    MCSA 2008 is 2008 R2, 70-640, 70-642 & 70-646 then upgrade to 2012 (upgrade exam is all three 2012 exams rolled into one)

    the VCP5-DCV would also be good in my current position as we are about to start a small investment in ESXI hosts and shared storage, however the cost of the mandatory training course is somewhat frustrating

    I take it you have a few certs under your belt Splash?

  15. #15
    Senior Member
    Join Date
    Mar 2005
    Posts
    4,825
    Thanks
    161
    Thanked
    358 times in 288 posts
    • badass's system
      • Motherboard:
      • ASUS P8Z77-m pro
      • CPU:
      • Core i5 3570K
      • Memory:
      • 32GB
      • Storage:
      • 1TB Samsung 850 EVO, 2TB WD Green
      • Graphics card(s):
      • Radeon RX 580
      • PSU:
      • Corsair HX520W
      • Case:
      • Silverstone SG02-F
      • Operating System:
      • Windows 10 X64
      • Monitor(s):
      • Del U2311, LG226WTQ
      • Internet:
      • 80/20 FTTC

    Re: vINCEPTION (or nested virtualisation)

    Quote Originally Posted by madman045 View Post
    I agree however I am terrible for getting distracted, creating servers, AD, DHCP, DNS etc.. not a problem, I am a hands on learner though, so i fall asleep reading the study books, be it a real book or e-book

    also do i get my MCSA 2008 or just go straight for 2012, just have to install 2012 R2 as we don't use it at work, also might buy a book on powershell

    too many decisions.....
    I agree on the not keeping too many plates spinning. One at a time is best for me too.
    What I will say is that it is a lot harder to recruit VCP certified engineers in the 30k to 60 k bracket than engineers with an MCSE. Make of that what you will

  16. #16
    HEXUS.social member Allen's Avatar
    Join Date
    Nov 2003
    Location
    Brighton
    Posts
    8,536
    Thanks
    363
    Thanked
    262 times in 168 posts
    • Allen's system
      • Motherboard:
      • ASUS Maximus VIII Gene
      • CPU:
      • Intel Core i5 6600K
      • Memory:
      • 2 x 8GB Kingston HyperX Predator DDR4-3000
      • Storage:
      • 256GB Samsung 950 PRO NVMe M.2 (OS) + 2 x 512GB Samsung 960 EVO in RAID 0 (Games)
      • Graphics card(s):
      • ASUS ROG Strix GeForce GTX 1080 Ti OC
      • PSU:
      • XFX P1-650X-NLG9 XXX 650W Modular
      • Case:
      • Fractal Design Node 804
      • Operating System:
      • Windows 10 Home 64-bit
      • Monitor(s):
      • 27" BenQ XL2730Z + 23" Dell U2311H
      • Internet:
      • Virgin Media 200Mbps

    Re: vINCEPTION (or nested virtualisation)

    I have now managed to borrow more RAM and I am running 16GB (well, 18GB, but thanks to Windows 7 Home Premium I am limited!). I have been doing a lot of work today to get this going and I have a question. I will give you a brief outline of what I have.

    Workstation is running vCenter, 2 x ESXi hosts, a storage server and an AD server.

    I have several networks set up in Workstation:

    VMnet0 - bridged - Home network connected to the AD server for internet access and to enable me to RDP to the AD server
    VMnet1 - host 10.0.1.0/24 - AD network connected to AD server and vCenter
    VMnet2 - host 10.0.2.0/24 - Management network connected to vCenter and my ESXi servers
    VMnet3 - host 10.0.3.0/24 - Storage network connected to my storage server and my ESXi servers
    VMnet4 - host 10.0.4.0/24 - vMotion network connected to my ESXi servers

    So far so good, my whole nested lab is completely isolated from the outside world and I can RDP to my AD server, run vSphere and administer the whole lab from there (with the exception of the storage server which I don't really need to access that often).

    However, my confusion comes with regards to the VM's. I have seen other people setting up another network for their ESXi VM's, but they seem to be host networks. I would prefer my VM's on the ESXi hosts to be able connect to the internet. I am therefore thinking that the ESXi hosts need another network, this time bridged, so that my ESXi VM's get their IP addresses from my router. Is this correct? Will this also expose my ESXi hosts to my home network? I am pretty sure I don't want to do that, especially in a live environment if Workstation wasn't involved. And I know for a fact that's it's possible as my last company which was a server hosting company had multiple ESXi hosts running VM's with access to the internet but the hosts were completely isolated.

    I am probably just struggling to get my head around it as it's nested and therefore virtual where as I should be thinking of everything inside Workstation as being physical. Anyone know the answer for the Workstation way of doing things?

Page 1 of 2 12 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •