ESXi 5 Home Lab Specs

Many people have asked me what hardware setup that I have in my current lab.  Note that this is not what I would consider a cheap, nor a standard lab.  This was built with a number of specific things in mind, including not only having an environment to train in, but also one for programming, practicing technologies like Highly Available MySQL, clustering, and so forth.  It is also a production environment where VMs for the house use live, and other items.  In choosing the hardware and networking, I wanted to have solid hardware compatible with ESXi, and leave plenty of room to grow.

“Physicalizing” VMs as Desktops

Also, I do something quite unique, that I believe I may be the only person on the Internet doing, at least in the method I am doing it.  I take VMs, pass video cards through to them, and then use them as “physical machines” for everyday use, including having a physical keyboard, mouse, and peripherals, such as printers.  I’ve coined the term “physicalized” for this practice.  This began with using XBMC as a VM, and evolved into using the same method to virtualize my son and wife’s PCs.  The only physical desktop that I have left in the house is my own.

My specialized method stands apart in that I pass video and peripheral data over CAT6, using keystone wall jacks with HDMI and USB plugs so these physical components simply plug right into a wall outlet.  For video, I run HDMI over CAT6 to the desktop monitor or wall-mounted TV (in the case of the HTPCs), and then run USB over CAT6 to a USB hub.  From that USB hub, I connect a keyboard and mouse for input, and any other peripherals that I may need, such as printers.  This has worked extraordinarily well, and has turned out to be no different, in use, than having a physical box.  My son even games on his, and has dual monitors.  Currently, I have four VMs physicalized like this.

As I write the article, I’ll also try to give some thoughts and background on why I did what I did, and also give some round prices on what I spent.  Almost everything was sourced off eBay, or Craigslist at discount prices.  Also, I will give links where relevant, and these are associate links for Amazon, just for transparency.  It doesn’t cost you anything to use them, and I get a small percentage of anything you purchase, so it helps out if you don’t mind.

Goals for VMWare ESXi 5 Home Lab

First, some thoughts that set the stage for how I shopped when I built the lab.  The biggest linchpin when building an ESXi Home Lab is memory.  Desktop memory is relatively cheap, but server memory is expensive, and don’t even think about trying to purchase branded memory from a vendor.  As for servers, you can pick up fairly cheap boxes, like Dell PowerEdge 2950s, even dual quad cores with 8-16GB of RAM, but they are power-hungry beasts, and in the long run, they’ll cost you more in your monthly electricity bill than simply going with newer hardware.  However, finding multi-core desktop CPUs and motherboards that are supported by ESXi and allow you access to things like USB and video card passthrough is quite a challenge.  There simply isn’t a need for the manufacturers to support it, and although the chipset may SAY it supports it, the support must be coded into the BIOS to work.

So let’s start with my requirements for my lab, and then figure out how to meet it.

  • Low cost (this is relative, but I didn’t want to spend over $2,500 in total)
  • Low power consumption (a little more up front would save me tons in the long run)
  • Minimum 8 cores per host
  • Minimum 32GB of RAM per host
  • Multiple Physical ESXi Nodes (Think vMotion, SRM, DRS, and Fault Tolerance)
  • 4 GB NICs per box (Minimum)
  • VT-x / AMD-V (Virtualized Enhancement)
  • VT-d / AMD IOMMU (Direct Path I/O, Passthru, and PCI Device Assignment)
  • vMotion capable
  • iSCSI Storage including Highly Available iSCSI (2 nodes)
  • NFS Storage
  • Quiet and cool
  • Everything rack mount
  • Two physical managed switches (network lab)
  • Modern hardware firewall (network lab)

Those are some heady requirements.  My solution to this was to do with AMD products.  AMD has multi-core chips for much cheaper than Intel, and although slower, they have a great bang for the buck for use in a ESXi environment, and I don’t personally think the price to performance ratio can be beat.  AMD’s version of VT-x is AMD-V, allowing virtualization enhancement, and like Intel, they have their own version of VT-d called IOMMU, which allows you to pass USB ports, video cards, and more through to virtual machines to use.  So, for example, you could create a virtual HTPC and passthrough a PCI-e graphics card for it to use.  However, even though this is enabled in the CPU, very few vendors actually enable it in the BIOS, and even less after that actually work with ESXi.  After much research and a couple of boards bought and sent back, I found a great AMD board that works out of the box with ESXi and gives you IOMMU support.  Later on, I’ll list my total lab hardware layout.

Total Virtual Specs for VMWare ESXi 5 Home Lab

  • Current as of September 2013
  • ESXi Physical Nodes: 3
  • Total Cores: 32
  • Total RAM: 128GB
  • Storage: 24TB RAID6 (Media Storage, Medium Performance iSCSI Target)
  • Storage: 1.2TB RAID0 on 15K SAS Drives (High Performance iSCSI Target)
  • Storage: 128GB SSD iSCSI Target (Highly Available, 2 nodes)
  • Storage: 1TB Local Drive per Node
  • Storage: 4GB USB Stick per Node (ESXi Boots From These)
  • Network Connectivity: 5 Gigabit NICs per Server (1 Motherboard + Quad Gigabit Intel Pro/1000 PCI-e)
  • Network Hardware: 2 x 24 Port Gigabit Smart Switches, 1 Netgear WNR3500L w/DD-WRT as my gateway, and 1 Juniper SSG5 firewall
  • Power: 2 x 20Amp dedicated power drops (each on own breaker)
  • Battery Backup: APC BR1500G + External Battery Pack (~30 min Runtime)

Hardware Specs for VMWare ESXi 5 Hosts for Home Lab

Below is my hardware list for my lab, including the sources that I got the items, and the prices at the time that I paid for them.  Shopping through eBay saved me a considerable amount of money, especially considering that you earn a form of “cash back” in the form of eBay Bucks.  I’d probably say I earned $200+ of eBay Bucks that knocked an additional amount of money off of my purchases.  Also, I could have made some cheaper purchases, like going with cheap, non-rackmount cases, but I spent a little more because I had a vision 😀  Finally, a note is that I purchased this over about a 3 month period; and bought it piecemeal, always getting what was on sale, rather than purchasing “whole” systems.  This saved me another solid chunk o’ money.

VMWare ESXi 5 Home Lab: Host 1

VMWare ESXi 5 Home Lab: Hosts 2 and 3

VMWare ESXi 5 Home Lab: Whitebox SAN #1 (Primary)

VMWare ESXi 5 Home Lab: Whitebox SAN #2 (Secondary)

VMWare ESXi 5 Home Lab: Miscellaneous Equipment

Total Cost for VMWare ESXi 5 Home Lab: $2,587.71

Space for VMWare ESXi 5 Home Lab

We purchased a house last year, and with it came a large, expansive, 1,300 square foot basement that stays cold, even in the summer.  Wanting a separate area to wall off the sound from the servers, and also have a room for inventory of hardware, I quickly claimed a corner of the basement.  Taking the corner, I framed and sheetrocked a 8’x20′ room with a door and one wall nothing but shelves, provided 2 dedicated 20Amp power drops, an air conditioning duct, networked the entire house with CAT6, and dropped all the networking in that room.  A small desk and chair with a monitor, keyboard, and KVM switch rounded out my “server room”.  A geek’s man cave.

Server Rack for VMWare ESXi 5 Home Lab

Wanting to not spend a lot of money here, and also not have a full 42U rack, I purchased 2 pair of 20U audio equipment rails, and built my own out of 2x4s with locking casters so that it’s mobile, yet I can lock it in place.  You can see the story, pictures, and plans at my post about my Home Server Rack.

House Computers/Hardware That Factored Into Decision

  • “Don-Windows7” (Primary Work/Personal Desktop): Windows 7 Ultimate, i7-2700K, 32GB DDR3-2100 RAM, dual Crucial M4 Sata 6GB/s 128GB SSDs (RAID0) for system/boot drive, 2 x 2TB green drives (RAID1) for extra storage, Seagate 3TB USB 3.0 External Drive for backups, 3 x 27″ Acer G276HL LED Monitors on Curved Triple Monitor Stand  and APC Smart 1500VA UPS.
  • XBMC HTPC VMs: 2 VMs, 2 Cores and 2.5GB of RAM each with HD6670s passed through w/HDMI and USB over CAT6 to standalone monitors and keyboards
  • Wife’s VM: 3 Cores, 4GB RAM, HD6670 Passed though w/HDMI and USB over CAT6 to standalone monitors and keyboards
  • Son’s VM: 3 Cores, 4GB RAM, HD7850 Passed though w/HDMI and USB over CAT6 to standalone monitors and keyboards
  • Various Electronics: Brother Multifunction Inkjet Printer w/LAN port, Brother Laser Printer w/LAN port, couple of Android tablets on WiFi, Android phones on wireless, as well as a DSI XL on wireless, and a couple of other devices like BluRay network players.

ESXi 5.0 Home Lab Diagram (Click to Enlarge)


ESXi 5.0 Home Lab Diagram

ESXi 5.0 Home Lab Diagram