All-In-One VM Lab / NAS / Mame Arcade / Media center / Workstation

With the increase of my power bills I thought I'd find a solution to having multiple computers doing multiple tasks. I've been working with VMware ESX/Vsphere for awhile now and thought there would have to be a way to incorporate all my requirements into a beefy home server setup. I surfed around the net finding ideas and other peoples experiences to see whats possible and whats not.. thanks to those threads!

First step was to determine what I wanted out of this server..

• Home LAB system for testing software/applications/Operating systems
• Active Directory/dns etc for my home LAN
• Central Storage for Home (NAS)
• Central storage for Dropbox/Skydrive shared through my LAN
• Home Media PC or Arcade Mame setup.

I recently brought a new desktop PC and thought this would be a good base little did I know that I purchased the correct parts for what I was trying to do. Since I wanted to use vspheres VMDirectPath passthrough for my Video card and USB devices for this my CPU/Motherboard required VT-D.

My Hardware

• Motherboard : ASUS P8Z77-M PRO (Bios version 1805)
• CPU : Intel i7 3770
• RAM : Geil 32GB Kit (4x8GB), PC-10666(1333MHz) EVO Veloce Hot Rod Red DDR3
• Raid HBA : ibm m1015 Crossflashed to LSI 9220-8i IT-mode Learn how at servethehome.com
• SATA Card : JMirco JMB362/JBM/363 (A cheap card I had laying about if you want to use this to install esxi on there is a patch so esxi can see this card i'll try and find the link later)
• SSD : 120gb Corsair SATA3
• SSD : 120gb Kingston SATA3
• 8 x : Western Digital Black 1TB
• Video : ATI Radeon HD 3600 PCI-E (Better success rate of vmware passthrough with ATI v's Nvidia so I've read may have changed now)
• Eaton UPS
• Intel dual 1gb PCI-e Network card
• Optional
• TV-Tuner Card (I can't fit one in this setup at the moment no more slots)
• Arcade MAME Hardware (buttons,joysticks,USB PC Encoder) source www.diyarcade.com

Software

• VMware ESXi 5.0.1 (Don't use 5.1 at this point it'll break the PCI Passthrough)
• FreeNAS 8.3.0 Release-P1-64bit
• Ditched using FreeNAS due to my Virtual server suffering, went back to NexentaStore and instant speed!
• Windows 7 64bit
• Server 2008R2 64bit
• Proxy Server (cache enabled for youtube etc.. save on data)
• Hyperspin
• Mediaportal
• Vyatta (Used for Virtual Routers will work with VMXnet3 10gb nics) Find out how here

BIOS settings Generic

• Make sure you enable vitalise technology on your CPU

BIOS Settings for Video Passthrough for ESXi
There are a few settings required to be set on my motherboard so the video passthrough will function correctly. More motherboard configs are on this awesome thread

• Primary Video set to PCIe
• Disable iGPU
• Disable onboard Sound

 Core Systems
Virtual Hyperviser
For my underlying Virtual Hyperviser I decided to use ESXi 5.0.1 simply because I use this software at work and am familiar with it. As stated above don't update to ESXi 5.1 because at this point in time (Jan-2013) it'll break your PCI Passthrough.

OS is installed on a 16gb Sandisk cruiser contour and set as first boot in BIOS. (OS will probably be moved to an SSD in the future)

Set Hardware for Passthrough
Before you start configuring VM's you'll have to enable the required hardware for passthrough. To do this run the vcenter client and connect to your server. Once loaded click on your server IP/HOST click Configurations tab and then Advanced Settings, If you don't have a compatible set of hardware this will be grayed out and not configurable if not click edit. Select which bits of hardware you want to put in passthrough and click OK

*NOTE: BIG NOTE about hardware passthrough, Once you've configured the passthrough on a bit of hardware after ESXi reboots its not accessible from the ESXi server anymore. Example I have my ESXi OS on my USB stick when I was testing I selected the wrong USB port to passthrough and after the server booted it enabled that port for hardware passthrough and my USB stick went offline. The server still booted and I could use it as normal but I couldn't save anything back to the USB stick and when I added this USB as passthrough to a VM I would see the USB stick with all my ESXi partitions within my VM. Lucky I could just reboot the ESXi with the USB stick in another port that wasn't in passthrough and fix my mistake.

Also if you put your primary video card your using for the ESXi server in passthrough it'll look like ESXi has stalled on boot, this isn't the case its just up to the part where the ESXi set the video card in passthrough.

Core VMS
I've configured two VM's which are stored on the Corsair SSD, The first one is the Primary Domain Controller / DNS for my domain, Since FreeNAS is configured to use windows ACLS without the domain up and running before FreeNAS the Active Directory Service will fail. Second VM is FreeNAS which has the IBM m1015 raid card and the JMicro SATA card set in hardware passthrough. The IBM m1015 raid card will be used to host the 8x1TB WD disks and the JMicro with the SSD will be used for a ZFS LOG disk. This will speed up my NFS exports to my Vmware server (Highly recommend this config if you'll be using NFS)

FreeNAS   Using NexentaStore out of box configuration

The first attempt at building a VM FreeNAS 8.3.0 and passthrough my m1015 card failed, there seems to be a error when you use FreeNAS with this type of configuration, the server will boot but will hang with an error message "run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config mps_startup" So off to google I went..... It seems a few people have had this and the solution is pretty simple.

• Build your FreeNAS VM (Remove any unwanted devices from the VM and also disable unwanted devices in the VM BIOS)
• Install FreeNAS (WIHTOUT the ibm m1015 raid card installed.
• Open the FreeNAS Admin website
• browse to System --> Tunables (You only need to add one of them which enables/disables but i can't remember which so I just add both)
• Add hw.pci.enable_msix 0
• Add hw.pci.enabled_msi 1
• shutdown your VM
• Add the ibm m1015 raid card and boot your system.

As stated above I've got the M1015 hosting the 8 WD 1TB disk running a Raidz2 config (Gives me 2 x parity) I dont have the room for a spare disk at the moment and the SSD running as LOG. I've configured the server with 6GB of RAM this should handle my working set. I've not enabled de-dupe although I'd love to since I'm using VM's but that would totally blow out my RAM which I'm not willing to do.

ZFS CRC Error gotcha
When I first configured Nexenta/FreeNAS with a ZFS data type I noticed my raid kept producing CRC errors on the which followed me over Nexenta/FreeNAS. I thought it may have been the RAID card I got my ebay and flashed all the way to the PSU, In the end a work mate said "You've tested the memory right" umm I was going to but never go around to it. I downloaded ultimatebootcd and ran memtester straight away RED RED RED well theres my problem. This was on my old 16gb Cosair RAM so I sent that back and brought the Geil 32gb.

(IF YOU WANT TO USE FREENAS) Adding 10gb Vmxnet3 module for FreeNAS
Another good addition to an All-In-One system is to add a 10gb Network card since most of your systems will be within the same box it makes sense to open up the bandwidth. Where I found the answer and what I did is below.

• In the ESXi Console of the FreeNAS server select the VM -> Guest -> Install/Upgrade VM Tools
• logon to the FreeNAS shell (Option 9)
• mkdir /mnt/cdrom
• mount -t cd9660 /dev/cd0 /mnt/cdrom
• cp /mnt/cdrom/vmware-freebsd-tools.tar.gz /mnt/thepit/data/    (thepit is my volume I created yours will differ)
• tar zxvf /mnt/thepit/data/vmware-freebsd-tools.tar.gz
• mount -rw /
• cp /mnt/thepit/data/vmware-tools-distrib/lib/modules/binary/FreeBSD8.0-amd64/vmxnet.ko /boot/modules/
• Go back to FreeNAS Admin Web site
• browse to System -> Tunables and add vmxnet3_load and set to yes
• reboot and configure the new network device.

(IF YOU WANT TO USE FREENAS) *NOTE: If your using NFS theres an option to enable Asynchronous Mode which says it will help performance beyond gigabit speeds, I've done no testing if this  helps but I enabled it anyway since I have a 10gb network now active.

(IF YOU WANT TO USE FREENAS) Now you can Configure your FreeNAS ready to present datastores to your ESXi. I use NFS datastores and use the follow base NFS exports for esxi.

• vmdk - For storing the Virtual Machines
• vmdk_page - For Storing the Page files of the Windows machines so they dont chew up data in the snapshots.
• iso - Used to store all my OS/Application iso
• vmdk_templates - Used to store templates for deployment (when you have a vcenter server installed

(IF YOU WANT TO USE FREENAS) Once the volumes are created and your choice of protcol is configured NFS/iSCSI connect them to your ESXi server and we're ready to start building some VM's on the FreeNAS Datastores.

Configure Workstation
VM Settings

• Dual CPU
• 2gb RAM (When using passthrough for video it seems theres a issue with going over 2gb ram there are some possible workarounds but I found 2gb gave me enough)
• PCI Device - USB Controller (Used to connect Keyboard/mount/MS Remote/Arcade Controllers)
• Network vmxnet3 10gb
• DONT ADD THE ATI VIDEO YET
• Install OS and patch up to date
• turn off and take a snapshot (will remove later once we're happy with config)
• Optional : PCIDevice - TV-tuner if space is an issue check theses out I personally haven't used them yet but looks good.

UPDATE: If you want more than 2gb of ram on your VM add the following to to your advanced settings of your VM

• pciHole.start="1200"
• pciHole.end="2200"

Adding Video Card

With my configuration if I add the video card and try and boot into windows it'll blue screen and reboot which isn't great. To get around this and install the video drivers theres a process I have to follow.

• Once the snapshot is taken start the machine again and login
• click start -> run and type msconfig.exe
• click boot tab and select safeboot  in the boot options
• If you dont select Network you'll have to copy over the ATI drivers to the machine before restarting. The drivers I used were  13-1-legacy_vista_win7_win8_64_dd_ccc.exe
• shutdown the computer
• Add the video card
• PCIDevice - ATI Video
• PCIDevice - ATI HDMI Audio
• start VM
• Install the drivers, dont install the control center
• run msconfig.exe and remove the safemode settings
• Reboot

Upon reboot you may not see the login window on your VM screen you might get the extended screen, go have a look at your monitor connected to the video card you've put in passthrough mode on the VM it should now show the desktop of the VM and if your mouse and keyboard have been installed correctly you can use it as a workstation. Since I'll be using this screen only I opened the display settings and selected to only show on screen two (which was my physical screen) DON'T go into hardware settings and uninstall the VMware display adapter.

Now that you've got your base install of your Video passthrough workstation ready you can shutdown and remove the old snapshot. Also if you want to you can put in multiple video cards and multiple USB ports in passthrough which can be setup as two workstation in a study on one computer. I haven't tried this but I've read others using this with success.

Home Theatre Computer
Now that the base system is configured you can install your favourite Home threatre software. I've read that some video cards have issues with display video when using hardware DXVA on my setup above I dont have any issues.

I use mediaportal on my setup but a mate uses Plex since he doesn't use it to watch live tv, I'd probably use Plex to if I didn't have the live tv and record requirement. I was a mythtv fanboy for a long time but since using mediaportal I've found the community alot larger and has alot more plugins.

Arcade Computer
My future project is to create a arcade computer for all the old consoles/arcade. I was going to use an old pc and have it dedicated for the arcade box but I wanted to use Hyperspin. Hyperspin was created in Flash 10 which didn't support GPU offloading so its very CPU heavy so my old PC just didn't cut it, Sure theres other arcade frontends which would work but they don't look as cool as Hyperspin. Since my arcade machine will be in the games room and will be a arcade machine and also a bit of furniture I wanted it to look pretty. On my setup I can use a VM with dual CPU and 2gb RAM and it works great.

I have a USB controller board to connect my buttons and joysticks upto which is plugin and play with Windows7 and shows up as two game controllers so that should much effort to implement. You can find them on ebay if you search for "PC & PS3 USB ARCADE CONTROLLER"

UPDATE:
I've connect the USB Controller and arcade buttons and joysticks and configured Hyperspin, This seems to work very well under the Virtual Machine even plays Dreamcast games. Since the systems are all-in-one I dont have to load the hyperspin and ROMS onto the virtual machine I just play them off a network share.

I'm also running rocksmith PC game within the VM also, When I first installed the game I had very poor sound crackling and was slow. I googled and found ubisofts troubleshooting page with some things to try to clear up audio Link to Ubisoft . The option that I used was ForceWDM=1 within the rocksmith.ini I also had to change the resolution to 800x600 from 1080p since my video card couldn't handle it. I played some songs fine but as soon as it came to the performance with the people in the back ground I did notice some lag, I assume thats to do with the video card.

Dropbox and Skydrive Shared Over a LAN
I have two account one with Dropbox (4.25gb) and another with Microsoft Skydrive (25gb) I like the idea of having some data accessible to me at all times over the "cloud" but I didn't want my multiple computers at home setup with the client on each. My thinking was easy, Share it out from a central server and just use it as normal windows shares, Sounded easy enough.

First problem the clients only run when the user that configures them is logged on, Second problem dropbox or skydrive doesn't like to install to a network share and I didn't want to store the files within the VMDK.

Solution - Doesn't the folder located on network shares

• Created two iSCSI LUNS on the NAS
• Connected to the Server and formatted as required and give drive letters.

Solution - Runs only when user is logged on.

• Created a service account for dropbox and skydrive (i.e service-cloud) mine was a domain account
• logged onto the computer with svc-cloud
• Install and configure Dropbox and Skydrive clients and store the data on the iSCSI drives.
• download and install Windows 2003 Resource Kit (need srvany.exe)
• Copied srvany.exe c:\apps\
• Created two Services
• sc create "<Service Name i.e. dropbox>" binpath= "c:\apps\srvany.exe"   <- note the space after the =
• Run regedit
• HKLM\system\currentcontrolset\services\dropbox
• create new key "parameters"
• create new string "Application"
• edit Application with path of dropbox, mine was "C:\Users\<username>\AppData\Roaming\Dropbox\bin\dropbox.exe"
• repeat for skydrive my path was "C:\Users\<username>\AppData\Local\Microsoft\SkyDrive\skydrive.exe"
• open services , Start -> run -> services.msc
• find dropbox and skydrive
• change the logon user and password to your service account.
• Set to automatic (I set mine to automatic with delay just to make sure)
• leave them as stopped
• logout and back in with a different user
• open services.msc and start the services.
• open task manager and make sure that your service account user now has two srvany.exe programs and dropbox.exe and skydrive.exe running.
• Check your iSCSI drives to see if they're now starting to fill with your files.

Summary
So after that long build session I've learnt alot and now have a all-in-one nas/lab/arcade/htpc all running with minimal power. The LAB system is currently running with about 20 VMs.. System memory running about 70% and cpu currently 13% so room to move. My 'cloud' storage (dropbox/skydrive) shared locally over windows shared folder which is always active since its running as a srvany.exe so I can upload a file/picture/document and tell my wife its on the skydrive shared folder...which is all controlled with GPOs from my AD.

The only concern I have is that it is an all-in-one which means a huge single point of failure, being a home lab system etc I'm not so worried.The important data is backed up externally and a ZFS snapshot schedule is also configured. Other things like pictures we store on skydrive/dropbox and also backup. The great thing with the ZFS file system is if the FreenNAS nexenta dies I just bring up another nexenta FreeNAS VM import the ZFS volume and restore the settings file (Backed off externally from my NAS ofcourse)

I hope this blog post helps other to make a nice little all-in-one Lab box and I'd like to hear how you go..

Gav.