Friday, September 3, 2021

Migrating TrueNAS / FreeNAS Virtual Machines to Proxmox

Purpose : I'm migrating from TrueNAS Core to TrueNAS Scale and underlying OS has change from FreeBSD to Debian. This means they've changed from using the FreeBSD Bhyve Hypervisor to Debian with KVM. 

I want to migrate to the new system with as little downtime as possible, I host a couple of things locally which I don't want to be down to long.

Pre-Req

We use a program called qemu-img to conver the raw VM disk images to qcow2. On my TrueNAS install the external sources were disable and local were enabled. To install the program I enabled the external and disable local temporarily 
  1. SSH to TrueNAS server
    1. nano /usr/local/etc/pkg/repos/local.conf 
      1. change enabled: no
    2. nano nano /usr/local/etc/pkg/repos/FreeBSD.conf
      1. change enabled: yes
    3. pkg install qemu-utils
    4. If it asks you to update your pkg package say no
    5. Remember to change back the sources to the original when qemu-utils have been installed.

Exporting VM from TrueNAS 12

  • SSH to your TrueNAS Server
  • change to /dev/zvol
  • change to your folder where the VMs are stored. As an example mine is /dev/zvol/SSD-Data
  • ls the directory to get the vm names, as an example i have openproject
  • run qemu-img convert -f raw -O qcow2 openproject /mnt/SSD-Data/openproject.qcow2
NOTE: you will need to output your qcow2 file to the /mnt/<folder> or else it'll fail if you try to store in /dev/zvol/<folder>

Importing VM to Proxmox


Copy file to Proxmox

  • SSH into proxmox 
  • create folder /root/vm.import
  • Copy the exported qcow2 file to the proxmox server /root/vm.import directory
For the copying I used WINSCP program to copy the file locally and then uploaded to the proxmox server

Build destination VM 

There needs to be a Virtual Machine created before importing so you can get the VM ID number.
Create a VM with the required specs
NOTE: For my TrueNAS VM's I need to create a VM with BIOS of OVMF (UEFI) since that's how they were built on the TrueNAS Server.

Once the server is build you can remove the hard disk which was created with it. 

Importing VM Disk

  • SSH to proxmox
  • change to /root/vm.import
    • run qm importdisk <vm id> <diskname> <proxmox storage>
    • example qm importdisk 104 openproject.qcow2 QNAP
      • QNAP is the storage name on proxmox
      • 104 is the VM ID of the new VM i created
  • When the transfer is 100% and successful you can continue.
  • remember the name of the file: i.e. QNAP:104/vm-104-disk-0.raw

Attaching the VM Disk

  • SSH into proxmox
  • type qm set <vm id> --scsi0 <vm image name>
    • example : qm set 104 --scsi0 QNAP:104/vm-104-disk-0.raw
NOTE:If you get an error with booting the machine you may need to change the boot device like I did.

Changing Boot Order

  • SSH to proxmox
  • cd etc/pve/qemu-server
  • nano <server id>.conf
    • example nano 104.conf
  • change boot: order=ide2;net0 to boot: order=scsi0
  • save file
  • start virtual machine

Updating Network

Depending on your virtual machine you may need to change the network.
Im using ubuntu 18 and 20 and the network adapter has changed from enp0s4 to ens18
  • ssh into virtual machine
  • find new network name
    • type dmesg | grep eth
    • this will show what the network has changed to. 
      • example virtio_net virtio2 ens18: renamed from eth0
  • nano /etc/netplan/<filename>.yaml
    • example : sudo nano /etc/netplan/50-cloud-init.yaml
  • change the ethernet name from the old to the new 
    • example change enp0s4 to ens18
    • save and exit
  • Make sure the old system is powered off then....
  • type sudo netplan apply

REFERENCES : 

Friday, August 27, 2021

How to Install bareos in a TrueNAS/FreeBSD Jail with bareos-webui

*NOTE: These are quick notes for when I've been installing the software - they're not extensive, its more for record and to help anyway else.

Software : bareos

Planform : TrueNAS CORE 12.0-U5

Jail : Release 12.2

Purpose : Backup local virtual machines and remote computers/files/sql

Login to Jail

  1. SSH to TrueNAS 
  2. type iocage list
  3. find the jail and type iocage console <name> i.e. iocage console bareos

Install pre-reqs

  1. pkg install -y wget postgresql12-server libxml2 nano php74 curl ZendFramework-php74 mod_php74
  2. Setup postgresql
    1. sysrc postgresql_enable=yes
    2. /usr/local/etc/rc.d/postgresql initdb
    3. service postgresql start
  3. Setup php
    1. ln -s /usr/local/etc/php.ini-production /usr/local/etc/php.ini
    2. sysrc php_fpm_enable=yes
    3. service php-fpm start
  4. Edit /usr/local/etc/php.ini
    1. nano /usr/local/etc/php.ini
    2. Under the section ;Paths and Directories add
      1. include_path = ".:/usr/local/share/ZendFramework/library"
    3. service php-fpm restart

Install bareos

  1. Install bareos using script below
    1. nano install_bareos.sh (copy paste script below)
    2. chmod +x install_bareos.sh
    3. ./install_bareos.sh

<<SCRIPT>> - start

#!/bin/sh

# See https://download.bareos.org/bareos/release/
# for applicable releases and distributions

DIST=FreeBSD_12.2
# or
# DIST=FreeBSD_12.1
# DIST=FreeBSD_11.4

RELEASE=release/20
# or
# RELEASE=experimental/nightly

URL=https://download.bareos.org/bareos/$RELEASE/$DIST

# add the Bareos repository
cd /etc/pkg
wget -q $URL/bareos.conf

# install Bareos packages
pkg install --yes bareos.com-director bareos.com-storage bareos.com-filedaemon bareos.com-database-postgresql bareos.com-bconsole

# setup the Bareos database
su postgres -c /usr/lib/bareos/scripts/create_bareos_database
su postgres -c /usr/lib/bareos/scripts/make_bareos_tables
su postgres -c /usr/lib/bareos/scripts/grant_bareos_privileges

# enable services
sysrc bareosdir_enable=YES
sysrc bareossd_enable=YES
sysrc bareosfd_enable=YES

# start services
service bareos-dir start
service bareos-sd start
service bareos-fd start

<<SCRIPT>> - end

 

Post Install Tasks

  1. Update bconsole password
  2. Copy password cat /usr/local/etc/bareos/bareos-dir.d/director/bareos-dir.conf
  3. Insert into /usr/local/etc/bareos/bconsole.conf
  4. test by typing bconsole should show successful connection
Installing bareos web-ui (apache24)
  1. pkg install -y apache24
  2. sysrc apache24_enable=yes
  3. service apache24 start
    1. NOTE * If you get an error talking about "Could not reliably determine the servers's fully qualified domain name" you'll need to edit /usr/local/etc/apache24/httpd.conf and search for ServerName and update as directed. i.e. i didn't have a FQDN so put my local ip address.
  4. pkg install -y bareos-webui
  5. edit /usr/local/etc/apache24/httpd.conf
    1. backup defeault httpd.conf mv /usr/local/etc/apache24/httpd.conf /usr/local/etc/apache24/httpd.conf.bck
    2. Create new httpd.conf file nano httpd.conf
      1. add the below test and edit the ServerName to reflect the servers IP
    3. restart the service service apache24 restart
    4. open a broswers and go to http://<serverip>/bareos-webui
    5. Login with
      1. username : admin
      2. password : admin
      3. NOTE* if you get a login error 
        1. run bconsole
        2. run reload
      4. try logging back in
  6. Update BackupCatalog job to refresh bvfs cache - this is to show the files within the webui there is an issue if you don't run this after the job that no files show up.
    1. nano /usr/local/etc/bareos/bareos-dir.d/job/BackupCatalog.conf
    2. under RunAfterJob input the following.
  Run Script {
    Console = ".bvfs_update"
    RunsWhen = After
    RunsOnClient = No
  }

NOTE : Changing the ALIAS within the httpd.conf file from bareos-webui breaks the rewrite you'll also need to update the RewriteBase value in httpd.conf to.

FILE httpd.conf

ServerRoot "/usr/local"
Listen 80


LoadModule mpm_prefork_module libexec/apache24/mod_mpm_prefork.so
LoadModule authn_file_module libexec/apache24/mod_authn_file.so
LoadModule authn_core_module libexec/apache24/mod_authn_core.so
LoadModule authz_host_module libexec/apache24/mod_authz_host.so
LoadModule authz_groupfile_module libexec/apache24/mod_authz_groupfile.so
LoadModule authz_user_module libexec/apache24/mod_authz_user.so
LoadModule authz_core_module libexec/apache24/mod_authz_core.so
LoadModule access_compat_module libexec/apache24/mod_access_compat.so
LoadModule auth_basic_module libexec/apache24/mod_auth_basic.so
LoadModule reqtimeout_module libexec/apache24/mod_reqtimeout.so
LoadModule filter_module libexec/apache24/mod_filter.so
LoadModule mime_module libexec/apache24/mod_mime.so
LoadModule log_config_module libexec/apache24/mod_log_config.so
LoadModule env_module libexec/apache24/mod_env.so
LoadModule headers_module libexec/apache24/mod_headers.so
LoadModule setenvif_module libexec/apache24/mod_setenvif.so
LoadModule version_module libexec/apache24/mod_version.so
LoadModule unixd_module libexec/apache24/mod_unixd.so
LoadModule status_module libexec/apache24/mod_status.so
LoadModule autoindex_module libexec/apache24/mod_autoindex.so
<IfModule !mpm_prefork_module>
        #LoadModule cgid_module libexec/apache24/mod_cgid.so
</IfModule>
<IfModule mpm_prefork_module>
        #LoadModule cgi_module libexec/apache24/mod_cgi.so
</IfModule>
LoadModule dir_module libexec/apache24/mod_dir.so
LoadModule alias_module libexec/apache24/mod_alias.so
LoadModule php7_module        libexec/apache24/libphp7.so
LoadModule rewrite_module libexec/apache24/mod_rewrite.so

 


# Third party modules
IncludeOptional etc/apache24/modules.d/[0-9][0-9][0-9]_*.conf

User www
Group www

ServerAdmin you@example.com
ServerName 192.168.1.210

# Bareos WebUI Apache configuration file
#

# Environment Variable for Application Debugging
# Set to "development" to turn on debugging mode or
# "production" to turn off debugging mode.
<IfModule env_module>
        SetEnv "APPLICATION_ENV" "production"
</IfModule>

Alias /bareos-webui  /usr/local/www/bareos/public

<Directory /usr/local/www/bareos/public>

        Options FollowSymLinks
        AllowOverride None

        # Following module checks are only done to support
        # Apache 2.2,
        # Apache 2.4 with mod_access_compat and
        # Apache 2.4 without mod_access_compat
        # in the same configuration file.
        # Feel free to adapt it to your needs.

        # Apache 2.4
        <IfModule mod_authz_core.c>
                <IfModule mod_access_compat.c>
                    Order deny,allow
                </IfModule>
                Require all granted
        </IfModule>

        <IfModule mod_rewrite.c>
                RewriteEngine on
                RewriteBase /bareos-webui
                RewriteCond %{REQUEST_FILENAME} -s [OR]
                RewriteCond %{REQUEST_FILENAME} -l [OR]
                RewriteCond %{REQUEST_FILENAME} -d
                RewriteRule ^.*$ - [NC,L]
                RewriteRule ^.*$ index.php [NC,L]
        </IfModule>

        <IfModule mod_php5.c>
                php_flag magic_quotes_gpc off
                php_flag register_globals off
        </IfModule>

<IfModule dir_module>
    DirectoryIndex index.php
</IfModule>



</Directory>

<Files ".ht*">
    Require all denied
</Files>

#
# ErrorLog: The location of the error log file.
# If you do not specify an ErrorLog directive within a <VirtualHost>
# container, error messages relating to that virtual host will be
# logged here.  If you *do* define an error logfile for a <VirtualHost>
# container, that host's errors will be logged there and not here.
#
ErrorLog "/var/log/httpd-error.log"

#
# LogLevel: Control the number of messages logged to the error_log.
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
#
LogLevel warn

<IfModule log_config_module>
    #
    # The following directives define some format nicknames for use with
    # a CustomLog directive (see below).
    #
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    <IfModule logio_module>
      # You need to enable mod_logio.c to use %I and %O
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>

    #
    # The location and format of the access logfile (Common Logfile Format).
    # If you do not define any access logfiles within a <VirtualHost>
    # container, they will be logged here.  Contrariwise, if you *do*
    # define per-<VirtualHost> access logfiles, transactions will be
    # logged therein and *not* in this file.
    #
    CustomLog "/var/log/httpd-access.log" common

    #
    # If you prefer a logfile with access, agent, and referer information
    # (Combined Logfile Format) you can use the following directive.
    #
    #CustomLog "/var/log/httpd-access.log" combined
</IfModule>

<IfModule alias_module>
    #
    # Redirect: Allows you to tell clients about documents that used to
    # exist in your server's namespace, but do not anymore. The client
    # will make a new request for the document at its new location.
    # Example:
    # Redirect permanent /foo http://www.example.com/bar

    #
    # Alias: Maps web paths into filesystem paths and is used to
    # access content that does not live under the DocumentRoot.
    # Example:
    # Alias /webpath /full/filesystem/path
    #
    # If you include a trailing / on /webpath then the server will
    # require it to be present in the URL.  You will also likely
    # need to provide a <Directory> section to allow access to
    # the filesystem path.

    #
    # ScriptAlias: This controls which directories contain server scripts.
    # ScriptAliases are essentially the same as Aliases, except that
    # documents in the target directory are treated as applications and
    # run by the server when requested rather than as documents sent to the
    # client.  The same rules about trailing "/" apply to ScriptAlias
    # directives as to Alias.
    #
    ScriptAlias /cgi-bin/ "/usr/local/www/apache24/cgi-bin/"

</IfModule>

<IfModule cgid_module>
    #
    # ScriptSock: On threaded servers, designate the path to the UNIX
    # socket used to communicate with the CGI daemon of mod_cgid.
    #
    #Scriptsock cgisock
</IfModule>

#
# "/usr/local/www/apache24/cgi-bin" should be changed to whatever your ScriptAliased
# CGI directory exists, if you have that configured.
#
<Directory "/usr/local/www/apache24/cgi-bin">
    AllowOverride None
    Options None
    Require all granted
</Directory>

<IfModule headers_module>
    #
    # Avoid passing HTTP_PROXY environment to CGI's on this or any proxied
    # backend servers which have lingering "httpoxy" defects.
    # 'Proxy' request header is undefined by the IETF, not listed by IANA
    #
    RequestHeader unset Proxy early
</IfModule>

<IfModule mime_module>
    #
    # TypesConfig points to the file containing the list of mappings from
    # filename extension to MIME-type.
    #
    TypesConfig etc/apache24/mime.types

    #
    # AddType allows you to add to or override the MIME configuration
    # file specified in TypesConfig for specific file types.
    #
    #AddType application/x-gzip .tgz
    #
    # AddEncoding allows you to have certain browsers uncompress
    # information on the fly. Note: Not all browsers support this.
    #
    #AddEncoding x-compress .Z
    #AddEncoding x-gzip .gz .tgz
    #
    # If the AddEncoding directives above are commented-out, then you
    # probably should define those extensions to indicate media types:
    #
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz

    #
    # AddHandler allows you to map certain file extensions to "handlers":
    # actions unrelated to filetype. These can be either built into the server
    # or added with the Action directive (see below)
    #
    # To use CGI scripts outside of ScriptAliased directories:
    # (You will also need to add "ExecCGI" to the "Options" directive.)
    #
    #AddHandler cgi-script .cgi

    # For type maps (negotiated resources):
    #AddHandler type-map var

    #
    # Filters allow you to process content before it is sent to the client.
    #
    # To parse .shtml files for server-side includes (SSI):
    # (You will also need to add "Includes" to the "Options" directive.)
    #
    #AddType text/html .shtml
    #AddOutputFilter INCLUDES .shtml
</IfModule>

#
# The mod_mime_magic module allows the server to use various hints from the
# contents of the file itself to determine its type.  The MIMEMagicFile
# directive tells the module where the hint definitions are located.
#
#MIMEMagicFile etc/apache24/magic

#
# Customizable error responses come in three flavors:
# 1) plain text 2) local redirects 3) external redirects
#
# Some examples:
#ErrorDocument 500 "The server made a boo boo."
#ErrorDocument 404 /missing.html
#ErrorDocument 404 "/cgi-bin/missing_handler.pl"
#ErrorDocument 402 http://www.example.com/subscription_info.html
#

#
# MaxRanges: Maximum number of Ranges in a request before
# returning the entire resource, or one of the special
# values 'default', 'none' or 'unlimited'.
# Default setting is to accept 200 Ranges.
#MaxRanges unlimited

#
# EnableMMAP and EnableSendfile: On systems that support it,
# memory-mapping or the sendfile syscall may be used to deliver
# files.  This usually improves server performance, but must
# be turned off when serving from networked-mounted
# filesystems or if support for these functions is otherwise
# broken on your system.
# Defaults: EnableMMAP On, EnableSendfile Off
#
#EnableMMAP off
#EnableSendfile on

# Supplemental configuration
#
# The configuration files in the etc/apache24/extra/ directory can be
# included to add extra features or to modify the default configuration of
# the server, or you may simply copy their contents here and change as
# necessary.

# Server-pool management (MPM specific)
#Include etc/apache24/extra/httpd-mpm.conf

# Multi-language error messages
#Include etc/apache24/extra/httpd-multilang-errordoc.conf

# Fancy directory listings
#Include etc/apache24/extra/httpd-autoindex.conf

# Language settings
#Include etc/apache24/extra/httpd-languages.conf

# User home directories
#Include etc/apache24/extra/httpd-userdir.conf

# Real-time info on requests and configuration
#Include etc/apache24/extra/httpd-info.conf

# Virtual hosts
#Include etc/apache24/extra/httpd-vhosts.conf

# Local access to the Apache HTTP Server Manual
#Include etc/apache24/extra/httpd-manual.conf

# Distributed authoring and versioning (WebDAV)
#Include etc/apache24/extra/httpd-dav.conf

# Various default settings
#Include etc/apache24/extra/httpd-default.conf

# Configure mod_proxy_html to understand HTML4/XHTML1
<IfModule proxy_html_module>
Include etc/apache24/extra/proxy-html.conf
</IfModule>

# Secure (SSL/TLS) connections
#Include etc/apache24/extra/httpd-ssl.conf
#
# Note: The following must must be present to support
#       starting without SSL on platforms with no /dev/random equivalent
#       but a statically compiled-in mod_ssl.
#
<IfModule ssl_module>
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
</IfModule>

Include etc/apache24/Includes/*.conf

<FilesMatch "\.php$">
    SetHandler application/x-httpd-php
</FilesMatch>
<FilesMatch "\.phps$">
    SetHandler application/x-httpd-php-source
</FilesMatch>





                                     







Thursday, February 21, 2013

All-In-One VM Lab / NAS / Mame Arcade / Media center / Workstation

With the increase of my power bills I thought I'd find a solution to having multiple computers doing multiple tasks. I've been working with VMware ESX/Vsphere for awhile now and thought there would have to be a way to incorporate all my requirements into a beefy home server setup. I surfed around the net finding ideas and other peoples experiences to see whats possible and whats not.. thanks to those threads!

First step was to determine what I wanted out of this server..
  • Home LAB system for testing software/applications/Operating systems
  • Active Directory/dns etc for my home LAN
  • Central Storage for Home (NAS)
  • Central storage for Dropbox/Skydrive shared through my LAN
  • Home Media PC or Arcade Mame setup.
I recently brought a new desktop PC and thought this would be a good base little did I know that I purchased the correct parts for what I was trying to do. Since I wanted to use vspheres VMDirectPath passthrough for my Video card and USB devices for this my CPU/Motherboard required VT-D.

My Hardware
  • Motherboard : ASUS P8Z77-M PRO (Bios version 1805)
  • CPU : Intel i7 3770
  • RAM : Geil 32GB Kit (4x8GB), PC-10666(1333MHz) EVO Veloce Hot Rod Red DDR3
  • Raid HBA : ibm m1015 Crossflashed to LSI 9220-8i IT-mode Learn how at servethehome.com
  • SATA Card : JMirco JMB362/JBM/363 (A cheap card I had laying about if you want to use this to install esxi on there is a patch so esxi can see this card i'll try and find the link later)
  • SSD : 120gb Corsair SATA3
  • SSD : 120gb Kingston SATA3
  • 8 x : Western Digital Black 1TB
  • Video : ATI Radeon HD 3600 PCI-E (Better success rate of vmware passthrough with ATI v's Nvidia so I've read may have changed now)
  • Eaton UPS
  • Intel dual 1gb PCI-e Network card
  • Optional
    • TV-Tuner Card (I can't fit one in this setup at the moment no more slots)
  • Arcade MAME Hardware (buttons,joysticks,USB PC Encoder) source www.diyarcade.com

Software
  • VMware ESXi 5.0.1 (Don't use 5.1 at this point it'll break the PCI Passthrough)
  • FreeNAS 8.3.0 Release-P1-64bit
  • Ditched using FreeNAS due to my Virtual server suffering, went back to NexentaStore and instant speed!
  • Windows 7 64bit
  • Server 2008R2 64bit
  • Proxy Server (cache enabled for youtube etc.. save on data)
  • Hyperspin
  • Mediaportal
  • Vyatta (Used for Virtual Routers will work with VMXnet3 10gb nics) Find out how here
BIOS settings Generic
  • Make sure you enable vitalise technology on your CPU
BIOS Settings for Video Passthrough for ESXi
There are a few settings required to be set on my motherboard so the video passthrough will function correctly. More motherboard configs are on this awesome thread
  • Primary Video set to PCIe
  • Disable iGPU
  • Disable onboard Sound

 Core Systems
Virtual Hyperviser
For my underlying Virtual Hyperviser I decided to use ESXi 5.0.1 simply because I use this software at work and am familiar with it. As stated above don't update to ESXi 5.1 because at this point in time (Jan-2013) it'll break your PCI Passthrough.

OS is installed on a 16gb Sandisk cruiser contour and set as first boot in BIOS. (OS will probably be moved to an SSD in the future)

Set Hardware for Passthrough
Before you start configuring VM's you'll have to enable the required hardware for passthrough. To do this run the vcenter client and connect to your server. Once loaded click on your server IP/HOST click Configurations tab and then Advanced Settings, If you don't have a compatible set of hardware this will be grayed out and not configurable if not click edit. Select which bits of hardware you want to put in passthrough and click OK

*NOTE: BIG NOTE about hardware passthrough, Once you've configured the passthrough on a bit of hardware after ESXi reboots its not accessible from the ESXi server anymore. Example I have my ESXi OS on my USB stick when I was testing I selected the wrong USB port to passthrough and after the server booted it enabled that port for hardware passthrough and my USB stick went offline. The server still booted and I could use it as normal but I couldn't save anything back to the USB stick and when I added this USB as passthrough to a VM I would see the USB stick with all my ESXi partitions within my VM. Lucky I could just reboot the ESXi with the USB stick in another port that wasn't in passthrough and fix my mistake.

Also if you put your primary video card your using for the ESXi server in passthrough it'll look like ESXi has stalled on boot, this isn't the case its just up to the part where the ESXi set the video card in passthrough.

Core VMS
I've configured two VM's which are stored on the Corsair SSD, The first one is the Primary Domain Controller / DNS for my domain, Since FreeNAS is configured to use windows ACLS without the domain up and running before FreeNAS the Active Directory Service will fail. Second VM is FreeNAS which has the IBM m1015 raid card and the JMicro SATA card set in hardware passthrough. The IBM m1015 raid card will be used to host the 8x1TB WD disks and the JMicro with the SSD will be used for a ZFS LOG disk. This will speed up my NFS exports to my Vmware server (Highly recommend this config if you'll be using NFS)

FreeNAS   Using NexentaStore out of box configuration

The first attempt at building a VM FreeNAS 8.3.0 and passthrough my m1015 card failed, there seems to be a error when you use FreeNAS with this type of configuration, the server will boot but will hang with an error message "run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config mps_startup" So off to google I went..... It seems a few people have had this and the solution is pretty simple.
  • Build your FreeNAS VM (Remove any unwanted devices from the VM and also disable unwanted devices in the VM BIOS)
  • Install FreeNAS (WIHTOUT the ibm m1015 raid card installed.
  • Open the FreeNAS Admin website
  • browse to System --> Tunables (You only need to add one of them which enables/disables but i can't remember which so I just add both)
    • Add hw.pci.enable_msix 0
    • Add hw.pci.enabled_msi 1
  • shutdown your VM
  • Add the ibm m1015 raid card and boot your system.
As stated above I've got the M1015 hosting the 8 WD 1TB disk running a Raidz2 config (Gives me 2 x parity) I dont have the room for a spare disk at the moment and the SSD running as LOG. I've configured the server with 6GB of RAM this should handle my working set. I've not enabled de-dupe although I'd love to since I'm using VM's but that would totally blow out my RAM which I'm not willing to do.

ZFS CRC Error gotcha
When I first configured Nexenta/FreeNAS with a ZFS data type I noticed my raid kept producing CRC errors on the which followed me over Nexenta/FreeNAS. I thought it may have been the RAID card I got my ebay and flashed all the way to the PSU, In the end a work mate said "You've tested the memory right" umm I was going to but never go around to it. I downloaded ultimatebootcd and ran memtester straight away RED RED RED well theres my problem. This was on my old 16gb Cosair RAM so I sent that back and brought the Geil 32gb.

(IF YOU WANT TO USE FREENAS) Adding 10gb Vmxnet3 module for FreeNAS
Another good addition to an All-In-One system is to add a 10gb Network card since most of your systems will be within the same box it makes sense to open up the bandwidth. Where I found the answer and what I did is below.
  • In the ESXi Console of the FreeNAS server select the VM -> Guest -> Install/Upgrade VM Tools
  • logon to the FreeNAS shell (Option 9)
  • mkdir /mnt/cdrom
  • mount -t cd9660 /dev/cd0 /mnt/cdrom
  • cp /mnt/cdrom/vmware-freebsd-tools.tar.gz /mnt/thepit/data/    (thepit is my volume I created yours will differ)
  • tar zxvf /mnt/thepit/data/vmware-freebsd-tools.tar.gz
  • mount -rw /
  • cp /mnt/thepit/data/vmware-tools-distrib/lib/modules/binary/FreeBSD8.0-amd64/vmxnet.ko /boot/modules/
  • Go back to FreeNAS Admin Web site
  • browse to System -> Tunables and add vmxnet3_load and set to yes
  • reboot and configure the new network device.
(IF YOU WANT TO USE FREENAS) *NOTE: If your using NFS theres an option to enable Asynchronous Mode which says it will help performance beyond gigabit speeds, I've done no testing if this  helps but I enabled it anyway since I have a 10gb network now active.

(IF YOU WANT TO USE FREENAS) Now you can Configure your FreeNAS ready to present datastores to your ESXi. I use NFS datastores and use the follow base NFS exports for esxi.
  • vmdk - For storing the Virtual Machines
  • vmdk_page - For Storing the Page files of the Windows machines so they dont chew up data in the snapshots.
  • iso - Used to store all my OS/Application iso
  • vmdk_templates - Used to store templates for deployment (when you have a vcenter server installed
(IF YOU WANT TO USE FREENAS) Once the volumes are created and your choice of protcol is configured NFS/iSCSI connect them to your ESXi server and we're ready to start building some VM's on the FreeNAS Datastores.

Configure Workstation
VM Settings
  • Dual CPU
  • 2gb RAM (When using passthrough for video it seems theres a issue with going over 2gb ram there are some possible workarounds but I found 2gb gave me enough)
  • PCI Device - USB Controller (Used to connect Keyboard/mount/MS Remote/Arcade Controllers)
  • Network vmxnet3 10gb
  • DONT ADD THE ATI VIDEO YET
  • Install OS and patch up to date
  • turn off and take a snapshot (will remove later once we're happy with config)
  • Optional : PCIDevice - TV-tuner if space is an issue check theses out I personally haven't used them yet but looks good.
UPDATE: If you want more than 2gb of ram on your VM add the following to to your advanced settings of your VM
  • pciHole.start="1200"
  • pciHole.end="2200"

Adding Video Card

With my configuration if I add the video card and try and boot into windows it'll blue screen and reboot which isn't great. To get around this and install the video drivers theres a process I have to follow.
  • Once the snapshot is taken start the machine again and login
  • click start -> run and type msconfig.exe
  • click boot tab and select safeboot  in the boot options
    • If you dont select Network you'll have to copy over the ATI drivers to the machine before restarting. The drivers I used were  13-1-legacy_vista_win7_win8_64_dd_ccc.exe
  • shutdown the computer
  • Add the video card
    • PCIDevice - ATI Video
    • PCIDevice - ATI HDMI Audio
  • start VM
  • Install the drivers, dont install the control center
  • run msconfig.exe and remove the safemode settings
  • Reboot
Upon reboot you may not see the login window on your VM screen you might get the extended screen, go have a look at your monitor connected to the video card you've put in passthrough mode on the VM it should now show the desktop of the VM and if your mouse and keyboard have been installed correctly you can use it as a workstation. Since I'll be using this screen only I opened the display settings and selected to only show on screen two (which was my physical screen) DON'T go into hardware settings and uninstall the VMware display adapter.

Now that you've got your base install of your Video passthrough workstation ready you can shutdown and remove the old snapshot. Also if you want to you can put in multiple video cards and multiple USB ports in passthrough which can be setup as two workstation in a study on one computer. I haven't tried this but I've read others using this with success.

Home Theatre Computer
Now that the base system is configured you can install your favourite Home threatre software. I've read that some video cards have issues with display video when using hardware DXVA on my setup above I dont have any issues.

I use mediaportal on my setup but a mate uses Plex since he doesn't use it to watch live tv, I'd probably use Plex to if I didn't have the live tv and record requirement. I was a mythtv fanboy for a long time but since using mediaportal I've found the community alot larger and has alot more plugins.

Arcade Computer
My future project is to create a arcade computer for all the old consoles/arcade. I was going to use an old pc and have it dedicated for the arcade box but I wanted to use Hyperspin. Hyperspin was created in Flash 10 which didn't support GPU offloading so its very CPU heavy so my old PC just didn't cut it, Sure theres other arcade frontends which would work but they don't look as cool as Hyperspin. Since my arcade machine will be in the games room and will be a arcade machine and also a bit of furniture I wanted it to look pretty. On my setup I can use a VM with dual CPU and 2gb RAM and it works great.

I have a USB controller board to connect my buttons and joysticks upto which is plugin and play with Windows7 and shows up as two game controllers so that should much effort to implement. You can find them on ebay if you search for "PC & PS3 USB ARCADE CONTROLLER"

UPDATE:
I've connect the USB Controller and arcade buttons and joysticks and configured Hyperspin, This seems to work very well under the Virtual Machine even plays Dreamcast games. Since the systems are all-in-one I dont have to load the hyperspin and ROMS onto the virtual machine I just play them off a network share.

I'm also running rocksmith PC game within the VM also, When I first installed the game I had very poor sound crackling and was slow. I googled and found ubisofts troubleshooting page with some things to try to clear up audio Link to Ubisoft . The option that I used was ForceWDM=1 within the rocksmith.ini I also had to change the resolution to 800x600 from 1080p since my video card couldn't handle it. I played some songs fine but as soon as it came to the performance with the people in the back ground I did notice some lag, I assume thats to do with the video card.

Dropbox and Skydrive Shared Over a LAN
I have two account one with Dropbox (4.25gb) and another with Microsoft Skydrive (25gb) I like the idea of having some data accessible to me at all times over the "cloud" but I didn't want my multiple computers at home setup with the client on each. My thinking was easy, Share it out from a central server and just use it as normal windows shares, Sounded easy enough.

First problem the clients only run when the user that configures them is logged on, Second problem dropbox or skydrive doesn't like to install to a network share and I didn't want to store the files within the VMDK.

Solution - Doesn't the folder located on network shares
  • Created two iSCSI LUNS on the NAS
  • Connected to the Server and formatted as required and give drive letters.
Solution - Runs only when user is logged on.
  • Created a service account for dropbox and skydrive (i.e service-cloud) mine was a domain account
  • logged onto the computer with svc-cloud
  • Install and configure Dropbox and Skydrive clients and store the data on the iSCSI drives.
  • download and install Windows 2003 Resource Kit (need srvany.exe)
  • Copied srvany.exe c:\apps\
  • Created two Services
    • sc create "<Service Name i.e. dropbox>" binpath= "c:\apps\srvany.exe"   <- note the space after the =
    • Run regedit
      • HKLM\system\currentcontrolset\services\dropbox
      • create new key "parameters"
      • create new string "Application"
        • edit Application with path of dropbox, mine was "C:\Users\<username>\AppData\Roaming\Dropbox\bin\dropbox.exe"
      • repeat for skydrive my path was "C:\Users\<username>\AppData\Local\Microsoft\SkyDrive\skydrive.exe"
  • open services , Start -> run -> services.msc
  • find dropbox and skydrive
  • change the logon user and password to your service account.
  • Set to automatic (I set mine to automatic with delay just to make sure)
  • leave them as stopped
  • logout and back in with a different user
  • open services.msc and start the services.
  • open task manager and make sure that your service account user now has two srvany.exe programs and dropbox.exe and skydrive.exe running.
  • Check your iSCSI drives to see if they're now starting to fill with your files.

Summary
So after that long build session I've learnt alot and now have a all-in-one nas/lab/arcade/htpc all running with minimal power. The LAB system is currently running with about 20 VMs.. System memory running about 70% and cpu currently 13% so room to move. My 'cloud' storage (dropbox/skydrive) shared locally over windows shared folder which is always active since its running as a srvany.exe so I can upload a file/picture/document and tell my wife its on the skydrive shared folder...which is all controlled with GPOs from my AD.

The only concern I have is that it is an all-in-one which means a huge single point of failure, being a home lab system etc I'm not so worried.The important data is backed up externally and a ZFS snapshot schedule is also configured. Other things like pictures we store on skydrive/dropbox and also backup. The great thing with the ZFS file system is if the FreenNAS nexenta dies I just bring up another nexenta FreeNAS VM import the ZFS volume and restore the settings file (Backed off externally from my NAS ofcourse)

I hope this blog post helps other to make a nice little all-in-one Lab box and I'd like to hear how you go..

Gav.