Found this quite interesting… using a motherboard soley to power the PCI-e SAS Expander card…
http://www.servethehome.com/sas-expanders-diy-cheap-low-cost-jbod-enclosures-raid/
The guy also recommended using Histachi Deskstar:
http://www.newegg.com/Product/Product.aspx?Item=N82E16822145373
Maybe can use the above drive instead of Western Digital which is plagued with TLER issues… But strange.. the manufacturer only claims compatibility with HBA and not RAID cards were mentioned…
Well, so now that I have aggregated the links for my ESXI, guess it is time to aggregate my links for my SAN box. However, I face the same problem as:
http://www.nexentastor.org/boards/1/topics/874
I guess I will need to follow their procedure. Looks easy enough to understand but really wish someone would have provided the commands to type… especially for a solaris newbie like me.
After a long and painful struggle….
Well. so how do we do it. Basically, I just login as root from the console, and started meddling with:
1. Unconfigure Network Interfaces by typing setup network and following through the text wizard.
2. Setup Link Aggregation by typing setup network and following the wizard.
3. Type setup appliance init.
I was kind of confused in the beginning about this nmc thing. It did not look familiar at all… until I read:
http://serverfault.com/questions/237975/how-do-i-login-to-nexenta-in-expert-mode
Ah… so the nmc is some kind of supposedly user friendly shell. No wonder all my ifconfigs were suddenly not working… 
Commands to enter Expert Mode
option expert_mode=1
!bash
Seems like there is some kind of network bandwidth. Seem to see over 200+MBs of sporadic reads from the system when LARC is enabled.
Sometime back, I identified a network bottleneck between ESXI and my openfiler application. There I realised that one of my NICs seem to be only communicating with OpenFiler at 100 Mbps.
I quickly narrowed down the problem to my MOBO NIC being not as good as the Intel ones… and set off wanting to replace it.
Today before I replaced, it I did a reinstallation of ESXI on my thumbdrive and Lo and Behold:
Well did some googling and it seems that there might be the following problems:
Network Cable Not Properly Crimped
http://forums.whirlpool.net.au/archive/1561188
Cat5 are not rated to gigabit. And the quality of even cat5e patch cables needs to be very good.. gigabit uses all 4 pairs to connect.
Yeah! That solved all my problems. Check out my ESXI Network Adaptors! All Gigabyte Quality!
Quite an intelligent RAID card. I tried configuring RAID 0, RAID 1, RAID 5s.. move the entire RAID card + Hard Drives from one computer to another… and there was absolutely no problem. The new computer detected the RAID volumes accordingly and I could easily power up my Virtual Machines.
The most amazing thing is that… you do not even need to ensure that the same cables are attached to the same drives when you move from PC to PC. Really quite idiot proof…
Addendum: Hmm… apparently ZFS has such functionality too! We can just export out a pool and import it back in while moving the hard drives between servers.
http://superuser.com/questions/97754/zfs-moving-drives-around
Backing up ESXI USB Flash Drive. I was a little paranoid. There were a few boxes in my office that could be converted into an ESXI hosts simply by booting up from a USB disk. I kept a USB disk for each box.
Also, as new hardware is wired to the ESXI boxes, the configuration of the ESXI on the Flash Drive will change. To keep snapshots of such changes, I actually kept multiple USB Flash Drives for each device.
It was really stupid.
Today, I finally did some googling on how we can backup USB Flash Drives… and got my answer:
And… hey, I had that software all along. Time to convert all my ESXI backups into Acronis images….
Configuration Backup
Well besides backing up on the thumbdrive, we might want to backup the configuration of ESXI itself. Might save us some trouble from adding all the virtual machines again…
http://professionalvmware.com/2010/05/how-to-backup-esxi-configuration-the-missing-piece/
(Especially useful when u are always going about repairing or installing ESXI updates)
Well tested it out.. and yeap.. did restore the VMs configuration and license…
Some simple scripts after installing the VMWare CLI (which I dump in my bin folder):
backup-esxi.bat
set DATESTAMP=%DATE:~10,4%_%DATE:~4,2%_%DATE:~7,2%
set filename= "F:\Isos\Hypervisors\server_%DATESTAMP%.bak"
vicfg-cfgbackup.pl -save -server 192.168.1.2 -username root -password bugbear123 %filename%
restore-esxi.bat
set filename= "F:\Isos\Hypervisors\%1%"
vicfg-cfgbackup.pl -load -server 192.168.1.2 -username root -password bugbear123 %filename%
Okay.. after some really tiring contribution to ESXI whitelist. Lets get on to looking at using my MOBO for the SAN/NAS. I am looking at Nexenta.. and being really poor… will be trying out the Community Edition. You can see the difference between the various editions at:
http://www.nexenta.com/corp/nexentastor-overview/nexentastor-versions
Hmm.. the community version is a total of 576MB. Definitely a much bigger download than FreeNAS. Going for version 3.0.4. Hope it goes well…
Wanted to get a new motherboard with more SATA cables and extension slots for my home made NAS/SAN. But since I got the board, might as well do the ESXI community some good by testing if the hardware is compatible with ESXI 4.0.
Well, I managed to install it… so I guess it must be! Got it installed on Esxi 4.1 Update 1 (10 Feb 2010).
There might hit 2 snags though in the installation though:
Problem 1: System hangs at loading module usb
Solution 1: Disable C1E Support in BIOS. Press Delete to enter Bios. Go to Advanced. Go to CPU Configuration and Disable C1E Support (4th Option)
No idea why this is so but got some help from a useful post at:
http://communities.vmware.com/thread/272255
Problem 2: Unable to load module /usr/lib/vmware/vmkmod/vmfs
Well, if you are like me, trying to cheapo and get lucky by hoping that you can get by without the Intel PRO/1000 GT Adaptor, you will probably encounter the above error.
Soution 2: Plug in some Intel NIC that is supported by VMWare. We should also try disabling the motherboard’s in-built LAN.
http://communities.vmware.com/message/1571740
Hope this will help some other ESXI tinkerers.
Hurray! I contributed a bit back by posting at the forum:
http://www.vm-help.com/forum/viewtopic.php?f=13&t=2954&e=0
But I wonder why they did not include my motherboard in the main compatibility list at:
Just when I thought I had done enough evaluation on NAS/SAN… I just chanced on another one – Nexenta. Lots of reading ahead of me…
which one shall it be?
So I was trying to set up my ESXi to run off ISCSI on an OpenFiler SAN. I got terribly performance. Sequential Reads and 512k Reads were capped at 10MBps.
Hmmm… this reeked strangely of a bottleneck in the network card.. and sure enough the network connection in ESXI is only at 100Mbps.
Upgraded the connection with a Intel Pro GT1000 Adaptor and the benchmarks went right back to over 40MB/sec.
I suspect the bottleneck is now on the MOBO NIC on openfiler itself. Well, did not have enough high quality NICs to rectify the issue… so this will have to suffice… until…. my Intel Pro 1000 PT Quad Server Adaptor comes to the rescue…
