I was just kind of curious. Will there be any performance difference when we run bonnie on a single disk vs a mirror disks array?
Mirror Disk Array Performance
Single Disk
Dun really see a any performance bonus using a mirrored disk array over a single disk using bonnie…
Very relevant and interesting PDF slides..
http://wikis.sun.com/download/attachments/102400619/S304013_OpenStorage_ZFS_COMSTAR.pdf
http://wikis.sun.com/display/OpenSolarisInfo200906/COMSTAR+Prerequisites
http://www.nexenta.org/projects/site/wiki/SettingISCSI
http://support.nexenta.com/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=132&nav=0,12
Well based on those resources, I was able to configure my open solaris to share ISCSI just like how Nexenta has done it. Hmm… so my suspicion is confirmed. Nexenta Community is bascially a snazzy frontend over all the solaris terminal commands…
Installation Command:
wget -O – www.napp-it.org/nappit | perl
Addendum: Even Linux seems to have iSCSI…
Hmmm.. this guy face the exact bandwidth limit!
I was wondering if 75MB/s means I did something wrong.. because if you multiply 75 by 8.. you still fall far short of a 1000.
Hmmm.. maybe the infiniband network is something I should look at too… looks like u can connect direct between 2 PCs without a switch! That is real cool…
Imagine I can connect it with ZFS – ESXI. That will be so cool!
Hardware Supported List:
http://communities.vmware.com/cshwsw.jspa?sortField=1&sortOrder=1&start=25
Seems to suggest we need a switch with subnet manager.
Also explains how we can download the right drivers for the mellanox Connect-X HBA cards
EBay Options
Okay… looks.. not too easy
http://www.nexentastor.org/boards/1/topics/1113
Needs cards with memory
is that Mellanox also has working drivers which is good. Also important to note, Solaris needs cards with onboard memory as you cannot use the memory free ones which utilize system memory instead. Finally, FreeBSD and OpenSolaris were ROUGH to set up
Well, my Link Aggregation is definitely not working.. which is kind of expected… ‘cos I am currently using a cheapo 35 USD 8 port switch… I mean.. if that works… it will be kind of amazing.
I tried running my benchmarks simultaneously on both VMS, i.e. Picasso and Davinci which are both on a LUN on my ZFS computer. Their bandwidth is like halfed…
(Compared to the blazing fast response at http://james.com.sg/2011/04/10/esxidisk-reads-benchmarking/)
You might be thinking.. maybe its your harddisk that is slow? Well, I actually did a
show performance network
on the nexenta system … and guess what… most of the traffic is still routed through e1000g0. Almost nothing at all goes to e1000g1. 
Guess will have to work on it tonight and try to upgrade it to a much nicer switch… this will have to wait till at night.. when nobody is at the office.. .and yeah! I got the whole office to myself again!
TRENDnet TEG-240WS
Okay. Just swapped in my TRENDnet…
After Trunk Settings
Link Aggregation Results
Hmmm… not sure if this means it is partially working… somehow for ZFS system, it magically dedicates one port to Tx and the other to Rx. So I guess the operation for ZFS is sort of working on the ZFS server.
However, for ESXI, it is definitely not working. Both VMs are still hitting at the same PortID 4.
Found this quite interesting… using a motherboard soley to power the PCI-e SAS Expander card…
http://www.servethehome.com/sas-expanders-diy-cheap-low-cost-jbod-enclosures-raid/
The guy also recommended using Histachi Deskstar:
http://www.newegg.com/Product/Product.aspx?Item=N82E16822145373
Maybe can use the above drive instead of Western Digital which is plagued with TLER issues… But strange.. the manufacturer only claims compatibility with HBA and not RAID cards were mentioned…
Well, so now that I have aggregated the links for my ESXI, guess it is time to aggregate my links for my SAN box. However, I face the same problem as:
http://www.nexentastor.org/boards/1/topics/874
I guess I will need to follow their procedure. Looks easy enough to understand but really wish someone would have provided the commands to type… especially for a solaris newbie like me.
After a long and painful struggle….
Well. so how do we do it. Basically, I just login as root from the console, and started meddling with:
1. Unconfigure Network Interfaces by typing setup network and following through the text wizard.
2. Setup Link Aggregation by typing setup network and following the wizard.
3. Type setup appliance init.
I was kind of confused in the beginning about this nmc thing. It did not look familiar at all… until I read:
http://serverfault.com/questions/237975/how-do-i-login-to-nexenta-in-expert-mode
Ah… so the nmc is some kind of supposedly user friendly shell. No wonder all my ifconfigs were suddenly not working… 
Commands to enter Expert Mode
option expert_mode=1
!bash
Sometime back, I identified a network bottleneck between ESXI and my openfiler application. There I realised that one of my NICs seem to be only communicating with OpenFiler at 100 Mbps.
I quickly narrowed down the problem to my MOBO NIC being not as good as the Intel ones… and set off wanting to replace it.
Today before I replaced, it I did a reinstallation of ESXI on my thumbdrive and Lo and Behold:
Well did some googling and it seems that there might be the following problems:
Network Cable Not Properly Crimped
http://forums.whirlpool.net.au/archive/1561188
Cat5 are not rated to gigabit. And the quality of even cat5e patch cables needs to be very good.. gigabit uses all 4 pairs to connect.
Yeah! That solved all my problems. Check out my ESXI Network Adaptors! All Gigabyte Quality!
Quite an intelligent RAID card. I tried configuring RAID 0, RAID 1, RAID 5s.. move the entire RAID card + Hard Drives from one computer to another… and there was absolutely no problem. The new computer detected the RAID volumes accordingly and I could easily power up my Virtual Machines.
The most amazing thing is that… you do not even need to ensure that the same cables are attached to the same drives when you move from PC to PC. Really quite idiot proof…
Addendum: Hmm… apparently ZFS has such functionality too! We can just export out a pool and import it back in while moving the hard drives between servers.
http://superuser.com/questions/97754/zfs-moving-drives-around
Just when I thought I had done enough evaluation on NAS/SAN… I just chanced on another one – Nexenta. Lots of reading ahead of me…
which one shall it be?
