iSCSI - fast, cheap and reliable?

iSCSI is often touted as the new way forward for enterprise storage needs. This is mostly a rant/information about what I have found useful so far.

To begin with iSCSI support in linux is pretty good and you will find a lot of resources on how to set it up etc. Here are the quick steps.

• yum install iscsi-initiator-utils(or whatever tool you use to install stuff)
  
• chkconfig iscsid on; chkconfig iscsi on (or update-rc)
  
• echo "iscsi initiator name" > /etc/iscsi/initiatorname.iscsi (but put the real name in, make one up thats unique to this host). Make sure that the iscsi volume you want to mount now allows this initiator name to connect to it.
• service iscsid start
• iscsiadm -m discovery -t sendtargets -p IPofIScsiHost
• service iscsi start (now you should see a device sdx appear in dmesg)
• when you put it in /etc/fstab, you need "_netdev" in the options so that it doesn't try to mount it before the network is available.

(you can get fancy with CHAP initiator names etc, but for most cases they aren't needed).

That should present an iSCSI block device to your system. Check your dmesg for the device name. Unfortunately, I was unable to find a simple iSCSI command that lists the mapped device name and the corresponding iSCSI lun information. In the old days (with the cisco? iSCSI tools), there used to be a nice little iscsi-ls command that would list all your iSCSI devices and where they came from etc. You now have to manually map /proc/scsi/scsi entries with the /sys/class/scsi_disk/lun-number/ and dig for the entries that way.

I have found that iSCSI in general is not all that fast. You can get decent performance out of it, but nothing close to dedicated local disk and a good hardware raid solution. I have seen asymmetric read-write performance on iscsi disks - with writes being a lot faster than reads. Also, don't waste your money on hardware iSCSI initiators. The don't help a whole lot with speed and are a pita to maintain (with out-of-kernel drivers in some cases). Most modern servers have plenty of horsepower to drive iscsi using the CPUs. I have heard that running iscsi on a 10Gbps network helps (we run it on a 1 Gbps network) with the performance.

iSCSI is in general cheaper than comparable full SAN solutions, but, performance does suffer. This is okay for most cases, except maybe for i/o heavy db applications etc. Choosing the right vendor for your NAS device matters a lot. Don't believe the specs you get from the vendors, try a few of them, perform some tests (bonnie, plain old dd, iozone) and decide how much you want to sink into it.

HTH someone out in the ether!