December 3, 2008

Exchange 2007 SCC Clustering in VMWARE using ISCSI for the Masses (and how I went very wrong) – part 1…

I was reading the other day with great interest an article by a fellow blogger and online friend Elan Shudnow which takes you through how you can create an SSC Exchange 2007 Cluster using Windows 2008 and StarWind ISCSI – this excellent article is well worth a read.

Now as many of you well know, if you are regular readers of my blog that I tend to focus more on the CCR side of things however I have been considering an article about SCC (Single Copy Cluster) for a little while – but considering the excellent quality of Elan’s post I thought that it would be shoddy of me to put a regurgitated article out which would potentially read pretty much the same.

So I decided that I would try putting an SCC cluster together in VMWARE just to see how it all hangs together under Windows 2008 (essentially not planning to write a blog post on the subject) – it was at this point where the reason as to why Elan had elected to use ISCSI in conjunction with VMWARE with Windows 2008 became very (and embarrassingly) apparent.

Essentially I setup the VMWARE machines under VMWARE server in the usual way – creating both the local and shared disks – I then made the usual changes to the VMWARE VMX files on my cluster nodes (these are the files which contain each machines configuration) – just as an a side – when using Windows 2003 and you wish to setup shared cluster storage you enter in the following commands to each nodes VMX file:

SCSI.sharedBus = “virtual” (where x is the number of your SCSI controller – this allows for SCSI 2.0 protocol reservations to be made on the bus)

Disk.Locking = “false” (this allows multiple hosts to access a single VMWARE disk file)

I had assumed that this was all that would be needed to make the configuration work under Windows 2008 (I even ignored the section in the VMWARE 2.0 Admin’s guide which said “Windows 2008 Failover Cluster are not supported by VMWARE Server 2.0“).

I cracked on with the installation of Windows 2008 – did all of the usual bits and bobs (configuring Interfaces, domains, disks, accounts, etc) until I got to the point where I was ready to install Failover clustering.

I created the Cluster in Windows 2008 (there will be much more detail on this in a later part) and got to the point where I wanted to configure the Quorum shared disk resource (remember that I had at this point already done all of disk configurations).

In Windows 2008 before you can configure any shared disk for your cluster you have to add it to the storage pool – like so;


Which presents you with the following window;


Upon adding the disk I received the following error message and the disk that I had just added was crossed out with a red X:


Reviewing the Event Logs led to nothing of any real substance therefore I decided to have a look in the server storage management console which revealed the following status for the Quorum disk (which I had just tried to add):


Which was very odd as I had formatted the disk correctly.

I decided that my only option at this point was to go to my good friend Google for an answer, which is where I ended up being more than just little embarrassed – as it turned out I already knew the answer but just didn’t put two and two together.

Essentially VMWARE server only supports the SCSI-2 Protocol whereas Windows 2008 will only support SCSI-3 complaint disk access applications. When creating the reservation in the VMX file for the cluster node I was purely trying to get a SCSI-P2 disk to work with Windows 2008 as a persistent reservation – there was more chance of the hole in my bottom healing before this was going to work.

I knew this, and indeed it would appear that Elan knew this as well, it certainly explained his use of StarWind ISCSI – (as an ISCSI Initiator is built into Windows 2008 and indeed is a supported protocol for both Failover clustering and Exchange 2007 SP1) and by using StarWind he could present disks to his servers – this presented me with a conundrum.

In his article Elan had used the paid for version of the StarWind software as the Free Evaluation edition only allows for two ISCSI targets (essentially disks) and a single host to connect to the target therefore if you wanted to have more hosts and more Targets you would need to use either the 15 day evaluation – or – buy the server edition.

From my own personal perspective I could not afford $395 just for testing purpose so I set off in search of another way of doing this.

Before I go any further with the article I would like to cover a little bit technical information on ISCSI for readers whom may of not had a lot of exposure to the protocol and also provide a little information as to what actually changed in Windows 2008 which dropped support for SCSI-2 reservations;

What is ISCSI?

ISCSI (or Internet SCSI) is a TCP/IP based protocol which typically operates over ports 860 and 3260. Essentially there are two component parts to ISCSI – Initiators – essentially the clients whom make requests and Targets typically published disks which respond to the requests. The long and shot of the situation is that ISCSI allows for the SCSI protocol to be transmitted over IP networks.

You will find many articles which correctly explain that one of the major benefits to ISCSI is that it requires no further cabling beyond what you have already rolled out in terms of your CAT5 data network, but one tip I can give you is that when using Exchange server and ISCSI you should dedicated a VLAN which will give you a dedicated broadcast domain and IP subnet to both the ISCSI Initiators (the Exchange Servers) and the ISCSI Targets this should also be combined with a well defined Qos (Quality of service) policy on your network. This way you avoid general network contention associated with your Client / Server VLANS with the ISCSI traffic.

ISCSI can be secured via IPSEC or CHAP or use no form of security or authorisation.

For more information on ISCSI have a look here:

What does Microsoft say about ISCSI?

Well in Exchange 2003 you might be wise not consider it as a viable option which is mainly to do with the amount of disk I/O that Exchange 2003 could generate – however in Exchange 2007 ISCSI is considered a very viable option as long as best practices are adhered to – the following is quoted from the MS Exchange team blog in the following article:

While iSCSI does connect a server to storage over Ethernet, it is important to treat it as your storage connection, and completely isolate your iSCSI storage network from all other network traffic. If available, options such as flow control, quality of service, and jumbo frames can further increase performance. The Microsoft iSCSI Initiator 2.0 and later supports MPIO, and in our test labs we have pushed over 250MB/second over 3 network cards, proving iSCSI as a capable storage transport for scenarios where high throughput is required.

I had a reservation – where is my table (erm… disk)?

In Windows 2008 Microsoft decided that a total rewrite of the cluster disk driver was required (clusdisk.sys). In Windows 2003 the Cluster Disk driver had a direct path to the storage whereas in Windows 2008 it communicates with the partition manager (partmgr.sys) which essentially looks after (protects) cluster disk resources.

However the key thing which affected me was that in Windows 2003 all the SCSI commands (SCSI-2) or more to the point the SCSI-2 Reserve\Release actions were performed by the cluster disk driver writing directly to the disk sectors. In Windows 2008 SCSI-3 persistent reservation commands are the only method of access to protect a nodes rights to the disk. If the underling software (such as VMWARE) does not support SCSI-3 PR then you are stuffed (like I was).

Ok – got it – what now then?

It became clear to me that I was not going to be able to build an Exchange SCC cluster in VMWARE using Windows 2008 via VMWARE disk reservations so my only options was to use ISCSI – however, as I mentioned I did not wish to purchase StarWind so what could I do.

Well, I thought that StarWind could not be the only Windows based ISCSI Target software so I started looking around for other free based alternatives.

After a few hours of searching I found OpenFiler which is a Linux (OMG shock horror – this is an Exchange blog man for god’s sake) based NAS solution which allows for ISCSI targets to be configured and present via the Windows ISCSI Initiator.

OpenFiler is released under the GNU license and is released pretty much free for personal use (although you don’t get a manual nor any support however considering that I was using VMWARE I decided to download OpenFiler ISO, install it into a virtual machine and see if I could present ISCSI disks to my Windows 2008 servers (soon to be Exchange Servers) via OpenFiler.

You can imagine by virtue of the fact that I am blogging about it – I had some success! ;-)

In the Next Part

Ok, now the I have perhaps whetted your appetite (or bored you ridged) I would like to leave this here – however in the next part of this series I will be covering the following:

Installing and Configuring OpenFiler for ISCSI based Exchange Disks

Exchange 2007 Articles One Reply to “Exchange 2007 SCC Clustering in VMWARE using ISCSI for the Masses (and how I went very wrong) – part 1…”
Andy Grogan
Andy Grogan


One comment on “Exchange 2007 SCC Clustering in VMWARE using ISCSI for the Masses (and how I went very wrong) – part 1…

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.
%d bloggers like this: