Suggested Hub Transport Hardware Config for Exchange 2007 Installations of 5000 users…

by Andy Grogan on February 3, 2008 · 0 comments

in Exchange 2007 (General), Exchange 2007 (Hub Transport), Exchange 2007 Design

As many of you know I have been working through the gradual transition of my organisations Exchange installation from Exchange 2003 to Exchange 2007. This as I am sure you are all aware requires many hours of research, testing, specification then implementation on all aspects of the migrated system.

One of the things that I have noticed when researching the various aspects of Exchange 2007 is that it is very hard to find a reference which answers questions simply at first glance. For example;

“I wish to know what is a good Basic Hub Transport server specification for up to 5000 users, whom send and receive on average 68,245 messages a day (aver –Total);”

In order to ascertain the answer to the above question you will need to spend large amounts of time on various web sites using Sizing Calculators, analysing IO requirements – determining RAM sizes and CPU overhead – which for me is fun, however I have noticed that many Exchange administrators would like to find an article that essentially says –

“Look, buy this server – stick these processors in, with this amount of RAM, and this disk config and you won’t go far wrong…”

In this article I would like to provide a base specification for people whom are looking to size their Hub Transport servers for mail transport of up to 5000 mail users, whereby I cut out the need to in-depth server performance analysis (although I will include much for the technical detail for those whom are interested).

The recommendations contained within this article cover the following aspect of HT server hardware design:

  • CPU
  • Memory
  • Disk (Size, RAID)

I have based this article around the HP ProLiant DL365 Server which is pretty much the dedicated Opteron version of the DL 360. I have chosen this model as it offers a good trade off between cost / performance and capacity, you can view the full specifications of this server here: http://h18004.www1.hp.com/products/quickspecs/12564_div/12564_div.html

It is at this stage that I need to point out that I am a HP (Compaq) server man, this is not because I have shares in the company, nor is it down to HP asking me to write this article – I just like HP in the same way that others like IBM and DELL servers – the specification of the HP server that is provided at the end of this article can easily be replicated into specifications for most other server vendors, so please do not feel that I am encouraging you to run out and buy HP servers – stick with what you are happy with.

Hub Transport – CPU Metrics:

The following information is based upon sizing recommendations as feature in: http://technet.microsoft.com/en-us/library/aa998874(EXCHG.80).aspx

Choosing the Correct Processor;

Choosing the correct processor for Exchange 2007 is an important business, however it is perhaps not as complicated as designing Disk Sub System requirements – as in terms of your processors in Exchange 2007 you have a set of pre-defined rules right from the outset for all production installations of Exchange server:

 

So essentially at this stage you have a toss up between modern Intel Xeons or the AMD Opteron range. I have always been an Intel person myself, however I do urge you to seriously look at the AMD Opteron offerings as they can save you a reasonable amount of money on the total cost of the server unit for very little (if at all noticeable) performance drop.

How Many Cores?

We all know that over the last 24 months the big new thing in processor technology is additional processor cores on a single chip – but, does it actually improve Exchange performance?

Microsoft and the Exchange team state very clearly that multiple CPU cores can have a significant impact on the overall performance of Exchange Server and they provide some really interesting reading on the subject here: CPU and Memory Scalability for Exchange 2003 (which can be applied to Exchange 2007) and if you are really interested there is a good article on the MAPI Messaging Benchmarking 3 and how Dual Core processors fare have a look here: http://www.microsoft.com/technet/prodtechnol/exchange/2003/performance.mspx.

However whilst keeping in mind the purpose of the article (keep it simple and HT) – multiple core processors mean better performance at a competitive price – I recently received a quote where the Quad core option was cheaper than the Dual – so shop around is the key.

In practice terms of looking at Microsoft “best practice” for Hub Transport Processors and Cores the following recommendations are made:

Minimum:

x 1 processors cores – Essentially this is the absolute base processor configuration that you will require in order to get Microsoft PSS to talk to you should you have a problem (one would wonder how you could have less than a single processor core) – one assumes that this is based upon being able to handle a Light user message profile (5 messages sent and 20 received at around 50KB per message) – you could potentially also be looking at handling a Average user message profile (10 sent and 40 received at around 50 KB per message) – but be prepared for some potentially small latency under this scenario.

If we assume 5000 LIGHT users sending and receiving mail on an average day here you would be looking at: 125000 messages per day – I have left out the potential space implications as 50KB per message is highly subjective.

If we assume 5000 AVERAGE users sending and receiving mail on an average day here you would be looking at: 250000 messages per day – I have left out the potential space implications as 50KB per message is highly subjective.

Note: The figures given above are based around 5000 people sending and receiving that amount of mail per day (again given as an average based upon traffic that I have seen in my organisation) – it is unlikely that all 5000 people would consistently amount to that level traffic per day – so I would recommend reducing the figures given by around %20 or base them on your own understanding of your mail system.

Recommended:

x 4 processor cores – this recommendation is taken on the basis of the following formula price + memory configuration = performance, essentially if you are using x 4 cores and have the correct memory configuration (see more on this later) – this would the be the best practice configuration you should aim for.

Essentially here you would be looking to effectively handle the Average user message profile (10 sent and 40 received at around 50 KB per message) or the Heavy User Message Profile (20 sent and 80 received at around 50KB per message) but both comfortably rather than in the previous section potentially looking at some form of latency.

If we assume 5000 AVERAGE users sending and receiving mail on an average day here you would be looking at: 250000 messages per day – I have left out the potential space implications as 50KB per message is highly subjective.

If we assume 5000 Heavy users sending and receiving mail on an average day here you would be looking at: 500000 messages per day – I have left out the potential space implications as 50KB per message is highly subjective.

Note: The figures given above are based around 5000 people sending and receiving that amount of mail per day (again given as an average based upon traffic that I have seen in my organisation) – it is unlikely that all 5000 people would consistently amount to that level traffic per day – so I would recommend reducing the figures given by around %20 or base them on your own understanding of your mail system.

Maximum:

x 8 Processor Cores – One thing to note about the maximum requirement about Hub Transport processor cores is that it should not Unnecessarily be considered the “final say” so to speak. This recommendation is based around HT performing message transport in a very busy environment, however if third party applications are used which interface with the HT role there might be scope for additional processing power – but generally this processor configuration can be considered appropriate for most medium to large scale transport messaging environments of greater than 500000 messages per day.

Summary of CPU Metrics:

There are a number of CPU considerations that can be applied to the Hub Transport role, however given the scenario of the article and indeed the base for the message transactions I would personally recommend a two physical processor server where the two processors in the slots have a pair of cores each (resulting in the recommended four cores) if you are on a budget the AMD Opteron™ Processor 2220 2.80 GHz is a good choice, however if you are a devout Intel server admin then most of their Dual core offerings above 2GHz will also suffice (bear in mind that you might at this stage be able to acquire Quad cores at competitive rates).

Hub Transport – Memory Metrics:

In Exchange 2007 – the more physical RAM that you have generally means more that can be cached – therefore a greater performance gain.

As we all know that server memory comes in many forms however as a basis I would recommend that the modules that you choose confirm to 667MHz PC5300 standard. When looking at the amount of RAM that you place in your Hub Transport the following metrics are recommended:

  • If you have a server with 4 processor cores you should consider 4 GB of RAM (working on the recommendation of 1GB per core)
  • If you have a server with 8 processor cores your should consider 8 GB of RAM

 

The above are based upon Microsoft best practice recommendations – however I would suggest that if you have 4 processor cores consider 6 GB or RAM – this allows for the O/S and Message transport to be taken care of given our 5000 user scenario (you will also find that 6GB or RAM is future proofed for any upturn in user population or message traffic).

Reading though Microsoft white papers you may also see see that the maximum memory recommendation (although as Microsoft are again quick to point out this is not a physical maximum, merely a trade off over performance and price) for Hub Transport RAM is 16 GB – bear in mind that this is based upon 1 million messages averaged on the number of recipients so for the purposes of the article 6GB should be more than enough – even Microsoft states “The Edge Transport and Hub Transport Roles do not require substantial amounts of memory to perform well in optimal conditions”

Hub Transport – Disk Metrics:

Getting the Disk configuration of the Hub Transport correct is a key part of implementing a successful Exchange 2007 implementation – lots of focus is given to getting the Mailbox Server Disk metrics correct, but many people over look the fact that the HT makes use of the same Database Technology as the mailbox server in order to transport Messages.

During my time working with Exchange I have found that the main gripe that users have with e-mail (aside from it not being available) is that it took over 5 minutes for their message inviting everyone out for dinner to arrive with a recipients (Man I remember that days of Microsoft Mail Post Offices – where users were happy if their message arrived at all!). Essentially transport lag annoys people.

Disk Type / Speed and Size:

There are a number of options that you can pursue in terms of the Disk Technology that can be used with Exchange in general. Speaking openly from my own personal perspective there has been a swing from SAN based storage (in Exchange 2003) back to DAS being a recommendation in Exchange 2007 (however you still have the option of SAN storage in 2007 in fact it is likely for large installations with many storage group it will potentially be a requirement – that is unless you wish to consider the now supported iSCSI option).

For the purposes however of the Hub Transport (and this article) we will be looking at Direct Attached Storage.

Firstly what technology should you use? – well disk technology has moved on in recent years with the introduction of SATA and SAS;

SATA:

SATA in my humble opinion is still a little slow for Enterprise class solutions, don’t get me wrong, I know that there is the SATA E class drives which are designed to run 24 x 7 and at speeds of 10K RPM, but I am not certain if there is much of a financial saving between buying SATA E or just opting for SAS.

SAS:

SAS (Serial Attached SCSI) is designed for Enterprise class performance and reliability. Access times and platter speeds are excellent (although when using the smaller form factor drives you are limited to smaller disk sizes – however given the topic of the article the drive sizes should be more than enough the HT role) – HP provide SAS 2.5” and 3.6” in the following capacities:

10K RPM = 36 GB / 72 GB / 146 GB

15K RPM = 36 GB / 72 GB / 146 GB / 300 GB

My personal preference for the HT role would be to use x 6 15K 146 GB or 300 GB drives (this will potentially vary according to budget – if you can opt for the 300 GB drives) the DL 365 G1 has capacity for x 5 2.5” SAS drives (which when configured represent x 3 RAID 1 pairs).

The following diagram illustrates how the disks can be allotted to RAID arrays via the built in P400i – each Array configured on the controller represents two separate spindles configured as RAID 1 (this size of the Array will either be 146GB or 300GB) where each Array is labelled A,B and C – this is all configured from the HP Arrays Configuration utility.

HP DL 365 G1 Copyright HP 2008

The P400i is supplied with 256 MB of battery backed cache which not only provides backup for the Transport Databases in the event of a dirty shutdown, can when configured correctly provide and additional performance increase in terms of Database transactions – below is a picture of the 400i – this little “mouse thingy” is the battery.

HP DL 365 G1 Copyright HP 2008

Ok – thats great – but how do you know that 146 GB (or 300 GB) is enough for day to day operation with 5000 system users?:

Firstly lets look at the figures that I have seen within my existing Exchange 2003 environment:

Average Tracking Log Size Per Day (Protocol Logs): 150 MB view to keep for 15 Days = 150 * 15 = 2.25GB + 450 (%20) = 2.7GB

Transport Transaction Log: This by default utilises circular logging Microsoft does suggest that you can leave this on the O/S Mirror, however old habits are hard to break with me, therefore I always give them their OWN Mirror which is shared with the Protocol Logs (see above)

Transport Database: As per Microsoft the Transport Database does not store items Indefinitely, essentially you can derive a rough estimate of space required by taking an average message size (my environment is around 200KB) and then multiply it by the maximum queue size (which I tend to base on the worst case scenario that I have seen in my 2003 environment which was 10,000 items) – so:

200(KB) * 10,000(Items) = 2000000(KB) = 1.953 GB + 20% (Fluff factor) = (about) 2.3GB

You can also work this out against the Microsoft worst case which is like so:

200(KB) * 500,000(Items) = 100000000(KB) = 95 GB + 20% (Fluff factor) = (about) 114GB

Even in the worst case scenario where you have 500,000 items in your queue you will still maintain 32 GB free on the drive (if you are using 146 GB drives), this is clear of the 4GB threshold which invokes Back Pressure.

I use CCR do I need to consider the Transport Dumpster?

The “Transport Dumpster” is a special feature that is located on HT servers within sites that contain CCR or LCR Mailbox clusters.

Essentially the HT will need to have enough capacity factored into the disk subsystem to store mail long enough for all Storage groups located on CCR / LCR clusters in your site. This is used to recover messages that were in transport and destined for mailboxes on cluster node but then the node failed – full details of the Transport Dumpster are beyond the scope of this article – but if you would like to read more have a look here http://msexchangeteam.com/archive/2007/01/17/432237.aspx.

In order to calculate the size of the transport dumpster use the following metrics:

Largest possible message size (which in my environment is 18MB) then add 50% of its size = 27 MB (50% of 18 is 9 – therefore 18 + 9 = 27)

You would then set the MaxDumpsterSizePerStorageGroup to 27 (see the article above) – the total capacity required by the Transport Dumpster is then derived by the number of storage groups contained within the CCR environment – for example if you have 20 Storage groups that calculates as 20(SG) * 27(MB) = 540 MB.

Under best practice guidelines you should set your increase your largest message size by 1.5 before using this calculation – however as you can see, the disk requirements that we have established for our DL 365 G1 are more than happy even in the worst case scenario.

Putting it all together:

Ok at this stage we have a design of a DL 365 G1 with 6 GB of RAM with x 4 Opteron 2220 cores, additionally we have x 6 146 GB (or 300 GB) SAS drives plugged into a P400i RAID controller.

Our Disks are configured on x 3 RAID 1 sets (with two disks per RAID 1 mirror – obviously – this totals x 6 disks).

By rights the above configuration should easily perform that role of transporting messages around your environment. The article was based around 5000 users and average traffic of 68,000 messages per day, however the specification would realistically handle far more than that and arguably sustain quite a few years worth of growth.

So there you have it, if you are considering how to specify your Hub Transport Servers and support between 5000 – 9000 users and have a daily transport rate of around 68 – 80 thousand messages a day – have a think about the server configuration above.

Social

{ 0 comments… add one now }

Leave a Comment

Previous post:

Next post: