Exchange installs and upgrades the good, the bad and the ugly…

Copyright (c) - United Artists 1966Over the years, I have installed Exchange Server probably thousands of times. In big infrastructures, little infrastructures and everything in between. Luckily the vast majority of those installs have gone without a hitch – however, there have been some (and those tend to be the ones that I remember the most) that didn’t go so well ~ I affectionately call these the “Smashed” installations).

What I would like to do in this post is take you through some of my own tips when both installing, upgrading and where necessary – recovering from a smashed Exchange install which covers versions of Exchange from 2007 through to 2013. Please note that this is not a Disaster Recovery article – the tips here are about getting new Exchange servers up and running, either in Lab or Production environments and avoiding problems before you encounter them.

First things first – take your time and plan accordingly

Yep, this seems very obvious – I know the feeling, you have installed Exchange countless times, you already have an environment up and running and you just want to get the ISO down and dive into the installation wizard. Long and short of this is don’t do it ~ not just yet.

Planning can be equated to “staking out” your existing environment – e.g. gathering information and researching potential “gotchas” before they happen.

It doesn’t matter if you are working in a vanilla environment, adding new servers to one that already exists or working within a co-existence environment for an upgrade, you should still ensure that you have a plan.

Now there are many levels of planning – but the most pertinent of which I have found is the spend time reviewing almost every aspect that you can think of within the environment that you plan to build, upgrade, or char before you begin the install.

Please do:

  • Try any new configuration / setup or upgrade in a test lab where possible.
  • Make backups of you AD infrastructure and Exchange Servers / Databases before you make any changes.
  • Read all the release notes for known issues, this is especially relevant in the case of Service Packs.
  • Check product compatibility with any 3rd party software that might be interfacing or be integrated with your Exchange environment, this mainly effects products such as 3rd party Archival Software – but can extend to CRM / ERP systems, room bookings software etc.

Also take the time to review the following:

  • That all Domain Controllers are online within your environment – having a domain controller offline that you are not aware of could cause Exchange Setup to fail. This does not necessarily means just key FSMO role holders (like the Schema master for example), but Exchange setup could try to talk to a “non FSMO role holder” that is a GC which is not working, this could “kill” the install.
  • Ensure that Directory Replication is working correctly – DCDIAG is your friend here, if you have replication problems between AD replication partners – then you will probably have issues during Exchange setup, especially during the Schema preparation stages.

However – if you would like to get some more detail which gives you an excellent and very detailed overview of the state of your Active Directory infrastructure – have a look at the following script on the TechNet scripts library by Jon Knapp: http://gallery.technet.microsoft.com/scriptcenter/28812d92-30be-4f37-80cb-ab21b1eeaace

The code is two scripts in one – if you do decide to use it – read the information on the link carefully. Once it is up and running it produces a very good overview of your Directory environment.

  • Review all Domain Controller Event Logs for unusual errors or warnings.
  • Ensure that DNS is working correctly and that you can resolve both forwarders and reverse – for all DC’s and the Servers on which you intend to install Exchange.
  • Like above – ensure that all your existing Exchange servers are online and functioning correctly if you are adding a new server to an existing installation – or performing an upgrade.
  • Ensured that all relevant operating pre-requisites and patches are installed BEFORE you install Exchange. Whilst it is true that you can asked Exchange setup to install O/S pre-requisites during the install, this can lead to the installer stopping with the Message “A reboot is pending from a previous installation (for example if you let Setup install the RSAT tools). It is always a good idea to get all reboots and installs out of the way before you install Exchange.
  • Linked to the above tip – if you have installed any recent O/S updates via Windows Update or a management tool such as System Centre Configuration Manager – ensure that any outstanding reboots have completed.
  • Take the time to ensure that relevant system mailboxes (these are called arbitration mailboxes) are working as you would expect. In the past I have had Exchange setups (and upgrades) fail, because certain arbitration mailboxes have been orphaned from their mailboxes store (this is often caused by someone forcibly deleting a mailbox store using ADSI edit from AD)

You can check this by running the following Exchange PowerShell cmdlet before you run setup:

Get-Mailbox –arbitration | Select name,Database

If there is a problem you should see an error message similar to the following:

Name                                                        Database
----                                                        --------
SystemMailbox{1f05a927-ac84-48a8-9e9b-03a88581d504}         prod-ex2010-01-General-2
FederatedEmail.4c1f4d8b-8179-4148-93bf-00a95fa1e042         pri_Commercial_01
SystemMailbox{e0dc1c29-89c3-4034-b678-e6c29d823ed9}         pri_Technology_01
Migration.8f3e7716-2011-43e4-96b1-aba62d229136
WARNING: The object prepAD.local/Users/Migration.8f3e7716-2011-43e4-96b1-aba62d229136 has been corrupted, and it's in
an inconsistent state. The following validation errors happened:
WARNING: Database is mandatory on UserMailbox.
WARNING: Database is mandatory on UserMailbox.

04-01-2013 17-53-54

You often can repair errors like the above by running Exchange Setup again with the setup /prepareAD command line – full details of the process are detailed here: http://social.technet.microsoft.com/wiki/contents/articles/5317.recreate-and-enable-missing-arbitration-user-accounts-and-mailboxes-in-exchange-server-2010.aspx

  • If you are using any “Agent Based” monitoring tools such as System Centre Operations manager which maintain locks on critical files on a monitored host, or any monitoring tool that will take corrective action when a key service goes offline (MSExchangeIS for example) – disable them before you begin setup.
  • If you are running any File Level anti-virus products – you should ensure that these are not monitoring any Exchange files, critical paths or critical services (as per http://technet.microsoft.com/en-us/library/bb332342(v=exchg.141).aspx (for Exchange 2010) or http://technet.microsoft.com/en-us/library/bb332342.aspx (for Exchange 2013) – these can interfere with the setup process.

Preserve your log trail for troubleshooting later on

Before you install Exchange, review and preserve your Event and Setup logs – in particular:

  • Before you begin an install – Export the event logs from the servers that you are working on and save them to a safe location.
  • Clear the event logs – this might seem like a crazy idea – but if you have exported them (as in the previous tip) – then you have preserved the historic data. Clearing the logs just before you begin setup means that if something should go wrong it is easier to filter out the relevant events.
  • When Exchange setup runs a folder called ExchangeSetupLogs is created in the root of the system boot drive (typically C:\). This folder is like gold dust if something goes wrong – however there are some things that you should know about the contents of this folder:
    • Each time Exchange Setup runs a large amount of data is input into the files in this folder. These files are a mix of text log files, PowerShell files, and Dr Watson Reports. As a tip I recommend that you take a copy of this folder after each setup run. Each time setup is executed a number of the files are appended to – which can make them harder to search.
    • If you take a copy after each setup run you effectively create a point in time set of logs for each install attempt which you can look at in isolation and compare if you need to.
    • If you need to effectively analyse the logs created in the ExchangeSetupLogs it is worth taking a look at the Get-SetupLog.ps1 script which is located in the $exscripts folder in Exchange 2007 and 2010 versions of Exchange.

The full overview of how you can use the script is provided here in the following post on the MSExchangeTeam blog – but the loose gist is as follows:

Open an Exchange Management Shell and type in:

cd $exscripts

.\Get-SetupLog.ps1 <path to Exchange Setup Log file>

The above above command will produce verbose output from the script.

If you use the following command

.\Get-SetupLog.ps1 <path to Exchange Setup Log file> –error $true

This will scale down the output from the script to relevant warnings and errors.

04-01-201319-09-15

As you can see from my output, the problem with the Exchange setup that I was using in the example above was down to missing arbitration accounts – see below:

0.  ErrorRecord: Microsoft.Exchange.Data.DataValidationException: Database is mandatory on UserMailbox.
[ERROR] The following error was generated when "$error.Clear();
[ERROR] The following error was generated when "$error.Clear();
[ERROR] Database is mandatory on UserMailbox.
1.  ErrorRecord: Database is mandatory on UserMailbox. Property Name: Database
1.  ErrorRecord: Microsoft.Exchange.Data.DataValidationException: Database is mandatory on UserMailbox. Property
Name: Database
[ERROR] The following error was generated when "$error.Clear();
[ERROR] The following error was generated when "$error.Clear();
[ERROR] Database is mandatory on UserMailbox. Property Name: Database

The article on the MSExchange team’s site mentions some further formatting scripts which can dump the output to HTML – however clicking on the link redirects to a 404 page. However, I have found that if you need to keep the data you can use the PowerShell Start-Transcript cmdlet – like so:

Start-Transcript –path <path to transcript file>

.\Get-SetupLog.ps1 <path to Exchange Setup Log file> –error $true

Stop-Transcript

You can then use a text editor such as Notepad++ to review the formatted output file.

Just for information – this script does not appear to be included with Exchange 2013 – however from some of the testing that I have done, it works with the Exchange 2013 setup logs as well.

Other tips that you might find useful

  • If you are having trouble restarting setup after a failed installation on a specific role, and you are getting an error message similar to the following:

Setup previously failed performing action: x

Setup might then terminate.

In order to fix this problem you will need to open up the Registry Editor and navigate to the following location:

HKLM\Software\Microsoft\ExchangeServer\V(14, 15)

Under this key you will see entries for each Exchange Role that has been installed – click on each key and look for the “Action” and “Watermark” entries – delete them and then begin setup again.

  • If you are adding servers to an existing Exchange 2010 environment, co-existing with Exchange 2007 take the time to run the Exchange BPA ~ this should identify any major issues before you make any changes to your environment.

There are – of course many other tips available on the web to help you with a safe install or upgrade for you environment which I urge you to seek out – but I hope that you find my own personal brain dump useful.

Sharing is caring!:

2 thoughts to “Exchange installs and upgrades the good, the bad and the ugly…”

  1. Another thing to be wary of is if you enforce Powershell Execution Policy via GPO on your Exchange servers, if so you may run into serious issues, as I’ve documented in the past here https://www.angryadmin.co.uk/?p=327

    Essentially the Exchange 2010 upgrade process stops the WMI service and the WMI service is required to handle Powershell MachinePolicy & UserPolicy via GPO, so halfway through the upgrade the setup program can no longer execute the Powershell scripts it needs to complete the install and you get a nice error along the lines of:

    The following error was generated when “$error.Clear();
    & $RoleBinPath\ServiceControl.ps1 EnableServices Critical
    ” was run: “AuthorizationManager check failed.”.

    Worst case, it’ll hose your install completely.

    1. Excellent – thanks Adam. This is an issue that I have read about – but not personally come across. Thank you for adding it.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.