Odd Problem when trying to add a second DAG member Server (A Server-Side database availability group administrative operation failed)…

Today I came across a very odd problem whilst trying to add a second node to an existing DAG environment.

In essence when either using a custom script that I have written – or indeed via the Exchange Management Console I get the following error when trying to add a second node to the DAG:

Summary: 1 item(s). 0 succeeded, 1 failed.
Elapsed time: 00:00:04 

LAB-EX21K-02
Failed 

Error:
A server-side database availability group administrative operation failed. Error: The operation failed with message: An error occurred while attempting a cluster operation.
Error: Cluster API '"AddClusterNode() (MaxPercentage=12) failed with 0x80070005. Error: Access is denied"' failed. [Server: LAB- EX21K-01.justice.lab.com]
An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"AddClusterNode() (MaxPercentage=12) failed with 0x80070005. 

Error: Access is denied"' failed. 

Access is denied 

Warning:
The operation wasn't successful because an error was encountered. You may find more details in log file "C: \ExchangeSetupLogs\DagTasks\dagtask_2010-04-05_17-27-34.291_add-databaseavailabiltygroupserver.log". 

Exchange Management Shell command attempted:
Add-DatabaseAvailabilityGroupServer -Identity 'LABDAG' -MailboxServer 'LAB-EX21K-02' 

Elapsed Time: 00:00:04

Upon further investigation via Google, I found that the most common cause of this error was that the “Exchange Trusted Subsystem” group was not a member of the local administrators on the node which failed to join the DAG.
Job done” you (and I) might be forgiven for thinking, but upon opening up [ Server Manager –> Configuration –> Local Users and Groups –> Groups –> Administrators ], I was presented with the following:

AdminsIssueDAG

In essence, all of the Security principles that one might expect to see here (Domain Admins, Exchange Trusted Subsystem, & Organization Management) were merely represented by their respective SIDS.
Initially I thought that I would remove them, and then re-add them – which at the first look seemed to be have been successful – but alas; upon re-entering the Properties of the Administrators group revealed the same issue that I had seen above.

At this stage, I thought that un-installing Exchange 2010 from the DAG node which had the problems and then removing it from the domain, and then re-joining it would solve the this problem (I had checked the Event Logs, confirmed the DNS settings, reviewed the logs on the Domain Controllers all to no avail) so I proceeded with the uninstall and re-join – which, I was at that point glad to say – worked! (in truth if time was not against me I would have tried more troubleshooting; but my wife and I were going to dinner and I at the very least wanted to have the Security issue fixed before we left for the night – and I was pretty sure that she was going to cut my dangly bits off if I opened up the Exchange Setup Logs).

I tried to re-install Exchange on the failed node; however – at the point where it wanted to install the Hub Transport role (this server would be a HT, MBX and CAS) I was greeted with the following error:

Summary: 11 item(s). 7 succeeded, 1 failed.
Elapsed time: 00:03:39 

Preparing Setup
Completed 

Elapsed Time: 00:00:01 

Stopping Services
Completed 

Elapsed Time: 00:00:01 

Copy Exchange Files
Completed 

Elapsed Time: 00:01:10 

Language Files
Completed 

Elapsed Time: 00:01:37 

Restoring services
Completed 

Elapsed Time: 00:00:00 

Languages
Completed 

Elapsed Time: 00:00:16 

Management Tools
Completed 

Elapsed Time: 00:00:16 

Hub Transport Role
Failed 

Error:
The following error was generated when "$error.Clear(); if (($server -eq $null) -and ($RoleIsDatacenter -ne $true) ) { 

Update-RmsSharedIdentity -ServerName $RoleNetBIOSName }" was run: "RMS Shared Identity user FederatedEmail.4c1f4d8b-8179-4148-93bf-00a95fa1e042 not found.". RMS Shared Identity user FederatedEmail.4c1f4d8b-8179-4148-93bf-00a95fa1e042 not found. 

Elapsed Time: 00:00:16 

Client Access Role
Cancelled 

Mailbox Role
Cancelled 

Finalizing Setup
Cancelled

Bugger! – again I decided to have a look for information on the new error above via my mate Google and found a number of articles which were closely related to the issue which I was experiencing above – for example here, here and here but none hit the nail exactly on the head – e.g. the articles that I had found seemed to deal with either the Mailbox Role not installing or the UC role – whereas in my situation the setup process was failing on the Hub Transport Installation.

From the information that I had found – two articles from 3 suggested that this was to do with an arbitration mailbox having been removed (not sure how that could of happened by uninstalling Exchange from one node and then removing the node from the domain and the re-joining it) – and that the solution was to run the Exchange 2010 setup again using the following command:

Setup.com /prepareAD

So, I did, and low and behold it worked! – after executing the above and then attempting to run the setup on my DAG node again – it all went very smoothly.
I then went back to my original objective of trying to add my sickly node to a DAG configuration using my script (which is to appear in a article very soon) – and finally it worked.

I realise that the events that I have experienced above will not be common place (in fact I am still at a loss to figure out what went wrong in the first instance – and I guess that I was lucky that I was in the position to be able to remove the second DAG node completely from the domain (although I suspect that I might have been able to move mailboxes should there have been any as to all intents and purposes it was still functioning correctly in terms of Exchange)).

However I believe that it was worth putting the experience down so; if anyone else should find themselves in this situation they at the very least have a last ditch resort to get them back up and running

Sharing is caring!:

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.