Quick Tip – Symantec Enterprise Vault Services Automatically Stop

You learn something new everyday! – There I was bumbling away today minding my own business – then … bang! – my TAC queue lit up with calls from customers complaining that they could not access their archives in Symantec Enterprise vault from Outlook and OWA.

After a quick attempt to access my own vault it became very apparent that the furore that had woken me from my mid-afternoon snooze whilst contemplating the arrival of my new Exchange database servers was not at all misplaced as I too could not access KVS – every time that I clicked on a shortcut from within my mailbox I would get the familiar, but annoying "The Vault is unavailable" message.

You learn something new everyday! – There I was bumbling away today minding my own business – then … bang! – my TAC queue lit up with calls from customers complaining that they could not access their archives in Symantec Enterprise vault from Outlook and OWA.

After a quick attempt to access my own vault it became very apparent that the furore that had woken me from my mid-afternoon snooze whilst contemplating the arrival of my new Exchange database servers was not at all misplaced as I too could not access KVS – every time that I clicked on a shortcut from within my mailbox I would get the familiar, but annoying "The Vault is unavailable" message.

I jumped onto the Vault server and had a quick look at all the services – all of which to my surprise had stopped, so considering that it was the middle of the afternoon (one of our peak periods) I thought that this was perhaps a glitch and it was best to just restart them and check to see if I could access my own personal archives and then do a proper analysis later on – see below:

services

We are running Enterprise Vault 2007 SP1 – in a fairly simplistic configuration (when we implemented it I didn't fancy bogging us down with multiple sites / servers with complex business rules) as a result generally I find that Enterprise Vault looks after itself (this was one of the reasons why we purchased it).

Upon restarting the services normal operation seemed to be resumed – or at least it was for about 4 minutes – where, again my TAC alerts queue flashed up again telling me that KVS was down again – bugger!

I logged back onto the server – only this time it was obvious that I was going to have to do some more in depth troubleshooting – starting with the Event Logs.

I opened up the Enterprise Vault Event log and had a look through – there were a number of errors – however the one which stood out immediately was the following entry logged by the Admin Service:

everror

This puzzled me as we monitor all the drives on the Enterprise Vault server using Microsoft Operations Manager and had not received any notifications that we were getting low on space – so I had a quick look at the V: (as per the Event Log) and found that it still had around 14GB of free space remaining. I did some further digging around and found that the Enterprise Vault Admin service amongst other things has the responsibility of monitoring resource utilisation (which includes Disk Space) if a resource enters into a critical threshold the Admin Service will close down all of the other Vault Services to preserve the integrity of the system.

It would seem that in terms of Disk Space (although I could not find a variable anywhere which confirmed this) if you get to the stage where you have only 1 – 2 % of free space left on a Disk which the Admin Service is monitoring this will trigger a close down – in my case the V: drive was 1.5 TB and had 14GB remaining – which is give or take about 1%.

This also explained why I did not get a warning message from Operations Manager as its Disk Space thresholds are set to completely different values (we are not using the Enterprise Vault Management Pack – after this we might though!).

So – how was I going to fix this?

I decided that this best course of action was to close down all of the Journal and Vault Archive partitions which were resident on the V: Drive, then add another LUN to the Server of around 500GB and create new partitions. To close down both Archive and Journal store partitions (which should be done at least every year if you have quite a lot of mailboxes) open the Enterprise Vault Management console and navigate to [ Enterprise Vault -> Directory on <your server> -> Vault Stores ] you will be presented with your defined Vault stores – click on either your defined Journal or Archive stores and the right hand window will change to display the partitions that you have configured:

kvsmanagement

This would mean that the older partitions would become read-only (no further items would be added to them) and there would still be enough space for them to be read (when a user opens an item from a closed partition it is expanded from its CAB file – therefore even on closed partitions there still needs to be some space left to cope with the expansion of CABS).

Now you have to remember that in order to close down the partitions and indeed create a new one on the new LUN the following Enterprise Vault Services need to be running:

  • Enterprise Vault Admin Service
  • Enterprise Vault Directory Service
  • Enterprise Vault Storage Service

Because you are in the position where the Enterprise Vault Admin Service keeps closing down the other services – it is slightly difficult to both close down a partition and create a new one before the services stop (as I found out) – so you will need to turn off monitoring on the drives.

In order to do this open up the Windows services MMC and locate the Enterprise Vault Admin Service – ensure that it is stopped (if you are going through this it should be stopped anyway) – open its properties and on the General Tab (the default) locate the "Start up" parameters area and add in the following switch /NOMONITOR – then start the service FROM THE PROPERTIES page – if you do not start the service from the Properties page the changes that you have made will be lost.

I then closed down the partitions which were on the drive which was considered to be a problem to KVS and created new partitions on the new LUN that I had created. However although the system was now backup and running there was still something that needed to be done.

Even though I had created a set of partitions on a new drive – I still had partitions (although closed) on the V: drive – the next time that the Enterprise Vault Admin Service was restarted (which would be that night during the backup) I would end up in the same position that brought me to this point – e.g. KVS would detect that one of its drives was low on space and close down the system.

So I went back to the properties of the Enterprise Vault Admin Service and in the Start up parameters box I added in the following Switch:

/DISK=<all of the disks that I want monitored> /SAVE – therefore my KVS server has the following drives: C, E,I,V,X and I did not want to ever monitor the V: drive again the switch was /DISKS=CEIX /SAVE.

Again start the service from the properties dialog – but on this occasion Enterprise vault will create the following registry key: [ HKLM\SOFTWARE\KVS\Enterprise Vault\AdminService\Disk ] where all of the disk which need monitoring will be stored – subsequent restarts of the service will not cause an issue.

I hope that the above helps someone along the way!

Sharing is caring!:

One thought to “Quick Tip – Symantec Enterprise Vault Services Automatically Stop”

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.