Enterprise Vault v10.0.x Index marked as failed 'The failed reason on the index volume indicates the sequence number of the index is higher than that in EV storage. Will check the sequence numbers before continuing with the synchronization.'

Index gets marked as failed and synchronization of the index job gets stuck.  When you open synchronization of the index report you see following in the job report:

………………

15/05/2013 22:02:53 Checking index volume for failed state.

15/05/2013 22:02:53 The failed reason on the index volume indicates the sequence number of the index is higher than that in EV storage. Will check the sequence numbers before continuing with the synchronization.

15/05/2013 22:02:53 The synchronize has been asked to stop and is stopping.

15/05/2013 22:03:03 The synchronize has resumed its processing.

15/05/2013 22:03:03 The synchronize has been asked to stop and is stopping.

………………

The problem was to just flip the status of the index in SQL by following queries ran against Directory DB:

update indexvolume

set Failed = 0 where Failed = 1 and foldername = '1126AEF7E8493C346B70D777F8A052E09_1592'

update indexvolume

set FailedVolumeReason = 0 where FailedVolumeReason = 3 and foldername = '1126AEF7E8493C346B70D777F8A052E09_1592'

where foldername is taken from the synchronization job report .

To monitor indexing catching up use following query against the Jounaling Vault database:

Select * from journalarchive where indexcommited = 0

This should return zero rows or more - number goes up and down when you constantly run it - less than 100.  When we had a problem it was 200,000 ish items waiting to be indexed.

For one of two Vaults it also needed reboot of the server to catch up.

Enterprise Vault Discovery Accelerator Client is stuck on "Loading the item set"

 Symptom: DA user complaints that items are not loading for review with "Loading the item set" message is stuck on the screen

Following is the fix procedure - performed by knowledgeable sysadmin or dba admin on DA database::

1. Close the affected user's accelerator client
2. Find the PrincipalID of the affected user. Change 'smith' to that user's last name.
SELECT PrincipalID, PrincipalName FROM tblPrincipal WHERE PrincipalName LIKE '%smith%'
3. Find PrincipalIDs of users who have not used the DA Client:
SELECT PrincipalID, PrincipalName FROM tblPrincipal
WHERE ADSync = 1 AND SyncError = '' AND PrincipalID NOT IN (
SELECT PrincipalID FROM tblUserPreferences)
4. Update the user preferences table to change the PrincipalID of the affected user (A) to a user that does not have a DA client profile (B - one of the users found in step 3)
UPDATE tblUserPreferences SET PrincipalID = B where PrincipalID = A
5. Have the affected user log into the DA Client and then log out again. Logging out will create a new Profile in the user preferences table.
6. Have the affected user log back in and test the DA Client.
7. If this works remove the entry associated with user (B)
DELETE FROM tblUSerPreferences WHERE PrincipalID = B

If it doesn't fix the issue, Here is the query to revert the change:
DELETE FROM tblUSerPreferences WHERE PrincipalID = A
GO
UPDATE tblUserPreferences SET PrincipalID = A where PrincipalID = B

Exchange 2010 Hub/Cas Servers require space on C: drive or mail submission fails

Exchange 2010 Hub/Cas server needs extensive amount of free disk space on C: drive.  I had 1.8G free on C: drive, but it was not enough: mail submission from mailbox server was failing with event id 1009, none of emails sent from the mailbox server get processed.