Enterprise Vault v10.0.x Index marked as failed 'The failed reason on the index volume indicates the sequence number of the index is higher than that in EV storage. Will check the sequence numbers before continuing with the synchronization.'

Index gets marked as failed and synchronization of the index job gets stuck.  When you open synchronization of the index report you see following in the job report:

………………

15/05/2013 22:02:53 Checking index volume for failed state.

15/05/2013 22:02:53 The failed reason on the index volume indicates the sequence number of the index is higher than that in EV storage. Will check the sequence numbers before continuing with the synchronization.

15/05/2013 22:02:53 The synchronize has been asked to stop and is stopping.

15/05/2013 22:03:03 The synchronize has resumed its processing.

15/05/2013 22:03:03 The synchronize has been asked to stop and is stopping.

………………

The problem was to just flip the status of the index in SQL by following queries ran against Directory DB:

update indexvolume

set Failed = 0 where Failed = 1 and foldername = '1126AEF7E8493C346B70D777F8A052E09_1592'

update indexvolume

set FailedVolumeReason = 0 where FailedVolumeReason = 3 and foldername = '1126AEF7E8493C346B70D777F8A052E09_1592'

where foldername is taken from the synchronization job report .

To monitor indexing catching up use following query against the Jounaling Vault database:

Select * from journalarchive where indexcommited = 0

This should return zero rows or more - number goes up and down when you constantly run it - less than 100.  When we had a problem it was 200,000 ish items waiting to be indexed.

For one of two Vaults it also needed reboot of the server to catch up.