Repeatedly Corrupted Search Catalogs on multiple DAG members

Status
Not open for further replies.
J

Jason Huitt

For several days now we've been seeing corrupted search catalogs on a number of mailbox databases on our Exchange 2010 8 member DAG. Each time this happens I'm seeing the following events in the Application Logs of the affected servers:

MSExchange Search Indexer : Event ID 118

Exchange Search Indexer has temporarily disabled indexing of the mailbox database EX24R12 (GUID = 6644034b-2058-478e-bfde-442423cce41b) due to an error (Microsoft.Mapi.MapiExceptionMdbOffline: MapiExceptionMdbOffline: Unable to read events. (hr=0x80004005, ec=1142)

- and -

ExchangeStoreDB : Event ID 123

At '7/9/2010 9:54:22 AM' the Microsoft Exchange Information Store Database 'EX12S12' copy on this server experienced a corrupted search catalog. Consult the event log on the server for other " ExchangeStoreDb" and " MSExchange Search Indexer" events for more specific information about the failure. Reseeding the catalog is recommended via the 'Update-MailboxDatabaseCopy' task.

Each time this happens I'm able to resolve the issue by running " Update-MailboxDatabaseCopy MDB\Server -CatalogOnly" , however this is continuing to happen and I can find to root cause. Today it happened nearly immediately after we performed a database switchover of a number of databases, however not all databases (and not all mailbox servers) involved in the switchover experienced the corruption. Additionally, the databases themselves all report as health, and I'm not seeing any disk or I/O errors in the System Logs of the affected mailbox servers. This doesn't appear to be any kind of physical disk corruption.

This has happened with enough frequency (and has occurred without having been preceded by a database switchover event) that while all databases are reporting their ContentIndexState as healthy at present (after running the Update-MailboxDatabaseCopy -CatalogOnly cmdlet), I am reasonably sure that this condition will occur once again. What can I do to determine the root cause of this?
 
J

jader3rd

Do you run a chkdsk against the hard drive before seeding again?
 
J

Jason Huitt

I hadn't done that previously, however I just spot checked a couple of the mailbox servers and CHKDSK reports zero errors on our data volumes.
 
J

jader3rd

And do these remain failed permenatly?

Event ID 118 is normal for a database dismount. Does the copy become corrupted everytime you dismount the database? You don't need to move it, just dismount, wait a second, check the status, and then mount again.
 
J

Jason Huitt

No, it doesn't appear to have anything to do with mounting or dismounting the database. I just dismounted a database, waited a few seconds and then remounted it, with no negative effects on the database ContentIndexState. Note that this has also happened to a number of databases when there is no administrative function going on, say at like 9:30 in the morning when we are just watching the system, not making changes.
 
J

jader3rd

Is this a geo located Dag, and is this only happening on the copies on the other side of the planet?
 
J

Jason Huitt

Nope. The four of the DAG members are located in one datacenter on one part of campus, the other four are located in a DR datacenter in another part of campus. They are connected via 1GB fiber.

And, this is happening on servers in both locations.
 
B

Blackuke

What"s your memory usage like when this happens, can you use perfmon to check the percentage used, this sounds very much like a batch backlog. Are there any event in the event log like search pausing
 
J

Jason Huitt

All of our Ex2010 MB servers have 48GB of RAM - current physical utilization is at 42.4GB. I can't speak to whether that's abnormal for our deployment or not, as these are all new boxes for us (we're migrating for Ex2007).

I get the following error from MSExchangeSearch Indexer (Event ID 118) prior to the Content Index going corrupt:

Exchange Search Indexer has temporarily disabled indexing of the mailbox database EX13S12 (GUID = 6285e2ec-5e60-4d1e-acda-0c6dbde8eda1) due to an error (Microsoft.Mapi.MapiExceptionMdbOffline: MapiExceptionMdbOffline: Unable to read events. (hr=0x80004005, ec=1142)

We were fine over the weekend, no failed indexes. Today just to see where we were at, I moved all active DB copies on one server to another, and shortly thereafter I started seeing more of the above events, and eventually nine of the fourteen DBs " homed" on that server ended up with a Failed ContentIndexState on some of the DAG members.
 
J

Jason Huitt

Gavin -

Thanks for getting back to me. I'm happy to try this, and I will the next time we have a corrupt index event. However my concern isn't so much how to resolve the problem (I've gotten quite good at that already). My concern is that this continues to happen, and it appears that it may be related to database switchover events. We plan to apply patches to our mailbox servers tomorrow evening, at which point we'll incur another several switchover events, and I'll have a better gauge on whether switchovers specifically are causing this issue.

In the meantime, can you give me a few reasons why we might be experiencing a corrupt content index when there are no underlying physical disk issues?

Thanks.

- Jason
 
B

Blackuke

In all honesty I would suggest you open a Microsoft support case before tomorrow so that an engineer can work with you to try and understand whats going on, they will be able to help better than this thread using traces and other tools IMHO.
 
G

Gavin-Zhang

Hi Jason,
Sure, the issue maybe caused by several reason, and it is difficult to confirm it just according to the information you refered.
It is a better way to do some update.
I have run into a issue looks same as yours, the cuse is drive controller issue. Yet it would need retrieve many kinds of log to confirm it, you could open a ticket from MS other team.
Regards!
Gavin
 
S

Stevie_Admin

I am experiencing exactly the same issue....it seems to occur after switchover in a dag but this in turn is being triggered by something in the night also.....
 
B

busbar

I am having the issue with lots of my customers and no RAID or disk issues, we see it also a lot on the forum with no apparent reasonRegards, Mahmoud Magdy Watch Arabic Level 300 Videos about Exchange 2010 here: http://vimeo.com/user3271816 Read pretty advanced Exchange stuff I post here: http://www.enowconsulting.com/ese/blog.asp, follow my blog: http://autodiscover.wordpress.com , corp blog: http://ingazat.wordpress.com and if you Liked my post please mark it as helpful and accept it as an answer
 
J

Jason Huitt

Well we patched our entire Exchange 2010 infrastructure last night (including upgrading to UR4) and didn't incur a single corrupt content index. So I can no officially say we have no clue what is causing this to randomly occur, and I can confirm that as recently as three days ago we were still having this problem.

We're hoping that we aren't waiting on a Service Pack to get this problem fixed, though that seems to be looking increasingly likely...
 
A

A_RASHID_IBM

Hi,

We're experiencing this same problem, however no solution was reported in the thread. Did the events appear again?

Thanks,

Atif
 
J

Jason Huitt

Yes I had this happen randomly at 4AM on Saturday morning without any identifiable cause. If anyone has additional things to check out with respect to content index replication/corruption, please let us know. Thanks.
 
P

Pantherfan

add me to the list of people that randomly see this error pop up, with " healthy and mounted" databases after moving an active db.
 
Status
Not open for further replies.
Thread starter Similar threads Forum Replies Date
R Shortcut for attaching the same file repeatedly Using Outlook 6
Fozzie Bear Repeatedly corrupting PST file Using Outlook 1
D Outlook 2007 Repeatedly Requesting Password via Outlook Connector Using Outlook 1
V pst repeatedly needing repair Using Outlook 2
B Sign-in dialog pops up repeatedly for shared calendars Using Outlook 1
C exchange repeatedly requests login info Using Outlook 4
A How to stop .pst getting email repeatedly ? ( outlook 2010 ) Using Outlook 3
S Outlook Calender Items Repeatedly showing in Deleted Items Using Outlook 2
R Repeatedly prompted to enter your credentials when trying to connect to an Exchange mailbox by using Using Outlook 2
P Had Outlook Connector working, it asked for upgrade repeatedly. Uninstalled and tried to download-reinstall and it doesn't read my live.com email or Using Outlook 5
S email stuck in outbox and sends repeatedly Using Outlook 9
B Outlook 2007 repeatedly asks for passord in one profile only Using Outlook 2
M Outlook 2007: Unable to connect to Exchange Server, repeatedly prompted for credentials Using Outlook 1
M Outlook 2007: Unable to connect to Exhange Server, repeatedly prompted for credentials Using Outlook 4
C Microsoft Office hangs up repeatedly during the day and is otherwise slow since installing Windows 7. Using Outlook 1
W Outlook 2010 repeatedly crashes and restarts Using Outlook 4
I Email with largish attachment sent repeatedly Using Outlook 0
T In Office 2010, Outlook repeatedly freezes (not responding) when transitioning between Calendar and Using Outlook 1
1 Outlook recipients receiving same email repeatedly Using Outlook 1
D Outlook Profiles becoming corrupted Using Outlook 0
G .ost file corrupted / solved the issue with some posts already said, but .ost file is corrupted agai Using Outlook 2
S How to restore corrupted PST file over 2GB in Outlook 2007 Using Outlook 2
G Outlook calendar entry corrupted. Constant pop up when you open outlook. Unable to delete or remove. Using Outlook 2
E Outlook contact group lists corrupted Using Outlook 8
L Outlook 2010 - Keep getting "Corrupted File" Msgs Using Outlook 4
N Outlook 2007 Categories Problem (Most Recentley Used List Corrupted) Using Outlook 3
smokiibear Mobile Contacts Corrupted Using Outlook 3
C crash during account setup - can't delete corrupted account/data file Using Outlook 13
T PST file keeps getting corrupted Using Outlook 3
Y I Have A Corrupted In Box In Outlook 2007 Using Outlook 1
G PST file corrupted during move Using Outlook 1
M my mails are corrupted... Using Outlook 1
D Corrupted Outlook 2010 Address book Using Outlook 4
D Mail profile keeps getting corrupted Using Outlook 1
L Importing Emails from corrupted Profile Using Outlook 1
C Outlook pst file corrupted Using Outlook 3
F Correct email dates corrupted during importing from Outlook Express Using Outlook 1
D File attached to an e-mail message stored in Outlooks Express is corrupted after some time Using Outlook 3
3 Corrupted Forms Folder - Error message Using Outlook 1
A corrupted Outlook 2007 UPdated today 09/19/10 Using Outlook 2
M Error: This mailbox exceeded the maximum number of corrupted items that were specified for this move Exchange Server Administration 9
O When I try to open Outlook, I get a message saying the program is corrupted and it directs me to use scanpst.exe. However, I can't run the scan, becau Using Outlook 1
D Corrupted Outlook 2007 data files Using Outlook 2
T Outlook 2007 corrupted PST file Using Outlook 1
T what to do to fix outlook 2010 once corrupted by dbx files? Using Outlook 9
Z PST file corrupted Outlook VBA and Custom Forms 1
L New Location for the Search Bar Using Outlook 7
T How can you include Junk Email in Search Results like you can include Deleted Items? Using Outlook 3
J Outlook 2016 After a search in all mailboxes, where is each message that was found? Using Outlook 6
King Mustard Sort search groups by amount of items? Using Outlook 1
Similar threads


















































Top