Wednesday, January 13, 2010

Fixing SharePoint Search that is down due to error 1000

Jeremy Williams, Sr. Director, Modern Workplace

Situation: A SharePoint server experienced some really nasty search errors (error #1000, 5214, 6398, 6482, 7888, 10036).  The most annoying of all these (and one I hadn’t seen before was the error 1000, shown below).  A quick internet search turns up a number of blog posts and threads regarding general SharePoint search troubleshooting (ranging in recommendations from the classic multi-threaded IIS issue w/available hotfix, to full-blown SharePoint reinstall).  At the end of it all, it seems that a hybrid approach helped me to solve the issue.

 

[Screenshot clipped to protect the innocent (servers)J].

Solution (In the form of those old pick-your-own adventure books):  The short of it is that SharePoint was pretty busted up by search; and I’m (unfortunately) still unsure what caused it to blow up.  As always, before you start really messing with Central Administration, be sure that you have a good SQL backup of your SharePoint content.  I cannot stress this enough, if you don’t have a backup, you’re asking for trouble.  Alright, now that the necessary disclaimer is out of the way, the solution involved the following steps:

1.       Ensure your install user account has full access to SQL (takes some permission issues out of the equation).

2.       Attempt to stop then restart Office SharePoint Server search (net stop osearch, net stop osearch in a command prompt).

a.       Did that fix the issue?  If yes, go out and buy a ‘That was easy!’ button

b.      If it didn’t fix it, Continue to 3 (Unless you’ve been here already, then go to 5).

3.       Restart the SharePoint Server box/boxes; if you’re really paranoid, you’d want to restart SQL too.

4.       Repeat step 2.

5.       Okay, things are bad, but there’s still hope.  Read On!

6.       Run the SharePoint Configuration wizard; leave the farm attached.  (Basically, we just want SharePoint to ‘repair’ any of its configurations that may have been corrupted).  When the configuration wizard finishes, run an iisrest for good measure.

a.       Did that fix the issue?  If yes, be thankful you aren’t like the other guys having to read on...

b.      If that didn’t fix it, don’t give up yet; there are still a few more things we can try, read on to 7.

7.       Wow, SharePoint is really broken here.  Before we bring out the big guns, let’s try running that configuration wizard again.  This time, you’re going to detach your server from the farm.  You will lose all of your configuration settings, so before you click the ‘Destroy’ button, be sure you’re comfortable rebuilding SharePoint’s configuration.  Detach SharePoint from the farm and restart your MOSS box.  Once the server comes back up, you’ll want to run the Configuration Wizard again (it will look just like a new SharePoint install).  Remember, all your old configuration databases are still on SQL’s file system, so you’ll either have to remove/rename the old files or rename your new configuration databases.

a.       Once you get a new SSP spun up, does search work?  If yes, see 8a.

b.      Didn’t work?  Let’s go for broke in step 8.

8.       You’re going to repair your SharePoint installation now.  Detach SharePoint from the farm again (instructions in step 7, but don’t reattach this time).  Go to Control PanelàAdd/Remove Programs (<=Server 2003) OR Programs and Features (>=Server 2008).  Find the SharePoint line item and select the ‘Repair’ option.  This is (effectively) going to reinstall your SharePoint bits and bytes.  Once the repair has finished, go ahead and reattach and reattach your farm’s major content databases.  Set up the new SSP and verify that SharePoint search is now functioning. 

a.       If yes, I recommend a congratulatory beverage.

b.      Still nothing?  Check out the ‘Things to look for…Antivirus’ below.

 

Things to look for…Antivirus

If these steps still don’t fix search, take a look at the antivirus running on your SharePoint box.  Certain AV products don’t ‘play nice’ with SharePoint, especially if the AV product isnt’ specifically designed for servers like SharePoint.  If you happen to discover some antivirus that isn’t SharePoint specific, see this Microsoft article for paths that you should exclude from the A/V scan: http://support.microsoft.com/kb/952167

Once you’ve done that, try a restart and go through the troubleshooting steps again.