Application Server Administration job failed for service instance Microsoft.Office.Server.Search.Administration.SearchAdminSharedWebServiceInstance (6b98c51f-116c-49f0-9aa6-4207555ed2f8).

Reason: The handle is invalid.

Techinal Support Details:

System.Security.Cryptography.CryptographicException: The handle is invalid.
at System.Security.Cryptography.Utils.CreateProvHandle(CspParameters parameters, Boolean
andomKeyContainer)
at System.Security.Cryptography.Utils.GetKeyPairHelper(CspAlgorithmType keyType, CspParameters parameters, Boolean randomKeyContainer, Int32 dwKeySize, SafeProvHandle& safeProvHandle, SafeKeyHandle& safeKeyHandle)
at System.Security.Cryptography.RSACryptoServiceProvider.GetKeyPair()
at System.Security.Cryptography.RSACryptoServiceProvider..ctor(Int32 dwKeySize, CspParameters parameters, Boolean useDefaultKeySize)
at Microsoft.SharePoint.Utilities.CertificateManager.CreateSelfSignedSslCertificate(CspParameters parameters, X500DistinguishedName name, DateTime expiresAfter)
at Microsoft.SharePoint.Administration.SPProvisioningAssistant.ProvisionIisWebSite(String serverComment, String[] serverBindings, String[] secureBindings, AuthenticationMethods authenticationMethods, String[] authenticationProviders, String path, AccessFlags accessFlags, String applicationName, String applicationPoolId, String[] scriptMaps, String sslCertificateSubjectName)
at Microsoft.SharePoint.Administration.SPMetabaseManager.ProvisionIisWebSite(String serverComment, String[] serverBindings, String[] secureBindings, Int32 authenticationMethods, String[] authenticationProviders, String path, Int32 accessFlags, String applicationName, String applicationPoolId, String[] scriptMaps, String sslCertificateSubjectName)
at Microsoft.Office.Server.Administration.SharedWebServiceInstance.Synchronize()
at Microsoft.Office.Server.Administration.ApplicationServerJob.ProvisionLocalSharedServiceInstances(Boolean isAdministrationServiceJob)
After some troubleshooting, googling and talking to the windows team I found that the folder:
C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys most likely had the wrong permissions. One of the files located in the folder didn’t seemed to like the move and when trying to look at the permissions I got this:

So, what I did was that I gave the ownership of this folder to the local administrators group which is done in the following way:

Click Ok on the warning that is shown above.

Click on Advanced and then on the Owner tab.

Select administrators, and click Ok until all windows are closed

Open up the security dialogue again and add the local administrators group and click ok. Note that if you have moved the server from one AD to another you will most likely see the old AD account in the security list which should be deleted.

You should add the local Administrator group to the MachineKeys folder above an make sure that it inherits the permissions.

This should solve the problem for you.

The reason for that is that the rebuild of the index is quite aggressive in it’s nature and does not have any logic built in to it. It simply rebuilds the whole index daily.

If you look at the information from Microsoft at technet (http://technet.microsoft.com/en-us/library/cc678870.aspx) about the job it says:

If you have installed Office SharePoint Server 2007 with SP2:

• The job updates the query optimization statistics by sampling key tables every time that it runs, instead of performing a full scan.
• If you are running SQL Server 2005 or SQL Server 2008, the job rebuilds all indexes in the content databases every time that it runs.
• If you are running an Enterprise edition of SQL Server 2005 or SQL Server 2008, the job rebuilds most indexes online.
• If you are running a Standard edition of SQL Server 2005 or SQL Server 2008, the job rebuilds the indexes offline.
• If you are running SQL Server 2000, the job does not rebuild any indexes.

It actually also specifies that the job should only run weekly if you are running MOSS and daily if your using WSS. I assume that this is just a mistake in the documentation because in all environments I looked in it’s set to daily no matter if you are using Moss or WSS with SP2.

So what happens if you have a large environment where your have backups, or maybe a crawl running at the same time. Or maybe your SQL DBA’s have done their job and have their own reindex and update statistics job put directly in SQL? Well, you could run in to performance problems and extreme slow environments together with backups taking longer time then they should, error messages and maybe even corrupt databases. The later I have not seen but potentially it’s possible.

I’m not saying that bad things will happen to your environment with this job, but more of be aware and have a look at how this is configured in your environment. Maybe you should reschedule it if it’s in conflict with any other SQL job. And if you are not the SQL DBA talk to the DBAs and make sure that they don’t have any similar already in place. If they have, is it better with built in logic where they look at the fragmentation level? Then maybe you should disable the SharePoint Timer Job and only run the job put on the SQL server.

If you want to find out what settings you have you could first look in Central Administration. You have one Database Statistics job for each Web Application and you can see that under Operations and Timer Job Status. In the Timer Job Definition, found at the same place in Central administration, you can see if the job is run weekly or daily.
The best  way is however to use STSADM or SPDiag to get the information on when it’s scheduled. To reschedule jobs you need to use STSADM.

This is how you get the settings you have using STSADM:

stsadm -o getproperty -pn job-database-statistics -url http://YourSPSite

And to set a new value you do like this:

stsadm -o setproperty -pn job-database-statistics -pv “Weekly between Fri 22:00:00 and Sun 06:00:00″ -url http://YourSPSite

Here you have the STSADM reference for this particular task: http://technet.microsoft.com/en-us/library/cc424963.aspx

After some troubleshooting and discussions with Microsoft I finally got confirmed that this is a known issue, not very frequent but still. This is caused when the worker process updates or deletes a cached file and a request comes simultaneously to the WFE server for a MOSS page that contains the cached item.

This is corrected in August Cumulative update even if it’s not specifically mentioned in the kb. http://support.microsoft.com/kb/973400/
Below you see how it was showned in my case.

Event Type: Error
Event Source: Office SharePoint Server
Event Category: Publishing Cache
Event ID: 5538
User: N/A
Description:
An error occured in the blob cache. The exception message was ‘The process cannot
access the file
‘D:\BlobCachefolder’ because it is being used by another process.’

They ran the binaries without any problem but when running the SharePoint Products and Technologies Configuration Wizard the installation quickly failed with the error “Failed to connect to the database server”.

We did all testing and troubleshooting with local Firewalls, loop back checks, accounts etc, and when accessing the internal database with SQL Management Studio it worked fine. In the log files the following was found that did not gave us that much.

The timeout period elapsed prior to completion of the operation or the server is not responding.

After some testing we started to notice that the server was behaving strange when doing other things. It turned out that Firewall ports between the server where we should install WSS 3.0 and AD had been blocked causing problems to authenticate the user that ran the wizard.

Synchronization for Shared Services Provider ‘[your shared service provider]‘ has failed. The operation will be retried.
Reason: Collection was modified; enumeration operation may not execute.
 

And to solve it I did the following:

stsadm -o sync -listolddatabases 1

This gives us a list of databases that has not been syncronized for 1 day. You can change the values if you want to check further back in time.

stsadm -o sync -deleteolddatabases 1

This sounds scary right but it actually just deletes the record to the databases.

Restart the Windows SharePoint Timer Service

stsadm -o execadmsvcjobs

Hope it might help someone to quicker solve the issue.

What we first tested was to see if the settings page worked at all and it did. This test was done by manually adding formEdt.aspx?List=[Your list guid] to the URL and everything worked fine.

We then found out that SharePoint removes this option, quite logically after you have set the list to “Allow management of content types”. This is done under Advanced Setting in List Settings. If you set that to No, the link will show up again.

This is because when you select to use Content Types in your list the columns becomes a new custom Content type. And If you need to have the content type then you need to change the Column order through the Content type. It’s basically just moved into the Content Type settings.

The error message I got was:

Exception calling “ProvisionGlobally” with “0″ argument(s): “An object of the type Microsoft.SharePoint.Administration. SPWebApplicationProvisioningJobDefinition named “Provisioning 0615878b-f543-43ab-aeb8-c59080cd52a8″ already exists under the parent Microsoft.SharePoint.Administration.SPWebService named “”. Rename your object or delete the existing object.”

This is because SharePoint creates a  timer job that runs once to provision the Web Application out to all WFE’s. So in this case when we extend the same Web Application, the job will have the same name and sometimes (almost all the time if you have more then one WFE) the timer job doesn’t get finished and there is your conflict. You can follow this process in Central Administration under Operations and Timer Job Status while you run your script.

The easiest way to solve this is to insert a sleep in to your script looking something  like this:

Start-Sleep -s 30

-s indicates how many seconds to wait so in this case 30 seconds.

But I wanted to make it a bit more sophisticated so I made a for each loop where I check if there is any provisioning job running, and if it is, wait another 30 sec.

To be able to do this we first need a reference to the farm and then to the Service running the Provisioning jobs. The Service is called “Windows SharePoint Services Web Application” so by looping through the JobDefinitions of that service, and look for a one named “Provisioning Web Application” we know if we need to wait some more or we can continue extending the Web Application.

This is how the script looks like:

$farm=[Microsoft.SharePoint.Administration.SPFarm]::Local

$Service = $farm.Services | Where { $_.TypeName -eq “Windows SharePoint Services Web Application”}

 while ($Service.jobdefinitions | where { $_.Title -like “*Provisioning Web Application*” } ){

 Write-Host “The ‘” $Service.jobdefinitions.Title “‘ job are still running. Waiting 30 sec to provision ” $_.WebAppDisplayName

  Start-Sleep -s 30

  }

I sometimes use Fiddler (which is a great tool by the way) when inspect the HTTP traffic, but without a custom HTTP Header it makes the hole process much more difficult.

Once again, I’m not much for repetitive tasks and in this case I had a lot of WFE’s and web applications. Therefore I put together a small script that enumerates and adds ServerName: [Servername] as custom HTTP header to all your sites on the server. You find it in my download section under IIS. And please don’t hang me for not making a beautiful script, It was a quick and dirty one and if you want to shape it up please feel free to do so.

There has been an error creating the personal site. Contact your site administrator for more information.

when you open up My Site for the first time? I have heard about this before and also read the solution (for instance found here). This is usually related to permissions not set up the right way but the other day I found out that it can happen during other circumstances. It’s likely that you will not get it but I thought that it would be good to share the knowledge.

The strange thing this time was that it only happened to one user and not everyone. That indicates that it’s not a permission problem, or most likely not.

I looked in the SSP and the user was imported in the profile import and the info from the AD was there. I decided to delete the site collection and found out that there was already another My Site Site collection there but with another name. It turns out that a former employee had the same AD account name. The account had been deleted from the AD since it was a long time ago but the My Site was still there. So SharePoint were not able to create the My Site because it already had Site Collection with that name.

Lesson learned. An updated routine for when deleting AD accounts on former employees. Best practise would be to never delete AD accounts in terms of traceability but it might also be a good thing to have a policy for how long My Sites should be stored. A suggestion is to have a retention period for when you delete the content and then have a Custom Property connected to AD where you manage the employment status like “full time”, “contractor”, “Parental leave” and Left the company yy-mm-dd etc.

Once again it was one of those problems that are hard to troubleshoot since it does not happen to all users, it happens sometimes and not on all our farms. This is usually indicating that there is something wrong with the client or the network like firewalls or load balansers, but you can never be sure. However, this time it seemed to be.

What happens when you try to import the spreadsheet is that Excel (in this case Office 2003 but we also had it on the 2007 version) gives you a “Cannot connect to the server at this time. Your list cannot be published” when you click on import.

There are a number of posts on the topic if you google it and it seems like there is a lot of things that can cause this problem. I therefore would like to have a word about troubleshooting the Import spreadsheet.

On the client side

First of all, the window that handles the import commes from the Excel client and not some SharePoint thing. The file that is used is the EXPTOOWS.XLA found in Program files\Microsoft Office\OFFICE11\1033 (If you have Office 2007 then the folder will be office12)

Instead of writing it all down I’ll give you a couple of links that shows what you need to change:

http://blog.sharepointhosting.com/Lists/Posts/Post.aspx?List=399ed944%2D72d2%2D4932%2Dbc13%2Dde0586abf2d1&ID=64

http://rdacollab.blogspot.com/2008/05/importing-excel-spreadsheet-to.html

Or a thread on the MSDN forum with a lot of info:

http://social.technet.microsoft.com/forums/en-US/sharepointgeneral/thread/75b2be6b-44df-4389-bf34-c7a01c12372c/

We can draw the conclusion that when mixing different Office version we can get a problem with the EXPTOOWS.XLA.

You should also make sure that you have the latest patches. Especially if you are running Office 2003. You atleast need to have SP3 and the following patches:

http://support.microsoft.com/kb/939269
http://support.microsoft.com/kb/952608

If you have Office 2007 try this one:

http://support.microsoft.com/kb/956510
What also is good practise in this kind of troubleshooting is to remove old cache files and entries. Specifically in the following locations:

Windows XP:
 c:\documents and settings\[Username]\nethood

Windows Vista
c:\users\[Username]\nethood

And you should also delete the cache files located in:

Windows XP
c:\documents and settings\[Username]\application data\microsoft\web server extensions\cache

Windows Vista
C:\Users\[Username]\AppData\Roaming\Microsoft\Web Server Extensions\Cache

In addition, remove all entries in the registry location:

HKEY_CURRENT_USER\Software\Microsoft\Office\11.0\Common\Internet\Server Cache

When you have completed that, make a reboot of your client.

Finally check in Internet Explorer so that the Option Check for server certificate revocation in Internet Options ->Advanced-> security is not checked, if it is, uncheck it and restart IE.

 
So let’s move over to the SharePoint server side

Check that you in IIS has a HTTP header for SharePoint. You find this by right click on the web site in IIS and then click on the tab HTTP Headers. You should have something like:

Custom header name
MicrosoftSharePointTeamServices

Custom header value:
12.0.0.6219 (or the version number of SharePoint installed)

You should also check so that not the ASMX extension itself is missing from the list of Application Mappings in IIS. You find it in the Properties of your IIS site -> Home Directory -> Configuration. 

You should do a IIS Reset after this change and you need to do it on all your web front end servers.

The network

In our case we found out that the problem was related to the load balancer. We have a couple of WFEs and when we changed the stickiness type to source ip instead of cookie it started to work. What it does is that it makes the Layer type to change from Layer 5 to Layer 4, and apperently this is a known bug in Cisco CSS Load Balancers. We found this after reading an excellent post from the MSDN guys:

http://blogs.msdn.com/vsofficedeveloper/pages/SharePoint-Cisco-CSS-Switch-Issue.aspx
If you are using ISA server or any other network boxes between the server and client you need to check them as well.

Hope this gives you a good summary and that after you have read this you get rid of the problem. However, if I have missed anything please post me and I will update this post.