Had an issue the other day with an already installed and configured agent refusing to start.
When trying to start the service through command prompt I got this error:
C:\>net start healthservice
The System Center Management service is starting.
The System Center Management service could not be started.
A service specific error occurred: 2164195395.
And in the System event log I found this entry:
The System Center Management service terminated with service-specific error %%-2130771901
Went back to look in the Operations Manager event log entries to see what happened right before the agent service stopped and found at least these two error entries a few days earlier:
Operations Manager has received a bad configuration file for management group XYZ which could not be parsed. Configuration has been re-requested. A copy of the bad configuration file will be placed in C:\Program Files\System Center Operations Manager\Agent\Health Service State\\Connector Configuration Cache\XYZ\OpsMgrConnector.Config.Error.xml
The Management Group XYZ failed to start. The error message is 0xC00CEE04(0xC00CEE04). A previous message with more detail may have been logged.
Right after these the agent had stopped.
Usually first thing to start is to check the registry for the WindowsAccountLockdownSD entry:
This entry should be present. Otherwise copy it from another machine in the same management group with a working agent.
In this case this was all there and good.
Next checked the Registry again, but looking for the State Directory here:
This should have an entry called State Directory and please check if the directory path stated there exists and is correct.
Next I went to this directory and removed the contents.
In my case everything located in:
C:\Program Files\System Center Operations Manager\Agent\Health Service State\
Tried to start the healthservice again through the services.msc or by command prompt net start healthservice
And it started working again.
Very soon it will be the time for the System Center Universe 2014 event again.
This year there is a contest for the SCU Jedi. The one who wins will speak about his topic of interest at the event.
Voting is still possible until the 15th of December, so don't forget to vote!
Here is the playlist with all three remaining contestants:
Play the videos and vote for your favorite by using the like options.
My favorite is Dieter Wijckmans with a great monitoring related subject!!
Jalasoft has been creating great network monitoring and visualization tools on top of SCOM for years now and I have been working with them for a long time. They are quite active in creating new products and new versions of existing products, which evolve with customer demand and SCOM product updates. These weeks we are seeing a number of product updates:
Xian Network Manager 2012 SP2
A little less than 2 weeks ago Jalasoft announced the release of Xian Network Manager 2012 Service Pack 2. This latest version of the product supports SCOM 2012 R2. It features improvements to the Configuration Monitoring feature and adding support for the latest version of VMware Virtual Center. In addition, one of the main features called Netflow also got a lot of upgraded functionality and can handle V9 templates and adds dashboard improvements. Also it can now filter data to a top-N set to send to SCOM in order not to overflow SCOM with too much data which is marginal in importance.
More information on Xian Network Manager can be found here: http://www.jalasoft.com/xian/networkmanager
Xian Wings 2.0
The next thing to get exited about is that Jalasoft is releasing Xian Wings 2.0 this week. It will be released on Wednesday 30 October 2013! Xian Wings brings all kinds of SCOM views to your mobile and tab devices, including state, alerts, performance graphs, tasks and notifications. This is not limited to the Xian Network Manager objects in SCOM, but all objects. Now I am pleased to early-announce (with permission) some of the improvements in Wings 2.0:
Xian Wings HD for iPad
- Remote Shell: it is now possible to run PowerShell and cmd prompt commands directly from your iPad against the SCOM server or another computer. This means that additional tasks or actions can be directly done as a response to some alerts received in SCOM. I will surely be showing you what this looks like in one my upcoming blog posts.
- Notifications Widgets: besides alerts, states and performance graphs, it is now possible to add notifications widgets to any Wings dashboard. This widget will display a new notification object for each alert generated or modified on SCOM associated to any selected object within the scope of this particular widget.
Xian Wings for Windows Phone
- Full support for Windows phone 8: a new Xian Wings client for Windows phone 8 has been created in order to fully support this new version. Besides this latest version, Wings still supports phone 7 as well. It is very nice to see both versions of the Windows Phone OS are now supported.
Xian Wings for Android and iPhone
- Class icon for each monitored object: each monitored object now displays its associated type icon along with its status, making it easier to know which object has problems and what kind of component it is; for instance a database, computer or network device.
Xian Wings 2.0 Server
- The Xian Wings server now fully supports .Net Frameworks 4.5 in order to improve performance and compatibility with the latest Microsoft platforms and technologies.
- The Xian Wings server and/or console can now be installed on any computer running Windows 7 or 8.
More information on Xian Wings can be found here: http://www.jalasoft.com/xian/wings
Like I said this calls for an additional blog post on the possibilities of the latest Xian Wings 2.0 on my iPad mini. Stay tuned for that one to come in the near future!
The case of not being able to uninstall SCOM agent due to Failed to load Intercept Scripting LibrarySCOM, System Center, SCOM 2012 Send feedback »
Had a strange case today. On some servers in a DMZ we wanted to upgrade the existing SCOM 2007 R2 agents to SCOM 2012 SP1. However something strange happened during upgrade and somehow it looked like both the old agent and the new agent were installed, but not working. Upon removing of the old agent entry from the control panel we still found the SCOM 2012 agent unable to connect and even to start the service (we had both cases actually on a few machines). We also had one machine where we first removed the old agent followed by a reboot and a clean install of the new agent and still had below problems.
Next course of action is to go to Control Panel and uninstall the SCOM 2012 agent. Only to receive an error like the one below. Failed to load Intercept Library.
Troubleshooting led to all kinds of things
First because I couldn't remove the agent, I thought lets try to heal it first.
Lets go through a few items:
- The system environment variable Path still contained the path to the 2007 agent folder.
- The state directory of the agent found in the registry was pointing to the 2007 folder at hklm\system\currentcontrolset\services\healthservice\parameters\
- There were a few old entries in the registry pointing to wrong folder names. However looking at these they seemed not to be that important. Cleaned them anyway.
Try to start the SCOM agent.
Results in two errors seen below. Especially the 2147221164 error seemed to be the main one and we got that one on all the machines.
The error 2147221164 can be caused by different things and I have some links at the bottom of this post that have some relation. However in this case it was something different. We were missing a registry entry for a connector.
We ran the following registry file to merge into the registry:
Windows Registry Editor Version 5.00
@="C:\\Program Files\\System Center Operations Manager\\Agent\\MOMConnector.dll"
Well, at least the SCOM agent service starts after this. However we are immediately greeted in the Operations Manager event log by hundreds of 4507 errors. They talk about not being able to register a class ID in the registry (the agent tries to register some entries in the HKEY_classes_root\CLSID for different types of workflows). And with hundreds I mean hundreds of these errors.
So I guess we get back to uninstalling the agent again, because this can not be good.
This time I went out and got the log file where we uninstall the agent. I see the following entries:
CExecutorHelper::GetPath: FAILED: Executor not registered.. Error Code: 0x80070057.
UninstallAPMMonitoringConfiguration: Failed to load Intercept Scripting Library. Error Code: 0x80004005.
MSI (s) (B8!D8) [14:19:26:606]: Product: System Center 2012 - Operations Manager Agent -- Failed to load Intercept Scripting Library
Well, I have looked and could not find anything about this error "Failed to load Intercept Scripting Library" or anything in these lines basically.
What we can see in here is what I expected when I saw this error. Intercept Studio is the old name for the APM piece of the SCOM agent. And we can see it is trying to uninstall that piece (or at least gather all related info).
At this point I decided to call out to my fellow MVP's and MS Product Team members. Somebody must have an idea
And sure enough, out of the Product Group there came a suggestion. He actually looked at the first line in the error (Executor not registered). So he suggested to register an APM dll and see if that got it moving.
Alright here goes:
In written form:
- In a (elevated) command prompt on the server go to the directory where the SCOM agent APM component is installed.
C:\Program Files\System Center Operations Manager\Agent\APMDOTNETAgent
- Next register the DLL
Now try to uninstall the SCOM agent again.
Next we close all open programs and clean out the SCOM agent related directories in "C:\Program Files". I am guessing they were left over after the issues.
Next we could proceed to install the SCOm 2012 SP1 agent and UR3 and run momcertimport.exe to import the already existing certificate. And everytyhing was running again!
A very big thank you to Vlad, Daniele and Vitaly for getting me in touch with Dmitry, who gave the suggestions pointing to the solution!! And of course a very big thank you to Dmitry himself for trying to think of ways how this could happen and what might be done to get it rolling again!
Some of the articles I have used in troubleshooting and which I freely interpreted during troubleshooting and trying to solve this:
AT least some of the things mentioned in these linked articles pointed me to somewhat similar situations for smaller pieces of the puzzle and where to look for troubleshooting.
Hope this helps the very few that might run into a similar issue.
As many of you know the last version of the SharePoint 2013 management pack had an issue with its reports. They couldn't be published to the report server and thus caused the health state of the management server to turn to a red state. Yesterday an updated version of the SharePoint 2013 packs have been published that contain the fix to that issue.
You can find the Packs here:
- System Center Management Pack for SharePoint Server 2013
- System Center Management Pack for SharePoint Foundation 2013
Very happy this got fixed now, so we can get to monitoring again
Last week I installed a SCOM agent on a Windows 2008 R2 print server, only to discover that the SCOM agent went wild and kept crashing. Furthermore I found out it was generating a lot of data (about 18 milion entries in a few hours). So what had happened? Well the reason I found in this blog post from Daniel Mueller dating from April 2013.
In short the Printer Server 2008 mp (version 6.0.7004.0) has a mistake in a few performance collection rules. They are targetted at "Printer" in stead of print server role. Because it collects data from all instances, this meant in my case it saw about 500 printer queues and multiplied that by 500 times, because every printer queue monitored all instances of all other printer queues as well. As you can see with a high number of printer queues this thing will go wild.
The Printer Server 2008 management pack dates from December 2012 and it looks like even after the blog post from Daniel nothing has happened with it. So I decided a few days ago that I will post a hotfix management pack which will disable the original faulty collection rules and create new rules with the same settings but correct targetting.
The "Hotfix Management Pack: Print Server 2008" can be found in the TechNet Gallery.
It is specifically to override the 6.0.7004.0 version of the Print Server 2008 management pack from Microsoft, so you do not run into the performance and agent issues.
And thanks to Daniel Mueller for recognizing this issue in the first place.
According to a post from Brad Anderson the release date for several products will all be the same: 18 October 2013.
These are of course Windows Server 2012 R2, Windows 8.1, SYstem Center 2012 R2 as main components.
Because these products were developed together and work together to form a number of great new scnearios these are also now release together and will work together as well.
So lets get ready for this!
Especially for our friends from the Netherlands and Belgium:
The next user group meeting from WMUG NL is at 17 September in De Meern in the Netherlands. The evening's topic is Virtualization and has great speakers like Ruben Spruijt, Henk Arts and James van den Berg !!!
Get more information and a registration link through this link (in Dutch): http://wmug.nl/2013/07/30/%ef%bb%bf-17-september-wmug-nl-bijeenkomst-met-als-thema-virtualisatie/ .
More information on the specific subjects these speakers will talk about will be shared on that page soon.
Hope to see you there!
A few days ago the UR3 for DPM 2012 SP1 was released together with a few other updates (also for VMM, SCOM and AppController). A few days after it seems that a number of people have had issues with this UR3 update.
Issues are not completely clear yet, but seem to include things like not being able to upgrade the agent and not being able to delete or mofiy the membership of a protection group.
This issue is already being taken care of by the DPM team currently. In the meantime they have pulled the update down. I assume we will soon get information on a fresher version of the same and guidance on what to do if you are affected already. This is a message in the comments of a few blog articles from the DPM team:
We are actively working on the issues that have been reported here. We have pulled down the update from Microsoft Update. Given the issues seen at this point we would strongly advise against any further installation of Update Rollup 3 for DPM 2012 SP1.
We will shortly update you on the issues and the next steps. Stay tuned.
- DPM Team
As soon as I hear more information on the issue or if and when it is fixed I will update this post of course.
Microsoft released a KB article containing a fix for one of the following cases:
In System Center Operations Manager 2007 R2 environments you may see the following:
- The Data warehouse size grows, and upon investigation you discover that partitions are not being dropped/optimized even though grooming jobs are showing as running successfully.
- Report results do not match the resource availability and do not match the visible data in the Operations Manager console.
Keep in mind that the script to fix these items requires first that SCOM 2007 R2 with CU7 is installed a minimum version.
The KB article can be found here: