BICTT

We had something funny today in a non-production environment.
We were simply installing the great product nWorks 5.6.
Installing the EMS first. During the install have to enter a service account. So we did, existing account in the domain, local admin and everything. It would simply not acknowledge that account!!! told me that the account or password was not correct. Well we were logged on with that account!
WHen running it as my own account (and installing the service with my account) the install went fine. So had to be the service account!

You never believe what the reason was...
There was an AT (@) sign at the end of the account name.
So in AD Users and Computers. Open an account properties. Go to the tab Account.
In the user login name part there are two fields.. one for the account name and one for the logon domain.
In the logon domain field there was just one AT sign !! no domain there... It should have been @domain.something.local for instance.
When we selected the correct domain, everything worked as expected.
This is what that looks like:

We also found the cause already, it was due to a script based account creation in the past.

Guess windows did just continue with this issue, but the installer of for instance nWorks could not handle that kind of abuse

Funny thing!
Glad we found and fixed it.
I dont expect anybody else to ever run across this as its a chance of one in multiple milions
But perhaps you can laugh about it as well, just like we did (after finding out what the issue was).

Greets from Bob Cornelissen

Another thing we saw in a SCOM 2007 R2 CU4 environment we were playing in to create new management servers, break them, migrate them, restore them and so on and so on.

What happens...

Installed a new management server and on top of that installed the SCOM 2007 R2 CU4 (that was the current version for that environment). All good. Next after a few minutes we saw the following error on that management server:

Source: HealthService
Event ID: 7022
The Health Service has downloaded secure configuration for management group (management group name), and processing the configuration failed with error code 0x80FF003F(0x80FF003F).

and

Source: HealthService
Event ID: 1220
Received configuration cannot be processed. Management group (management group name). The error is 0x80FF003F(0x80FF003F).

I knew I had seen this before so I checked my document where I try to keep track of these things and found two entries to KB articles. One is about when this happens on an agent which happens to be a domain controlelr as well, and the other one was about a newly installed management server. Bingo!

New Management Server unable to get configuration in System Center Operations Manager 2007
http://support.microsoft.com/kb/2027535

So as stated in the KB article we changed the "Default Action Account" runas profile for that server to Local System Action Account and everything was running again. Waited a few minutes for the server to go ahead and come to peace with itself. Next we wanted to move the runas account back to the action account it was running under in that profile as stated in the article as well. Immediate error and it would not let me save this new config. If that happens to you, just go to that runas account and re-type the password (even though it is still the same!). Next change the runas profile again.. working!

Not sure why this would have gone wrong, but it sure helped to get stuff working again!
I have sometimes seen something go wrong with a runas account on the password and with re-typing it stuff started to work again.

If you run into it, good luck!
Bob Cornelissen

Here is another thing I saw happening in a test environment, which does not happen if you follow the guide :-) But always good to know what it looks like. Thought it is informative for somebody who might run into that and search for it.

This is situation. Test environment for SCOM 2007 R2 CU4. wanted to do a migration and wanted to swing the RMS role to another management server in order to break down the old RMS and re-install. Simple procedure. Take encryption key to the other management server, import it and use the managementserverconfigtool.exe to PromoteRMS and swing the RMS role to that management server. That worked out and after some cleaning and so on a new machine could be installed and installed SCOM on that machine. So now the machine needed to be promoted again.

AT some point in the procedure the guy who did this procedure ran the managementserverconfigtool.exe PromoteRMS command. This wizard asks you twice to confirm with an Y. Next it will continue and do the promotion for you. The first step is to enable the scomsdk service on the management server and start it in order to get the management group information to continue the promotion. However this machine was stuck at this step. They could see the scomsdk service was started and running, so good. Nothing on processor. It was just sitting there waiting for the next step it seemed. After killing this process and reverting the service to disabled state and rebooting both the MS and RMS they tried again. Same result. It was stuck at the first step of starting the SCOMSDK service (which it did do). As it was the end of the day they just left it there and the next day the command had stopped working with the message that "Unable to retrieve management group information on (local machine name here)". Aha! now we have information!

So what happened here...

The management group information gets retrieved through the sdk service and needs to have the encryption key for that!
Guess what.. the step to import the RMS encryption key was not done on that management server before trying to promote it.

So now you know what happens when you try to promote a management server without the RMS encryption key.

Good luck if you run into this.
Bob Cornelissen

Just a short note this time. I have blogged about the SCOM 2012 CEP (community evaluation program) a lot already. And yes, I have yet to blog about the last two sessions, but I have been swamped with stuff to do. I will get to it for sure as they are very interesting subjects and near to the heart Like I have said before I really like the CEP program in general, because it gives access basically to information provided by the product team of that product about what is new and how to use it and see demo's and ability to ask questions and such. This makes evaluating the product that much better for the public. It is also why I have been blogging about what is being said and things that strike out to me during those sessions. If you are serious about evaluating the beta products, join a CEP program if they have one!!

This week I got a note from Nicole Pargoff from the CEP program:
Congratulations! You are the October winner of the Operations Manager 2012 Community Evaluation Program Community Participation Contest. Thanks for your great blogs and all your on-going participation!

Thanks Nicole for your kind words and thanks to the CEP program for extending into the community like this.
Bob Cornelissen

A few weeks ago I have been part of an Inside Central podcast on cross platform monitoring with SCOM, with Dan Kregor and Pete Zerger. It was lots of fun to do. Also good news is that the website for inside podcast network has been renewed and revamped. Great work Dan!! There are several kinds of shows on this channel and more to come. Go over and take a look at the site and scroll through.

Also here is a link to the podcast that features Bob Cornelissen about the cross platform monitoring.
http://insidepodcastnetwork.tv/show/ic10/

Have fun checking out the podcasts!
Bob Cornelissen

Early next year there will be a new Mastering SCOM 2012 book from Sybex. I have been asked to participate as a writer for this book together with a few others. So I will be writing a few chapters for that book the coming months. I love the challenge and the opportunity to work with the other writers of the book and the great people at Wiley. Means it will be a bit more busy for me and a bit less time to blog. Although I will try to keep up and post usefull information, tricks, errors, successes and so on.

It is already ready for pre-order over at Amazon:
http://www.amazon.com/Mastering-System-Center-Operations-Manager/dp/1118128990/ref=sr_1_1?ie=UTF8&qid=1319009875&sr=8-1
As seen from Amazon the expected release date is in April.
The book page over at WIley website:
http://eu.wiley.com/WileyCDA/WileyTitle/productCd-1118128990.html

As things progress with the book I will do some more postings about the book, the authors team, the cover and perhaps release date and ways to get your hands on it We will see what happens

Happy that the RC of SCOM 2012 is out now (see my post from yesterday) so can now re-build some test machines and get to work!

For later in a few months.. happy reading!
Bob Cornelissen

Just got word over at the Server Cloud blog that SCOM 2012 RC is available for download.
http://blogs.technet.com/b/server-cloud/archive/2011/11/10/system-center-operations-manager-2012-release-candidate-from-the-datacenter-to-the-cloud.aspx

This is the link where you can download it:
http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=27974

On the download page you can also download the updated docs. If you are upgrading the SCOM 2012 Beta version to RC have a quick read in the RC documentation - deployment guide, page 172 for things to keep in mind and required steps.

Enjoy your testing!!
Bob Cornelissen

Another interesting case I ran into at a customer site. In this site they use a Default Action Account for their SCOM agents. We rolled out two new SCOM agents and did not specify the agent action account during deployment. So went into SCOM administration -> Run As Configuration -> Profiles and found the Default Action Account profile. The two machines were using the Local System as their default action account as expected, while all others were using a specific account for that. Easy enough, click Edit and use the dropdown list to specify the same account as what all other agents are using. When trying to confirm this I got the following error. I will first pate the text version and followed by a screenshot followed by stack trace .

Cannot Associate a credential with a blank password to a health service. Accounts with blank passwords are inserted by a health service and cannot be used by users until they are updated to include a password explicitly.

Here is the stack trace belonging to this error:

Code:

This would suggest that the account I am trying to assign to this machine has a blank password. Funny because all others are using it already.
But anyway, I went into the SCOM console - administration pane - Run As Configuration -> Accounts and found the Action Account we were assigning. Opened the properties and typed the password again.
Right after this we could change the Default Action Account Run As Profile.

If it happens to anybody... hope it helps. Make sure you check that that password you are typing in the box is the right one first! Dont want to get errors from all agents....

Bob Cornelissen

The SCCM team have reached another milestone in getting the release candidates for SCCM 2012 and FEP 2012 released to the public. Also interesting that they are moving the antivirus product name from ForeFront to System Center as it seems. See the extract below.

AN extract from the email we got about this:

We are extremely excited to announce the availability of the release candidates for System Center 2012 Configuration Manager and System Center 2012 Endpoint Protection (formerly known as Forefront Endpoint Protection 2012) today. Both releases are available through a single download package on the Microsoft Download Center. You can learn more about this release at our Server and Cloud Platform blog.

Looks good eh? Now we can download and test the RC!
Bob Cornelissen

Less than 2 weeks ago Microsoft released a new version of the Base-OS management pack for Windows Server 2000/2003/2008). This MP contains a few cool new features and updates. Also there were a number of community feedbacks implemented, which is a very good thing! These have been blogged about and I will mention these in a short version as well below. However the community and MS PFE's quickly discovered some issues with that version of the MP and a community effort was started to quickly fix these issues. During the weekend a number of us had frequent contact with a friend at the SCOM product team, Barry Shilmover, who took up the task to fix the stopping issues and get an updated version out to the public. A big thumbs up for him from us!!! If you are running version 6.0.6957.0 or less, please consider upgrading to the newest version 6.0.6958.0.

So what are the new things?

CSV monitoring

It now discovers and monitoring Cluster Shared Volumes (CSV). This is a separate .mp file in the download which you can choose to import. Good one!

BPA analyzer

Also a separate .mp file in the management pack download. This monitor basically connects to the system BPA in windows 2008 and gives the output from that tool back to SCOM. Now there is a bug in the WSUS BPA that causes that one to give an error for each machine in the environment, which causes some confusion and some additional noise, so in the newest version of the management pack it was chosen to disable this monitor by default. You can use an override to enable it again. For the WSUS BPA thing you can check this following thread for more info and a workaround (look for entries from Pavel for this).

Some changed and disabled monitors and rules

A number of monitors were changed to reflect more realistic monitoring scenario's. Also a number of performance collection rules were disabled. These mainly caused a lot of performance data to be gathered and stored which nobody seemd to report on. These rules are still there, but set to disabled. This is also a community request. You can turn on any performance collection you want by using an override. These are mentioned in the management pack guide.

New reports!

There are two new reports, which are included in a separate .mp file in the management pack download. These are really great reports and very useful!! Simply great! One catch though... if you are running SQL 2005 as your SCOM backend you can not deploy these and you will get an alert in SCOM like "Data Warehouse failed to deploy database component. Failed to deploy Data Warehouse component. The operation will be retried" and your RMS will turn red. If you are running SQL 205, just go to SCO administration - management packs. Find and remove the mp called "Windows Server Operating System Reports". This only contains those two reports and not the rest of the known reports. If you are on SQL 2008 you will see these reports in the reporting console in the folder Windows Server Operating System Reports.

An example of the quick run of the Performance By Utilization report is below:

These reports are absolutely great!

So thats the short version of the recap on this new version of the new Base-OS management pack.
Keep in mind that it is best to use the downlaod link to download and extract all the .mp files to local disk before importing them (from disk). This way you will see all the mp files and you can select which ones to take and you will have a backup of that specific version available locally.

Here is the download link to the management pack with version 6.0.6958.0 :
http://www.microsoft.com/download/en/details.aspx?id=9296

Again a big thank you to the SCOM product team (and of course I mean especially you Barry for giving up your sleep for this) in taking quick action and providing the required updated version and feedback. I also want to thank the community for reacting to the call to action by Marnix and myself last week.

Now go on and enjoy the new and updated MP!!

Bob Cornelissen