BICTT

Lately have been investigating some strange behavior at a customer site where SCOM 2007 would not send out notifications. At the same time we were investigating why a test of promoting a management server to RMS would not work correctly. As it turned out the reason was the same. It’s all about the Operations Manager Internal Connector. It took some time to find both explanations, which can be frustrating to others, so I will post some information about this so it might help other people who happen to run into the same thing. This prompted a few questions on the Operations Manager Internal Connector… Where is it, what does it do, how can it be broken, What does it look like when it gets broken, what can be done if it is gone? So here we go!

Where is it?

Open SCOM console and go to Administration. In the menu on the left go to Product Connectors -> Internal Connectors. Now you will see one or more connectors listed there. The Operations Manager Internal Connector should always be there. If it is not, you are sure you want to keep reading! There can be other connectors listed here if you have installed and configured features that use connectors, for instance Schedule Maintenance Mode Connector or Savision LiveMaps Vx Connector. In SCOM 2012 this connector is also installed by default, next to the Network Monitoring Internal Connector and the SMASH Discovery Internal Connector.

What does it do?

Documentation on this connector is a bit limited. What the description of the connector says is that it is used by Operations Manager components to insert discovery data and that you should not use this connector yourself. In general connectors are used to synchronize alerts with other monitoring/alerting/ticketing systems, but they can be used for more uses. This connector is also used as part of the SCOM notification process for the notification channels you create through the console. This is not visible really, so the relationship is not clear when looking at it.

How can it get broken or removed?

1. By trying to forcibly remove it through a script or removing it directly from the database.
2. By not correctly following the steps while updating SCOM to a new Cumulative Update (CU) level.

I will give references and explanations below of both situations.

1) Removing connectors by script

Through the SCOM console you cannot remove a connector. So once it is there are you want to get rid of a connector you can delete it through a script. This method is described in KB article 2626670 http://support.microsoft.com/kb/2626670 . Keep in mind the notes mentioned in this article to not remove Internal Connectors and the note near the bottom stating the following:

Note If you attempt to delete the Operations Manager Internal Connector from the database it will generate the following error:

Exception content:
Exception calling "Setup" with "1" argument(s): "Discovery data generated by invalid connector:7431E155-3D9E-4724-895E-C03BA951A352."

It is always advisable to create backups of the database before attempting to run these kinds of scripts as you can mess up your management group if you make a mistake!

2) By not correctly following the steps while updating SCOM to a new Cumulative Update (CU) level.

There is another way to wreck the Operations Manager Internal Connector, which can happen in rare cases and if the right procedures are not followed. In this case we are talking about SCOM 2007 R2 CU4 mainly, but CU5 might have the same effect.

The correct order is (taken partly from one of Kevin Holman’s posts):
1. Backup the Operations and Warehouse databases, and all unsealed MP’s.
2. Apply the hotfix to the RMS
3. Run the SQL script(s) update against the OpsDB AND Warehouse DB.
4. Import the updated management packs provided.
5. Apply the hotfix to all secondary Management Servers.
6. And so on and so on…

I cut the list short a bit, because the most important things in this discussion are there already. Now step 1 is the important part in case you need to do a roll-back in case you make a big mistake or the system causes some problem during installation. Better be safe than sorry! It will become clear later what the alternative is to restoring the database.
Steps 2 and 3 and 4 are very important and must be done in that order. Install the hotfix on the RMS, run the SQL scripts coming with the update and install the management packs coming with the update. After that it’s just updating the rest of the management group components.

Now why is this important you might ask? Well because the Operations Manager Internal Connector also gets touched in the procedure in the background.

Thing is that during the CU4 upgrade the “System Center Core Library” management pack get updated to the 6.1.7221.61 version. In CU5 this will upgrade it to the 6.1.7221.81 version. Now if you follow the steps 2 and 3 and 4 as described in the short version of the list above there will not be a problem.

However it is possible that somebody upgrades the bits on the RMS and forgets to run the SQL scripts directly after that step. If you then import the 6.1.7221.61 or .81 version of the Microsoft.systemcenter.library.mp file into the management group you can wreck the system.

Now, normally you would not have this management pack version as an .mp file. The reason is that the SQL script coming with CU4 and CU5 actually contains this management pack and updates it during the procedure. This is just one of the many steps the scripts do by the way.

So why would you have this file anyway? Well quite simply if you are an management pack author and use the Authoring Console and open a management pack with a reference to the System Center Core Library it will ask you for that .mp file, which you don’t have! After many people got this error and could not get the mp file the easy way a KB article was published where you can get that management pack for the purpose of using with the Authoring Console while editing management packs. This is KB 2590414 http://support.microsoft.com/kb/2590414.

This management pack should not be used to import directly in the SCOM management group. As explained before this is part of the CU 4 and CU5 upgrade process. There is an important note near the top of the KB article:

NOTE: Do not import the Microsoft.SystemCenter.Library management pack into your management group. You do not have to do this. In fact, in a very limited set of circumstances, importing this management pack can result in the need to restore the Operations Manager database from backup.

And another note near the bottom:

When a new management pack is created in the Operations Console, the management pack automatically includes a dependency on the latest version of the Microsoft.SystemCenter.Library management pack. In Cumulative Update 4 (CU4), a change was made to the Microsoft.SystemCenter.Library management pack to prevent connectors from appearing as Distributed Applications in the Distributed Applications state view and in other locations where Distributed Applications could be displayed, such as the Distributed Application Designer. This change was made in response to the feedback of customers who did not want connectors to be shown in the Distributed Applications state view.

WarningThe updated version of the Microsoft.SystemCenter.Library management pack is embedded in the SQL script that is included with each new cumulative update. The management pack is then imported into the management group when the administrator runs the SQL script. The script bypasses the verification code.

If the CU4 binary update is applied without the CU4 SQL update, you will be able to import version 6.1.7221.61 or a later version of the Microsoft.SystemCenter.Library. This results in the removal of all connector instances from the management group. The only way to recover from this situation is to restore from a DB backup. This is why access to Microsoft.SystemCenter.Library is limited.

Aha! There we go. This states that if you forget to run the CU4/5 SQL scripts and import this mp anyway it will remove all connector instances. So also the Operations Manager Internal Connector.

So quite simply make sure you follow the procedure for updating SCOM and make sure you have backups of course. At first this mp file was only made available through customer support, so it could be checked that the SQL scripts belonging to the cumulative update had been run correctly before handing out this file to a customer. However, due to so many authors having problems while trying to edit management packs the KB article later included the download of the file.

What does it look like when it gets broken/removed?

Well mostly you don’t see it. I mean you really don’t see it in the console, but also in other ways. First let’s look at the console:

There is an empty space where it should be. In this case another MP was imported after this action which also got a connector, but since it was distracting I removed the name from the screenshot. In any case the Operations Manager Internal Connector was not there.

So what is the effect on notifications?

Well quite simply nothing happens. In a test environment where this had happened a few months earlier I was trying to add notification channels and subscribers and subscriptions. All done without any problems. Except that it simply did not send out any notifications. So I tried to create a command channel to dump all alerts into a text file. Nothing. No errors, no alerts, no events in event logs.

What is the effect on the RMS promotion process?

Next situation is during the promotion of a management server to RMS in that management group.
When you do the PromoteRMS command you will see that services get stopped on the old RMS and started on the new RMS, it will update an entry in the DW. And if you do not have the Operations Manager Internal Connector installed you will see the following, followed by a stop of the PromoteRMS command:
An object of type MonitoringConnector with Id 0d234406-0029-4605-b376-86b78dcac3b2 was not found.
And guess what… 0d234406-0029-4605-b376-86b78dcac3b2 is the Operations Manager Internal Connector. It tries to change a reference in that row and cannot find it.

And now the important question:

What can be done if the Operations Manager Internal Connector is no longer there due to one of the situations describer above?

The answers to this question are easy to give, but not much fun.

• Restore the database from your last backup. I do hope you have a very recent backup!
  You will find several guides on how to restore a database online, one of which is found in the Operations Administrator’s guide over here: http://technet.microsoft.com/en-us/library/cc540396.aspx
• OR
• Completely re-build your management group. I hope you have some spare time!

Wrap up

So now we know what it is there for, what happens if it is gone, what could be the cause of breaking it and how to get back up to a running state.
So I hope this has cleared up a bit about the Operations Manager Internal Connector.

Thanks to Lukasz Rutkowski, Marnix Wolf, Simon Skinner and a big thanks to Vlad Joanovic for helping me gather more info about these items and gathering some understanding about it. Thanks guys!

Bob Cornelissen