Saving dogs while monitoring

SCOM, System Center, Active Directory, SCOM Tricks, SCOM 2012, Windows 2012, Configuration Manager Send feedback »

I know this blog is all about my technical career in IT, with System Center and Microsoft products and so on and so on. But I also have a secondary life which is very important to me and a lot more. In fact I am somewhat of a shy person and so is my wife. We do not ask for favors much, but do like to give them. I am talking about our dog and animal sanctuary in Thailand. Currently my wife and our staff Oarn are mainly taking care of about 60 dogs in 3 locations. Also some of our family members like the Mother and sister of my wife in one location and her father in another are doing their best to help out. We have to do all of this out of our own pocket and I can tell you it is not possible to do this alone. Also we have some huge set backs in the past few months. We are in the process of emergency relocation and are in a big stress to find funding to be able to buy a new small plot of land in a less expensive province where we can relocate to, build fencing, build a roof, build a 1 room shelter for staff and relocate most of the dogs to that location. Of course this is next to our ongoing push for health (sterilization, anti-tick program, helping sick and wounded, vaccinations, vitamins and trace elements and iron in their food somehow to get them healthier).

Two days ago I did a webinar with my good friend Cameron Fuller for our friends at Jalasoft. The webinar was of course SCOM and monitoring related. Read about the webinar and also some of the things which Cameron said about the dog sanctuary which I would not have said myself and humbles us:
http://blogs.catapultsystems.com/cfuller/archive/2015/04/16/what-is-new-and-coming-soon-to-the-microsoft-monitoring-world-and-helping-a-good-cause-opinsights-scom-opsmgr-sysctr.aspx

We all donated the proceedings of this webinar to the good cause. Lek's House Of Tails.
More information can be found at following locations:

A website with some background information (see the tabs near the top below the picture) and information on how to donate (there are other methods, feel free to get in touch if you want to know how):
http://lekshouseoftails.azurewebsites.net/?page_id=222

The facebook page of the sanctuary:
https://www.facebook.com/LeksHouseOfTails

My personal Facebook Page (have lots of pictures from several years of the dog sanctuary):
https://www.facebook.com/bob.cornelissen.5

Feel free to contact us!

Also if you want to help out by whatever means (on location help in Thailand, financial, words of advice, toys, sharing our pages and posts, liking a post or whatever you can think of it is very much appreciated!
These dogs we have known for years, we have known their grandparents and all. They got deserted by people. They were in dire need. We try to help them have a life, eat food, have a roof for the rain, have a blanket to sleep on, have a bowl to eat from, be safe from getting hit by people or cars or getting poisonned or caught for some terrible plight. We just want to keep them safe, fed, healthy. They deserve a chance and all of them are really sweet and kind.
Due to bad circumstances we had to relocate to a temporary place and will soon move to our new place, which is a piece of empty land with nothing on it and start all over.

Now to get back to the webinar at Jalasoft. We have done this webinar and I feel it went great :p We want to thank Jalasoft for giving us this opportunity. I want to thank again my co-presenter Cameron Fuller, don't even know how to say how thankful we are mate.

Also a big thank you to several community members out of the technical community and MVP's who have helped in several ways in the past year.

During the webinar we had a surprise for the great folks and team at Jalasoft and that was that we named one of the pups without a name after their product range: Xian.
See below for a picture of Xian (on the left, with his brothers), it is a very smart and active dog and we thought it is a perfect fit :D

One more thing: The webinar we did was recorded and I am sure Jalasoft will publish it soon. Of course you can Always send a message to sales@jalasoft.com to get access to the webinar recording or the slides and of course to get more information on one or more of their great products. Of course in the near future I will be writing about several products and monitoring and System Center.

Xian and brothers

Thank you all so much for your patience and help and shares on behalf of me, my wife, our families and 60 sets of teeth from our dogs!
Bob Cornelissen

SCOM 2012 Importing a list of ICMP Devices

SCOM, System Center, SCOM Tricks, SCOM 2012 Send feedback »

Today we had a case where we wanted to make an additional ICMP ping monitoring for all the Unix/Linux servers we were already monitoring in SCOM. There was a specific reason for that. I wanted to quickly get all the IP addresses of the Unix/Linux machines out of SCOM and into the Device Discovery.

First I started by wanting to export the IP addresses through Powershell. So I started from a query I had made earlier over here and use that to pick up only the IP address and create a CSV file.

get-scomclass -Displayname "UNIX/Linux Computer" |get-scomclassinstance |select @Label="IPAddress";Expression={$_.'[Microsoft.Unix.Computer].IPAddress'}} |Export-CSV -notype C:\Scripts\nixexport.txt

Alright, Now I have a TXT file I can play with. I used Notepad to replace the double quote with nothing. And I removed the first line (which was the header). This leaves us with a simple txt file with a list of all IP addresses.

Next in the SCOM Network Device Discovery, I opened an existing discovery rule with some devices in it already and selected the Import button in that wizard (step 1 in below picture). Select the txt file and all the devices will be added to the bottom of the devices list of the discovery rule. Next Multi-select all of them (step 2 in below picture) and select Edit (step 3). Change discovery mode to ICMP. The picture was taken after they all got changed to ICMP already. And now click through the rest of the discovery wizard and let it run.

So this went quite quickly and with very little remembering of IP addresses and typos.

Good luck!
Bob Cornelissen

SCOM 2012 R2 UR5 and SCOM 2012 SP1 UR9 released

SCOM, System Center, SCOM 2012 Send feedback »

Last night there were some update rollups released for SCOM. One for SCOM 2012 SP1 and one for SCOM 2012 R2. Will list them below.

For SCOM 2012 SP1 this is Update Rollup 9 and can be found at KB3023167. The description does not include much at the moment except the mention of Operational Insight and the support for SUSE Linux 12. The article can be found here: http://support.microsoft.com/kb/3023167

For SCOM 2012 R2 this is Update Rollup 5 and can be found at KB3023138. The description has several items listed and there is also the Linux packs separately with an update. The article can be found here: http://support.microsoft.com/kb/3023138

Items in the UR5 rollup for SCOM 2012 R2:

  • Monitoringhost process crashes because of bind failures against Active Directory
  • RunAs accounts cannot be edited because of "Specified cast is invalid" exception
  • Application crashes when a search is finished without filter criteria on Distributed application designer
  • MonitoringHost crashes with the exception - System.OverflowException: Value was either too large or too small for an Int32
  • Support to troubleshoot PowerShell scripts
  • Operational Insights through Operations Manager
  • "Reset the Baseline," "Pause the baseline," and "Resume the baseline" tasks fail when they are run against an Optimized performance collection rule
  • An exception occurs when you edit subscriptions that contain deleted monitors or objects
  • You can't set Widget column width
  • Event 4506: Data was dropped because of too much outstanding data
  • Support for handling Datawarehouse time-outs
  • Can't continue scan with NOLOCK because of data movement
  • $ScriptContext.Context does not persist the value in PowerShell widgets
  • Evaluation version alert

And the Unix/linux Pack:

  • Updating the SCX Agent causes deep monitoring status for the JEE Application Server to be reset.
  • By default, the Rpcimap monitor for Red Hat Enterprise Linux 5 is now disabled.
  • Monitoring in UTC +13 causes Unix Process Monitoring template to fail.
  • Using scxadmin -log-rotate causes logging to stop after log rotation.

As Always please first test the update rollups and supporting management packs and any newly released management packs first in a test environment!

Bob Cornelissen

New KB - Alerts may not get forwarded as expected via a connector in Operations Manager

SCOM, System Center, SCOM Tricks, SCOM 2012 Send feedback »

New KB

Alerts may not get forwarded as expected via a connector in Operations Manager

When using connectors to forward alerts in System Center Operations Manager 2007 (OpsMgr 2007) or System Center 2012 Operations Manager (OpsMgr 2012, SP1 and OpsMgr 2012 R2), in certain situations such as an alert storm (defined as a large number of alerts being generated in a very short period of time) there may be alerts that are not forwarded via a connector. When this occurs, these alerts will never be forwarded and will remain in a "New" state.

Cause and potential fix can be found in the KB article here:
http://support2.microsoft.com/kb/2277825/en-us

Good luck :p

WMUG NL first webinar of the year. Monitoring SQL with SCOM and Operational Insights

SCOM, SQL, System Center, SCOM 2012, Windows 2012 Send feedback »

Want to know about Monitoring SQL with SCOM and Operational Insights?
Then you have to check in with this webinar hosted by the WMUG NL user group and presented by a fellow MVP and friend Simon Skinner. Save your calendars for 28 January 2015 at 20:00 CET (GMT+1). For more information and registering for this webinar please follow this link and do not be shy to spread the word:

http://wmug.nl/2015/01/14/wmug-nl-webinar-1-2015-deep-dive-into-monitoring-sql-server-with-system-center-and-operational-insights-hosted-by-simon-skinner/

All are welcome to join!
Bob Cornelissen

System Center Orchestrator 2012 license error after moving database

System Center, SCORCH 2012 Send feedback »

Last week we got a fresh database server at a customer site to put the SCOM and Orchestrator databases on. Actually two servers in ALways-on setup. Should be a lot better and faster than the "temporary" servers we had been using before.
So, last week we started with the Orchestrator database simply because it is smaller and because there are less people looking in that console.

We had a few lessons learned and one of them is listed as the subject of this post. But I will go through it in a few steps. Don't try to follow the steps exactly as described, because I am listing them in the order it happened... So lessons learned are in between!

First of all I used the following page on TechNet:
How to Change the Orchestrator Database
http://technet.microsoft.com/en-us/library/hh473578.aspx

This page talks about what to do when moving the database and it looked really easy.

So made a backup of the orchestrator database and restored it on one of the two new nodes. Looked nice. So next was adding it to the always on availability group. There is a wizard for that so clicked through that thing. Needed to specify a share that both servers could reach.
Failed.

After investigation it turns out the service account which SQL db engine is running should have access to the share from both machines, not my account. Lesson learned.

Run through the wizard again to put it in availability group. What is does is to create backups of database and transaction logs and moving it to the second server and restoring them into there and getting it in sync. It will place the database in the availability group.
Failed.

How is this possible? the backups seemed to have worked. However the second database was in a retoring mode somehow. It looked like the synchronization did not start. We had our DBA check this with us and we found that it should be connection related. So we checked the firewall settings and it turned out that the Endpoint ports used for syncronization in the cluster were not listed here. We added the firewall ports and checked the one for SQL 1433 was there as well. Lesson learned.

We turned back and removed the second copy and the entry in the Availability group and we tried it again. This time it worked.
By the way if you do it manually to copy the database to be made highly available to the secondary server, do not forget to use both the full backups of the database and ALSO create a transaction log backup and roll that one into the second one too. Else it will not work.

Alright, now the database was running and highly available. We tried a failover to the other node and that worked.
Meanwhile we had Orchestrator itself stopped.

So ran the steps in the technet article and started the services again. Started the Runbook Designer.
Error.

The following is this error:

The license for System Center 2012 Orchestrator has expired or is invalid. Enter Product Key.

It asks me to enter the Orchestrator license. Huh? Actually when doing that and pressing Enter it also did not want to continue.
This was very irritating. In the end I think we managed to cancel through it and get to the runbooks, most of which would not start. Or would start and within 5 seconds stop again.
:no:

Checking for the errors in the runbook designer it turns out that the connections could not be made to for instance the SCOM server.

We investigated and finally I found this article on the TechNet site:
Migrate Orchestrator Between Environments
http://technet.microsoft.com/en-us/library/hh913929.aspx

Among the very first things I saw was the first step involved in bold:
Back up SQL Server service master key in environment A

Now at first I did not know what this was or what it does, but I am pretty sure this is what our problem relates to! :roll:

It did irritate me that those TechNet pages did not reference each other though! Also as you will see they do not take into account what happens when you have multiple target servers such as when using Always On functionality. :!:

First thing I did was think that it was too late already to get that thing moved over. So I went ahead and created a fresh key. Next thing to do was enter the encrypted info again. So go find the connection settings for connecting to SCOM and e-mail and SCCM and so on and re-enter passwords. I opened up all runbooks and re-selected those connections to be sure. Also the System Center license key.

Restart the Orchestrator management service and we were back up and running.

However, hold on! 88|

A little later this went wrong again and it was due to a manual fail over of the availability group the database was living in. And again we had runbooks failing and again the enter your license key error!
Turns out I missed something important there!

Alright this is how it works:
- the data you enter like passwords and so is stored encrypted in the Orchestrator database.
- a key from the database is used for this.
- however this key can only be read or decrypted using the SQL Server Service Master key!

Yeah and this Master key is not inside the datasbase but is stored in the SQL system tables. Meaning it is related to the Instance and not the database. Now in a normal failover cluster we would not have run into the second time to see the Orchestrator go nuts on this. However with this always on cluster there are two machines with their own SQL instance. And the databases are made in sync. But there are two instances!
If the Server Master key is not the same (which it isn't) the data can not be descrypted with failover. So I should have used the key from the original server in the first place or just have replaced it after seeing the error. I did not know it could still be done afterwards. And also did not know that in this case

SO what to do:
Open up SQL management studio and run this command to check the master key:

USE master
SELECT * FROM sys.symmetric_keys

(by the way you see here that its not database related by instance related because it is asking the master database for this info and of course its taking it from the system tables).
You will see the servicemasterkey and the Guid belonging to that key.
And of course this was different between the two servers and would have been different from the original database server as well.

So to fix it we need to create a backup of the first key and restore it onto the second server.
First create a directory on the first SQL server to store the backup of the key in. Next run a backup command to backup the key. I have changed the password a bit in this post.

BACKUP SERVICE MASTER KEY
TO FILE = 'C:\backup\service_master_key'
ENCRYPTION BY PASSWORD = '3dH85Hhk004GHk2597ghefj5';

Next I created the same directory on the second server and copied the key bakcup file to it. Open a SQL query on the second server and import the key:

RESTORE SERVICE MASTER KEY
FROM FILE = 'C:\backup\service_master_key'
DECRYPTION BY PASSWORD = '3dH85Hhk004GHk2597ghefj5' FORCE
GO

By the way this last command uses the FORCE option because it will find a database which is already using encrypted data, so we need to force it to use this master key. Make sure you take the key from the working machine and put it on the not working machine though!

Now we check again for the master key and see that they are the same. I restarted SQL Server Service.
Restarted orchestrator services on the orchestrator management server and runbook servers.
Worked! :D

So there were a few lessons learned. Sure wish I had read the second article earlier as it could have prevented part of this.

A few of these lessons learned came back when we did the SCOM operational database, but that is a story for next time.

Good luck!
Bob Cornelissen

Im back and blog errors

Active Directory Send feedback »

Hi all, I am back from my trip to the states for the MVP summit and the MMS. It was great to see everybody again and hear a lot of stuff. Totally worth the trip.

Last few days I had some issues with this blog. Basically it is running a PHP type blogging software with a MYSQL backend. Something went wrong with the MYSQL and it wasnt easy to fix. Somehow this is working again, but I do not trust it enough. This means I will be transferring the whole blog to a freshly installed server and trying to move the software and databases over, so everything remains the same. Probably with higher versions of both software parts and of course also higher Windows version and so on. If it is down for a number of hours or even a day it will probably be because im moving it at that time.

There was somebody who sent me an email about a page he was trying to get while it was down. Thats great! Feel free to do so if that happens.

SCOM-Bob

Looking forward to the Midwest Management Summit

SCOM, DPM, Hyper-V, SCVMM, Service Manager, SCCM, SCE, System Center, SCOM Tricks, SCOM 2012, Configuration Manager, SCORCH 2012 Send feedback »

Since Microsoft stopped their Microsoft Management Summit and integrated it into their TechEd it still felt like a void was left there in the System Center space. Of course the MMS in the old days started as a community thing and in the end it ended up as a Microsoft conference with some community input. And of course now it was gone. :-/

Well, the community decided to step back in! B) :D :p :) :)) :>> (yeah i better not include too many smileys here).

User groups and MVP's and other community leaders and experts are jumping in and now the Midwest Management Summit (so also MMS in short) has been born. It is filled with great speakers, many from the community and MVP sides from all over the world. It is gearing up to be a great event and specialized into the field of System Center in connection also with PowerShell, Clouds and so on.

Have a look at this great lineup of sessions in the schedule: http://mms2014.sched.org/

Next I can inform you that I also will be speaking there in two sessions along with my long time friend, SCOM specialist and MVP Cameron Fuller

Check out our sessions here: http://mms2014.sched.org/speaker/bob.cornelissen

MMS2014SPeaker

If you have not registered yet for this event, give it a second thought and go to: http://mms.mnscug.org/ and book your ticket http://mms2014.sched.org/tickets :yes:

I hope to see you there. If you are attending the MMS, please feel free to catch me there and have a chat! That is also what these community based events are all about. I am very much looking forward to meeting as many of you as possible during the upcoming events in November. I will be attending the MVP Summit and this MMS of course!

Bob Cornelissen

SCOM 31552 Data Warehouse configuration synchronization process failed to write data

SCOM, System Center, SCOM Tricks, SCOM 2012 Send feedback »

Today I investigated a case where SCOM had an alert with the following name and contents:

Data Warehouse configuration synchronization process failed to write data

Data Warehouse configuration synchronization process failed to write data to the Data Warehouse database. Kan geen gegevens in de datawarehouse opslaan.
Uitzondering SqlException: Sql execution failed. Error 2627, Level 14, State 1, Procedure ManagementPackInstall, Line 2879, Message: Violation of UNIQUE KEY constraint 'UN_ManagementGroupManagementPackVersion_ManagementGroupRowIdManagementPackVersionRowId'. Cannot insert duplicate key in object 'dbo.ManagementGroupManagementPackVersion'. The duplicate key value is (1, 2020, Jun 18 2014 3:15PM).

There are event log entries in the Operations Manager event log with ID 31552 with the same kind of contents.

Now I want to give a big shout out to the guys in the SCOM Support team, who are writing Support Tip entries on their blog for common issues and solutions. I found their Support Tip quickly and the contents is very clear on how it probably happened and what not to do next time and how to solve the issue at hand.

Support Tip: Data Warehouse synchronization failures following restore of the OperationsManager DB

And yes in my case we did decide during some problems earlier not to restore the DW database because it was huge and so on :> And yes that probably was the cause. Was during an upgrade of SCOM and the upgrade wizard failed and killed the management server and touched the operational database so decideed to restore the OpsDb because well they wouldn't have touched the datawarehouse db yet right? Wrong, they do :roll: So lesson learned for sure, when restoring one database just restore the other one to the same point in time.

So ran the SQL script provided and pasted in the correct key value pair into the script and ran it. Sure enough it was the Notifications Internal Library. Exported the pack, increased the version number. Imported the pack. And few minutes later the 31554 event popped up in the event log.

Thanks again to the SCOM Engineering Blog and the escalation engineers behind it for publishing these kind of support tips.

Bob Cornelissen

Microsoft Ignite

SCOM, SQL, Hyper-V, Exchange, System Center, Active Directory, Windows 2012 Send feedback »

Just now Microsoft announced the wave of main events of next year. The expected event which brings together several tech events like TechEd and Management Summit and Exchange/Lync/Sharepoint/Project conferences into one big event. Well here it is and it is called Microsoft Ignite. Scheduled May 4 to May 8 in Chicago. Read more about it on this page:

http://blogs.microsoft.com/blog/2014/10/16/introducing-microsoft-ignite-lineup-top-conferences-2015/

This page also lists some of the other conferences in the year like Convergence, Build and WPC.

Enjoy!
Bob Cornelissen

Contact / Help. ©2015 by Bob Cornelissen. blog software.
Design & icons by N.Design Studio. Skin by Tender Feelings / Evo Factory.