How to monitor e-mail data sources with SCOM and Orchestrator

SCOM, System Center, SCOM Tricks, SCOM 2012, SCORCH 2012 Send feedback »

While chatting with some MVP friends of mine about a specific scenario where data from e-mails needed to be read and monitored, there are multiple possibilities to do it. I proposed one possibility which I implemented at a customer a while ago and got asked to blog about the solution, so here it is. Because SCOM is not built to natively read from a mailbox, one has to come up with a workaround, and in my case I used System Center Orchestrator to do part of the job.

Challenge:

Following is the situation. A number of servers monitored by another company and using another monitoring product. That product monitors servers from several customers of theirs, so we can not directly access it. We could not access or query the product directly either through scripts or commands or database queries. So in the end the result was that the other company would send e-mails from their several monitoring systems to one of our mailboxes. Resulting in 3 e-mails every 15 minutes. The e-mails contained an XML formatted body containing a list of servers and their state.

  1. So, we have to read 3 e-mails from a mailbox every 15 minutes. Pull out the body of the e-mails. Next merge the content to make it 1 XML file placed on a server with a SCOM agent on it. These steps are not native to SCOM, but a combination or Orchestrator and PowerShell
  2. After that we can use one of several methods to monitor a text based file on a server to create the monitoring part. For this we can use SCOM.

SO let us start with the first part

Using Orchestrator to get our e-mails into an XML file

I bet there are also other methods of doing this, but this was the method I selected and due to Orchestrator having some flexibility and some built-in actions in the intelligence packs this is very versatile.

Let us check out the email for a second:

We see the XML body there. In this case there are two servers mentioned in the email, however with longer names than how we know them so we need to play around with that too. Also with XML there is a header (first line) and a wrapper (second line start and end of last line), with the two actual content lines in the middle of it. Notice there are carriage returns and also spaces and potential tabs in there, which make it “nice” to filter those out while pulling the XML apart and creating a new XML file from that!

Ingredients needed:

  • A destination File share where the final XML file will be placed for being monitored.
  • A mailbox where those messages arrive and we can read them from
  • We created an automatic rule to place those e-mails in a specific named folder in the mailbox.
  • We created a second folder where we can move the already read messages to.
  • An account able to read in that mailbox.
  • Orchestrator to create a runbook and bring it all together.
  • An intelligence pack for Orchestrator which can read from a mailbox. I used the “SCORCH Dev - Exchange Email” IP for this which can be found at https://scorch.codeplex.com/

First import the Orchestrator IP needed to read the email and distribute it to the runbook servers as usual. Next start a fresh runbook and name it appropriately and place it in a folder where you can actually find it within Orchestrator. Advice is to use a clear folder structure within Orchestrator to place your runbooks in. This is not for the benefit of Orchestrator, but for yours!

Now we create the runbook. I will put the picture of the finished runbook here first before going through the activities:

Let’s now cut up the pieces:

Monitor Date/Time

Well this one simply says to check every 15 minutes

Format Date/Time

This one takes the current time from the first activity and at the bottom there subtracts 15 minutes from it. The story behind this is that we want to read all emails which came in between now and 15 minutes ago. So this gives us that point in time.

Rename File

We wanted our monitored xml file to always have a fixed name. So when we are about to create a new version of that file we first go out to that file share and take the current XML file and rename it by adding a date-time format in the name to make it unique. We wanted to be able to look back in history here, else we would have chosen to just delete it. This makes the folder look like this:

Read mail from folder

Now this is a custom activity coming from the Exchange Email IP we imported earlier.
From the top we see we have to define a configuration. We will get back to that in a second. Next you can see that we are looking for Unread emails in a certain folder (keep in mind folder name must be unique in that mailbox else it just takes the other one, which you did not want to). Now on the left hand side we see Filters:

We also want those emails to have a certain subject line. And we want those emails to be received after the time from the Format Date/Time activity above. Meaning the email was received after 15 minutes ago. So in the last 15 minutes.

Now to get back to the Configuration part. Many IP’s in Orchestrator have a place where you can centrally set some parameters. For instance a login account, a server connection, and so on. This can be found on the top menu bar of the Orchestrator Runbook Designer under the Options menu. Find the item with the same name as the IP you are trying to configure. In this case it needs us to setup a connection to an email server. Type is Exchange Server, type a username, password, domain, and a ServiceURL. For an exchange server this could be https://webmail.domain.com/EWS/Exchange.asmx for example, but check this for your own environment.

Retry Read mail from folder

This one will only run if the first read mail from folder activity fails. You can set properties on those connecting arrows between the activities to make it go here it the first one fails. I made the line color red and set a delay on the line of 20 seconds. Else it will follow the other line and go to the script. This activity does exactly the same as the previous one. We had some time-outs during certain times so this extra loop slipped in there.

So those Read mail from folder activities should contain 3 e-mails received in the last 15 minutes from that folder, unread, with a subject line, and Orchestrator now knows what the body of those emails contains. This also means that the next activity (the script) will run three times.

Run .net script

At the top we define this to be a PowerShell script. So first we pull in the variable, which is the body of the email from the previous step. Next thing we do in the script is remove all excess stuff that we do not need. Empty spaces before and after several lines and entries. Also we will take out those headers and surrounding entries. We can add them ourselves to a clean file, right? SO this should give us a new string which only contains the XML entries for those servers with their state.

Next thing we needed to do is build in some tricks into this script. We know it is going to run three times and we need to stitch the contents together into one file.

Line of thought:

If there is no xml file there to write to this means this is the first time we run the script after the old file got renamed. So we need to create the xml file right now and add the headers to it. Next we add the body to it (server names with state).

If there is a file there with the correct name it means we are either in the second or third run. So what we do is simply write down the body (servers and state) and add the trailing end tag to it. This can be done on the second and third run. However, if this happens to be the third run, we will first check if that trailing tag is there and remove it. And next dump the body again and add the end tag.

So that part takes care of dumping the contents into the file following the above thought process (with the first thought coming at the end as the Else statement). Sorry for the Dutch comments, but you get the idea.

Move mail

Next we take the e-mails found by the Read mail from folder activity and move them to the other folder in the mailbox.

So, that is the whole runbook to get a few emails and merge them together so we can monitor the thing!
There is a separate runbook which cleans old files from that file share and which cleans old emails from that folder in the mailbox by the way. At least we can look a few days back what happened.

The monitoring part in SCOM

Now I am not going into all the details of this part. I had a reason to not link these entries directly to the monitored servers, or to write the xml file to those servers. I opted to create a watcher node (and its discovery from a registry entry on that machine). That watcher node is the server with that file share and the xml file on it.

Next I created watchers in a class, and discovered them through registry as well. Containing the names of the servers we wanted to check for in the XML.

For each watcher it runs a PowerShell monitor which goes into the XML file and finds its corresponding entry (server name). Next it picks up the State (which is a number) and we translate the 12 possible numbers into green/yellow/red type entries and place them into the property bag. That gets evaluated into the three states we know so well.

Next we could throw those watcher entries for each server and also some other entries onto a dashboard. We could see the state the other party saw from their monitoring system and the state we see from SCOM side on one dashboard for those servers and monitored entries. We have the hardware/OS layer with a few extras, and they have an OS layer and application layers which we could not pick up.

Conclusion

As you can see sometimes we run into situations where there is no other way to get monitoring data than through workarounds and the long way. This is not ideal. As you can understand there is dependencies left and right for this whole chain to work. If there is no other way then that is the way it has to be. Direct monitoring or direct connecting is preferred.
But this shows how you can get monitoring data from e-mails into SCOM, in this case through the use of Orchestrator and watchers because that was what we needed.

Shout-out to amongst others Cameron Fuller for making me write this post!
Happy monitoring!
Bob Cornelissen

Hold off on the SCOM Base OS pack version 6.0.7303.0

SCOM, System Center, SCOM 2012, Windows 2012 Send feedback »

About a bit ago the Base OS pack has been updated again to a version 6.0.7303.0
It is not advised to upgrade your SCOM packs to this version yet, because it has some bugs in it.

Kevin Holman writes about it here:
http://blogs.technet.com/b/kevinholman/archive/2016/02/24/base-os-mp-s-have-been-updated-version-6-0-7303-0.aspx

So would hold off until a newer version comes out (hopefully soon) and please check it in your test environment before you let it hit production. A lot of other packs have references to Base-OS packs so it is hard to remove it when something goes wrong.

Good luck!
Bob Cornelissen

System Center DPM 2012 R2 UR9

DPM, System Center Send feedback »

If you are running DPM 2012 R2 you will want to look at the UR9 update rollup which was released a month ago. They are classing it now as an important update. Includes a whole list of improvements.

Check out the blog post from the DPM team here for more information:
https://blogs.technet.microsoft.com/dpm/2016/02/25/backup-of-enterprise-data-made-easier-with-system-center-data-protection-manager/

Have fun backing up your stuff!
Bob Cornelissen

Bridgeways releases newer version of VMWare MP for SCOM

SCOM, System Center, SCOM Tricks, SCOM 2012 Send feedback »

Bridgeways has been working very hard this last year in coming back into their speed for creating management packs, updating existing ones, making them more intuitive and useful to work with, adding support staff and also hiring a very competent CTO (a good friend of mine and fellow MVP Simon Skinner :p ) who takes management pack quality very seriously. It is exciting to see the progress being made.

Today the new version of their VMWare monitoring management pack was released. It contains updated views and dashboards. The new style dashboards also were augmented with a few core dashboards:

  • Host Performance: includes CPU Usage, Memory Usage, Swamp Memory Usage, Balloon Usage, Network Usage and Storage Usage data.
  • CPU Performance: includes CPU Usage, Average CPU System Time, Average CPU Ready and CPU Wait data.
  • Memory Performance: includes Active Memory, Balloon Memory, Shared Memory and Swapped Memory data.

Also there is an expanded set of reports. The combination gives better and quicker insights into the VMWare environment in order to troubleshoot issues and to proactively find upcoming issues.

Now as some of you probably know I have always been a big fan of a competing vendors management pack for VMWare (Veeam) and I still am. However it is good that a few other vendors have been looking seriously at creating a good management pack which can cover this monitoring scenario. I know a few have been working on this and are becoming serious contenders when it comes to product selection for this purpose. The Bridgeways pack will be one of them now in every selection process. Especially when there is a price difference to be found between solutions our customers (and you) will be looking at price/quality points and if it covers that what you are initially looking for. I know all my customers always look at price very closely and I do not think its a pure Dutch thing to do that.
I will be examining functionality and pro's and con's of these packs closely in the future.

So for now congratulations to Bridgeways for taking the step forward and we are watching you closely B)

For any questions regarding this article or any management packs feel free to contact me.
Bob Cornelissen

UR9 for System Center 2012 R2 available

DPM, SCVMM, Service Manager, System Center, SCOM 2012, SCORCH 2012 Send feedback »

We noticed the release of UR9 for Service Manager a day before and WSUS also already had the SCOM agent updates, but today the KB article is released giving the overview of the UR9 for System Center 2012 R2 products.

The KB article for System Center

The products receiving the update are:

Do not forget to test first and to read the KB articles of whatever you are upgrading.
Have fun!
Bob Cornelissen

Citrix aquiring Comtrade management packs for Citrix products

SCOM, System Center, SCOM 2012 Send feedback »

LJUBLJANA, January 11, 2016 – Comtrade System Software & Tools (Comtrade), a leading provider of IT infrastructure and application management and monitoring solutions, today announced that its Citrix management packs were acquired by Citrix, a strategic, long-term partner of Comtrade.

Read more through this link:

http://www.comtradeproducts.com/news/citrix-acquires-comtrades-scom-management-packs-ip/

Orchestrator 2012 web console not working

System Center, SCORCH 2012 Send feedback »

We have had an issue with the Orchestrator web console not working at a customer site for a while. Since this feature was not used it got pushed back to lower priority. However for instance the Infront management pack for Orchestrator did not work anymore due to their connection to the web service and we could not follow the status of runbooks anymore. It bugged me today again so went into fixing it. I used some websites/blogposts/KBarticles/forums to get all the items together and will post links at the bottom. The following steps were needed to get it up and running again:

There are a number of known issues for instance after upgrading Orchestrator from a previous version. Lets start with the first:

Database

The Orchestration Console cannot be opened after upgrading to System Center 2012 R2

Error Executing the current operation
[httpwebrequest_webexception_remoteserver]
Arguments: NotFound

The SQL statement to run is included in the KB article above.

Next thing to check while you are there right now is in SQL mgmt Studio click to the database - security and find the Orchestrator service account used to connect to the database. Make sure it is assigned the DB_owner role.

Web Services

On the Orchestrator web server go into the IIS manager. You will two websites relating to Orchestrator. Click each website and next select the "Basic Settings" button. Make sure the websites are not using the default applicationpool but the "System Center 2012 Orchestrator Web Features". In my case this was set to default applicationpool as well, so had to change it.

Another step you may want to do (and it is part of what got it fixed for me!) is in IIS Manager go to that application pool "System Center 2012 Orchestrator Web Features" and click "Advanced Settings". Find the Process Model -> Identity. This should be the service account from Orchestrator. Please go into that by selecting the ... button in that line and re-enter the username and password.

Next step:
We need to check the web.config files.

How to Change the Orchestrator Database
I had to use the second part of that post to decrypt on config file and change it to use the correct database server and instance and database (we have changed the SQL server at some point to be places on an Always-On cluster) and encrypt it again. Keep in mind the syntax of the two commands. The "connectionStrings" part is case-sensitive so wite it exactly like that! Also the path to the web.config file may need adjustment. For instance in our case the folder was "C:\Program Files (x86)\Microsoft System Center 2012 R2\Orchestrator\Web Service\Orchestrator2012" You will see when it works the way as intended.

Next one to check out while you are at it:
C:\Program Files (x86)\Microsoft System Center 2012 R2\Orchestrator\Orchestration Console\web.config

In that file I took the line:

Code

<add key="ScoServiceUri" value="http://SERVERNAME1:81/Orchestrator2012/Orchestrator.svc"/>

And made a copy of it to the line below it. Next add the FQDN to it so it reacts also when using full server names:

Code

<add key="ScoServiceUri" value="http://servername.domain.name:81/Orchestrator2012/Orchestrator.svc"/>

And save the file.

After any of these changes make sure you restart IIS completely.

Other stuff:

Somebody had a problem with Locale localization of the server:
Orchestration console fails to start with [Arg_TargetInvocationException] error

Hope it helps somebody who runs into Orchestrator web console problems.

Throughout these fixes I have used posts from these links:

http://blogs.technet.com/b/canitpro/archive/2013/03/26/system-center-2012-orchestrator-orchestration-fix-the-web-console-login-issue-after-upgrading-to-sp1.aspx
Thanks Brad!

http://windowsitpro.com/orchestrator/solve-orchestrator-web-problems

https://support.microsoft.com/en-us/kb/2919041#/en-us/kb/2919041

https://social.technet.microsoft.com/Forums/en-US/c3afc430-5811-4c8c-a797-79c98459d792/console-fails-to-open-error-executing-the-current-operation?forum=scogeneral

https://technet.microsoft.com/en-us/library/hh473578.aspx

https://social.technet.microsoft.com/Forums/en-US/64dc86b3-fa3b-413c-a5cd-9d8a19cd76a4/orchestration-console-fails-to-start-with-argtargetinvocationexception-error?forum=scodeployment

Good luck to all!
Bob Cornelissen

Jalasoft Device Simulator upgraded to version 5.5

SCOM, System Center, SCOM Tricks, SCOM 2012, Windows 2012 Send feedback »

I have written and talked about Jalasofts great tool called Device Simulator. Many people in the monitoring space have been using this tool in previous versions. Today they released a fresh version of this tool to the 5.5 version.

The main additions to this version are:

  • An OID editor meant to customize the behavior of certain OID's within the simulation file of your to be simulated device.
  • The SNMP Dump Manager to more easily work with al the simulation files you are using
  • The number of simulated devices out of the box has been expanded to over 100!

This tool can be used for free provided you only simulate a few devices at a time. And for a few dollars an expanded license can be bought to do a lot more testing with this.

For demo and test environments this can be used a lot as a primairy tool to simulate network devices without needing to have all of that infrastructure in production. Also you could take a recording of an existing device in your production and even including real time data and play it back through the simulator.

Everybody needs a copy of this piece of software!

Get the Jalasoft Device Simulator over here.

Bob Cornelissen

Spend a minute for animal charity and your effort will be doubled

SCOM, Exchange, Windows 7, System Center, Active Directory, Windows 2012, Charity Send feedback »

As you may be aware I have been very involved with an animal charity. It is called House of Tails (www.houseoftails.org ) and is about taking care of abandoned dogs in Thailand. This is also the reason why the volume of my tech blog posts went down for a while. I will however pick up the pace again, I promise! Lots of ideas here. Because the dog sanctuary has a very hard time to make things happen this has taken a lot of time. We have setup an official Foundation (Stichting in Dutch language) with bank and email and PayPal and so on and we are working on a fresh website.

Yesterday a dear friend Pete Zerger wrote a blog posting about our story and he has taken it upon himself (and his friends and family) to bring a challenge. Any donation given to our House of Tails charity within a 72 hour period (starting last evening, so for the coming 52 hours or so still) will be DOUBLED. So any donation you make will get a double effect for the animal charity and all of it will go to the wellbeing of these dogs!

Please read how you can help too:
http://insidethecloudos.azurewebsites.net/a-story-about-a-woman-a-man-and-their-dogs/

Also I will include a link here to the PayPal/Creditcard possibility of the charity Stichting House of Tails.




Take a minute to check it out through that post and through the House of Tails website.

Seeing how my geeky family and my online families are willing to help out a good cause is truely heart warming. It means soo much for us and for the dogs!

a truely gratefull Bob Cornelissen

SCOM 2012 R2 Web Console Configuration Required updated

SCOM, System Center, SCOM Tricks, SCOM 2012 Send feedback »

This is an update to the page describing a solution to the "Web Console Configuration Required" message for SCOM 2012 and SCOM 2012 R2.

If you get this message on your workstations and you do not have the rights to simply run the configuration wizard it asks you to run, it has to be configured through another way.
What the configuration wizard does is set a few registry keys and add a code signing certificate (can also be done through registry). For instance a customer of mine this needs to be entered into a VDI image to make it work for most people.

At SCOM 2012 R2 UR3 a new code signing certificate was introduced which was valid until 22 July 2015. ANd now at SCOM 2012 R2 UR7 a new code signing certificate is introduced. It is now valid until somewhere in September 2016. Bit unfortunate there was a lack of overlap there.

The article I made last year SCOM 2012 Web Console Configuration Required describes the fix and how to find the registry keys. AT the bottom of the article there are updates for a registry key we found as an extra potential blocker of access to the SCOM web Console and there is another update created today for UR7. It has the registry files you need for either UR3/4/5/6 or for the UR7 now.

Good luck and enjoy!
Bob Cornelissen

Contact / Help. ©2017 by Bob Cornelissen. blog software.
Design & icons by N.Design Studio. Skin by Tender Feelings / Evo Factory.