Category: "SCORCH 2012"
While chatting with some MVP friends of mine about a specific scenario where data from e-mails needed to be read and monitored, there are multiple possibilities to do it. I proposed one possibility which I implemented at a customer a while ago and got asked to blog about the solution, so here it is. Because SCOM is not built to natively read from a mailbox, one has to come up with a workaround, and in my case I used System Center Orchestrator to do part of the job.
Following is the situation. A number of servers monitored by another company and using another monitoring product. That product monitors servers from several customers of theirs, so we can not directly access it. We could not access or query the product directly either through scripts or commands or database queries. So in the end the result was that the other company would send e-mails from their several monitoring systems to one of our mailboxes. Resulting in 3 e-mails every 15 minutes. The e-mails contained an XML formatted body containing a list of servers and their state.
- So, we have to read 3 e-mails from a mailbox every 15 minutes. Pull out the body of the e-mails. Next merge the content to make it 1 XML file placed on a server with a SCOM agent on it. These steps are not native to SCOM, but a combination or Orchestrator and PowerShell
- After that we can use one of several methods to monitor a text based file on a server to create the monitoring part. For this we can use SCOM.
SO let us start with the first part
Using Orchestrator to get our e-mails into an XML file
I bet there are also other methods of doing this, but this was the method I selected and due to Orchestrator having some flexibility and some built-in actions in the intelligence packs this is very versatile.
Let us check out the email for a second:
We see the XML body there. In this case there are two servers mentioned in the email, however with longer names than how we know them so we need to play around with that too. Also with XML there is a header (first line) and a wrapper (second line start and end of last line), with the two actual content lines in the middle of it. Notice there are carriage returns and also spaces and potential tabs in there, which make it “nice” to filter those out while pulling the XML apart and creating a new XML file from that!
- A destination File share where the final XML file will be placed for being monitored.
- A mailbox where those messages arrive and we can read them from
- We created an automatic rule to place those e-mails in a specific named folder in the mailbox.
- We created a second folder where we can move the already read messages to.
- An account able to read in that mailbox.
- Orchestrator to create a runbook and bring it all together.
- An intelligence pack for Orchestrator which can read from a mailbox. I used the “SCORCH Dev - Exchange Email” IP for this which can be found at https://scorch.codeplex.com/
First import the Orchestrator IP needed to read the email and distribute it to the runbook servers as usual. Next start a fresh runbook and name it appropriately and place it in a folder where you can actually find it within Orchestrator. Advice is to use a clear folder structure within Orchestrator to place your runbooks in. This is not for the benefit of Orchestrator, but for yours!
Now we create the runbook. I will put the picture of the finished runbook here first before going through the activities:
Let’s now cut up the pieces:
Well this one simply says to check every 15 minutes
This one takes the current time from the first activity and at the bottom there subtracts 15 minutes from it. The story behind this is that we want to read all emails which came in between now and 15 minutes ago. So this gives us that point in time.
We wanted our monitored xml file to always have a fixed name. So when we are about to create a new version of that file we first go out to that file share and take the current XML file and rename it by adding a date-time format in the name to make it unique. We wanted to be able to look back in history here, else we would have chosen to just delete it. This makes the folder look like this:
Read mail from folder
Now this is a custom activity coming from the Exchange Email IP we imported earlier.
From the top we see we have to define a configuration. We will get back to that in a second. Next you can see that we are looking for Unread emails in a certain folder (keep in mind folder name must be unique in that mailbox else it just takes the other one, which you did not want to). Now on the left hand side we see Filters:
We also want those emails to have a certain subject line. And we want those emails to be received after the time from the Format Date/Time activity above. Meaning the email was received after 15 minutes ago. So in the last 15 minutes.
Now to get back to the Configuration part. Many IP’s in Orchestrator have a place where you can centrally set some parameters. For instance a login account, a server connection, and so on. This can be found on the top menu bar of the Orchestrator Runbook Designer under the Options menu. Find the item with the same name as the IP you are trying to configure. In this case it needs us to setup a connection to an email server. Type is Exchange Server, type a username, password, domain, and a ServiceURL. For an exchange server this could be https://webmail.domain.com/EWS/Exchange.asmx for example, but check this for your own environment.
Retry Read mail from folder
This one will only run if the first read mail from folder activity fails. You can set properties on those connecting arrows between the activities to make it go here it the first one fails. I made the line color red and set a delay on the line of 20 seconds. Else it will follow the other line and go to the script. This activity does exactly the same as the previous one. We had some time-outs during certain times so this extra loop slipped in there.
So those Read mail from folder activities should contain 3 e-mails received in the last 15 minutes from that folder, unread, with a subject line, and Orchestrator now knows what the body of those emails contains. This also means that the next activity (the script) will run three times.
Run .net script
At the top we define this to be a PowerShell script. So first we pull in the variable, which is the body of the email from the previous step. Next thing we do in the script is remove all excess stuff that we do not need. Empty spaces before and after several lines and entries. Also we will take out those headers and surrounding entries. We can add them ourselves to a clean file, right? SO this should give us a new string which only contains the XML entries for those servers with their state.
Next thing we needed to do is build in some tricks into this script. We know it is going to run three times and we need to stitch the contents together into one file.
Line of thought:
If there is no xml file there to write to this means this is the first time we run the script after the old file got renamed. So we need to create the xml file right now and add the headers to it. Next we add the body to it (server names with state).
If there is a file there with the correct name it means we are either in the second or third run. So what we do is simply write down the body (servers and state) and add the trailing end tag to it. This can be done on the second and third run. However, if this happens to be the third run, we will first check if that trailing tag is there and remove it. And next dump the body again and add the end tag.
So that part takes care of dumping the contents into the file following the above thought process (with the first thought coming at the end as the Else statement). Sorry for the Dutch comments, but you get the idea.
Next we take the e-mails found by the Read mail from folder activity and move them to the other folder in the mailbox.
So, that is the whole runbook to get a few emails and merge them together so we can monitor the thing!
There is a separate runbook which cleans old files from that file share and which cleans old emails from that folder in the mailbox by the way. At least we can look a few days back what happened.
The monitoring part in SCOM
Now I am not going into all the details of this part. I had a reason to not link these entries directly to the monitored servers, or to write the xml file to those servers. I opted to create a watcher node (and its discovery from a registry entry on that machine). That watcher node is the server with that file share and the xml file on it.
Next I created watchers in a class, and discovered them through registry as well. Containing the names of the servers we wanted to check for in the XML.
For each watcher it runs a PowerShell monitor which goes into the XML file and finds its corresponding entry (server name). Next it picks up the State (which is a number) and we translate the 12 possible numbers into green/yellow/red type entries and place them into the property bag. That gets evaluated into the three states we know so well.
Next we could throw those watcher entries for each server and also some other entries onto a dashboard. We could see the state the other party saw from their monitoring system and the state we see from SCOM side on one dashboard for those servers and monitored entries. We have the hardware/OS layer with a few extras, and they have an OS layer and application layers which we could not pick up.
As you can see sometimes we run into situations where there is no other way to get monitoring data than through workarounds and the long way. This is not ideal. As you can understand there is dependencies left and right for this whole chain to work. If there is no other way then that is the way it has to be. Direct monitoring or direct connecting is preferred.
But this shows how you can get monitoring data from e-mails into SCOM, in this case through the use of Orchestrator and watchers because that was what we needed.
Shout-out to amongst others Cameron Fuller for making me write this post!
We noticed the release of UR9 for Service Manager a day before and WSUS also already had the SCOM agent updates, but today the KB article is released giving the overview of the UR9 for System Center 2012 R2 products.
The products receiving the update are:
- DPM 2012 R2 UR9 Data Protection Manager
- SCOM 2012 R2 UR9 Operations Manager
- SCSM 2012 R2 UR9 Service Manager
- SPF Service Provider Foundation
- VMM 2012 R2 UR9 Virtual Machine Manager
- WAP Windows Azure Pack
Do not forget to test first and to read the KB articles of whatever you are upgrading.
We have had an issue with the Orchestrator web console not working at a customer site for a while. Since this feature was not used it got pushed back to lower priority. However for instance the Infront management pack for Orchestrator did not work anymore due to their connection to the web service and we could not follow the status of runbooks anymore. It bugged me today again so went into fixing it. I used some websites/blogposts/KBarticles/forums to get all the items together and will post links at the bottom. The following steps were needed to get it up and running again:
There are a number of known issues for instance after upgrading Orchestrator from a previous version. Lets start with the first:
Error Executing the current operation
The SQL statement to run is included in the KB article above.
Next thing to check while you are there right now is in SQL mgmt Studio click to the database - security and find the Orchestrator service account used to connect to the database. Make sure it is assigned the DB_owner role.
On the Orchestrator web server go into the IIS manager. You will two websites relating to Orchestrator. Click each website and next select the "Basic Settings" button. Make sure the websites are not using the default applicationpool but the "System Center 2012 Orchestrator Web Features". In my case this was set to default applicationpool as well, so had to change it.
Another step you may want to do (and it is part of what got it fixed for me!) is in IIS Manager go to that application pool "System Center 2012 Orchestrator Web Features" and click "Advanced Settings". Find the Process Model -> Identity. This should be the service account from Orchestrator. Please go into that by selecting the ... button in that line and re-enter the username and password.
We need to check the web.config files.
How to Change the Orchestrator Database
I had to use the second part of that post to decrypt on config file and change it to use the correct database server and instance and database (we have changed the SQL server at some point to be places on an Always-On cluster) and encrypt it again. Keep in mind the syntax of the two commands. The "connectionStrings" part is case-sensitive so wite it exactly like that! Also the path to the web.config file may need adjustment. For instance in our case the folder was "C:\Program Files (x86)\Microsoft System Center 2012 R2\Orchestrator\Web Service\Orchestrator2012" You will see when it works the way as intended.
Next one to check out while you are at it:
C:\Program Files (x86)\Microsoft System Center 2012 R2\Orchestrator\Orchestration Console\web.config
In that file I took the line:
And made a copy of it to the line below it. Next add the FQDN to it so it reacts also when using full server names:
And save the file.
After any of these changes make sure you restart IIS completely.
Somebody had a problem with Locale localization of the server:
Orchestration console fails to start with [Arg_TargetInvocationException] error
Hope it helps somebody who runs into Orchestrator web console problems.
Throughout these fixes I have used posts from these links:
Good luck to all!
Last week we got a fresh database server at a customer site to put the SCOM and Orchestrator databases on. Actually two servers in ALways-on setup. Should be a lot better and faster than the "temporary" servers we had been using before.
So, last week we started with the Orchestrator database simply because it is smaller and because there are less people looking in that console.
We had a few lessons learned and one of them is listed as the subject of this post. But I will go through it in a few steps. Don't try to follow the steps exactly as described, because I am listing them in the order it happened... So lessons learned are in between!
First of all I used the following page on TechNet:
How to Change the Orchestrator Database
This page talks about what to do when moving the database and it looked really easy.
So made a backup of the orchestrator database and restored it on one of the two new nodes. Looked nice. So next was adding it to the always on availability group. There is a wizard for that so clicked through that thing. Needed to specify a share that both servers could reach.
After investigation it turns out the service account which SQL db engine is running should have access to the share from both machines, not my account. Lesson learned.
Run through the wizard again to put it in availability group. What is does is to create backups of database and transaction logs and moving it to the second server and restoring them into there and getting it in sync. It will place the database in the availability group.
How is this possible? the backups seemed to have worked. However the second database was in a retoring mode somehow. It looked like the synchronization did not start. We had our DBA check this with us and we found that it should be connection related. So we checked the firewall settings and it turned out that the Endpoint ports used for syncronization in the cluster were not listed here. We added the firewall ports and checked the one for SQL 1433 was there as well. Lesson learned.
We turned back and removed the second copy and the entry in the Availability group and we tried it again. This time it worked.
By the way if you do it manually to copy the database to be made highly available to the secondary server, do not forget to use both the full backups of the database and ALSO create a transaction log backup and roll that one into the second one too. Else it will not work.
Alright, now the database was running and highly available. We tried a failover to the other node and that worked.
Meanwhile we had Orchestrator itself stopped.
So ran the steps in the technet article and started the services again. Started the Runbook Designer.
The following is this error:
The license for System Center 2012 Orchestrator has expired or is invalid. Enter Product Key.
It asks me to enter the Orchestrator license. Huh? Actually when doing that and pressing Enter it also did not want to continue.
This was very irritating. In the end I think we managed to cancel through it and get to the runbooks, most of which would not start. Or would start and within 5 seconds stop again.
Checking for the errors in the runbook designer it turns out that the connections could not be made to for instance the SCOM server.
We investigated and finally I found this article on the TechNet site:
Migrate Orchestrator Between Environments
Among the very first things I saw was the first step involved in bold:
Back up SQL Server service master key in environment A
Now at first I did not know what this was or what it does, but I am pretty sure this is what our problem relates to!
It did irritate me that those TechNet pages did not reference each other though! Also as you will see they do not take into account what happens when you have multiple target servers such as when using Always On functionality.
First thing I did was think that it was too late already to get that thing moved over. So I went ahead and created a fresh key. Next thing to do was enter the encrypted info again. So go find the connection settings for connecting to SCOM and e-mail and SCCM and so on and re-enter passwords. I opened up all runbooks and re-selected those connections to be sure. Also the System Center license key.
Restart the Orchestrator management service and we were back up and running.
However, hold on!
A little later this went wrong again and it was due to a manual fail over of the availability group the database was living in. And again we had runbooks failing and again the enter your license key error!
Turns out I missed something important there!
Alright this is how it works:
- the data you enter like passwords and so is stored encrypted in the Orchestrator database.
- a key from the database is used for this.
- however this key can only be read or decrypted using the SQL Server Service Master key!
Yeah and this Master key is not inside the datasbase but is stored in the SQL system tables. Meaning it is related to the Instance and not the database. Now in a normal failover cluster we would not have run into the second time to see the Orchestrator go nuts on this. However with this always on cluster there are two machines with their own SQL instance. And the databases are made in sync. But there are two instances!
If the Server Master key is not the same (which it isn't) the data can not be descrypted with failover. So I should have used the key from the original server in the first place or just have replaced it after seeing the error. I did not know it could still be done afterwards. And also did not know that in this case
SO what to do:
Open up SQL management studio and run this command to check the master key:
SELECT * FROM sys.symmetric_keys
(by the way you see here that its not database related by instance related because it is asking the master database for this info and of course its taking it from the system tables).
You will see the servicemasterkey and the Guid belonging to that key.
And of course this was different between the two servers and would have been different from the original database server as well.
So to fix it we need to create a backup of the first key and restore it onto the second server.
First create a directory on the first SQL server to store the backup of the key in. Next run a backup command to backup the key. I have changed the password a bit in this post.
BACKUP SERVICE MASTER KEY
TO FILE = 'C:\backup\service_master_key'
ENCRYPTION BY PASSWORD = '3dH85Hhk004GHk2597ghefj5';
Next I created the same directory on the second server and copied the key bakcup file to it. Open a SQL query on the second server and import the key:
RESTORE SERVICE MASTER KEY
FROM FILE = 'C:\backup\service_master_key'
DECRYPTION BY PASSWORD = '3dH85Hhk004GHk2597ghefj5' FORCE
By the way this last command uses the FORCE option because it will find a database which is already using encrypted data, so we need to force it to use this master key. Make sure you take the key from the working machine and put it on the not working machine though!
Now we check again for the master key and see that they are the same. I restarted SQL Server Service.
Restarted orchestrator services on the orchestrator management server and runbook servers.
So there were a few lessons learned. Sure wish I had read the second article earlier as it could have prevented part of this.
A few of these lessons learned came back when we did the SCOM operational database, but that is a story for next time.
Since Microsoft stopped their Microsoft Management Summit and integrated it into their TechEd it still felt like a void was left there in the System Center space. Of course the MMS in the old days started as a community thing and in the end it ended up as a Microsoft conference with some community input. And of course now it was gone.
Well, the community decided to step back in! (yeah i better not include too many smileys here).
User groups and MVP's and other community leaders and experts are jumping in and now the Midwest Management Summit (so also MMS in short) has been born. It is filled with great speakers, many from the community and MVP sides from all over the world. It is gearing up to be a great event and specialized into the field of System Center in connection also with PowerShell, Clouds and so on.
Have a look at this great lineup of sessions in the schedule: http://mms2014.sched.org/
Next I can inform you that I also will be speaking there in two sessions along with my long time friend, SCOM specialist and MVP Cameron Fuller
Check out our sessions here: http://mms2014.sched.org/speaker/bob.cornelissen
I hope to see you there. If you are attending the MMS, please feel free to catch me there and have a chat! That is also what these community based events are all about. I am very much looking forward to meeting as many of you as possible during the upcoming events in November. I will be attending the MVP Summit and this MMS of course!
This week I added an Orchestrator 2012 Runbook server to an existing one for scale-out and high availability reasons. Very soon it was ready to go and I was making some additional runbooks to use together with SCOM. In these runbooks were Run .Net Script activities with PowerShell scripts in there. And I noticed the script activities would refuse to run. Except when I ran them separately as a normal PowerShell script. SO I went in the history and checked what had happened:
File C:\Program Files\System Center Operations Manager 2012\Powershell\OperationsManager\OperationsManager.psm1 cannot be loaded because the execution of scripts is disabled on this system. Please see "get-help about_signing" for more details.
O right! So I opened up a PowerShell prompt using Run As Administrator and I typed "Set-ExecutionPolicy Unrestricted".
And the script failed to run again! Wait a second perhaps it is because I forgot to run the same command on the other runbook server. Oops. OK running that command again and....
I went searching for it and I saw a comment in a thread somewhere saying it could be that the same command needs to be done for the 64 bit version of PowerShell as well!
Did this on all Runbook servers this time as well
And try again. Working fine!