Bridgeways has been working very hard this last year in coming back into their speed for creating management packs, updating existing ones, making them more intuitive and useful to work with, adding support staff and also hiring a very competent CTO (a good friend of mine and fellow MVP Simon Skinner ) who takes management pack quality very seriously. It is exciting to see the progress being made.
Today the new version of their VMWare monitoring management pack was released. It contains updated views and dashboards. The new style dashboards also were augmented with a few core dashboards:
- Host Performance: includes CPU Usage, Memory Usage, Swamp Memory Usage, Balloon Usage, Network Usage and Storage Usage data.
- CPU Performance: includes CPU Usage, Average CPU System Time, Average CPU Ready and CPU Wait data.
- Memory Performance: includes Active Memory, Balloon Memory, Shared Memory and Swapped Memory data.
Also there is an expanded set of reports. The combination gives better and quicker insights into the VMWare environment in order to troubleshoot issues and to proactively find upcoming issues.
Now as some of you probably know I have always been a big fan of a competing vendors management pack for VMWare (Veeam) and I still am. However it is good that a few other vendors have been looking seriously at creating a good management pack which can cover this monitoring scenario. I know a few have been working on this and are becoming serious contenders when it comes to product selection for this purpose. The Bridgeways pack will be one of them now in every selection process. Especially when there is a price difference to be found between solutions our customers (and you) will be looking at price/quality points and if it covers that what you are initially looking for. I know all my customers always look at price very closely and I do not think its a pure Dutch thing to do that.
I will be examining functionality and pro's and con's of these packs closely in the future.
So for now congratulations to Bridgeways for taking the step forward and we are watching you closely
For any questions regarding this article or any management packs feel free to contact me.
We noticed the release of UR9 for Service Manager a day before and WSUS also already had the SCOM agent updates, but today the KB article is released giving the overview of the UR9 for System Center 2012 R2 products.
The products receiving the update are:
- DPM 2012 R2 UR9 Data Protection Manager
- SCOM 2012 R2 UR9 Operations Manager
- SCSM 2012 R2 UR9 Service Manager
- SPF Service Provider Foundation
- VMM 2012 R2 UR9 Virtual Machine Manager
- WAP Windows Azure Pack
Do not forget to test first and to read the KB articles of whatever you are upgrading.
LJUBLJANA, January 11, 2016 – Comtrade System Software & Tools (Comtrade), a leading provider of IT infrastructure and application management and monitoring solutions, today announced that its Citrix management packs were acquired by Citrix, a strategic, long-term partner of Comtrade.
Read more through this link:
We have had an issue with the Orchestrator web console not working at a customer site for a while. Since this feature was not used it got pushed back to lower priority. However for instance the Infront management pack for Orchestrator did not work anymore due to their connection to the web service and we could not follow the status of runbooks anymore. It bugged me today again so went into fixing it. I used some websites/blogposts/KBarticles/forums to get all the items together and will post links at the bottom. The following steps were needed to get it up and running again:
There are a number of known issues for instance after upgrading Orchestrator from a previous version. Lets start with the first:
Error Executing the current operation
The SQL statement to run is included in the KB article above.
Next thing to check while you are there right now is in SQL mgmt Studio click to the database - security and find the Orchestrator service account used to connect to the database. Make sure it is assigned the DB_owner role.
On the Orchestrator web server go into the IIS manager. You will two websites relating to Orchestrator. Click each website and next select the "Basic Settings" button. Make sure the websites are not using the default applicationpool but the "System Center 2012 Orchestrator Web Features". In my case this was set to default applicationpool as well, so had to change it.
Another step you may want to do (and it is part of what got it fixed for me!) is in IIS Manager go to that application pool "System Center 2012 Orchestrator Web Features" and click "Advanced Settings". Find the Process Model -> Identity. This should be the service account from Orchestrator. Please go into that by selecting the ... button in that line and re-enter the username and password.
We need to check the web.config files.
How to Change the Orchestrator Database
I had to use the second part of that post to decrypt on config file and change it to use the correct database server and instance and database (we have changed the SQL server at some point to be places on an Always-On cluster) and encrypt it again. Keep in mind the syntax of the two commands. The "connectionStrings" part is case-sensitive so wite it exactly like that! Also the path to the web.config file may need adjustment. For instance in our case the folder was "C:\Program Files (x86)\Microsoft System Center 2012 R2\Orchestrator\Web Service\Orchestrator2012" You will see when it works the way as intended.
Next one to check out while you are at it:
C:\Program Files (x86)\Microsoft System Center 2012 R2\Orchestrator\Orchestration Console\web.config
In that file I took the line:
And made a copy of it to the line below it. Next add the FQDN to it so it reacts also when using full server names:
And save the file.
After any of these changes make sure you restart IIS completely.
Somebody had a problem with Locale localization of the server:
Orchestration console fails to start with [Arg_TargetInvocationException] error
Hope it helps somebody who runs into Orchestrator web console problems.
Throughout these fixes I have used posts from these links:
Good luck to all!
I have written and talked about Jalasofts great tool called Device Simulator. Many people in the monitoring space have been using this tool in previous versions. Today they released a fresh version of this tool to the 5.5 version.
The main additions to this version are:
- An OID editor meant to customize the behavior of certain OID's within the simulation file of your to be simulated device.
- The SNMP Dump Manager to more easily work with al the simulation files you are using
- The number of simulated devices out of the box has been expanded to over 100!
This tool can be used for free provided you only simulate a few devices at a time. And for a few dollars an expanded license can be bought to do a lot more testing with this.
For demo and test environments this can be used a lot as a primairy tool to simulate network devices without needing to have all of that infrastructure in production. Also you could take a recording of an existing device in your production and even including real time data and play it back through the simulator.
Everybody needs a copy of this piece of software!
As you may be aware I have been very involved with an animal charity. It is called House of Tails (www.houseoftails.org ) and is about taking care of abandoned dogs in Thailand. This is also the reason why the volume of my tech blog posts went down for a while. I will however pick up the pace again, I promise! Lots of ideas here. Because the dog sanctuary has a very hard time to make things happen this has taken a lot of time. We have setup an official Foundation (Stichting in Dutch language) with bank and email and PayPal and so on and we are working on a fresh website.
Yesterday a dear friend Pete Zerger wrote a blog posting about our story and he has taken it upon himself (and his friends and family) to bring a challenge. Any donation given to our House of Tails charity within a 72 hour period (starting last evening, so for the coming 52 hours or so still) will be DOUBLED. So any donation you make will get a double effect for the animal charity and all of it will go to the wellbeing of these dogs!
Please read how you can help too:
Also I will include a link here to the PayPal/Creditcard possibility of the charity Stichting House of Tails.
Take a minute to check it out through that post and through the House of Tails website.
Seeing how my geeky family and my online families are willing to help out a good cause is truely heart warming. It means soo much for us and for the dogs!
a truely gratefull Bob Cornelissen
This is an update to the page describing a solution to the "Web Console Configuration Required" message for SCOM 2012 and SCOM 2012 R2.
If you get this message on your workstations and you do not have the rights to simply run the configuration wizard it asks you to run, it has to be configured through another way.
What the configuration wizard does is set a few registry keys and add a code signing certificate (can also be done through registry). For instance a customer of mine this needs to be entered into a VDI image to make it work for most people.
At SCOM 2012 R2 UR3 a new code signing certificate was introduced which was valid until 22 July 2015. ANd now at SCOM 2012 R2 UR7 a new code signing certificate is introduced. It is now valid until somewhere in September 2016. Bit unfortunate there was a lack of overlap there.
The article I made last year SCOM 2012 Web Console Configuration Required describes the fix and how to find the registry keys. AT the bottom of the article there are updates for a registry key we found as an extra potential blocker of access to the SCOM web Console and there is another update created today for UR7. It has the registry files you need for either UR3/4/5/6 or for the UR7 now.
Good luck and enjoy!
I wish to direct your attention to a blog post made by my friend and fellow System Center MVP Tao Yang. He is talking about community efforts and contributions and on how some of them get turned into commercial products. If you are working with SCOM for instance you will find many management packs and scripts and solutions and thoughts right from the community. You will also find some great paid solutions written by serious software vendors who add value. There are also cases where both options are available and this is a point where you have to think on how to spend your time and money wisely. (By the way a great way is supporting a dog charity, look to the right of this post!). Power to the community. Please read his blog post:
Regards, Bob Cornelissen
Recently I received a request to be able to monitor all Windows Services running under a specific user account (service account) one certain machines, and to bring them together in a overview. I will show you how I did this.
This blog post will show you a complete way to monitor a list of services running using the same account.
I will show you at the bottom of the post how to monitor a list of services with a similar name (which is just a few words different from this whole story).
First of all we have of course multiple ways to monitor services as I have explained in a blog post series about the subject: SCOM - Monitoring a Service - Part 1 intro. However all of this is geared towards monitoring a single service on a single machine or a group of machines for instance.
But sometimes I get requests to:
- Monitor multiple services with somewhat similar names (for instance "System Center *").
- Monitor multiple services running under a certain account.
- Alert if any service on any server is running using a certain account (which they dont want).
The first item has been blogged about years ago. The second one is what I was asked to do in this case, so I will pick that one up first. And the last one is something we did in a completely diffeent manner and I might blog about that as well later if you like that (the question was actually not nonly for Windows Services, but also for Windows Scheduled Tasks). I will get back to that in a minute.
The next thing is to bring these items for the first two bullets together in a dashboard or console. As example in the case I ran into at a certain customer the easy way would be to open up a Savision Live Map and add them through an automatic population rule. That way they are of course added to a group and are visible and have a rollup, all happy. In this case we were limited in the amount of maps we could use. So a next likely alternative is to create a distributed application (DA) in SCOM. From there we can add the found services into the DA's. Problem is this does not populate automatically by default. But we can change that behavior to enable automatic population of items on a DA. Think it is best to put that in a separate post how to do that. But the idea was to: find these services + monitor them + put them in a DA + automatically. So also if a service gets added which uses the same criteria it should automatically be brought into monitoring and should be added to the DA! And in his specific case we added the resulting DA's as items on an existing Live Map dashboard.
You know what, I can blog about all these things These kind of workarounds are not always easy to find, and sometimes it is an adjustment of what has been described by one or two people before.
We need to have a discovery and monitoring of Windows Services running on a certain machine, for all services which are using the same service account. And we also wanted to do this for both Automatic and Manual services.
Now we need to create a similar use case to do this blog post with. Lets go and find all Windows Services on the SCOM management server using the SDK service account (On my test machine im running a few services with that account to make it easy for me).
I will write this down mostly step by step and expect a little knowledge on how to do stuff in the SCOM Console and there will be XML editting involved, but still no crazy stuff.
Let's do this!
I am going to use a method posted about by Brian Wren wayyyyy back in 2008 for using wildcards with the Windows Service template, and adjust it to my needs. By the way the method in the post mentioned is the way to monitor multiple services with a similar name and a wildcard, using the same template as we use below. Lucky we can keep using the same methods as before and even modify them for other similar needs!
I want to have a new fresh management pack, and I want to have a group of servers to target this service monitoring to. So I start with creating a New Group in SCOM and meanwhile when it asks me where I want to save it I will have it create a New Management Pack for me.
The name is easy to recognize and I am using a versioning format as 1.year.month.anumber. This way with the year and month placed there when I create updates to a pack it is always a higher version than the last time I editted it (if it is in the same month I just move the last digit up a version number). Just one of the things I use for a versioning strategy.
And now we have a group in the same management pack. From this point I walk through the wizard for the group and just add the SCOM Management server to it. So this is my group of servers where I want to monitor the services on.
Moving on, now we will go to the Authoring pane in the SCOM console.
In the Authoring pane find the Management pack templates -> Windows Service. Click the Add Monitoring Wizard. Select Windows Service to start the correct wizard.
Enter a name and description and of course place this in the newly created management pack.
Now on the Service Details page we enter a temporary service name. We will change this later, but we need to have something we can find back quickly enough. So I entered test123 as service name. Next we select the targeted group, which is the group we created earlier and contains the application servers. Because we want to monitor these services both when they are in automatic state and manual startup state we de-select the checkbox at "Monitor only automatic service". I am not that interested in performance data for each service so I just walk through the rest of the wizard in a Next Next Next Finish kind of way.
So now we have a service monitor created in the new management pack and targetted at a group of servers.
Now go to the SCOM Admin pane and go to the Management Packs list. Find this management pack and export it to for instance the desktop.
First we need to do something else before we start editting the code within the exported XML of the management pack.
Now the default Windows Service monitoring template uses a different method to find the specific service (discover it!) than we would use to collect several services in one query. FOr this we want to use WMI to query windows for all the services we are interested in and put that in the Discovery item in the management pack we just created. For this to work of course we first need to find the correct WMI query and test if this gives us the desired result. For this we can use a tool already installed on your machine.
Go to the server where there are these different services running using the same service account. Open a command prompt. Type wbemtest and press enter.
Press the Connect button, make sure it says "root\cimv2" near the top and press Connect button again.
We should now be connected and all the buttons enabled. Find the Query button and click it.
Now the trick is to make the right query to get your data returned to you. In my case I had to search a little bit. Also because I wanted to have the extra filter not only for account name but also wanting to see manual and automatically starting services.
select * from win32_service where (Startname like '%sdk%') AND (Startmode != 'Disabled')
I came up with that query. So what does it do. Well it requests the list of windows services and filters it (where) to find any service with a Startname containing sdk. The percent % signs around it are the wildcards. In WMI they use a percent sign as wildcard and not the star. And I added a second filter where I filtered out all disabled services. That should give me what I want. Lets run the query by clicking the Apply button.
Alright. These are the internal names for the services and indeed there are two SCOM related services and 3 SQL related services which are running with an account with "sdk" in the name somewhere. So after some playing and trying to find how to write WMI queries we finally have a query which will return what we want. Now please copy the query somewhere, so we can use this later.
Now the management pack Authoring part! We will now edit the XML of the management pack we saved earlier. You can do this in any editor and even in Notepad, but I suggest to use at least an editor with some code highlighting. For instance I use Notepad++ for this.
And of course I found out that I did not have it installed on this test server, so I downloaded it (free !) and installed it.
Open up the XML file in the preferred tekst editor. We want to find the Discovery for our custom service we created. So find "test123" and you will have found the discovery.
This looks like a lot of stuff and to some it is not Always too clear what it all means, but stay with me. I will highlight the pieces of code within this block we want to change:
I am sorry to report that this blogging software messes up the XML code blocks and the codeblock option makes it unreadable. So I put the whole thing in MS Word and made two screenshots of the whole discovery. The red parts of the code is what we are looking for (in three places).
We are going to replace the first red part with:
(I left the Frequency tag in there so you recognize it).
The next red piece was where it said: 'BinaryPathName'
We replace that by: 'PathName'
And the third red piece was for the DisplayName and in my case it said: SDK Services
We will replace that by: $Data/Property[@Name='Name']$
Next I scroll to the top of the code and find the version number and give the last digit a higher number:
Now I found out that the template can also make a strange management pack reference. We are using discovery modules living in the Windows Library and the reference has an alias. We need to change something to make this work. Go to near the top of the management pack code (first 30 lines or so) and you will see a list of references. Look for the one listing the Microsoft.Windows.Library as such:
See the here purple colors alias? We need to copy that alias name (so everyting between the quotes and not including those quotes). Next go to the first block of tekst you replaced.
Now the bit where I put the red line under is a reference, we can recognize it by the ! exclamation mark ! Now replace the part before the exclamation mark (windows) with the alias you had in your reference.
Now it should look like the above, and referencing the reference alias you found in your code.
Next I save the file using the exact same name (do not rename management pack files!). And I go to SCOM Admin pane - Management Packs and I import this management pack from disk.
Now in the SCOM Console go to the Monitoring pane. On the left hand side scroll all the way down until you find "Windows Service and Process Monitoring" and go to the state view called "Windows Service State". Within minute I saw these 5 services appear and a few minutes later they were in a monitored state as well. The two at the top just happened to be in there already.
Alright, there we have it! We are now monitoring all windows services - running under a service account with sdk in the name of the account - for both automatic and manual start services - on all machines in a custom group we created.
How to monitor multiple services with a similar name in SCOM
Alright as promissed how to monitor a group of services with a similar display name, such as this example:
You see the group of Jalasoft services there. I want to find all of them and monitor them all. Well you have seen that we can use wbemtest tool to create a WMI query to get that data. In this case we will filter on the name (this is the displayname as you see it in the screenshot above).
select * from win32_service where (name like '%Jalasoft%')
There is our string and the result while querying WMI for this. Exactly what we want.
Well in the story above we know where the query ended up, so we can just insert this WMI query in the same place to have it monitor all services with Jalasoft in the name.
In this case I want to add this monitoring set to the existing pack I created above. So in short the order of actions:
- Create a Service monitoring template entry and name it Jalasoft Services.
- Store it in the Custom Appllication Monitoring management pack.
- Complete the wizard and put in a recognizable name in the what service to monitor field.
- Export the management pack
- Verify through wbemtest that our WMI query gives the desired result and copy the correct query
- Open up the exported management pack in an XML editor or text editor and find the discovery relating to this in the way shown above and replace the three blocks of code for the three new ones and of course insert the right WMI query.
- Find the management pack reference Alias pointing to Microsoft.Windows.Library and replace the reference in the new code block for that one.
- Raise the number of the management pack version and save the management pack.
- Import the management pack
- Inspect the results in the monitoring pane
And there we go:
A group of services with something similar in the name, running on a group of servers we defined. Any service with similar name being added to this group of machines will get picked up within a few minutes and be monitored.
SO now we have had the story on how to add either a group of services with a similar name or running under a similar account.
Brian Wren's post was all we need to cover the second example and also put us on the right path for the first example. I just thought it was time to blog about this in an extensive post to show how this is done and what things we can run into. Also using the SCOM 2012 R2 interface of course and showing the wbemtest interface.
Next on the agenda:
Well I said in the beginning that I wanted these objects placed in a distributed application in SCOM and have it pull in these services automatically. So I will create a simple Distributed Application and I will add all Jalasoft services to it.
Of course this will be in my next post coming very soon!
It has been a while since I wrote about Veeam and their management pack for monitoring VMware products (did you know they also added Hyper-V to that now?). Lately I got a chance to install this pack in a reasonable size environment, so this is a good chance to write more about the product and what it can do for us.
This environment has two VMware infrastructures. One for the servers based on 60+ hosts and one for the VDI desktops environment also roughly based on 60+ hosts. This situation also requires that we have different monitoring for each environment, because we are not really interested in VDI desktops which get destroyed and rebuilt every day. But in the server infra we are interested in the guests as those are the ones we also monitor with SCOM agents. I will discuss as well how this was achieved, because Veeam has a way of doing this which not every organization will use, but it is very useful in the scenario we are dealing with.
When designing your Veeam-SCOM monitoring infrastructure there are a few things to keep in mind which affected our design in this scenario.
- Numbers. A bit of an open door, but the amounts of hosts/datastores/VM’s you monitor has an effect on the sizing of your Veeam Collectors. Deciding whether or not to monitor VM’s makes a big difference.
- Wanting two separate methods of monitoring for the two environments is best done with the use of Monitoring Groups. It is a feature included in the Enterprise Plus license of Veeam and gives possibility to create in our case two Monitoring Groups. Each with their own collection policy (one not interested in collecting VM information for instance).
- A Collector can only be a member of one Monitoring Group.
- High Availability, even though a collector might be able to handle the load it might be desired to have an additional collector in each monitoring group for high availability purposes and during normal situations a way to spread the load.
One good piece of advice is to download the resource kit and use it. It has an excel sheet in it which is a calculator for the estimated number of collectors you need for a certain environment!
By the way you will find that a collector can handle a very nice load. The stability of this platform has been strengthened throughout the years (And I have been stress testing previous versions of the pack for years in some big environments just to see when it would break down under the load and give the feedback to the developers).
In our case we went with 3 collectors for the server VMware environment and for 2 collectors for the VDI environment. Because we were not interested in the VDI VM’s this was enough to also provide high availability for the collectors within the monitoring groups.
Because not many are using the monitoring groups feature I will show you what it looks like.
This picture shows on the left hand side the monitoring groups. The top one is the default and the other two were created manually. The example shows the VDI monitoring group with 2 collectors and some stats. In the middle of the picture there are two important things we adjusted as a start. First of all to change the collection interval from the default 5 minutes to 10 minutes (this basically gives a bit less data flow simply because it collects for instance performance data less frequently). The second important setting is setting the Collect VM Data to false for this monitoring group. With thousands of VDI VMs this makes quite a difference.
If you open the monitoring group collector settings you see first of all the possibility to stop the inheritance from the settings of the top level to flow down to this monitoring group. Next we set the EnableVMDiscovery option to False to stop the discovery of VMs and their data from going towards SCOM.
Something else to note in these settings is the CollectionInterval which in this case we set to 10 minutes. And as an example the DataStoreScanInterval that is set to a number, which is actually a multiplier for the CollectionInterval. So when I changed the CollectionInterval from 5 minutes to 10 minutes, also other collections which happen less often are dependent on this setting. In this case the Datastores will be scanned every 12x10 minutes, so every 2 hours instead of every 1 hour. Keep in mind that when tinkering with the collector settings there might be multiple things happening in the background. In case you want to play with the default settings make sure you read up on what these settings do. Veeam MP is very well documented from design to install to adjusting to monitoring.
Now, when starting out with the installation and connecting the collectors to the VCenter servers we deliberately set the collection interval to 10 minutes and also the VM discovery to disabled on both monitoring groups. We figured the amount of data coming in from only the hosts, VCenter, clusters, datastores and hardware and so on and the resulting alerts from that would be enough to start with in the first few days. This was a good decision as there was enough to start with all these layers without already putting the VMs layer on top of it. For larger environments I think it is useful to phase in the deployment this way to give opportunity to do tuning and tweaking and discuss the resulting alerts and views with the operators. It also gives you opportunity to check on the performance and dataflow of the collectors and their agents and the SCOM infrastructure. Check also for most common alerts and health state changes and the amount of performance data coming in and see if this suits your needs. After that continue to the next phase and bring in more monitored stuff.
Hope this was useful to show a case where monitoring groups within the Veeam Management Pack can be used and why. Actually I wrote this post a while back and did not get to posting it until now. Meanwhile we have done a lot of tuning and dashboarding for instance.