Experiences implementing the Veeam management pack with monitoring groups

SCOM, System Center, SCOM Tricks, SCOM 2012 Send feedback »

It has been a while since I wrote about Veeam and their management pack for monitoring VMware products (did you know they also added Hyper-V to that now?). Lately I got a chance to install this pack in a reasonable size environment, so this is a good chance to write more about the product and what it can do for us.

Situation sketch

This environment has two VMware infrastructures. One for the servers based on 60+ hosts and one for the VDI desktops environment also roughly based on 60+ hosts. This situation also requires that we have different monitoring for each environment, because we are not really interested in VDI desktops which get destroyed and rebuilt every day. But in the server infra we are interested in the guests as those are the ones we also monitor with SCOM agents. I will discuss as well how this was achieved, because Veeam has a way of doing this which not every organization will use, but it is very useful in the scenario we are dealing with.

Design

When designing your Veeam-SCOM monitoring infrastructure there are a few things to keep in mind which affected our design in this scenario.

  • Numbers. A bit of an open door, but the amounts of hosts/datastores/VM’s you monitor has an effect on the sizing of your Veeam Collectors. Deciding whether or not to monitor VM’s makes a big difference.
  • Wanting two separate methods of monitoring for the two environments is best done with the use of Monitoring Groups. It is a feature included in the Enterprise Plus license of Veeam and gives possibility to create in our case two Monitoring Groups. Each with their own collection policy (one not interested in collecting VM information for instance).
  • A Collector can only be a member of one Monitoring Group.
  • High Availability, even though a collector might be able to handle the load it might be desired to have an additional collector in each monitoring group for high availability purposes and during normal situations a way to spread the load.

One good piece of advice is to download the resource kit and use it. It has an excel sheet in it which is a calculator for the estimated number of collectors you need for a certain environment!

By the way you will find that a collector can handle a very nice load. The stability of this platform has been strengthened throughout the years (And I have been stress testing previous versions of the pack for years in some big environments just to see when it would break down under the load and give the feedback to the developers).

In our case we went with 3 collectors for the server VMware environment and for 2 collectors for the VDI environment. Because we were not interested in the VDI VM’s this was enough to also provide high availability for the collectors within the monitoring groups.

Monitoring Groups

Because not many are using the monitoring groups feature I will show you what it looks like.

Veeam Monitoring Groups

This picture shows on the left hand side the monitoring groups. The top one is the default and the other two were created manually. The example shows the VDI monitoring group with 2 collectors and some stats. In the middle of the picture there are two important things we adjusted as a start. First of all to change the collection interval from the default 5 minutes to 10 minutes (this basically gives a bit less data flow simply because it collects for instance performance data less frequently). The second important setting is setting the Collect VM Data to false for this monitoring group. With thousands of VDI VMs this makes quite a difference.

If you open the monitoring group collector settings you see first of all the possibility to stop the inheritance from the settings of the top level to flow down to this monitoring group. Next we set the EnableVMDiscovery option to False to stop the discovery of VMs and their data from going towards SCOM.

Something else to note in these settings is the CollectionInterval which in this case we set to 10 minutes. And as an example the DataStoreScanInterval that is set to a number, which is actually a multiplier for the CollectionInterval. So when I changed the CollectionInterval from 5 minutes to 10 minutes, also other collections which happen less often are dependent on this setting. In this case the Datastores will be scanned every 12x10 minutes, so every 2 hours instead of every 1 hour. Keep in mind that when tinkering with the collector settings there might be multiple things happening in the background. In case you want to play with the default settings make sure you read up on what these settings do. Veeam MP is very well documented from design to install to adjusting to monitoring.

Now, when starting out with the installation and connecting the collectors to the VCenter servers we deliberately set the collection interval to 10 minutes and also the VM discovery to disabled on both monitoring groups. We figured the amount of data coming in from only the hosts, VCenter, clusters, datastores and hardware and so on and the resulting alerts from that would be enough to start with in the first few days. This was a good decision as there was enough to start with all these layers without already putting the VMs layer on top of it. For larger environments I think it is useful to phase in the deployment this way to give opportunity to do tuning and tweaking and discuss the resulting alerts and views with the operators. It also gives you opportunity to check on the performance and dataflow of the collectors and their agents and the SCOM infrastructure. Check also for most common alerts and health state changes and the amount of performance data coming in and see if this suits your needs. After that continue to the next phase and bring in more monitored stuff.

Hope this was useful to show a case where monitoring groups within the Veeam Management Pack can be used and why. Actually I wrote this post a while back and did not get to posting it until now. Meanwhile we have done a lot of tuning and dashboarding for instance.
Bob Cornelissen

Free SCOM MP Authoring University opportunity

SCOM, System Center, SCOM Tricks, SCOM 2012 Send feedback »

Our friends at Infront Consulting Group are organizing another great set of sessions on SCOM MP Authoring. Watch below for an announcement and registration information. The event is capped to 100 attendees for get there on time:

Silect Software and Infront Consulting are hosting a FREE MP University on August 12th. Registration is capped at 100 people so we wanted to ask you to spread the word to your many followers. Brian Wren and Kevin Holman will both be presenting.

Jump to the registration at This Link

A big thank you to Infront Consulting and Silect Software for organizing this and for the presenters such as Brian Wren and Kevin Holman for being there! B)

Microsoft acquires BlueStripe

SCOM, System Center, SCOM Tricks, SCOM 2012 Send feedback »

I am sure many of you know about BlueStripe solutions and how they connect to SCOM for application discovery and so on. Today we hear that Microsoft has aquired BlueStripe. I think this is a very good move indeed! Read the announcement here:

http://blogs.microsoft.com/blog/2015/06/10/microsoft-acquires-bluestripe-to-help-customers-improve-application-visibility-and-management-across-the-datacenter-and-cloud/

We will probably soon see the effects of this in several areas around monitoring and management B)

SCOM and Savision Extension as a package for locked down workstations

SCOM, System Center, SCOM Tricks, SCOM 2012 Send feedback »

For those of you wanting to provide the SCOM full console, possibly augmented with Savision Live Maps extensions, to desktops which are locked down, as a package. It might help B)

Case
We want to create a software package for the SCOM Console with Savision Live maps extensions so it can be run on managed locked down desktops.
Used are Windows 7 desktops and ThinApp and AppSense.
The desktops are locked down for users.

Package
We started with the prerequisites and installation files which needed to be added to the package (or not):
- .Net Framework 4, was already available in the system, so not included in building this package
- SQL CLR Types, was already present in the image as package, so not included
- ReportViewer 2012, was already present in the image as package, so not included
+ SCOM 2012 R2 Console installation
+ The at that time current Update Rollup 5 for SCOM 2012 R2
+ Savision Live maps extension for the at time current version 7.3

Problem
After the package was created and installed there were problems with SCOM and specifically with the Savision extension for viewing Savision maps directly from the full SCOM console.
It looked strongly like a rights issue, because the problem would occur on the restricted desktops with normal users.

Solution
The issue were the below mentioned two dll files, which do not get created at installation, but at login. The first user to login seems to create these two files in the ProgramData path. The creation of dll files in that path was blocked by AppSense.

The solution is to WhiteList the specific path in AppSense Application Manager so a user can write the dll there.
It seems that the subdirectory where these are created are dependent on the version of Savision used and when updating it to a newer version an additional subdirectory is created and the new two files placed there. So a wildcard was used for the folder name.
There was also an exclusion made for the SCOM directory:

As soon as it was made possible for a user to create these files upon first login to SCOM Console and opening a Live Map these were created and everything worked fine.

A big thanks to Jerry Chen for checking and fixing this.

Bob Cornelissen

SCOM Object reference not set to an instance of an object

SCOM, Service Manager, SCOM Tricks, SCOM 2012 Send feedback »

I had some problems with a State view in SCOM for a certain class, which used to work and now produced an error saying:
Object reference not set to an instance of an object

So I opened up the Discovered Inventory view and selected the appropriate class and waited for the screen to populate.
The same error appeared.

Just for fun I thought I would pick up the instances of the class with powershell by using the get-scomclass and get-scomclassinstance combination to get a table of all objects of the class and that did turn out a list of data.
So I suspected a faulty entry somewhere in the database for one of the discovered objects. Actually I had 1 top level class created and 3 child classes and this happened to two of the child classes.

Updating the pack to a newer version did not work and it was confirmed in the test environment that the pack should not be the problem.

I created a state view of the not monitored top level class and added columns for the other classes to it and that one looked fine.

So I wanted the discovered objects to be all gone from SCOM
The way you usually speed up the process is by setting overrides to the discovery rules for these classes and disabling them for all objects.
Next you run the command remove-scomdisabledclassinstance
And wait for it to remove all the entries.

But while running this command I got an error (see below in text and picture form):

PS C:\> remove-scomdisabledclassinstance
**WARNING** This operation WILL result in class instances and relationships being permanently deleted. This operation i
s irreversible and will likely result in a significant amount of activity in the operational database. Are you sure you
want to do this? [Y/N]

y
Started. This operation may take a very long time to complete.
remove-scomdisabledclassinstance : Discovery data has been received from a rule targeted at a non-existent object ID.
Object ID: 8375a67f-5750-53e1-bd9f-7468152934b0
Rule ID: 3147a861-3d91-74a6-675e-e41be8f92658
At line:1 char:1
+ remove-scomdisabledclassinstance
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (Microsoft.Syste...nstancesCommand:RemoveSCDisabledClassInstancesComma
nd) [Remove-SCOMDisabledClassInstance], DiscoveryDataFromRuleTargetedToDeletedObjectException
+ FullyQualifiedErrorId : ExecutionError,Microsoft.SystemCenter.OperationsManagerV10.Commands.RemoveSCDisabledClas
sInstancesCommand

If you get that error the best you can first do is in SCOM open up a Discovered Inventory view and check how many objects are still there (if it does not throw the same exception as the one I started with!).

Check the number of entries there.
Now run the remove-scomdisabledclassinstance command again.
If it throws the error refresh the screen to check if discovered inventory shows a lower number of discovered instances.
Keep doing that until it finishes. Took me 25 times :>

Next I waited for a grooming interval and let it populate the classes again one by one by turning the discoveries back on.

Bob Cornelissen

Saving dogs while monitoring

SCOM, System Center, Active Directory, SCOM Tricks, SCOM 2012, Windows 2012, Configuration Manager Send feedback »

I know this blog is all about my technical career in IT, with System Center and Microsoft products and so on and so on. But I also have a secondary life which is very important to me and a lot more. In fact I am somewhat of a shy person and so is my wife. We do not ask for favors much, but do like to give them. I am talking about our dog and animal sanctuary in Thailand. Currently my wife and our staff Oarn are mainly taking care of about 60 dogs in 3 locations. Also some of our family members like the Mother and sister of my wife in one location and her father in another are doing their best to help out. We have to do all of this out of our own pocket and I can tell you it is not possible to do this alone. Also we have some huge set backs in the past few months. We are in the process of emergency relocation and are in a big stress to find funding to be able to buy a new small plot of land in a less expensive province where we can relocate to, build fencing, build a roof, build a 1 room shelter for staff and relocate most of the dogs to that location. Of course this is next to our ongoing push for health (sterilization, anti-tick program, helping sick and wounded, vaccinations, vitamins and trace elements and iron in their food somehow to get them healthier).

Two days ago I did a webinar with my good friend Cameron Fuller for our friends at Jalasoft. The webinar was of course SCOM and monitoring related. Read about the webinar and also some of the things which Cameron said about the dog sanctuary which I would not have said myself and humbles us:
http://blogs.catapultsystems.com/cfuller/archive/2015/04/16/what-is-new-and-coming-soon-to-the-microsoft-monitoring-world-and-helping-a-good-cause-opinsights-scom-opsmgr-sysctr.aspx

We all donated the proceedings of this webinar to the good cause. Lek's House Of Tails.
More information can be found at following locations:

A website with some background information (see the tabs near the top below the picture) and information on how to donate (there are other methods, feel free to get in touch if you want to know how):
http://lekshouseoftails.azurewebsites.net/?page_id=222

The facebook page of the sanctuary:
https://www.facebook.com/LeksHouseOfTails

My personal Facebook Page (have lots of pictures from several years of the dog sanctuary):
https://www.facebook.com/bob.cornelissen.5

Feel free to contact us!

Also if you want to help out by whatever means (on location help in Thailand, financial, words of advice, toys, sharing our pages and posts, liking a post or whatever you can think of it is very much appreciated!
These dogs we have known for years, we have known their grandparents and all. They got deserted by people. They were in dire need. We try to help them have a life, eat food, have a roof for the rain, have a blanket to sleep on, have a bowl to eat from, be safe from getting hit by people or cars or getting poisonned or caught for some terrible plight. We just want to keep them safe, fed, healthy. They deserve a chance and all of them are really sweet and kind.
Due to bad circumstances we had to relocate to a temporary place and will soon move to our new place, which is a piece of empty land with nothing on it and start all over.

Now to get back to the webinar at Jalasoft. We have done this webinar and I feel it went great :p We want to thank Jalasoft for giving us this opportunity. I want to thank again my co-presenter Cameron Fuller, don't even know how to say how thankful we are mate.

Also a big thank you to several community members out of the technical community and MVP's who have helped in several ways in the past year.

During the webinar we had a surprise for the great folks and team at Jalasoft and that was that we named one of the pups without a name after their product range: Xian.
See below for a picture of Xian (on the left, with his brothers), it is a very smart and active dog and we thought it is a perfect fit :D

One more thing: The webinar we did was recorded and I am sure Jalasoft will publish it soon. Of course you can Always send a message to sales@jalasoft.com to get access to the webinar recording or the slides and of course to get more information on one or more of their great products. Of course in the near future I will be writing about several products and monitoring and System Center.

Xian and brothers

Thank you all so much for your patience and help and shares on behalf of me, my wife, our families and 60 sets of teeth from our dogs!
Bob Cornelissen

SCOM 2012 Importing a list of ICMP Devices

SCOM, System Center, SCOM Tricks, SCOM 2012 Send feedback »

Today we had a case where we wanted to make an additional ICMP ping monitoring for all the Unix/Linux servers we were already monitoring in SCOM. There was a specific reason for that. I wanted to quickly get all the IP addresses of the Unix/Linux machines out of SCOM and into the Device Discovery.

First I started by wanting to export the IP addresses through Powershell. So I started from a query I had made earlier over here and use that to pick up only the IP address and create a CSV file.

get-scomclass -Displayname "UNIX/Linux Computer" |get-scomclassinstance |select @Label="IPAddress";Expression={$_.'[Microsoft.Unix.Computer].IPAddress'}} |Export-CSV -notype C:\Scripts\nixexport.txt

Alright, Now I have a TXT file I can play with. I used Notepad to replace the double quote with nothing. And I removed the first line (which was the header). This leaves us with a simple txt file with a list of all IP addresses.

Next in the SCOM Network Device Discovery, I opened an existing discovery rule with some devices in it already and selected the Import button in that wizard (step 1 in below picture). Select the txt file and all the devices will be added to the bottom of the devices list of the discovery rule. Next Multi-select all of them (step 2 in below picture) and select Edit (step 3). Change discovery mode to ICMP. The picture was taken after they all got changed to ICMP already. And now click through the rest of the discovery wizard and let it run.

So this went quite quickly and with very little remembering of IP addresses and typos.

Good luck!
Bob Cornelissen

SCOM 2012 R2 UR5 and SCOM 2012 SP1 UR9 released

SCOM, System Center, SCOM 2012 Send feedback »

Last night there were some update rollups released for SCOM. One for SCOM 2012 SP1 and one for SCOM 2012 R2. Will list them below.

For SCOM 2012 SP1 this is Update Rollup 9 and can be found at KB3023167. The description does not include much at the moment except the mention of Operational Insight and the support for SUSE Linux 12. The article can be found here: http://support.microsoft.com/kb/3023167

For SCOM 2012 R2 this is Update Rollup 5 and can be found at KB3023138. The description has several items listed and there is also the Linux packs separately with an update. The article can be found here: http://support.microsoft.com/kb/3023138

Items in the UR5 rollup for SCOM 2012 R2:

  • Monitoringhost process crashes because of bind failures against Active Directory
  • RunAs accounts cannot be edited because of "Specified cast is invalid" exception
  • Application crashes when a search is finished without filter criteria on Distributed application designer
  • MonitoringHost crashes with the exception - System.OverflowException: Value was either too large or too small for an Int32
  • Support to troubleshoot PowerShell scripts
  • Operational Insights through Operations Manager
  • "Reset the Baseline," "Pause the baseline," and "Resume the baseline" tasks fail when they are run against an Optimized performance collection rule
  • An exception occurs when you edit subscriptions that contain deleted monitors or objects
  • You can't set Widget column width
  • Event 4506: Data was dropped because of too much outstanding data
  • Support for handling Datawarehouse time-outs
  • Can't continue scan with NOLOCK because of data movement
  • $ScriptContext.Context does not persist the value in PowerShell widgets
  • Evaluation version alert

And the Unix/linux Pack:

  • Updating the SCX Agent causes deep monitoring status for the JEE Application Server to be reset.
  • By default, the Rpcimap monitor for Red Hat Enterprise Linux 5 is now disabled.
  • Monitoring in UTC +13 causes Unix Process Monitoring template to fail.
  • Using scxadmin -log-rotate causes logging to stop after log rotation.

As Always please first test the update rollups and supporting management packs and any newly released management packs first in a test environment!

Bob Cornelissen

New KB - Alerts may not get forwarded as expected via a connector in Operations Manager

SCOM, System Center, SCOM Tricks, SCOM 2012 Send feedback »

New KB

Alerts may not get forwarded as expected via a connector in Operations Manager

When using connectors to forward alerts in System Center Operations Manager 2007 (OpsMgr 2007) or System Center 2012 Operations Manager (OpsMgr 2012, SP1 and OpsMgr 2012 R2), in certain situations such as an alert storm (defined as a large number of alerts being generated in a very short period of time) there may be alerts that are not forwarded via a connector. When this occurs, these alerts will never be forwarded and will remain in a "New" state.

Cause and potential fix can be found in the KB article here:
http://support2.microsoft.com/kb/2277825/en-us

Good luck :p

WMUG NL first webinar of the year. Monitoring SQL with SCOM and Operational Insights

SCOM, SQL, System Center, SCOM 2012, Windows 2012 Send feedback »

Want to know about Monitoring SQL with SCOM and Operational Insights?
Then you have to check in with this webinar hosted by the WMUG NL user group and presented by a fellow MVP and friend Simon Skinner. Save your calendars for 28 January 2015 at 20:00 CET (GMT+1). For more information and registering for this webinar please follow this link and do not be shy to spread the word:

http://wmug.nl/2015/01/14/wmug-nl-webinar-1-2015-deep-dive-into-monitoring-sql-server-with-system-center-and-operational-insights-hosted-by-simon-skinner/

All are welcome to join!
Bob Cornelissen

Contact / Help. ©2017 by Bob Cornelissen. blog software.
Design & icons by N.Design Studio. Skin by Tender Feelings / Evo Factory.