Monitoring IBM i with IBM Systems Director

As other POWER IT Pro articles have highlighted, IBM Systems Director is a robust tool for tackling a significant number of management tasks in the IT environment. In this article, I’ll focus on the capabilities that Systems Director provides for monitoring IBM i and IBM i resources. I'll also explore the use of Systems Director to monitor various system metrics, subsystems, message queues, and processes.

Background: Getting Ready

Before Systems Director can monitor an IBM i system and its resources, the endpoint (a Systems Director term for a component that's being managed) must be discovered, accessed, and inventoried. The protocols discovered on the IBM i endpoint should include the Common Information Model (CIM), aka the Platform Agent, and Common Agent Services (CAS), aka the Common Agent. For IBM i, the Common Agent runs in the IBM Portable Application Solutions Environment (PASE) and can be installed either manually or from Systems Director itself. The code is available for download from the Systems Director website. The Platform Agent is provided through the IBM i 5770-UME Licensed Program Product (LPP).

The agents (or protocols) available on the endpoint have a direct impact on both Systems Director's management functionality and the monitoring that can be performed. Table 1 provides a summary of the available agent options. In most cases, the Platform Agent will provide sufficient management and monitoring support. However, you should consider installing the Common Agent when you want to use the genevent utility to develop customized, client-side event generation scripts, or when you need additional monitors, including process and job monitors. The agent requirement for Message Monitoring depends on the version of the Platform Agent installed on the IBM i endpoint. Message Queue monitoring was moved into the Platform Agent with version 6.3; if you're using an earlier version, you'll need to use the Common Agent for Message Queue Monitoring.

	Discovery	Inventory	Systems Director Navigator	Update Manager	CIM monitors	Asset ID	Message queue monitoring	Use genevent	Process and job monitors	Director agent monitors
Agentless	X	X	X	X
Platform agent	X	X	X	X	X	X
Common Agent	X	X	X	X	X	X	X	X	X	X

The agents discovered on the IBM i endpoint can be verified in a couple of different ways. One way is to simply review the endpoint's properties, as Figure 1 shows.

The other way is by using the Security, Configure Access function, as shown in Figure 2.

Notice that some of the protocols shown in Figure 2 indicate "No access." This status can be a bit misleading. In this case, access has been successfully configured for the highest-level protocol (CAS), so all Systems Director communication will occur through CAS, and CAS will communicate with the other protocols on the endpoint as needed.

Monitoring System Metrics

The first thing to look at with monitoring is which monitors are provided through the standard monitor view in Systems Director. When you select the System Status and Health, Monitors function for the endpoint, a list of monitor views is displayed, as Figure 3 shows.

Systems Director 6.3 added an IBM i Monitors view, which contains a set of popular IBM i monitors (Figure 4).

As you can see, selecting the monitor view will display the monitors in the view as well as the current value for each monitor. Staying on this page will cause the monitor values to be updated periodically.

You can create additional monitor views by clicking the Create Filter button on the Monitors page, then drilling down into the specific endpoint and protocol you want from the displayed list. This process will make additional monitors available. After you drill down into the endpoint, a list of protocols that provide monitors is displayed, as shown in Figure 5.

     

The Agent protocol shown in Figure 5 refers to the Platform Agent, whereas the CIM Agent refers to the Common Agent. Drilling down into one of these protocols will display a list of monitor categories that you can continue to drill down into. Monitor categories are different from the actual monitors in a number of ways. First, a monitor category is a blue hyperlink that can be selected, and it can’t be added to the monitor view being created. A monitor, however, is black, and it can be added to the monitor view being created.

The monitors provided by the Platform Agent comprise two broad categories: File System Monitors and i5/OS System Monitors. Navigating into File System Monitors displays the hierarchy of the directory tree (starting at the root, or /). As directories are opened, monitors for the following directory properties are provided:

• Directory Attributes
• Directory Exists
• Directory Owner
• Directory Size
• Last Modified
• Object Type

Likewise, when a file is selected, the following properties that can be monitored are provided:

• Checksum
• File Attributes
• File Exists
• File Owner
• File Size (Bytes)
• Last Modified
• Object Type

There are nine monitor categories in the i5/OS System Monitors:

• I/O Processors
• Job Queues
• Job Statistics
• NetServer Statistics
• Physical Disks
• Storage Pools
• Subsystems
• System Statistics
• User Statistics

Like the File System Monitors, each of the i5/OS System Monitors categories has additional categories and monitors. For example, drilling down into the Subsystems category will display a list of subsystems from the IBM i endpoint. Drilling down further into a subsystem will present the list of actual monitors—which in this case are Subsystem % of Job Limit, Subsystem Active Jobs, and Subsystem Status.

      Monitors available in System Statistics include:

• CPU Utilization %
• Current Temp Storage Used (MB)
• Max Temp Storage Used (MB)
• Permanent Addresses Used %
• System ASP Used %
• Temporary Addresses Used %

And monitors available in User Statistics include:

• Users Disconnected
• Users Signed Off, Output Waiting
• Users Signed On
• Users Suspended by Group Jobs
• Users Suspended by System Request

In addition to the Platform Agent–provided monitors, the Common Agent provides numerous monitors, which you access by drilling down into the CIM agent. It’s somewhat difficult to determine which Common Agent monitor provides what information from the IBM i managed endpoint. To more easily identify information provided by a Common Agent monitor, you can navigate into the various monitor groupings (root/cimv2, root/ibmsd, root/PG_Internal, and root/PG_Interop), then use the table search capability to search for strings such as "IBMi" and "IBMOS400." Keep in mind that the monitors provided in the predefined IBM i Monitors view are provided by the CIM agent and have been renamed for usability.

Multiple monitors can be added to the same monitor view, and the monitors in the monitor view can come from different monitor categories and even different protocols. Although you must use a specific endpoint to create the monitor view itself, you can use the resulting monitor view with different endpoints (or groups) by specifying the endpoint on the Monitors page.

What Can You Do with a Monitor?

You can perform several functions with monitors, including Add to Dashboard, Activate Threshold, and creating event automation plans.

Add to Dashboard. The Add to Dashboard function is used to display a graph of the values for the selected monitor on the Health Summary page. When you add a monitor to the dashboard, you need to define a name for the monitor (you can also provide a description). Additionally, you must specify the type of chart to display. The default is a line chart; however, you can change the chart type to bar chart, area chart, stacked bar chart, scatter chart, or pie chart.

Once a monitor has been added to the dashboard, you can display it by selecting Health Summary from the System Status and Health function list, as shown in Figure 6. You can display monitored values for a group of endpoints on a single graph; to do so, select a group rather than an individual endpoint on the monitors page.

Activate Threshold. Thresholds are used to define the upper and lower values on which an event should be generated in Systems Director. As an example, Figure 7 shows the definition box for the threshold for CPU % Utilization.

Checking the Enable event generation check box causes an event to be generated in Systems Director whenever the defined threshold is exceeded. By default, the event generation will cause an entry to be added to the event log and an indication to be added to the scoreboard at the top of the Systems Director web interface. A bit later in this article, I’ll show you additional actions that can be taken against an event. Selecting the Generate events when the value changes check box will cause a new event to be generated every time the monitored value changes. Typically, this option would be used only for certain items, such as file-existence monitors.

The Maximum queued events setting defines the maximum number of events that will be held in the queue. Minimum duration indicates the amount of time a threshold needs to be exceeded before an event is generated. Resend delay indicates the amount of time a threshold is exceeded before another threshold event for the monitored value is generated (assuming the defined threshold is exceeded again). Finally, the monitor values indicate the high and low values for both warning and critical events.

Three separate actions occur when a monitored value exceeds the defined threshold, as shown in Figure 8, Figure 9, and Figure 10:

Event automation plans. So far, the monitoring items we've looked at—graphs and thresholds—are useful for showing the value of monitors and raising events for exceeded values. However, these items are useful only if you’re actively displaying the Systems Director web interface. Event automation plans can be used to cause, among other things, an external notification when an event occurs so that you can get monitoring information outside of the Systems Director web interface.

An event automation plan links an event definition with one or more event actions. Keep in mind that event definition and event action are two separate items in Systems Director. Making event actions generic allows them to be used for multiple events.

There are two ways to create event automation plans: from the Automation task list and from a defined monitor. Creating the event automation plan from a defined monitor has the advantage of bringing any defined threshold values into the plan. Let's look at the steps you perform to create an event automation plan.

You first define a name for the event automation plan, then define parameters for the event itself, as shown in Figure 11. Notice that in this case, because the Create Event Automation Plan wizard was started from a defined monitor, the monitor and its threshold values were brought into the event definition.

Once you've defined the event, the next step is to select the event actions you want to take when the event occurs. Systems Director includes a predefined event to add an entry to the event log. You can create additional event actions from within the Event Automation Plan wizard—basically a wizard inside of a wizard. When you create an event action, a template of predefined actions, shown in Figure 12, is used to specify the necessary values for the event action.

As you can see, there are many actions that can be taken for an event, such as starting programs, logging the event, sending notifications to upstream management systems, and sending email alerts. Each event action has a form, or template, that lets you specify values specific to that action. For example, Figure 13 shows the form for the Send an e-mail action.

Most of the values on this form are self-explanatory; however, two that should be discussed are the Subject of message and Body of message fields. Notice that these two fields have event variables (as denoted with the ampersand [&] character). Additional event variables can be added to either of these fields through the Event variable and Target text field drop-down lists. In addition to the event variables, you can also add free-flow text to these fields. It's always a good idea to use the Test button to ensure that the values for the event action are defined correctly. Clicking OK will add the event action to the list of actions that can be selected for an event automation plan.

The final item you define for an event automation plan is the time frame for enforcing the plan. Typically, an event automation plan is defined to be enforced 24 × 7 (the default). However, you can define a more specific time frame, including day of the week.

Message Queue Monitoring

One of the really powerful monitoring capabilities that Systems Director provides is the ability to monitor message queues. Message queue monitoring requires the Platform Agent to be installed and discovered on the IBM i endpoint. As I’ll demonstrate, message queue monitoring enables monitoring of messages based on the message queue, the message ID, or even a string within the message text.

You define the message queue monitor by creating an event filter, which is part of the Automation task. After you provide a name and an optional description for the filter, your next step is to define the filter type. This step includes the check box Include IBM i message queue events, which for our purposes needs to be checked.

For message queue monitoring, the next step—defining the message queue—is really the key, as shown in Figure 14. The message queue itself needs to be defined; you can define an optional message ID as well. Note that more than one message queue/message ID definition can be provided to the filter.

You can define additional items in the filter, including the event Severity (Fatal, Critical, Minor, Warning, Informational, and Unknown), event category (Alert, Resolution), event sender (managed endpoint that issued the event), event text, and time range. The event text definition (if provided) can specify that any word match triggers the event, that all words need to be in the message to trigger the event, or that the exact phrase provided needs to be in the message to trigger the event. Also, the event text definition lets you specify case sensitivity. The default setting for all these additional items is Default, which indicates that the item doesn't further define, or restrict, the filter. As an example, the default setting for event severity and event category indicates that all severities and categories are included in the filter.

Once you've created the event filter, you can use it to trigger an event automation plan. Unlike the monitor events I demonstrated earlier, in which the event automation plan could be created directly from the monitor, for event filters you need to create the event automation plan via the Event Automation Plans automation task. The main point to note here is that after you provide a name for the event and define the endpoint (or group of endpoints) for the event automation plan, you need to select Advanced Event Filters for the event. Doing so will provide a list of event filters (including any created/customized event filters). The remainder of the event automation plan definition is the same as I described earlier.

Process Monitoring and Event Log Entries

Another item that can be monitored with Systems Director is processes. The Manage Processes task from System Status and Health provides a list of processes that are currently running on the endpoint. From this list, you can create a monitor on the process to generate an event for one of three conditions: Start, End, or Fail to start after reboot.

Another way to create an event definition is through entries in the event log. For example, you can create an event filter against an entry in the event log, then define an event automation plan against the filter.

Powerful Monitoring

The monitoring capability in IBM Systems Director provides a powerful ability to both watch for and react to a wide range of IBM i characteristics, including message queues, subsystem status, and system statistics. With help from Systems Director's diverse and granular monitoring capabilities, you'll be able to keep a close watch on the health of your IBM i system. For more information about IBM Systems Director on IBM i, drop me a note at askerwin@poweritpro.com.