This guide addresses the various problems an administrator might encounter when configuring and/or using the IBM® Lotus® Domino® server monitoring tools. The article especially focuses on how to use the Domino Domain Monitoring feature and includes advanced troubleshooting tips as well.
Contents
1 Introduction
2 Monitoring troubleshooting “101”
2.1 General troubleshooting techniques
2.2 Advanced troubleshooting techniques
3 Statistics
3.1 General
3.2 Platform statistics
3.3 Disk statistics
4 Debugging commands for monitoring
5 Troubleshooting monitoring performance
6 Other troubleshooting considerations
7 Conclusion
8 Resources
1 Introduction
Domino Domain Monitoring (DDM) is a simple way for monitoring servers in a single domain. The setup for DDM is configured in the database Events4.nsf (see figure 1).
Figure 1. View of Events4.nsf
DDM is a new feature as of Lotus Domino release 7 and is part of the Event task that runs on the Domino server. In Domino versions 7 and later, the Event task is loaded by default and is no longer configured via the ServerTasks line in the Notes.ini file, as was the case in previous versions of Lotus Domino.
Let’s begin by discussing the two different kinds of events, enhanced and simple.
Excerpt from the Administrator Help:
Enhanced events and simple events
A collection server collects two classes of event information, enhanced events and simple events. When you open an event, the Event document contains a Class type field in the upper-right corner which indicates whether the event is a simple event or an enhanced event. Enhanced events include events generated by a DDM Probe document, events generated by a Domino event generator, or any other event with specific target information appearing in the DDM event report. Target information includes servers, databases, agents, or a user-specified target. A simple event is any event that is not associated with or that does not contain specific target information. For example, most of the events that are reported to the Domino server console are simple events.
DDM filters are one of the methods of determining what kinds of events get reported to the DDM database. It is important to set the filters correctly as this determines what gets reported to the DDM.nsf database. DDM filters are similar to the event handler method of logging to a database.
The server collection hierarchy represents a way to organize data in DDM.nsf in a multi-tiered design. Collection is divided up into servers that collect from other servers and servers that are collected from. All servers collect/report data to their own DDM.nsf database.
Thus, when a server is designated as a collection server, it will contain data from both its own DDM.nsf as well as the data that is collected from the subordinate servers. The information is “rolled up” to the collections server via a form of selective replication.
DDM is designed for a single domain and for monitoring related to that single domain.
2 Monitoring troubleshooting “101”
This section addresses the basics of monitoring troubleshooting.
2.1 General troubleshooting techniques
Situation #1: DDM rollup is not working.
Solution: Is a server collection hierarchy set up to accurately reflect the rollup (see figure 2)? Do the Events4.nsf replica IDs match? Do the DDM.nsf database replica IDs match?
Figure 2. Server collection hierarchy showing collection-to-child relationship structure
Situation #2: Replication of DDM continues to occur despite a server being removed from the domain.
Solution: Check whether the problem server(s) have been removed from the server collection hierarchy (see figure 3). If so, then refer to the Advanced Troubleshooting Techniques section below.
Figure 3. Collection hierarchy showing the problem child server as the second entry
Situation #3: An admin is unable to assign events or change the state of an event, or to use a corrective action.
Solution: Check the ACL to verify the appropriate roles are assigned to the user within the DDM.nsf database (see figure 4).
Figure 4. ACL to DDM
Situation #4: An admin sees only one document, even though multiple events of the same type have occurred.
Solution: DDM combines all events of the same type into the same document and
increases the occurrence counter within the event itself, rather than creating a large number of the same document (see figure 5).
Figure 5. Event showing occurrence counter
Situation #5: After an event occurs, nothing appears to be reported about the event.
Solution: In order for an event to be generated, an event generator must first be created, after which a corresponding event handler must be created, to determine what should be done with the event when it occurs.
Once a handler is created for a particular event, we can handle it in several ways, including—but not limited to—logging it to a database like Statrep.nsf or sending an email response.
2.2 Advanced troubleshooting techniques
Situation #1: Replication of DDM continues to occur despite a server being removed from the domain.
Solution: Verify the servers are not still listed in the collection hierarchy. If they are not, then examine the hidden view called $vwStatus within DDM.nsf. There will be several documents there, but focus on any that references the old server as well as any documents that have the number 2 next to them.
Examine the documents within the $vwStatus view and click on the grey box once one of the documents are opened. Notice that the name of the undesired server to replicate with will be listed in either the ChildServers field (see figure 6) or ChildServerPend field (for example, 2:cn:collectionservername/o=organization).
Figure 6. Example document from the $vwstatus view showing the field of a child server
Remove those documents and then restart the Event task. When the Event task is reloaded, the DDM collection hierarchy will be repopulated in DDM.nsf. See IBM Support Technote #1255418 for additional information.
Situation #2: The ISPY task is loading automatically and runs, despite there being no entry in the Notes.ini file.
Solution: This is something new to DDM and Domino 7, whereby the ISPY task will
load automatically when specific DDM probes are enabled. Using the command from the Domino console screen “sh sched –ddm”, you can see what probes are enabled and which of those are using/needing the ISPY task.
Situation #3: Lotus Domino is reporting a message that duplicate documents were found in a specific view within Events4.nsf. Figure 7 shows an example of $Messages dealing with duplicate documents in a view.
Figure 7. Example $Messages view
Solution: To resolve this, issue a “tell event clean ($Messages)” command from the Domino console screen, or delete the duplicate manually.
3 Statistics
3.1 General
Statistics in general are created based on features used within Lotus Domino and, as such, are generated by usage. For example, a server that only routes mail via NRPC will only generate mail statistics based on the NRPC routing (see figure 8). This means that Lotus Domino will not contain statistical information for SMTP stats like Mail.TotalRouted.SMTP.
Figure 8. Sample NRPC mail stats
Monitoring statistics and their threshold values can help administrators monitor for potential problems in their environment. For example, administrators could monitor for the available space left on their hard drive, to warn them when the server is approaching a critically low level of available free space.
An administrator might also use statistics to monitor things like the number of dead mail messages, which can indicate a possible mail routing issue.
Follow these steps to set up stats and events to monitor server statistics for a particular threshold value:
1. The server tasks Event and Collect must be running to allow statistics to be monitored, alarmed, and reported.
2. Set up a Statistic Collection document for the server (see figure 9), setting the alarm interval to how frequently the statistics should be examined, to determine it they exceed the threshold.
Figure 9. Server Statistics Collection document
3. Create an Event Generator to generate an event when a threshold is reached. In the example in figure 10, we use Mail.Dead.
Figure 10. Event Generator
4. Create an Event Handler to handle the Mail.Dead event. Now in the Statrep.nsf database, an alarm will be generated, according to our alarm interval and based on that an event will be generated and handled accordingly (see figure 11).
Figure 11. Example results after setting up several statistics to monitor and alarm
After the alarm has been written to the Statrep database, subsequent alarms will only increase the occurrence rate’s numeric value; further event generators and handlers will not fire off until the next day. Basically only one event will be reported per day for a specific statistic setup.
If a statistic needs to be reported more than once, then the Alarms view within Statrep.nsf would need to purged and not contain any entries for that alarmed statistic (see figure 12).
Figure 12. Blank view of a recently purged Statrep database
Restarting the Collect task would allow the entries to be purged out, and according to the alarm interval, they would be reported as needed again.
3.2 Platform statistics
Situation: Platform statistics are not being monitored and do not appear in the Domino console screen when the command “show stat platform” is issued.
Solution: Platform statistics must have a Windows registry value updated in order to be able to generate platform statistics. This update is not created or added by default by Domino version 7 or 8; however, the notesreg.bat file located in the Domino program directory allows for an automated way to update the registry.
An example of how to run this would be to shut down the server, navigate to the Domino program directory, and then run the bat file:
notesreg.bat "c:\Program Files\Lotus\Domino"
where everything in quotes is the program directory for Lotus Domino.
Next, restart the entire machine. Now the platform statistics on the server should be created when the server is running, and after you issue a “sh stat platform” all the platform stats should display (see figure 13).
Figure 13. Example platform stats
3.3 Disk statistics
Situation: Issuing a “sh stat disk” command does not correctly display all remote drives the server may be using.
Solution: Using the Notes.ini parameter EnableRemoteDiskStats=1 will enable an
administrator to see the needed information regarding remote drives (see figure 14).
Figure 14. Example “sh stat disk” display
4 Debugging commands for monitoring
The following are the debugging commands for monitoring and a description of what each does:
§ sh sched –ddm (shows the schedule and what is responsible for triggering the event ISPY, Event, etc.)
§ debug_event=1 (displays debug about events in general)
§ tell event dump (shows event information and memory allocated, as well as the current amount that can be used for events)
§ tell event dumpprobes (displays the list of currently enabled ddm probes)
§ tell Event fire 0x0000 (kicks off a probe manually for testing purposes)
§ debug_ddm=1 (includes additional debug information in the Event document for ddm)
5 Troubleshooting monitoring performance
Sometimes the number of events occurring can cause the available event pool to become filled with messages, in which case the server cannot accurately process the current events before additional events are created. An example error message is:
“Warning: Cannot record event - cannot keep up with event occurrence rate!”
You can resolve the issue by removing the amount of processes being monitored, by disabling the event generator and handlers so that less is being monitored, or by simply increasing the pool size to be used.
To increase the pool size, add one of the following to your Notes.ini:
· Event_Pool_Size= size size = x*1024*1024, where x is an integer between 5 and 100 (Must be using Domino 7.0 or later)
· Event_Pool_Size= size size = x*1024*1024, where x is an integer between 5 and 10 (Must be using Domino 6.5.5 or later)
After adding the above INI entry, restart the server. Now the memory pool can grow beyond what was previously set in the default, allowing additional events to be created and handled without the error.
6 Other troubleshooting considerations
Starting in Domino version 7, if replication needs to be monitored, then DDM must be used because the older methods of checking for replication with event generators will no longer be accurate.
To set this up, you must configure and enable a DDM scheduled replication check probe (see figure 15)
Figure 15. Replication probe
After setting up and enabling a scheduled replication check probe within DDM, you will see the information shown in figure 16 reported on the console screen.
Figure 16. Console screen report
For example, let’s say we want to handle this message automatically using an Event Handler (see figure 17).
Figure 17. Event Handler
In this case, figure 19 shows what some reporting would like each time the message is seen on the console screen. In this example, it is being logged to a database (Statrep).
Figure 19. Statrep logging screen
7 Conclusion
Monitoring tools are a powerful way for an administrator to be proactive, and they can be used as a first-line defense alert system for potential problems in any Domino environment.
DDM is one of the new mechanisms available that further add to the suite of monitoring tools. DDM can help to distribute the load of administrators through the assigning of events and is a much more powerful method of information delivery compared with pre-Domino 7 methods of event monitoring. In general, the monitoring tools help an administrator be aware of potential problems, helping to maintain a healthy server.
8 Resources
Lotus Notes and Domino wiki article “Domino Domain Monitoring (DDM)”:
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/domino-domain-monitoring
IBM Redpaper publication, “Lotus Domino Domain Monitoring”:
http://www.redbooks.ibm.com/abstracts/REDP4089.html?Open
developerWorks Lotus article, “Troubleshooting application performance: Part 2: New tools in Lotus Notes/Domino 7”:
http://www.ibm.com/developerworks/lotus/library/app-troubleshooting2/
IBM Support Technote: “DDM attempts replication with servers not in hierarchy or in replication schedule”:
http://www-01.ibm.com/support/docview.wss?uid=swg21255418