DAOS: How To Set Up Resynchronization Events With DDM

If DAOS determines that any of the reference counts on the attachments it manages are no longer trustworthy (which is not to say they're not necessarily accurate, but rather that a condition has been introduced such that they can no longer be relied upon with 100% confidence), it puts the catalog into a safe mode of sorts and refuses to permanently delete NLOs until the catalog has been resynchronized – a process of full accounting issued by the administrator at an appropriate time.  An example of how this could come about is when a DAOS nsf has been copied at the OS-level rather than through the Notes client.

In order to determine the state of the DAOS catalog and whether it needs resynchronization, you can use the "tell daosmgr status catalog" command from the server console.  If all is well, it will be in a "SYNCHRONIZED" state:


And, if you should find it in a "NEEDS RESYNC" state:


you would only have to issue a "tell daosmgr resync" in order to start the healing.

NOTE:  The duration of a resync can be significant and depends on the number of DAOS-enabled databases, the number of NLO files in your environment, and your system configuration.  It can take several hours to complete causing its execution to overlap into normal business hours.  Although Domino and DAOS are functional while a resync is in progress, there may be a degradation in performance while it is running.

If you find it necessary to halt the resync operation, you can interrupt it by issuing “tell daosmgr quit” from the server's console.  However, for continued operation of DAOS, immediately restart the DAOSMGR task using the console command “load daosmgr.”  When it is convenient to continue the resync operation, issue “tell daosmgr resync” again from the server's console and resync processing will continue where it left off.

This document describes how such a resynchronization can be set up to run automatically whenever the catalog goes into a "NEEDS RESYNC" state.

The way this state change is recognized and subsequent processing occurs is via the DDM (Domino Domain Monitoring) and event handling system.  DDM enables the administrator to select a set of conditions to monitor or "probe," and define a means of reacting to the messages under watch, including resolving without manual intervention.

Both the probe and the event are created using the Monitoring Configuration database (events4.nsf).  Let's step through a typical scenario as it relates to the DAOS catalog needing  resynchronization.

From the DDM "Probes By Type" view, hit the "New DDM Probe" button and select Database.  Choose "Database Error Monitoring" as the Subtype and provide a description.  You can then target specific servers or generalize.  As for the severity, in this case the Needs Resync message we wish to handle is considered a "Warning (low)."


Save and Close.  The checkmark indicates the probe is enabled.



So far, we have it so all "Warning (low)" severity database errors are tracked and logged to ddm.nsf.  The next step is to configure a response, to set up some automation surrounding the specific case of the DAOS catalog needing resynchronization.  That could be as simple as sending the administrator an email or it could involve actually performing or scheduling the resynchronization itself.

In short, we want to create a new event handler.  To do this, open the Event Handlers folder below the DDM Probes in the left-hand pane and select By Type.  Hit the "New Event Handler" button.  


The first section asks whether you'd like to restrict the notification to a single server or all on the domain.  Under the triggers, choose "A built-in or add-in task event," and proceed to the Event tab.

From the Select Event dialog, we're going to pick the message with code 0x1BFA, which indicates the catalog needs resynchronization due to an NSF being modified or deleted from the filesystem.  


The full list of DAOS DDM messages is comprised of the following:

0x1BF0        "The DAOS catalog does not exist."
0x1BF1        "The DAOS catalog cannot be created. DAOS cannot operate normally."
0x1BF2        "The DAOS catalog cannot be updated."
0x1BF3        "The database %p attempted to access a missing file: %p"
0x1BF4        "The database %p was unable to open or read the file %p"
0x1BF5        "The database %p was unable to write to file %p"
0x1BF6        "The database %p could not be read by DAOS."
0x1BF7        "The database %p appears to have changed at an OS level."
0x1BF8        "DAOS was unable to rebuild the list of external files while trying to resynchronize."
0x1BF9        "DAOS was unable to scan the database %p to gather its DAOS tickets while trying to resynchronize."
0x1BFA        "Informational - The database %p has caused the DAOS catalog to become out of sync. Deletions will be postponed. Please run 'tell  daosmgr resync' at the next convenient opportunity to re-synchronize."
0x1BFB        "DAOS encountered an initialization failure."
0x1BFC        "The DAOS catalog cannot be opened. DAOS cannot operate normally."
0x1BFD        "The DAOS catalog cannot be resynchronized. DAOS deletions will be postponed."
0x1BFE        "Invalid ticket encountered in database %p."
0x1BFF        "Informational - The DAOS catalog is not synchronized. Deletions will be postponed. Please run 'tell daosmgr resync' at the next convenient opportunity to re-synchronize."


On the next tab, we'll specify the action to take when this error is encountered.  For purposes of this demonstration, let's have it issue a console command to immediately resynchronize the catalog.  In a production environment, you may want to schedule this for less active periods of server use as it can take some time.  In the next section, we'll cover how this can be accomplished using "program documents."


To test this, shutdown the server, rename a DAOS-enabled database and restart.  The missing database will be detected during DAOS initialization and the error will be fired.

Further recommended reading: Lotus Domino Domain Monitoring Redpaper for Domino 7, by Philip Monson, Thomas Gumz, Frank Nostrame, Leah Busque, December 2005 - http://www.redbooks.ibm.com/redpapers/pdfs/redp4089.pdf


Many production systems are simply too large or mission critical to resynchronize the DAOS catalog immediately as just described.  To that end, you may just wish to send yourself a page or email using a DDM notification, and let a regularly scheduled process handle the resynchronization at a more convenient time.
The best way to do this is via a "program document" defined in the Administrator.  It causes no harm for the resynchronization to be run when the catalog is already synchronized.  In such a case, the fact that no action is required is quickly recognized and the process exits gracefully.

Even run during non-business hours, there are other factors to consider.  The DAOS "prune" task and database deletion scan are similarly configured to execute around 2 AM, and neither will attempt their assignment if the catalog is in need of or is currently being resynchronized.  They will merely exit and try again the next evening.  

Let's take a look at how you would create a regularly scheduled catalog resynchronization.  From the Administrator, select Programs under the Server menu in the left-hand pane.


Press the "Add Program" button and complete the Basics tab as follows.  The program name is the server itself (enter "server" without the quotes).  In the Command line field, we'll use the   -c option to pass a tell command to the server.  In this case, that looks like this:

  -c "tell daosmgr resync" (this time you do want the quotes).

Indicate the appropriate server and leave an optional comment.


Further to the right of this same tab is the scheduling section.  The best times to run the resynchronization will be different for each administrator, so this is just an example of how it would appear for the scenario we've been discussing.

No Comments Found