DAOS: How To Setup An
Automatic Catalog Resynchronization Event
If DAOS determines that any of the reference counts on the attachments
it manages are no longer trustworthy (which is not to say they're not necessarily
accurate, but rather that a condition has been introduced such that they
can no longer be relied upon with 100% confidence), it puts the catalog
into a safe mode of sorts and refuses to permanently delete NLOs until
the catalog has been resynchronized – a process of full accounting issued
by the administrator at an appropriate time. An example of how this
could come about is when a DAOS nsf has been copied at the OS-level rather
than through the Notes client.
In order to determine the state of the DAOS catalog and whether it needs
resynchronization, you can use the "tell daosmgr status catalog"
command from the server console. If all is well, it will be in a
And, if you should find it in a "NEEDS RESYNC" state:
you would only have to issue a "tell daosmgr resync" in order
to start the healing.
NOTE: The duration of a resync can
be significant and depends on the number of DAOS-enabled databases, the
number of NLO files in your environment, and your system configuration.
It can take several hours to complete causing its execution to overlap
into normal business hours. Although Domino and DAOS are functional
while a resync is in progress, there may be a degradation in performance
while it is running.
If you find it necessary to halt the resync
operation, you can interrupt it by issuing “tell daosmgr quit” from the
server's console. However, for continued operation of DAOS, immediately
restart the DAOSMGR task using the console command “load daosmgr.” When
it is convenient to continue the resync operation, issue “tell daosmgr
resync” again from the server's console and resync processing will continue
where it left off.
This document describes how such a resynchronization can be set up to run
automatically whenever the catalog goes into a "NEEDS RESYNC"
The way this state change is recognized and subsequent processing occurs
is via the DDM (Domino Domain Monitoring) and event handling system. DDM
enables the administrator to select a set of conditions to monitor or "probe,"
and define a means of reacting to the messages under watch, including resolving
without manual intervention.
Both the probe and the event are created using the Monitoring Configuration
database (events4.nsf). Let's step through a typical scenario as
it relates to the DAOS catalog needing resynchronization.
From the DDM "Probes By Type" view, hit the "New DDM Probe"
button and select Database. Choose "Database Error Monitoring"
as the Subtype and provide a description. You can then target specific
servers or generalize. As for the severity, in this case the Needs
Resync message we wish to handle is considered a "Warning (low)."
Save and Close. The checkmark indicates the probe is enabled.
So far, we have it so all "Warning (low)" severity database errors
are tracked and logged to ddm.nsf. The next step is to configure
a response, to set up some automation surrounding the specific case of
the DAOS catalog needing resynchronization. That could be as simple
as sending the administrator an email or it could involve actually performing
or scheduling the resynchronization itself.
In short, we want to create a new event handler. To do this, open
the Event Handlers folder below the DDM Probes in the left-hand pane and
select By Type. Hit the "New Event Handler" button.
The first section asks whether you'd like to restrict the notification
to a single server or all on the domain. Under the triggers, choose
"A built-in or add-in task event," and proceed to the Event tab.
From the Select Event dialog, we're going to pick the message with code
0x1BFA, which indicates the catalog needs resynchronization due to an NSF
being modified or deleted from the filesystem.
The full list of DAOS DDM messages is comprised of the following:
0x1BF0 "The DAOS catalog does not exist."
0x1BF1 "The DAOS catalog cannot be created.
DAOS cannot operate normally."
0x1BF2 "The DAOS catalog cannot be updated."
0x1BF3 "The database %p attempted to access
a missing file: %p"
0x1BF4 "The database %p was unable to open
or read the file %p"
0x1BF5 "The database %p was unable to write
to file %p"
0x1BF6 "The database %p could not be read
0x1BF7 "The database %p appears to have
changed at an OS level."
0x1BF8 "DAOS was unable to rebuild the
list of external files while trying to resynchronize."
0x1BF9 "DAOS was unable to scan the database
%p to gather its DAOS tickets while trying to resynchronize."
0x1BFA "Informational - The database %p
has caused the DAOS catalog to become out of sync. Deletions will be postponed.
Please run 'tell daosmgr resync' at the next convenient opportunity
0x1BFB "DAOS encountered an initialization
0x1BFC "The DAOS catalog cannot be opened.
DAOS cannot operate normally."
0x1BFD "The DAOS catalog cannot be resynchronized.
DAOS deletions will be postponed."
0x1BFE "Invalid ticket encountered in database
0x1BFF "Informational - The DAOS catalog
is not synchronized. Deletions will be postponed. Please run 'tell daosmgr
resync' at the next convenient opportunity to re-synchronize."
On the next tab, we'll specify the action to take when this error is encountered.
For purposes of this demonstration, let's have it issue a console
command to immediately resynchronize the catalog. In a production
environment, you may want to schedule this for less active periods of server
use as it can take some time. In the next section, we'll cover how
this can be accomplished using "program documents."
To test this, shutdown the server, rename a DAOS-enabled database and restart.
The missing database will be detected during DAOS initialization
and the error will be fired.
Further recommended reading: Lotus Domino Domain Monitoring Redpaper for
Domino 7, by Philip Monson, Thomas Gumz, Frank Nostrame, Leah Busque, December
2005 - http://www.redbooks.ibm.com/redpapers/pdfs/redp4089.pdf
DAOS: How To Setup A Regularly Scheduled Resynchronization
Many production systems are simply too large or mission critical to resynchronize
the DAOS catalog immediately as just described. To that end, you
may just wish to send yourself a page or email using a DDM notification,
and let a regularly scheduled process handle the resynchronization at a
more convenient time.
The best way to do this is via a "program document" defined in
the Administrator. It causes no harm for the resynchronization to
be run when the catalog is already synchronized. In such a case,
the fact that no action is required is quickly recognized and the process
Even run during non-business hours, there are other factors to consider.
The DAOS "prune" task and database deletion scan are similarly
configured to execute around 2 AM, and neither will attempt their assignment
if the catalog is in need of or is currently being resynchronized. They
will merely exit and try again the next evening.
Let's take a look at how you would create a regularly scheduled catalog
resynchronization. From the Administrator, select Programs under
the Server menu in the left-hand pane.
Press the "Add Program" button and complete the Basics tab as
follows. The program name is the server itself (enter "server"
without the quotes). In the Command line field, we'll use the
-c option to pass a tell command to the server. In this case, that
looks like this:
-c "tell daosmgr resync" (this time you do want the quotes).
Indicate the appropriate server and leave an optional comment.
Further to the right of this same tab is the scheduling section. The
best times to run the resynchronization will be different for each administrator,
so this is just an example of how it would appear for the scenario we've