Configuring and testing the IBM Lotus Domino DAOS feature

Introduction

Introduced in IBM® Lotus® Domino® 8.5, the Domino Attachment and Object Service (DAOS) feature allows attachments to be stored outside the NSF file in Notes Large Object (NLO) files. This provides two major benefits in that it:

Allows multiple copies of the same attachment to be stored as a single copy to save storage.
Segregates the relatively static attachment data from the NSF data, allowing flexibility in data storage and backup processing.

In databases that use DAOS, Lotus Domino no longer saves a separate and complete copy of every document attachment. Instead, the server saves a reference to each attached file in an internal repository (DAOS Catalog), and it refers to the same file from multiple documents in one or more databases on the same server.

When a message containing an attached file is large and is broadcast to thousands of users, creating a separate copy of the message for each recipient could require several gigabytes of disk space. Also, multiple copies of the same attachment are often proliferated in mail threads with multiple replies.

With DAOS enabled, disk space usage is substantially reduced, and you can realize these additional advantages:

• Optimized mail routing of attachments
• Optimized object copying for any same-server copy operations
• Faster compacting and backing up of databases

IBM’s Domino System Verification Test (SVT) team performs extensive testing of the DAOS feature under load, to ensure that it works in a stressed environment. This article discusses some of the statistics you can use to ensure DAOS is running correctly, provides a few tips on how to prevent corruption, and describes how the SVT team configures DAOS in their environment.

Typical environment in SVT

The configuration used within the System Verification Team is as follows (see table 1):

• Number of mail files = 3000 mail files across one Domino Partition (DPAR)
• Size of attachments = 50 KB and 10 MB
• Number of NLO’s = 40,000
• Mail files and the DAOS directory containing NLOs reside on a Storage Area Network (SAN)

Table 1. SVT environment details

System	IBM System z990 – 2084-D32
Processor	64-way processor
Memory	6023MB
LPAR	CPUs: 8 Shared Memory: 64G Central storage, 16G Expanded storage
Guest	CPUs: 2 Virtual Memory: 6GB
Fiber channel adapter	18 Shared 2-GB FICON adapters
Disk drive	180 emulated 3390-9 and 16 emulated 390-3 on multiple DS8000 (roughly 1TB)
First-level operating system	ZVM 6.1
Guest operating system	RedHat Linux® (64bit) Release 6.1
Lotus Domino server	Domino 8.5.4 Build for zLinux (64bit)

Configuring the server and mail files to support DAOS

Before we start with the scenarios, we need a basic understanding of how DAOS works, how to enable it, etc.

Enabling DAOS on the server
1. In the Basics tab of the server’s Server document, set “Transaction logging” to Enabled (see figure 1).

Figure 1. Enable Transaction logging


2. In the DAOS tab of the Server document, enable DAOS with the settings shown in figure 2.

Figure 2. DAOS Settings


where:

• The “Minimum size of object before Domino will store in DAOS” field is set to 4096 for testing purposes only and is significantly lower than it should be for a typical production environment. Setting the minimum participation size that low in a production system would cause too many NLO files to be created, which would make the system difficult to manage.

• Setting the “Store file attachments in“ field to DAOS (that is, the DAOS repository will be stored under the Data directory) in a production environment can cause utilities to unnecessarily traverse the DAOS repository looking for NSF files. If there are a significant number of NLO files, there can be a significant performance impact on those utilities, so it's recommended they be in a location other than under the Domino data directory.

3. From the server console, issue a “show server” command, to report the status of DAOS.
4. By default the new DAOS folder is created under the Data folder; however, we can change that by specifying the directory under the DAOS tab in the Server document (see figure 3).

Figure 3. DAOS tab



A DAOS Catalog (daoscat.nsf) will be created under the Data directory (see figure 4).

Figure 4. DAOS Catalog listed in Data directory


Enabling DAOS on mail files
To enable DAOS on an individual mail file, run the command, “load compact –c mailfile.nsf –daos”. Note that compact can also be run against multiple mail files, using “load compact –c mail –daos on”.

Important DAOS commands

Some of the important DAOS commands and what they do are listed below:

Tell DAOSMgr Status. Displays status of various DAOS Manager Operations.

Tell DAOSMgr Status Catalog. Displays status of DAOS Catalog.

Tell DAOSMgr Resync. Resynchronizes DAOS-enabled databases with DAOS objects in the storage repository. This command won’t work if the DAOS Catalog is in a synchronized state.

Tell DAOSMgr Resync Force. Runs the resynchronization command, regardless of whether the DAOS catalog is in a synchronized state.

Tell DAOSMgr Resync quick. Rebuilds the DAOS Id Table (DIT) and DAOS Object Index (DOI) but will not scan the databases, where:

• DIT is a list of all NSF files participating in DAOS, containing the total number of DAOS references for each NSF,and the timestamp of the last DAOS activity in the NSF.

• DOI is a list of all NLO files in the DAOS repository, containing the total number of DAOS references to each NLO.

Tell DAOSMgr Resync Force quick. This command is a combination of the above two; that is, it forces the resynchronization process, even though Catalog is in a synchronized state and just rebuilds the DIT and DOI.

Table 2 lists the various Catalog states with their brief descriptions.

Table 2. Catalog states

UNAVAILABLE​	DAOS Catalog is not available.​
REBUILDING* ​	The DAOS Object Index (DOI) is being rebuilt.​
RESYNCING ​	Databases are being scanned for DAOS references.​
SYNCHRONIZED ​	All DAOS-enabled databases are in a known state.​
NEEDS_RESYNC ​	There is at least one DAOS-enabled database in an unknown state.

Table 3 lists the various Database states with their brief descriptions.

Table 3. Database states

Resync: Pending ​	Database is waiting to be scanned for DAOS tickets.​
Resync: Scanning ​	Database is currently being scanned for DAOS tickets.​
Resync: Updating Refs ​	Database's tickets are being processed, DAOS references incremented.​
Synchronized ​	Database is in a known state, all DAOS tickets counted in the catalog.​
Deleted ​	Database has been deleted and is being scanned for DAOS tickets.​
Deleted: Updating Refs ​	Deleted database's tickets are being decremented in DAOS catalog.​
Adopt: New DB* ​	A new database has been discovered and is waiting to be scanned.​

Test scenarios & evaluation criteria

In this section we discuss the scenarios that are tested by the SVT team to verify DAOS is running correctly.

Scenario 1: DAOS Resync time window (Performance test)

This scenario tests that Resync picks up where it left off when resync is confined to a time window.

1. First, we must add the following settings to the server’s Notes.ini file:


2. Before starting the test, we must also start performance monitoring (show stat & Open session) tools.
3. Apply load to some mail files on the server.
4. Issue a “tell daosmgr resync quick force” a few hours before the Resync window opens (see figure 5).

Figure 5. “tell daosmgr resync quick force” command


5. Issue a “tell daosmgr status” and confirm DAOS is resync'ing (see figure 6).

Figure 6. “tell daosmgr status” command


6. When quick resync is complete, issue a “tell daosmgr status“. DAOS will still be in a resync'ing state, which is expected. Before the resync window opens, issue a “tell daosmgr resync” (see figure 7). Resync should not start as outside the window.

Figure 7. “tell daosmgr resync” command


7. When the resync window opens, issue a “tell daosmgr resync”, which will start the resync process, and monitor the server around the end of the resync window. Resync will start to skip databases after the window closes.
8. Issue a “tell daosmgr status” and confirm DAOS is still resync'ing.
9. Before the resync window opens, issue a “tell daosmgr resync”. Resync should not start as outside of the window. When the resync window reopens, issue a “tell daosmgr resync”.
10. Confirm that DAOS skips the databases already resync'ed and starts at the databases from which it left off.
11. Monitor that the window ends, or confirm the resync is complete, as necessary.

Evaluation criteria:
Pass (if):

• DAOS resync stops when window closes and restarts where it left off when restart resync completes successfully.
• DAOS resync completes within a reasonable time for the number of mail files and NLO’s on the server (resync can take quite a long time, depending on the number of NLO’s), without significant impact on user response or test load.

Fail (if):

• DAOS resync carries on past the window.
• DAOS resync restarts from the first database when run on Day 2.
• DAOS resync runs outside of the window.
• There are signs of a server hang or crash during the test.