DAOS

Abstract

ShowTable of Contents

HideTable of Contents

1 Planning
- 1.1 DAOS Estimator
2 DAOS Configuration
3 Enable DAOS for Databases
4 Enable DAOS for Mail Box
5 DAOS Maintenance and Troubleshooting
6 DAOS Backup and Restore
7 Further Reading

Table of Contents | Previous Page | Next Page

Domino Attachment and Object Service (DAOS) is a large object store for Domino where its content is outside of the Notes database. The common content is stored only once, meaning that it consolidates the storage of multiple attachments for all the users hosted on a particular Domino server. On the shared content, DAOS manages reference counts. Since DAOS is a server-based feature, the access to its objects is totally transparent to the Domino server tasks (HTTP, Replication, FT Indexing) and applications (C-API calls, Agents, LotusScript).

A DAOS object is called a Notes Large Object (NLO). For each attachment processed by DAOS, an NLO file is created in the DAOS store, that is a simple folder at the operating system level. Each NLO file is created based on the checksum of the content of the attachment represented. Using this checksum process, DAOS can determine if the attachment is the same or different than an existing one. For security purposes, each NLO is encrypted by default with the server's key to protect it.

Planning

Now that we know what DAOS is, what steps should be taken in the planning for its deployment?

First, we need to determine where we will put the DAOS store. The DAOS store is one folder at the operating system level, under which all the NLO files are stored.

We need also to plan for the minimum size of the attachments that will be put into the DAOS store. If we choose a size too small, the DAOS server will create lots of NLO files, thus using more disk space than really needed. On the other hand, if we choose a size too large, we lose the potential the savings that DAOS was designed to provide.

Another thing we need to determine is how long we hold on to DAOS objects once the last reference is deleted. One key consideration is the frequency of database restores. An important thing to understand is that databases and NLO files are backed up independently, meaning that you have to change your Domino backup jobs to include the DAOS store, just like any other folder. When you restore a database, you have to make sure that all the necessary NLO files are present, or, if not, restored from backup as well. Finding the right balance will save lots of time and effort when restoring databases. Once the last reference is deleted, if you purge the NLO files too soon, you'll have to restore them when you restore a database, requiring more operational administration. If you keep them too long, you're losing valuable disk space and might have to add extra disks for growth, which reduce the disk space savings.

The last thing to plan is security for the DAOS repository. Should it be encrypted or not? This depends on how secure the Domino server is. By default, NLO files are encrypted using the server key.

This means that you cannot copy a .nlo file to another DAOS directory for another server. It won’t work. Encryption can be turned off by using the variable “DAOS_ENCRYPT_NLO=0” in the server's Notes.ini file.

DAOS Estimator

The DAOS Estimator tool will assist in the DAOS planning phase. This tool examines a set of Notes databases and provides a summary with the following information:

Total size of NSFs examined
Total attachments found
Total duplicate attachments found
Total DAOS eligible attachments
Estimate size of DAOS store (folder)
Total disk savings

This tool is compatible with Domino 6.x and higher, meaning that you can run this tool and get a detailed listing of the above information for every Domino 6.x and higher server in your environment.

To download the DAOS Estimator tool, please follow the instructions in IBM Technote 4021920 “Download the Domino Attachment and Object Service Estimator Tool version 1.4”

Now that we know how to get DAOS estimates, let's continue with our planning!

To plan for DAOS disk space needs, we need to understand the following concepts:

DAOS only supports a single base path and it manages subdirectories below this base path. The base path can be anywhere, but it is recommended to be in a location other than under the Domino data directory. The base path should have enough capacity to hold all existing attachments and some projected growth. Use the information provided by the DAOS Estimator. It is recommended to use flexible, configurable and expandable disk storage to avoid running out of disk space for the base path.
Since the typical I/O rates to DAOS are very low, it is a good candidate for low-cost storage. Because typical users are not constantly opening or detaching attachments all the time, there's no need for high-speed disks. One exceptions would be if you are using Full-text indexing on attachments, or if you are running third-party applications that may run agents or scan attachments like anti-virus softwares.

DAOS Configuration

Now that we meet the requirements and have our plan in place, we can configure the DAOS feature on a Domino 8.5 server.

The settings we need are the following:

Minimum size of object before Domino will store in DAOS - This is the minimum size (bytes) of attachments that will be stored in DAOS. Anything below this size will remain as a fully independent attachment in the Notes database.
DAOS base path - This is the base path where the DAOS store will be located.
Defer object deletion for - Once the last reference is deleted, this is the number of days that the NLO file will remain in the DAOS store until it is purged.

For the minimum size of object, use the DAOS Estimator to choose your best cost/benefit model. Based on feedback from early adopters, 64KB (64000 bytes) is a good starting minimum, but keep in mind that this value can be changed anytime later.

For larger deployments, we recommend to set a higher value first, and later decrease ( and compact again)

For defer object deletion, this interval should be tuned to your restore frequency. For example, if your average database restore frequency is once a month or less, use 30 days (default). If it's more than once a month, evaluate your own cost/benefit ratio of keeping NLOs versus restoring them on an as-needed basis.

To configure and activate DAOS on a server:

Modify the server's NOTES.INI file (or change the servers configuration document / notes.ini section) by adding the following setting:
CREATE_R85_DATABASES=1
Edit the Server document, go under the DAOS tab and set the following settings:

Note: Although the default value for “minimum size…” of 64kByte is good for new deployments, we recommend to start using an higher value (for example, 1024000) first, and decrease it at a later stage. When applying changes to this value, actions in “Enable DAOS for Databases” (below) must be performed again.

Save the Server document and Shutdown the server.
Restart the Domino server
Configure your OS Backup to back up the DAOS repository

Since DAOS files will never change, a monthly full backup and daily incremental backups are enough.

Note: You may want to configured your operating system Antivirus scanner to exclude the DAOS directory, or at least exclude *.NLO from scanning, otherwise the server performance will be impacted.

Enable DAOS for Databases

Now that the server has been configured to support DAOS, you can enable the first databases for DAOS.

To enable DAOS on all databases use the following server command.
load COMPACT -daos on
OR
Use the Administration Client to multi-select databases under the File tab and use the Advanced Properties option.
Note: Both methods will actually not enable DAOS, instead a setting within the database properties will be set for being used when compacting the database (see next step)
To DAOSify existing databases
load COMPACT -c
To enable DAOS and compression in one single step:
load COMPACT -c -v -n -daos on

This command will move the existing qualifying attachments into DAOS, compress existing design documents and compress existing data documents

Enable DAOS for Mail Box

Starting with Version 8.5.1, Domino is able to benefit

Run compact on the template, mailbox.ntf on your server in order to upgrade it to ODS 51. By default, it is ODS 43. Use copy style compact to upgrade the template to the current level (ODS 51).

Run compact on mail.box
Still working with mail.box, click File > Application Properties and select Use Domino Attachment and Object Service. LZ1 is selected by default.
Exit the Administrator client. Shut down the server and delete your mail.box at the OS level.
Start the Domino server. A new mail.box will be created based on the altered mailbox.ntf.
Start the Administrator client.
Verify that the newly created mail.box is configured for DAOS:

1. Select mail.box in the Files tab

2. SelectFile > Application Properties

3. Verify the information tab shows ODS 51

4. Verify Disable transaction loggingis not checked

5. Verify “Use LZ1 compression for attachments” is checked

Important Note: For the option “Use LZ1 compression for attachments”, make sure to select the same value as you configured for all the databases on the same server. If not the same, you will waste disk space.

6. Verify “Use Domino Attachment and Object Service” is checked.

DAOS LZ1 Settings

DAOS Maintenance and Troubleshooting

This section provides some concepts and tips on how to maintain and troubleshoot DAOS on a server.

One important component of DAOS is its catalog. By default, it's called "daoscat.nsf". This catalog is an index that keeps track of all the references between NSF files and NLO files. This file is created when the DAOSMGR task is launched for the first time, when DAOS is first configured on a Domino server.

The DAOS catalog can have either a status of "synchronized" or "unsynchronized". One way of becoming "unsynchronized" is if an administrator manipulates the NLO files at the OS level, or if your restore NLO files from a backup. To re-synchronize it, run the following command at the server console:

tell daosmgr resync

Each Domino 8.5 server with DAOS enabled maintains their own DAOS store and catalog, independently from other servers. DAOS-enabled databases are not transportable from one server to another because all its attachments are referenced only to its main hosting server. If you wish to create a new replica or move a DAOS-enabled database to another DAOS server, there are two ways to do it:

Create a new copy or replica of the database locally or on a non-DAOS server
Run the following command at the server console:

load compact -c -daos off

These actions will put all the attachments back into the database. Then, you'll be able to create a new replica or move the database to another DAOS-enabled server.

Note: You do not need to do these actions if you create a replica or move a DAOS-enabled database to a Domino server 6.x, 7.x or 8.0.x, or a Domino 8.5 server without DAOS configured. The attachments will come back into the database automatically with no further action required.

Although the physical file size of the database is reduced, if quotas are implemented in an environment, they will still be enforced. Even with DAOS implemented, the quota still checks against the Notes database size, which is calculated based on the current database size plus the attachments. This is working as designed to take into consideration the possibility of rolling back a database to a non-DAOS state.

In a clustered environment, it is important to keep in mind that the DAOS catalog and DAOS store are server-independent, meaning that every server has their own separate DAOS catalog and store. These components are not synchronized in a clustered environment. Even though the Notes databases are clustered, their attachment references in the DAOS catalog are different, depending on which server you open the database.

DAOS Backup and Restore

This section provides some information on how backup and restore works when DAOS is enabled on a Domino 8.5 server. This information is applicable to both single and clustered Domino servers because the DAOS catalog and store are specific for each server.

All backup solutions supported on Domino 8.5 should still work on DAOS-enabled servers. As mentioned earlier, DAOS is transparent to existing C-API calls. If you run a backup client on your Domino server that supports Transactional Logging, backups and restores remain unchanged from an NSF perspective. For a server with DAOS configured, you will have to modify your Domino server backup job to include the DAOS store. One recommendation would be to do full backups of the DAOS store during weekends, and incremental backups during weekdays because NLO files are either added or deleted, not modified.

Running DAOS on a Domino server provides important cost savings from a backup perspective:

Reduced backup volume - Takes less tapes to backup a server
Reduced backup processing - Takes less time to backup a server

Restoring a Notes database on a Domino 8.5 with DAOS may require some manual steps in some specific cases, because there is no existing backup APIs integrating with DAOS. For example, when you do a point-in-time database restore, the NSF gets restored as expected, but you have to make sure that all NLO files contained in the NSF at the time it was backed up, are present in the DAOS store. This step is done manually and separately, meaning that once the NSF is restored, you have to restore the NLO files from DAOS store. There is a simple command that will let you know which NLO files are missing from the restored NSF file. This will assist you in only selecting the missing NLO files from your backups instead of restoring the whole DAOS store. The command is the following:

tell daosmgr listnlo missing

IMPORTANT NOTE: NLO files are encrypted with the server key by default, thus not portable between servers.

Once restored, you run this command from the server console to resynchronize the DAOS catalog with the DAOS store: