How do I determine when the DAOS catalog needs resync?
When the DAOS catalog transitions to the state of "NEEDS RESYNC" a DDM event is fired that will identify the database that first caused the catalog to go into this state. This information can be found in the DDM database or in the log.nsf.
The following is a list of operations that cause the DAOS catalog to transition to the state of "NEEDS RESYNC":
- copy a DAOS enabled database into the data directory from the OS
- delete a DAOS enabled database from the OS
- rename a database from the OS
- delete an NLO from the OS (will transition to "NEEDS RESYNC" when the NLO is attempted to be opened.)
- restore a database directly to the data directory
If you can not determine what action caused the catalog to "NEEDS RESYNC" and it continues to occur, debug code should be enabled. Setting
DAOS_LOGGING=c:\debug\daos.txt STATE_CHANGE
will log all DAOS state change activity. Also enabling DAOS_LOGGING will cause DAOS to output the call stack to the server console that caused the catalog to transition to this state. That should be given to support along with the daos.txt file to help troubleshoot the problem.
To clear the "NEEDS RESYNC" state run a resync by entering the command 'tell daosmgr resync'. Note that while the catalog is in this state DAOS will continue to operate normally. However deletion of NLO files by the PRUNE task will be postponed until the catalog is in the "SYNCHRONIZED" state again.
How do I determine if objects are going into DAOS?
You can look in the DAOS directory to see that NLO files are being created. Under the DAOS directory are subdirectories for the containers. Each subcontainer holds 40,000 NLO files by default.
Why are new attachments added to a database not creating new NLO files?
Verify that the database is DAOS enabled and the status is Read/Write. This is done by entering the command 'tell daosmgr dbsummary
dbname'
It is also possible that the attachment does not meet the minimum DAOS size requirement. Remember that the size of the object is compared to the DAOS Minimum after it is compressed.
The attachment already exists. Attachments are written to a temporary NLO file so that the MD5sum can be calculated. If that checksum matches an existing NLO, the temporary file is deleted. In this case the refcount for the object is incremented.
A problem was encountered when creating an NLO file. In this case the fallback is to immediately create a standard, inline NSF attachment. If anything goes wrong along the way (can't create NLO file due to out of disk space, write error, permission, etc.), the DAOS operation returns control to the normal NSF attachment handling code. Worst case, it'll get in-lined in the database.
How do I determine which databases are DAOS enabled?
Run the command 'tell daosmgr dbsummary'. It will display the list of DAOS enabled databases, their ticket counts and their DAOS state.
Why are NLO files not being deleted?
NLO files are cleaned up (deleted) by the DAOS Prune process that is scheduled to run nightly at 2:00 AM. NLO files are only removed when the DAOS catalog is in a 'Synchronized' state, the NLO refcount is zero, and the NLO was marked deleted longer than
the prune interval ago.
You can also run Prune from the console with the command 'tell daosmgr prune
prune interval'. An interval of zero will cause all NLO files with a refcount of zero to be deleted immediately. Use caution when here as you could potentially delete attachments that have never been backed up.
Why can't I open attachmen ts when I open a DAOS enabled database locally?
The DAOS references are only resolved through the Domino server. When you open the database locally and you try to open a DAOS attachment you will get the error “The object store database is disabled. : Could not save to file c:\tmp\notesEA312D\daosdb.nsf”
How do I troubleshoot DAOS related database problems?
In order to troubleshoot a DAOS related problem off line, or for support to diagnose the problem, the database along with all NLO files will need to be provided. You can obtain a list of all NLO files referenced by a database by running the command 'tell daosmgr listnlo '
Note that this is only necessary if the problem is with the attachment. Otherwise a resync of a database with missing attachments will result in 'Ghost' entries in the DAOS catalog, allowing for diagnosis of issues not related to the attachments.
How do I troubleshoot non-DAOS related database problems?
A DAOS enabled database will function normally as long as attachments are not opened. So if you are troubleshooting a problem that is not DAOS related you do not need to provide the NLO files.
Why are some DAOS NLO files missing?
One possible cause is the Anti-virus software found a virus and 'quarantined' the file. This can happen when compacting a database into DAOS and a virus is found on an attachment within the database.
You should set your anti-virus settings to match you data directory.
Also deleting NLO files from the OS will cause problems with DAOS.
Why can't I compact a database containing tickets with missing attachments?
This can happen due to the reasons above. When you encounter this problem you will need to run 'fixup -j -D'. The -j switch is for transaction logged databases and the -D switch will cause documents with DAOS tickets pointing to invalid or missing NLO files to be deleted. These documents are deleted without a deletion stub so they may replicate back in.
Why don't I see the "Use Domino Attachment and Object Service" option under File -> Application -> Properties?
If you are not running a Notes 8.5 client or later you will not see that option.
If the database is not ODS 51 or greater you will not see that option. Make sure CREATE_R85_DATABASES=1 is set in the notes.ini and compact -c the database. Adding the '-DAOS on' flag will directly enable DAOS before the compact.
Why do I get error when compacting a large database out of DAOS?
If the resulting database would exceed the 64GB NSF limitation with the attachments in-lined within the NSF then you can not compact the database out of DAOS.
Why am I getting the error "File Not Found" when replicating a DAOS enabled database?
If an NLO file is missing on one system and that document is replicated, the replicator will report this error. This will cause the replicator to continue to attempt to replicate this document on every replication. If this error is encountered three times in a row the replicator will abort replicating the database. This can be avoided by setting DEBUG_REPL_TOLERATE_ERRORS but does not resolve the problem.
There are two ways to resolve this problem:
- Restore the NLO file.
- Run 'fixup -j -D' to delete the document referencing the missing NLO.
Why does DAOS require Transaction Logging to be enabled?
Storing attachments in DAOS involves coordinated actions for three different files: the Notes database receiving the attachment, the DAOS file, and the daos catalog (index). Transaction logging is used to guarantee that we can put these distributed pieces of the transaction back into a consistent state after a server crash (T/L redo) or catastrophic failure (recover roll T/L forward). Transaction logging is a requirement for DAOS and you simply cannot use the feature without it. Overall, using transaction logging increases the consistency and integrity of all NSF data.
The one area where T/L is optional with DAOS is mail boxes (e.g. MAIL.BOX). T/L mail boxes is optional. There is a performance enhancement that allows optimized routing of messages with attachments to DAOS enabled mail files.
Not enabling T/L on mailboxes does not affect the disk space savings from using DAOS. It only disables a performance optimization.
If I make the minimum participation size larger, will attachments stored in DAOS be moved back into the NSF?
Yes, but not automatically. Changing the minimum participation size in the DAOS tab of the server doc will affect the storage location of any attachments written from that point forward. If you run a copy-style compact (compact -c somefile.nsf) on an NSF that is already participating in DAOS, attachments that are in DAOS but no longer meet the minimum participation size will be stored inline in the resulting NSF. The reference count for the NLO file will be decremented to reflect this. If the resulting reference count reaches zero, the NLO file becomes a candidate for deletion.
Is there one NLO file per NSF, or one per attachment, or...?
There is one NLO file created for each unique attachment among the NSF files participating in DAOS on a single server. Each NLO file contains the data for a single unique attachment. If there are multiple attachments in an NSF, or even in a single document, a separate NLO file will be created for each unique one.
I have clustered servers, and they all have DAOS enabled. Can I share a single DAOS repository between clustermates?
No. Each server participating in DAOS must have its own separate DAOS storage path.
Do I need a special client version to use DAOS?
No. DAOS is a Domino server feature, and is transparent to the client. Any client (older Notes versions, web, mobile, etc.) will be able to read and write attachments to an NSF with DAOS enabled on the Domino server. In 8.5.1, a new feature was introduced to the Notes client that takes advantage of DAOS to eliminate transmitting attachment data that already exists in the destination DAOS server repository. Other clients will still work, but they can not take advantage of this, and will always transmit the attachment data.
How can I enable DAOS automatically for all new mailfiles?
To enable DAOS on all new mailfiles created on the server, modify the mail85.ntf (template) to have 'Use Domino Attachment and Object Service' enabled in the advanced properties. When you select the mail85.ntf, it will automatically include settings. This setting will not take effect until a database is at ODS 51. To create databases by default as ODS 51, enable the parameter 'Create_R85_Databases=1' in the server's notes.ini file. (While you are editing the template, it's a good idea to enable the 'Use LZ1 compression for attachments' property also to save even more space.)
Is it OK to delete daoscat.nsf and/or daos.cfg?
In most cases, no. For details, see technote 1503667:
http://www-01.ibm.com/support/docview.wss?uid=swg21503667
Is there a detailed DAOS rollout document available?
Yes! Please see the DAOS Deployment Guide:
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/DAOS_Deployment_Guide