Table of Contents
Without an index, finding the data you are looking for would be extremely difficult. There are 3 different types of indexes used when using Lotus Domino -- view indexes, database full text indexes, and Domain indexes. There are also multiple tasks involved with Domino indexing including update, updall, chronos, kvoop and domidx. If you are new to Domino administration, this can be overwhelming. This article gives you the information you need to get started with and manage Domino indexing.
What is a view index? Anytime you open a Domino database, you are seeing information presented based on a view index. You could be looking at the Inbox of a mail file, a personal folder or a view in a custom application, but how that data is presented to you and whether or not you are seeing the latest update depends on the view index. Since view indexes are stored within a Domino database, you may wonder how to see if a view index exists, if it is up to date or the size of the index. The Domino Administrator provides a tool to easily see this information. To access the tool you will go to the Files
tab in the Domino Administrator client and right-click on the database you want to see. You should then select Manage Views
as seen in figure 1.
Within the Manage the views of this database
window, you can see the name of each view index, the index size, the creator of the view, refresh interval, discard interval and the internal note id representing the view. In figure 2 you can see the views for a typical mail file.
Now that you understand how view indexes are used and where they are stored, you may want more information on how the view indexes are updated and the tasks responsible. For this information refer to the Domino administration help topics “Indexer tasks: Update and Updall
” and “Keyboard shortcuts that update or rebuild views
Full Text Indexes
A full text index is an index designed to be used for searching within a Domino database. The full text index may only contain words found within a document or may optionally include text within numerous supported attachment types. A full text index is created and stored outside of the application in a .ft folder. A user can easily see if their mail database has a full text index by the icon displayed above the search bar as shown in figure 3.
When creating a full text index, there are several choices that need to be made including the update frequency and whether or not attachments should be read and included in the index. Be default, attachment indexing is turned off, but the default update frequency is different depending on the method used to create the index as shown in figure 4. If a user selects to create the index from within their mail file by clicking Not indexed
as seen in figure 3 the index will be created with an immediate update frequency. This is typically not recommended for most applications. When creating the index from the Domino Administrator, the default is daily updates, which is recommended for most applications.
You will also notice that the wording for the attachment indexing is also different. The most complete attachment indexing is binary indexing performed by the kvoop task which implements KeyView technology
for reading the attachments. This type of indexing is referred to as with file filter
in the Domino Administrator client and called conversion filters
on the database properties Create Full-Text Index
There are additional options which can include indexing encrypted fields, sentence and paragraph breaks and case-sensitive searches. In general it is best to create the index with the fewest index options needed based on the application to minimize the size of the full text index.
Having a full text index on a database has some pros and some cons. Creating and maintaining a full text index requires additional resources on your system. However, if a database is not full text indexed and a user needs to perform a search, a temporary index is created and then destroyed. If this is done many times, it would be more efficient to have a permanent full text index. If an agent performs full text searches and a full text index is not created for that database, the server will advise you to create a full text index by logging the message Warning: Agent is performing full text operations on database 'mail/database.NSF' which is not full text indexed. This is extremely inefficient.
Because of the cost involved with building and maintaining temporary and full text indexes, there are numerous options available to Domino Administrators to control who can create an index, how often and what can be full text indexed. See the table below for the options.
|All users allowed to create full text indexes at their discretion with any refresh interval they choose.||In order to create an index, the user must have a minimum of designer access to the database. The user can create the index within the Application properties, the Full Text tab.||Pro: Does not require administrator intervention|
Con: This is not best practice. IBM recommends a maximum access setting of editor in the ACL of the mail file.
Con: Users can create indexes with an “automatic” or “immediate” update frequency on large mail files which can impact the performance of the entire server.
|No full text indexing is allowed except if created by an administrator using the Domino Administrator client.||As an administrator you can prevent users from creating their own full text indexes by setting UPDATE_NO_FULLTEXT=1 into the NOTES.INI on your server. With this setting the following will occur:
- Current full text indexes will continue to be updated.
- Administrators can create new full text indexes from the Domino Administrator client.
- Users cannot create a full text index.
|Pro: Administrators can fully control when a full text index is created and the update frequency used. |
Pro: Prevent performance problems created by having mail files with full text indexes set for “automatic” or “immediate” updates.
Con: Users may be upset if they have been able to create full text indexes in the past.
|Available disk space on the system is low and you want to conserve space by removing unused indexes.||By default, view indexes are purged if they have not been accessed in 45 days. You can reduce this by settingDEFAULT_INDEX_LIFETIME_DAYS=<# of days> in the NOTES.INI of your server.||Pro: Disk space used by indexes that are not being used can be reclaimed and reused.|
Con: If you set the value too low, views may need to be rebuilt more frequently than before and thus it will have a negative performance impact.
|No temporary indexes are allowed on the server. If a database/application must be searched, then a full text index must be created.||As an administrator, you can prevent all temporary indexes from being created and deleted by setting FT_FLY_INDEX_OFF=1||Pro: Conserve server resources by maintaining full text indexes rather than creating and deleting a temporary index each time the user performs a full text search.|
Con: If you do need to search a database only one time, you will need to manually create and delete the full text index.
|No temporary indexes are allowed to be created on the specific application.||At ODS version 48 or higher you can set a database property called “Don’t allow simple search”. When this property is selected, the database does not contain a full text index and a user attempts a full text search a message stating “Application must be full text indexed before search is allowed.”||Pro: You can avoid the server overhead of creating a temporary full text index, but not prevent it for all databases.|
Con: May lead to help desk calls for users that want to be able to perform full text searches.
|It is desired to force or prevent the full text indexing of attachments or attachment types.||There are several ways that an administrator can control attachment full text indexing.|
To disable all attachment indexing set FT_Index_attachments=2
To force all full text indexes to include attachments set FT_Index_attachments=1
To exclude specific attachments types from being indexed set FT_INDEX_IGNORE_ATTCHMENT_TYPES=<list of file types separated by commas
|Pro: As an administrator you can set a standard for all attachments on the server.|
Pro: You can easily disable attachment indexing which will lead to smaller full text index sizes as well as fewer system resources needed when building and maintaining indexes.
Con: Unable to accommodate situations where applications have different requirements which determine whether or not attachment indexing should be used.
|Need to disable binary attachment indexing (kvoop process).||As an administrator you can prevent the binary attachment indexing process, also known as the keyview filter, from indexing attachments by setting FT_BINARY_FILTER_OFF=1||Pro: You can easily prevent the kvoop process from running on your server.|
Con: This does not work in Domino version 8.5.0 and 8.5.1 (SPR JSTN825PAV)
|You search an application and when reviewing the resulting documents you observe that when opening a document containing an attachment it is slower than opening a document without an attachment.||When opening a document that is a result of a search, Domino will highlight all of the search words found within the search. Domino will also search the attachment for the search string to determine whether or not the attachment should be highlighted. You can prevent this search and thus prevent the attachment from being highlighted by setting FT_LIMIT_HIGHLIGHT_FILTER=1 in the server notes.ini file.||Pro: Opening documents after a search is faster for users and require less server resources.|
Con: Attachment will not be highlighted so user may not realize the result of their server is contained in the attachment.
|It is desired that temporary files created and used by the indexing process are created outside of the data directory.||During the indexing process the server will need to create and manipulate temporary files. You can specify the path where you would like these temporary files stored by setting view_rebuild_dir=<complete path to desired directory|
into the server notes.ini file.
|Pro: Fewer files in the data directory will make the server more efficient.|
Con: Administrator must be aware of where the files are located to ensure the desired storage location is available for the server.
|Need to temporarily disable the chronos task.||Chronos can be disabled by setting Debug_Disable_Chronos=1.||Pro: Occasionally may be necessary when troubleshooting or can be used to compare system performance with and without chronos.|
Con: Chronos is responsible for updating view and full text indexes set for hourly updates. With chronos disabled, those views and indexes will not get updated until the nightly updall process runs.
A domain index is a full-text index over multiple Domino applications or databases. The domidx task is responsible for building and maintaining a domain index. A domain index can be especially helpful in legal situation where auditors may need to be able to search your entire organization for specific data. For information on planning and creating a domain index refer to an article that is still as relevant in Domino 8.5.x as it was in Domino 5 entitled “Domino R5: Domain Search
” or the Domino Administrator help topic Planning the Domain index
or the wiki article Key points to consider when creating a domain index