6.1 View index performance considerations 

Previous | Next

Introduction

• Server:
  • Domino Server Version 8.5.2 (with FixPack 2 applied)
  • Operating System:  Windows Server 2008 Enterprise Edition (running in a Hyper-V environment
  • Memory:  4GB RAM
  • CPU:  Intel Zeon Processor (3.33GHz)
  • Disk:  Three disk drives, expanded as follows:
    • C: Drive:  Operating system
    • D: Drive:  Lotus Domino 8.5.2
    • E: Drive:  Domino Transaction Logs (more on this topic in section 6.1.4)
• Client:
  • Lotus Notes and  Domino Designer 8.5.2 (with the FixPack 2 applied)
  • Operating System:  Windows 7
  • Memory: 4GB RAM
  • CPU: Intel Core-Duo (1.8GHz)
  • Disk:  One C: drive to accommodate the Operating System,  and Lotus Notes and Domino Designer 8.5.2
• Application:
  For the purposes of illustration, we use the newly-revised Domino Discussion template that is installed with the Domino server, as created below:
  

All references to this section (and others, as noted) are made to the same application, server, and data set for consistent findings. We create 10,000 documents in the application (using  a LotusScript agent) to show the  performance metrics.
 
Factors impacting view indexes
From an application development perspective, there are many design decision that we make every day that impact view indexing.   Depending on the decisions made, view indexing (thus, view performance) can be effected.  Here, we discuss how the view indexes are maintained and their performance implication when these design decisions are implemented:

• Column sorting and categorization
• Using @Functions, and especially, date/time based functions
• Transaction logging
• Controlling Domino view index maintenance
• Using XPages 

Now that we have our environment established, let us dive into the topic of one of the most frequently ignored facets of Domino development: the View index.

View index sorting considerations

• What documents to display in a view
• What order the documents appear in
• What other column information is displayed in a view

It is our goal to design applications to keep this index as small as possible, for, the larger the index, the more work that Domino has to do – hence, the slower the view response time will be for the user. How the views are sorted impacts the size of the index.  You have the following view sorting options:

• No sorting
• Single column sorting
• Multi-column sorting
• Dynamic (user-controlled) sorting

You can display a list of view and the size of each view index using the following command:  

SHOW DATABASE dataBaseName.nsf v

• No sort options: Index size is 2,612,056 bytes.
• Sort on one column: View index size is 2,694,576 bytes (increase of 3.16% over non-sorted views).
• Sort on two columns: View index size is 3,022,256 bytes (increase of 15.7% over non-sorted views).
• Dynamic sorting: View index size is 6,796,120 bytes (increase of 160.18% over non-sorted views).

• Consider the view design for performance and scalability.
• Since there is significant performance impact, don't automatically make every column sortable. Use dynamic sorting as needed to give users ways to find documents. Also consider that the performance impact of one resortable view is less than that of two separate views. So, if you can eliminate a view by adding a re-sort to another view, that's a performance win. It's also a UI win, since users are accustomed to being able to click column headings to sort in various applications. It makes more sense to them than having to use an outline to choose among lists of the same documents sorted in a different way. In cases where you do need separate views (for instance, because you want to use categorization) consider using the view column option that lets you link to another view by clicking the column heading. This is intuitive for users, provided the views you link in this way select the same set of documents.
• Learn how to use the view column option "Defer index creation until first use." This minimizes the performance impact of resortable columns by waiting to do the work of sorting them until someone actually uses that re-sort. The first user to request that sort may have a long wait, but once the index exists it will be updated automatically by the server, like regular indexes, so subsequent use is quick. If the index is unused for a length of time, the index will be erased, saving the server work. It's a good choice for columns you want to allow to be dynamically sorted, but which you don't expect to be sorted very often.

• A category row is an "entry" in the view table just like a document is. The categorized column in the document row also contains a copy of the category value (it may be different because a category contains one value, whereas the field you're categorizing may contain multiple values).
• If a document does contain multiple category values, there are multiple row entries for that document, which also takes additional space and makes maintaining the list of entries more work.
• Categorization automatically implies sorting, either in ascending or descending order, so there is view index overhead because of the sort option selected.
• When you categorize a column, all documents related to that category are referenced (or linked) back to the category.   This information is stored in the view index to provide the “parent/child” hierarchy between the category and associated documents.

If we add a second categorized column to the same view (say, on the Author field), the results are as follows:


Note that we added the Author (Bruce Lill) across all documents, so that we have the same second-level category value across our application. The results of the view index size are shown below:


Our view index for the By Category view is now 4,400,924 bytes in size, an increase of 31.97% over a single category, and an increase of 43.55% over a non-categorized view.    As mentioned earlier, the larger the view index, the more work that Domino has to do to maintain that index, and open the view.

In addition to minimizing the view index size, the Collapse all when database is first opened setting on a view can aid performance.   This setting is under the view properties when you open a categorized view in Domino Designer, shown below:


The “Collapse all when database is first opened” setting instructs Domino to collapse the "Twisties" when a view is first opened by a user, thus, instructing Domino to display less information to the user upon view open (for our demo application, Domino would show two categories: Lotus Redbooks and Lotus Technotes, as opposed to these categories, the sub-categories, and document column data).  

The view will be displayed faster because Domino does not have to initially display as much data to the user. The consideration is that the user must expand a category in order to see the documents it contains.

Best practices

• Plan categorization carefully, for over-use of categorized views can lead to performance degradation and database size increases.
• If you use categorized views, keep the number of columns per view that you will categorize on as few as possible.
• Use the “Collapse all when database is first use” setting for categorized views.  
• Because Domino must maintain each view index, excessive numbers of categorized views may drag down the performance of the server generally.

• The view selection formula that  Domino uses to determine which documents to display in the view.
• Column formulas that are used by Domino to determine the data to display in the columns for each document.

The Domino server tries to save work by storing the view index, including the values of all columns for each document indexed. When the contents of the view are needed -- for instance, when a user tries to open the view -- the server only has to look at any documents which were created or modified since the view index was last updated, to decide whether they should be in the view and update or insert the row entries for those documents.

But this cannot work if the selection or column formulas are such that they might give a different answer at different times even if the document has not been modified. This can occur if the selection formula or a column formula uses any of the following functions:

• @Now
• @Today
• @Yesterday
• @Tomorrow
• @Accessed

If any of these functions are used, the server cannot assume that the value it calculated previously is still valid. So, every time the view is used, it creates the view index from scratch, evaluating the selection formula against every document in the database, then calculating the column values for each matching document. Naturally, this takes much longer than just examining recently modified documents.

Some notes regarding this situation:

• This rule applies only to column and selection formulas. Other formulas associated with the view, such as column hide formulas, action button formulas, and so on, are not used in calculating the view index, but are instead calculated by the client when the view is used. You can use @Now in those without any problem.
• It would seem to make sense for the Domino server to distinguish between the use of @Now, which always returns a different value, versus @Today, which returns the same value for 24 hours at a stretch. If only @Today is used, the server really only needs to recalculate the index from scratch once per day. However, that's not how it works; @Today is just as bad for performance as @Now.
• You may have come across a tip that tells you to use @TextToTime("today") instead of @Today, for better performance. This is incorrect. You do get better performance, but the view contents are incorrect. All this does, is fool the view indexer into not realizing that you're referencing the current date in your formulas. So, the indexer doesn't know that it has to re-evaluate the formulas against documents that weren't modified, and the values in the view get out of date.
• The @Random function returns a different value every time it's called, but the view indexer doesn't include it in the functions that force an index rebuild on each use. That's probably because using the @Random function in a view is kind of weird anyway, and it's not clear whether there's really a need to re-evaluate it for old documents.
• There are some functions that you might think would be changeable over time, but really are not. As an example, the @DocNumber function; if there are three documents in the view, the doc number for the last document is 3; if you create a new document, and the view sorting puts it at position 2, the old document number 3 becomes document number 4, even though that document hasn't been modified. There's a trick with @DocNumber and similar functions, though; they actually return a static string with special codes in it that instruct the client to calculate a value and insert it at that point. So the value in the server index doesn't change because of changes to other documents.
• There are several other functions, listed in the documentation, that don't work in views at all. They will not force a rebuild, but they also don't do anything useful.
• Any other functions that return or manipulate date/time values, are not a problem. Feel free to use @Modified, for instance.

As an example of that last point, consider the following formula added to the By Category view:


The formula used in the "Custom Formula Column" is as follows:

“This document was last modified on ” + @Text(@Modified)

• One of the underlying documents is modified.
• Domino performs maintenance on the view index.

Suppose that we make the following modification to our column formula:

“The current date and time is ” + @Text(@Now)

• Create a view that simply sorts documents by the date/time value, so that the documents of interest appear at the top of the view. Of course, the view also will contain a lot of documents you didn't want there, but it may be an acceptable compromise for best performance.
• Create a view categorized by date, and use the "single category" function to display only today's category.
• Use a folder instead of a view. Write an agent that executes nightly on one server, shortly after midnight, and adds and removes documents in the folder. Folder membership replicates, so it need not run on all servers. Depending what the selection criteria are, you might also have to have a "new and modified documents" agent to assign folder membership for documents created during the day.
• If the date-related formula is in a column rather than in the selection formula, use a "user programmable" column to read the formula from a profile document. Write a nightly agent to update the formula to contain today's date as a hard-coded value. Use @Date function to express a date in a way that doesn't depend on the date formatting settings of the server or client (for a local replica).
• The user-programmable column option can also be used for selection, though it is a bit of a kludge. If a column formula uses @Return, the document is excluded from the view.

• In the event of a server crash, the Domino application is left in an ‘inconsistent state’.   In short, the data in cache was lost, and, the data in the NSF was not written to completely (between steps 2 and 3, above). Thus, upon restart, the server had to perform consistency checks on each application.  The consistency checks means that the application could be off-line for minutes to hours, depending on the size and complexity of the application.
• In addition, since the application state is inconsistent, each view index has to be rebuilt, which, depending on the size and complexity of the application, could take minutes to hours.

• Domino  opens the transaction logs, looking for any transactions not marked as Complete.
• If the entire transaction was written to the log, then Domino writes the data into the NSF, marking the transaction in the log as "Complete".  This process is called a "roll forward".
• If the entire transaction was not written to the log, then Domino puts the data in the NSF back to the state it was in before that data was edited, and, mark the transaction ‘Failed’.   This process is called a "roll back".

The important piece for us to understand is that Domino will, when transaction logging is enabled by the Domino Administrator, protect the data from corruption, getting the server back on-line as fast as possible.   However, there is one more piece we have to consider: view index.

View logging does not impact the view indexing performance unless you explicitly design your views to do so.   By default, the view logging is disabled on a view that you build using Domino Designer. The view logging is enabled through Domino Designer, view properties, on a per-view basis, as shown:


Once enabled, view indexes can now take advantage of Domino transaction logging, assuring that the database is brought back on-line fast, and, the indexes are updated without a full-scale view index rebuild.   Combined with database transaction logging, a production server can be on-line in minutes after a server crash (instead of hours).

Best practices

• Transaction logging must be enabled and configured by the Domino Administrator before view logging can be successfully enabled.
• View logging does incur additional overhead on the Domino Server, so work with your Domino Administrator to make sure that the server resources (memory, CPU, and disk) can adequately process view logging requests.

• Benefit:  Active views are kept reasonably up to date, so that users don't have to wait much for the view to open. Inactive views eventually stop causing extra work for the server, so that unused databases don't slow your server merely by existing on the disk.
• Consideration: Once users start using the view, its existence causes work for the server even if the users aren't using it often. Enough views of this type that are in occasional use, can drag down server performance.

Automatic 
The UPDATE process will create an index for this view, and keep it up to date, whether anyone is using the view or not. In addition, anytime the view is opened, the server updates the view index to make sure of catching recent changes.

• Benefit:  The view is always up to date, and opens quickly for the user .
• Consideration:  The presence of this view causes work for the server even if nobody ever uses the view. Only select this option if performance is a key concern for that specific view.

Manual
The view index is never generated or updated automatically. It is created the first time the view is used, but not updated every time a user opens the view. This option allows the view to be opened very quickly because the stored index information is displayed without checking for recent modifications. Since the view index is not refreshed, the data displayed in the view might not match the data currently in the documents. The user will likely see the view refresh icon ( ) when they open the view, advising them that the data are out of date. They can refresh the view by clicking this icon or pressing the [F9] key.

• Benefit:  The view open quickly, a feature especially useful in large databases.
• Consideration: The data may be stale. Refreshing the view index is a manual operation for the user.

This is a good choice for views whose contents change seldom or never, such as help documents.

Auto, at most every
This option allows you to control the maximum hourly interval at which the view index will be refreshed. When selected, the application designer must enter a number of hours between view index refreshes.  Once applied, the view index is refreshed automatically, but not more often than the number of hours entered.   For example, if you set the Auto, at most every option to  four hours, the UPDATE process will refresh the index if it's been four hours or more since the last update. Like Manual updating, the view index will not be updated when a user opens the view, but they will see the refresh icon and can manually refresh. The time between updates may also be more than the selected interval, since UPDATE is a polled process that scans through all databases repeatedly, and if there's a lot of work to be done in other databases, it can fall behind.

• Benefit: The view index is automatically refreshed at controlled time increments.   This can decrease indexing time, and assure that server resources are not being consumed by a more frequent refresh interval. The view also opens very quickly.
• Consideration: When you open a view, the view index is likely stale, and may need to be refreshed manually.

Because of the combination of rapid access and eventual updating, this is a good choice for views whose contents change only occasionally, such as keyword documents accessed by @DbLookup. Even though the database is active generally, we don't need to update this index with every use; it's acceptable if it takes a few hours in exchange for better performance overall.

“Discard” Index Option
By default, Domino performs an incremental "refresh" of the view index, looking for new and changed documents since the last refresh and inserting their information among the existing view entries, or deleting existing entries if the document no longer qualifies for inclusion. You have the option to “rebuild” the view index, destroying the existing index and creating a new one from scratch (you can do this manually from the view via Shift+F9, or via calls to the Notes APIs). Naturally, unless there have been a huge number of modifications, a refresh is much faster than a rebuild. While it may sound attractive to "start fresh," you can generally rely on it that the index for any documents that haven't been modified, is correct and up to date. Use refresh and don't worry about it.

To control how often the view index is rebuilt from scratch, use the Discard options in the view properties. See the figure below:

If you "discard" the view index, Domino must create the index from scratch the next time someone wants to use the view. This doesn't make the view more reliable, only slower to open. The advantages of discarding the index are that it's not taking up space on disk, and the UPDATE process no longer has to update it, so that frees up server resources.

The options for automatically discarding the view options are described below:

If Inactive For 45 Days
This is the default option. If Domino finds that a view is inactive (no users access) for 45 days, it  deletes the view index from the application.   This can save storage space on infrequently accessed views. 45 days is a good length of time for most views; if they become disused for that long, there's a good chance that they'll continue to be disused for much longer, so there's no point in expending resources to keep them updated.

Note: this option is ignored if the Refresh setting is "Automatic".

After Each Use
This setting instructs Domino to not store the view index at all, but to calculate it from scratch every time the view is opened. This is rarely a good choice, but might be appropriate in some special cases. This is the same behavior that occurs automatically if the selection or column formulas use @Now; we know that any stored view index is immediately invalid because @Now is not the same the next time we look at the view.

If Inactive For 
This option is similar to the "If Inactive For 45 Days" setting, except you  get to define the number of days between view index discards.  If Domino finds that a view is inactive (no users access) for the number of days specified, it will delete the view index from the application. This can save storage space on infrequently accessed views, or in case you know enough about the usage pattern of the view to select a more specific number than the default 45 days. For example, a view containing sales data that's needed once per quarter to prepare a report; once it's been inactive for a week, we know they're done with it until next quarter.

Note: this option is ignored if the Refresh setting is "Automatic".

Best practices for Discard settings

• Given the important relationship between view indexing and performance, first try to minimize the number of views when designing  your Domino application.
• For those views you do need, at least consider whether the default indexing options are appropriate.
• In existing applications, if you suspect views are unused but aren't sure enough to just delete them, set their discard settings to a low number of days to reduce the performance penalty if someone opens it by accident.

• Logical information ordering
• Ease of navigation
• Robustness that users expect in modern applications

We now explore how XPages can give us multiple, useful views of the data presented above without physically designing another view in the application.

Creating the All Documents view

• Show data from:  This field instructs the View Control from where to show data.  We use Domino view.
• Application:  The application for this view.  We use our demo application.  You can link this View Controller to a view in another Domino application.
• View:   The name of the view to display in this View Controller (the All Documents view)
• Choose Domino view columns to display:    This is where the power of the View Controller comes in to play.  You can select which columns from the view that you want to see, ignoring those that you do not.  

Since the original All Documents view contained the creation date, author, and subject fields, we include only those in our XPages design for the All Documents View Controller, as shown: 

 
Your XPage appears as follows, where you can "tweak" the results to your needs.


Through the View Controller, you can control virtually all aspects of the view display, including:

• The navigation bar at the top allows the user to page through the documents in a clean, easy to use navigation system.
• The view columns can be rearranged to suit your needs.
• The view sorting options can be set here on the XPage – a more efficient method of processing view column sorting than traditional Domino views.

NOTE: XPages are not magic; the work of sorting still has to be done at some point.
For example, you can rearrange our columns, as needed, by simple cut and paste operations.  Then, you can open the properties of the Created column and instruct XPages to sort the data, as shown: 

Upon preview, you obtain the following XPage-based view:

Note that we did not make the interface attractive, for our focus is on view design and performance.

In summary, here is what XPages design brings to views, view performance, and server resource utilization:

• We created one view in Domino Designer, thus, Domino has one index to maintain.
• We inserted the view onto our XPage,  using the XPage View Control, removed the columns we do not want, rearranged the column we do, and sorted on the appropriate column – giving us the custom view we need without physically designing another view in the application.
• By selecting different sets of columns to display, we can create different front-ends to the same Notes view. To display different selections of documents, however, you usually still need different views.

To illustrate the last point, consider building an XPages-based view for the following views:

• documentsAuthor.xsp:  All documents, sorted and categorized by the Author, and containing the Subject, Creation, and Last Modified fields
• documentsCreated.xsp:  All documents, sorted and categorized by the Creation date, and containing the Subject, Author, and Last Modified fields

Using traditional Domino development, you would have to design two additional views, so the server would have to maintain two additional view indexes.   Using XPages, you can repeat the process above, inserting only the columns you need, giving users three views from one physical view design.
XPages interfaces have drawbacks as well as advantages.

• It's somewhat more work to design.
• Full-text search and other functions that users take for granted in a Notes view, are not available unless you design them in each time.
• It's difficult to navigate longer views, where you may need to advance to the next page many times to get into the middle of the view. The Notes view's "quick search" function and the scroll bar that can jump you to any place in the view, are a big help when working with large amounts of data.
• There is no way to do a manual full-text search on the whole view, select all matching documents, and execute an action on them, if the results are more than one page long. This is routine in a Notes view.

• Limit the number of views.
• Limit the complexity of views -- fewer columns, simpler formulas.
• Avoid using @Now, @Today, etcetera in view selection and column formulas.
• Take time to consider whether you have set the appropriate indexing options for each view.
• Limit use of user-sortable columns, while bearing in mind that a single re-sortable view has less performance impact than two views with no re-sort.
• Don't allow end-user creation of shared views. The users don't understand the performance impact of creating views, and may create many duplicates.
• Avoid excessive column categorization in the design. A categorized column affects performance significantly more than a simple sorted column.
• When categorizing multiple columns in a single view, it is best to have a single categorized column, using the value "Column1Value\Column2Value\etc". Because Domino only has to maintain one category index. [really?]
• When displaying columns that are categorized, or responses that appear beneath their main document, use the view design option "Collapse View When Database Is First Opened" to  make views open faster.
• Using XPages and the View Control can  enhance the application performance by re-using the same view in different display or sort orders, minimizing index maintenance and size in the application.
• Work with  your Domino Administrator to enable Transaction Logging on the server to take the advantage of view index logging (a property of the design of each view) to allow server restart and application availability to be optimal in the event of a server crash.

The main consideration in designing views, is the tradeoff between giving the user complete information, and performance of the application and the server.

As a designer, the Rapid Application Development (RAD) nature of Notes permits you to add a view with ease.   While at times this may be necessary, you must really think about simply adding additional views “on the fly” without considering the impact on the overall application, and, the overall environment.

First learn what tasks users need to do in your application, and think through the application design. Design each view carefully to give users the information they need for their specific tasks. Adding unneeded columns and re-sorts hurts performance without helping the user. Unnecessary columns can even make your application more difficult to use by cluttering the screen with trivia.

It's quite common to see applications with dozens of views, many of which are probably unused. Nobody, not the developer, not any one user, knows what all these views are for or which documents they select. The view name in the outline is often not enough information for the user to determine what view to use. Even if they go to a given view, they may not be able to tell whether it contains all and only those documents they need for their task. This sometimes results in users requesting creation of new views that duplicate existing views, simply because the name of the existing view doesn't suggest the task they want to do there.

This can be addressed in part by intelligent design of the navigation area of your application. You don't have room in the outline to describe the view in detail. What you can do instead is identify the tasks that different users do in your application, and create views that directly address those tasks. Sometimes (often, we hope) different groups of users may find the same views useful. Bear in mind that the label of a view in the outline doesn't have to be the same as the name of the view, and that multiple outline entries might point to the same view. Also consider that certain tasks may find it useful to see multiple views at the same time as a document preview. Perhaps a frameset or composite application would be helpful. Sometimes, views are identical but for different sets of actions, directed to different tasks. Consider merging these views and using action hide formulas based on @UserRoles instead.

If you learn what tasks your users do in your application, and design around those tasks, you will maximize productivity while also keeping the number of views under control and being able to tell when a view is obsolete, so that you can delete it and improve overall performance. If you already have an outline that isn't task-oriented and you want to let people find views in the old way also, consider a tabbed UI in the navigation space that would let users select the "legacy" navigation or the new task navigation.

There's no way to directly test whether a given view is unused. As we saw in some earlier examples, you can view the index information of specific views and tell whether the index has been deleted for disuse. However, because Notes data are replicated but view indexes aren't, this information may be different on different servers. Also, especially in applications with a large number and confusing organization of views, users may open the wrong view by mistake. So the existence of an index is no guarantee that the view is needed. If you think a view might be unused and you'd like to delete it, you might add code to the Postopen view event to display a dialog that the view is to be deleted on such and such a date if you don't hear from anyone who wants to keep it. Or even delete the view and change the outline entry to point to a page that says "this view has been deleted because we didn't think anyone was using it. If you want it back send mail to ___ explaining why you need it. Or, have a look at this other view that we think fulfills the same purpose: (link here)." If you don't hear from anyone for a while, delete the outline entry also.

The whitepaper Performance basics for IBM Lotus Notes developers contains more on view performance.

Parent topic: 6.0 Performance considerations