Managed Replicas Explained

Introduction & Overview

Notes 8.5.2 introduces the managed replica, which is a variation of a standard local replica of the user's mail file. The managed replica has the special characteristic of being used "automatically" when the user is connected on a server-based location. It also has the capability of containing full documents for a section (based on time/date) of the mail file and truncated content for the rest (older data). The concept of a local replica of a user's mail file and the use of a local outgoing mailbox to send mail has been with Notes almost since its inception. Previously these features were utilized by configuring a location to use a local mail file, causing the client to use a local outgoing mailbox automatically. In addition, the replication schedule has to be configured so that the local replica will replicate with the server-based mail file.

At a fundamental level, the managed replica extends the capabilities of a local mail replica and a local outgoing mailbox to those locations where the user remains connected to a Domino server. In this configuration, (connected location with managed replica) when the user opens the mail file the client code will detect that a managed replica for the file exists, open that managed replica (locally), and present the user with the view of the mail from the local file. Updates, sending of mail, and receipt of new mail will happen as when the Notes user uses a standard local replica with a local outgoing mailbox. Synchronization occurs just as always, in the client replicator between the server-based files (mailbox, mail file) and local mail. The client will be configured to perform this synchronization in as timely a manner as possible (details in a later section). The target user for this feature is a user who is connected to a Domino server for mail but on a network or line that can have high latency issues (for example, LLN2). By moving all network I/O between local and server files to the background client replicator, the user's interface will never be blocked waiting for an I/O to complete. All the user's operations will occur on the local files.

Configuring Managed Replicas

There are two methods for creating a managed replica (or standard local replica) of the user's mail file.

1. Setting the appropriate value(s) in the NOTES.INI file.
2. Having the Domino Administrator set the appropriate fields in the Desktop Settings policy document. This method requires the release 8.5.2 Domino Directory NAMES.NSF.

In either method, the information is sampled by the Notes Dynamic Client configuration tool (DynConfig), and the local replica (managed or not) is created and added to the user's desktop and to the replicator. An initial replication (build) of the file is then triggered.

The following settings are recommended for the initial creation of a managed replica. Default settings can be used for the remaining parameters, which are detailed in the following table.

• CacheMail=3 (Create the managed replica if a local replica does not already exist.)
• OutgoingMailSendThreshold=1 (Set the mail threshold to 1. This means every time a new message is deposited in the local mailbox it will immediately be sent to the server.)
• ReplicateOnNewMail=1 (Replicate new mail from the server every time we detect new mail has been delivered. This is crucial in keeping the cache "up to date")

These settings can be configured explicitly in the NOTES.INI file by the end user or they can be pushed down by the administrator, using the Notes Desktop settings policy:

Notes.ini Parameter/Policy Field	Type	Description
LocalReplica (Policy) CacheMail (INI)	Bit mask	1 - Create a local replica of the user's mail file. If a local managed replica exists, convert it to a standard local replica. 3 - Create a local managed replica of the user's mail file. If a local replica already exists, do nothing. 7 - Create a local managed replica of the user's mail file. If a local replica already exists, convert it to a managed replica. 8 - Delete the local replica of the user's mail file (managed or not).
CacheDBInitialFullDoc	Number	Applies to initial managed replica creation only. If set (to n) the most recent n days worth of data will be replicated with full content (attachments and rich text). (Default is 30 days). Documents older (not modified) than this will be truncated. (Default is summary data only).
CacheDBRemoveDocsAfter	Number	Applies to initial creation of the local replica. If set (to n) documents not modified in the last n days will not be retrieved from the server and will be removed as the replica ages. (This is the same setting as seen on the Space Savers panel of the replication options dialog box). Default is none (all documents will be retrieved).
DocAutoRetrieve	Number	0 - No auto-retrieve of truncated documents. Entire documents may be retrieved the same as in selective replicas. 1- Rich text (content) is automatically retrieved when a document is opened. Attachments are retrieved on demand (when accessed). 2- All truncated content (rich text and attachments) is automatically retrieved from the server when the document is opened. Default value if not specified is 1 (auto-retrieve of rich content). This setting controls the auto-retrieval of truncated documents in any local replica. :
CacheDbTruncatedRTF	Number	Amount of rich text (in KB) to retrieve in docs older than CacheDbInitialFullDoc days during cache creation. Default 0 (summary only)
CacheDbTruncatedAtt	Number	Amount of attachments (in KB) to retrieve in docs older than CacheDbInitialFullDoc days during cache creation. Default 0 (summary only)
CacheDbFreeSpaceNeededInMB	Number	Amount of free disk space required on the user's drive where Notes data resides in order to create the managed replica. Default is zero. (the client will always try to create the managed replica)
OutgoingMailSendThreshold	Number	Note: It is recommended for all users of LLN2 that this value be set to 1 (transfer on each message deposited to the local mailbox). If set, the client will use a local outgoing mail box for sending mail from the user interface. The client replicator will transfer the sent messages from the local mailbox to the server's mailbox. The value of this parameter will indicate how many messages need to be queued in the local mailbox before triggering the replicator to transfer them to the server. The default is zero (transfer directly to the server's mailbox).
ReplicateOnNewMail	Number	If non-zero and a local replica (managed or not) exists. The client replicator will be triggered to pull down new mail from the server upon detection of that mail being deposited in the user's server-based mail file (asynchronous notification from the server to client).
FailoverSilent	Number	If non-zero the client will silently fail over to a different server if the current server is no longer operational (no prompts). Default is 0 (prompt). Best practices suggest this should be set to 1 for all server based mail users in the LLN2 environment.
($PrefReplDefFullText)	Number	If set to non-zero full text indices will be created for all local replicas (including managed replica). Default is 0 (no full text indices)
($PrefReplDefEncrypt)	Number	If set to non-zero local replicas (including managed replica) will be created with encryption turned on (default).
($PrefReplDefEncryptType)	Number	Specifies the strength of the encryption for the local replicas

The policy settings that would be recommended for Managed Mail Replicas / MMRs as follows.

These settings cover:

• Initial creation/conversion.
• Settings that apply at MMR creation.
• Settings that control behavior
• Settings that control replication schedule and replication trigger on new mail

The recommended changes are shown in the table below.

The table is followed by screen shots of each of the areas on the desktop policy that need to be changed, with a focus on specific settings highlighted with a red box.

Settings on the Desktop Policy	Set Value To	How to apply should be set to	Applicability	Comments
Mail Tab\Mail Settings
Local mail file	Create managed replica or convert local replica to managed replica	Set value whenever modified	At MMR creation/conversion only.	A conversion of a local replica to a managed replica allows a customer to standardize on managed replicas.
Mail file location	Server	Set value whenever modified	Always on mail application open	The Notes Client will automatically use the local copy once created and available. At other times, the Notes Client will use the server copy.
Use local mail.box to send messages (faster)	1 (This is already enforced by LLN)		On mail send	A sent mail message is placed in the local mail.box and sent in the background
Mail Tab\Managed Replica Settings
Amount of free space required before cache is created	<   in Mb		At creation of MMR only.	At the Customer's discretion. Setting this to a value such as 1,000 (1 Gb) would ensure that a managed replica does not use the remaining free space on initial creation.
Mail Tab\Client Settings
Auto-retrieve document setting	Enable document without attachment		Always on open of a truncated (partial) document.
Enable server to poll for new mail and trigger replication on notification of new mail	Enable		Always when client is notified of new mail receipt on server.
Preferences Tab\Replication\Default settings for a local replica
Create a full-text index for faster searching	Enable	Set value whenever modified	At creation of MMR only.	At the Customer's discretion
Encrypt local replicas	Locally encrypt	Set value whenever modified	At creation of MMR only.	At the Customer's discretion
Preferences Tab\Replication\Default replication schedule
The normal schedule should be configured.			Always while Notes Client is open
Preferences Tab\Mail
Check for new mail	Enabled	Set value whenever modified	Always. Needed for when the Notes Client reconnects to a network after no network access or after a hibernation, suspend or client shutdown.
Mail checking interval	60	Set value whenever modified		This needs to be set to some value. The frequency is unimportant since the client maintains an active connection to the server in order to be notified of new mail after receipt on the server.

Desktop Settings policy document > Mail tab > Mail Settings & Managed Replica Settings

Desktop Settings policy document > Mail tab > Client Settings

Desktop Settings policy document > Preferences > Replication -> Default settings for a local replica

Desktop Settings policy document > Preferences > Replication -> Default replication schedule

Desktop Settings policy document > Preferences > Mail

Creating & Using Managed Replicas

After the client is started, with the above values set in the NOTES.INI file, the managed replica will be created. The processing of these values and the initial replica creation is handled by the Dynamic Client Configuration tool (DynConfig). If a policy is received, that contains a setting for the LocalReplica, the setting will be interpreted and the other fields (in the table above) will be pulled from the policy and copied to the NOTES.INI file. If the CacheMail setting is in the NOTES.INI, that setting is interpreted and the other values (if set) are assumed to be also set in the NOTES.INI. In this way, the same code can process the settings regardless of whether they were pushed down through a policy or explicitly set in the NOTES.INI file.

The trigger is the value of CacheMail (NOTES.INI file) or LocalReplica (policy). In the case of a policy being pushed down all the other values are extracted from the policy and copied into the Notes.INI and the code from that point forward will interrogate the INI variables to determine what to do. If specified, the local managed replica is created (in the same relative path as on the server), and the desktop entry is created and added to the workspace. A replication entry is also created and added to the replication page; then a replication is initiated.

The managed replica is initially marked out of service, using the database option (DBOption) bit, until the initial replication pass is complete. In addition, a new database option bit (DBOPTBIT_LOCAL_CACHE) that identifies the local replica as a managed replica is set in the database. This is essentially the only on-disk difference between a standard local replica and a managed local replica. A managed replica is initially set to retrieve summary data only from the server. As the replicator makes its first pass, it checks to see, for each note to be replicated, whether the last modified time is more recent than (CacheDBInitialFullDoc) days. If so, the replica is signaled, on the fly, to retrieve the entire note from the server. Once the first replication pass is complete, the replica is updated always to retrieve entire documents from then on. In addition, the replicator will build the most accessed views (Inbox, Sent, Calendar...) before marking the replica back in service. After the replica is marked in service, it will be used as the user's mail file.


Using mail during the creation of the managed replica
All the processing in the preceding paragraphs takes place in background processes (Task Loader, DynConfig) on the Notes client. The end user will continue to access mail and calendar directly on the server. If a value for the OutgoingMailSendThreshold (recommended value is 1 for use with a managed mail replica), the local outgoing mail box (mail.box) will be created immediately. Any mail sent by the end user from then on will be deposited in the local mail box and then immediately replicated to the server via the background replicator. Once the managed mail replica is completely replicated and is marked in service, it is ready for use.

Using the managed replica as the mail file
The managed mail replica is essentially intended to be used as an abstraction of the user's server based mail file. Even though the location will still indicate that the user's mail file is on the Domino server, user gestures which access the mail file will transparently access the local managed replica rather than the server-based file. The following gestures will all access the local replica rather than the server replica:

The user can still directly access the server based copy by explicitly opening it. There are two methods by which the end user can explicitly open a specific server-based replica of the mail file:

Synchronizing the managed replica
As already mentioned, a managed replica is essentially a local replica with a couple of special attributes. It will use the same background replicator as other local replicas to sync changes between the local file and the server-based replica. Because the managed replica is an abstraction of the server-based file, it should receive updates to the local file as soon as possible. For that reason, the managed replica should always be used in conjunction with the ReplicateOnNewMail setting. Updates in 8.5.2 always leave a network session open to the user's home mail server. In this way, an asynchronous notification is always received at the client, when new mail is deposited in the server mail file. If the ReplicateOnNewMail setting is enabled, this configuration will trigger a replication of the mail file to the local managed replica and the new mail will become available at the client.

Because the goal is for the end user always to work in the local file, updates made locally are not as time-critical to send to the server. If the user already had replication enabled, when the managed replica was created, the settings are not changed. For all server based locations, for which replication is not enabled, the client will create a default schedule (once an hour, seven days a week) for the replicator. Updates made to the local replica will be pushed out to the server on this schedule.

One of the goals of the managed replica is to never block the user interface due to network latency/connection issues. This is especially important in the cloud environment. For this reason, a managed replica should always be used with the SendOutgoingMailThreshold parameter set to one. When the user sends an e-mail, it will be dropped locally in the mail.box file and the user can continue on to the next operation without waiting for the transfer to the server to complete. If the threshold is set correctly (to one) the background replicator will immediately transfer the message to the server's mailbox. Because this happens in a background process, the user interface is not blocked or delayed.

Full text index searching
When the local managed replica is created, a full text index will also be created if either of the following is true:

The managed replica is created and replication is performed in the same manner , whether a full-text index is created or not. That is to say, the CacheDBInitialFullDoc setting controls how many days of full content (default 30) is replicated down to the client. All remaining data is summary only (by default). Once this initial replication pass is complete, the indexer is kicked off and the content that has been retrieved from the server is indexed. The end user can now perform full text searches against the local, managed replica. Because the data is incomplete the index will be as well.

The first time the user searches the mail file, this will signal the background replicator to begin to pull down the rest of the data (the notes that were truncated) from the server. This happens invisibly to the user and will be interrupted if the client exits before it is done. The "trickle down" of the truncated data will continue on client restart. The data is indexed as it is retrieved. The user can continue to search the local managed replica while this process is occurring. The user will receive an indication with the search results, that the hits may be incomplete as the index is not fully built. Once all of the truncated documents are retrieved and indexed this indication is removed and the user can search with impunity. The screen shot below illustrates a typical result said with the (Incomplete...) added to the header line. Hovering over this line will indicate that the reason the results may be incomplete is that the index is still being built.



NOTE: The managed replica's index uses document data plus any attachment data already retrieved, if the "Index attached files" option was set on the server replica.

Falling back to the server-based replica
The goal of the managed replica is that the user constantly works on the local file, the data is synchronized in the background, the user is never blocked due to network latency and never has to explicitly connect to the server based replica of the mail file. One of the issues with the Notes client and local files has to do with an abnormal client exit. Any local NSFs that were open are now in a potentially inconsistent state and need to be fixed up. The end user will typically see this when, upon restarting the client they go to access a local database (e.g. their local mail file) and receive a status message to the effect that the code is performing a consistency check on the file. The user is prevented from being able to access the file until the consistency check has completed. Unfortunately, depending on the state of the NSF when Notes terminated, and particularly for large files, this can be a lengthy delay. For managed replicas, on access, the code will check if the file is an inconsistent state and if it is, will automatically redirect the user to the server based replica. The consistency check, or fixup will occur in a background thread. Once this check is complete, a replication with the server based file is triggered. Subsequent accesses of the mail file will then go back to using the local managed replica.

If managed replica got badly corrupted or deleted or somehow got in a state where the code is unable to complete the consistency check, the existing managed replica will be deleted. The code will then recreate the file from scratch, honoring the settings as outlined in the previous sections. Again, after the replication completes and the views are built, the managed replica is ready for use and will be transparently switched to, on subsequent mail file accesses.

Architectural Change: Multi-threaded Background Replicator

The creating and the initializing of the managed replica simply leverages the long standing replication technology in Notes. No architectural changes were required here either. The notion of switching between various replicas of the user's mail file depending on availability is also a longstanding capability of Notes utilized by fail over. The transparent switching to and from the local managed replica based on its state leverages and extends this capability. The one major area where architectural changes were introduced in support of the managed mail replicas is in the Notes client's background replicator.

Prior to Notes 8.5.2, entries on the client's replicator page were processed in sequence, by the background replicator process, in a single thread of execution. The main goal of the managed mail replica is to shield the user from any network latency issues by performing all mail operations in this background process. However, the user expectations is that these operations occur in real time (when they hit send, the message is sent immediately, when new mail arrives in their mail file, they see it immediately...).

In the single threaded model, mail operations can be blocked for a significant amount of type. Imagine the end user has other local replicas. If a large number of documents, say from a specs database, are being replicated down to the client machine, all other replicator operations are blocked until the first replication is complete. Even if the end user utilizes the background replicator solely for mail operations scenarios are encountered where a large amount of data is being replicated in one direction (incoming or outgoing) and pending operations in the other direction are blocked until the current operation is complete.

The solution for this problem in Notes 8.5.2 is to re-architect the background replicator to be multi-threaded so that more than one operation can be executed simultaneously. In particular, higher priority is given to operations involving the end user's mail than other replicator operations. To that end, two threads of execution are reserved for mail operations, essentially sending and receiving of mail. In general, just one thread will be utilized, but if that thread is busy for longer than 15 seconds, the second thread will be utilized to perform the other mail operation.

All other replicator functions are performed on a pool of worker threads. Operations against the same server (e.g. replicating 3 databases, all against the same remote server) are grouped together, in a single thread and performed sequentially. Failover will occur exactly as in earlier versions of Notes.

Because multiple operations can be concurrently executing the user interface for the client to reflect that. The major change is that rather than have a single status area at the bottom of the page to reflect the current operation, each entry in the replicator page has its own status area and an individual button to control (cancel) its operation. The major button at the top allows the user to start or stop all enabled operations. The various menu commands and capabilities (Send/Receive Mail, replicate with a specific server, etc..) remain the same.

The screenshot below illustrates a typical scenario during a scheduled replication, on the Notes client:



Note that the mail operations have already completed, on the mail thread. Two databases are concurrently replicating with the respective servers. Each database has its own stop button and one can be cancelled independently of the other. A third database is pending, which means it is waiting for another operation to complete. There are two reasons for an operation to be pending:

In the example shown above, the database that is pending (V85 Fixes) is waiting to replicate against the same server that the entry above it (Clearcase) is currently replicating with. Once the replication of the first NSF completes, the second NSF will be processed, on the same thread of execution.


Intended audience for Managed Replicas
The sweet spot for managed replica usage is end users who work exclusively connected to a Domino server but are frustrated by delays in processing mail operations, waiting for network operations to complete. Users in a remote location and cloud users are two good examples. Anyone who experiences network latency issues would benefit from having a managed replica. In addition, because the Notes replicator streams notes from the server to the local machine in a single NRPC transaction, usage of managed replicas can reduce load on the Domino server. The managed replica has to be created and completely replicated and have its view indices built before it can be utilized. Newly provisioned users or users with small mail files are excellent candidates for managed replica usage as they can enjoy the benefits of the local replica almost immediately.

Users with large mail files can also utilize managed replicas. One potential consideration for these users is to configure the setting CacheDBRemoveDocsAfter to create a managed replica that is also a selective replica. For example setting CacheDBRemoveDocsAfter to 365, the end user will have a managed replica containing documents for the most recent year, and full content for the most recent 30 days. This will allow such users to get online and use the managed replica for new mail in a more timely fashion. The caveat here is that in order to see documents older than one year the end user must explicitly open the server replica.

As noted in an earlier section, the managed replica can be configured by the Domino Administrator via the Desktop settings policy. (eg. Open->Mail->Open Replica-> <Home server replica> or go to the File->Open->Lotus Notes application dialog and explicitly browse to the server based replica).

Users of local mail replicas can continue to use these replicas as they always have. The behavior of standard local mail replicas do not change with the addition of managed replicas. In general users who are comfortable with local mail replicas and understand the functionality and limitations should continue to use them in the manner they are accustomed to. Notes 8.5.2 will support the ability to convert a standard local replica to a managed replica. In local locations, the managed replica will behave much the same as a regular local replica. The benefits of conversion will add the capability to use the managed replica even in a server based location, as outlined in the functional section of this document. In addition, if the local replica needs fix up, the code will automatically switch back to using the server based replica. Converting a standard replica to managed or vice versa does not change any of the data parameters but simply toggles the database option bit that identifies the file as a managed replica. What ever the truncation and selective replication parameters are for the existing file, they will not be altered when converting between a standard and a local replica.


Configuring Managed Replicas for large number of users
Users can have managed replicas setup for them by the Domino Administrator via the Desktop Settings policy document. One thing to consider is that the potential for a large number of users to have the same policy pushed down to their clients at roughly the same time. This could result in a large number of managed replicas being created and replicating with the Domino mail server simultaneously, causing significant load on the server. To reduce the likelihood of this, the Administrator may want to consider a phased approach. Notes 8.5.2 has built-in functionality to mitigate this risk.

For hosted users whose Domino mail server is in the cloud, the server load when creating a managed replica is modulated based on the current server load. This modulation occurs as a cooperative exchange between the Notes client and the Domino SAAS (LLN2) server. When establishing a session to the LLN2 server for initially replicating the managed replica, the Notes client will indicate that this is a background session. The LLN2 server has built in controls to award higher priority to foreground sessions. If the server becomes heavily loaded it may send asynchronous notifications back to the client, on this session to indicate that it should modulate the streaming of the data (notes) from server to the client. Based on the data in the asynchronous notification the client code will delay before asking for the next piece of data in the replication stream. If the server becomes too heavily loaded, it may, without warning terminate the session to the client. For this reason, the client periodically stores away checkpoint results during the replication operation. If the replication is interrupted (either by the end user or by the server terminating the session), the client code will, upon the next replication, pick up the latest checkpoint results. The replication operation will pick up from there, rather than starting over again to stream the data down to the local machine.

A different mechanism needs to be managed replica users whose mail servers are on premise. For Notes 8.5.2, the managed replica creation code will leverage the same throttling mechanism that exists for the Notes Smart Upgrade on the server. The administrator will have to enable the Smart Upgrade Governor in the mail server's configuration document:



The maximum concurrent downloads setting indicates how many client machines can be replicating with the mail server for the purpose of building the managed mail replica at any given time.
In addition to allowing a maximum number of clients to connect to the server for the purposes of managed replica creation and initial replication, the replication code in the end user's client will receive an indication of how many concurrent managed replica operations are being performed on the server. Based on this number the replicator will delay between operations that occur during the initial replication of the managed replica. The delay behavior will conform to the following algorithm:

Examples for setting maximum number of users

• The limit can be set at what ever the administrator would like to set it at, and the client handles it in the following way.
• If the default of 100 is used, the first 20 users (20%) can replicate at the highest performance level at the same time, users 21 through 90 are still allowed to replicate, except at a lower level, and (at 90% usage) users 91 to 100 (and above) are limited until less users are currently replicating (Access to this server has been restricted due to excessive load).
• If 1000 user limit is used, the first 200 users (20%) can replicate at the highest performance level at the same time, users 201 through 900 are still allowed to replicate, except at a lower level, and (at 90% usage) users 901 to 1000 (and above) are limited until less users are currently replicating (Access to this server has been restricted due to excessive load). 90% was chosen so Smart Upgrade has at least 10% of the user limit to use for Smart Upgrade purposes.
• So 90% of the user limit setting, can replicate at the same time. The first 20% of the user limit will replicate at the highest performance level, users from 21% to 90% of the limit will replicate at a lower performance level.
• Setting the maximum users to a large number will essentially disable the governing/limiting for replication and rely on the standard server performance to handle its maximum number of users that can replicate at the same time.

If a client is refused a connection, it will attempt again, according to the above algorithm on the next replication (scheduled or explicit, via user gesture).


Enabling full text searching for Managed Replicas
As already mentioned, if the server based replica of the user's mail file has a full text index, the local managed mail replica will be created and a local full text index will be created so that the user will have the same search capabilities as on the server. However, a number of current customers do not have full text enabled on their servers for performance reasons. For end users who have sufficient free disk space, the administrator has an alternate method to be able to provide full text capabilities for managed replicas. On the desktop settings policy document, the Preferences->Replication Tab has a setting to create full-text indices for local replicas (Create a full-text index for faster searching):



Note: Enabling this setting will create a full-text index for all local replicas, not just the managed mail replica. It should also be noted that the data will be retrieved for building the full text index in a background session, much the same as the data that is retrieved when the managed replica is initially built.

Screencasts

You may also watch the video in full size and in HD on You Tube

How to Identify Truncated Documents


Error when Archiving Truncated Documents

Related Content

Technote #1448134: Understanding Managed Mail Replicas (Q&A, Troubleshooting, and Known Issues)
Technote #1437957 - Configuring managed replicas using the Desktop Settings document
Technote #1505469 - Recommendations to achieve the best results with Notes/Domino Failover functionality
Help topic: What's New in Lotus Notes 8.5, 8.5.1, and 8.5.2?

About the Author

This article was written by Mike Kudla, a Sr. Software Engineer on the IBM Lotus Notes Client Adoption Team.

---