IBM® Traveler® (formerly IBM Notes Traveler) is mobile e-mail software that provides quick access to mail, calendar and contacts. It gives two-way, over-the-air syncing between IBM Domino® servers and wireless handheld devices, including Android devices, and select mobile devices running the Exchange ActiveSync protocol, such as Apple, Windows Phone and BlackBerry 10 devices. IBM Traveler syncs mail, calendar and address book data in real time. On some selected clients, To Do and Journal data are also synced.
IBM Traveler provides a simple, easy-to-use interface with a minimal number of configuration settings. You can customize how much data is synced with the device to optimize the use of device memory and communication bandwidth.
New mail messages from the IBM Domino server arrive on your device automatically and can trigger a notification event, such as a tone or device vibration. Updates made on the device, such as sending a new mail message or changing a calendar entry, sync with the server as soon as a network connection is available.
The IBM Traveler server is installed on a computer running IBM Domino 853 FP6 and above. For basic configurations, the IBM Traveler component operates immediately following installation with minimal input required from an administrator. For highly available configurations, additional steps are required to enable multiple servers to operate in a high availability pool. Day-to-day administrator activities are performed using the IBM Domino Administrator client or IBM Traveler Web Administration and the IBM Domino remote administration console. IBM Traveler uses the Domino directory to automatically look up and find users, as a result there is no manual enrollment procedure.
If you are an IBM Notes® or IBM iNotes® user, then you are already enabled as an IBM Traveler user. Mobile handheld device users must only install the client software depending on the device, and direct the device to an IBM Traveler server. The device automatically registers with the server and syncing begins immediately.
The primary method for IBM Traveler clients communicating with the IBM Domino server is through an over-the-air communication channel. Examples include, cellular General Packet Radio Service (GPRS), WiFi (802.11x) or 3G. The IBM Traveler client works with any secure virtual private network (VPN) installed on the device. It also provides integrated support with IBM Mobile Connect. By using IBM Mobile Connect, you take advantage of the roaming and secure communication features that logically extend the enterprise network to the mobile device, regardless of the physical network that the device is using. The IBM Traveler client can connect using public GPRS or GSM (Global System for Mobile communications) networks and syncs data in transit over a security-rich Hypertext Transfer Protocol/Secure Sockets Layer (HTTP/SSL) connection, helping to safeguard sensitive data while enhancing compliance with your corporate policies and enhances administrator productivity while helping optimize the network throughput.
- Supports remote wipe for lost or stolen devices. You can select full wipe of a device or a partial wipe of only IBM Traveler data.
- Enables security policies that control password length and strength specification, an option to deny access to unencrypted devices and an option to prohibit camera usage.
- Encrypts data at rest (on device) on Apple iOS, Android and Nokia Symbian devices.
- Enhances administrator productivity.
- Provides a single point of control to monitor the IBM Traveler community by user name, device type and operating system version; and to allow or deny access based on company security policies.
- Provides automated client updates.
- Supports scheduled synchronization and data filtering policies to optimize network throughput.
- Supports Linux®, Microsoft Windows® and IBM i servers using supported mobile devices.
- IBM Traveler Version 18.104.22.168 supports the latest mobile devices, documented here.
IBM Traveler syncs mail, calendar, contacts, and journal and to-do data through wireless networks with an IBM Domino server. The software provides basic mail collaboration features, such as create, reply, forward and delete (including attachment support). It also provides meeting-request support, including accepting and rejecting meeting invitations with comments, as well as attachment handling. Eligible IBM Notes customers can download the software free of charge. IBM Traveler software uses a Secure Sockets Layer (SSL) connection for encrypting data that travels over the air using an HTTPS protocol.
Traveler Architecture without High Availability:
Traveler Architecture with High Availability:
How IBM Traveler works
IBM Traveler is installed on an IBM Domino server and runs as a separate add-in task. Having a dedicated infrastructure is preferable, which means nothing else should be deployed on the IBM Traveler servers to ensure adequate system resources are available.
Each mobile device has an IBM Traveler client installed (Apple Active Sync profile, IBM Verse for Android, and Apple devices and Windows Phone using active sync) which communicates with the IBM Traveler server over HTTP or HTTPS. All mobile devices must download the IBM Verse client, except for Windows Phone devices which download a configuration a profile and use the native mail client.
The following components are involved when a user accesses the IBM Traveler server:
- User mail file
- Mobile device
- Domino directory
The following figure shows the main components of a simple IBM Traveler configuration and how they interact:
New mail messages arriving in your Inbox on the Domino server arrive on the device without you needing to do anything (that is, are automatically pushed) and can trigger a notification event, such as a tone or a device vibration. Updates made on the device, such as sending a new mail message or changing a calendar entry are synced with the server as soon as a network connection is available, and are reflected in the user's mail file and Notes client.
The IBM Traveler client provides a simple, easy-to-use interface with a minimal number of configuration settings. You can customize how much data is synced with the device to optimize the use of device memory and server resources.
The IBM Traveler server checks the Domino Directory for the user home server and mail file information and subsequently connects to it. The IBM Traveler server does not store any data, only the user's designated mail server has the user mail file.
In a stand-alone implementation (non high availability) of an IBM Traveler server, there is a local Derby database where information about the user's subscribed folders, devices, and sync status is stored. This derby database also stores any security information about the devices on the system if the administrator of the server has implemented specific settings in regards to devices security. In a high availability implementation of the IBM Traveler server, this information is kept in a central database on a separate enterprise database server (IBM DB2 for Linux, UNIX, and Windows or Microsoft SQL Server) and shared amongst all the IBM Traveler servers in the same high availability pool.
There is a friendly way to see the majority of information stored in the IBM Traveler database referenced above: the LotusTraveler.nsf file, located in the root folder of the IBM Traveler server for stand-alone implementations or a web interface found at http(s)://server_name.domain.com/LotusTraveler.nsf for a high availability implementation. In this view, you can see all devices and users. You can also use this interface to administer these devices and users. In essence, this is a graphical user interface for administering the IBM Traveler's database information.
Daily Administrator Task: Monitoring activities of Traveler Server
The Traveler task is part of the "Server tasks" within Domino Administrator.
You can run the “Show Task” command on a server console to reveal whether the Traveler task is up and running.
For tracking the performance of IBM Traveler, you can check its status through the command “Tell Traveler Status”
Traveler status can be any of the below:
1. GREEN: Indicates server is performing Good.
2. YELLOW: Indicates that there are some errors/issues on the server that need to be addressed.
3. RED: Indicates there are critical server issues that need immediate intervention.
A Domino program document can be used to establish regular output of IBM Traveler server statistics and usage.
In addition, the command "tell traveler stat show"
will give detailed information about the statistics that IBM Traveler maintains. A Domino program document can be used to establish regular output of IBM Traveler server statistics and usage. Statistics can be viewed in Domino Administrator client.
When any of the IBM Traveler statistics (for example, memory or CPU) exceed a given threshold, IBM Traveler goes into a constrained state. Once the constrained state is detected, IBM traveler will not allow new device sync or prime sync threads to start, but existing threads will be allowed to complete. Apple device users may experience a "Cannot connect to server"
error or slow responsiveness, while Android users may see error 503 (server busy)
At the Domino Console on the IBM Traveler server, run the command "show stat traveler.constrained.state" → where 1 indicates the server is running in constrained state.
You can also Configure Event Probes to send a notification to the helpdesk team at times when IBM Traveler's performance degrades.
Setting up a program document
To run the command "Tell traveler status"
at regular intervals, you must set up a program document.
With a program document, you can set Traveler to run the command "Tell traveler status"
at regular intervals, and mail a designated ID with an alert if, at any given point of time, the IBM Traveler status is not GREEN.
Setting up a program document:
A sample mail notification received when there is an event handler in place for when the IBM Traveler status changes to Yellow or Red:
With this alert in place , an administrator can take corrective actions and thus proactively monitor server issues.
Different Kinds of Logs
By default, IBM Traveler is configured to log the messages to the following locations:
SEVERE and WARNING messages are logged to this Domino database
Anything logged to Log.nsf will appear on the Domino console as well, so SEVERE and WARNING messages appear here. The output from the domino console is not logged to a file by default, but by setting the key DEBUG_OUTFILE=filename in your server NOTES.INI, you can save the console output. This can be helpful as there are many messages that appear on the console that are not saved anywhere else
This file contains all messages and traces that are logged by Topsail. If the logging level is set to FINEST, then this file can wrap quickly depending on how many users are using the system and the wrapping settings that are being used.
This file contains on logs with the SEVERE level
This file contains only the usage log records. Each time a device synchronizes or a user does a prime sync, then a record indicating the time, user, sync status, device type and the number of changed records is logged to this file.
This file contains all messages and traces that are logged by Traveler's servlet.
IBM Traveler records all device and user activity in text filse under \IBM_TECHNICAL_SUPPORT\traveler\logs\:
The timestamp corresponds to the date and time that the log file was created. It will wrap when it reaches the maximum defined usage log size, but the logger can be configured to keep older versions as well. Usage logger limits are defined in the file
Messages logged at FINE, FINER and FINEST levels are debug traces. These levels are generally only enabled during testing or by a customer at the direction of an IBM support representative for diagnosing issues. The logging system that Topsail is using is extremely configurable. You can modify the logging level the system is using and change where the log messages are stored.
The IBM Traveler server can log messages with the following levels:
Understanding SystemDump, UserDump, and NTS logs
An unexpected condition, but this could be normal
Debug trace, lowest level
Debug trace, medium level
Debug trace, highest level
IBM Traveler captures its transactions, server statistics, and user statistics in different sets of logs. Based on the type of issue, each of these logs can be read to find the root cause of an issue.
Understanding User Dump Log:
The User Dump log file provides quick information on User statistics, User profile details, missing documents and folders, ad security related issues for the user.
User Dump logs are stored on the IBM Traveler server in the IBM Technical Support folder under the Traveler/dumps directory. These logs get saved with name format of Username, followed by the date and time when the log was created.
User Dump logs can be reviewed as a first attempt at troubleshooting user sync issues, access issues, missing documents on the device, and so on.
There is no actual data stored in User Dump logs from the mail file. However, it has a reference for each document in the mail file, which are used by the IBM Traveler task while syncing that document from the mail file to the device.
The User Dump log can be generated by running the command "Tell traveler dump Username" on the IBM Traveler server.
The user name can be either the mail address or the Notes canonical name.
There are many different sections in a dump log, and are denoted by a line of Hash keys having with the section name in the center.
For example, the first section is identified with the IBM Traveler user name:
# IBM Traveler user dump for #
This section contains basic server information, such as the Domino and Traveler server versions, CPU details, and so on. This is basically the same as what is shown in a SystemDump.
The next section is "Show" section:
This section has 3 different piece of information:
- User configuration
- User profile information
- User device information
IBM Traveler has validated that it can access the database mail/administ.nsf.
Monitoring of the database for changes is enabled.
Encrypting, decrypting and signing messages are disabled because the Notes ID is not in the mail file or the ID vault.
Canonical Name: CN=Administrator/O=Acme
Internet Address: Administrator@acme.com
Home Mail Server: CN=Server1/O=Acme
Home Mail File: mail/administ.nsf
Current Monitor Server: CN=Server1/O=Acme Release 9.0.1FP3
Current Monitor File: mail/administ.nsf
Mail File Replicas:
[CN=Server1/O=Acme, mail/administ.nsf] is reachable.
ACL for Administrator/Acme: Access=Manager Capabilities=create,update,read,delete,copy Missing
ACL for Server1/Acme: Access=Manager Capabilities=create,update,read,delete,copy Missing
As you can see from the example, the user configuration section displays information if IBM Traveler is able to connect and access the user mail database.
IBM Traveler also validates whether the user is having appropriate access to the Traveler server.
If there is an issue with access, if the database is over quota, or any other reason for which the database cannot be accessed, the error will be displayed here and on the profile information section, which displays the user address, Mail server and cluster server details. This will display the path of all servers on which a replica of the user database exists. As a result, this information is referred by the Traveler task to switch to a secondary replica of the user mail database when the primary server goes down.
There is also a section providing information about all configured devices for the user. We have not listed this above, but this page contains information about Device type, Device ID, Device OS, Traveler application version and synch state. In this section, there are two areas for Sync State: "Auto Sync User State" and "Auto Sync Connection State".
Auto Sync User State is for active monitoring of the user's mail database. The Traveler server will monitor a user's mail database for 24 hours after the device last connected to server.
Auto Sync Connection State displays whether or not a particular device has a current active push connection to the server. In other words, if the device is actively connected and syncing.
This section tracks any security action requests that may be applied to the device. For example, wipe, device approval, and so on. If any Security Flag is set for the user's device, the device is denied access to the system. As an example, if you have device approval set, and the user configured a new device which is not yet approved by an Administrator, the security flag for this approval restriction will display here. The Security status section of user dump and system dump is similar.
This section provides information about whether or not the user mail database is actively monitored by the Traveler server. If this is disabled for any reason, the device will not be able to sync mail.
For example, suppose monitoring is disabled for a database. In that case, IBM Traveler will not monitor the user mail database and will not have any new updates to push to the device. Even if the device connects it will show as updated but not sync any new mail that might have been received in the database.
You can quickly reenable database monitoring by running the command "Tell traveler push enable Username".
This gives information on the running threads assigned to this user. It also gives details about how long the thread has been running. A PS thread running over 60 minutes or any other thread running over 10 minutes would be suspicious, but it might not necessarily be a problem.
This section contains cached information about a user's Access Control List, Domino database information, and preferences. Sometimes it may happen that the user was over quota or had an incorrect ACL which was causing mail to not sync. While the issue was being addressed, Traveler might still be referring to old information of the user. In that case it must be because the cache information for the user has not been updated by Traveler. Generally, cache information refreshes automatically, but if you notice it is not updating, perform a "Tell traveler show user" command to force Traveler to refresh the cache for that user.
Profile documents exist in the user's mail database to store information about each device's preferences and security settings. When you set a policy to control Traveler data or apply security settings, the Traveler profile is pushed to the user mail database and this section displays what profiles exist for the user. Profiles are not deleted if you do a "rest" command for the user. They can be deleted using the delete profile tell commands.
Security Policy Status
This section denotes the status of devices complying with any assigned security policy settings. If you are having issues where a policy is not updating correctly, you can validate the user policy status here.
GUID (Global Unique Identifier) Map
This section shows the references of all the documents from the user's mail database that could be synced with at least one device for the user.
Suppose you have 1000 mail messages on your mail file, but your Traveler settings allow the syncing of only the last 30 days. In this case, all mail in this 30 day limit will be mapped in this section.
LGUID is the Local GUID that the device and server use as the key. BACKEND_GUID is the Domino UNID. If a document on a device is missing, you can look to see if the UNID for the document is listed here. If the document is listed, you can see if the folder assigned is the one expected and being synced. You can match the document in this table with the document UNID in the actual mail database of the user. You can find this document UNID (which can be a contact, mail or meeting) in the document properties tab. The first two lines are the DOC UNID, excluding the OFF and ON text.
LGUID: 82615 BACKEND_GUID: 38D46BF5E8F08834852564B500129B2C TYPE: Folder FOLDER: Inbox PCOUNT: 0 FCOUNT: 0
***This section gives Information on how many records the user has in the Traveler database. The more records a user has, the
more capacity the user is consuming.
After the record counts, the current filters are included. So If a user has a large number of records, that might be
caused by the filter settings that are undesirable
You can see one sample output below:
Document Usage by Type
To Do: 0
ApplDLXFJGG3DFHY Mail 0130228receiveddate>=last:3:days
ApplDLXFJGG3DFHY Calendar 0130228startdate>=last:30:days
ApplDLXFJGG3DFHY Contacts 0130228
ApplDLXFJGG3DFHY To Do 0130228complete=f
ApplDLXFJGG3DFHY Notebook 0130228modtime>=last:7:days
ApplDLXFJGG3DFHY Folder 0130228
Where We can see user has 31 Emails, 7 Folders, and 8 meetings
This will also display device filters.
So administrator can understand about how much data user is synching with each device. eg: 30 days data or last 7 days data etc
Understanding SystemDump Logs:
SystemDump log Provides quick information on Traveler status, performance statistics of last 24 hours, All traveler configuration settings. Helps in understanding
performance and configuration issues.
SystemDump log can generated on Traveler server by running command: “tell traveler SystemDump”
While above command is run a Log file gets generated in the Traveler logs/dumps directory
SystemDump log file Dumps all the variables of Traveler server including Domino server document configuration, Traveler
statistics, running threads
You can read the important sections inside the SystemDump that will help to analyze issues at first level
General Information section
This section contains the basic server information like, domino and traveler server version, CPU etc.
This is basically Same as what is shown in User Dump information section
# IBM Traveler Server system dump ##
IBM Traveler Version: 22.214.171.124 Build 201508211840_20
Domino Version: Release 9.0.1FP3|January 12, 2015
Domino Platform: Windows/64 with 4 processors
Domino Install Type: Domino Enterprise Server License (4)
Domino Server Name: Server1/Acme
Database Connection URL: jdbc:derby:ntsdb;create=true
Current Time (Local): Tue Sep 01 20:30:59 IST 2015
Current Time (GMT): Tue Sep 01 15:00:59 GMT 2015
IBM Traveler Started: Tue Sep 01 12:37:45 IST 2015 (running for 0 days, 7 hours, 53 minutes, and 13 seconds)
Last Defrag: Thu Aug 28 12:05:24 IST 2015; Defrag Interval in Days: 30
This section Gives a quick snap shot of the overall system. If the status of the machine is not in a "Green"
state, a list of reason(s) why the system is in Yellow and Red states are listed
The administrator can take corrective actions based on the status responses to make server status back to Green.
The IBM Traveler task has been running since Sun Sep 06 16:07:29 IST 2015.
The IBM Traveler availability index is currently 100 while servicing 1 users.
The last successful device sync was on Wed Sep 09 12:31:13 IST 2015.
The overall status of IBM Traveler is Green.
Current CPU Utilization: CPU running at 0.37 pct over 522.659 Secs
CPU and Memory (MB) Usage History
Date CPU Pct Java Mem C Mem Avl Indx # Users # Errors # DB Conn
2015-09-07 06:15:39 IST 0.00 43 2711 100 0 0 0
2015-09-07 06:30:39 IST 0.05 37 2726 100 0 3 1
2015-09-07 06:45:39 IST 0.03 39 2790 100 0 3 0
2015-09-07 07:00:39 IST 0.04 41 2854 100 0 3 0
If for any reason the Traveler status is not Green the administrator can review the reason and take necessary steps to make the server status GREEN.
Memory Usage Section:
This section gives quick overview on memory availability on server. Helpful to understand if there is more memory fine tuning requirement on server or if there are any memory leak issues that is causing the server performance to degrade.
Java Memory Usage ← Total memory used by Notes Traveler process
Max Total 512 MB ← Maximum memory as configured on the IBM Traveler tab of server document
Current Total 96 MB ← JVM heap size
Free 440 MB (86 percent of Max Total) ← JVM free allocated memory
Allocated 72 MB (14 percent of Max Total) ← Memory being used currently by Traveler
C Memory Usage ← Amount of allocatable memory left on the machine
Total Virtual 2047 MB ← Total virtual memory available left on the machine
Total Physical 2047 MB ← Total physical memory available left on the machine
Allocated 1147 MB (57 percent of Total Physical) ← Allocated memory on the machine
Current Usage ← Summary of the above two values
Java 72 MB ← Traveler used memory
C 1147 MB ← Machine's memory being used
Looking at above statistics, administrator can determine if there are any memory related performance issues on server.
Thread Manager section
This section Lists the active threads, the mapping's of tokens to threads, and an overall summary
The Running time for a thread being high could explain a long sync for a user or higher than expected CPU utilization. If any threads appear hung, a Stop Sync command may be possible if the thread is related to a sync (DS or PS).
Otherwise and for all other issues, a restart of Traveler is probably needed to clear up the issue
You can find a long running thread by looking at this value in this section:
DS-25b4[Email_4][0C4D16EC129F332480257B1F002806AF] [CN=Notes Admin/O=Acme]
[ApplC37JW7PFDTWD] [CN=Notes Admin/O=AcmeApplC37JW7PFDTWDsyncASSyncEmail4] [CN=Notes
Admin/O=AcmeApplC37JW7PFDTWDsyncASSyncEmail4] [0 runnables] [Busy? true] [Last Runnable: Thu Sep 11 18:00:01
IST 2015] [Running: 7125ms] [Idle: 0ms]
A long running PS thread indicates that the user is synching a large amount of data (no limits are set) Thread types to be concerned about:
DS = Device Sync (Connection thread between Mobile device and Traveler server)
PS = Prime Sync (Connection thread between Traveler server and User Mail server)
This section List of all the Traveler statistics that have been collected since the stats were last reset (the statistics are automatically
reset each time the Traveler process is restarted) . Generally the systemdump displays statistics for last 24 hours or since when the task was last restarted whichever is lower.
Some important Statistics to monitor.
Traveler checks the CPU usage on a periodic basis. This stat is a histogram showing how many times the CPU
percentage was in the specified range or bucket. The bucket values are "000-010", "010-020", "020-030", "030-040",
"040-050", "050-060", "060-070", "070-080", "080-090", "090-100". As an example, CPU.Pct.040-050 would show the
number of times the CPU usage was between 40% and 50%
Histogram of the time spent (in seconds) to open a database on the given server using the Domino Java API call. Server
is the name of the Domino server on which the database was opened.
Buckets (In seconds) are "000-001", "001-002", "002-005", "005-010", "010-030", "030-060", "060-120", "120-Inf".
If any server list many request in more than bucket of 2 seconds that denotes some connectivity issues between traveler and that server which could lead to server performance to degrade.
Histogram of the time spent (in seconds) to execute the look-up of a user against Traveler's internal database. A high
number in the larger bucket indicates the need for a defrag.
The total number of devices registered on the server
This statistics give number based overview on how many errors occurred on server and how much errors each user occurred.
Errors = 34
Errors.Administrator = 3
Errors.Test User1 = 30
Errors.Test Admin = 1
Helpful to understand Traveler red status issue due to high error count. Administrator can find here which user is encountering more errors and narrow down the issue to
resolve the Red status issue due to errors.
Prime Synch and Device Synch statistics
These statistics are helpful in understanding success rated of all the prime synchs and device synchs processed by server.
There have been 1300 prime syncs.
The average prime sync took 168 ms.
100 percent (1300) of the prime syncs were successful.
The average successful prime sync took 168 ms.
0 percent (0) of the prime syncs failed.
The average failed prime sync took N/A ms.
There are an average of 0 prime syncs running at any given time.
There have been 285 device syncs.
The average device sync took 4,366 ms and transferred 84,928 bytes.
100 percent (285) of the device syncs were successful.
The average successful device sync took 4,366 ms.
0 percent (0) of the device syncs failed.
The average failed device sync took N/A ms.
Section Show Active
This section List all the users who are actively synching and the type of sync
Section Mail Replicas
This section provides Mail replica information stored in the Traveler database for all users
CN=User/OU=Orig Unit/O=Org = , , , etc.
This section lists Status of the users being enabled for monitoring of mail database changes and the current state for those being monitored
Monitoring is the process by which Traveler detects changes in the user's mail database on a periodic basis. When a
change is detected, the user is queued for a prime sync which may result in push messages if push is enabled
If a user database is not being actively monitored, changes in the user's mail database will not be known to IBM Notes Notes
Traveler and all sync's will be made empty on server side data and when device tries fetching emails, no updates will be
synched as well as no errors
Section Banned Documents
This section provides List of documents that have been banned by traveler server that have been in past caused traveler task to crash.
A document is automatically banned if it causes the server to crash per the value set (default 2)
* NTS_BAN_DOC_LIMIT can be set in the notes.ini (0 to disable this)
Section High availability
High Availability (HA)
This section provides High availability information stored in the Traveler database.
--- HA Servers ---
Domino Name ID Hostname IP:Port Alive Reachable Last Heartbeat AI Users Devices Build Level Startup DB Version
All SMS Migrated Status
--- HA Devices ---
--- HA Users ---
NTS Activity/Audit/Usage/Error Logs:
NTS Logs Captures overall Traveler transactions, helpful in analyzing synch related, performance, crash issues
NTUSAGE Logs all the usages of the Traveler server. NTSUsage logs entry for each transaction that system performs on behalf of the devices. Users synchronize their device with IBM Notes Notes Server to update their mail, contacts and calendar information and this activity is logged in the NTSUsage log
The transactions are logged once they complete (not when they begin) and this information can be useful to the administrator to determine if devices are synchronizing with the server and how long this process is taking. The administrator can monitor the log for issues such as when the device last connected, what was synchronized between the device, the return codes and the time the synchronization took to complete
"10/15 15:28:44.584" 192.168.56.1 192.168.56.2 "CN=Test User/O=Acme" "http://192.168.56.2/mail%2Ftuser.nsf?
SendMail" dp 200 328 "Apple-iPad2C1/902.206" ApplDLXFJGG3DFHY 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 "D2S Mail 1A"
Reading the transaction
[Device IP Address]
[Server IP Address]
[User, Database or Action]
"CN=Test User/O=Acme" "http://192.168.56.2/mail%2Ftuser.nsf?SendMail"
Identifies who requested the sync (ps = prime sync, dm = device manual, dp = device push, sp = server push)
[Status Return Code on request]
200=OK, 408=Request Timeout (Device did not respond before the Server timed out the session), 409=Conflict
(device started a new session which caused the session to be aborted), 500=Unknown Error, 503=Server Busy
[Time (ms) taken for request to complete]
[User Agent involved in transaction]
[Changes from (Device to Server) (18)]
1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
[Changes (Server to Device) (18)]
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
[Summary of Changes (only present if there are non-zero Changes)]
"D2S Mail 1A"
Device 2 Server, this was a Mail, and 1 add was done
→ Mail-Add Mail-Update Mail-Delete Contact-Add Contact-Update Contact-Delete Calendar-Add Calendar-Update
Calendar-Delete ToDo-Add ToDo-Update ToDo-Delete Journal-Add Journal-Update Journal-Delete Folder-Add
NTSErrors log files contain any errors that IBM Traveler has encountered. These are just the SEVERE level messages that were encountered by server.
[TimeStamp] [Level] [Thread Name] [User] [Class.Function#Line] [Message]
[02/27 14:43:40.325] SEVERE PS-3eec CN=User/OU=Org Unit/O=Org BackEndManager.validateDBAccess#563
Exception Thrown: com.lotus.sync.dca.CAException: Notes error: The server is not responding. The server may
be down or you may be experiencing network problems.
Additional debug messages (not always present)
Stack trace (not always present)
at lotus.domino.local.Database.NqueryAccess(Native Method)
at lotus.domino.local.Database.queryAccess(Unknown Source)
In case of any issues on server, administrator can read these logs to understand detailed transaction of errors.
NTSACTIVITY logs Lists all the errors and the usage activity that are seen in the NTSErrors, and NTSUsage logs. Further it Lists all tracing that is done on a user/server
Below is the Basic layout of the log message
[TimeStamp] [Level] [Thread Name] [User] [Class.Function#Line] [Message]
[10/15 15:28:44.584] FINE DS-0f18[Email_4] CN=User/OU=Org Unit/O=Org ActiveSyncSyncSession.run#232
ActiveSync starting for user CN=User/OU=Org Unit/O=Org on device ApplDLXFJGG3DFHY
[10/15 15:28:44.584] FINE DS-0f18[Email_4] CN=User/OU=Org Unit/O=Org ActiveSyncSyncSession.run#232
ActiveSync starting for user CN=User/OU=Org Unit/O=Org on device ApplDLXFJGG3DFHY
Timestamp: The local date and time to millisecond precision
Level: SEVERE, WARNING, INFO, FINE, FINER, FINEST
Thread Identifier: The thread name with an optional correlator in brackets [ ]
User Name: The user's ID in canonical name format
Source Code ID: The class, method, and line number that originated the message log
Message: The message can be any text
NTSAudit files logs information of all changes to the system, such as settings changes (server document, notes.ini, and so on) and server state changes (up, down, and so on)
They also Log all the tell commands fired on the server
Sample details in the NTSAUDIT Log file.
"10/15 19:27:10.513" Profile *System* "Profile updated: devPmaxAtt - oldValue=20480, newValue=4000"
"10/15 19:27:10.513" Profile *System* "Profile updated: devPbTrunc(devPbTrunc) - oldValue=5000, newValue=4000"
"10/17 08:19:12.545" TellCommand *Console* "tell traveler Status: status=OK, time=188 ms"
"10/19 07:29:08.664" TellCommand *Console* "tell traveler Memory show: status=OK, time=15 ms"
"10/20 11:47:57.056" Configuration *System* "Non-default value: NTS_64_BIT - oldValue=true (Default),
"10/20 11:47:57.056" Configuration *System* "Non-default value: NTS_BUILD - oldValue= (Default), newValue=126.96.36.199
These logs are helpful to understand if any changes were made to server.
How to analyze the logs: Log analysis to resolve Traveler issues
1. Scenario 1: Performance issue on server
1. Users experiencing Very slow mail synchronization on devices via Traveler Server
2. CPU spike / High CPU utilization by Traveler server
3. Memory time-out errors on Traveler Server
4. Devices not synchronizing
Generally, whenever you see performance degradation, always issue system dump command and collect system dump at the time of issue.
System dump log will provide a fair idea about what could be the cause of traveler performance degradation.
However not necessarily system dump may give clear hints on issue sometimes and further finest logging might need to be set and logs to be collected.
You can see below sample example where the Traveler status shows RED.
→ Example: Traveler status section in System dump
[0C6C:3711-3A8C] The IBM Traveler task has been running since Mon Jul 27 01:16:34 IST 2015.
[0C6C:3711-3A8C] The IBM Traveler availability index is currently 70 while servicing 1,869 users.
[0C6C:3711-3A8C] The last successful device sync was on Mon Aug 24 11:59:55 IST 2015.
The overall status of IBM Traveler is Red.
When Traveler status is not Green, the systemdump will clearly indicate all reasons why it is Not green. Eg: either it is Yellow or RED.
As you can see in this example, there are multiple reasons seen causing the Traveler status to turn RED.
Reason 1. The IDX table or index has a large number of pages allocated that are unfilled.
Which means Traveler NTSDB requires a defrag.
Reason 2. The peak number of HTTP connections are more than what currently are allocated.
You need to calculate required HTTP thread based on how many devices are configured with server.
If HTTPS threads set on server are low, then they will cause performance issue.
Reason 3. We can see that response time taken by traveler for communicating with one of the mail server is too high. This can lead to threads being held up which impacts performance.
Reason 4 and 5: you can see Traveler reported many errors in last interval as well as there are few threads which are running for a longer time.
Not necessarily you will see all these errors all together on server.... but we have just included it here for demonstration
Post checking status section, check the CPU and memory table in system dump.
This gives information on last 24 hours usage statistics. which will help understand. at what times the performance is at peak.
As you can see on screen in morning around 11:22 AM the usage was at peak
So overall observations from this scenario you can conclude that:
1.HTTP threads utilization are high
2.NTSDB Requires maintenance
3.There are Connectivity issues between Traveler and Mail Server in particular
4.There are some Hung thread
So correcting this observation should resolve the performance crunch
Scenario 2: Traveler task configuration or startup issues.
You can come across issue where Traveler task is not starting.
Whenever you see startup issues, check console log carefully. Most of the time for startup related error console log itself will give clear observation on what is wrong.
Try to see any abnormal message that shows during startup of Traveler and HTTP task.
As you can see below sample example, there are many errors that shows connection error to NTSDB.
IBM Traveler: Server starting...
IBM Traveler: SEVERE *system Connection to database ntsdb unable to be made.
Verify that you have properly created the remote database. Exception Thrown: java.sql.SQLException: Failed to start database 'ntsdb' with class
loader sun.misc.Launcher$AppClassLoader@72b072b, see the next exception for details.
IBM Traveler: SEVERE *system The component com.lotus.sync.dca.BackEndManager could not be started. Internal
error: com.lotus.sync.db.PersistenceException: java.sql.SQLException: Failed to start database 'ntsdb' with class
loader sun.misc.Launcher$AppClassLoader@72b072b, see the next exception for for details.
IBM Traveler: SEVERE *system IBM Traveler Server could not be started. The exception was
com.lotus.sync.util.ComponentNotStartedException: A TrueSync Server startup timeout has occurred.. Exception
Thrown: com.lotus.sync.util .ComponentNotStartedException: A TrueSync Server startup timeout has occurred.
So with this observation we can conclude that Traveler task is unable to read NTSDB.
Traveler task reads NTSDB while starting up. So if that database is corrupt that will restrict Traveler task from starting up.
So in this case, you can try starting Traveler task with command line "Tell traveler -defrag" which forces a maintenance on NTSDB before starting the Task.
This should correct any corruption issue with NTSDB and further task should start.
Sometime it might happen that Traveler task starts but users cannot synch emails or Traveler page does not load.
In such case as well check the console for any abnormal error during Traveler and HTTP task startup.
Traveler has a dependency on HTTP. So if there is any error even with starting any of HTTP component that will not allow traveler to service users.
See below example where Traveler and HTTP both tasks were started.
However, HTTP even while it was started, it was unable to initialize XSP which is one of the HTTP component.
[1798:0002-2200] 09/06/2015 04:03:41 PM Traveler: Loading HTTP server.
[10D8:0002-0870] 09/06/2015 04:03:41 PM HTTP Server: Using Web Configuration View
[1798:0002-2200] 09/06/2015 04:03:42 PM Traveler: Server started.
[10D8:0002-0870] 09/06/2015 04:03:45 PM JVM: Java Virtual Machine initialized.
[10D8:0002-0870] 09/06/2015 04:03:45 PM HTTP Server: Java Virtual Machine loaded
[10D8:0002-0870] 09/06/2015 04:03:45 PM HTTP Server: DSAPI Domino Off-Line Services HTTP extension Loaded successfully
[10D8:0002-0870] 09/06/2015 04:03:50 PM Xsp Initialization error - Could not load class or methods
[0738:000C-2B40] 09/06/2015 04:03:51 PM This server is currently a member of a cluster
[10D8:0002-0870] 09/06/2015 04:03:55 PM HTTP Server: Started
For detailed debug why xsp initialization fails, you can start Traveler task with debug command line “load
For this example, this must be because any of JVM file is corrupt and that's why the xsp is not initializing. You can copy the JVM folder from other good working server on same OS and release and see if it helps.
Else you might need to reinstall domino which will correct these JVM files.
Scenario: Traveler servers in HA not communicating with each other.
While having Traveler in High Availability setup there can be issues when Traveler within the HA pool are not communicating with each other.
In above example both of the servers are up but one server is not reachable to other traveler server.
The command to check all server’s status in HA pool is Tell traveler HADR show
It might be that no communication is received from other server which is causing to show its availability as false.
As a basic troubleshooting step in such scenarios, you can run the command “tell traveler HADR ping unreachable servers name from the console of other working traveler server.
If ping command displays correct results, then rerun the command tell traveler hadr show. and Both servers should be reachable now
If ping command displays incorrect results, then there may be a network connectivity problem between the two servers or there might be Traveler or http task startup issues on other server which is why it is not reachable
Scenario: User connectivity / configuration issues.
There may be different kind of these issues like:
User receives error while logging in to traveler home URL or connecting to Traveler Server from device.
Few mails do not synchronize body properly otherwise synch is working normally
Unable to download certain type of attachment
Traveler client crash on Device or has very slow response
For any of these Traveler issue related to particular user, always issue a user Dump command against affected
user and check the User dump key sections as we discussed earlier to see if user has correct access and Traveler is able to reach and
access user mail database
User dump will provide further information on the synch issue related to particular folder, document
For example, if user complains about any missing document or folder, check in the GUID map section of user dump to understand if Traveler was able to read that document from user database.
Sometime user may report issues that they are unable to configure Traveler on their device.
You can ask user to login to Traveler homepage and see if user can login or if any error is displayed on home screen.
You can refer below sample errors by traveler on user homepage if there is any configuration issue.
Like we see, in this case either the database is over quota or Mail server is not reachable from Traveler server or User does not have sufficient access which caused user connection/configuration to fail.
In such scenarios of user configuration issues, user dump will provide detailed information on the error.
Check in user dump if Traveler server is able to read and resolve Notes User ID
Check if Traveler server is able to read user database and its replicas across cluster
Verify that there is no security Flag, restricting user and its accessibility to system
Verify that the monitoring of database for changes is enabled
You can see one below example on screen where we can see that user mail database cannot be accessed which will cause the user configuration to fail.
Description: Gives a quick snap shot of the user and any devices.
IBM Traveler could not open the database mail/administ.nsf. Verify that the server CN=Server1/O=Acme and the database
grant access to server CN=Server1/O=Acme and that there is a network connection available between these servers.
Monitoring of the database for changes is disabled.
Canonical Name: CN=Administrator/O=Acme
Internet Address: Administrator@acme.com
Home Mail Server: CN=Server1/O=Acme
Home Mail File: mail/administ.nsf
The IBM Traveler server cannot connect to your mail database mail/administ.nsf on server CN=Server1/O=Acme. Verify that
your mail server mail database grants access to server CN=Server1/O=Acme and is operational. If this does not resolve the
problem, your administrator may need to verify the network connection between the servers and that the IBM Traveler server
is allowed to access your mail server.
[CN=Server1/O=Acme, mail/administ.nsf] is not reachable, status(0x246) "You are not authorized to perform that operation".
In case of user level connectivity issues, the device and server logs needs to be collected most of the time.
It might be a new user configuration or an existing user unable to connect to server and synchronize mails
AT first level,
Check if, Traveler home URL is accessible from user's device. Just to ensure there is no mobile internet issue reaching Traveler server
Check if, any error is seen in User profile statistics of User DUMP log.
As a next step Collect logs from device for Traveler application (Enable logging in Traveler application on device and then collect logs)
It may be necessary at times to also enable HTTP debug by running http debug command as shown on screen to collect HTTP logs.
Http logs help understand if the request generated from device is getting modified somewhere at network level before it reaches traveler server.
Always enable finest logs while tracing issue and then ask user to reproduce issue before collecting logs.
Scenario: Mail synchronization issues
user might report issue wherein certain mail body gets truncated or format is altered when email is synched to device or replied from device.
Certain attachments do not synch to device
Some documents for example Email or Calendar or a contact or folder do not synchronize on device
Device hangs synching certain type of emails or may be Traveler client unexpectedly closes on device.
Issue like Email body truncation or attachment download issue
For this kind of issues, it is unlikely to get any observation in user dump.
So you will need to enable Traveler debugs for affected user and recreate issue and then collect logs.
However, as a first step towards resolution, please try to replace data on the device so that it synchronize everything on mobile device.
You can run reset command on server to initiate resynchronization of data for user.
Further the NTS Activity and Error logs will help understand what goes wrong for affected document causing issue.
You can see below sample error, where the transaction fails while getattachment call.
Which means while retrieving attachment it failed.
[10/09 20:06:54.452] SEVERE
DS-0048[92EshAAA][23DA9CACB55713DB85257BB40073F6B2] Test User/Acme ContentStore.getAttachment#1363
serverName=CN=Test/O=Acme, databaseName=mail/testuser.nsf, and
dcaDocument=NoteId=830726, 0xcad06, wantsAttachments=true,wantsHtmlBody=false, wantsPlainTextBody=true,
itemName=,isSigned=false, m_FullRetrieve=true, m_retrievedAttachments=true,m_fieldMap.size(0),
Attachments(2)=[name=actn130.gif, size=1085,mimeType=image/gif referenceId=actn130.gif@icon
embedded=truestreamable=true hasData=true, name=, size=0, mimeType=imageeferenceId=@ embedded=false streamable=true
database=mail/sedgecom.nsf Exception Thrown: java.lang.NullPointerException: Exception occured in ILE native method, see
joblog. java.lang.NullPointerException: Exception occured in ILE native method,see joblog.
But just as an initial troubleshooting try to narrow down issue.
Check if issue is specific to certain mail type or device type etc.
If any error is printed on console try to verify if already this is reported for Traveler and any fix already exists for same.
Always keep handy the below URL mentioned to look for known reported errors and there remedies.