Analyzing IBM Lotus Notes Client hangs and crashes
Uttam Kumar
Staff Software Engineer
IBM Software Group
Pune, India
June 2009
Abstract: Learn how to distinguish between an IBM Lotus Notes client crash and hang, and what you can do to analyze and correct both. The material in this article applies to Notes 6.x, 7.x, and the Notes 8.x Basic Client.
Introduction
There are three different types of Notes clients:
· Lotus Notes Client. Used as a messaging client and for accessing an application.
· Lotus Domino Designer Client. Used to create a database or application.
· Lotus Domino Administrator Client. Used to manage and monitor the servers, groups and users.
Here we discuss the Lotus Notes client, one of the most popular messaging clients. It is rich with user-friendly features but, of course, it can sometimes fail or shut down, rendering it unusable. This article provides some basic tips on how to analyze and fix Notes client crashes and hangs.
To get the most from this article, you should be familiar with basic Notes/Domino concepts and terminology or be an experienced Domino administrator.
Notes client overview
When the Notes Client is launched, its executables get added into the Windows Task Manager. Depending on the Notes client type, the executables that run are Nlnotes.exe and Notes.exe (for Notes Client 7).
When launched, Notes.exe does three main things and then calls Nlnotes.exe:
· Displays the splash screen
· Determines whether Lotus Notes is already running (and aborts, if it is)
· Loads Qnc.exe, if you have it configured to do so
The Ntaskldr.exe file is also loaded with Notes Client process. Ntaskldr.exe is the Notes Task Loader in Notes Client 7.x and serves as the replacement process for multiple Notes R5 Client tasks such as nupdate, namgr, nhldaemn, and naldaeam.
In the Notes R5 Client a separate process runs for each individual Notes task. In contrast, Ntaskldr.exe is a single process that runs on Microsoft Windows operating systems, carrying out the various tasks by spawning threads instead of loading individual processes.
The executables corresponding to the Notes Client 7.x and later versions on the Processes tab of the Windows Task Manager are Nlnotes.exe and Ntaskldr.exe. Note that the Ntaskldr executable does not load until a Notes Client task is started.
Lotus Notes Client 8.x
Notes 8.x is designed differently from earlier Notes client versions. First of all, there are two different types, the Basic Notes Client and the Standard Notes Client.
When you launch the Notes Client from Start > All Programs > Lotus Applications, it will launch the Standard client, and the Basic client will run in the background. If you want to launch the Basic client, you can do so from the installed location or by using Start > Run, and typing “notes –sa”.
The Standard Client is a Java / Eclipse client and visible to users, whereas the Basic Client is a headless client (invisible to users). You can see two different executables, for example, notes2.exe/notesw.exe for the Standard Client and nlnotes.exe for Basic client in the Task Manager.
There are many plug-ins that can be added to the Notes 8.x Client, and below are some of the new features/options:
· A new, Java-based client option with a newly designed UI, including an "Open" button to access your bookmarks
· Name change from 'Welcome Page' to 'Home Page'
· Hover Help/Tool Tips for Action bar items
· Folder name change from "Databases" to "Applications"
· Multi-level "Undo" operations, including more operations that can be undone
· Additional "Undo" operations for tables
· Usability improvements in the Application Open dialog
· Enhanced Attachments dialog allows access to Windows-based places, for example, My Documents, Desktop, or My Computer (Windows only)
· Property infobox has better contextual graying
· Names and text fields have been enhanced to show ellipses, instead of allowing multiple lines in a field.
· An enhancement to support URL lengths greater than 250 characters
· Viewer support for Open Office-formatted documents
· Support for multiple monitors (Windows only)
· An Eclipse-based Help system that lets you display context-sensitive help in a side panel while you work. In addition to help for Lotus Notes, this Help includes sections on Sametime Connect, Activities, the Composite Application Editor, and the IBM Word Processing, Spreadsheet, and Presentation tools, all of which you can choose to install as part of the client install.
What is a Notes Client crash?
In a crash, the Notes Client is terminated and is no longer running. When Lotus Notes crashes, it creates a Diagnostic file (NSD or RIP, depending on the Notes release) for troubleshooting, with the .log extension. The NSD file is under the Notes > Data > IBM_TECHNICAL_SUPPORT directory and starts with “NSD”; for example:
Nsd_W32I_UTTKUMAR_2008_04_15@17_03_32.log
where the format is:
nsd__name>_@.log
A Notes Client crash can be one time, reproducible, or repetitive.
One-time crash. This type of crash is quite difficult to analyze; for example, when Lotus Notes crashes once while launching.
Reproducible crash. In this case you have the steps to reproduce the issue. For example, say a particular document is corrupted and the client crashes every time you open it.
Repetitive crash. In this case, the crash occurs on a particular schedule or event. For example, say there’s an agent written to run on a particular event and, whenever the event occurs, the agent runs and crashes Lotus Notes.
Some of the common reasons for a client crash are:
- A software problem in the code (on the client)
- Corruption in a database or document
- Insufficient memory
- Customized code
What is a Notes Client hang?
A hang is a situation in which the Notes Client seems to be running, but one or more tasks is not responding to requests. Most of the time, a hang occurs due to a low-level loop or a permanent unavailability of a resource, causing serious performance issues.
During a hang, the program seems to freeze, or the application does not respond to users' actions. Keyboard input or mouse clicking has no effect, regardless of where the cursor is placed, yet the program is still running.
Sometimes a hang will resolve itself, and the application resumes its normal execution without your involvement, in which case it might be considered more of a performance issue than a hang.
NOTE: In a Notes Client hang situation, an NSD is never generated automatically; you must manually create an NSD file using these steps:
- Open a Command Prompt window (Start > Run > and type cmd.exe).
- Navigate to the Notes Data directory, and issue the nsd command. If the Notes program directory is not included in the system's path statement, you must specify the full path to the NSD executable.
IMPORTANT: NSD should not be run from a Notes program directory because it may not attach to the Notes processes.
- Once NSD has completed, it will return you to an NSD prompt:
- Retrieve the NSD output file from the IBM_TECHNICAL_SUPPORT directory.
Must-gather information
When the Notes Client crashes, the following information must be gathered:
· How many users are getting the crash?
· Were there any error messages displayed during the crash? If so, what were they and where were they seen?
· What was the user doing at the time of the crash?
· How often does the crash occur?
· Can the crash be reproduced at will? If so, what are the exact steps to reproduce it?
· What other programs, applications, and operating system tasks were running at the time of the crash?
Once we have the NSD we can start troubleshooting, using the NSD and the above information. But first, we need to understand which part of the NSD is applicable:
· If one user is getting the crash, ask that user to use a different workstation with his ID and see whether he is still getting the crash. If yes, then the crash is related to the User ID or database. If not, then it must be related to the workstation and/or local databases like the Desktop6.ndk, Bookmark.nsf, and cache.ndk. So in this case, the user needs to make a backup and recreate these files.
· If the crash is happening in only one workstation for multiple users, then we need to look at the Notes configuration.
· If the crash is related to a database or application, run the maintenance tasks (fixup, compact, and updall) on it.
· If the crash is related to one document, we need to review the document’s design element (size and limitation) as every element has some limitation.
How to troubleshoot using the NSD
The Notes Client 7.x NSD is stored in the Notes > Data > IBM_TECHNICAL_SUPPORT directory. The format used in NSD files is:
nsd___@.log
Let’s review the key sections of the NSD and learn to analyze them with respect to client crashes.
Workstation detail. From this section of the NSD you can see the Notes version (Release 7.0) and operation system (Windows XP 5.1 SP2), and from the Release Notes database you can verify whether the configuration is supported for this client version. You can also see the time of the Notes Client crash, in this case, Tue Apr 15 17:03:32 2008.
Host Name : UTTKUMAR
User Name : Administrator
Date : Tue Apr 15 17:03:32 2008
Windows Dir : C:\WINDOWS
Arguments : "C:\Lotus7_old\notes\nsd.exe" -dumpandkill -termstatus 1
NSD Version : Release 7.0
OS Version : Windows XP 5.1 (Build 2600), PlatID=2, Service Pack 2 (2 Processors)
Notes detail. From this section we can see where the Notes Client is installed—which must be in the local drive. Keeping the data folder in the file server or Network drive is not supported. (As a workaround, however, you can configure a Citrix environment.)
<@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@>
Section: Notes Process Summary (Time 17:03:42)
<@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@>
Ini File : C:\Lotus7_old\notes\notes.ini
Data Dir : C:\Lotus7_old\notes\data
Prog Dir : C:\Lotus7_old\notes
Fatal Stack. Here, search for the keyword “Fatal” to find the function responsible for the crash. Once you have the function, you can review it from the top and, if you have knowledge of database or source code, you can check the functionality yourself, or report it to the Support or Development team for further action.
### FATAL THREAD 1/12 [ NLNOTES:01ec: 1156]
### FP=0x0012f50c, PC=0x7700a324, SP=0x0012f504, stksize=8
### EAX=0x00000000, EBX=0xc0000000, ECX=0x61000000, EDX=0x00000000
### ESI=0x611ad840, EDI=0x00000000, CS=0x0000001b, SS=0x00000023
### DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010202
Exception code: c0000005 (ACCESS_VIOLATION)
[ 1] 0x7700a324 MFC42u.Ordinal4151+411 (611ad840,1046a,849,0)
[ 2] 0x77e16b21 USER32.ScreenToClient+213 (611ad840,1046a,849,0)
[ 3] 0x77e25010 USER32.CallWindowProcW+25 (611ad840,1046a,849,0)
[ 4] 0x717521f0 COMCTL32.Ordinal363+457 (1046a,849,0,0)
[ 5] 0x7175276f COMCTL32.Ordinal413+595 (1e9d50,1046a,849,0)
[ 6] 0x71752574 COMCTL32.Ordinal413+88 (1046a,849,0,0)
[ 7] 0x7172c11b COMCTL32.InitializeFlatSB+2301 (1046a,849,0,0)
[ 8] 0x7175276f COMCTL32.Ordinal413+595 (1e9d50,1046a,849,0)
[ 9] 0x71752643 COMCTL32.Ordinal413+295 (1046a,849,0,0)
[10] 0x77e3a2d0 USER32.SetWindowPlacement+80 (717525ac,1046a,849,0)
[11] 0x77e145e5 USER32.TranslateMessageEx+1541 (12f7c0,0,61222003,12f7c0)
[12] 0x77e1a816 USER32.DispatchMessageW+11 (401000,12f15c,1340bc,0)
@[13] 0x0040156d NLNOTES._WinMain@16+1389 (400000,0,1340bc,1)
@[14] 0x00401dfa NLNOTES._WinMainCRTStartup+308 (12f15c,7c590615,7ffdf000,134878)
[15] 0x7c5987e7 KERNEL32.ProcessIdToSessionId+381 (401cc6,0,c8,100)
Memory Dump. In this section you can get clues as to why the client is crashing, such as Notes crashing due to a malformed email address, which you can see here. Also, if a .dll file is corrupted, you’ll also be able to see the .dll files’s name in this section.
Once you find the document with the offending email address, you can delete the document or use other workarounds to avoid the crash.
#
PASS 2 : FATAL THREAD with STACK FRAMES 1/19 [ nlnotes: 4480: 5740]
### FP=0013d8d0, PC=61378bf3, SP=0013d8c8, stksize=8
Exception code: c0000005 (ACCESS_VIOLATION)
#
Top of the Stack
# 0013d8c8 0013da98 05f48814 0013d9f0 612365e6 |.............e#a|
@[ 1] 0x61378bf3 nnotesws.CNEMOutlineView::BroadcastUpdate+51 (5f48814,8,0,5f48814)
Short Callstack detected, dumping extra stack data
# 0013fff0 00000000 00000000 00401cc6 00000000 |..........@.....|
# 00140000 78746341 00000020 00000001 00002498 |Actx ........$..|
# 00140010 000000c4 00000000 00000020 00000000 |........ .......|
# 00140020 00000014 00000001 00000006 00000034 |............4...|
# 00140030 00000114 00000001 00000000 00000000 |................|
# 00140150 00000000 00000000 00000000 00000000 |................|
# 00140160 00000000 00000000 00000002 00000024 |............$...|
# 00140170 00000034 003a0043 0057005c 004e0049 |4...C.:.\.W.I.N.|
# 00140180 004f0044 00530057 0057005c 006e0069 |D.O.W.S.\.W.i.n.|
# 00140190 00780053 005c0073 00000000 00000000 |S.x.s.\.........|
Open Database. In this section you can see all the databases that were open at the time of the crash and verify whether all the databases have a correct ODS or not. If not, then update the ODS.
<@@ Notes Memory Analyzer (memcheck) -> Open Databases (Time 17:03:48) @@>
C:\Documents and Settings\Administrator\Local Settings\Temp\notesB720D4\~editclp.ncf
Version = 43.0
SizeLimit = 0, WarningThreshold = 0
ReplicaID = 0x6525742c:0x00378139
bContQueue = NSFPool [ 00049e65]
FDGHandle = 0xf02401ed, RefCnt = 1, Dirty = Y
DB Sem = (FRWSEM:0x0244) state=0, waiters=0, refcnt=0, nlrdrs=0 Writer=[ : 0]
SemContQueue ( RWSEM:#0:0x029d) rdcnt=-1, refcnt=0 Writer=[ : 0], n=0, wcnt=-1, Users=-1, Owner=[ : 0]
By: [ nlnotes: 4480: 2] DBH= 70, User=CN=Uttam Kumar/OU=India/O=IBM
CN=CAMDB07/OU=CAM/OU=A/O=Lotus!!o_dir\OS2SPRS.NSF
Version = 43.0
SizeLimit = 0, WarningThreshold = 0
ReplicaID = 0x85256084:0x004e8881
bContQueue = NSFPool []
FDGHandle = 0x00000000, RefCnt = 11, Dirty = N
DB Sem = (FRWSEM:0x0244) state=0, waiters=0, refcnt=0, nlrdrs=0 Writer=[ : 0]
SemContQueue ( RWSEM:#0:0x029d) rdcnt=-1, refcnt=0 Writer=[ : 0], n=0, wcnt=-1, Users=-1, Owner=[ : 0]
By: [ nlnotes: 4480: 2] DBH= 32, User=CN=Uttam Kumar/OU=India/O=IBM
By: [ nlnotes: 4480: 2] DBH= 77, User=CN=Uttam Kumar/OU=India/O=IBM
By: [ nlnotes: 4480: 2] DBH= 87, User=CN=Uttam Kumar/OU=India/O=IBM
C:\Lotus7_old\notes\data\headline.nsf
Version = 43.0
SizeLimit = 0, WarningThreshold = 0
ReplicaID = 0x652572bd:0x003ffd45
bContQueue = NSFPool [ 0001b525]
FDGHandle = 0xf024004e, RefCnt = 1, Dirty = Y
DB Sem = (FRWSEM:0x0244) state=0, waiters=0, refcnt=0, nlrdrs=0 Writer=[ : 0]
SemContQueue ( RWSEM:#0:0x029d) rdcnt=-1, refcnt=0 Writer=[ : 0], n=0, wcnt=-1, Users=-1, Owner=[ : 0]
By: [ntaskldr: 5072: 6] DBH= 9, User=CN=Uttam Kumar/OU=India/O=IBM
CN=d23ml171/OU=23/OU=M/O=IBM!!mail9\uttkumar.nsf
Version = 43.0
SizeLimit = 0, WarningThreshold = 0
ReplicaID = 0x65256e3d:0x007b65a6
bContQueue = NSFPool []
FDGHandle = 0x00000000, RefCnt = 28, Dirty = Y
DB Sem = (FRWSEM:0x0244) state=0, waiters=0, refcnt=0, nlrdrs=0 Writer=[ : 0]
SemContQueue ( RWSEM:#0:0x029d) rdcnt=-1, refcnt=0 Writer=[ : 0], n=0, wcnt=-1, Users=-1, Owner=[ : 0]
By: [ nlnotes: 4480: 2] DBH= 64, User=CN=Uttam Kumar/OU=India/O=IBM
By: [ nlnotes: 4480: 2] DBH= 59, User=CN=Uttam Kumar/OU=India/O=IBM
By: [ nlnotes: 4480: 2] DBH= 61, User=CN=Uttam Kumar/OU=India/O=IBM
By: [ nlnotes: 4480: 2] DBH= 65, User=CN=Uttam Kumar/OU=India/O=IBM
Local Disk. Here you can see the different drives available on your workstation and can also determine whether you are using any network drive or mapping.
INFO (0): X:\ volume info error
DRV Total (KB) Used(KB) Free(KB) % Used Remote Connection
C:\ 32860924 26086784 6774140 79%
D:\ 25736096 18831284 6904812 73%
H:\ 209712476 209315476 397000 17% \\moh1\premak
I:\ 83883364 74836556 9046808 38% \\moh2\psd
L:\ 83883364 63421920 20461444 24% \\moh2\division shared
N:\ 13639152 7214968 6424184 52% \\sgmoha004\spss$
Q:\ 47102548 36101920 11000628 76% \\dwweb\data_entry
Shared Memory. From this section you can determine whether the Notes Client is crashing due to a memory issue. If so, you must kill some of the process to free up memory.
Total Physical Memory: 1014.0M
Avail Physical Memory: 264.7M
Memory Usage : 73%
Total Paging File : 1.6G
Avail Paging File : 1.0G
Total Virtual Memory: 2.0G
Avail Virtual Memory: 1.8G
DBG (0ad8) 08:53:29
Conclusion
You should now understand the differences between a Notes Client crash and a hang, and should be able to perform troubleshooting procedures to analyze and fix client crashes. Hopefully, you can use this article as a helpful resource whenever you encounter a hang or crash with the Notes 6.x, 7.x, or 8.x Basic Client.
About the author
Uttam Kumar is a Staff Software Engineer currently working with the Notes Client team at IBM’s Pune, India, facility. He has five years of IT experience, and has previously worked on teams such as Lotus Workflow, Lotus Support , Domino Designer, Lotus SmartSuite and Domino Access for Microsoft Outlook.