This article will describe areas in an NSD log to gather information on memory related crash on HTTP process.
Memory exhaustion defined by “Insufficient memory” messages
Memory Handles defined by “Out of Handles” messages
NSD log contains memory summary for Domino processes. Start by reviewing the Process Memory Mappings section for HTTP process. For NSDs generated from Unix OS, use “Process Heap” size under “Process Heap Memory Stats” instead.
The Process Memory Mappings section above shows HTTP private memory is over 1G in size.
Next, identify where is majority of HTTP memory allocated. Overall HTTP memory is made up of the following areas:
1. Private Static-DPools
3. Thread stack Space
Memory allocations performed by Notes using various DPools (4 MB to 8 MB). These pools are not freed back to the OS after being used.
NSD or Memcheck log look at “Process Heap Memory Stats”
Debug for troubleshooting high memory usage in private DPools:
- Check Top 10 section for HTTP in NSD log to identify any memory blocks
- Memcheck logs can be used to track memory usage over time. These can be automated by setting up Program document to run "-memcheck -perf" over period of time.
- DPool debug for HTTP process will generate call stacks in Console log when DPool is allocated. Note, DPool debug can also be used for Shared memory which does not apply in this case.
Example of DPool debug for HTTP on Windows OS uses 'nhttp' as value. In this example, debug will track DPool allocations for memory block 0x044
Memory blocks allocated directly from the OS and released back to the OS when freed by the caller. From NSD log search for “Directly Allocated Memory” under memcheck for HTTP process.
If Directly Allocated Memory is showing high memory usage with blk_local or blk_local_block, add Notes.ini parameter inotesuseNotesMemory=2 to help manage memory more efficiently. In some cases when using this parameter, Admins may notice high handle count reported. It is then recommended to change setting to 0.
Memory is allocated directly from the OS
Memory is allocated through BLK_LOCAL or BLK_LOCAL_BLOCK
Memory is allocated through BLK_HTTPINPROCESSMEM
Memcheck logs can be used for troubleshooting to track memory usage over time. These can be automated by setting up Program document to run "-memcheck -perf" over period of time. This can be correlated with activity on server during time frame where spike in direct memory allocation is observed.
Thread stack Space
Generally, each HTTP thread uses about 1 MB of memory per thread. Check NSD stat section for number of HTTP worker threads configured on Domino server: Http.Workers = 40
If the value is higher, review document #1173877 to determine if this should be lowered to default of 40.
Also check field "Optimize HTTP performance based on the following primary activity:" is set to => "Advanced (Custom Settings)" found in Server document – Internet Protocols tab – HTTP tab.
Any other value can potentially increase number of worker threads based on number of CPUs.
java.lang.OutOfMemoryError messages can be reported in Console when JVM memory has been exhausted.
Factors causing high memory usage in JVM could be:
Java heap size
Java heap size for JVM used by HTTP process can be increased through Notes.ini parameter JavaMaxHeapSize or HTTPJVMMaxHeapSize new in Domino 8.5.1. HTTP memory will be impacted in cases when Java heap size has been increased.
NSD log has Notes.ini section to check current settings for HTTP JVM.
Default for 32-bit of Domino:
*Note Domino (32-bit) on iSeries uses 1024M as default for HTTPJVMMaxHeapSize
Default for 64-bit of Domino:
HTTPJVMMaxHeapSizeSet=1 new in Domino 8.5.2 maintains HTTPJVMMaxHeapSize value.
Troubleshooting Java agents, Admins can implement Java agent debug:
Doc #: 1290419 What is the significance of using the notes.ini parameter, debugshowjavaagenttid=1
Memcheck logs can be generated over period of time to track memory allocations and correlate with Java agent debug.
LotusScript memory currently found in a memory.dmp file. The “HTTP” process section has “LotusScript Memory Usage” section
- LotusScript agents: Memory dumps
Memory dumps can be generated over period of time to track memory allocations and correlate with agent debug.