This article will describe areas in an NSD log to gather information on memory related crash on HTTP process.
Memory exhaustion defined by “Insufficient memory” messages
Memory Handles defined by “Out of Handles” messages
NSD log contains memory summary for Domino processes. Start by reviewing the Process Memory Mappings section for HTTP process. For NSDs generated from Unix OS, use “Process Heap” size under “Process Heap Memory Stats” instead.
In the example below, Process Memory Mapping shows that HTTP private memory is over 1G in size:
HTTP overall memory is comprised of the following areas:
1. Private Static-DPools
3. Thread stack Space
Next step is to identify which of these areas is making up majority of HTTP overall memory.
Memory allocations performed by Notes using various DPools (4 MB to 8 MB). These pools are not freed back to the OS after being used.
NSD or Memcheck log look at “Process Heap Memory Stats”
DPools are found within an NSD or Memcheck log under “Process Heap Memory Stats” section:
Check Top 10 section for HTTP in NSD log to identify any memory blocks that are large in size or have high handle count.
DPool debug for HTTP process will generate call stacks in Console log when DPool is allocated. Note, DPool debug can also be used for Shared memory which does not apply in this case.
Example of DPool debug for HTTP on Windows OS uses 'nhttp' as value. In this example, debug will track DPool allocations for memory block 0x044
Memcheck logs can be collected to track memory usage and identify time frame memory spike occurs or correlate high memory usage with activity on server. The memcheck logs can be automated by setting up Program document to run "-memcheck -perf" over period of time.
MBlocks are memory blocks allocated directly from the OS and released back to the OS when freed by the caller.
From NSD log search for “Directly Allocated Memory” under memcheck for HTTP process.
If Directly Allocated Memory is showing high memory usage with blk_local or blk_local_block, add Notes.ini parameter inotesuseNotesMemory=2 as this will help manage direct memory allocation more efficiently. In some cases when using this parameter, Admins may notice high handle count reported. It is then recommended to change setting to 0.
Thread stack Space
Memory is allocated directly from the OS
Memory is allocated through BLK_LOCAL or BLK_LOCAL_BLOCK
Memory is allocated through BLK_HTTPINPROCESSMEM
Generally, each HTTP thread uses about 1 MB of memory per thread. Check NSD stat section for number of HTTP worker threads configured on Domino server:
Using more than 40 worker threads for HTTP may not be necessary and taking up overhead memory that can be used for processing requests.
Doc #: 1173877 Domino HTTP Active Thread recommendations
Setting "Optimize HTTP performance based on the following primary activity:" can also impact number of HTTP worker threads configured. Check that this is set to => "Advanced (Custom Settings)" in Server document – Internet Protocols tab – HTTP tab.
Java heap size by default is 64MB however, this can be modified via Notes.ini parameter JavaMaxHeapSize or HTTPJVMMaxHeapSize in Domino 8.5.1 or later. Overall HTTP memory will be impacted in cases when Java heap size has been increased to high number.
NSD log has Notes.ini section to check if these parameters are being used on Server.
System Data -> Environment -> C:\Lotus\Domino\notes.ini (Time 09:20:58)
LotusScript memory currently found in a memory.dmp file. The “HTTP” process section has “LotusScript Memory Usage” section