An HTTP crash on Domino will be either memory or code related (non-memory).
Memory exhaustion defined by “Insufficient memory” messages
Memory Handles defined by “Out of Handles” messages
Custom Web Application or Agent Code
3rd party applications, i.e. DSAPI, other apps running on top of Domino
HTTP memory exhaustion report messages such as “Insufficient memory”, "out of memory" or "out of handles”. The following information identifies areas in an NSD log for troubleshooting an HTTP memory crash scenario.
HTTP runs out of handles: Console log reports "out of handle" messages.
NSD log has
Overall HTTP memory exhaustion crash
: Console log or the Panic message will report messages related memory such as "Insufficient memory" or "Out of memory".
NSD log from crash will contain memory information for the Domino processes. Below is memory summary for HTTP process from Domino server running on Windows OS.
Note: NSDs from Unix OS, use “Process Heap” size under “Process Heap Memory Stats”
Example below shows HTTP private memory has exceeded 1G:
First, overall HTTP memory is made up of the following areas:
1. Private Static-DPools
3. Thread stack Space
In troubleshooting high HTTP memory, need to identify which of these areas is impacting overall memory usage.
Memory allocations performed by Notes using various DPools (4 MB to 8 MB). These pools are not freed back to the OS after being used.
NSD or Memcheck log look at “Process Heap Memory Stats”
Memory blocks allocated directly from the OS and released back to the OS when freed by the callerFrom NSD log search for “Directly Allocated Memory” under memcheck for HTTP process.
Thread stack Space
Generally, each HTTP thread uses about 1 MB of memory per thread. NSD logs has "Server Statistics" section that will list the stat for number of active threads for HTTP called "Http.Workers"
Notes Data -> Server Data -> Server Statistics (Time 09:56:25)
Http.Workers = 80
Java heap memory by default is 64MB. This can be modified via Notes.ini parameters “JavaMaxHeapSize=” and new to Domino 8.5.1 is addition of parameter “HTTPJVMMaxHeapSize=” which impacts Java heap size used by HTTP.
NSD log has Notes.ini section to view if any of these parameters are being used.
NSD does not list information on total LotusScript memory usage, rather this can be found in a memory.dmp file. In the Memory dump file, search for keyword “HTTP” and review “LotusScript Memory Usage” section for Total Heap Usage size.
Get the NSD
Look at the "Fatal" Stack
Generalize what HTTP was doing at the time of crash based on callstack (i.e. full-text index, running an agent...)
Take the thread from the fatal and match to HTTHR thread log (find last request in log that was incomplete)
Look at the console log and see if any error messages reported, search for the thread number from the bottom up and see if the thread printed out any other messages
Search the SPR and TN for a matching callstack (General rule: <5 match = maybe match 5-10 match=possible match >10 match=likely match)
If you find no match, discuss with customer the calls seen in fatal thread and request running, gather more info on the request i.e. does it kick off an agent, etc.
Use the "Resource Usage Summary" section of the NSD and find the DBs, docs or views that the thread had open at the time of the request, this could point to other DBs that the request interacted with.