Troubleshooting Tools To Help Determine Why A Server Not Responding Error Message May Occur
There may be times when you encounter messages in your log or on your Server Console that say either “Server not responding”, “Remote server not responding”, or “Remote server no longer responding” when one Server in your Domain is trying to communicate with another Server in your Domain, either replicating or routing mail. Also, users may see these errors in their Notes Client when simply trying to access a database on your Server. There are several Technotes that discuss all different causes and possibilities. The purpose of this document is to try and contain all that information in one place. If you do not find the answer to your problem here, or at least a better understanding of where to begin your investigation, several links to the more commonly used and useful Technotes and documentation are included at the end this page.
First you will want to take a look at some of the basic Domino areas that could be problematic; the Server Document is the best place to start.
While in the Server Document, navigate to the Ports... Tab and, under Notes Network Ports, ensure the Net Address field for the source or destination server has an IP address or Fully Qualified Host name that matches what is in the Fully Qualified Internet Host Name field in the Basics tab:
Next, there are several command prompt based utilities that can be useful in determining where the problem actually lies.
*These can be done from machines inside or outside the firewall
IPConfig /all provides a list of Nic’s and IPaddress’s, DNS used etc.
Netstat –anb provides a list of local address, foreign address and associated task. (NOTE: If using Windows Server 2000, the Command to enter is netstat -an)
Ping –a resolves an IP address to a Host Name.
Ping -f -l 1470 IPAddress - More information available through TechNote # 1086718 (Add link)
NSLookup set type= ( a, mx , ptr ) type of record you are looking up. MX is the most common of records and if one does not exist there must be an A record available.
Tracert domain.com ( This is a traceroute from DOS )
In addition to the command prompt tools we also have a GUI based tool called Nping or NotesConnect.
NPing- Available through Technote # 4004434. NPing/NotesConnect needs to be placed in the Domino program directory (Add link)
A final tool available directly from the Domino console is the simple Trace command.
Trace NotesServerName ( This is a successful trace route to a Domino server ).
02/09/2010 11:10:46 AM Network: Determining path to server VEC
02/09/2010 11:10:46 AM Network: Available Ports: TCPIP
02/09/2010 11:10:46 AM Network: Checking normal priority connection documents only...
02/09/2010 11:10:46 AM Network: Allowing wild card connection documents...
02/09/2010 11:10:46 AM Network: Enabling name service requests and probes...
02/09/2010 11:10:46 AM Network: Address found in local TCPIP names table for VEC
02/09/2010 11:10:46 AM Network: Connecting to VEC over TCPIP
02/09/2010 11:10:46 AM Network: Using address 'vec' for VEC on TCPIP
02/09/2010 11:10:46 AM Network: Requesting IP Address for vec from DNS
02/09/2010 11:10:46 AM Network: DNS returned address 9.00.123.189
02/09/2010 11:10:46 AM Network: DNS returned address 192.000.123.1
02/09/2010 11:10:46 AM Network: DNS returned address 192.000.12.1
02/09/2010 11:10:46 AM Network: Connected to server VEC
When using these troubleshooting tools keep in mind the OSI Layers:
Application – Where Domino resides
Network – Network Interface Cards, Firewalls etc.
The above tools work at the Network Layer. Therefore, if one of them fail, Domino can almost be ruled out at this point.
Other areas that can be checked as well. If drives are on a SAN, you will want to review your Application and System Event Viewer logs, along with the SAN log.
These files may contain errors that have been missed in the Domino Console.log file that show a break in connectivity between Domino and it’s data that is stored in the SAN.
Lastly, there is a free Network Sniffer available called Wireshark (formerly Ethereal) and is available on their Website (http://www.wireshark.org/
). This utility can be placed on the actual User Workstation or Server experiencing connectivity issues, but it is best to be placed on an Aggregating (Full-duplex) Tap. Once a Trace file is captured, we can begin to review if and where the conversation was broken or interrupted. All this information as well as a helpful Tutorial is available on Wireshark's Website.
Steps To Follow To Gather Data For Initial Troubleshooting
In the event that your users are experiencing the error “Server Not Responding”, you may find that the information outlined above in this document has not resolved your issue and need to engage IBM Technical Support for troubleshooting assistance.
Prior to opening a PMR, there are several things that can be done so that once the PMR is opened, troubleshooting can begin immediately. Following these steps will overall result in a faster resolution to the issue as you will not have to open a PMR and then wait for response back with instructions on what data to gather; you will already have much of the needed initial data for the Software Engineer that will be assisting you.
First, if you have worked with Server-side IBM Technical Support before, you may have found that it is often recommended that you add a few notes.ini parameters to your to your Domino Server. The parameters to enable on your Server for initial debugging are outlined below:
Enabling these parameters will provide additional details for IBM Technical Support if the problem occurs again and are good general debug parameters for many Server problems.
Console_log_enabled=1 enables Console Logging and creates the Console.Log file, which is located in the Domino\Data\IBM_TECHNICAL_SUPPORT Folder. This file will log all the information shown in the Console to an organized Text File that IBM Technical Support can review.
Console_log_max_kbytes=102400 is a parameter that is used to set the Maximum Size of the Console.Log File. This value can be set to any value. NOTE:
The value of the parameter is set in kilobytes and is commonly set to 50 – 100 MBs. Here, we have it set to a Maximum Size of 100 MBs. Once the Maximum Size limit is reached, the Console Log simply wraps the log information within the same file; it does not create a new Console Log file until the Server is restarted.
Debug_ThreadID=1 outputs ThreadID information to the Console Log file and, often times, allows you to correlate information between the NSD, Console Log, and Semdebug Files. Once enabled, you will see a Hexadecimal value placed before each line on the Console. See example output below:
[0F30:0002-13B8] 01/27/2010 09:31:40 AM Database Replicator started
[1E84:0002-0D18] 01/27/2010 09:31:42 AM Index update process started
[1A78:0002-1F14] 01/27/2010 09:31:44 AM Agent Manager started
Debug_capture_timeout=1 and Debug_show_timeout=1 are two parameters that are set to enable the capturing of Semaphore information, which is output to the SEMDEBUG.TXT File, which is also located in the Domino\Data\IBM_TECHNICAL_SUPPORT Folder. Output in a Semdebug file can be annotated by IBM Technical Support and will look similar to the following:
ti="006B22ED-8625768C" sq="0008BDAB" THREAD [1434:028E-15C0] WAITING FOR WRITE LOCK ON RWSEM 0x0924 DbServerQueue semaphore (@012830FE) (R=0,W=1,WRITER=1434:13B0,OWNER=1434:13B0) FOR 30000 ms
ti="006B22F2-8625768C" sq="0008BDAC" THREAD [1434:0360-0324] WAITING FOR WRITE LOCK ON RWSEM 0x0924 DbServerQueue semaphore (@012830FE) (R=0,W=1,WRITER=1434:13B0,OWNER=1434:13B0) FOR 30000 ms
Notice the Thread IDs, which can often be correlated with the Console Log and NSD files. For more information on Semaphores and Semaphore Timeouts, see Technote 1094630 - "Semaphores and Semaphore Timeouts" below:
Lotus Software Knowledge Base Document
Title: Semaphores and Semaphore Timeouts
Doc #: 1094630
All of these parameters can be enabled on the Server Console while the Server is running by using the "set config" command (i.e. set config debug_threadid=1). However, the two timeout parameters for Semaphore Debugging will not begin logging until the Domino Server is restarted, but the Console Log and Thread ID parameters are dynamic and do not require a Domino restart.
When you shutdown Domino to make sure the timeout parameters begin logging, it is also recommended that you update the Version of NSD running on the Server if you are running an older Release of Domino or have not Updated your NSD Version recently. The download link to the latest Version of NSD is found in Technote 4013182 and can be found in the direct link below:
Lotus Software Knowledge Base Document
Title: Updated NSD for Domino releases
Doc #: 4013182
Once the latest NSD Version is downloaded, extract the executable from the Zip File. Shutdown your Domino Server and Double-click on the executable and follow the prompts to direct it to the correct Domino Install Path. After the NSD Update completes, Domino may be started again and does not require an OS Restart.
NOTE: If you cannot find your Release of Domino in the list within the Technote, then your NSD does not need to be Updated. Additionally, Updating your NSD Version will not affect your Server Version or Hot Fixes Installed, as the NSD Update only affects the NSD Files (NSD.exe and NSD.sym).
Now that the Server has been configured to gather some of the important debug information, data can be gathered so IBM Technical Support may begin troubleshooting this issue. If your users begin encountering the “Server Not Responding” error, the following tests should be completed prior
to bringing the Server down.
NOTE: Output from your results and responses to the tests and questions below should be captured in a Text Document (Symphony Word Document, Microsoft Word, etc) so that it can be Uploaded to Support for review.
1) Database open to Server IP Address
1a) Using a Notes Client that cannot connect to the Domino Server, go to File --> Database --> Open and type in the IP Address of the "problem" Domino Server. Can you connect using the IP Address?
1b) Using a Notes Client that cannot connect to the Domino Server, go to File --> Database --> Open and type in the Fully Qualified Host Name (FQHN) of the "problem" Domino Server. i.e.: Boston/Mail - Can you connect using the FQHN?
2) Ping the Server with 100 packets from a Client PC
2a) Open a command prompt on a problem user's PC and type in the following command:
ping -n 100 -l 8192 ServerNameorIP
<--For example - ping -n 100 -l 8192 220.127.116.11
This will send 100 packets of data (8192bytes) to the destination Server and report back on how many packets were lost and received. What is the resulting percentage received, lost, etc?
3) Telnet To The Server Over Various Ports
When the Domino Server exhibits this issue, try to connect to that Server via Telnet following the steps outlined below.
3a) Try to connect to the server via Telnet over Port 25 (SMTP) using the IP Address. (i.e.: telnet 8.67.53.09 25) Is this successful?
3b) If that is succeeds, try to connect to the server via Telnet over Port 25 (SMTP) using the Internet Address. (i.e.: telnet MailServer.Boston.Acme.com 25) Is this successful?
3c) Try to connect to the server via Telnet over Port 1352 (Replication) using the IP Address. (i.e.: telnet 8.67.53.09 1352) Is this successful?
3d) If that is succeeds, try to connect to the server via Telnet over Port 1352 (Replication) using the Internet Address. (i.e.: telnet MailServer.Boston.Acme.com 1352) Is this successful?
3e) Try to connect to the server via Telnet over Port 80 (HTTP) using the IP Address. (i.e.: telnet 8.67.53.09 80) Is this successful?
3f) If that is succeeds, try to connect to the server via Telnet over Port 80 (HTTP) using the Internet Address. (i.e.: telnet MailServer.Boston.Acme.com 80) Is this successful?
4) Run a Trace from a Client PC
4a) Using a Notes Client that cannot connect to the Domino Server, go to File --> Preferences --> User Preferences --> Ports --> Trace
4b) From there, type in the Server's IP Address, Hierarchical name, and Fully Qualified name into the "Destination" Field and select Trace. Provide all 3 traces for review. If you cannot connect, select "Copy" and paste the data into a Text file and save it so that you may send this data to IBM Technical Support for review.
5) Use the NPing/NotesConnect Utility
5a) Follow the Steps outlined in Technote 400434 below to run the NotesConnect Utility.
Lotus Software Knowledge Base Document
Title: NotesConnect utility for use when troubleshooting IP connectivity issues
Doc #: 4004434
As the Technote states:
"NotesCONNECT (NPing) is a TCP/IP diagnostics tool designed to verify that a service on a given machine is available. This is accomplished establishing an end-to-end TCP/IP connection with the target host without using the Notes address book or address resolution logic. This tool requires that Notes be installed on the local machine.
This tool is ideal for determining if an TCP/IP connectivity problem is Notes related or an IP infrastructure problem. This connection that is established is a TCP/IP connection, as opposed to ping which uses an IP connection over ICMP. No application or service specific protocol/data is exchanged during the "ping" connection.”
6) General Questions
6a) If Domino Web Access is in use, can users open their Mail Files over the Web?
6b) Does the Server seem to be running slowly or is it running fine, but users just cannot connect?
6c) If you remote into the Server at the time of the issue, can you see the Server Console moving, or is it hung?
6d) Try to issue a "show server" command on the Server; did it respond with output on the Console?
6e) Are you able to enter Commands on the Domino Console at all?
6f) Do you see any error message(s) displayed?
6g) Can you see if you are experiencing any network issues at the time of the problem?
6h) Are any agent, backup, or maintenance utilities running at the time of the issue?
6i) Are there any additional details regarding this issue that you feel are relevant?
7) Commands to Enter and a Manual NSD with -detach switch **VERY IMPORTANT PIECES OF DATA**
At the time of the issue, if the Server Console is responsive, IBM Technical Support will need you to run the following commands on the Server Console (this will provide some additional information - especially if the NSD does not complete successfully):
show tasks and/or show tasks time
Next, multiple Manual NSDs using the '-detach' switch
will need to be run while the issue is occurring. Do not
do an NSD -kill unless absolutely necessary to bring down the Server; always be sure an NSD -detach has completed running first). If you are unfamiliar with how to do this, follow the procedure outlined below (a link to Technote 1204263 - "How to run a manual NSD for Notes/Domino on Windows" as well as specific instructions are included):
Lotus Software Knowledge Base Document
Title: How to run a manual NSD for Notes/Domino on Windows
Doc #: 1204263
All you do is simply use a Command Prompt on the Server, path out to the Domino\Data Directory. From there, type the path back to Domino\nsd -detach. Allow it to complete, and gather the data. See example below:
After following the tests outlined above and running nsd -detach, browse to the IBM_TECHNICAL_SUPPORT
directory inside the Domino Data directory
and collect the following three files:
If you restart Domino before obtaining these files, the Console.log
will have been renamed
with the same naming format as the nsd, except the date and time will be the time the server was last brought up PRIOR
to the problem. For example, if you started your Server on 1/1/2010 at 12:00:30 PM and the Server Crashes on 1/15/2010 at 10:15:45 AM, the files will be named:
· Console_servername _2010_01_01@12_00_30.log
· Semdebug_servername _2010_01_01@12_00_30.txt
In addition to the Domino side files that are collected above, a few OS Level Files are also required. First, capture a WINMSD Report
and save it as a System Information File, .nfo. To do this, go to Start --> Run and type in winmsd. Once this opens, go to File --> Save and save it as WinMSD.NFO (not txt)
Next, collect your Application and System Event Viewer Logs and save them as .evt files. To do this, go to Start --> Control Panel --> Administration Tools --> Event Viewer. Once there, Click on Application
, then Action
--> Save Log File As
... and save it as "App.evt
". Next, once you save the Application, click on System
, then Action
--> Save Log File As
... and save it as "Sys.evt
Overall, the files that will be initially required by Support are seen below:
1) Text Document with your results and responses to the above tests and questions
5) WINMSD Report
Now that this Data has been collected, place the above 7 Files into a Zip File and open a PMR with IBM Support. If using the ESR Tool, then attach the Zip File to the PMR upon opening it. If you open the PMR by calling 1-800-IBM-SERV, then take note of the Exact PMR Number provided
. Once you have your PMR number, go to the following website - http://www.ecurep.ibm.com/app/upload
- and fill in the fields as seen in the screen shot below with your information and click "Continue":
On the next screen, browse to the Zip File containing the 7 Files that you just created and submit it. The Software Engineer that will be troubleshooting your issue will be notified of the files uploaded to your PMR and will begin reviewing this information.
Error: '...The server is not responding...' appears on console every minute http://www-01.ibm.com/support/docview.wss?uid=swg213...
"Remote System No Longer Responding" Dialog Box when Sending a Mail Message http://www-01.ibm.com/support/docview.wss?uid=swg211...