ShowTable of Contents
Table of Contents
Lotus products typically run smoothly. To ensure that they continue to run smoothly, administrators should be equipped with the right set of tools for analyzing problems when they occur. This article provides considerations and recommendations on what administrators can do to improve efficiency or optimizing their system.
General recommendations
As a general recommendation, administrators should always be equipped with the latest knowledge in the Domino area, understand how to analyze system performance, and be able to troubleshoot server crashes.
Knowledge
One of the very basic elements is to know about capabilities of an environment, know about bugs or problems, and practical methods to improve system functionality.
Optimizing an environment requires continuous improvement and fresh ideas. However, fresh ideas are not always written down in a book yet. So you should always keep track of the up-to-date information. We recommend few hints to help you or an administrator to do this:
- Sign up for “My Notifications” within the IBM Support portal.
With My Notifications you can receive daily or weekly announcements through e-mail, custom Web pages and RSS feeds. These customizable communications can contain important news, new or updated support content, such as publications, hints and tips, technical notes, product flashes (alerts) and downloads and drivers. The tool allows you to customize and categorize the products you want to monitor and any of the available delivery methods to suit your support needs. http://www-01.ibm.com/software/support/einfo.html
Since the early beginning Lotus Domino administrators have been collaborating with each other to exchange information, best practices, hints, etc. One of the key elements is to share knowledge with the community. This is why the biggest source of information is the community itself. Make use of this valuable information by:
- Signing up for the IBM Developerworks Lotus Community to read about Lotus software products and strategy from those who develop it.
The Lotus Blogs are brought to you from IBMers who focus on Lotus software.
http://www.ibm.com/developerworks/lotus/community/
- Actively participate in the product discussion forums.
Do not hesitate to ask your questions and feel free to answer questions from other people.
ttp://www.ibm.com/developerworks/lotus/community/
- Reading the blog posts of the Domino community.
This following site provides a good start because it consolidates Blog posts from various Lotus community blogs. You may want to use the RSS Feed Reader embedded in your Lotus Notes client in order to stay up to date.
www.planetlotus.org
- Getting in touch with other Domino Administrators in your region by attending community meetings or round tables.
- Setting up your own Blog and post about your experience with Lotus products.Encourage the community to provide feedback by registering your Blog at:
www.planetlotus.org
Performance
Troubleshooting performance related issues is not a simple task. Usually, these kind of problems do not show up over night. Most likely, it is a phenomenon which develops over time and this is also the reason why a performance analysis must take into account that the root cause may be located outside of the Lotus Domino ecosystem.
Different tools and techniques apply to different platforms or areas that you want to focus. IBM already provided a detailed article about performance best practices. To get an idea of which tools you need for your platform, take a look at
Server Crash Analysis
For troubleshooting server crashes, the following tools are valuable to have. Be familiar with their capabilities, functionality and usage.
Actions
To help administrators in maintaining smooth Domino operations, optimizing server stability, performance, and security, we provide recommendations for daily/weekly/monthly tasks that are to be carried out.
Actions outlined in the sections below represent general best practice, but do not include maintenance activities such as compact or fixup tasks which typically run scheduled. Focus is set on what administrators are required to do to keep the Domino environment healthy.
This is not an all-in-one perfect list as individual actions vary based on your environment. If any of the described actions is performed, but does not result as expected, administrators need to investigate further as there may be undiscovered problems. On the other hand, not every action needs escalations or emergency actions. So even if you are not investigating further, it is highly recommended to at least document an abnormality with date and time of when the event occurred followed by comments or clarifications.
Actions can be automated or may even be part of an existing monitoring solution. In this case, the task shall be understood as task to “verification of functionality”.
To get an understanding of efficiency, where valuable time is spent, we recommend documenting how much time administrators spend for which reoccurring action and activity. After a certain period of time, the reported hours should be reviewed to identify where (e.g.) small tasks take up a lot of the administrator’s time. In return, a conclusion would be to evaluate if such a task or action can be automated in some way to optimize your environment.
Daily Actions
The following list represents the daily actions an administrator should carry out for optimizing server runtime, performance and security:
- Check and resolve problems reported in Domino Domain Monitoring database (DDM).
The type and number of issues shown are depending on your DDM configuration.
- Check if there were any servers crashing.
If there is a problem, find the root cause by analyzing NSD files using tools referenced in the general recommendations section earlier.
- Check available free disk space.
This daily check can be automated by creating an event in a properly configured Domino Domain Monitoring database (DDM).
- Verify daily backup jobs ran successfully.
How to do this depends on the backup software used in the environment. All backup software vendors provide log files which administrators can check. Some can report success and failures through mail or other notification methods.
- Check if anti-virus software is running properly and patterns are up to date.
Keep in mind to include the operating system anti-virus software in this check because it is as important as the anti-virus software on Domino servers.
- Check for replication problems by reviewing the replication log of your server(s).
The amount of time spent for this action can be minimized by setting up DDM replication monitors which only reports failures to DDM.
- Monitor mail routing by checking mail routing queues.
- Check key statistic values on your servers and compare them with values from past days.
For the number of peak transactions per minute, a fixed limit can not be provided because the capabilities depend on the underlying hardware. Over time, you will get an idea of when workloads become an issue and you will get to know how to balance workload.
For the daily check, focus on these statistics:
- Server.trans.PerMinute.peak
- Replica.cluster.WorkQueueDepth.avg
- Replica.cluster.WorkQueueDepth.Max
Note: This list does not claim to be complete. Depending on your environment, additional or fewer actions may apply e.g. caused by third party tools which either requires administrators to manage them separately or which help them to perform some of the tasks mentioned above automatically.
Weekly Actions
In addition to the daily actions, administrators should perform the following actions on a weekly basis, preferable in the beginning of a week to review the previous weekend:
- Monitor Administration Requests database (Admin4.nsf),
Check the views “Errors by Date” and “Individual approval required” and take appropriate action.
- Review Domino server statistics and statistic trends. Especially search for workload peaks and document them.
Take actions if you can see an unexplainable peak by reviewing log files further and explore options to balance workload within your environment.
For more details, refer to Notes/Domino Best Practice Statistic and Events
http://www-01.ibm.com/support/docview.wss?uid=swg27009310
- Clean up your server and remove restored NSF files which are no longer required on your server.
Typically, a restore is only required to be kept for a certain period of time, e.g. 7 days after this time, the restored file can be removed from disk.
Once again, additional actions may apply to your environment.
Monthly Actions
In addition to the weekly actions, administrators should perform the following actions on a monthly basis:
- Monitor Domino server memory consumption and take actions to provide sufficient memory for the Domino server to avoid performance problems. The total amount of memory required depends on more than just the Domino server alone. Third party tools on the operating system level that consume additional memory must also be taken into account.
- Check for new releases or patches affecting your environment.
Sign up to mailing lists as described in “General Recommendations” above.
- If any changes are applied to the infrastructure, update the documentation to reflect the current environment.
- Run Lotus Domino Configuration tuner against all servers in the environment and review recommendations made by DCT.
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/domino-configuration-tuner
- Check Certlog.nsf for expiring certificates and ID files,
If required initiate the recertification process as described here http://www-01.ibm.com/support/docview.wss?uid=swg21326765
- Check server disk fragmentation, because fragmented disks may result in slow server performance.
Note: You can run defragmentation tools on a server, but Lotus Domino must be down to avoid damage to your data!
For more details, see IBM Technote #1229817
http://www-01.ibm.com/support/docview.wss?uid=swg21229817
- Review problems (if any) that have been reported in the previous month. Dig into problems and try to find and fix the root cause.
- For Domino Web servers, monitor web server requests.
The following IBM technotes can be helpful when working with Domino HTTP logs:
- How can I gather and sort data from HTTP access?
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/09182009035115PMJALR7A.htm
- 1161104 – How to reduce the size of Domino Web Server Log (domlog.nsf)
http://www-01.ibm.com/support/docview.wss?uid=swg21161104
As mentioned previously, this list does not claim to be complete. Depending on your infrastructure, the focus may vary and additional actions may apply.
Yearly Actions
In addition to the monthly actions, administrators should perform the following actions on quarterly or yearly basis:
- Perform tape backup restoration tests to ensure valid recovery data.
Just checking backup logs and reacting on errors alone is not everything. There is nothing worse than a restore which cannot be done because (e.g.) backup media is broken or it is empty for whatever reason. Once in a while, such as on a quarterly or yearly basis, backups should be verified by restoring to a full server. The restore can be done on an isolated server to avoid effect on your production environment.
- Update your documentation.
We recommend updating documentation whenever a change is applied to the infrastructure or at least with only a small delay. A quarterly or yearly update is recommended to verify essential parts are correct. The larger an environment is, the more frequent administrators should schedule this task.
- Perform a server health check of your environment.
For more information on this topic, refer to the IBM OpenMic Call “Health Check for Lotus Domino servers” in IBM Technote #1432995
http://www-01.ibm.com/support/docview.wss?uid=swg21432995
Conclusion
In this article, we recommended a set of actions you should perform on daily, weekly, monthly, quarterly, and yearly basis. These are not complete lists of actions. Depending on your specific environment, come up with your own lists to perform on regularly basis, to ensure smooth Domino operation and optimized system performance.