1.1 Documentation 

Table of Contents

How many times have you come across a project or new client, where you have to migrate hundreds of databases, application, and servers? You assume that everything is documented and has always been well documented. As you begin to work on the project, you realize that the existing documentation does not contain what you needed or expected. This article describes how to properly document your Lotus Domino environment and what kind of documentation you should have based on the size of your corporation. The article serves as a valuable resource as you begin to document your environment or assist you in evaluating your current documentation.

Before you start, here are some questions to consider regarding your existing documentation:

• Where is the documentation stored in? Is there a defined place for “all” documentation?
• Have you ever inherited an environment from someone else? If so, is the document about this environment up to date as you were told?

Documentation helps the current and the next administrator or technical project manager to quickly understand the environment. Properly and up-to-date documentation helps the continuation and growth of your environment.

Let’s look at what documents should be created and what information should be documented.

Content to document

What makes a good documentation? Just listing the number of servers and host names is certainly not enough in all cases.
A well documented environment helps to identify problems more efficiently, resolve tickets faster, and helps to find bottlenecks.

The following table lists the kind of information that should be part of your documentation and what implementation size benefits most from this type of documentation.

What Content to Document	Small	Medium	Large
Server details	+	+++	+++
Architectural overview	+	++	+++
Naming standards	(+)	++	+++
Instructions for end users	(+)	++	+++

Server Details

The very first and basic documentation is to have the server configuration described. Often, the Domino Directory is misunderstood to be online documentation of the server configuration. However, there is much more to be detailed for example, to allow someone who is new to the environment to understand the building blocks.

The server configuration is essential for upgrade projects because it can uncover pitfalls such as incompatibilities before they would cause a problem.

For each logical Lotus Domino server in your environment, the following (but not limited to) configuration elements should be detailed:

• Server names and IP addresses
• Platform details such as hardware, operating system
• Lotus Domino version with fix packs or hot fixes installed (if any)
• Additional Lotus products with name, version and patch level
• 3^rd party software installed with information on licensing details, install instructions and describing how to access the installation package
  

For this part of the documentation, it is advisable to use the server name as the primary key, e.g. have one document per server where you keep track of the current server configuration details.

The resulting document must be able to be referenced in other parts of the documentation. That's why a Notes application is a good choice for this kind of information. Document links can be used to cross reference information.

Architecture Overview

An architectural overview diagram is a graphical overview of your servers, clusters and their inter-connections. The main focus is set on Lotus Domino servers and how their logical communication paths like mail and replication or SMTP connections.

The main purpose of the architecture overview diagram is to communicate a simple, brief, clear and understandable overview of the environment that the administrators are working with. Most likely, this is a drawing, much larger than one sheet of paper, which is the building of your environment.

This diagram should include:

• Lotus Domino Servers with their name and primary usage type and cluster membership (if applicable)
• Mail routing connections
• Replication paths
  For one way connections like "Pull only", use arrows to point out the direction of the data flow
• Inbound and outbound SMTP connections
• Incoming and outgoing 3^rd party connections like ODBC or other type of connections
• Major non-Domino systems which Domino is communicating with

Some recommendations for creating an architectural overview diagram are:

• Do no try to put too much information into the same drawing. It is important to get the concept rather than all the tiny little details at this point in time. Focus on the Domino level, ignore details such as hardware and operating system, patch level, etc.
• Do not use abbreviations without explaining them in (e.g.) a legend. Make sure when you build this overview, others can understand it easily. Especially for complex environments, it is a key element to a successful documentation because even the person who created this overview might not remember all the details later.
• Work with your application development team to get an understanding of the 3^rd party connections. There is a high chance that administrators do not know what developers have done in the past. It is essential to get a full understanding of these inter-connections to avoid problems when applying changes to the infrastructure.
• Make sure to describe the type of connection, e.g. by using different colors for each connection type. Include a legend within the drawing for more details. An example for this legend is shown below.

Keep this overview up to date by making sure changes in the infrastructure are reflected in the drawing as soon as possible. In best case, the documentation is updated as part of the change implementation.

Tools and Method to Create an Architectural Overview Diagram

To create a drawing, any vector oriented graphic software is acceptable. Even CAD software or simple bitmap oriented graphic software will work. A broad number of software products are available on the market, from commercial software products to freeware applications. The product choice is yours. Just make sure to be familiar with its usage and be comfortable with the licensing details.

Some examples for applications which are free of charge are listed below.

• Dia
  http://projects.gnome.org/dia/
• yED
  http://www.yworks.com/en/products_yed_about.html

After you have selected the software of your choice, create a drawing in the following way:

1. Create an overview of the Lotus Domino Domains and outline how they are connected.
   For this first step, the Domain can be represented by a cloud icon or similar.
2. Review the replication topology within the Domain by looking into the Replication Topology of each Domain. This information can be retrieved from the Lotus Domino Administrator client within the “Replication \ Replication Topology” tab as shown on the picture below.
   
   
   
   Note: The Domino Administrator client will retrieve this information from the Domino server which runs the maps task. This task is not automatically started on a Domino server. For details on how to enable the maps task, refer to the Lotus Domino Administrator Help http://www.ibm.com/developerworks/lotus/documentation/domino/

• Add the administration server of each Domain and walk along the replication path.
  Ensure that in the end of this process, all servers of a Domain were added.

Example

Here is an example of how a simple architectural overview diagram can look like:

Standards

Especially in large organizations, it is important to describe standards that apply to the entire corporation. These standards can apply to every single detail of the environment, sometimes predefined by other people in your company e.g. your operating system standard or similar. Even if there are no regulations, it is advisable to define simple but effective software standards, giving you and your peers the opportunity to work towards a common target.

Hardware Standards

Start with documenting the hardware type and size, and used for what purpose.

From the Domino point of view you start by Defining server classes, this is where you defer the server usability according to registered users or user access, location size, server main task, etc.

For example, in Company A, the architect has defined that:

• Small Servers should not exceed 150 users or host application for small locations (up to 150 users);
• Intermediate Servers can work as administration/application hub and should not exceed 1000 registered users or host application where the concurrent users should not exceed 1000 users;
• Large Servers should not exceed 3000 registered users or host application where the concurrent users should not exceed 3000 users. They may also be used for high performance infrastructure servers (e.g. central SMTP gateways).

In a second step, for each of the server classes above, define:

• Server Parameters: This is where you define based in the Server Class how your servers’ configuration are going to be; type and amount of CPUs; how much memory they should have, etc.
• Hard Disk Layout: In here, you identify how your server’s hard disk is or should be configured, where would the operating system be installed; where the binaries are going to be; and where is your data configured. Also in here, you should determine which type of disk array you are going to be using or used according to each different Server Class.

Be aware that vendors change their server hardware models quite often. New and more powerful servers are being offered while the older ones might not be available anymore. This is why you should consider the hardware standards as a rough guidance for new Domino administrators or people who are not familiar with Domino itself. Keep them updated on a yearly basis and do not hesitate to brainstorm smaller adjustments, e.g. to use a more powerful CPU if it becomes available.

Note: Domino is a very I/O intensive application. When you choose a server model, choose the system with best I/O thruput for best performance!

Software Standards

An important part of the environment documentation is the software standard. You determine what will be the server operating system according to each Server Class. Again, we can see that when you determine the Server Class correctly, it facilitates everything that comes after.

Below is the suggested software standards that help you in your documentation:

• Operating System (OS): You determine what should be the OS version and which service pack needs to be installed according to the product’s requirements. Depending in your Server Class, you can determine a specific OS.
• Anti-virus Software: For anti-virus software, you have to differentiate between anti-virus software for the OS and anti-virus software for Lotus Domino:
• Anti-virus software at the OS level: You document the software version, patch level applied, how the virus pattern files are being updated (how often they should be updated), and also the files to be ignored by the OS anti-virus.
• Anti-virus software at the Lotus Domino level: You document the software version, patch level applied, and how the virus pattern files are being updated (how often they should be updated).
• Lotus Domino server: Document what is the Lotus Domino Server version and Fix Packs applied according to each sub-software requirements. We all know that for every product, there is a recommended Lotus Domino Server version with a specified Fix Pack, which is very important to follow. Also, you document what should be the folder naming convention for your Lotus Domino Server binary folder and data folder
• Backup software: This is a tricky area. In many circumstances in a large company, the Domino administrators do not work with the backup administrators. It is very critical for every environment to have a good backup standard and policies very well defined:
• Domino server backup tool and policies: Document what are covered and how your Domino server backups occur (daily, weekly, monthly) and what type (Incremental, Full, the use or not of Transaction Logging Archiving).
• OS backup tool and policies: Document what are covered and how your OS backup occur (daily, weekly, monthly and what type for each Incremental and/or Full).
• Special backups: If your company follows some sort of rule that request a longer retention period or any special tasks that need to be done with the Lotus Domino Server backup, this is the place where you should document it.
• Environment monitoring: You document how your environment is being monitored (which tool, and what if being monitored at the OS level and at the Lotus Domino Server level).
• Server reporting: You have to differentiate between reporting of operating system specific data and reporting of Domino specific data. This document only covers the Domino related reporting;
• OS settings: Document the mandatory OS settings for all Domino servers. These mandatory settings are necessary to support a stable and secure environment and to minimize the support efforts.
  
  Document what and how your OS should be set (e.g. drive letter, volume name, if Windows update should be automatic or not, network card naming convention, registry keys for OS tuning, etc):
• Drive Indexing: If option is turned off as per Lotus recommendation (for each drive) or if the service is turned off in the control panel (which covers all disk drive).
• System Page file: Document how your system’s page file is set.
• Time settings: To assure a smooth mail routing and replication between all Domino servers, it is important that the servers have been set to the correct time.
• Time zone: The operating system’s time zone has to be set to the correct value according to the physical location of the server. In addition, the setting "Automatically adjust clock to daylight saving changes" should be enabled and Domino servers should be set to "use OS time zone settings".
• Regional settings: Document how your server’s Regional Settings are set, when working on Windows OS environment;

Not all of the information must be described for a small environment. In large and growing environments with multiple administrators and teams working in different time zones, it is clearly a benefit to have common settings lay out.

Security Standards

One of the key components in a documentation is a description of the security standards in your environment.
This information is not only essential in case of a disaster, its also supposed to clearly describe whats allowed and whats not.

• Describe the different roles and the limitations for each role.
  E.g. User Administrator, Database Administratror, Application developer
• Defined server access level, e.g. how the server security tab is supposed to be set up.
  Part of this is already described in the naming standards, so the focus should be on the concept rather than the naming as such.
• Limitations and permissions within the environment defined in form of a corporate policy
  Think about who is allowed to run agents, who is allowed to sign applications, script libraries, etc. Have you got a specific role defined who will perform this work?
• Handling of user id's and passwords, where are they stored, who (which role)
• Handling of access deny groups
• Where backup copies of key system (cert.id, server.id's, administrator id's.) files are located and who got access to them in case of an emergency

Please note that the list above wont be a full list of items to be considered - additoinal elements may apply to your environment, so please ensure to update this chapter of your documentation on a regular basis.

Backup Standards

Documentation should also include a description of your environment is backed up and what users can expect from a restore. This should include:

• Backup policy, descrbing your corporate rules for how long a backup is retained
• "What" explaining what kind of information is backed up how often. Here the focus should be on Domino data and not so much on other elements such as the operating system
• "When", backup jobs are starting and when your backup window ends.
• "Where", describing which servers do perform which role within your backup concept, e.g. if one server is backing up data for all servers, etc.
• "How", describing what backup software is used, where its installed
• Details about the restore process and an estimate of how long it will take to have the restore available

For more details about backup concepts for Domino please refer to the chapter "Backup a Lotus Domino Environment"

Naming Standards

Naming standards allow people who are new to the environment to understand how to maintain consistency while extending or changing elements within. The following list represents configuration details of a Lotus Domino environment where naming standards could be applied:

• Server names
  e.g. Define when to use what kind of server name.
  As the previous example, Company B uses CCNNNNXXX where CC stands for ISO country code, NNNN server Type (MAIL, HUB, APPL, etc), and XXX stands for the server sequential number according to its Type.
• Domain names.
• Cluster names.
• Domino named Networks.
• Access control lists.
• Directory names.
  e.g. Describe the directory structure of a Domino server and where to put which type of applications.
  Note: You should document your company standards for server’s directories (folders) names for system databases, user’s Mail database, mail-in databases, and application databases.
• File names.
  e.g. Define how to get a unique file naming in place.
  Document the naming convention used by your company for file naming convention at the Domino side. You can cover the naming convention for: user’s mail database, mail-in database, common Domino application database, Domino application based in the default templates; etc.
• Group names.
  Document the naming convention used by your company for groups used in the Domino environment. Make sure to define groups following Domino standards: multi-purpose, access control list only, mail only, server only, and deny list only.
  Also, clearly define which characters are allowed and which are not.
• Rooms and resources.
  For information, see http://www.ibm.com/support/docview.wss?rs=899&uid=swg21251010
• User names.
  Describe the naming convention used by your company for users names’ naming convention. Cover all the details that are in use in your environment, such as: First Name, Last Name, Full Name, Short name, Internet Address and any other important field related to user that is in-place at your company.
  e.g. Which characters are allowed or how and when to use middle initials.
• Email addresses.
  e.g. How an email address is computed if users can choose any address of their choice or if there are restrictions within your organization.
• Agent signer names
  If you use specific ID files to sign or run your agents, then describe what syntax to be used for them.

Note: There are limitations defined by IBM Lotus for each of the elements listed. For more detail, refer to IBM Technote 1091216.

Access Control List

In this section of the documentation, include information on the database access control lists (ACLs). They provide a flexible mechanism for the database owner to determine the rights that individuals and groups have with respect to a Notes database. The ACLs for a database are server specific, in that each server enforces the ACL as specified in its version (replica) of the database.

For ACL documentation, include how your company’s default template ACLs are set and how the ACL of the system databases are set.
With this kind of information, you can improve your level of standardization and fine tune your monitored environment.

To document ACL settings, we provide the following suggestions as how this can be done:

• Define basic ACL rules, e.g. Define ACL entries which must be present in all applications or must be present in all mail files.
  This can be all the groups and default IDs that should be in all databases ACL.
  Normally, these are groups that are related to IT Administrator and Administrative Server groups such as Domino Administration Process server, Server Backup, and so on.
• Databases that are under investigation. In most companies, the legal department can request access to any database that may be part of some sort of litigation process. Create and document the standards for this scenario.
• Define how the ACL for Domino System databases should look like in your environment.
  Include every ACL entry with its access rights and roles from the following system databases:
  names.nsf, admin4.nsf, catalog.nsf, log.nsf, certlog.nsf, events4.nsf, ddm.nsf, domlog.nsf, domcfg.nsf, busytime.nsf / clubusy.nsf, report.nsf, statrep.nsf, mail.box / mailX.box (where X stands for the number from 1 to n depending on the amount of mail.box set in the configuration document), statmail.nsf, mtstore.nsf, and da-CC.nsf (where CC stands for the naming convention used for your company’s Country code or Site code).

Format to be used to document naming standards

It does not matter what format you use to document naming standards; however, it is important that this part of the documentation is available to everyone. It must be simple and easy to access and should be versioned.

There are multiple options to achieve this. Some examples are:

• Stored in a Notes application, e.g. A discussion database where all users have READER access.
• Create a file based document (e.g. PDF) and store it in (e.g.) Lotus QuickR or any other existing document management system.
• An internal web page, where users can access the information by using their web browser.

There are other methods. Use your favorite ones. Again, make sure the result is accessible for all your users who need the documentation.

The following recommendations should be kept in mind when detailing naming standards:

• In small implementations of Domino, naming standards most likely do not need to be defined because the number of servers and domains are relatively consistent. Nevertheless, it might be useful to define some naming standard to set the scene for future growth.
• Within larger Domino environments, corporate naming standards are defined or need to be defined to define precise rules and limits for a team of administrators. This is why it should include all the elements we mentioned in their naming standards.
• Especially in large environments, allow users and IT people to improve these naming standards by offering a discussion forum where people can ask why a certain standard is defined in this way. Keep answers accessible for others and be open-minded for changes suggested by your users.

Instructions for End Users

To reduce the number of help desk tickets and service requests, end users need to be provided with a completely different kind of documentation. Most likely, end users do not want to know about the detailed server configuration or their replication paths. Instead, they look for guidance on how to manage standard requests in an efficient way. Descriptions of common processes such as “How to reset request new User ID” or “how to reset a password” is what end users are usually looking for.

The list of processes can be endless. Always aim to reduce the time support staff needed to spend for answering the same question over and over again.

Consider this part of the documentation to be one of the first places for people who are new to your organization. Its purpose is to be used for self assistance platform or a form of guidance for your end users:

• Process descriptions
• “What-if” scenarios
• Background information and cross reference (e.g. to naming standards)
• Help desk phone number(s)

Additionally, keep the following recommendations in mind:

• Include information for 1^st level help desk, e.g. where to route tickets or how to reach self service portals.
• Do not mix end user instructions with training material. If people are new to Lotus Notes, there are better existing resources to refer them to. You do not have to create your own training material.
  For more information please take a look into:
• http://www-01.ibm.com/software/lotus/training/
• http://www-01.ibm.com/software/lotus/training/multimedialibrary.html
• Define the language which is to be used.
  Depending on your corporation and regional distribution, not all users might understand English.
  The language is defined by what users understand best, not by what administrators would like to use.
• Extend this part of the documentation as needed.
  Whenever you experience a growing amount of questions or requests in a certain area, add one more instruction and cross reference them where needed.
• Do not send mass mails to your users communicating how a new process looks like.
  Instead, put the process description into this part of your documentation and refer users to it by sending a link to the respective entry.

Format to be used for documenting end user instructions

Although there are many different ways to provide information, not all method provide the required functionality:

• Simple to access,
  e.g. via web browser or Lotus Notes client
• Distributable to a broader population
• Accessible even when having a problem
  e.g. Stored locally on the users desktop

One advisable method is to set up a new Lotus Domino application based on the Help template. IBM Technote 1164526 describe how to do this.

Keep in mind that process descriptions are likely to change. Especially in larger organizations, they are not easy to categorize because the processes differ e.g. between countries, regions, business units or even between departments.

Conclusion

With a well documented environment, any infrastructure changes and any emergency situation can be faced with more efficiency and more professionalism. It is important to understand that documentation is not a static document. It is a living document with your environment and needs to be updated on a regular basis to keep its value.