The IBM System Verification Test (SVT) objective is to execute a set of test scenarios against a test configuration that contains the key requirements and components that will create a load on Windows 64-bit/SUSE 32-bit/RedHat 64-bit machines.
This testing will use test scripts currently used by the Domino system test team, and also the automation scripts developed by Traveler testing team to simulate Traveler device sync.
One's perception of system quality is governed under the statement of overall system reliability. A widely accepted definition of software reliability is the probability that a computer system performs its destined purpose without failure over a specified time period within a particular execution environment. This execution environment is known formally as the operational profile, which is defined in terms of sets of possible input values together with their probabilities of occurrence. An operational profile is used to drive a portion of the system testing.
A reliability test is one that focuses on the extent to which the feature or system will provide the intended function without failing. The goal of all types of testing is the improvement of the reliability program with specific statements about reliability-specific tests. Reliability is the impact of failures, malfunctions, errors and other defect related problems encountered by customers. Reliability is a measure of the continuous delivery of the correct service (and, the time to failure).
2. Configuration Diagram
This environment consisted of one SUSE 11 32-bit machine with two clustered Domino Mail servers. Each mail server has 2000 Domino users and would replicate from/to each other.
This environment consisted of one Windows 2008 64-bit machine with one Windows 2003 64-bit Domino mail server. The mail server has 2000 Domino users.
The environment consisted of one Redhat 5 64-bit machine with two clustered Domino Mail servers. Each mail server has 2000 Domino users and would replicate from/to each other.
2.1 Evaluation Criteria
Traveler 8.5.2 Reliability was evaluated under the following criteria:
- Server CPU: The overall CPU of the server will be monitored over the course of the experiment. The aim is for the server CPU not to go above 75% over the course of the experiment allowing the server to function appropriately. It is acceptable for the CPU to occasionally spike at this level for a short period of time, but it must return to a lower level. High CPU results from the server being stressed due to processes running such as compact, replication or from user load or Traveler device sync loading.
- Traveler Processes CPU: The previous metric monitors the overall CPU of the server, however, the CPU consumption of Traveler specific processes will also be monitored individually.
- Traveler Server Memory: The server memory metric represents the amount of physical memory available on the server. If the available memory becomes low the server performance could be compromised.
In order to simulate user activity and the Traveler device sync loading:
- Server.Load: The Server.Load is a capacity-planning tool that is used to run tests against a targeted Domino server to measure server capacity and response metrics.
- Domino showstats data: The Domino showstats captures important server metrics. A Testnsf client driver may be used to execute the showstats console command at regular intervals for each server in the configuration and will provide Domino-specific data. The resulting data is logged in a text file and may be graphed for analysis.
- Automation tool: This automation tool was developed by Traveler development team to simulate the real device. The automation script is used to create sync loading and to put stress on Traveler server side.
2.3 Evaluation Process
We use Domino testnsf script to create user load on Domino server. Also, automation tool scripts are used to simulate the real device sync which would put stress on Traveler server. In order to simulate realistic load on the Domino and Traveler server, a total of 7 client drivers running testnsf load/automation scripts was used. Both testnsf and automation tool will be running for 24 hours x 5 days as the long run for Traveler SVT Reliability. Server needs to stay up for 5 days without crashing.
3. Test Drivers
These client drivers are used to create load for both Domino and Traveler server.
4. Conclusion and Summary
The test result demonstrates that the SUSE 11 32-bit machine configured as described in this report (a remote Traveler server with 2 clustered mail servers) and Win 2008 64-bit machine (Traveler server) with one remote mail server of Win 2003 64-bit were able to support concurrent 200 Traveler users out of 2000 Domino users. For the third configuration, Traveler server on RedHat 5 64-bit and 2 clustered mail servers of Win 2003 64-bit and RedHat 5 32-bit, it was able to support concurrent 400 Traveler users out of 2000 Domino users.