Base portal tuning (Tuning Guide 6.1.x) 

The base portal scenario covers user login, page navigation, and interaction with simple portlets. Users can see a small set of pages, some of which are visible to all authenticated users, with access to others based on their group membership.

We have also benchmarked a number of other scenarios, which focus on different functions or use cases for IBM® WebSphere® Portal. For example, there are scenarios which make use of IBM Lotus® Web Content Management, and a scenario where users have access to thousands of pages. While we have used different tuning to optimize performance for some of those scenarios, the tuning is all based on the tuning done in the Base Portal Scenario.

In all of our measurement environments, we use a separate database server and directory server, in addition to the WebSphere Portal server. We run these servers on separate systems to avoid resource contention on the system running the WebSphere Portal server. This helps improve the maximum capacity achievable.

Contents

     WebSphere Application Server tuning
        JVM Initial and Maximum Heap Size
        JVM Heap Large Page
        JVM Heap New Area Size
        Additional SUN JVM Arguments
        Session Timeout
        Web Container Thread Pool Size
        Security Attribute Propagations
        VMM Context Pooling
        ORB Service Tuning for z/OS
     WebSphere Portal Services tuning
        Navigator Service
        Registry Service
        Cache Manager Service
     Database tuning
        Datasource Tuning for DB2
        DB2 Database Server Tuning
        Oracle Database Server Tuning
        Other Database Considerations
     Directory server tuning
     Web server tuning
     Operating system tuning
        AIX
        Linux
        Windows 2003
        Solaris
        z/OS
     Required Fixes

WebSphere Application Server tuning

There are many aspects to configuring and tuning an application server in IBM WebSphere Application Server. We found that those aspects presented here were critical to a correctly functioning and optimally performing WebSphere Portal in our laboratory environment.

For more details on tuning WebSphere Application Server, refer to the Tuning section of the Information Center at: http://www-01.ibm.com/software/webservers/appserv/was/library/

Accessing the Administration Console

There are two methods to get to WebSphere Application Server Administration Console:

Method 1: Start Server1 and use port 10001

1. Open a command prompt to wp_profile/bin
2. Enter and run the following command:
   • On Linux: ./startServer.sh server1
   • On Windows: \startServer.bat server1
3. Access the Administration Console at: http://yourhost:10001/admin

Method 2: Start WebSphere Portal and use port 10027

1. Open a command prompt to wp_profile/bin
2. Enter and run the following command:
   • On Linux: ./startServer.sh WebSphere_Portal
   • On Windows: \startServer.bat WebSphere_Portal
3. Access the Administration Console at: http://yourhost:10027/ibm/console

Customer ports can differ from the ports 10001 or 10027 mentioned on this page. To find out the ports in use for your installation, look for adminhost in wp_profile/config/cells/cell_name/nodes/node_name/serverind ex.xml.

The following are settings based on our experience with the Base Portal workloads described above:

JVM Initial and Maximum Heap Size

Java™ Virtual Machine heap size: The value of the JVM Heap size is directly related to the amount of physical memory on the system. Never set the JVM heap size larger than the physical memory on the system.

Setting the parameters: In the WebSphere Application Server Administration Console, select Servers > Application Servers > WebSphere Portal > Server Infrastructure: Java and Process Management > Process Definition > Java Virtual Machine

Locate the following fields where you can set the JVM Heap Size:

• Initial Heap Size
• Maximum Heap Size

See JVM Max Heap Size Limits for further discussion.
See instructions on Accessing the Administration Console.

JVM Maximum Heap Size Limits

When setting the heap size for an application server, consider the following:

• Make sure that the system has enough physical memory for all of the processes to fit into physical memory, plus enough for the operating system. When more memory is allocated than the physical memory in the system, paging will occur, and this can result in very poor performance.


• We set the minimum and maximum heap sizes to the same values since we are using the gencon garbage collection policy available in 1.5 IBM JDK which avoids heap fragmentation, this may not be the best choice if you plan to use a different garbage collection. In our measurement runs, the system is under load for a relatively short time (around 3 hours), and it is running with portlets which do not have large memory requirements. When using portlets which have larger memory requirements, or for continuous operation, it may be possible to reduce heap fragmentation by setting the initial heap size to 320 megabytes.


• After doing any tuning of heap sizes, monitor the system to make sure that paging is not occurring. As mentioned above, paging can cause poor performance.


• 32-bit operating systems have an address space limit of 4GBytes, regardless of the amount of physical memory in the system. This space limits the maximum size of each individual process in the system. In addition, some operating systems restrict the size of processes to be even less than this limit. Many versions of Windows limit processes to 2GBytes in size; you can find more information at http://support.microsoft.com/default.aspx?scid=kb;en-us;555223 .


• The address space limit further restricts the size of the JVM process. If the process grows larger than the limit imposed by the operating system, it may terminate unexpectedly.

Due to the demands on native memory by WebSphere Portal V6.1 and its underlying components, we chose a maximum heap size of 1408MB in our Windows environments. There is a balance between JVM heap and native memory, all of which must fit within the 2GB restriction in 32-bit Windows. 1408MB was the largest value we could use to successfully measure all of our Windows configurations and workloads. If your application has additional native memory requirements then you may need to choose a smaller maximum heap size. For more information, see the WebSphere Application Server information center.

On Solaris and zLinux, we use 3.5GB heap size in 64-bit environment.

Parameter	AIX POWER5	Linux	Solaris	Windows 2003	z/Linux	z/OS
Initial and Maximum heap size (Mbytes)	1792	2048	3584	1408	3584	2048

JVM Heap Large Page

Large pages can reduce the CPU overhead needed to keep track of heap. With this setting we have seen 10% throughput improvement in our measurements. This setting does improve performance on Windows, we did not set it for our measurements because Portal doesn’t start reliably when –Xlp is set, sometimes it requires a system reboot to get the JVM to start.

Setting the parameters: In the WebSphere Application Server Administration Console, select Servers > Application Servers > WebSphere Portal > Server Infrastructure: Java and Process Management > Process Definition > Java Virtual Machine > Generic JVM Argument. Add –Xlp.

Large pages are supported by systems running Linux kernels V2.6 or higher. See JVM Large Page Tuning for AIX Operation System.

JVM Large Page Tuning on AIX Operating System

To use JVM Large Page, AIX operating systems must be configured to support large pages.

Setting the parameters:

1. We use the following steps to allocate 4GB of RAM as large pages (16MB) . We chose this amount based on having 8GB of physical memory in these systems. These values may need to be adjusted on systems with different amounts of physical memory.
   vmo -r -o lgpg_regions=256 -o lgpg_size=16777216
   bosboot -ad /dev/ipldevice
   reboot -q
   vmo -p -o v_pinshm=1
   chuser capabilities=CAP_BYPASS_RAC_VMM,CAP_PROPAGATE $USER
2. Add the -Xlp command-line option as described above.
3. In the WebSphere Application Server Administration Console, select Servers > Application Servers > WebSphere Portal > Server Infrastructure: Java and Process Management > Process Definition > Environment Entries > New.
   1. Enter the following in the Name field: EXTSHM.
   2. Enter the following in the Value field: OFF
      NOTE: When the value of EXTSHM is ON it prevents the use of large pages.
   3. Enter an optional description in the Description field.
   4. Select OK.
4. Restart Portal Server. To verify if large pages are being used, run the AIX command vmstat -l 1 5 and check the alp column which is the active large page used. It should be a non-zero value if large pages are being used.

Parameter	AIX POWER5	Linux	Solaris	Windows 2003	z/Linux	z/OS
JVM Heap Large page	-Xlp	-Xlp	Not Applicable	Not Applicable	Not Applicable	Not Applicable

JVM Heap New Area Size

The Generational Garbage Collector introduced in Java 5.0 is efficient to Portal application JVM memory management, and it is set as default by installation with the –Xgcpolicy:gencon command-line option. Use –Xmn to further fine tune the Java heap new area (Nursery).
The –Xgcpolicy:gencon option does not apply to Solaris.

   

Setting the parameters: WebSphere Application Server Administration Console, select Servers > Application Servers > WebSphere Portal > Server Infrastructure: Java and Process Management > Process Definition > Java Virtual Machine.

In the Generic JVM Arguments field, add the following: –Xmn256m.

Parameter	AIX POWER5	Linux	Solaris	Windows 2003	z/Linux	z/OS
New Area Size	-Xmn320m	-Xmn256m	-Xmn768m	-Xmn256m	-Xmn1024m	-Xmn320m

Additional SUN JVM Arguments

On the Solaris platform, we use the following Java HotSpot parameters to achieve optimum performance:

Table 1: Additional Sun JVM Settings

Parameter	Value	Additional Information
-server		Offers higher throughput than the "client" mode.
-XX:MaxPermSize	768m
-XX:+UseConcMarkSweepGC		Use concurrent mark-sweep collection for the tenured generation. The application is paused for short periods during the collection; we found this collector works best in Portal.
-XX:SurvivorRatio	6
-XX:+UseParNewGC		By default concurrent low pause collector uses the default, single threaded young generation copying collector. Set this parameter to use parallel young generation collector for new area.
-XX:ParallelGCThreads	5	Reduces the number of garbage threads. On the Chip multithreading processor based system, we set the threads no higher than one quarter of the hardware threads. We also distribute the threads for 6 JVMs. Our system has 128 virtual processors, we set a total of (128/4)=32 GC threads across all the JVMs. So 5 or 6 GC threads per JVM.
-XX:+PrintGCDetails		Print more details at garbage collection. This does not improve performance, but it provides additional information related to garbage collection activity, which is useful in tuning garbage collection.
-XX:+PrintGCTimeStamps		Print timestamps at garbage collection. See above.

Session Timeout

Session timeout: The default value of Session Timeout is 30 minutes. Reducing this value to a lower number can help reduce memory consumption requirements, allowing a higher user load to be sustained for longer periods of time. Reducing the value too low can interfere with the user experience. For Solaris, on a T5240 hardware, we used a much lower thinktime, 5 seconds, than was used for other platform hardware measurement of 12 seconds. With a lower thinktime, fewer virtual users will result in a heavier load on the system. The reason we lowered the thinktime was specifically to decrease the number of virtual users required for this measurement. Our pool of LoadRunner virtual user licenses was inadequate to generate enough load with the higher think time. With a shorter think time than is used in the other measurements, the duration of each virtual user's interaction with the site is shorter by approximately 2 minutes. To compensate for this, and keep the sessions live on the server for the same period of time, we increased the session timeout by 2 minutes, to 12 minutes.

Setting the parameters: In the WebSphere Application Server Administration Console, select Servers > Application Servers > WebSphere Portal > Container Settings: Web Container Settings > Web Container > Additional Properties: Session Management.

In the Session Timeout field, select Set Timeout.

Parameter	AIX POWER5	Linux	Solaris	Windows 2003	z/Linux	z/OS
Session timeout	10 minutes	10 minutes	12 minutes	10 minutes	10 minutes	10 minutes

Web Container Thread Pool Size

Servlet engine thread pool size: Set this value and monitor the results. Increase this value if all the servlet threads are busy most of the time.

Setting the parameters: In the WebSphere Application Server Administration Console, select Servers > Application Servers > WebSphere Portal > Additional Properties: Thread Pools > Web Container.

In the General Properties section, set the thread pool size in the following fields:
Minimum size threads
Maximum size threads.

Parameter	AIX POWER5	Linux	Solaris	Windows 2003	z/Linux	z/OS
Web container thread pool size	50	50	50	50	50	50

Security Attribute Propagation

To reduce the Security Attribute Propagation (SAP) overhead, use the following custom property: disable Callerlist. If you do not use SAP, then ensure it is disabled to remove the extra overhead, which improves login performance.

If Subject has not been customized, then there is no need to enable Security Attribute Propagation. Security Attribute Propagation can add extra overhead due to some extra processing that is required. However, there are certain configurations where performance might be better with security propagation enabled due to reduction of remote registry calls. See the WebSphere Portal 6.1 InfoCenter (use the following search terms: security attribute propagation) for a discussion of when propagating security attributes is desirable. If you want to enable SAP for functional reasons, you can improve the performance with CallerList tuning described as follows.

Note: The following settings apply to all platforms.

Setting the parameters: In the WebSphere Application Server Administration Console, select Security > Secure Administration, Applications, and Infrastructure > Custom properties.

Table 2: WebSphere Security Attribute Propagation Settings

Name	Value
com.ibm.CSI.disablePropagationCallerList	true
com.ibm.CSI.rmiOutboundPropagationEnabled	false
com.ibm.CSI.rmiInboundPropagationEnabled	false
com.ibm.ws.security.WebInboundPropagationEnabled	false

For com.ibm.CSI.disablePropagationCallerList, create a new property. For the other properties, change their values to false.

Note on WebSphere Application Server V7.0:
In our WebSphere Application Server V7.0 environment, we add com.ibm.CSI.disablePropagationCallerList = true, and use the other 3 default true attributes. For WebSphere Application Server V7.0, this field is accessed through the following: Security > Global Security > CustomProperties > New.

VMM Context Pooling

Tune VMM Context Pooling to improve the performance of concurrent access to an LDAP server.

We changed the following Context Pooling settings line in: wp_profile/config/cells/cell_name/wim/config/wimconfig.xml
<config:contextPool enabled="true" initPoolSize="10" maxPoolSize="0" poolTimeOut="0" poolWaitTime="3000" prefPoolSize="30"/>

You can also set them via the Administration Console as described in the following topic: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.w ebsphere.base.doc/info/aes/ae/uwim_ldapperfsettings.html

Table 3: VMM Context Pool Setting

Context Pool Setting	Default Value	Value
initPoolSize	1	10
prefPoolSize	3	30 Number of open connections to maintain to LDAP server.
maxPoolSize	20	0 A value of 0 allows the pool to grow as large as needed. If access to the LDAP server is shared by many systems, this setting may allow an excessive number of connections to the LDAP server; in such a case, set the maximum pool size to a value appropriate to your environment.

ORB Service Tuning For z/OS

In the WebSphere Application Server Administration Console, set the ORB Service to pass by reference instead of pass by value (default) for both server1 and WebSphere_Portal

Setting the parameters: In the WebSphere Administration Console, do the following:

For WebSphere Application Server:
Select Servers > Application Servers > server1 > Orb Service, then locate and select Pass by Reference

For WebSphere Portal:
Select Servers > Application Servers > WebSphere_Portal > Orb Service, then locate and select Pass by Reference

WebSphere Portal Services

WebSphere Portal has a number of configurable "services"; each service has several parameters available to it. This section describes which services we tuned, the tuning values used, and the rationale for those changes.

Setting the parameters:

1. Edit wp_profile/PortalServer/config/properties/xxxService.properties
2. Uncomment the line, then change the size.
3. Run the following command: wp_profile/ConfigEngine/ConfigEngine.sh update-properties

The changes should appear in the following location in the WebSphere Application Server Administration Console: Resources > Resource Environment > Resource Environment Providers > WP_xxxService > Additional Properties: Custom properties

Navigator Service

The navigator service manages the content model for unauthenticated users, which controls the pages those users are able to see. This content model is periodically reloaded by WebSphere Portal; new pages which are visible to unauthenticated users will not be available until the next reload occurs. Our environment assumes a low rate of change for pages, so we set this reload to only occur once per hour. In a production environment where new pages for unauthenticated users are rarely created, setting this reload time to an hour or more will give better performance. In a test or staging environment where updates to unauthenticated pages need to be seen more often, a lower reload time is more appropriate.

This service also controls the HTTP cache-control headers which will be sent on unauthenticated pages. While our environment did not exploit HTTP page caching, increasing these cache lifetimes in a production environment can reduce load on the portal. For more discussion of the use of HTTP cache-control headers with WebSphere Portal, refer to the Caching section of the Tuning topic in the WebSphere Portal V6.1 InfoCenter.

Table 4: Navigation Service Settings

NavigatorService.properties
Parameter	Default Value	Value Used	Definition
public.expires (seconds)	60	3600	Determines cache expiration time for caches outside of WebSphere Portal and for unauthenticated portal pages only. If the setting remote.cache.expiration is also set to a value greater than or equal to 0, the smaller one of the two values is used.
public.reload (seconds)	60	3600	Determines cache expiration time for the portal internal cache for unauthenticated pages.
remote.cache.expiration(seconds)	60	28800	Determines cache expiration for caches outside of portal server for authenticated as well as for unauthenticated pages.

Registry Service

WebSphere Portal maintains information about many resource types in its databases. Some of these resources are replicated into memory for faster access; this is provided by the registry service. This replicated information will be periodically reloaded from the database, thus picking up any changes which may have been made on a peer node in a clustered environment.

The registry service allows configuring a reload time, in seconds, for each type of data which it is managing. In a production environment, we expect this type of information changes very infrequently, so we used very long reload times for the registry service. A full list of the types of information managed by the registry service is provided in Table 4.

Table 5: Registry Service Settings

RegistryService.properties
Parameter	Default Value	Value Used	Definition
default.interval	1800	28800	Reload frequency for any object types not explicitly specified in the file.
bucket.application.interval	600	28800	Reload frequency for application definitions.
bucket.portlet.interval	600	28800	Reload frequency for portlet definitions.
bucket.theme.interval	3000	28800	Reload frequency for theme definitions.
bucket.skin.interval	3500	28800	Reload frequency for skin definitions.
bucket.client.interval	19000	28800	Reload frequency for client definitions.
bucket.markup.interval	20000	28800	Reload frequency for markup definitions.
bucket.transformation application.interval	600	28800	Reload frequency for transformation application definitions.
bucket.transformation.interval	600	28800	Reload frequency for transformation definitions.

Cache Manager Service

The cache manager service in WebSphere Portal is used to cache a wide variety of types of information in memory. These caches are somewhat similar to the registries maintained by the registry service, as each type of information gets its own cache. The key differences are as follows:

• The information stored in the cache manager service’s caches tends to be more dynamic than the information stored in the registry service’s registries.
• The caches used by the cache manager service are limited in size, and entries will be discarded when the caches become full. The registries used by the registry service are not size-limited; they contain all entries of the specific data type.
• Expiry times are managed individually for each entry in the cache, managed by the cache manager service. In contrast, when the reload time is reached for a registry, the entire contents of that registry are reloaded.

Each cache has several configurable options. A full discussion of these options, along with a list of the caches in WebSphere Portal V6.1, is given in Chapter 2 of this document. Table 5 lists the changes which we made to the cache manager service configuration file. Size values are specified in number of objects and lifetime values are specified in seconds.

Table 6: Cache Manager Service Settings

CacheManagerService.properties
Cache Name	Default Value	Value Used
com.ibm.wps.model.factory.ContentModelCache.live.size	1000	2500
com.ibm.wps.ac.ExplicitEntitlementsCache.USER_GROUP.size	1000	2000
com.ibm.wps.model.factory.NavigationSelectionModelCache.live.size	1000	2500
com.ibm.wps.ac.OwnedResourcesCache.enabled	true	false
com.ibm.wps.ac.ProtectedResourceCache.lifetime	5000	14400
com.ibm.wps.datastore.services.Identification.SerializedOidString Cache.size	2500	5000
com.ibm.wps.puma.DN_OID_Cache.size	500	5000
com.ibm.wps.puma.DN_User_Cache.size	500	3000
com.ibm.wps.puma.DN_Group_Cache.size	500	1500
com.ibm.wps.puma.OID_DN_Cache.size	1500	3000
com.ibm.wps.puma.OID_User_Cache.size	1500	3000
com.ibm.wps.puma.OID_Group_Cache.size	1500	5000
com.ibm.wps.ac.groupmanagement.NestedGroupCache.enabled	true	False
com.ibm.wps.ac.RolesCache.enabled	true	False
com.ibm.wps.ac.ChildResourcesCache.lifetime	7200	28800
com.ibm.wps.policy.services.PolicyCacheManager.lifetime	7780	43200

Database Tuning

Datasource Tuning For DB2

Multiple databases are used to hold information in WebSphere Portal V6.1. We used six separate IBM DB2® databases, each representing a separate database domain and having their own datasources. These databases are as follows:

Table 7: DB2 Database Domains

Database	Database name	Datasource name
Release	release	reldbDS
Community	community	commdbDS
Customization	custom	cusdbDS
Feedback	fdbkdb	fdbkdbDS
Likeminds	Lmdb	lmdbDS
JCR	jcrdb	jcrdbDS

All datasources are configured in a similar manner by logging on to the WebSphere Application Server Administration Console. For the prepared statement cache size, the path is as follows: Resources > JDBC > JDBC Providers > provider_name > Data Sources > datasource_name. The provider name and datasource name are based on the names selected for that database during the database transfer step. Locate the following parameter: Statement cache size.

For the connection pool settings, the path in the WebSphere Application Server Administration Console is as follows: Resources > JDBC > JDBC Providers > provider_name > Data Sources > datasource_name > Connection Pools. The settings are Minimum connections and Maximum connections.

The default settings were used for the prepared statement cache size, and connection pool minimum and maximum sizes.

DB2 Database Server Tuning

WebSphere Portal V6.1 uses database servers for core functionality. In our measurement environment, we used DB2 database server for the Portal application. The LDAP server, IBM Tivoli Directory Server also included a DB2 database as a repository, but it is largely unseen and was operated as an out of box configuration.

We recommend using a remote database server for the largest capacity. For our measurements we used IBM DB2 Enterprise Edition V9.1 fixpack 5 as our database server. WebSphere Portal V6.1 uses the concept of Database domains to designate either groups of tables belonging to one domain, or even entirely separate databases to store the data specific to each domain.

We built six separate databases within one database server to house the tables and data needed to support each domain. For the Base Portal and Many Pages measurements, the Release domain is the primary database being exercised.

The databases and related domains supported by Portal V6.1 are as follows:

1. Release (release domain). This is the primary database domain used by the Base Portal and Many Pages Scenarios.
2. Customization (customization domain). This database receives some light traffic in our scenarios.
3. Community (community domain). This database receives some light traffic in our scenarios.
4. JCR (JCR domain). The JCR database is used heavily in Lotus Web Content Management scenarios. This database receives light traffic in all other scenarios measured in our Benchmark report.
5. Likeminds database, used for Likeminds enabled systems. This database is not used in the scenarios measured in our Benchmark report.
6. Feedback database, used by the feedback subsystem. This database is not used in the scenarios measured in this document.

DB2 on AIX Setup

We configured our DB2 database on AIX using the following settings:

• Set the filesystem which will hold the Portal databases to be a Enhanced Journal File System (JFS2) because a large file system is limited to 64GB.
• Turn on concurrent I/O (CIO) for Enhanced Journal File System as this improves performance.
  To enable CIO, run the following command to mount the database fileset: Mount –o cio /portaldb
• Increase AIX maximum number of processes per user to 4096.
  The default 500 processes per user is too low for database server, we increase it to 4096 in our AIX environment.
  To increase the maximum number of processes per user, run the following command: chdev –l sys0 –a maxuproc=’4096’

While the Portal databases are configured for high capacity performance, various tuning adjustments may be necessary from time to time. Typically these tuning needs are based on the volume of database traffic and the size of table populations.

Our database tuning settings is documented in the Portal Info Center under Creating Remote Database section.

DB2 on Z/os Setup

After transferring the database tables, first identify what tables need to be reorganized.
Perform a re-org check to improve performance.

• Run the EJPSDBTC job after the database transfer. This job contains the DB2 check and RUNSTATS utility for the JCR, Likemind and Feedback database.
• For details on re-org DB2 database, visit the WebSphere Portal Info Center.

Create a Re-org job to re-org all table spaces in the WPSDBJCR database.

Recommended Database Maintenance for DB2 LUW

Two of the database attributes, which DB2 relies upon to perform optimally, are the database catalog statistics and the physical organization of the data in the tables. Catalog statistics should be recomputed periodically during the life of the database, particularly after periods of heavy data modifications (inserts, updates, and deletes) such as a population phase. Due to the heavy contention of computing these statistics, we recommend performing this maintenance during off hours, periods of low demand, or when the portal is off-line. The DB2 runstats command is used to count and record the statistical details about tables, indexes and columns. We have used two techniques in our environment to recompute these statistics. The form we recommend is as follows:
db2 runstats on table tableschema.tablename on all columns with distribution on all columns and sampled detailed indexes all allow write access

These options allow the optimizer to determine optimal access plans for complex SQL.

A simpler, more convenient technique for recomputing catalog statistics is:
db2 reorgchk update statistics on table all

Not only does this command count and record some of the same catalog statistics, it also produces a report that can be reviewed to identify table organization issues. However, we have found instances where this produces insufficient information for the optimizer to select an efficient access plan for complex SQL, particularly for queries of the JCR database.

We have determined a technique that has the same convenience of the reorgchk command and provides the detailed statistics preferred by the optimizer.
db2 -x -r "runstats.db2" "select rtrim(concat('runstats on table ',concat(rtrim(tabSchema),concat('.',concat(rtrim(tabname),' on all columns with distribution on all columns and sampled detailed indexes all allow write access'))))) from syscat.tables where type='T'" db2 -v -f "runstats.db2"

The first command is used to create a file, runstats.db2, which contains all of the runstats commands for all of the tables. The second command uses the db2 command processor to run these commands.

To determine which tables might benefit from reorganization, we use the following command:
db2 reorgchk current statistics on table all > "reorgchk.txt"

For those tables which require reorganization, we use the following command to reorganize the table based upon its primary key:
db2 reorg table tableschema.tablename

Tips:

• Ensure that your database servers have an adequate number of disks. Multiple disks allow for better throughput by the database engine. Throughput is also improved by separating the database logs onto separate physical devices from the database.
• Ensure that the database parameter MaxAppls is greater than the total number of connections for both the datasource and the session manager for each WebSphere Portal application server instance. If MaxAppls is not large enough, you will see exceptions in your connection pools.
• Use System Managed Storage (SMS) for temporary table spaces to benefit complex SQL which require temporary tables to compute their result sets. This saves time in buffer writes and improves disk utilization.

Database performance is very important for obtaining good performance from WebSphere Portal. The maintenance tasks and practices mentioned here were found to be critical to the performance and correct operation of WebSphere Portal in our lab environment. Additional database maintenance and tuning may be needed in your production environments. For further information on DB2 administration and tuning, refer to the DB2 Information Center.

Oracle Database Server Tuning

WebSphere Portal V6.1 uses database servers for core functionality. In this measurement environment, we used Oracle database server for the Portal application. The LDAP server, IBM Tivoli Directory Server included a DB2 database as a repository.

Planning for Oracle Enterprise Edition Database

For our Solaris platform measurements we also used Oracle 10g R2 as our database server. WebSphere Portal V6.1 uses the concept of Database domains to designate either groups of tables belonging to one domain, or even entirely separate databases to store the data specific to each domain.

On Oracle, we built a single database and create Oracle users to own the tables and data needed to support each domain. The domains are listed in PortalDatabaseDomain, above. For the Base Portal measurements, the Release domain is the primary database being exercised.

A well designed database can save a lot of trouble later down the road, and improve database performance. We recommend that you refer to the Oracle Administrator’s Guide to help you make informed database design decisions. The following are the key choices we have implemented in our Oracle database:

• To avoid I/O contention and allow for better throughput, you should ensure your database server have adequate number of disks. Our database is on seven stripped disks.
• For better management and performance of storage structures, Oracle-Managed Files are used for database, as well as redo logs, and control files.
• Database block size: 8k
• The following tablespace sizing was required to support roughly a medium sized Portal, with 100,000 authenticated users, approximately 180 installed portlets and 220 pages, which the load generally consisting of database read operations. We recommend monitoring your tablespace sizing and growth on a regular basis. We used DBCA to create database with the following Tablespace size:
  • SYSAUX: 800MB
  • SYSTEM: 800MB
  • TEMP: 800MB
  • UNDOTBS: 1024MB
  • USERS: 2048MB
• Redo log groups: 500MB each.

Oracle on AIX Setup

We configure our Oracle database on AIX using the following setup:

• Set the filesystem which will hold the Portal databases to be a Enhanced Journal File System (JFS2).
• Turn on concurrent I/O (CIO) for database filesystem as this improves performance. Do not enable CIO for Oracle product filesystem, ie, /u01, as Oracle could fail to start.
  To enable CIO, run the following command to mount the database fileset: Mount –o cio /u02
• Increase AIX maximum number of processes per user to 4096.
  The default 500 processes per user is too low for database server, we increase it to 4096 in our AIX environment. To increase the processes per users, run the following command: chdev –l sys0 –a maxuproc=’4096’
• Enable AIX async I/O, and increase MinServer to 5.
  smitty aio Change/Show Characteristics of Async I/O MinServers = 5
• We also set the following in Oracle user profile as the Oracle Installation Guide for AIX recommends:
  AIXTHREAD_SCOPE=S

Oracle Enterprise Edition Database Parameter Tuning

Database performance is very important for obtaining good performance from WebSphere Portal. Below is a list of tuning applied on our Oracle database server with the alter system command. Additional database tuning maybe needed in your production environments. For further information on Oracle database tuning, refer to Oracle Performance Tuning Guide at: http://www.oracle.com/technology/documentation/database10g.html.

Command used: Alter system set parameter scope=spfile;

Table 8: Oracle Database Tuning

Parameter	Value
sessions	900
sga_target	1813M
pga_aggregate_target	604M
processes	750
open_cursors	1500
db_files	1024

Recommended Oracle Database Maintenance

Optimizer statistics are a collection of data about the database and the objects in the database, these statistics are used by the query optimizer to choose the best execution plan for each SQL statement. Because the objects in a database can be constantly changing, statistics must be regularly updated so that they accurately describe these database objects, particularly after periods of heavy data modifications (inserts, updates, and deletes) such as a population phase. We have used the following command in our environment to recompute these statistics:
execute dbms_stats.gather_database_stats(dbms_stats.auto_sample_size, method_opt=>'FOR ALL INDEXED COLUMNS SIZE AUTO',cascade=>TRUE);

Other Database Considerations

WebSphere Portal maintains some information about users in its database tables, which increase when a user first logs in. We were interested in the steady-state performance of WebSphere Portal, not the performance of a user’s first login to the site. For this reason we evaluated performance after all the users logged in at least once.

One of the most important database tuning factors is bufferpool sizing. It is important to evaluate the database's physical versus logical reads and size the bufferpools to achieve as much as a 95% logical read rate if possible.

Directory Server Tuning

Our measurements used IBM Tivoli Directory Server versions 6.0 as the directory server. These products use a database for storing user information; DB2 Enterprise Server was used for this database in our environment. This database is typically located on the same system as the directory server. If your workload involves creating, updating, or deleting users, then database maintenance described above may be needed on this database.

The following table shows the tuning values used for the directory servers in our Solaris Base Portal Scenario measurements.
You must set these parameters in the following file: /opt/IBM/ldap/V6.0/etc/SchemaV6.0/ibmslapd.conf.

Restart:< /b> You must restart the LDAP server after changing these values.

Table 9: IDS Tuning

Parameter	Value
Ibm-slapdACLCacheSize	250000
Ibm-slapdEntryCacheSize	250000
Ibm-slapdFilterCacheSize	250000
Ibm-slapdFilterCacheBypassLimit	7500

The IBM Tivoli Directory Server uses IBM DB2 as the database server. The database instance and alias are named IDSLDAP. We applied the following tuning to this database:
db2 update db config for idsldap using dbheap 4800
db2 update db config for idsldap using num_ioservers 10
db2 update db config for idsldap using num_iocleaners 5
db2 alter bufferpool LDAPBP size 3690
db2 alter bufferpool IBMDEFAULTBP size 88500

Web Server Tuning

We used IBM HTTP Server 6.1 in our measurement environment. The cluster configuration and the Solaris configuration has a remote Web server, find the tuning in Web Server Tuning in Cluster Tuning section. All other configurations have the Web server running on the same system as the WebSphere Portal application server. If, during your monitoring, you notice insufficient processor capacity on the system when running the Web server and the portal application server on a single system, consider separating the servers onto different systems. We used the following tuning on our Web servers:

Table 10: Web Server Tuning

Parameter	AIX POWER5	Linux	Windows 2003	z/Linux	Additional Information
KeepAliveTimeout	5	5	5	5	This value is less than the think time defined in our scripts to ensure that testing is conservative. Each user is assumed to open a new TCP connection for each page view. However, in a live environment, it can be helpful to increase the KeepAlive Timeout. A higher timeout value can increase contention for HTTP server processes, if you are running out of HTTP processes, decrease this value.
ThreadsPerChild	25	25	2000	25	The higher number of threads per child on Windows is due to a different process model for IHS on Windows.
MaxKeepAliveRequests	0	0	0	0	Selecting 0 lets an unlimited number of requests on a single TCP connection.
MaxRequestsPerChild	0	0	0	0
StartServers	2	2	N/A	2
Access logging	off	off	off	off	This was turned off by commenting out the following configuration line: CustomLog /usr/HTTPServer/logs/access_log common
ThreadLimit	25	25	2000	25
ServerLimit	150	120	N/A	180	Set it MaxClient/ThreadsPerChild.
MinSpareThreads	25	25	N/A	25
MaxSpareThreads	3750	4500	N/A	4500	Set it same as MaxClients.
MaxClients	3750	4500	N/A	4500

We also enabled the server-status module so that we could monitor the number of running and available Web server processes. This enables appropriate tuning of the MaxClients and ThreadsPerChild parameters.
We did additional Web Server tuning in Web 2.0 Scenario. Refer to the Web 2.0 Tuning Chapter of this document for details.
Note: For z/OS, no Web Server was configured.

Operating System Tuning

In any high-load environment, the network must be closely monitored to ensure that its performance is acceptable and consistent. Note that the following is not to suggest that all network parameters are set to these values, but merely make the reader aware that the network is also an entity in the performance environment and bottleneck resolution process.

AIX Network Tuning

To change the settings, run smitty > Performance and Resource Scheduling > Tuning Kernel and Network Parameters > Tuning Network Option Parameters > Change/Show Current Parameters. The changes will take effect immediately and improve the network layer performance in high volume environments.

Remember: Select Save current parameters for Next Boot.

Table 11: AIX Network Settings

Parameter	Value
tcp_sendspace	131072
tcp_recvspace	131072
udp_sendspace	65536
udp_recvspace	655360
somaxconn	10000
tcp_nodelayack	1
rfc1323	1

Linux Network Tuning

For Red Hat Linux and z/Linux (Suse Linux on zOS), add the following settings to the /etc/sysctl.conf file, then run the following command: sysctl -p
To inspect the current TCP parameters, run the following command: sysctl -a | fgrep tcp

Table 12: Linux Network Settings

Parameter	Value
net.ipv4.ip_forward	0
net.ipv4.conf.default.rp_filter	1
net.ipv4.conf.default.accept_source_route	0
net.core.rmem_max	16777216
net.core.wmem_max	16777216
net.ipv4.tcp_rmem	4096 87380 16777216
net.ipv4.tcp_wmem	4096 65536 16777216
net.ipv4.tcp_fin_timeout	30
net.core.netdev_max_backlog	3000
net.core.somaxconn	10000
net.ipv4.tcp_keepalive_intvl	15
net.ipv4.tcp_keepalive_probes	5

Windows 2003 Network Tuning

Using the regedit command, the following registry settings were made in the section HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ Tcpip\Parameters.
Create a new REG_DWORD as specified in the following table:

Table 13: Windows Network Settings

Parameter	Value
MaxFreeTcbs	dword:00011940
MaxHashTableSize	dword:0000ffff
MaxUserPort	dword:0000fffe
TcpTimedWaitDelay	dword:0000001e
TcpWindowSize	dword:0000ffff (65535)

Solaris Network Tuning

For Solaris, use the ndd command to set the following TCP layer parameters. These will take effect immediately, improving the network layer performance in high-volume environments. We use the following settings in Portal server running Solaris 10:

To add these setting, run the following command: ndd -set /dev/tcp PARAMETER VALUE

Table 14: Solaris Network Settings

Parameter	Value
tcp_time_wait_interval	60000
tcp_keepalive_interval	15000
tcp_fin_wait_2_flush_interval	67500
tcp_conn_req_max_q	16384
tcp_conn_req_max_q0	16384
tcp_xmit_hiwat	400000
tcp_recv_hiwat	400000
tcp_cwnd_max	2097152
tcp_ip_abort_interval	60000
tcp_rexmit_interval_initial	4000
tcp_rexmit_interval_max	10000
tcp_rexmit_interval_min	3000
tcp_max_buf	4194304

Kernel Tuning

Our Portal server is running on Solaris 10. In Solaris 10, we use the following projmod commands to set system parameters. After making the changes, we must logout then login to take these changes into effect. To examine your current settings, run the following command: cat /etc/project.
projmod -s -K 'project.max-shm-memory=(privileged,4294967296,deny)' user.root
projmod -s -K 'project.max-shm-ids=(privileged,1024,deny)' user.root
projmod -s -K 'project.max-sem-ids=(privileged,1024,deny)' user.root
projmod -s -K 'process.max-sem-nsems=(privileged,4098,deny)' user.root
projmod -s -K 'process.max-sem-ops=(privileged,16384,deny)' user.root
projmod -s -K 'process.max-file-descriptor=(privileged,16384,deny)' user.root

Solaris Container

Use Solaris Containers to better utilize your modern, powerful T2 server with hundreds of virtual processors. In our lab, we use Processor Sets to partition virtual processors. We create a vertical cluster with six Portal members, then bind each member to a Solaris Processor Set, this configuration gives the optimum performance result.

We used the following commands perform the setup:

1. pooladm –e to enable pool facility
2. pooladm –s to create a static configuration file that matches the current dynamic configuration
3. poolcfg –c ‘create wp_pset1 (unit pset.min=20; unit pset.amx =21)’
Create a processor set, named wp_pset1 or your choice, with between 20 and 21 processors. Create one per processor set.
4. poolcfg –c ‘create pool wp_pool1’
   Create resource pool named wp_pool1 or your choice. Create one per pool.
5. poolcfg –c ‘associate pool wp_pool1(pset wp_pset1)’
   Join the pool and the processor set with an association. Do this for each Processor set.
6. pooladm –c
   Commit the configuration at /etc/pooladm.conf.
7. poolbind –p wp_pool1 PortalPID
   Bind the resource pool to a Portal process.

Refer to the IBM Redbook IBM WebSphere Application Server V6.1 on the Solaris 10 Operating System, sg247584.

Z/OS System Tuning

In the PARMLIB member BPXPRMxx check the values of the following parameters:

Table 15: z/OS System Tuning

Parameter	Value	Additional Information
MAXPROCSYS	15000	System will allow at most 15000 processes to be active concurrently.
MAXPROCUSER	15000	Allow each user (same UID) to have at most 15000 concurrent processes active.
MAXUIDS	200	Allow at most 200 z/OS UNIX users to be active concurrently.
MAXFILEPROC	65535	Allow at 65535 open files per user.
MAXPTYS	800	Allow up to 800 pseudo-terminal sessions
MAXTHREADTASKS	5000	System will allow at most 5000 threads tasks to be active concurrently in a single process.
MAXTHREADS	10000	System will allow at most 10000 threads to be active concurrently in a single process.
MAXMMAPAREA	40960	System will allow at most 40960 pages to be used for memory mapping.
MAXFILESIZE	NOLIMIT	Unlimited file size.
MAXCORESIZE	4194304
MAXASSIZE	2147483647 This scenario illustrates the tuning configuration done on the base portal after the installation is complete. Depending on your desired configuration, additional tuning may be required, but this scenario reflects the essential tuning that should be done on all installation.