Performance management tools for IBM WebSphere Portal

Introduction

This article describes the tools and how they were used to evaluate an IBM® WebSphere® Portal 7.0 performance and/or problem determination issue during a recent engagement at a customer site.

The list of tools addressed here is not complete; specifically, it does not include well known commercial tools used in many environments for load testing, such as CA Wily Introscope, IBM ITCAM, and HP LoadRunner. Instead, the tools we will discuss are a supplement to the customer-supplied tools. The process of performance management is complex and beyond the scope of this document.

Fiddler

Fiddler is a “freeware” tool that captures all the HTTP(s) traffic between your computer browser and a network device. So, with Fiddler installed and any supported browser (most are), you can see exactly what your workstation (client) and the Web device (server) are doing relative to HTTP (Web) traffic.

For example, if you point your browser at a WebSphere Portal 7.0.0.2 default Welcome page, a Fiddler session would display the screen shown in figure 1.

Figure 1. Portal Welcome page in Fiddler session default theme



On the left-hand side of the above screen are each of the requests made to the server from my workstation. On the right-hand side is the timeline for the responses. If we open the Statistics tab for this panel, we see what's shown in figure 2.

Figure 2. Statistics tab data


So the simple process of loading the WebSphere Portal page resulted in 29 requests to the server, totaling 16K, and 873K being returned from the server. Total time to load into a fresh browser was 8.4 seconds. The term “fresh browser” means that this page has never been accessed, so there is no cache used to display content.

If we were to switch the portal over to the new version 7.0.0.2 theme as default, the screen shown in figure 3 would display.

Figure 3. Portal 7.0.0.2 theme as default


And if we do another fresh load of the browser, it would result in the screen in figure 4.

Figure 4. Portal Welcome page as default in Fiddler session 7.0.0.2 default theme


The loading time of the Welcome page is much less, as evidenced by the Statistics (see figure 5).

Figure 5. Statistics data


The number of request to the server has been reduced to 19 from 29, the data sent is now 10K, the data received is now 141K, and the time to load the page has been reduced to 1.85 seconds. The improvement is even more dramatic on a slower link.

With a reload of a page that has already been visited, content is delivered from the browser cache (see figure 6).

Figure 6. Fiddler session of a reloaded (cached) page



This theme is well cached; that is, there are only two requests to the server and the load time of the Welcome page is now 0.36 seconds (see figure 7). For more information, refer to the WebSphere Portal wiki article, “Optimizing Portal 7 Page Builder Theme for Performance.”

Figure 7. Statistics from cached page


Now, looking at the statistics from a fresh load, we see that the site could be improved (see figure 8). The content size is 1.3MB, which on a fast network probably isn’t an issue; however, it still took almost 5 seconds to load.

Figure 8. Statistics from fresh load


If you perform a reload, you'll see the data in figure 9.

Figure 9. Statistics for reload


The amount of data is reduced; however, look at the requests shown in figure 10. There are 74 requests to the server, and this figure shows numerous 304 responses.

Figure 10. Requests example


So, one way to improve end user response would be to change how the CSS content is downloaded; specifically, combining this content---where possible---would reduce the number of requests. This is what is being done on the WebSphere Portal 7.0.0.2 theme.

In addition, a 304 response indicates that the content was retrieved from the server when it was already in the browser cache. One possible cause for this is that the proper header was not configured to set and file expiration date.

To summarize: Fiddler can be used to view the traffic between the workstation browser and the server. In the above case, Fiddler was used to demonstrate the improvement in the new 7.0.0.2 theme, which is made possible by less traffic latency and less traffic downloaded. Fiddler has also shown that the browser properly loaded and used the cache.

Wireshark

Wireshark is another freeware tool that is a network protocol analyzer, letting you capture and interactively browse the traffic running on a computer network. Wireshark captures data directly or, in many cases, it can analyze the data that is captured from the server. In the Linux® world, the data is captured with a simple command:

The only caution is that this capture file can become large. Figure 11 shows a sample file loaded into Wireshark in which Wireshark filters were set to show LDAP traffic flowing to port 389 on the LDAP server.

Figure 11. Sample Wireshark capture file



In the above figure, you can identify the bind request and then follow the TCP process flow as WebSphere Application Server (WAS) and WebSphere Portal make subsequent calls to determine group membership. Figure 12 shows a rearrangement of the columns used in figure 11.

Figure 12. Columns rearranged



As you can see, there are many configuration elements. After working with the product, you can generate filters and follow TCP traffic, for example, follow a log-on request as it works it way through the LDAP. Another example would be to identify the WebSphere Portal Server default HTTP port and verify traffic flow.

To summarize: Wireshark is a freeware tool that allows you to analyze network traffic.

IBM Support Assistant

IBM Support Assistant, (ISA) is a no-charge tool that includes these two components: The server data collection tool, ISA Lite; and the workstation tool, ISA Workbench.

The Workbench is used to provide workstation analysis of data collected using the ISA Lite collector running in the server environment. ISA Lite does the collection portion of the data that ISA workbench uses, does specific trace-file data collection, and sends the results to IBM Support.

The process of collecting the base files for Support is done with another tool supplied with WebSphere Portal, wpcollector, which collects and transfers files to Support automatically.

There are a number of plugins that can be used with the ISA Workbench for specific detail analysis of JavaTM Virtual Machines (JVMs), including garbage collection, thread dumps, and configuration management. Let's now discuss these processes.

Garbage collection

Java or JVM memory is a limited resource in any environment. WebSphere Portal uses the memory to run its processes, defined as shown in figure 13 in the WAS console. Refer to the wiki article, “IBM WebSphere Portal V 7.0 Performance Tuning Guide,” for more details on setting the memory used by the JVM.

Figure 13. Memory settings


After collecting the required files from the JVM file system (not necessarily a WebSphere Portal JVM), the memory used by the JVM can be analyzed to determine whether the above settings are optimum for the environment.

It would seem that, logically, you would take all the memory available; however, the garbage collection memory-management process attempts to keep the JVM memory clear of unused items. If the heap (the term used to specify the JVM memory) is set too high, then the process to actually clean up the memory can take a long time.

Note that, during the time in which the garbage collection process is in operation, WebSphere Portal and other applications will not be accessible.

The ISA Workbench can be used to analyze the Verbose Garbage Collection log (see figure 14). This output is from a lightly loaded WebSphere Portal Server. The collection period was 14 days, the heap exhaustion was zero (good), and the garbage collection overhead was 0.02 % (very good). Clearly, if the heap is being exhausted or the overhead is significant, adjustments would need to be made.

Figure 14. Verbose Garbage Collection log


Figure 15 shows a graph of the JVM memory usage example.

Figure 15. JVM memory usage


The tool also does a good job of recommending changes to the JVM settings (see figure 16).

Figure 16. Analysis and Recommendations report

Thread dump

When analyzing performance, there will come a time when you want to know exactly what is happening with the WebSphere Portal processes.

The processes on which Portal performs work are called threads. When you log on to WebSphere Portal, that process is running on a thread and, as your user base grows, more threads are required. Normally, a thread is used and very quickly returned to the thread pool. The problem comes when Portal--- for any number of reasons---runs out of these threads because they are not being returned to the thread pool.

You can always increase the number of threads available (see figure 17), but this requires more resources, which impacts the JVM memory.

Figure 17. Thread sizing


To understand what the threads in the JVM are actually doing, you can perform what is called a thread dump of the JVM:

1. First, determine what process your JVM is using (see figure 18).

Figure 18. Determine Process ID


2. Then perform a ‘kill -3 4269’. This command provides a “javacore…..” that contains the thread information (see figure 19).

Figure 19. javacore thread information


When contacting Support, it's expected that you obtain a minimum of three thread dumps within a short period of time. The thread usage is normally active, and it is important to compare several dumps to determine if there is a problem.

Using the ISA Workbench, we can see a summary of the threads in our sample dump (see figure 20). This thread dump is from a lightly used or idle system.

Figure 20. Thread dump from low-use server


The thread dump in figure 21 is from an active server.

Figure 21. Thread dump from active server


Note the larger number of total threads (263 vs. 140), and the “Blocked” threads count is a greater percentage of the total (33%). Looking at the thread analysis in figure 22, you can see that some of the WebContainer threads are in a Wait state.

Figure 22. Thread analysis


In fact, there are 27 threads waiting on WebContainer 20 from above to complete (see figure 23).

Figure 23. Waiting Threads


If you look further (see figure 24), Thread 20 is actually at the “org/eclipse/emf/ecore/impl/EPackageRegistryImpl” class.

Figure 24. WebContainer 20 thread details



The ISA Workbench tool does not provide details into the reason this class is an issue, but one possibility is that it is non-serializable, meaning that there is only one path through the process. In this case, there was a problem under load, and remaining threads were held.

IBM Visual Configuration Editor

IBM Visual Configuration Explorer (VCE) enables application developers, system administrators, and technical support personnel to visually explore and analyze configuration information from diverse sources. The VCE can be used to perform the following tasks:

• Compare development, test, and production configurations
• Search one or more configurations for specific properties
• Export selected property sets and the results of searches and comparisons
• Create diagrams that show configuration information in a graphical format, including key relationships between property sets
• Define rules to validate configurations (including cross-product configurations)
• Save, restore, and share the content of configuration analysis sessions

The immediate value is that, typically there are changes made to the environment, and when the time comes to move to the next environment, this tool allows you to confirm visually that all the changes have been moved forward.

For example, in the JVM memory analysis case used above, the Maximum heap size field was set to 1024MB, but during analysis of load tests, you are also able to change WebContainer settings and any number of other items.

Let's compare two example WebSphere Portal configurations, one of which had the Maximum heap size set to 1024, the other to 1536 (see figure 25).

Figure 25. Output when comparing the two configurations



There are many configuration parameters within the WAS file system, and fixes are applied to one environment and not another. VCE is a tool that can help resolve the differences.

To summarize: The IBM Support Assistant tool has a number of functions to support an IBM server environment.

Other tools

Let's discuss several other tools that can be useful.

Cache monitor

There many caches in WebSphere Portal, and a cache monitor application is provided with WAS. A version of this application is also available from IBM Support that has a few more features. The Welcome screen of the cache viewer portlet shows some of the configuration options (see figure 26).

Figure 26. Welcome screen


The application also makes suggestions as to whether a particular cache is set correctly or needs to be changed (see figure 27).

Figure 27. Cache monitor recommendations



WAS also has caches that can be monitored (see figure 28).

Figure 28. WAS Cache Monitor

Netstat

When you work with network applications, the netstat command provides information on what connections are being made on a platform (see figure 29).

Figure 29. Netstat command output


The right-hand column side refers to the state of the connections, as illustrated in figure 30.

Figure 30. Diagram of connection states


Finally, it is always nice when you can get the system to do the work for you. By adding code to the Theme, you can display Portlet and Page Render Timers (see figure 31).

Figure 31. Portlet and Page Render Timers

Conclusion

When working within a complex environment the tests can include a complete system in which there are multiple HTTP and WebSphere Portal servers. As a general rule the process becomes somewhat more manageable if you can configure a single path, such as to a single HTTP and WebSphere Portal server. So, first make the process improvement there, and then move to a larger environment.

Resources

developerWorks® article, “Monitoring performance in a WebSphere Portal environment”:
http://www.ibm.com/developerworks/websphere/library/techarticles/0810_bickmore/0810_bickmore.html

developerWorks WebSphere Portal zone:
http://www.ibm.com/developerworks/websphere/zones/portal/

WebSphere Portal Discussion Forum:
http://www.ibm.com/developerworks/forums/forum.jspa?forumID=168

About the author

Richard Robbins has been a member of the IBM Software Services for Collaboration delivery team responsible for WebSphere Portal infrastructure for six years, working in both sales and development.

Acknowledgement
The author wishes to thank Phil Theiller (IBM), who assisted with the initial editing.