ShowTable of Contents
This document will explain what happens during a fixpack installation, where logging occurs, and best practices to keep you out of trouble. A WebSphere Portal upgrade failure is defined as a failure that occurs when you use the WebSphere Portal Update Installer to a cumulative fix or fixpack to your Portal server. This document applies to WebSphere Portal versions 6.0, 6.1 and 7.0.
What happens during a Fixpack Installation
When you install a cumulative fix or fixpack, most, if not all, components are updated. All Portal components are contained in the PortalServer root directory, for example:
The PortalServer directory structure is organized in the following manner: <PortalServer root>/<assemblyName>/<componentName>
In the above example, 'base' is the assemblyName and 'wp.base' is the componentName.
There are five major steps that occur when a WebSphere Portal Fixpack is installed:
1. Fixpack install only. All interim fixes and cumulative fixes currently applied are uninstalled
2. ConfigEngine code is updated
3. wp.ptf.preconfig component is updated. It is the first component updated and all this does is execute a ConfigEngine script against your system to perform various functionality tests. This helps ensure we do not have major failures later.
4. All Portal components are updated one at a time. This work occurs entirely in the PortalServer root directory. That process is:
- A backup is made of the current PortalServer/<assemblyname>/<componentName> contents and stored in the PortalServer/version/backup directory.
- New files for the component are copied into the PortalServer/<assemblyName>/<componentName> directory.
5. wp.ptf.config component is updated. This is the last component updated, and also takes the longest amount of time to complete. The wp.ptf.config component is a large ConfigEngine script that updates the Portal profile with any configuration changes, updates all portlet applications, and updates objects in the database.
The vast majority of Portal fixpack install failures occur during the wp.ptf.preconfig and the wp.ptf.config components. The following sub-sections describe these two components in more detail.
The wp.ptf.preconfig component is a series of ConfigEngine scripts. The most important of these is the PRECONFIG-WP-PTF script (PRECONFIG-WP-PTF-6105 for example). The purpose of this script is to execute a series of checks against your environment to catch any potential problem areas we might see later on. These checks include:
- WAS and JDK versions. Verify the WAS and JDK versions to make sure they meet minimum requirements
- Portal and WAS passwords. Checks explicitly for BLANK values and "ReplaceWithYour" values set for WasPassword and PortalAdminPwd in the following files:
- WP v6.0 => wpconfig.properties
- WP v6.1 => wkplc.properties
- WP v7.0 => wkplc.properties
- Database Connectivity. Test connections are made to each of the database domains. This is NOT based on the datasources and instead is based on the database properties in the following files:
- WP v6.0 => wpconfig_dbdomain.properties
- WP v6.1 => wkplc_comp.properties
- WP v7.0 => wkplc_dbdomain.properties
- Starts and stops WebSphere_Portal server
- Starts and stops Nodeagent (if cluster)
- DMGR Connectivity (if cluster). Tests to make sure we can connect to the DMGR if you are in a cluster. This is based on the WasRemoteHostName and WasSoapPort properties in the following files:
- WP v6.0 => wpconfig.properties (DMgrHostName and DMgrSoapPort here)
- WP v6.1 => wkplc.properties
- WP v7.0 => wkplc.properties
- Synchronizes node (if cluster)
- XMLAccess. Executes a basic XMLAccess script against your Portal server to validate the following:
- Portal server is functional
- PortalAdminId and PortalAdminPwd are correct
- XmlAccessHost and XmlAccessPort are correct
The wp.ptf.config component is the largest part of the fixpack upgrade, and the last component updated in a fixpack. It is another series of ConfigEngine scripts. The most important of these is the CONFIG-WP-PTF script (CONFIG-WP-PTF-7001 for example). This script is slightly different from fixpack to fixpack, but generally the following happens with each:
- Any database schema changes are applied
- All WAS configuration changes, including but not limited to:
- Application redeployment
- New or changed Resource Environment Provider properties
- New JVM system properties, if any, are applied
- Updates to Environment Variables
- Updates to classpath for new/updated classes
- Portlets are redeployed using XMLAccess
Fixpack Logging and Troubleshooting
The Portal Update Installer logs all cumulative fix and fixpack updates to this directory:
In that directory, you will find an individual log for each component that is updated. The log file name is formatted in the following way:
This tells you it is the WP_PTF_7001 fixpack, the log is for the wp.base component, it is an install log, and it was created on February 18th 2011 at 21:45:53.
Each fixpack has a main log that keeps track of each component being updated. This log is called the 'selective-install' log, for example:
If a fixpack installation fails, the error message will tell you to check this log. The purpose of this is so you can see which component failed to install. Once you know which component failed to install, you can check it's corresponding log. The selective-install log will tell you which log file you need to check next:
Failed to perform extraction.
Exception: WUPD0248E: Fix pack update failure: The processing of fix pack WP_PTF_6103, component wp.ptf.preconfig failed. See the log file C:\IBM\WebSphere\PortalServer\version\log\20091113_220533_WP_PTF_6103_wp.ptf.preconfig_install.log for processing details.
Saving History ...
Saving History ... Done
Time Stamp (End) : 2009-11-13T17:09:36-05:00
PTF Component Result : failed
PTF Component Result Message:
WUPD0248E: Fix pack update failure: The processing of fix pack WP_PTF_6103, component wp.ptf.preconfig failed. See the log file C:\IBM\WebSphere\PortalServer\version\log\20091113_220533_WP_PTF_6103_wp.ptf.preconfig_install.log for processing details.
Your next step is to check the log mentioned there for any failures. Most failures occur within the wp.ptf.preconfig and wp.ptf.config components so in the next sections, we will explore best practices for troubleshooting each of those.
Failures with the wp.ptf.preconfig component
The wp.ptf.preconfig component is the first component updated. If you're installing a fixpack and a failure occurs here, all that has happened is that any interim fixes and cumulative fixes have been uninstalled. If you are installing a cumulative fix and a failure occurs here, then no code has been updated at all yet.
Since wp.ptf.preconfig is ConfigEngine scripts, often the best log to start with is the ConfigTrace.log. This log is located in the wp_profile/ConfigEngine/log directory. Either open the log in a text editor and scroll all the way to the bottom, or use the IBM Portal Log Analzyer tool to parse the log.
The most common failures here are:
- WAS version is too low. Check the detailed system requirements to see what minimum WAS level you need for the Portal fixpack you are applying.
- Passwords are not set correctly. Check the wkplc.properties file and ensure that WasPassword and PortalAdminPwd are set and contain no typos.
- XMLAccess script fails. This could be due to an incorrect PortalAdminPwd, but could also mean the Portal server is not functional. Portal's SystemOut.log may be useful here to see if the Portal server is functional or not.
If you are unable to determine from the cause of the failure from the ConfigTrace.log or SystemOut.log, then you can use the following checklist to eliminate any problem areas:
1. Did you upgrade WAS?
2. Is the WebSphere Portal server functional? Start it up and try to render it in a browser.
3. Is WasUserid and WasPassword set correctly in wkplc.properties? Validate the credentials are good by using the exact same values you have in wkplc.properties to log in to the WAS admin console.
4. Is PortalAdminId and PortalAdminPwd set correctly in wkplc.properties? Validate the credentials are good by starting the WebSphere Portal server and logging in to it with the exact same credentials.
5. Cluster only. Is the DMGR up and running? Verify you can access the DMGR in a web browser.
6. Cluster only. Does WasRemoteHostName and WasSoapPort in wkplc.properties match your Deployment Manager values?
7. Cluster only. Are you able to synchronize the node?
8. Are the database properties correct in wkplc_dbdomain.properties (or wkplc_comp.properties if v6.1)? Execute the following ConfigEngine script to verify: validate-database-connection
Failures with the wp.ptf.config component
The wp.ptf.config component is the largest component and takes the longest to complete. If a failure occurs here, then you have a partially upgraded system and this failure needs to be resolved before you do anything else (unless you decide to restore a backup).
Like wp.ptf.preconfig, the wp.ptf.config component is ConfigEngine scripts so again it is usually best to start with the ConfigTrace.log. This log is located in the wp_profile/ConfigEngine/log directory. Either open the log in a text editor and scroll all the way to the bottom, or use the IBM Portal Log Analzyer tool to parse the log.
Since so much happens during this component upgrade, there are many points of failure. The most common ones are:
- XMLAccess script failures, due to:
- WebSphere_Portal startup failure. Check the SystemOut.log for any issues with the wps.ear application during WebSphere_Portal startup.
- Portlet deployment failures. Check the SystemOut.log for any issues.
- Ear deployment failures (wps.ear, wcm.ear, etc). If in a cluster, check the DMGR and NodeAgent SystemOut.log files for any issues.
- Database update failures. This will be logged directly in the ConfigTrace.log. Typically this is due to errors on the database server side, such as database user permissions.
- WCM and/or JCR 'import-nodes' failures. Check the SystemOut.log for errors.
- File permission issues. Make sure the user running the upgrade has proper access to the /tmp directory and the WebSphere file structure. If in a cluster, make sure the user that starts the DMGR server also has proper access to the /tmp directory and WebSphere directory.
Failures with any other component
If any other component fails, then you will need to check it's corresponding log in the PortalServer/version/log directory. Since all these components do is copy new files to the PortalServer root directory, failures here are typically operating system related. For example:
- File permissions prevent the user from writing to /tmp or the PortalServer root directory
- File system is full (/tmp or otherwise)
- ulimit settings are too low
Failures with the Portal Update Installer
If the Portal Update Installer throws any errors before it actually begins installing the cumulative fix or fixpack, then the only logging available is what's logged to the console window. There is no log file for the Portal Update Installer. If you need additonal debugging for failures here, you can add -debug to the command:
and run it again. Common problems here are:
- File permission issues that prevent the user from writing to /tmp
- Issues with the user running 'chmod 770 -R' on the PortalServer root directory (The Portal Update Installer will try to do this)
- Attempting to apply a fix, cumulative fix, or fixpack that is not applicable to the Portal version you have.
Always have a backup
If you have a tight window to apply a cumulative fix or fixpack, it is in your absolute best interest to have a full backup of the environment prior to applying any maintenance. If a cumulative fix or fixpack failure occurs, it may be faster to restore from a backup to get a functional system again. For full details on best practices for backup and restore, refer to this document:
WebSphere Portal Backup and Restore
At a minimum, you need backups of:
- The AppServer root directory
- The PortalServer root directory
- The wp_profile root directory
- Cluster only. The dmgr_profile root directory and it's AppServer root directory
- All Portal database domains
Read and understand all fixpack instructions
It is extremely important that you read and understand all fixpack instructions. Each fixpack may contain different 'extra' steps that you must complete before applying the fixpack. For example, in a clustered environment, you are required to manually copy the wp.base.jar file to the AppServer/lib directory of the Deployment Manager and restart it before proceeding.
Review supported hardware and software
Each Portal fixpack comes with a new list of minimum hardware/software requirements. Review this list to ensure the software you use (such as WebSphere Application Server, databases, LDAP) meet these new requirements. WebSphere Portal Detailed System Requirements
Upgrade WebSphere Application Server
If you do not upgrade the underlying WebSphere Application Server and its JDK to meet the new minimum requirements, the fixpack installation will fail. The wp.ptf.preconfig component explicitly checks for this. This is a commonly overlooked step and can lead to problems from the start.
Set passwords in property files
Make sure all password values are set in the wkplc.properties and wkplc_dbdomain.properties files. The relevant properties are:
This is something else that wp.ptf.preconfig explicitly checks for, and is another commonly overlooked step.
Note: Do not use encrypted passwords. The passwords must be in plaintext. The Portal Update Installer will automatically clear these password values from the property files after fixpack completion.
Note: The Derby database users do not have passwords, but the values cannot be blank. If you are using Derby, then make sure the database password values are any string you'd like, just not blank.
Use the latest Portal Update Installer
With each fixpack release, the WebSphere Portal Update Installer is generally updated and republished. Sometimes this is just minor changes to the Update Installer; other times major changes are put in that the fixpacks depend on. Therefore, it is always a good idea to get the latest Portal Update Installer whenever you are applying a Portal fixpack. WebSphere Portal Update Installer
Use the HealthChecker tool
A HealthChecker tool is provided for users to execute any time an upgrade is planned. The HealthChecker tool is the wp.ptf.preconfig component, separated from the fixpack so you can execute it at any time. It performs the exact same checks that the wp.ptf.preconfig component checks, and allows you to find potential problem areas and correct them before you attempt the fixpack installation.
HealthChecker tool for Portal 7.0
Health Checker for Portal 6.1
Test portlet deployment
If possible, try and test deploying a war file in Portal either using the Manage Web Modules portlet in Portal or with xmlaccess. Because of how invasive this process is, the HealthChecker script does not check this for you but it is a crucial part of the fixpack installation for the wp.ptf.config component.
If deployment fails for any reason, this should be resolved before attempting the fixpack or cumulative fix installation.
Upgrading as a non-root user
Unix only. If you have configured your Portal environment to run as a non-root OS user, then you should execute the upgrade as that same non-root user. Before doing so though, ensure all of the following is true for that user:
- ulimit -n setting is 20000 or higher
- The user owns the AppServer, PortalServer, and wp_profile directories
- The user has r/w/x access to the /tmp directory
Do not remove the contents of PortalServer/version/backup
When the Portal Update Installer installs any fix or fixpack, it makes a backup of the old component code and stores it in the PortalServer/version/backup directory. If you apply numerous fixes or fixpacks, this directory may grow in size and you may be tempted to delete its contents.
Do not delete the contents of this directory. The Update Installer uses this directory to uninstall these fixes and fixpacks. When a fix or fixpack is uninstalled, the corresponding component backup file is restored onto the system. If the backup file is not there, then the uninstall fails. This is especially important for fixpack installations, since the first thing that happens is all fixes are uninstalled.
Other Best Practices
For these and other Best Practices, review this link: Portal Upgrades: Best Practices