EDIT: Work in progress week of 6/20-/624. Expect frequent saves/updates during this time, as well as formatting changes.
Staging to Production is the process by which data from one WebSphere Portal environment is copied to a second WebSphere Portal environment. It is also known as a deployment process. This document will discuss many different questions that arise when devising a deployment process for your Portal environments.
This document is intended for Portal administrators with 1 year or more of experience and who are familiar with both WebSphere Application Server administration and Portal administration. New Portal administrators are welcome to read through this document to become familiar with the terminology and concepts for staging to production. However, some of the the more fundamental concepts of Portal administration will be assumed throughout this document and not explained. I would recommend reading through this document for an overview of Portal Administration if unfamiliar with any of the concepts discussed.
This document is intended to be a living document and will be updated regularly. The format will be a Frequently Asked Questions with multiple sections and questions/answers per section. Questions should not be considered ordered and information in different sections will repeat/overlap. Some questions will be posted without answers - these will be answered in time. The writing style will be a bit more informal / free-form vs. adhering to strict technical writing guidelines, i.e. technical content, the content accuracy and readability will be prioritized.
Community feedback is welcome - suggestions for improvement, questions/answers to add to the document (will attribute work!), corrections to mistakes - all welcome. See end of this document for author contact information.
Index of Topics
Section 1: Staging to Production Overview
Section 2: Portal Architecture Considerations
Section 3: XMLAccess and ReleaseBuilder - longest section
Section 4: Syndication
Section 5: Portal Appilication Archive
Section 6: Misc. Topics
Section 7: Fixing Common Issues
Appendix A: Terminology
Appendix B: References
Appendix Y: Acknowledgements
Appendix Z: About the Author of this Document
Section 1: Staging to Production Overview
Forward note - the discussion in this section is a high-level discussion / not specific to the WebSphere Portal product. i.e. You could substitute the term "WebSphere Portal" with "My cool website" and the same principals would apply. Skip ahead to additional sections if you want to see Portal specific considerations.
Q1) What is staging to production?
A1) Staging to production from a high-level is taking data environment from one environment and copying it to a second environment. In non-Portal terminology, this is also referred to as a deployment process. The terminology used with Portal implies two such environments that are typically involved in a deployment process, a staging environment and a production environment. However, any two environments may be used to copy data.
Q2) What environments are involved in staging to production?
A2) Minimum of two environments, no maximum. Environments are typically referred to by their roles
||A development environment. May be a standalone workstation used by a single developer or a shared server used by multiple developers
|STAGE / INTEGRATION
||Work from multiple DEV enviroments is brought in here and tested. If it does not work, rejected and sent back to developer for additional work.
|QA / UAT
||Quality Assurance / User Acceptance Testing environment. Much more tightly controlled changes and verification testing is performed in this environment after passing through STAGE and before promoting to PROD REND.
||Authoring environment where new web content originates
||Production rendering envirnoment which serves data to end users
||Disaster Recovery environment - should mirror PROD REND 100% identical. If PROD REND suddenly fails, you failover to this environment.
In practice we have seen some environments combined to reduce costs - e.g. STAGE and QA. It is not recommended to do so as it may introduce additional risk for performing deployments. Determine what risk level vs. cost is acceptable when architechting your deployment process.
Q3) How does data flow between environments?
A3) Typically we see two flows established - one for web content, and a second for all other data.
Web Content Flow:
PROD AUTH originates WCM content. Typically a single piece of content is not considered high risk and can be syndicated between environment freely.
PROD AUTH to DEV
PROD AUTH to STAGE
PROD AUTH to QA
PROD AUTH to PROD REND
PROD AUTH to DR
In summary, PROD AUTH contains the master copy and all copies of web content should originate from it. No other environment should originate web content.
DEV originates Pages, Portlets, WCM design elements (authoring+presentation templates), themes and skins changes. In summary - anything which can functionally touch / update multiple locations on the Portal site and present risk should originate in DEV. Why not PROD AUTH for authoring and presentation templates? One mistake and you end up impacting all content authors work / delaying new deliverables. Mitigate that risk by putting higher-risk elements through a more rigorous set of tests in QA before pushing to either production environment.
DEV to STAGE
STAGE to QA
QA to PROD AUTH
QA to PROD REND
Q4) Is there a way to push updates without affecting a live production environment?
A4) Yes. You may have two PROD REND enviroments available, one which is live to end users (PROD-A), the second which is not live and has changes made to it (PROD-B). This is known as an active/passive configuration. During a deployment window the changes are made to the passive environment. Once the changes are made update and validation testing passes - update your DNS records to have the external hostname point to the passive environment (PROD-B) and your passive environment now becomes the active environment. Should something unexpected go wrong on the recently updated environment (PROD-B), you may revert back to the environment which had no changes made to it (PROD-A) via a quick DNS changes.
Active/passive offers low risk means of pushing changes between environments. The primary disadvantage is cost - approximately double the hardware and software costs for a second PROD environment - in addition to overhead to maintain a second environment. Further, the Portal servers are not the only servers which would need to be duplicated in an active/passive consideration. Consider duplication of deployment managers, web servers, load balancers, LDAP servers, database servers, etc. in addition to the Portal servers.
Q5) How should I update my Disaster Recovery enviornment (DR)?
A5) Ideally your PROD REND and DR environments should be identical to each other such that if the PROD REND environment fails, you can failover to a disaster recovery environment within seconds. This is more commonly known as an active/active configuration.
When pushing updated data - do you update DR simultenously to ensure the enviornments remain identical? OR, do you wait to update the DR environment? The answer to this question is also dependent on your acceptable business risk level. From experience we have seen changes pushed to PROD and DR simultaneously multiple times with proper validation testing performed in QA and everything works as expected. We have also seen cases where proper validation performed in QA and the same changes pushed to PROD and DR simultaneously end up breaking both PROD and DR simultaneously, resulting in costly outages.
We would recommend updating PROD first during a maintenance window, allow normal end users to access the site following the maintenance window, then schedule the same updates for DR thereafter. While this will create a slight timing gap in PROD/DR being 100% identical, it also mitigates any risk associated with the updates completely breaking both environments. Arguably, better to have a working environment with some new features/updates missing rather than no working environment available. Less costly to failover to a backup than troubleshoot an extended outage.
Q6) When should I update these various PROD environments?
A6) We'll break this out based on some hypothetical scenarios.
Scenario #1 - active/active
- Peak hours are during normal business hours, 0800-1700, Monday-Friday.
- PROD - Friday night maintenance window
- DR - Friday night maintenance window the following week
*Rationale: ~45 total hours of busy activity on the system following the change. Ensures change does not cause unwanted side effects over a long period of time.
Scenario #2 - active/active
- Global site that is accessed with similar usage patterns 24x7. Schedule as follows:
- PROD - Friday night maintenance window
- DR - Sunday night maintenance window
*Rationale: ~48 hours of busy activity on the system following change. Ensures change does not cause unwanted side effects over a long period of time.
Scenario #3 - active/passive
- PROD-B - Friday night maintenance window
- DR - Friday night maintenance window simultaneously with PROD-B
- PROD-A - no changes made
*Rationale: PROD-A is your fallback in the update fails on both PROD-B and DR.
Note: These are not hard and fast schedules for updating systems but ones we have observed implemented in practice. In most cases, the discussion is based on cost vs. risk assessment with the business. For example - will your business permit changes to a PROD system during business hours? Probably not in scenarios #1 and #2, possibly in scenario #3.
Q7) How frequently should I be updating data?
A7) As often as your business risk will allow it.
- For WCM content updates, these are typically considered smaller / less risky changes, and hourly updates are normal to see.
- For all other updates, schedule the updates weekly, monthly and quarterly, etc. as business risks will allow. Hourly is technically possible but often not acceptable from a risk perspective.
Q8) Can I perform deployments with different code levels?
A8) Yes - so long as the major code level is the same (e.g. 7.0, 8.0, 8.5, etc.) this is supported. We recognize in practice keeping your PROD environments on the same code levels as your lower environments often is not feasible in the same timeframe as a given deployment cycle. i.e. We cannot stop deployments / stop meeting business deliverable while we schedule upgrades of the various environments to complete.
Across Portal code levels - in Portal 6.0-8.0 this was not as much of an issue as new features were not introduced in the middle of a release. In Portal 8.5 with a new philosophy of continuous delivery new features are being introduced as a part of cumulative fixes which may introduce new data into the system that will not promote between environments. Let's take an example scenario:
DEV: Portal 8.5 CF09
PROD: Portal 8.5 CF05
DEV has new work performed on a new feature introduced in CF09 - the Script Application. A ReleaseBuilder DIFF captures these changes, and is eventually moved up to PROD. The DIFF contains a command to update the Script Application. The Script Application does not exist in 8.5 CF05, and therefore the promotion of the data between systems fails. Could we workaround this by ensuring new features introduced are not utilized until all environments are on the same code levels? Yes, however, this requires some change control processes to be established and coordination between teams to ensure this scenario does not occur.
Q9) What should I do for backups?
A9) WebSphere Portal as a product does not offer any backup functionality out of the box. Backups are assumes to be handled by software external to Portal.
Backup the following:
- Deployment Manager if not located on same server as Portal
- Entire Portal filesystem
- Portal databases
On the filesystem backups, a full filesystem backup is recommended. We've seen from experience that updating only /opt/IBM/WebSphere* by itself can lead to issues when restoring from a backup because certain configuration elements NOT stored in this location are missing from the backup.
For PROD systems, a proper backup solution should be used.
For DEV systems, the author of this article has the DMGR, Portal and DB2 installed on the same virtual machine. Typically a snapshot of the virtual machine is "good enough" should something go wrong.
Assess your risk level vs. cost of backups to determine how extensive your backup strategy needs to be.
Section 2 - Portal Architecture Considerations
Q1) Does Portal support multiple installations in a single environment?
A1) Yes. The installer will detect existing running WAS/Portal instances and prompt you to install to a different directory. It will also recommended a different series of port numbers for the additional Portal installation to run on. The author of this article has successfully installed and runs simultaneously the following versions of all in a single environment - Portal 6.0, 6.1, 7.0, 8.0 and 8.5.
Q2) Does Portal support multiple profiles in a single environment?
A2) Yes - you may have a single set of WAS binaries (/AppServer) and Portal binaries (/PortalServer) with multiple Portal profiles. e.g.
Or possibly separated by line of business within a single environment:
This may help reduce software costs. However, one tradeoff is that application binaries are shared between environment. Meaning, if you update either WAS or Portal code levels, you must do so across all profiles simultaneously. We have found from experience this can work in a non-production environment, but not in a production environment. Why? Different lines of businesses have different timelines for their deliverables. Having to halt ALL deliverables for one line of business so a second line of business can perform a deployment is often unacceptable.
Example #1 - line of business #1 needs to add a new feature to allow single sign-on functionality via SAML login. This change needs to be configured at the DMGR cell level, meaning all profiles will need their Portal servers restarted once the change is made. While the other lines of business do not use SAML login - more importantly - they need to ensure the cell-level / global SAML login configuration change does NOT affect their line of business.
Example #2 - one line of business uses purely portlets. A second line of business uses purely web content. One of those lines of business receives an APAR iFix from IBM for a defect in the software. The APAR must be installed across all profiles simultaneously. One line of business is impacted by a change a second line of business needs and will in NO way benefit that line of business.
Q3) What is a virtual portal?
A3) Let's first expand on a different question ... "What is a WebSphere Portal server?". Its a series of .ear files, .war files, and .jar files that run on top of WebSphere Application Server. Several configuration elements of WebSphere Application Server - such as JDBC datasources for databases, global security for LDAP, resource environment providers for global variables independent of the code, etc. are utilized such that it becomes a bit more complex than a single traditional WAS application.
When you access the Portal server primary URL - /wps/portal - WebSphere Application Servers routes this URL to the wps.ear application based on the servlet-filter mappings inside of the web.xml file for the wps.ear application. WebSphere Portal code thereafter takes over and sends you to the "base" portal of the system - i.e. the out of the box site you see following installation of the product. The base portal is a constructions of HTML based on data in the Portal databases and the Portal filesystem.
Virtual portals are discussed in detail in the Portal Infocenter. Virtual Portals allow a logical separation of the Portal site into a different distinct areas, typically one virtual portal per line of business. For example, ibm.com would be a case portal, IBM Commerce would have its own virtual portal, IBM Watson its own virtual portal, IBM Cloud its own virtual portal etc. Each distinct area may have a different look and feel for the given LOB and also be managed differently per LOB. Virtual portals are distinct from virtual portal in that they may be accessed by a unique URL context, e.g. www.ibm.com/wps/portal/watson, OR, by a unique hostname, e.g. watson.ibm.com/wps/portal. Note a combination of unique URL context + hostname is not permitted, e.g. watson.ibm.com/wps/portal/watson could not be created.
Under the hood, virtual portals use the EXACT SAME binaries and database as the Base Portal. The ONLY difference between a base portal and a virtual portal is the manner by which data is loaded from the Portal database - that's it. How do we know whether we should be loading up a base portal or virtual portal from the database when accessing the portal server? Inside of the wps.ear application web.xml file exists a servlet-filter that executes on every URL access to the Portal server. That servlet-filter makes a determination at run-time whether or not the URL is accessing a base portal or a virtual portal and loads up database data accordingly thereafter. At present time, there is no means to separate out a virtual portal from a base portal. If you truly need them separated out so one LOB is not affected by another LOB - you may need a different architecture than virtual portals. See question #4 for those details.
Q4) Should I have multiple installations, multiple profiles, multiple clusters, or multiple virtual portals? Or some combination of all of them?
A4) Loaded question - but a common one asked. There is no right/wrong answer one on this one - so the most common response starts with "it depends". We'll list the primary pro/con of each.
- Pro: Least risky. Separation of lines of business ensure changes to one LOB do not affect a second LOB.
- Con: Most costly - hardware, software and overhead all go up significantly.
- Pro: Single common environment / configuration for multiple lines of business. e.g. Only need to configure LDAP once.
- Con: Changes to WAS or Portal code must be applied to all profiles simultaneously, unlike multiple installations.
Multiple clusters in single profile:
- Pro: Simpler to maintain than multiple profiles configuration - only a single profile to update.
- Con: Cell level changes - such as security changes - must be applied to all LOB's simultaneously. Changes to WAS or Portal code must be applied to all clusters simultaneously,
Multiple virtual portals:
- Pro: Unique to WebSphere Portal product. Simplest of configurations to maintain+update. Can separate LOB per virtual portal.
- Con: Cell+Cluster level changes - such as security+database changes - must be applied to all LOB's simultaneously. Changes to WAS or Portal code must be applied to all virtual portals simultaneously,
What does IBM typically use internally for its environments? Multiple virtual portals. Each LOB has their own virtual portal and can work on their own deliverables independent of the other LOB. Coordination between LOB is needed for some elements that are not separated out by virtual portal - for example WAS/Portal code levels.
Q5) What about Portal farms?
A5) In this author's experience, 99% of Portal environments will want to choose clusters for their architecture. Portal farms are not bad by any means - however, they are intended to fulfill a specific need as documented in the Portal Infocenter. However, they also have limitations associated with them that clusters do not have. One notable limitation - managed pages in a farm is not supported, and generally speaking, customers utilizing IBM Web Content Management (WCM) would not want to choose a farm architecture. A one-hour presentation given by two Portal architects goes into extreme detail on pros/cons of Portal Farms - the 11-15 minute marks discuss when to choose a farm vs. not choose a farm to answer this question.
Q6) What about shared Portal databases?
A6) WebSphere Portal supports sharing of the customization, community, feedback and likeminds databases. The release and jcr databases may not be shared between environments. Let's take a few example scenarios:
Scenario #1: DEV + STAGE: Separate release and jcr databases. Shared customization, community, feedback and likeminds databases.
- Create a new virtual portal in DEV. Not yet promoted to STAGE.
- Do some testing in DEV, including creation of customization data. Decide the virtual portal in DEV is no longer needed, decide to delete the virtual portal.
- Data in the customization database is not deleted, but instead orphaned. The Portal administrator forgot to specificy -DremoveResourcesInSharedDomains=true when deleting the virtual portal with shared domains.
* No functional issues with this scenario, but illustrates additional administrative overhead / deviation from common procedures involved in certain scenarios.
Scenario #2: STAGE + QA: Separate release and jcr databases. Shared customization, community, feedback and likeminds databases.
- Make a change to the STAGE release database. Not yet promoted to QA. Good here, release database is separated.
- Do some testing in STAGE, including a change to customization data. Now the data is in STAGE+QA shared customization DB, yet, QA can't utilize the customization data until the release data is promoted from STAGE to QA.
* No functional issues with this scenario, but the value-add of the shared databases is limited.
Scenario #3: PROD + DR: Separate release and jcr databases. Shared customization, community, feedback and likeminds databases.
- Make a change to the PROD + DR release databases close enough in time so they are 100% identical for failover purposes.
- Network in datacenter PROD is located in catastrophically fails - have to failover to DR datacenter.
- The customization, community, feedback, and likeminds databases are in the same datacenter as PROD. DR has no access to these databases given PROD datacenter is completely down.
* Functional issues with this scenario given the databases were shared.
Now the good news - for purposes of promoting changes between Portal environments, the four databases noted typically are NOT included in the promotion of changes between environments. Only release and jcr database data is typically promoted between environments. Therefore shared databases are typically not impacted by deployment scenarios.
So in this author's opinion - where does database sharing make sense? Typically niche/rarer configurations - such as multiple cluster or farm configurations - would make sense for sharing databases.
Q7) I've seen some terminology discussing "scoping" of resources? What is this?
A7) This goes back to our discussion on the base portal and virtual portals. The base portal and virtual portals use the same files on the filesystem, but different data in the database. There are some parts of the virtual portals which are completely separated from the base portal - most notably pages. There are other parts of the virtual portal which are shared with the base Portal that cannot be separated - most notably portlets. The "scoping" of resources refers to whether or not the artifact in reference can be isolated to a given virtual portal, or, if it must be shared in some means with the base portal.
The big three which are unscoped / must be shared - themes, skins, portlets
The big three which are scoped / are not shared - pages, web content libraries, search collections
A full listing is in the Portal Infocenter.
Q8) How does scoping affect my deployment process?
A8) Changes that are unscoped may affect a large number of teams. For example - let's suppose we redeploy the WCM Rendering portlet as a part of a deployment. Given the change is an unscoped change, it will impact the base portal and ALL virtual portals. Now, let's suppose we redeploy a custom portlet ... same idea, the change will impact the entire site because it is unscoped. If a custom portlet is only actively in use by the base portal or a single virtual portal - then the change probably won't impact other LOBs. If however the portlet is commonly used across locations in the Portal server (such as the WCM rendering portlet), then there is a higher risk associated with updating that particular portlet.
Changes with are scoped will only impact the base portal and/or virtual portal which they exist in. They will not impact other LOB's if updated . For example, if I create a page named "test page" in my "test virtual portal" - no other virtual portal will see the page, nor will the base portal see the page.
Special note: Web content libraries are scoped in Portal v8.0 and v8.5 only if managed pages is enabled - they are unscoped if managed pages is disabled. In Portal v7 and earlier, the base Portal and all virtual portals had access to all WCM libraries. Now, in v8.0 and v8.5 with managed pages enabled / scoping enforced, the base portal and each virtual portal MUST have its own copy of the WCM libraries it requires and they WCM libraries will function independent of each other. If WCM libraries need to be shared between two more more locations on the site (e.g. base portal + virtual portal, 2+ virtual portals, etc.) - this can be performed via a series of two-way syndication relationships. Chose one location as the "master" copy (preferably the base portal), and other locations to receive the updates once made to the master location. This is not quite a typical syndicator/subsriber pairing as we think of between systems, this is within the SAME system we are setting up these relationships. Once the syndication pairing are setup, all WCM libraries scoped to each location can be kept up to date with each other.
Q9) I want to use a Deployment Manager for administrative purposes. How does this work with Portal?
A9) WebSphere Portal installs to an unmanaged node out of the box. If you wish to utilize a managed node topology / federate to a deployment manager, we recommend following the steps outlined in the Portal Cluster guides per version, v7.0 link, v8.0 link, v8.5 link. Note - if you federate to a Deployment Manager, WebSphere Portal has an additional requirement that you created a cluster within that Deployment Manager for Portal. Even with a single-server in a DMGR, you must have a Portal cluster. Federated to a DMGR but not clustered is NOT supported by WebSphere Portal.
Why is it not supported? The issue is one with how Portal stores its global variables in a Resource Environment Providers within WebSphere. In an unmanaged node configuration all variables are scoped to the server level. When you federate to a DMGR they remain scoped to the server level. Now let's add a second Portal server to the DMGR ... the variables are still scoped to the server level. Therefore, the "global" variables are no longer global, e.g. a change in a variable on Portal server #1 at the server scope will not update the variables for Portal server #2 at the server scope. Portal's architecture requires a cluster scope to be created to ensure global variable updates can be made easily for all servers which require access to this information.
On a practical note - the Portal server in a single-server configuration federated to a DMGR but not clustered (unsupported configuration!) does work. Meaning you can start the Portal server and perform most functions correctly. However, many administrative operations will fail if we attempt them - in particular ConfigEngine configuration tasks will act erractically in a federated but not clustered scenario. This finding unfortunately flares up most frequently when performing a cumulative fix upgrade. The configuration must be unmanaged node / standalone system, OR, fully clustered for the cumulative fix upgrade to complete - this is no means to ensure a successful CF upgrade otherwise.
Q10) What is a database transfer? Why do I need to perform one?
A10) Portal installs to an out of the box database called Derby. Derby is lightweight database intended as a starting point only for proof of concept work / sandbox environments. An enterprise database - DB2, Oracle or SQL Server - is recommended for any significant development work. An enterprise database is required for PROD environments. A full listing of which versions of the enterprise databases Portal supports is available here (click the supported software tab). The database transfer procedure takes the content of the Derby database and copies the data via series of SQL commands to the enterprise database.
Special note #1: Derby does not scale well with large amounts of Web Content Management data. If you plan on performing a large amount of WCM authoring work, we recommend transferring an enterprise database in that environment before beginning any significant work.
Special note #2: You must perform a database transfer before creating a Portal cluster. Derby may functionalliy work in a single-server cluster scenario, but it is not a supported configuration. In a two-server cluster configuration, Derby absolutely will not work. Why? Derby is included on each installation of the Portal server ... so Portal server #1 has its own Derby database, Portal server #2 has its own Portal database. Both JVMs need to access the same database in order to keep their data synchronized. However, Derby does not support multiple JVMs accessing the Derby database.
Thus, we recommend performing the database transfer early in the system configuration process to avoid potential headaches.
Q11) I messed up creating my Portal site. I want to go back to the copy of the data in the Derby database which works. How can I do so?
A11) If the Derby database contains data on the SAME code levels you are presently on with your enterprise database, you may rerun database-transfer, which will drop and recreate all tables in the enterprise database and recopy the data from the Derby database. HOWEVER, if code levels are not the same, you may NOT perform a database-transfer as they will create unpredictable results with the database data being at a code level different than the .war/.ear files on the filesystem. Let's use a few example scenarios to illustrate this concept:
- Portal 8.5 GA is installed
- Upgrade to CF10. ==> Derby database contains CF10 data.
- Run database-transfer to DB2. ==> DB2 database contains CF10 data.
- Portal site gets messed up, want to restore.
Section 3 - ReleaseBuilder overview
Q1) What is ReleaseBuilder?
A1) ReleaseBuilder is an executable .bat|.sh file located in wp_profile/PortalServer/bin. It is a critical tool used as a part of the overall staging to production process. In some documentation, you may see references to a ReleaseBuilder process. This is a synonymous terminology for staging to production / deployment process with WebSphere Portal.
Q2) What does ReleaseBuilder do?
A2) From a high level - the releaseBuilder tool allows you to compare the data on a system from two different points in time, capture the changes made on the system, and create an output file noting the changes made. The output file, commonly called a DIFF file, can thereafter be imported to another environment and have the EXACT same changes made on that second environment. This ensures the exact changes made on one environment can be replicated to a second environment. i.e. We need only copy over the delta of changes made, not a full copy of data. ReleaseBuilder is a standalone java process that does not require a running WAS or Portal server to execute.
Q3) How does ReleaseBuilder actually work?
A3) Example Scenario:
January 1st, 2016 Take an ExportRelease.xml XMLAccess export from STAGE. Name it 2016.01.01_stage_exportrelease.xml
xmlaccess.sh -user wpsadmin -password wpsadmin -url http://localhost:10039/wps/config -in /opt/IBM/WebSphere/PortalServer/doc/xml-samples/ExportRelease.xml -out /tmp/2016.01.01_stage_exportrelease.xml
February 1st, 2016 Take an ExportRelease.xml XMLAcess export from STAGE. Name it 2016.02.01_stage_exportrelease.xml
xmlaccess.sh -user wpsadmin -password wpsadmin -url http://localhost:10039/wps/config -in /opt/IBM/WebSphere/PortalServer/doc/xml-samples/ExportRelease.xml -out /tmp/2016.02.01_stage_exportrelease.xml
Now run releaseBuilder against the two files
releasebuilder.sh -inOld /tmp/2016.01.01_stage_exportrelease.xml -inNew /tmp/2016.02.01_stage_exportrelease.xml -out /tmp/2016.01.01_2015.02.01_STAGE_DIFF.xml
The 2016.01.01_2016.02.01_STAGE_DIFF.xml file contains a capture of all changes made to the STAGE enviornment release database over a one month period of time. It may now be imported to another enviornment, such as a QA enviornment, and the QA environment will automatically have the 1 month of changes made in the STAGE enviornment applied to it. STAGE and QA release databases will be in sync / identical thereafter.
Q4) Can ReleaseBuilder be used with two different environments?
A4) No this is not supported. Let me repeat - THIS IS NOT SUPPORTED!!! (Author's note - I would use blink tags here if I could to emphasize this point). If you create a DIFF file with the intention of importing it another environment, the DIFF file _MUST_ be created from XMLAccess exports from the SAME environment. If you create a DIFF file using releasebuilder exports two different environments and import that DIFF file, this is NOT supported, and could create unpredictable results. We have seen from field experiences loss of 25%+ or more of Portal pages if ReleaseBuilder is used in this manner. ALWAYS use Releasebuilder with two exports from the SAME environment.
Q5) Can ReleaseBuilder be used with two different environments if I don't perform an import?
A5) Yes - this is supported. You may generate a ReleaseBuilder DIFF between two different environments so long as the DIFF is NOT used in the import. The tool itself does not prevent this action and actually - it can be helpful to compare two different enviornments. IBM Support performs this action regularly when analyzing customer data to check for differences between environments.
Q6) OK, how exactly does ReleaseBuilder "work"? Why can't I just do my deployments manually?
A6) Now we get into the heart of how WebSphere Portal works as a product and this will be a lengthy answer feeding into several other answers.
Dial back the clock several years - circa 2005, pre Web-2.0 days. Portlets were the primary focus back then - the capability to check your bank account balance, personalized health information specific to the user login, etc. were new and exciting. Today we consider them standard and boring but back then they were significant new features. WebSphere Portal as a product needed a means to copy the configuration of one environment to a second environment so what was tested in QA is what ended up in a PROD environment. Direct SQL statements to each environments databases were risky - not only were the SQL statements risky in and of themselves - disparate environments configurations - e.g. Derby in one environment (DEV) and DB2 in a different environment (STAGE) would result in different SQL statements. To ensure data in environment #1 made it to environment #2 successfully - an abstraction away from SQL statements was needed. Hence a scripting language specific to Portal - called XMLAccess - was created to provide a database representation of data that was not tied to a specific database implementation. As a result, data could be exported from one environment and imported to another environment, independent of the individual databases in use, and expect to produce the same end result at runtime.
One of the key principals of any database is primary keys - ensuring uniqueness of data in a given database table. The same is true with XMLAccess - each artifact in the Portal server must have some means of being uniquely identified. The title of the page itself is not considered unique - for example, environment #1 could have a page called "test page", and environment #2 could have a page called "test page". Both have the same title, but radically different contents of the page. Exporting/importing between environments would not produce the same result - nor should it. To ensure consistency of data between environments, in the XMLAccess scripting language Portal introduced the concept of "objectIDs". Each artifact in the Portal release database has an objectID, and, each objectID uniquely identifies the data it represents. There is a specific structure to objectID which you may read about in question #4 of the the XMLAccess FAQ. High-level summary, objectIDs unique identify data in the Portal database, and, objectIDs can be used to copy data between systems. Think of objectIDs like primary keys ... looking in the Portal database you will not find a primary key called objectID, but, think of the two concepts roughly equivalent.
How does this tie back to ReleaseBuilder and a deployment process? If you create a page named "test page" in environment #1, it may end up objectID/primary key "12345". Create a page named "test page" in environment #2 and you'll have objectID/primary key "67890". Exporting/importing the data between environments, you'll have two "test page" pages with the same name, but different objectIDs / primary keys. Our goal is to ensure the objectIDs / primary keys are the same across ALL environments for a successful deployment process. ReleaseBuilder and XMLAccess are both tools which can assist with guaranteeing the same objectIDs / primary keys exist across numerous environments. Thus, the database data in one environment is the same in a second environment independent of the actual database in use.
Fast forward the clock to today - while portlets are still used throughout the WebSphere Portal product - there has been an increasing emphasis on web content, mobile, and multimedia rich websites to entice end users. The architecture of Portal remains the same even with the shifts in the market - thus we still use ReleaseBuilder to move some data between systems while other forms of data (namely web content) may use a different tool - such as syndication - to move between systems. ReleaseBuilder has been in the WebSphere Portal product for a LONG time and continues to be a fundamental tool used as a part of a deployment process.
Q7) Am I required to use ReleaseBuilder / IBM tools for my deployments? What if I want to use my own tools?
A7) ReleaseBuilder and XMLAccess help guarantee the same objectIDs / primary keys between environments. IBM does not require the use of these tools - though does recommend them as a best practice for deployments to help ensure two environments remain the same with their database data. You may use any other tooling at your disposal to perform deployments - the primary focus is guaranteeing the objectIDs remain the same between systems.
Q8) What happens if the objectIDs are not the same between two different Portal systems?
A8) This is a condition we refer to as the systems being "out of sync". Let's take an example to illustrate an issue:
You deploy a portlet on DEV - it has objectID ABC
You deploy a portlet on QA - it has objectID XYZ
You create a page on DEV - it has object 12345. You add the portlet to the page. The portlet referenced has objectID ABC.
You export the page from DEV. You import to QA.
The QA system does not know about the objectID "ABC" ... it knows about the objectID of "XYZ". The page will properly import, but the page will NOT render the portlet correctly.
In summary, if the objectIDs are different between systems, export/import between systems becomes difficult to manage due to differences in objectIDs. Correcting the mismatched objectIDs is not a trivial task ... you MUST delete the "wrong" objectID / data in the database and reimport the "correct" objectID / data in the database for export/import to work.
Q9) My deployments are small. Do I have to know / care about the systems having the same objectIDs?
A9) If the number of changes you make are relatively small - say 10 total changes or less per deployment - we have advised many folks to perform their deployments manually rather than utilizing ReleaseBuilder / XMLAccess tooling. Why? Even with the manual changes - the overhead to perform the same tasks manually is often less than the overhead required to utilize the IBM tooling. XMLAccess and ReleaseBuilder - admittedly - is focused around larger environments. The more changes made to an environment - the more useful the tools become to capture those changes made and move them to a second environment. For example, if the number of changes increases from 10 to 100, would you feel confident in perform all 100 changes manually? Now what about 1000 changes? The scale of the changes is where the tooling has extraordinary benefit.
Q10) My deployments are large. How do I perform a proper ReleaseBuilder update?
A10) The following steps assume you have baselined your systems. See question #11 for how to baseline your systems if you have not done so already. The following instructions also assume two virtual portals are in use, though can be adapted for zero to hundreds of virtual portals in use.
0) Assume changes have been made in the SOURCE environment that need to be moved to TARGET.
1) Locate the three files that represent exports of your initial baseline
2) Create a new XMLAccess export from SOURCE of the base portal and both virtual portals, e.g.
./xmlaccess.sh -user wpsadmin -password wpsadmin -url http://SOURCE.portal.com:10039/wps/config -in /opt/IBM/WebSphere/PortalServer/doc/xml-samples/ExportRelease.xml -out /tmp/REV2_base.xml
./xmlaccess.sh -user wpsadmin -password wpsadmin -url http://SOURCE.portal.com:10039/wps/config/VP1 -in /opt/IBM/WebSphere/PortalServer/doc/xml-samples/ExportUniqueRelease.xml -out /tmp/REV2_vp1.xml
./xmlaccess.sh -user wpsadmin -password wpsadmin -url http://SOURCE.portal.com:10039/wps/config/VP2 -in /opt/IBM/WebSphere/PortalServer/doc/xml-samples/ExportUniqueRelease.xml -out /tmp/REV2_vp2.xml
3) In the same directory you ran xmlaccess from, do an "ls -la | grep -i release". You'll see a tool called releasebuilder.sh
4) Run releasebuilder for the base Portal as follows:
./releaseBuilder.sh -inOld /tmp/REV1_base.xml -inNew /tmp/REV2_base.xml -out /tmp/baseDIFF.xml
5) Run releasebuilder for the virtual portal #1 as follows:
./releaseBuilder.sh -inOld /tmp/REV1_vp1.xml -inNew /tmp/REV2_vp1.xml -out /tmp/vp1DIFF.xml -virtualPortalMode
6) Run releasebuilder for the virtual portal #1 as follows:
./releaseBuilder.sh -inOld /tmp/REV1_vp2.xml -inNew /tmp/REV2_vp2.xml -out /tmp/vp2DIFF.xml -virtualPortalMode
7) Backup the database and filesystem on TARGET.
8) Copy the portlets from from SOURCE to TARGET
*Note: The directory name which contains the portlets is:
This should be populated on both SOURCE and TARGET. Copy the entire directory over from SOURCE, overwriting existing contents on TARGET.
9) Import the DIFF file for the base portal to TARGET, e.g.
./xmlaccess.sh -user wpsadmin -password wpsadmin -url http://TARGET.portal.com:10039/wps/config -in /tmp/baseDIFF.xml -out /tmp/baseimportresults.xml
10) Import the DIFF file for the first virtual portal to TARGET, e.g.
./xmlaccess.sh -user wpsadmin -password wpsadmin -url http://TARGET.portal.com:10039/wps/config/VP1 -in /tmp/vp1DIFF.xml -out /tmp/vp1importresults.xml
11) Import the DIFF file for the second virtual portal to TARGET, e.g.
./xmlaccess.sh -user wpsadmin -password wpsadmin -url http://TARGET.portal.com:10039/wps/config/VP2 -in /tmp/vp2DIFF.xml -out /tmp/vp2importresults.xml
12) Sydndicate your updated WCM content between SOURCE and TARGET.
13) Run the activate-portlets configuration task on TARGET via ConfigEngine, e.g.
14) Test and validate the DIFF imports on the TARGET system.
15) Restart your Portal servers. This is not strictly required at this time, however, if you restart the severs as a part of normal maintenance anyways, now would be a good time to do so.
Q11) How do I "baseline" my systems?
A11) We talked about objectIDs previously. If you install a Portal 8.0 or 8.5 system, the IBM provided pages and portlets will have all the same objectIDs on installation. Cumulative fix upgrades will also preserve the same objectIDs. It is new data added to the system which may introduce differences in objectIDs between two different Portal servers. If we need to ensure the data in two different Portal servers is the same, proceed with the following steps: https://www.ibm.com/developerworks/community/blogs/portalops/entry/staging_to_production_portal_8_without_paa?lang=en
Steps repeated here with some slight modifications:
1) Backup the filesystems and databases on TARGET
2) Obtain an XMLAccess export of your SOURCE environment using ExportRelease.xml as the input file. Do NOT use Export.xml, e.g.
./xmlaccess.sh -user wpsadmin -password wpsadmin -url http://SOURCE.portal.com:10039/wps/config -in /opt/IBM/WebSphere/PortalServer/doc/xml-samples/ExportRelease.xml -out /tmp/SOURCEexport.xml
3) Delete all WCM libraries scoped to your Virtual Portals on TARGET. Delete all virtual portals on TARGET.
*Note #1: If you have any WCM libraries scoped to the VP, delete those prior to deleting the VP
*Note #2: You may use either the Manage Virtual Portals administration portlet, OR, the delete-virtual-portal command line to do so.
*Note #3: This step is needed on early code levels of 8.5 (before CF06) and 8.0 (before CF10). It is recommended to perform the steps on later code levels to avoid trouble, but it is not strictly required.
4) On TARGET, run XMLAccess with Task.xml as the input file, e.g.
./xmlaccess.sh -user wpsadmin -password wpsadmin -url http://TARGET.portal.com:10039/wps/config -in /opt/IBM/WebSphere/PortalServer/doc/xml-samples/Task.xml -out /tmp/taskresults.xml
5) Delete your WCM libraries in your base Portal on TARGET.
6) Run empty-portal on your TARGET environment via ConfigEngine, e.g.
*Note #1: You need to have the WasPassword and PortalAdminPwd filled out in wkplc.properties, or passed in as a parameter to run this configuration task.
*Note #2: WARNING - as the name implies, this will make your Portal inaccessible as all pages and all portlets will be wiped from the system.
7) Copy the portlets from from SOURCE to TARGET
*Note: The directory name which contains the portlets is:
This should be populated on SOURCE and empty on TARGET. Copy the entire directory over.
*Note: The portlets MUST come from the SOURCE system to ensure a perfect copy is made.
8) On TARGET, import the SOURCE xmlaccess export from step #2, e.g.
./xmlaccess.sh -user wpsadmin -password wpsadmin -url http://TARGET.portal.com:10039/wps/config -in /tmp/SOURCEexport.xml -out /tmp/importresults.xml
9) Run the activate-portlets configuration task on TARGET via ConfigEngine, e.g.
10) Restart Portal
11) Verify you can access your TARGET Portal via a web browser and login, e.g.
12) On SOURCE, run the following configuration task:
*Make note of the output here, we will need it in steps #14 and #17
13) On SOURCE, obtain an XMLAccess ExportUniqueRelease.xml export of each of your virtual portals, e.g.
./xmlaccess.sh -user wpsadmin -password wpsadmin -url http://SOURCE.portal.com:10039/wps/config/VP1 -in /opt/IBM/WebSphere/PortalServer/doc/xml-samples/ExportUniqueRelease.xml -out /tmp/SOURCEVP1export.xml
./xmlaccess.sh -user wpsadmin -password wpsadmin -url http://SOURCE.portal.com:10039/wps/config/VP2 -in /opt/IBM/WebSphere/PortalServer/doc/xml-samples/ExportUniqueRelease.xml -out /tmp/SOURCEVP2export.xml
14) On TARGET, update the wkplc.properties file for your first virtual portal with the following:
15) On TARGET, run the create-virtual-portal ConfigEngine command to create the first virtual portal, e.g.
And then import to that virtual portal via XMLAccess, e.g.
./xmlaccess.sh -user wpsadmin -password wpsadmin -url http://TARGET.portal.com:10039/wps/config/VP1 -in /tmp/SOURCEVP1export.xml -out /tmp/importresults.xml
*WARNING: Make SURE you specify /wps/config/VP1, and the VP1 is case sensitive! Otherwise you may risk importing your virtual portal to your base portal, which would not be good!
16) Verify you can access your TARGET virtual portal #1 via a web browser and login, e.g.
17) Repeat steps #13-#16 for each virutal portal in your configuration.
18) Setup a syndicator/subscriber pairing between SOURCE and TARGET and syndicate the libraries.
19) Restart Portal (not strictly required, but a good idea to clean out temporary caches, memory, etc.)
20) Create a new XMLAccess export from SOURCE of the base portal and all virtual portals, e.g.
./xmlaccess.sh -user wpsadmin -password wpsadmin -url http://SOURCE.portal.com:10039/wps/config -in /opt/IBM/WebSphere/PortalServer/doc/xml-samples/ExportRelease.xml -out /tmp/REV1_base.xml
./xmlaccess.sh -user wpsadmin -password wpsadmin -url http://SOURCE.portal.com:10039/wps/config/VP1 -in /opt/IBM/WebSphere/PortalServer/doc/xml-samples/ExportUniqueRelease.xml -out /tmp/REV1_vp1.xml
./xmlaccess.sh -user wpsadmin -password wpsadmin -url http://SOURCE.portal.com:10039/wps/config/VP2 -in /opt/IBM/WebSphere/PortalServer/doc/xml-samples/ExportUniqueRelease.xml -out /tmp/REV1_vp2.xml
etc. for each virtual portal
21) Save the files from step #20 for later use, as discussed in question #10
Q12) Do my LDAP schemas need to be the same between environments?
A12) Ideally yes - but not strictly required. Let's suppose we have two environments with the following LDAP structures:
It is permitted to create the DIFF file from STAGE, and perform a find/replace of "dc=ibmdev,dc=com" to "dc=ibmqa,dc=com". Other examples include a find and replace of the full username/groupDN in the DIFF file. This is also permitted. The key in the DIFF files are the objectIDs remaining the same between systems. The metadata - such as the LDAP DNs - can be modified as needed. However, note that the more manual changes that are made as a part of the deployment process (rather than allowing XMLAccess / ReleaseBuilder to perform all changes in a scripted manner) ... the more potential for error to be introduced and the end result in PROD not being functionally correct.
Q13) How to portlet deployments work with ReleaseBuilder?
A13) If you update the .war file by itself, ReleaseBuilder will NOT recognize the change. ReleaseBuilder will recognize changes in the Portal database, but not changes purely on the filesystem. There are three potential means of addressing this scenario:
1) Update the uid= or id= of the portlet.xml file with a versioning schema.
Pro: Each portlet deployment will force an update in the Portal database and Portal will redeploy the portlet in such a scenario.
Con: Most source control systems have their own versioning schema independent of the application id.
2) Manually update the .war file in the Portal or WAS administration console - commenting out the tags that in XMLAccess scripts that are used to deploy portlets
Pro: More control over portlet deployments if portlet updates are performed infrequently.
Con: Additional manual steps required during a deployment
Comment: We have seen this performed in practice on more than a few occasions. There is often a desire to tightly control when portlets are redeployed in a system and we have seen a number of DIFF files from ReleaseBuilder with the tags commented out to allow for this control. This is supported/permitted, so long as only the tags are commented out and other database changes for the portlet remain in the DIFF file to be promoted between environments.
3) Write a custom XMLAccess script to redeploy custom application on each deployment.
Pro: Guarantees updates each and every time for custom portlets. Scripts are relatively quick to author - no more than 15 minutes - a series of copy/paste.
Con: Murphy's Law / Bad luck could cause unforeseen issues on applications that have not changed and if redeployed now have issues.
As an example of scenarios #2 and #3 - a RelesaeBuilder DIFF file may show the following output to deploy a portlet - in this case the World Clock portlet
<web-app action="update" active="true" domain="rel" objectid="Z1_OHK812O0L0MSC0ALM7292L3004" removable="true" uid="com.ibm.wps.portlets.worldclock">
<access-control externalized="false" owner="uid=wpsadmin,o=defaultwimfilebasedrealm" private="false"/>
<servlet action="update" active="true" domain="rel" name="World Clock.$appuid.com.ibm.wps.portlets.worldclock.1" objectid="ZV_OHK812O0L0MSC0ALM7292L3006" remote-cache-dynamic="false"/>
<portlet-app action="update" active="true" domain="rel" name="World Clock" objectid="Z2_OHK812O0L0MSC0ALM7292L3002" uid="com.ibm.wps.portlets.worldclock.1">
<access-control externalized="false" owner="uid=wpsadmin,o=defaultwimfilebasedrealm" private="false"/>
<portlet action="update" active="true" defaultlocale="en" domain="rel" name="World Clock" objectid="Z3_OHK812O0L0MSC0ALM7292L3001" provided="false" servletref="ZV_OHK812O0L0MSC0ALM7292L3006">
<access-control externalized="false" owner="uid=wpsadmin,o=defaultwimfilebasedrealm" private="false"/>
For scenario #2 - you may comment out the second line, as follows:
For in scenario #3 - you may copy/paste similar stanzas for all of your custom applications and redeploy them during each deployment
Section 7: Fixing Common Issues
Q1) I created an artifact named "test". I deleted it. I want to recreate the same artifact named "test". It's throwing an error. Why is this happening?
Appendix A: Terminology
||staging to production
||system where data originates
||system where data is imported from SOURCE
||command line tool to export/import Portal release database data
||command line tool to compare two different XMLAccess exports of the same Portal server and produce a differential file showing changes made
|DIFF / Delta
||the differential file created by ReleaseBuilder
||Portal Application Archive
||a tool in the Portal administration console which can copy JCR data (WCM Libraries, Managed Pages, Managed Rules, etc.) between two different locations
||when syndication sends data from SOURCE to TARGET
||when syndication sends data from SOURCE to TARGET, or, TARGET to SOURCE.
||line of business
||The main portal site - typically /wps/portal
Separate area of Portal site with different look and feel, typically assigned per LOB. Uses either a URL context or a hostname, e.g.
wps/portal/hr OR hr.ibm.com/wps/portal
||Cumulative fix. Updated code levels and new features of WebSphere Portal delivered approximately every 8-12 weeks.
Appendix B: References
- Step-by-Step Cluster Guide for IBM WebSphere Portal v8.5
- Managing the ReleaseBuilder deployment process for IBM WebSphere Portal - written for Portal v7, content still applicable for v8.0 and v8.5
- Staging to Production Portal 8 without PAA - applicable for Portal v6.0 - v8.5
- Step-By-Step Guide to performing staging to production using Portal Application Archive in WebSphere Portal 8.5 - Portal v8.5 only
- Administering IBM WebSphere Portal 8.5: A comprehensive workshop - Portal v8.5 only; other versions exist for prior Portal versions
- How to generate a complete XMLAccess export of a Portal configuration
- XMLAccess Frequently Asked Questions
- Practical Advice On Deploying Portal as A Farm
- Virtual Portal Planning and Considerations
Appendix Y: Acknowledgements
- The WebSphere Portal Information Development team for providing the Product Documentation.
- David Batres, Staff Software Engineer at IBM. For countless whiteboard discussions to hash out various technical topics.
Appendix Z: About the Author of this Document
Travis Cornwell is an Advisory Software Engineer at IBM working out of Research Triangle Park, North Carolina, USA. He began supporting the WebSphere Portal product in 2009 and is a subject matter expert in the areas of installation, configuration, administration, security, and performance. Travis has written numerous technical documents for WebSphere Portal and has been the primary technical reviewer/editor for many additional items.
If you have any feedback about the content of this document, Travis can be reached at: firstname.lastname@example.org.
If you encounter any failures following the steps in this document or have questions on how the content of this document can be utilized in your deployment processl, you may open a PMR with WebSphere Portal Level 2 Support.