IBM Digital Experience wiki: WCM Troubleshooting: Troubleshooting Syndication

Troubleshooting Syndication

Added by Sunil Bhatnagar | Edited by IBM contributor

Dineshbabu Ramakrishnan on July 2, 2010 | Version 7

Abstract

Troubleshooting guide

ShowTable of Contents

HideTable of Contents

1 Understand the syndication environment
2 Verify the enviroment and configuration
3 Facts to keep in mind
4 Reading the traces
5 Eventlog Database and Syndication
6 Most Common Issues (and how to address them)
7 Syndication and Clustering
8 Syndication Resources

Understand the syndication environment

Is the syndication being done for the first time to the subscriber ?

Is the syndication being done between One Syndicator and One Subscriber or multiples ? If multiples, get details in terms of a topology of the syndication setup, detailing number of syndication servers, number of syndicator/subscriber objects on a server and the flow of syndication.

Is either the syndicator or the subscriber a part of a cluster ? If so is syndication being done from/to one node on a cluster or the entire cluster through httpserver ?

Is the syndication being done for one library or multiple libraries ? If multiple, do the items in one library reference the other ?

Is this synidcation being done for All items or All Live Items ?

Is this automatic or manual syndication ?

Are the syndicator and subscriber pointing to the same ldap ? If different ldaps then are the users and groups identical ?

Are the syndicator(s) and subscriber(s) at the same fix level ?

Has any authoring been doen on the subscriber ?

If doing All items syndication - is versioning enabled on the syndicator ?

Did syndication ever work completely ?

Are any of the libraries being syndicated, Migrated libraries, if so, from which version ?
Collect the Syndication MustGather information as below:

for V6.0.X
http://www-01.ibm.com/support/docview.wss?rs=688&uid=swg21243980
for V6.1.X
http://www-01.ibm.com/support/docview.wss?rs=688&uid=swg21316241

Capture a screenshot of the syndicator and subscriber objects

Verify the enviroment and configuration

From the MustGather files:
From the versioninfo.log verify both syndicator and subscriber have the same exact WCM version and WCM fixes installed. Both the syndicator and subscriber systems should match with regards to WCM version and WCM fix level.

Check for loose class files on the syndicator and subscriber system as this will impact the fix levels.
location for these loose class files:
V6.0.X: \wcm\shared\app
V6.1.X: \wcm\prereq.wcm\wcm\shared\app
Check for the following properties in the syndicator and subscriber's -WCMConfigService.properties file
location for this file:
V6.0.X: \wcm\shared\app\config\wcmservices\WCMConfigService.properties file
V6.1.X: \PortalServer\wcm\shared\app\config\wcmservices

deployment.itemDispatcherUrl=http://${WCM_HOST}:${WCM_PORT}/${WCM_CONTEXT_ROOT}/connect/?MOD=ItemDispatcher
deployment.syndicatorUrl=http://${WCM_HOST}:${WCM_PORT}/${WCM_CONTEXT_ROOT}/connect/?MOD=Synd
deployment.subscriberUrl=http://${WCM_HOST}:${WCM_PORT}/${WCM_CONTEXT_ROOT}/connect/?MOD=Subs
deployment.subscriberOnly=false (this should never be set to true on the syndicator)
deployment.itemChangedTaskDelay=30 ( in seconds – this is the default value)

Items to verify from the screenshots:
Syndicator Object

Name of the syndicator - (Keep a note of this as this will be referenced in the logs)

Name of the subscriber - (Keep a note of this as this will be referenced in the logs)

Subscriber ID The ID of the subscriber, used to setup the subscription - Make sure this matches the ID on the subscriber object

Subscriber URL The URL of the subscriber - this should be a fully qualified URL (in some cases an IP address) and should match the value default on the subscriber object. (Note: This value reflect the values of the variables WCM_HOST and WCM_PORT defined in WAS.These variables can be overridden by hardcoding the values in the WCMConfigService.properties file)

Libraries selected for syndication.

Configuration of the item gatherer - all items or all live items syndication

Syndicator is enabled or not – the checkbox should be checked

Subscriber Object

Name of the subscriber

The name of the syndicator

The ID of the syndicator used to setup the subscription.

Syndicator URL

Enable the subscriber

Facts to keep in mind

1.Syndication is just transferring content items stored in the JCR repository of the syndicator server to the JCR repository of the subscriber server

2.The UUIDs of each of these items is maintained the same across the Syndicator and Subscriber databases.

3.To have a simple picture in mind the syndicator and subscriber are sort of gatekeepers to the JCRDB that stores the WCM content and processes these items through syndication as deemed necessary by the event log.

4.Copying of the JCR repository itself from the syndicator to subscriber is a brute force method of ensuring that all the data on the syndicator showsup on the subscriber. Here are somethings to consider when doing so:
This will impact PDM/PZN as well since they use the same JCRDB
We suggest this ONLY if this is a first time syndication
Refer to the following link for details on performing this:
http://www-10.lotus.com/ldd/portalwiki.nsf/dx/18082009091727PMWEB3JG.htm

5.Syndication uses the http protocol and does not support the use of https although it may work technically as long as the certificates are setup correctly between the servers.

6. Resetting of the event log will cause a FULL REBUILD ( if automatic syndication is enabled or if manual syndication is initiated)

7.Once the event log gets created, any content item saved to the JCR repository for a library is logged to the event log as one or more entries. (depending on it's state and other settings, for example a live item is logged twice: once against the live item gatherer and again against the all items gatherer, there will also be additional entries for versions).

8.The event log provides critical information to determine if an object is not getting syndicated.
More information in the section: EventLog and Syndication

9.IBM Support tools portlet has some useful tool that can be used in syndication troubleshooting
The portlet can be downloaded from here (P.S: Requires registering with the site):
https://greenhouse.lotus.com/plugins/plugincatalog.nsf/assetDetails.xsp?action=editDocument&documentId=AE2BB2412F20AA318525772E006F7014
Some of the tools available in this portlet are:
DisplayNodes
DisplayEventLog
ClearPersistenceTable
Clearcaches

10.Authoring on the subscriber side can cause issues with syndication since the last modified date on the item on the subscriber will change, as a result when the subscriber processes this item being pushed from the syndicator it might skip saving this item as it has a current or newer version of the item in its database. (Authoring on the subscriber side can be ok if the client is aware of the limitations and manages them properly like in two way syndication scenarios where authoring is done on both servers but in different libraries)

11.Here is an example of a very typical syndication setup:Topology:Syndicator (Authoring server - Has the WCM Authoring portlet installed) =one way syndication=>Subscriber (Rendering server that has the WCM Local Rendering Portlet installed) to display the content - probably does not have the authoring portlet installed) .Syndicating two libraries one content and one design. The content library has the content items while the design library has the AT, PT, components etc.Typically the content library will reference the items in the design library. Both these libraries will need to be syndicated simultaneously for the syndication to succeed. Typically both these libraries are added to the syndicator in live items scope. For typical setups such as above it does make sense to syndicate live items only since there is no authoring being done on the rendering server side and it is being used only to render content - LRP/RRP will only display live content.So if a customer is doing ALL items syndication it is valid to ask the reason for it and if they have a setup similar to above whether they would like to consider doing live items

12.Both subscription partners should share the same user repository (such as LDAP) if they use automatic syndication. If this is not the case, disable automatic syndication and perform manual syndication. The latter should be followed by executing memberfixer. Please note that the purpose of automatic syndication is defeated if the LDAPs are not the same or at the very least the users and groups are not identical in both LDAPs with identical DNs.

13. Key traces to be enabled ( on BOTH syndicator and subscriber):
com.aptrix.deployment.*=all
com.aptrix.syndication.*=all
For event log related issues
com.ibm.workplace.wcm.services.eventlog.*=finest:com.ibm.workplace.wcm.services.transaction.steps.eventlog.*=finest
(The eventlog trace will show if an item got added to the event log and how.)

14.Syndication can handle the syndicator and subscriber servers being in different time zones as long as the server clocks for both are not more than a minute apart with the timezone differences factored in. For the servers on the same timezone it will be the same rule of the server times not being more than a minute apart.

15.Items that were deleted and then purged will not be syndicated if the event log is reset. To give an example if an item was deleted and purged on the syndicator side, the entry is recorded in the event log so that the same action can be performed on the subscriber side upon syndication. However if the eventlog is reset prior to the syndication then the rebuilt event log will not have this entry and hence this item will not get syndicated.

16. If you are resetting the event log then make sure that the event log traces: com.ibm.workplace.wcm.services.eventlog.*=finest:com.ibm.workplace.wcm.services.transaction.steps.eventlog.*=finest are applied prior to resetting the event log. Enable eventlog traces from WAS Admin console in config mode and save the configuration, then stop the portal server and reset eventlog. Make sure to remove traces from WAS Admin console in config and runtime mode, once eventlog rebuild is complete and the server has started up.The server startup with the eventlog traces set and the event log being built provides vital information.

17.If the repository( library) is large ( large would mean in the range of 20,000-30,000 items)then the customer would want to increase the number of historical trace files and trace files size in WebSphere Application Server admin console. A setting of 20 historical files and a file size of 50 MB is a good number and should be adequate for most scenarios. The idea is to ensure that trace information is not lost due to file rollovers.

18.Use the DisplayNodes tool included in IBM Support Tools Portlet for WCM to collect the output for any failing uuid. The displayNodes.jsp output will show all the references to that UUID and hence it is a useful tool in certain scenarios.

19.Library access permissions are not syndicated. If the library does not exist on the subscriber, it will be created during syndication but no access control settings are specified on the new library. On rendering servers you might want to specify different access control settings, for example you may want to disallow content entry for all users.

20.On the syndicator, the PackageGenerator class generates the packages to send to the subscriber and on the subscriber, the PackageProcessor class processes packages sent by the syndicator.

21.When eventlog is rebuild, it processes every WCM library (enabled) regardless of whether it is in a syndication definition or not. This makes it long time to rebuild eventlog depending on how many WCM libraries (enabled) are there. Disabled libraries are not processed and not added to eventlog.

22.Syndication cannot be scheduled but the frequency of syndication occurrence can be set. By default it is set for 30 seconds but can be changed by changing the value deployment.itemChangedTaskDelay in the WCMConfigService.properties file. Also the occurrence of syndication in certain scenarios can be controlled by using the publish date of content - the publishing of the content can be delayed and hence syndication can be delayed. This can used for example - to allow syndication to happen only during non peak hours

23.Prior to running the configuration task wcm-reset-event-log the Portal server needs to be stopped and then restarted when the task has completed. The config task merely clears the event log DB, the eventlog is rebuilt when the portal server is started and may take a while to comeup. To address this PK88848 introduced the module to reset the eventlog DB while the portal server is running. This fix is available in WCM Cumulative iFix 21(PK90387) for 6.0.1.X and WCM cumulative iFix 15 (PK90388) for V6.1.x. The FAQs related to this module are technoted here: http://www-01.ibm.com/support/docview.wss?uid=swg21396691

24.One of the common misconception is that LIVE items syndication means syndicating only published items. The Live Items syndication includes published items, expired items and non workflowed items (but not drafts, versions and deleted items)

25. When performing a first time or large syndication it is recommended to turn of the JCR text search.Refer to the following technote on how to do that:http://www-01.ibm.com/support/docview.wss?rs=6h88&uid=swg21299498

Reading the traces

1.Most activity occurs on the subscriber side since that is where the subscriber will go through the list of items to be saved, fetch these items from the syndicator database, add them to the respository and then save the document in its own respository. Basically here is what happens in very very layman's terms (this maynot be technically very accurate but the idea is to give a mental picture of the syndication functioning in the most common scenario): On the syndicator side the itemchangedtaskdelay value determines when the system will check for updates to the event log. If it finds some updated items it will then contact the subscriber and provide it with the list of items to be updated. The subscriber then starts processing each item in this list. It skips items that are already at the most current version (Note: you will see the statement -"skipping item" in the logs) and it adds the newer modified items to the items that need to be updated. After doing that it will fetch theses items from the syndicator server's database and then try to save these documents( items) on the subscriber JCRDB.Most syndication errors are seen during this time. If the subscriber is able to process all the items sent by the syndicator it will send the syndicator a 603 message code on receipt. If for some reason there are some items that failed to be processed by the subscribers those items will be processsed the next time the itemgatherer queries the eventlog.

2. New items are saved first, followed by updates and lastly by removals

3. Typically you will see a message at the end of the syndication in the subscriber trace logs such as this

[6/4/08 22:20:02:141 EDT] 00000084 PackageProces I ****************** Start of subscription update report ******************
[6/4/08 22:20:02:156 EDT] 00000084 PackageProces I Subscription update started: 2008-06-04T22:19:17.125
[6/4/08 22:20:02:156 EDT] 00000084 PackageProces I Subscription update finished: 2008-06-04T22:20:02.141 (45016 milliseconds)
[6/4/08 22:20:02:172 EDT] 00000084 PackageProces I Syndicator: syndicator6_to_subscriber4, URL=http://testsyndication1.raleigh.ibm.com:10038/wps/wcm/connect/?MOD=Synd
[6/4/08 22:20:02:172 EDT] 00000084 PackageProces I Subscriber: subscriber4_from_syndicator6, URL=http://testsyndication2.raleigh.ibm.com:10038/wps/wcm/connect/?MOD=Subs
[6/4/08 22:20:02:172 EDT] 00000084 PackageProces I Full update
[6/4/08 22:20:02:188 EDT] 00000084 PackageProces I Updates sent: 21
[6/4/08 22:20:02:188 EDT] 00000084 PackageProces I Removes sent: 0
[6/4/08 22:20:02:203 EDT] 00000084 PackageProces I Items up to date: 20
[6/4/08 22:20:02:203 EDT] 00000084 PackageProces I No errors were detected during this syndication.
[6/4/08 22:20:02:219 EDT] 00000084 PackageProces I

High Level Details

[6/4/08 22:20:02:234 EDT] 00000084 PackageProces I Start of library updates, time=2008-06-04T22:20:02.109, count=1
[6/4/08 22:20:02:250 EDT] 00000084 PackageProces I Total items: 1
[6/4/08 22:20:02:250 EDT] 00000084 PackageProces I End of library updates, time=2008-06-04T22:20:02.109, elapsed=0 milliseconds
[6/4/08 22:20:02:250 EDT] 00000084 PackageProces I Start of draft item removal, time=2008-06-04T22:20:02.109, count=0
[6/4/08 22:20:02:266 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:266 EDT] 00000084 PackageProces I End of draft item removal, time=2008-06-04T22:20:02.109, elapsed=0 milliseconds
[6/4/08 22:20:02:281 EDT] 00000084 PackageProces I Start of draft item removal, time=2008-06-04T22:20:02.109, count=0
[6/4/08 22:20:02:281 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:281 EDT] 00000084 PackageProces I End of draft item removal, time=2008-06-04T22:20:02.109, elapsed=0 milliseconds
[6/4/08 22:20:02:297 EDT] 00000084 PackageProces I Start of new published items, time=2008-06-04T22:20:02.109, count=0
[6/4/08 22:20:02:297 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:312 EDT] 00000084 PackageProces I End of new published items, time=2008-06-04T22:20:02.109, elapsed=0 milliseconds
[6/4/08 22:20:02:312 EDT] 00000084 PackageProces I Start of update published items, time=2008-06-04T22:20:02.109, count=0
[6/4/08 22:20:02:312 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:328 EDT] 00000084 PackageProces I End of update published items, time=2008-06-04T22:20:02.109, elapsed=0 milliseconds
[6/4/08 22:20:02:328 EDT] 00000084 PackageProces I Start of new draft items, time=2008-06-04T22:20:02.125, count=0
[6/4/08 22:20:02:344 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:344 EDT] 00000084 PackageProces I End of new draft items, time=2008-06-04T22:20:02.125, elapsed=0 milliseconds
[6/4/08 22:20:02:344 EDT] 00000084 PackageProces I Start of update draft items, time=2008-06-04T22:20:02.125, count=0
[6/4/08 22:20:02:359 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:359 EDT] 00000084 PackageProces I End of update draft items, time=2008-06-04T22:20:02.125, elapsed=0 milliseconds
[6/4/08 22:20:02:375 EDT] 00000084 PackageProces I Start of item removal, time=2008-06-04T22:20:02.125, count=0
[6/4/08 22:20:02:375 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:375 EDT] 00000084 PackageProces I End of item removal, time=2008-06-04T22:20:02.125, elapsed=0 milliseconds
[6/4/08 22:20:02:391 EDT] 00000084 PackageProces I Start of deferred new published content links, time=2008-06-04T22:20:02.125, count=0
[6/4/08 22:20:02:391 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:391 EDT] 00000084 PackageProces I End of deferred new published content links, time=2008-06-04T22:20:02.125, elapsed=0 milliseconds
[6/4/08 22:20:02:406 EDT] 00000084 PackageProces I Start of deferred new draft content links, time=2008-06-04T22:20:02.125, count=0
[6/4/08 22:20:02:406 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:422 EDT] 00000084 PackageProces I End of deferred new draft content links, time=2008-06-04T22:20:02.125, elapsed=0 milliseconds
[6/4/08 22:20:02:438 EDT] 00000084 PackageProces I Start of resolving references, time=2008-06-04T22:20:02.125, count=0
[6/4/08 22:20:02:438 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:438 EDT] 00000084 PackageProces I End of resolving references, time=2008-06-04T22:20:02.125, elapsed=0 milliseconds
[6/4/08 22:20:02:453 EDT] 00000084 PackageProces I Start of item renaming, time=2008-06-04T22:20:02.141, count=0
[6/4/08 22:20:02:453 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:469 EDT] 00000084 PackageProces I End of item renaming, time=2008-06-04T22:20:02.141, elapsed=0 milliseconds
[6/4/08 22:20:02:469 EDT] 00000084 PackageProces I Start of create versions, time=2008-06-04T22:20:02.141, count=0
[6/4/08 22:20:02:469 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:484 EDT] 00000084 PackageProces I End of create versions, time=2008-06-04T22:20:02.141, elapsed=0 milliseconds
[6/4/08 22:20:02:484 EDT] 00000084 PackageProces I Start of update versions, time=2008-06-04T22:20:02.141, count=0
[6/4/08 22:20:02:500 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:500 EDT] 00000084 PackageProces I End of update versions, time=2008-06-04T22:20:02.141, elapsed=0 milliseconds
[6/4/08 22:20:02:500 EDT] 00000084 PackageProces I Start of version removal, time=2008-06-04T22:20:02.141, count=0
[6/4/08 22:20:02:516 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:516 EDT] 00000084 PackageProces I End of version removal, time=2008-06-04T22:20:02.141, elapsed=0 milliseconds
[6/4/08 22:20:02:531 EDT] 00000084 PackageProces I Start of item status update, time=2008-06-04T22:20:02.141, count=0
[6/4/08 22:20:02:531 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:531 EDT] 00000084 PackageProces I End of item status update, time=2008-06-04T22:20:02.141, elapsed=0 milliseconds
[6/4/08 22:20:02:547 EDT] 00000084 PackageProces I

Low Level Details

[6/4/08 22:20:02:547 EDT] 00000084 PackageProces I Start of library updates, time=2008-06-04T22:20:02.109, count=1 [DepRef(id:409ae28049c897ccab84bf55786cef3c type: com.ibm.workplace.wcm.services.library.Library nonDraft:true draft:false purged:false parentId:409ae28049c897ccab84bf55786cef3c timeStamp:1211200237696 stateUpdate: false versions:null)]
[6/4/08 22:20:02:562 EDT] 00000084 PackageProces I Start of draft item removal, time=2008-06-04T22:20:02.109, count=0 []
[6/4/08 22:20:02:562 EDT] 00000084 PackageProces I Start of draft item removal, time=2008-06-04T22:20:02.109, count=0 []
[6/4/08 22:20:02:578 EDT] 00000084 PackageProces I Start of new published items, time=2008-06-04T22:20:02.109, count=0 []
[6/4/08 22:20:02:578 EDT] 00000084 PackageProces I Start of update published items, time=2008-06-04T22:20:02.109, count=0 []
[6/4/08 22:20:02:578 EDT] 00000084 PackageProces I Start of new draft items, time=2008-06-04T22:20:02.125, count=0 []
[6/4/08 22:20:02:594 EDT] 00000084 PackageProces I Start of update draft items, time=2008-06-04T22:20:02.125, count=0 []
[6/4/08 22:20:02:609 EDT] 00000084 PackageProces I Start of item removal, time=2008-06-04T22:20:02.125, count=0 []
[6/4/08 22:20:02:609 EDT] 00000084 PackageProces I Start of deferred new published content links, time=2008-06-04T22:20:02.125, count=0 []
[6/4/08 22:20:02:609 EDT] 00000084 PackageProces I Start of deferred new draft content links, time=2008-06-04T22:20:02.125, count=0 []
[6/4/08 22:20:02:625 EDT] 00000084 PackageProces I Start of resolving references, time=2008-06-04T22:20:02.125, count=0 []
[6/4/08 22:20:02:625 EDT] 00000084 PackageProces I Start of item renaming, time=2008-06-04T22:20:02.141, count=0 []
[6/4/08 22:20:02:641 EDT] 00000084 PackageProces I Start of create versions, time=2008-06-04T22:20:02.141, count=0 []
[6/4/08 22:20:02:641 EDT] 00000084 PackageProces I Start of update versions, time=2008-06-04T22:20:02.141, count=0 []
[6/4/08 22:20:02:641 EDT] 00000084 PackageProces I Start of version removal, time=2008-06-04T22:20:02.141, count=0 []
[6/4/08 22:20:02:656 EDT] 00000084 PackageProces I Start of item status update, time=2008-06-04T22:20:02.141, count=0 []
[6/4/08 22:20:02:656 EDT] 00000084 PackageProces I ******************* End of subscription update report *******************

which provides the subscription summary

4.Sometimes instead of the 603 -" no more confirmations to send" the subscriber responds back to the syndicator with a 200 message code where you will see the phrase "OK" being used. This phrase indicates that the subscriber is still processing the updates that the syndicator sent in earlier and doesnot wish to process any new updates. This will happen when say the syndicator sends in a large number of updates, the subscriber is processing each one of them ( and this is taking time) meanwhile the itemchangedtaskdelay which is set to 30 seconds causes the syndicator to contact the subscriber again for the eventlog updates.

5.Also while the subscriber is processing the updates sent by the syndicator the status of the subscriber will showup as active while the syndicator will showup as idle for most part.

6.To read the traces, start with the syndicator log to identify the syndication initiation point and co-relate the corresponding message on the subscriber systemout.log. Once identified, use the systemout.log to verify where the error occurs ( it will be easier to scan through the systemout.log than the trace.log to find this). Once identified then find the corresponding entry in the trace.log to see if additional information is available.

7.As mentioned the majority of the activity in the most common scenarios occurs on the subscriber side. Hence the subscriber logs typically are bigger than the syndicator ones. And the errors will most probably showup in the subscriber logs.

8.Edit the WCMConfigService.properties file and change the deployment.enableReport setting to "true". This enables high level reporting of syndication to the SystemOut log on the subscriber server. It provides a summary of items that were processed, and which items failed syndication to help with troubleshooting syndication issues.

Eventlog Database and Syndication

First of all syndication has changed dramatically in version v6 compared to v5 and although some parts use the same names (the item gatherers for example), they are completely different to what they were in previous releases.

Syndication begins at a library level. Each WCM library is associated with two item gatherers: the "All Items" gatherer and the "Live Items" gatherer. These item gatherers are unique for every library and provide the means to track items by status in specific libraries. Unlike in previous releases, the item gatherers do NOT contain any information about the items, they are simply a label used to associate a syndicator object with the items it must syndicate.

When an item is created or modified, the state of the item is updated in the event log database. It is important to understand that the event log stores the STATE of items, NOT the actions that were performed on them. To illustrate, let's look at the following example:

If you create a new content document, an entry similar to the following will be added to the event log:

Where:
ITEMID: The ID of the content document.
TYPENAME: The class type of the content document.
ITEMTYPE: Whether the content document is published (P) or draft (D)
STATE: A number representing the current state of the content document.
DELETED: Whether the content document has been deleted (Y), purged (P) or moved (M).
PARENTID: The ID of the content document's parent node (i.e. a sitearea id).
TSTAMP: The timestamp of when the content document was last modified.
ITEMGATHERERID: The ID of the appropriate item gatherer (All or Live) associated with the content document's library.

When the content document is modified, the entry in the event log is updated to reflect the updated state of the item, for example:

Syndication uses the information in the event log to determine what items it must send to a subscriber. When you create a syndicator object you will select a library and a scope (All items or Live item). This process will associate the syndicator object with a particular ITEMGATHERERID. When syndication occurs, the syndicator will obtain all the items in the event log that match it's associated item gatherer and send them to the subscriber to be processed. The full process will use the STATE, DELETED flag and other information in the event log to determine what action to perform on the subscriber.

Basically the event log maintains a list of items in the system by state and associated with an item gatherer id and that syndication uses the event log to determine what items it must send to a subscriber. When you run the wcm-reset-event-log task you are actually clearing all the entries from the event log database and telling WCM to rebuild the event log when the server restarts. This is possible because the event log database stores the current state of all the items in the system and not the actions that were performed on the items so no information is lost when the event log is reset, WCM will simply get every item in the system and add it to the event log using its current state.

Ideally the event log database should always be kept in sync with all the items in the system and never need to be reset. In practice however, sometimes an item is not added to the event log or an entry in the event log is not updated due to a defect or error in the system. This can cause the item to not be syndicated and resetting the event log will correct the problem by re-adding all items in the system in their current state.

Most Common Issues (and how to address them)

In this section we try to troubleshoot issues keeping the following in mind:

Questions to ask
Things to verify from the existing information
Additional information to request (if needed)
Actions to be taken

Automatic syndication is not happening

Symptoms Reported
Syndication not even initiating
Syndicator/subscriber connect but there is no data being transferred
Syndication starts up from the syndicator but no response from the subscriber
Syndication updates donot show up on the subscriber at all

a) On the syndicator machine
Check for 6.0.X
/wcm/shared/app/config/wcmservices/WCMConfigService.properties file
Check for 6.1.X
/PortalServer/wcm/shared/app/config/wcmservices/WCMConfigService.properties file
on syndicator machine to make sure that inittask for syndication is set to true connect.moduleconfig.syndication.inittasks=true

b) On the syndicator machine
Check for 6.0.X
\wcm\shared\app\config\wcmservices\WCMConfigService.properties file
Check for 6.1.X
\PortalServer\wcm\shared\app\config\wcmservices
on syndicator machine to make sure that subscriberOnly is NOT set to true
deployment.subscriberOnly=false

c) If a) and b) checkout ok then the issue can be one of the three:
a) EventLog is empty
b) Persistence Tables need to be cleared
c) EJBTimers are not firing

Check these using the following steps:

1. Make sure the following message logged in the syndicator SystemOut.log at startup.
SCHD0133I: Scheduler WPSTaskScheduler (wps/Scheduler) has acquired the lease and will run all tasks on this application server.
In a cluster environment, this message should log in either cluster member SystemOut.log. If this message not found in the SystemOut.log, then EJBTimer won't be able to fire the scheduled timers to send syndication update to subscriber. WAS L2 should help customer to configure scheduler for cluster environment.

2. Make sure there are no SQL exceptions in the trace. For example: com.ibm.websphere.ce.cm.DuplicateKeyException. If SQL exception found in trace use the Clear_WCM_Persistence_Service_table.jsp to clear the WCM_OBJECTS table from persistence database.
This tool is available in the IBM Support Tools Portlet.

3.Get the output of the following command and EJBTimers_nn.nn.log from the syndicator server:
$WAS_PROFILE_HOME/bin directory - ./findEJBTimers.sh WebSphere_Portal -all -username -password %WAS_PROFILE_HOME%\bin directory - findEJBTimers.bat WebSphere_Portal -all -username -password
Typically you should see something like this:

[4/30/09 19:43:25:916 CEST] 0000000a EJBTimersComm A      EJB    : wcm, WCM_EJBs.jar, EJBScheduler

[4/30/09 19:43:25:916 CEST] 0000000a EJBTimersComm A      Info   : com.ibm.workplace.wcm.util.scheduler.Schedulable|com.aptrix.syndication.business.TaskManager$CheckSessionsWork|com.aptrix.syndication.business.TaskManager$CheckSessionsWork|-5312fffa8c8dffc29c9092d19e8f8b8d9687d18c86919b969c9e8b969091d19d8a8c96919a8c8cd1ab9e8c94b29e919e989a8ddbbc979a9c94ac9a8c8c9690918ca8908d94fffffffffffffffefdffff878dffc79c9092d1969d92d188908d948f939e9c9ad1889c92d18a8b9693d18c9c979a9b8a939a8dd1be9d8c8b8d9e9c8bac9c979a9b8a939e9d939aaeaec3847c6da2cffdfff9a5fff493909198ad8a9191969198b5fff68b96929a8c8b9e928fb3fff699968d8c8bab96929a8bffefb3959e899ed08a8b9693d0bb9e8b9ac4b3ffef96918b9a8d899e93bb8a8d9e8b9690918bffefb3959e899ed0939e9198d0b3909198c4b3fffb919e929a8bffedb3959e899ed0939e9198d0ac8b8d969198c4b3fff6919e929a8c8f9e9c9a8eff81fffb878ffffffffedf08918d5c8c8dfff1959e899ed18a8b9693d1bb9e8b9a97957efeb4a68be6fcffff878f88f7fffffedf08918d5c878c8dfff1959e899ed1939e9198d1b3909198c4741b6f3370dc20fdfffeb5fffa899e938a9a878dffef959e899ed1939e9198d1b18a929d9a8d79536ae2f46b1f74fdffff878fffffffffffff8acf8bffc29c9092d19e8f8b8d9687d18c86919b969c9e8b969091d19d8a8c96919a8c8cd1ab9e8c94b29e919e989a8ddbbc979a9c94ac9a8c8c9690918ca8908d948eff81fff5

[4/30/09 19:43:25:916 CEST] 0000000a EJBTimersComm A   1 EJB Timer tasks found

If not found then try restarting Portal and run the command again.
If still not present then run cancelEJBTimers and restart Portal.

4. If there are many EJBTimer tasks scheduled, then run following command on the syndicator server
$WAS_PROFILE_HOME/bin directory - ./cancelEJBTimers.sh WebSphere_Portal -all -username -password %WAS_PROFILE_HOME%\bin directory - cancelEJBTimers.bat WebSphere_Portal -all -username -password (P.S: For ejbtimer issues enable the folowing trace: com.ibm.workplace.wcm.util.scheduler.*=all)

5. Make sure that the event log is populated with entries and is not empty. Run the displayEventLog tool from the IBM Support Tools portlet on the syndication server. Review the output of the jsp to ensure that it is not empty. If it is empty then make sure that you double check step2 on the subscriberOnly value set to true. Possibly a restart of the portal server maybe needed.

6. If the eventlog is empty and the subscriberonly is set to false and the restart doesnot fix the issue and everything above checked out okay then
a) Make sure additional traces are added on the syndicator server:
com.ibm.workplace.wcm.services.eventlog.*=finest:com.ibm.workplace.wcm.services.transaction.steps.eventlog.*=finest
(P.S: As mentioned earllier ensure that customer has set the traces in WAS and the number of historical files and file size of the trace files is adequate)
b) Reset the event log
c) Restart portal
d) Run the displayEventLog tool and check if the eventlog got populated now - if so then reattempt syndication, if not then e)
e) Collect the trace logs and the jsp output and provide it to IBM support

Syndication updates complete but with errors

These are some of the most typical errors seen in the logs:

DuplicatePathException
Refer to this technote: http://www-01.ibm.com/support/docview.wss?rs=688&uid=swg21260741
Solution is to do the following
1. Remove the failing object(s) from the Subscriber instance.
2. Then re-save the failing object on the syndicator and do an "Update", or you can simply do a "Full Rebuild" of the syndication pair.

ReferentialIntegrity Errors
This would indicate that the references for an item being saved could not be found. Typically this would mean that the an item may be referencing another item in another library and most likely that library is not getting syndicated over. Make sure to ask the customer to check if all the libraries that have items referencing each other are being syndicated.

DuplicateKeyException
com.ibm.websphere.ce.cm.DuplicateKeyException: [IBM][CLI Driver][DB2/AIX64] SQL0803N One or more values in the INSERT statement, UPDATE statement, or foreign key update caused by a DELETE statement are not valid because the primary key, unique constraint or unique index identified by "1" constrains table "JCR.EV_VERSION" from having duplicate rows for those columns.SQLSTATE=23505
Resetting of the event log should clearup this error. Also use the clear persistence table jsp available in the IBM Support tools portlet.

ItemNotFound
javax.jcr.ItemNotFoundException: A node does not exist with UUID: 60b1f58044f3bee49956b9f420fa6210 at com.ibm.icm.jcr.WorkspaceImpl.getNodeByUuid(WorkspaceImpl.java:1117)
While retrieving the an item from the syndicator database the subscriber logs throw this error is it is unable to find the item in the JCRDB of the syndicator server. This could possibly indicate that the eventlog is not in sync with the JCRDB for the syndicator. Ask the customer to reset the event log and then try the syndication again.

Parent Not Found [
[5/6/09 16:32:24:577 EDT] 00000041 PackageProces W Could not save item with id DepRef(id:16c567804590753c8bd68f0b6a6ea584 type: com.aptrix.pluto.site.SiteArea nonDraft:true draft:false purged:false parentId:6babef80458422afb958b9fb02923ee3 timeStamp:1227863963980 stateUpdate: false versions:null moved: false) because it could not find its parent.
Typically you will see this error as a symptom of the problem where the parent item or its parent could not be saved. Use the parent UUID mentioned in the string above and check the logs to see the reason for the parent failing. If a top level site or a workflow fails to be saved it will cause the failure to save several items and the above error will showup in the logs.

InvalidClassException
IWKSY1059X: detail: IWKPD1088X: Exception occurred while trying to retrieve document from item.IWKSY1060X:(Cause:java.io.InvalidClassException: com.aptrix.pluto.content.Content; local class incompatible: stream classdesc serialVersionUID = 1595762135600321749, local class serialVersionUID = -1089925949120930392)
at com.aptrix.deployment.subscriber.SerialisationTransformer.transformDocument(SerialisationTransformer.java:88)

http://www-01.ibm.com/support/docview.wss?rs=1041&uid=swg21265346

From the versioninfo.log verify the fixes installed on both the syndicator and subscriber match
If the fix levels match and we still see the error then, check for loose class files on the syndicator or subscriber system this will impact the fix levels.
Location for these loose class files:
V6.0.X: \wcm\shared\app
V6.1.X: \wcm\prereq.wcm\wcm\shared\app

Syndication performance is not good

1. Increase the default syndication frequency of 30 seconds for servers that don’t require frequent syndication. Take care not to space the intervals TOO far apart, 10 to 30 minutes is a good range.

2. Don’t enable a syndicator on the production rendering server.Set the subscriber.only property to “true” in WCMConfigServices.properties . This stops the monitoring task from running on the IWWCM instance that tracks object changes for later syndication to another server.Whenever the subscriber.only setting is changed, also reset the EventLog (\config\WPSConfig wcm-reset-event-log) before restarting the server

3. Avoid having too many Subscribers (>5) linked to the one Syndicator.Create a tiered syndication structure, e.g. the first syndicator syndicates to two servers and those servers further syndicate onto the rest.
This improves performance because you are decreasing the load on the syndicator server.

4. Consider disabling ‘Versioning’ for all items on both authoring and rendering servers.
If storing previous versions of items isn’t required, then disabling this feature can improve performance of saving and syndication. Refer to the following link in the infocenter that mentions how to achieve this:

http://publib.boulder.ibm.com/infocenter/wpdoc/v6r0/topic/com.ibm.wp.zos.doc/wcm/wcm_config_author_options.html

http://publib.boulder.ibm.com/infocenter/wcmdoc/v6r0/topic/com.ibm.lotus.wcm.doc/wcm/wcm_config_prop_authoring.html

Set the Version control options in the WCMConfigservice.properties file:
* versioningStrategy.AuthoringTemplate= never (or manual)
* versioningStrategy.Component= never (or manual)
* versioningStrategy.Content= never (or manual)
* versioningStrategy.PresentationTemplate= never (or manual)
* versioningStrategy.Site= never(or manual)
* versioningStrategy.Taxonomy= never (or manual)
* versioningStrategy.Workflow= never (or manual)
* versioningStrategy.Default= never (or manual)

Syndication changes donot show up on all cluster members

Typically a customer will report a scenario in which they are syndicating from their stand alone authoring server to a single node in a two node horizontal cluster. The syndication completes successfully but the changes are not reflected when accessing the cluster through the front end httpserver or when accessing the non syndicator node directly through its internal http transport port.

1. This is not a syndication issue but an issue with the WCM cache objects defined in WAS dynacache. A simple test to verify this is to either restart the non syndicator node ( the changes should reflect now) or use the clearcaches.jsp available here: https://w3.tap.ibm.com/w3ki08/display/wcm/Tools

2. A technote that helps in such a scenario:
http://www-01.ibm.com/support/docview.wss?rs=688&uid=swg21304020

3. This could be a WAS dynacache settings issue however if the customer is still reporting issues after following the technote, this needs to be investigated as a possible WCM cache issue.

Syndication of a particular item(s)is not happening

Sometimes in an already running syndication setup. Customer will report that when they editing and re-saving an existing content itemon the syndicator, it doesnot showup on the subscriber side. Ask the customer to do the following:

1.Use the IBM Support Tools Portlet to determine the UUID for the item that is not getting syndicated.

2.Use the displayEventlog from the portlet and get a screen capture of the output

3.Re-edit the item and re-save the item

4.Use the displayEventlog tool and get a screen capture of the output

5.Check between the screencaptures and see if the UUID got added to the eventlog or not - A lot of the times just resaving the item corrects the issue.

6. If it has not been added then there might be an issue with eventlog
Questions to ask:
6a. Does this happen for an item in a particular library ?
6b. If so, then does this happen for all items in that library ?
6c. If so, is this a migrated library ?
To resolve this do one of the following:
6.1) Make sure additional traces are added on the syndicator:
com.ibm.workplace.wcm.services.eventlog.*=finest:com.ibm.workplace.wcm.services.transaction.steps.eventlog.*=finest
6.2) Reset the event log
6.3) Restart portal ( please note - restarting portal after restting event log may take sometime doesnot mean that the portal is hung)
6.4) Use the displayEventLog tool to check if the eventlog got populated now with the UUID of the item - if so then retry syndication, If not, then 6.5)
6.5) Attempt the following steps for a new item with the above mentioned traces in place:
1. Create a new item in the library being syndicated.
2. Use the IBM Support Tools portlet to determine the UUID for this item
3. Use the displayEventlog tool and get a screen capture of the output - does this item get added to the eventlog ?
6.6) Collect the trace logs, review the traces and the jsp output and consult with IBM
P.S: In scenarios such as above always try the syndication with a new item to ensure its not content/library specific issue.

7.If the item is added to the eventlog, then attempt the syndication with the MustGather traces enabled. Review the subscriber logs to see the cause of the failure. Typically the item would have failed to be processed by the subscriber due to the syndication errors already covered in the Section "Common Issues->Syndication updates complete but with errors"

Syndication completes but not all items showup on the subscriber side or Need to verify if all items on the syndicator match with all items on the subscriber:

1.In this scenario, please consult IBM Support for a LibraryExport.jsp and run it against the syndicator and subscriber nodes. This will generate a CSV file each on the syndicator and subscriber servers.

2.IBM support will compare the outputs of these and identify if certain items are missing.

3.Use troubleshooting techniques discussed so far to determine why certain items didnot make it through the syndication. That is based on the difference identify the missing UUIDs. Review the logs and see the errors associated with the syndication of these UUIDs. If no errors in the logs then see the eventlog to determine if it got added to event log or not. And if not then enable traces to determine why.

4.Keep in mind a large number of syndication issues can be identified by comparing the LibraryExport output from both syndicator and subscriber along with the displayEventLog output on the syndicator side.

Syndication and Clustering

A clustered environment typically consists of several nodes and a load balancer but can include more complex relationships between several clusters with multiple levels of load distribution as shown in figure 2 below. It is therefore important to understand the topology of the environment before making any decisions about how to configure syndication.

The term load balancer is often used loosely to mean any system which distributes load to multiple sources. To prevent confusion, we will use the term load balancer to mean a system which distributes load across several nodes in a single cluster and IP sprayer to mean a system which distributes load across several clusters as shown in figure 2 above.
Configuring syndication in a clustered environment

When configuring syndication it is best to keep the ultimate goal in mind, which is to update the JCR repository of a subscribing environment with changes from the JCR repository of a syndicating environment. This means that for complex setups like the one shown above where you have more than one JCR repository to update in what is conceptually a single subscription environment, you need to treat each cluster as a separate environment and create multiple syndication pairs. For this reason IP sprayers should never be used in configuring syndication. In other words, the syndicator and/or subscriber URLs should never point to a system that distributes load between multiple servers or clusters with different JCR repositories.

The remainder of this section will describe configuring syndication between the single authoring cluster and one cluster in the rendering environment. This process should be repeated to achieve syndication to the other rendering cluster.

The syndicator and subscriber objects can be created by accessing the interface via the load balancer or any node directly. Because all nodes in a cluster share the same repository, it doesn't matter which sends (for a syndicator) or receives and processes (for a subscriber) the syndication request. This means that the syndication URLs can be configured to/from any single node in the cluster, however it is recommended that it be configured to/from the load balancer to facilitate fail over should a node go down. The syndication URLs can be set to any specific node or the load balancer irrespective of how you accessed the system to create the objects.

Syndication scheduling in a clustered environment

Automatic syndication it initiated by the scheduler service which runs on each node in a clustered environment. Each scheduler competes for a lease which is only granted to one node in the cluster. The node that obtains the lease is then responsible for running all tasks in the system until the lease is released and granted to another node (see the "High availability" section at http://www.ibm.com/developerworks/websphere/techjournal/0404_johnson/0404_johnson.htmlfor more information).

This means that any node in the cluster could initiate automatic syndication which might be confusing if you have configured the syndication URLs to a specific node in the cluster. The important thing to realise here is that the syndication URLs do not restrict which nodes are used to perform syndication but simply indicate how each environment should contact the other.

For example, the syndication pair may be configured between cluster 1, node 1 (C1N1) and cluster 2, node 3 (C2N3) but the scheduler may initiate syndication on cluster 1, node 2 (C1N2). C1N2 will contact the subscriber using the configured URL - C2N3 which will receive and process the initial packet. C2N3 will then use the configured syndicator URL - C1N1 to request updated items. Because all syndication operations happen as individual HTTP request/responses, it is ok for the process to switch between nodes in this fashion. Interestingly this is also why it is ok to configure syndication to a load balancer which may distribute requests for updated items across different nodes in the cluster.