ShowTable of Contents
Understanding the syndication environment
First, answer the following questions to help you understand your syndication environment:
- Is the syndication being done for the first time to the subscriber?
- Is the syndication being done between one syndicator and one Subscriber, or multiples? If multiples, obtain details about the topology of the syndication setup, including the number of syndication servers, the number of syndicator/subscriber objects on a server, and the flow of syndication.
- Is either the syndicator or the subscriber a part of a cluster? If so, is syndication being done from/to one node on a cluster or the entire cluster through httpserver?
- Is the syndication being done for one library or multiple libraries? If multiple, do the items in one library reference the others?
- Is the syndication being done for All items or All Live Items?
- Is the syndication automatic or manual?
- Are the syndicator and subscriber pointing to the same LDAP? If different LDAPs, then are the users and groups identical?
- Are the syndicator(s) and subscriber(s) at the same fix level?
- Has any authoring been done on the subscriber?
- If doing All items syndication, is versioning enabled on the syndicator?
- Did syndication ever work completely?
- Are any of the libraries being syndicated, for example, Migrated libraries; and if so, from which version?
Also, you should collect the syndication MustGather information as described in either Support Technote 1243980, "MustGather: Web Content Management (WCM) v6 syndication issues
," for V6.x or 1316241, "MustGather: Web Content Management (WCM) v6.1 Syndication
" for V6.1.x.
Finally, capture a screenshot of the syndicator and subscriber objects.
Verify the environment and configuration
From the MustGather files
From the versioninfo.log file, verify that both syndicator and subscriber have exactly the same WCM version and WCM fixes installed. The syndicator and subscriber systems must match with regard to WCM version and fix level.
Check for loose class files on the syndicator and subscriber system as this will impact the fix levels. These loose class files are located in:
Check for the properties listed below in the syndicator and subscriber's WCMConfigService.properties file, located in:
V6.0.X: \PortalServer>\wcm\shared\app\config\wcmservices\WCMConfigService.properties file
V6.1.X: \Profile Root>\PortalServer\wcm\shared\app\config\wcmservices
- deployment.subscriberOnly=false (this should never be set to true on the syndicator)
- deployment.itemChangedTaskDelay=30 (in seconds; this is the default value)
Items to verify from the screenshots
- Name of the syndicator. Make note of this as it will be referenced in the logs.
- Name of the subscriber. Make note of this as it will be referenced in the logs.
- Subscriber ID. This is ID of the subscriber, used to set up the subscription. Make sure this matches the ID on the subscriber object.
- Subscriber URL. This is the URL of the subscriber. It should be a fully qualified URL (in some cases, an IP address) and should match the default value on the subscriber object. (Note that this value reflects the values of the variables WCM_HOST and WCM_PORT defined in IBM WebSphere Application Server. These variables can be overridden by hardcoding the values in the WCMConfigService.properties file)
- Libraries selected for syndication.
- Configuration of the item gatherer. Determine whether it's All items or All Live items syndication.
- Enable the syndicator. The checkbox should be checked.
- Name of the subscriber.
- Name of the syndicator.
- The ID of the syndicator used to set up the subscription.
- Syndicator URL.
- Enable the subscriber.
Facts you should know about syndication
Syndication is merely transferring content items stored in the Java Content Repository (JCR) repository of the syndicator server to the JCR repository of the subscriber server. The UUIDs of each of these items is maintained the same across the syndicator and subscriber databases. Simply, you can think of this as the syndicator and subscriber being sort of gatekeepers to the JCRDB that stores the WCM content and processes these items through syndication, as deemed necessary by the Event log.
Copying the JCR repository itself from the syndicator to subscriber is a brute force method of ensuring that all the data on the syndicator shows up on the subscriber. However, consider the following points when doing so:
- This will impact Portal Document Manager (PDM) / Personalization (PZN) as well, since they use the same JCRDB.
- We suggest doing this ONLY if it's a first time syndication.
- Refer to the WCM wiki article, "Procedure for cloning WCM repositories," for more details on this topic.
Syndication uses the HTTP protocol and does not support the use of HTTPS (although it may work technically as long as the certificates are set up correctly between the servers).
General Event log information
Resetting the Event log causes a FULL REBUILD (if automatic syndication is enabled or if manual syndication is initiated). Once the Event log is created, any content item saved to the JCR repository for a library is logged to the Event log as one or more entries, depending on its state and other settings. For example, a live item is logged twice, once against the Live Item gatherer and again against the All Items gatherer. (There are also additional entries for versions.) The Event log provides critical information to determine whether an object is getting syndicated. More details on this information are provided in "EventLog and syndication" section below.
The IBM Support Tools portlet for Lotus WCM
has some useful tools you can download (after registering with the site) to help in troubleshooting syndication, including:
Authoring on the subscriber side can cause issues with syndication because the last modified date on the item on the subscriber will change. As a result, when the subscriber processes this item being pushed from the syndicator, it might skip saving this item because it has a current or newer version of the item in its database.
NOTE: Authoring on the subscriber side can be OK, if you are aware of the limitations and manage them properly, such as in two-way syndication scenarios in which authoring is done on both servers but in different libraries.
Syndication setup example
A typical syndication setup is as follows:
For the topology, the Syndicator (Authoring server, on which the WCM Authoring portlet is installed) = one way syndication => Subscriber (Rendering server, on which the WCM Local Rendering portlet is installed) displays the content; probably does not have the Authoring portlet installed).
When syndicating two libraries, one content and one design, the content library has the content items, whereas the design library has the AT, PT, components, etc. Typically the content library will reference the items in the design library, and both libraries must be syndicated simultaneously for the syndication to succeed.
Typically both libraries are added to the syndicator in Live items scope; in this case, it make sense to syndicate Live items only, since there is no authoring being done on the rendering server side, and it is being used only to render content (LRP/RRP only displays live content). So if you are doing All items syndication and you have a setup similar this example, you should consider doing Live items.
Both subscription partners should share the same user repository (such as LDAP), if they use automatic syndication. If this is not the case, disable automatic syndication, perform manual syndication, and then execute memberfixer. Note that the purpose of automatic syndication is defeated if the LDAPs are not the same or, at the very least, the users and groups are not identical in both LDAPs with identical DNs.
Key traces to be enabled (on both
syndicator and subscriber) are:
For Event-log-related issues:
(The Event log trace will show how and if an item was added to the event log)
Syndication can handle the syndicator and subscriber servers being in different time zones, as long as the server clocks for both are not more than a minute apart, with the time-zone differences factored in. This rule also applies for servers in the same time zone.
Event log resets and traces
Items that were deleted and then purged will not be syndicated if the event log is reset. For example, if an item was deleted and purged on the syndicator side, the entry is recorded in the Event log so that the same action can be performed on the subscriber side upon syndication. However, if the Event log is reset prior to the syndication, then the rebuilt Event log will not have this entry; hence, the item will not be syndicated.
Before resetting the Event log, make sure that these Event log traces are applied:
Enable Event log traces from the WebSphere Application Server Admin console in Config mode, save the configuration, stop the WebSphere Portal server, and then reset Event log. Be sure to remove
traces from the WebSphere Application Server Admin console in Config and runtime mode, after Event log rebuild is complete and the server has started up. The server startup with the Event log traces set and the Event log being built provides vital information.
- If the repository (library) is large (approximately in the range of 20,000---30,000 items), then you should increase the number of historical trace files and trace file sizes in the WebSphere Application Server Admin console. A setting of 20 historical files and a file size of 50 MB should be adequate for most scenarios. The idea is to ensure that trace information is not lost due to file rollovers.
- Use the DisplayNodes tool included in IBM Support Tools Portlet for WCM to collect the output for any failing UUID. The displayNodes.jsp output will show all the references to that UUID, which can be useful in certain scenarios.
- Library access permissions are not syndicated. If the library does not exist on the subscriber, it will be created during syndication, but no access control settings are specified on the new library. On Rendering servers you might want to specify different access control settings, for example, to disallow content entry for all users.
- On the syndicator, the PackageGenerator class generates the packages to send to the subscriber; on the subscriber, the PackageProcessor class processes packages sent by the syndicator.
- When Event log is rebuilt, it processes every WCM library (enabled), regardless of whether or not it's in a syndication definition. So, depending on how many WCM libraries (enabled) there are, the rebuild time can be quite long. Disabled libraries are not processed and not added to Event log.
- Syndication cannot be scheduled, but the frequency of syndication occurrence can be set. By default it is set for 30 seconds but you can change it by changing the value deployment.itemChangedTaskDelay in the WCMConfigService.properties file. Also, you can control the occurrence of syndication in certain scenarios by using the publish date of content; the publishing of the content can be delayed and hence syndication can be delayed. This can be used for, example, to allow syndication to occur only during non-peak hours.
- Before running the configuration task wcm-reset-event-log, stop the WebSphere Portal server and then restart it when the task has completed. The Config task merely clears the Event log DB; the Event log is rebuilt when the WebSphere Portal server is started and may take a while to come up. To address this delay, a module was introduced to reset the Event log DB while the WebSphere Portal server is running. This fix is available in WCM Cumulative iFix 21(PK90387) for 6.0.1.x and WCM cumulative iFix 15 (PK90388) for 6.1.x. Refer to Support Technote #1396691, "FAQs related to the reset event log module introduced with APAR PK88848."
- A common misconception is that Live Items syndication means syndicating only published items. In fact, the Live Items syndication includes published items, expired items, and non-workflowed items (but not drafts, versions, and deleted items).
Reading the traces
1.Most activity occurs on the subscriber side since that is where the subscriber will go through the list of items to be saved, fetch these items from the syndicator database, add them to the repository and then save the document in its own repository. Basically here is what happens in very very layman's terms (this maynot be technically very accurate but the idea is to give a mental picture of the syndication functioning in the most common scenario): On the syndicator side the itemchangedtaskdelay value determines when the system will check for updates to the event log. If it finds some updated items it will then contact the subscriber and provide it with the list of items to be updated. The subscriber then starts processing each item in this list. It skips items that are already at the most current version (Note: you will see the statement -"skipping item" in the logs) and it adds the newer modified items to the items that need to be updated. After doing that it will fetch theses items from the syndicator server's database and then try to save these documents( items) on the subscriber JCRDB. Most syndication errors are seen during this time. If the subscriber is able to process all the items sent by the syndicator it will send the syndicator a 603 message code on receipt. If for some reason there are some items that failed to be processed by the subscribers those items will be processed the next time the itemgatherer queries the eventlog.
2. New items are saved first, followed by updates and lastly by removals
3. Typically you will see a message at the end of the syndication in the subscriber trace logs such as this
[6/4/08 22:20:02:141 EDT] 00000084 PackageProces I ****************** Start of subscription update report ******************
[6/4/08 22:20:02:156 EDT] 00000084 PackageProces I Subscription update started: 2008-06-04T22:19:17.125
[6/4/08 22:20:02:156 EDT] 00000084 PackageProces I Subscription update finished: 2008-06-04T22:20:02.141 (45016 milliseconds)
[6/4/08 22:20:02:172 EDT] 00000084 PackageProces I Syndicator: syndicator6_to_subscriber4, URL=http://testsyndication1.raleigh.ibm.com:10038/wps/wcm/connect/?MOD=Synd
[6/4/08 22:20:02:172 EDT] 00000084 PackageProces I Subscriber: subscriber4_from_syndicator6, URL=http://testsyndication2.raleigh.ibm.com:10038/wps/wcm/connect/?MOD=Subs
[6/4/08 22:20:02:172 EDT] 00000084 PackageProces I Full update
[6/4/08 22:20:02:188 EDT] 00000084 PackageProces I Updates sent: 21
[6/4/08 22:20:02:188 EDT] 00000084 PackageProces I Removes sent: 0
[6/4/08 22:20:02:203 EDT] 00000084 PackageProces I Items up to date: 20
[6/4/08 22:20:02:203 EDT] 00000084 PackageProces I No errors were detected during this syndication.
[6/4/08 22:20:02:219 EDT] 00000084 PackageProces I
High Level Details
[6/4/08 22:20:02:234 EDT] 00000084 PackageProces I Start of library updates, time=2008-06-04T22:20:02.109, count=1
[6/4/08 22:20:02:250 EDT] 00000084 PackageProces I Total items: 1
[6/4/08 22:20:02:250 EDT] 00000084 PackageProces I End of library updates, time=2008-06-04T22:20:02.109, elapsed=0 milliseconds
[6/4/08 22:20:02:250 EDT] 00000084 PackageProces I Start of draft item removal, time=2008-06-04T22:20:02.109, count=0
[6/4/08 22:20:02:266 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:266 EDT] 00000084 PackageProces I End of draft item removal, time=2008-06-04T22:20:02.109, elapsed=0 milliseconds
[6/4/08 22:20:02:281 EDT] 00000084 PackageProces I Start of draft item removal, time=2008-06-04T22:20:02.109, count=0
[6/4/08 22:20:02:281 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:281 EDT] 00000084 PackageProces I End of draft item removal, time=2008-06-04T22:20:02.109, elapsed=0 milliseconds
[6/4/08 22:20:02:297 EDT] 00000084 PackageProces I Start of new published items, time=2008-06-04T22:20:02.109, count=0
[6/4/08 22:20:02:297 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:312 EDT] 00000084 PackageProces I End of new published items, time=2008-06-04T22:20:02.109, elapsed=0 milliseconds
[6/4/08 22:20:02:312 EDT] 00000084 PackageProces I Start of update published items, time=2008-06-04T22:20:02.109, count=0
[6/4/08 22:20:02:312 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:328 EDT] 00000084 PackageProces I End of update published items, time=2008-06-04T22:20:02.109, elapsed=0 milliseconds
[6/4/08 22:20:02:328 EDT] 00000084 PackageProces I Start of new draft items, time=2008-06-04T22:20:02.125, count=0
[6/4/08 22:20:02:344 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:344 EDT] 00000084 PackageProces I End of new draft items, time=2008-06-04T22:20:02.125, elapsed=0 milliseconds
[6/4/08 22:20:02:344 EDT] 00000084 PackageProces I Start of update draft items, time=2008-06-04T22:20:02.125, count=0
[6/4/08 22:20:02:359 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:359 EDT] 00000084 PackageProces I End of update draft items, time=2008-06-04T22:20:02.125, elapsed=0 milliseconds
[6/4/08 22:20:02:375 EDT] 00000084 PackageProces I Start of item removal, time=2008-06-04T22:20:02.125, count=0
[6/4/08 22:20:02:375 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:375 EDT] 00000084 PackageProces I End of item removal, time=2008-06-04T22:20:02.125, elapsed=0 milliseconds
[6/4/08 22:20:02:391 EDT] 00000084 PackageProces I Start of deferred new published content links, time=2008-06-04T22:20:02.125, count=0
[6/4/08 22:20:02:391 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:391 EDT] 00000084 PackageProces I End of deferred new published content links, time=2008-06-04T22:20:02.125, elapsed=0 milliseconds
[6/4/08 22:20:02:406 EDT] 00000084 PackageProces I Start of deferred new draft content links, time=2008-06-04T22:20:02.125, count=0
[6/4/08 22:20:02:406 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:422 EDT] 00000084 PackageProces I End of deferred new draft content links, time=2008-06-04T22:20:02.125, elapsed=0 milliseconds
[6/4/08 22:20:02:438 EDT] 00000084 PackageProces I Start of resolving references, time=2008-06-04T22:20:02.125, count=0
[6/4/08 22:20:02:438 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:438 EDT] 00000084 PackageProces I End of resolving references, time=2008-06-04T22:20:02.125, elapsed=0 milliseconds
[6/4/08 22:20:02:453 EDT] 00000084 PackageProces I Start of item renaming, time=2008-06-04T22:20:02.141, count=0
[6/4/08 22:20:02:453 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:469 EDT] 00000084 PackageProces I End of item renaming, time=2008-06-04T22:20:02.141, elapsed=0 milliseconds
[6/4/08 22:20:02:469 EDT] 00000084 PackageProces I Start of create versions, time=2008-06-04T22:20:02.141, count=0
[6/4/08 22:20:02:469 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:484 EDT] 00000084 PackageProces I End of create versions, time=2008-06-04T22:20:02.141, elapsed=0 milliseconds
[6/4/08 22:20:02:484 EDT] 00000084 PackageProces I Start of update versions, time=2008-06-04T22:20:02.141, count=0
[6/4/08 22:20:02:500 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:500 EDT] 00000084 PackageProces I End of update versions, time=2008-06-04T22:20:02.141, elapsed=0 milliseconds
[6/4/08 22:20:02:500 EDT] 00000084 PackageProces I Start of version removal, time=2008-06-04T22:20:02.141, count=0
[6/4/08 22:20:02:516 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:516 EDT] 00000084 PackageProces I End of version removal, time=2008-06-04T22:20:02.141, elapsed=0 milliseconds
[6/4/08 22:20:02:531 EDT] 00000084 PackageProces I Start of item status update, time=2008-06-04T22:20:02.141, count=0
[6/4/08 22:20:02:531 EDT] 00000084 PackageProces I Total items: 0
[6/4/08 22:20:02:531 EDT] 00000084 PackageProces I End of item status update, time=2008-06-04T22:20:02.141, elapsed=0 milliseconds
[6/4/08 22:20:02:547 EDT] 00000084 PackageProces I
Low Level Details
[6/4/08 22:20:02:547 EDT] 00000084 PackageProces I Start of library updates, time=2008-06-04T22:20:02.109, count=1 [DepRef(id:409ae28049c897ccab84bf55786cef3c type: com.ibm.workplace.wcm.services.library.Library nonDraft:true draft:false purged:false parentId:409ae28049c897ccab84bf55786cef3c timeStamp:1211200237696 stateUpdate: false versions:null)]
[6/4/08 22:20:02:562 EDT] 00000084 PackageProces I Start of draft item removal, time=2008-06-04T22:20:02.109, count=0 
[6/4/08 22:20:02:562 EDT] 00000084 PackageProces I Start of draft item removal, time=2008-06-04T22:20:02.109, count=0 
[6/4/08 22:20:02:578 EDT] 00000084 PackageProces I Start of new published items, time=2008-06-04T22:20:02.109, count=0 
[6/4/08 22:20:02:578 EDT] 00000084 PackageProces I Start of update published items, time=2008-06-04T22:20:02.109, count=0 
[6/4/08 22:20:02:578 EDT] 00000084 PackageProces I Start of new draft items, time=2008-06-04T22:20:02.125, count=0 
[6/4/08 22:20:02:594 EDT] 00000084 PackageProces I Start of update draft items, time=2008-06-04T22:20:02.125, count=0 
[6/4/08 22:20:02:609 EDT] 00000084 PackageProces I Start of item removal, time=2008-06-04T22:20:02.125, count=0 
[6/4/08 22:20:02:609 EDT] 00000084 PackageProces I Start of deferred new published content links, time=2008-06-04T22:20:02.125, count=0 
[6/4/08 22:20:02:609 EDT] 00000084 PackageProces I Start of deferred new draft content links, time=2008-06-04T22:20:02.125, count=0 
[6/4/08 22:20:02:625 EDT] 00000084 PackageProces I Start of resolving references, time=2008-06-04T22:20:02.125, count=0 
[6/4/08 22:20:02:625 EDT] 00000084 PackageProces I Start of item renaming, time=2008-06-04T22:20:02.141, count=0 
[6/4/08 22:20:02:641 EDT] 00000084 PackageProces I Start of create versions, time=2008-06-04T22:20:02.141, count=0 
[6/4/08 22:20:02:641 EDT] 00000084 PackageProces I Start of update versions, time=2008-06-04T22:20:02.141, count=0 
[6/4/08 22:20:02:641 EDT] 00000084 PackageProces I Start of version removal, time=2008-06-04T22:20:02.141, count=0 
[6/4/08 22:20:02:656 EDT] 00000084 PackageProces I Start of item status update, time=2008-06-04T22:20:02.141, count=0 
[6/4/08 22:20:02:656 EDT] 00000084 PackageProces I ******************* End of subscription update report *******************
which provides the subscription summary
4.Sometimes instead of the 603 -" no more confirmations to send" the subscriber responds back to the syndicator with a 200 message code where you will see the phrase "OK" being used. This phrase indicates that the subscriber is still processing the updates that the syndicator sent in earlier and doesnot wish to process any new updates. This will happen when say the syndicator sends in a large number of updates, the subscriber is processing each one of them ( and this is taking time) meanwhile the itemchangedtaskdelay which is set to 30 seconds causes the syndicator to contact the subscriber again for the eventlog updates.
5.Also while the subscriber is processing the updates sent by the syndicator the status of the subscriber will showup as active while the syndicator will showup as idle for most part.
6.To read the traces, start with the syndicator log to identify the syndication initiation point and co-relate the corresponding message on the subscriber systemout.log. Once identified, use the systemout.log to verify where the error occurs ( it will be easier to scan through the systemout.log than the trace.log to find this). Once identified then find the corresponding entry in the trace.log to see if additional information is available.
7.As mentioned the majority of the activity in the most common scenarios occurs on the subscriber side. Hence the subscriber logs typically are bigger than the syndicator ones. And the errors will most probably showup in the subscriber logs.
8.Edit the WCMConfigService.properties file and change the deployment.enableReport setting to "true". This enables high level reporting of syndication to the SystemOut log on the subscriber server. It provides a summary of items that were processed, and which items failed syndication to help with troubleshooting syndication issues.
Eventlog database and syndication
First of all syndication has changed dramatically in version v6 compared to v5 and although some parts use the same names (the item gatherers for example), they are completely different to what they were in previous releases.
Syndication begins at a library level. Each WCM library is associated with two item gatherers: the "All Items" gatherer and the "Live Items" gatherer. These item gatherers are unique for every library and provide the means to track items by status in specific libraries. Unlike in previous releases, the item gatherers do NOT contain any information about the items, they are simply a label used to associate a syndicator object with the items it must syndicate.
When an item is created or modified, the state of the item is updated in the event log database. It is important to understand that the event log stores the STATE of items, NOT the actions that were performed on them. To illustrate, let's look at the following example:
If you create a new content document, an entry similar to the following will be added to the event log:
If you create a new content document, an entry similar to the following will be added to the event log:
ITEMID: The ID of the content document.
TYPENAME: The class type of the content document.
ITEMTYPE: Whether the content document is published (P) or draft (D)
STATE: A number representing the current state of the content document.
DELETED: Whether the content document has been deleted (Y), purged (P) or moved (M).
PARENTID: The ID of the content document's parent node (i.e. a sitearea id).
TSTAMP: The timestamp of when the content document was last modified.
ITEMGATHERERID: The ID of the appropriate item gatherer (All or Live) associated with the content document's library.
When the content document is modified, the entry in the event log is updated to reflect the updated state of the item, for example:
Syndication uses the information in the event log to determine what items it must send to a subscriber. When you create a syndicator object you will select a library and a scope (All items or Live item). This process will associate the syndicator object with a particular ITEMGATHERERID. When syndication occurs, the syndicator will obtain all the items in the event log that match it's associated item gatherer and send them to the subscriber to be processed. The full process will use the STATE, DELETED flag and other information in the event log to determine what action to perform on the subscriber.
Basically the event log maintains a list of items in the system by state and associated with an item gatherer id and that syndication uses the event log to determine what items it must send to a subscriber. When you run the wcm-reset-event-log task you are actually clearing all the entries from the event log database and telling WCM to rebuild the event log when the server restarts. This is possible because the event log database stores the current state of all the items in the system and not the actions that were performed on the items so no information is lost when the event log is reset, WCM will simply get every item in the system and add it to the event log using its current state.
Ideally the event log database should always be kept in sync with all the items in the system and never need to be reset. In practice however, sometimes an item is not added to the event log or an entry in the event log is not updated due to a defect or error in the system. This can cause the item to not be syndicated and resetting the event log will correct the problem by re-adding all items in the system in their current state.
Most Common Issues (and how to address them)
In this section we try to troubleshoot issues keeping the following in mind:
Questions to ask
Things to verify from the existing information
Additional information to request (if needed)
Actions to be taken
Automatic syndication is not happening
Syndication not even initiating
Syndicator/subscriber connect but there is no data being transferred
Syndication starts up from the syndicator but no response from the subscriber
Syndication updates donot show up on the subscriber at all
a) On the syndicator machine
Check for 6.0.X
Check for 6.1.X
on syndicator machine to make sure that inittask for syndication is set to true connect.moduleconfig.syndication.inittasks=true
b) On the syndicator machine
Check for 6.0.X
Check for 6.1.X
on syndicator machine to make sure that subscriberOnly is NOT set to true
c) If a) and b) checkout ok then the issue can be one of the three:
a) EventLog is empty
b) Persistence Tables need to be cleared
c) EJBTimers are not firing
Check these using the following steps:
1. Make sure the following message logged in the syndicator SystemOut.log at startup.
SCHD0133I: Scheduler WPSTaskScheduler (wps/Scheduler) has acquired the lease and will run all tasks on this application server.
In a cluster environment, this message should log in either cluster member SystemOut.log. If this message not found in the SystemOut.log, then EJBTimer won't be able to fire the scheduled timers to send syndication update to subscriber. WebSphere Application Server support can help to configure scheduler for cluster environment.
2. Make sure there are no SQL exceptions in the trace. For example: com.ibm.websphere.ce.cm.DuplicateKeyException. If SQL exception found in trace use the Clear_WCM_Persistence_Service_table.jsp to clear the WCM_OBJECTS table from persistence database.
This tool is available in the IBM Support Tools Portlet.
3.Get the output of the following command and EJBTimers_nn.nn.log from the syndicator server:
$WAS_PROFILE_HOME/bin directory - ./findEJBTimers.sh WebSphere_Portal -all -username -password
%WAS_PROFILE_HOME%\bin directory - findEJBTimers.bat WebSphere_Portal -all -username -password
Typically you should see something like this:
[4/30/09 19:43:25:916 CEST] 0000000a EJBTimersComm A EJB : wcm, WCM_EJBs.jar, EJBScheduler
[4/30/09 19:43:25:916 CEST] 0000000a EJBTimersComm A Info : com.ibm.workplace.wcm.util.scheduler.Schedulable|com.aptrix.syndication.business.TaskManager$CheckSessionsWork|com.aptrix.syndication.business.
[4/30/09 19:43:25:916 CEST] 0000000a EJBTimersComm A 1 EJB Timer tasks found
If not found then try restarting Portal and run the command again.
If still not present then run cancelEJBTimers and restart Portal.
4. If there are many EJBTimer tasks scheduled, then run following command on the syndicator server
$WAS_PROFILE_HOME/bin directory - ./cancelEJBTimers.sh WebSphere_Portal -all -username -password
%WAS_PROFILE_HOME%\bin directory - cancelEJBTimers.bat WebSphere_Portal -all -username -password
(P.S: For ejbtimer issues enable the folowing trace: com.ibm.workplace.wcm.util.scheduler.*=all)
5. Make sure that the event log is populated with entries and is not empty. Run the displayEventLog tool from the IBM Support Tools portlet on the syndication server. Review the output of the jsp to ensure that it is not empty. If it is empty then make sure that you double check step2 on the subscriberOnly value set to true. Possibly a restart of the portal server maybe needed.
6. If the eventlog is empty and the subscriberonly is set to false and the restart doesnot fix the issue and everything above checked out okay then
a) Make sure additional traces are added on the syndicator server:
(P.S: As mentioned earlier ensure that the traces in WebSphere Application Server are set and the number of historical files and file size of the trace files is adequate)
b) Reset the event log
c) Restart portal
d) Run the displayEventLog tool and check if the eventlog got populated now - if so then reattempt syndication, if not then e)
e) Collect the trace logs and the jsp output and provide it to IBM support
Syndication updates complete but with errors
These are some of the most typical errors seen in the logs:
Refer to this technote: http://www-01.ibm.com/support/docview.wss?rs=688&uid=swg21260741
Solution is to do the following
1. Remove the failing object(s) from the Subscriber instance.
2. Then re-save the failing object on the syndicator and do an "Update", or you can simply do a "Full Rebuild" of the syndication pair.
This would indicate that the references for an item being saved could not be found. Typically this would mean that the an item may be referencing another item in another library and most likely that library is not getting syndicated over. Make sure to check if all the libraries that have items referencing each other are being syndicated.
com.ibm.websphere.ce.cm.DuplicateKeyException: [IBM][CLI Driver][DB2/AIX64] SQL0803N One or more values in the INSERT statement, UPDATE statement, or foreign key update caused by a DELETE statement are not valid because the primary key, unique constraint or unique index identified by "1" constrains table "JCR.EV_VERSION" from having duplicate rows for those columns.SQLSTATE=23505
Resetting of the event log should clearup this error. Also use the clear persistence table jsp available in the IBM Support tools portlet.
javax.jcr.ItemNotFoundException: A node does not exist with UUID: 60b1f58044f3bee49956b9f420fa6210 at com.ibm.icm.jcr.WorkspaceImpl.getNodeByUuid(WorkspaceImpl.java:1117)
While retrieving the an item from the syndicator database the subscriber logs throw this error is it is unable to find the item in the JCRDB of the syndicator server. This could possibly indicate that the eventlog is not in sync with the JCRDB for the syndicator.Reset the event log and then try the syndication again.
Parent Not Found [
[5/6/09 16:32:24:577 EDT] 00000041 PackageProces W Could not save item with id DepRef(id:16c567804590753c8bd68f0b6a6ea584 type: com.aptrix.pluto.site.SiteArea nonDraft:true draft:false purged:false parentId:6babef80458422afb958b9fb02923ee3 timeStamp:1227863963980 stateUpdate: false versions:null moved: false) because it could not find its parent.
Typically you will see this error as a symptom of the problem where the parent item or its parent could not be saved. Use the parent UUID mentioned in the string above and check the logs to see the reason for the parent failing. If a top level site or a workflow fails to be saved it will cause the failure to save several items and the above error will showup in the logs.
IWKSY1059X: detail: IWKPD1088X: Exception occurred while trying to retrieve document from item.IWKSY1060X:(Cause:java.io.InvalidClassException: com.aptrix.pluto.content.Content; local class incompatible: stream classdesc serialVersionUID = 1595762135600321749, local class serialVersionUID = -1089925949120930392)
From the versioninfo.log verify the fixes installed on both the syndicator and subscriber match
If the fix levels match and we still see the error then, check for loose class files on the syndicator or subscriber system this will impact the fix levels.
Location for these loose class files:
Syndication performance is not good
1. Increase the default syndication frequency of 30 seconds for servers that don’t require frequent syndication. Take care not to space the intervals TOO far apart, 10 to 30 minutes is a good range.
2. Don’t enable a syndicator on the production rendering server.Set the subscriber.only property to “true” in WCMConfigServices.properties . This stops the monitoring task from running on the IWWCM instance that tracks object changes for later syndication to another server.Whenever the subscriber.only setting is changed, also reset the EventLog (\config\WPSConfig wcm-reset-event-log) before restarting the server
3. Avoid having too many Subscribers (>5) linked to the one Syndicator. Create a tiered syndication structure, e.g. the first syndicator syndicates to two servers and those servers further syndicate onto the rest.
This improves performance because you are decreasing the load on the syndicator server.
4. Consider disabling ‘Versioning’ for all items on both authoring and rendering servers.
If storing previous versions of items isn’t required, then disabling this feature can improve performance of saving and syndication. Refer to the following link in the infocenter that mentions how to achieve this:
Set the Version control options in the WCMConfigservice.properties file:
* versioningStrategy.AuthoringTemplate= never (or manual)
* versioningStrategy.Component= never (or manual)
* versioningStrategy.Content= never (or manual)
* versioningStrategy.PresentationTemplate= never (or manual)
* versioningStrategy.Site= never(or manual)
* versioningStrategy.Taxonomy= never (or manual)
* versioningStrategy.Workflow= never (or manual)
* versioningStrategy.Default= never (or manual)
Syndication changes do not show up on all cluster members
Typically the scenario reported is syndication from a stand alone authoring server to a single node in a two node horizontal cluster. The syndication completes successfully but the changes are not reflected when accessing the cluster through the front end httpserver or when accessing the non syndicator node directly through its internal http transport port.
1. This is not a syndication issue but an issue with the WCM cache objects defined in WAS dynacache. A simple test to verify this is to either restart the non syndicator node ( the changes should reflect now) or use the clearcaches.jsp available here: https://w3.tap.ibm.com/w3ki08/display/wcm/Tools
2. A technote that helps in such a scenario:
3. This could be a Websphere Application Server dynacache settings issue however if the issue persists even after following the technote, then this needs to be investigated as a possible WCM cache issue.
Syndication of a particular item(s)is not happening
Sometimes in an already running syndication setup it is reported that when they editing and re-saving an existing content item on the syndicator, it does not show up on the subscriber side. Do the following:
1.Use the IBM Support Tools Portlet to determine the UUID for the item that is not getting syndicated.
2.Use the displayEventlog from the portlet and get a screen capture of the output
3.Re-edit the item and re-save the item
4.Use the displayEventlog tool and get a screen capture of the output
5.Check between the screencaptures and see if the UUID got added to the eventlog or not - A lot of the times just resaving the item corrects the issue.
6. If it has not been added then there might be an issue with eventlog
Questions to ask:
6a. Does this happen for an item in a particular library ?
6b. If so, then does this happen for all items in that library ?
6c. If so, is this a migrated library ?
To resolve this do one of the following:
6.1) Make sure additional traces are added on the syndicator:
6.2) Reset the event log
6.3) Restart portal ( please note - restarting portal after restting event log may take sometime does not mean that the portal is hung)
6.4) Use the displayEventLog tool to check if the eventlog got populated now with the UUID of the item - if so then retry syndication, If not, then 6.5)
6.5) Attempt the following steps for a new item with the above mentioned traces in place:
1. Create a new item in the library being syndicated.
2. Use the IBM Support Tools portlet to determine the UUID for this item
3. Use the displayEventlog tool and get a screen capture of the output - does this item get added to the eventlog ?
6.6) Collect the trace logs, review the traces and the jsp output and consult with IBM
P.S: In scenarios such as above always try the syndication with a new item to ensure its not content/library specific issue.
7.If the item is added to the eventlog, then attempt the syndication with the MustGather traces enabled. Review the subscriber logs to see the cause of the failure. Typically the item would have failed to be processed by the subscriber due to the syndication errors already covered in the Section "Common Issues->Syndication updates complete but with errors"
Syndication completes but not all items show up on the subscriber side
(or Need to verify if all items on the syndicator match with all items on the subscriber)
1.In this scenario, please consult IBM Support for a LibraryExport.jsp and run it against the syndicator and subscriber nodes. This will generate a CSV file each on the syndicator and subscriber servers.
2.IBM support will compare the outputs of these and identify if certain items are missing.
3.Use troubleshooting techniques discussed so far to determine why certain items didnot make it through the syndication. That is based on the difference identify the missing UUIDs. Review the logs and see the errors associated with the syndication of these UUIDs. If no errors in the logs then see the eventlog to determine if it got added to event log or not. And if not then enable traces to determine why.
4.Keep in mind a large number of syndication issues can be identified by comparing the LibraryExport output from both syndicator and subscriber along with the displayEventLog output on the syndicator side.
Syndication and Clustering
A clustered environment typically consists of several nodes and a load balancer but can include more complex relationships between several clusters with multiple levels of load distribution as shown in figure 2 below. It is therefore important to understand the topology of the environment before making any decisions about how to configure syndication.
The term load balancer is often used loosely to mean any system which distributes load to multiple sources. To prevent confusion, we will use the term load balancer to mean a system which distributes load across several nodes in a single cluster and IP sprayer to mean a system which distributes load across several clusters as shown in figure 2 above.
Configuring syndication in a clustered environment
When configuring syndication it is best to keep the ultimate goal in mind, which is to update the JCR repository of a subscribing environment with changes from the JCR repository of a syndicating environment. This means that for complex setups like the one shown above where you have more than one JCR repository to update in what is conceptually a single subscription environment, you need to treat each cluster as a separate environment and create multiple syndication pairs. For this reason IP sprayers should never be used in configuring syndication. In other words, the syndicator and/or subscriber URLs should never point to a system that distributes load between multiple servers or clusters with different JCR repositories.
The remainder of this section will describe configuring syndication between the single authoring cluster and one cluster in the rendering environment. This process should be repeated to achieve syndication to the other rendering cluster.
The syndicator and subscriber objects can be created by accessing the interface via the load balancer or any node directly. Because all nodes in a cluster share the same repository, it doesn't matter which sends (for a syndicator) or receives and processes (for a subscriber) the syndication request. This means that the syndication URLs can be configured to/from any single node in the cluster, however it is recommended that it be configured to/from the load balancer to facilitate fail over should a node go down. The syndication URLs can be set to any specific node or the load balancer irrespective of how you accessed the system to create the objects.
Syndication scheduling in a clustered environment
Automatic syndication it initiated by the scheduler service which runs on each node in a clustered environment. Each scheduler competes for a lease which is only granted to one node in the cluster. The node that obtains the lease is then responsible for running all tasks in the system until the lease is released and granted to another node (see the "High availability" section at http://www.ibm.com/developerworks/websphere/techjournal/0404_johnson/0404_johnson.htmlfor more information).
This means that any node in the cluster could initiate automatic syndication which might be confusing if you have configured the syndication URLs to a specific node in the cluster. The important thing to realise here is that the syndication URLs do not restrict which nodes are used to perform syndication but simply indicate how each environment should contact the other.
For example, the syndication pair may be configured between cluster 1, node 1 (C1N1) and cluster 2, node 3 (C2N3) but the scheduler may initiate syndication on cluster 1, node 2 (C1N2). C1N2 will contact the subscriber using the configured URL - C2N3 which will receive and process the initial packet. C2N3 will then use the configured syndicator URL - C1N1 to request updated items. Because all syndication operations happen as individual HTTP request/responses, it is ok for the process to switch between nodes in this fashion. Interestingly this is also why it is ok to configure syndication to a load balancer which may distribute requests for updated items across different nodes in the cluster.
1.Syndication MustGather, the learn more and analyze data section provides several technotes
2. Syndication Whitepaper:
3. Troubleshooting section for Syndication in the InfoCenter for V6.1
4. IBM support Tools portlet for Lotus Web content Management