Understanding the syndication environment

Syndication transfers content items stored in the JCR repository of the syndicator server to the JCR repository of the subscriber server. Here is what you should look for in the syndication environment

1. Is the syndication being done for the first time to the subscriber?
2. Is the syndication being done between one syndicator and one Subscriber, or multiples? If multiples, get the details of the syndication setup including the number of syndication servers, number of syndicator/subscriber objects on a server, and the flow of syndication.
3. Is either the syndicator or the subscriber part of a cluster? If so, is syndication being done from/to one node on a cluster, or from/to the entire cluster through HTTPserver?
4. Is Is the syndication being done for one library or multiple libraries? If multiple libraries are involved, do the items in one library reference the other?
5. Is this syndication being done for “all” items or “all live” items?
6. Is the syndication automatic or manual?
7. Are this syndication being done for “all” items or “all live” items?
8. Do the syndicator and subscriber point to the same LDAP? If different LDAPs are involved, are the users and groups identical?
9. Are the syndicator(s) and subscriber(s) at the same fix level?
10. Has any authoring been done on the subscriber?
11. If doing an “all items” syndication, is versioning enabled on the syndicator?
12. Did syndication ever work completely?
13. Are any libraries being syndicated migrated libraries. If so, from which version were they migrated?

Also,collect the syndication MustGather information as described in either Support Technote 1243980, "MustGather: Web Content Management (WCM) v6 syndication issues," for V6.x or 1316241, "MustGather: Web Content Management (WCM) v6.1 Syndication" for V6.1.x.

Finally, capture a screenshot of the syndicator and subscriber objects.

Verify the environment and configuration

From the MustGather files

From the versioninfo.log file, verify that both syndicator and subscriber have exactly the same WCM version and WCM fixes installed. The syndicator and subscriber systems must match with regard to WCM version and fix level.

Check for loose class files on the syndicator and subscriber system as this impacts the fix levels. These loose class files are located in:

Check for the properties listed below in the syndicator and subscriber's WCMConfigService.properties file, located in:

Items to verify from the screenshots

Syndicator Object:

• Name of the syndicator. Keep a note of this since it will be referenced in the logs.
• Name of the subscriber. Keep a note of this since it will be referenced in the logs
• Subscriber ID. Subscriber ID is used to set up the subscription. Make sure this matches the ID on the subscriber object
• Subscriber URL. This should be a fully qualified URL (or in some cases, an IP address) that should match the value default on the subscriber object. NOTE: This value reflects the values of the variables, WCM_HOST and WCM_PORT, defined in the WebSphere Application Server. These variables can be overridden by hardcoding the values in the WCMConfigService.properties file.
• Libraries selected for syndication.
• Configuration of the item gatherer. Determine whether it's All items or All Live items syndication.
• Enable the syndicator. The checkbox should be checked.

Subscriber Object:

• Name of the subscriber.
• Name of the syndicator.
• The ID of the syndicator used to set up the subscription.
• Syndicator URL.
• Enable the subscriber.

Facts you should know about syndication

Since syndication transfers content items stored in the JCR repository of the syndicator server to the JCR repository of the subscriber server, the UUIDs of each of these items is the same across the Syndicator and Subscriber databases.

The syndicator and subscriber can be seen as gatekeepers to the JCRDB that stores the WCM content and processes the items through syndication as deemed necessary in the Event log.

Copying the JCR repository itself from the syndicator to subscriber is a forceful method of ensuring that all data on the syndicator shows up on the subscriber. We suggest that you use this method ONLY for a first-time syndication. If you do, consider the following::

• This will impact Portal Document Manager (PDM) / Personalization (PZN), since they use the same JCRDB.
• We suggest doing this ONLY if it's a first time syndication.
• Refer to the WCM wiki article, "Procedure for cloning WCM repositories," for more details on this topic.

Syndication uses the HTTP protocol and does not support the use of HTTPS (although it may work technically as long as the certificates are set up correctly between the servers).

General event log information
Resetting the Event log will cause a FULL REBUILD if automatic syndication is enabled or if manual syndication is initiated. Once the Event log is created, any content item saved to the JCR repository for a library will be logged in as one or more entries, depending on its state and other settings. For example, a live item is logged twice: once against the live item gatherer and again against the all items gatherer. There will also be additional entries for versions. The Event log provides critical information to determine if and why an object is not being syndicated. For more information, refer to the section below, Event Log Database and Syndication.

The IBM Support Tools portlet for Lotus WCM has some useful tools you can download (after registering with the site) to help in troubleshooting syndication, including:

• DisplayNodes
• DisplayEventLog
• ClearPersistenceTable
• Clearcaches

As a result, when the subscriber processes the item being pushed from the syndicator, it might skip saving it if there is a current or newer version of the item in its database. Authoring on the subscriber-side can be OK if the client is aware of the limitations and manages them properly, for example, in two-way syndication scenarios in which authoring is done on both servers but in different libraries.

Syndication setup example

A typical syndication setup is as follows:

For the topology, the Syndicator (Authoring server, on which the WCM Authoring portlet is installed) = one way syndication => Subscriber (Rendering server, on which the WCM Local Rendering portlet is installed) displays the content; probably does not have the Authoring portlet installed).

When syndicating two libraries, one content and one design, the content library has the content items, whereas the design library has the AT, PT, components, etc. Typically the content library will reference the items in the design library, and both libraries must be syndicated simultaneously for the syndication to succeed.

Typically both libraries are added to the syndicator in Live items scope; in this case, it make sense to syndicate Live items only, since there is no authoring being done on the rendering server side, and it is being used only to render content (LRP/RRP only displays live content). So if you are doing All items syndication and you have a setup similar this example, you should consider doing Live items.

Both subscription partners should share the same user repository (such as LDAP), if they use automatic syndication. If this is not the case, disable automatic syndication, perform manual syndication, and then execute memberfixer. Note that the purpose of automatic syndication is defeated if the LDAPs are not the same or, at the very least, the users and groups are not identical in both LDAPs with identical DNs.

Traces
Key traces to be enabled (on both syndicator and subscriber) are:

com.aptrix.deployment.*=all
com.aptrix.syndication.*=all

For Event-log-related issues:

com.ibm.workplace.wcm.services.eventlog.*=finest:com.ibm.workplace.wcm.services.transaction.steps.eventlog.*=finest

(The event log trace will show how and if an item was added to the event log)

Syndication can handle the syndicator and subscriber servers in different time zones as long as the server clocks for both are not more than a minute apart with the time zone differences factored in. For servers in the same time zone, the same rule applies of server times not being more than a minute apart.

Event log resets and traces
Items that were deleted and then purged will not be syndicated if the event log is reset. For example, if an item was deleted and purged on the syndicator side, the entry is recorded in the event log so that the same action can be performed on the subscriber side upon syndication. However, if the event log is reset prior to the syndication, then the rebuilt event log will not have this entry; hence, the item will not be syndicated.

Before resetting the event log, make sure that these event log traces are applied:

com.ibm.workplace.wcm.services.eventlog.*=finest:com.ibm.workplace.wcm.services.transaction.steps.eventlog.*=finest

Enable event log traces from the WebSphere Application Server Admin console in Config mode, save the configuration, stop the WebSphere Portal server, and then reset event log. Be sure to remove traces from the WebSphere Application Server Admin console in Config and runtime mode, after event log rebuild is complete and the server has started up. The server startup with the event log traces set and the event log being built provides vital information.

Miscellaneous

• If the repository (library) is large (approximately in the range of 20,000---30,000 items), then you should increase the number of historical trace files and trace file sizes in the WebSphere Application Server Admin console. A setting of 20 historical files and a file size of 50 MB should be adequate for most scenarios. The idea is to ensure that trace information is not lost due to file rollovers.

• Use the DisplayNodes tool included in IBM Support Tools Portlet for WCM to collect the output for any failing UUID. The displayNodes.jsp output will show all the references to that UUID, which can be useful in certain scenarios.

• Library access permissions are not syndicated. If the library does not exist on the subscriber, it will be created during syndication, but no access control settings are specified on the new library. On Rendering servers you might want to specify different access control settings, for example, to disallow content entry for all users.

• On the syndicator, the PackageGenerator class generates the packages to send to the subscriber; on the subscriber, the PackageProcessor class processes packages sent by the syndicator.

• When event log is rebuilt, it processes every WCM library (enabled), regardless of whether or not it's in a syndication definition. So, depending on how many WCM libraries (enabled) there are, the rebuild time can be quite long. Disabled libraries are not processed and not added to event log.

• Syndication cannot be scheduled, but the frequency of syndication occurrence can be set. By default it is set for 30 seconds but you can change it by changing the value deployment.itemChangedTaskDelay in the WCMConfigService.properties file. Also, you can control the occurrence of syndication in certain scenarios by using the publish date of content; the publishing of the content can be delayed and hence syndication can be delayed. This can be used for, example, to allow syndication to occur only during non-peak hours.

• Before running the configuration task wcm-reset-event-log, stop the WebSphere Portal server and then restart it when the task has completed. The Config task merely clears the event log DB; the event log is rebuilt when the WebSphere Portal server is started and may take a while to come up. To address this delay, a module was introduced to reset the event log DB while the WebSphere Portal server is running. This fix is available in WCM Cumulative iFix 21(PK90387) for 6.0.1.x and WCM cumulative iFix 15 (PK90388) for 6.1.x. Refer to Support Technote 1396691, "FAQs related to the reset event log module introduced with APAR PK88848."

• A common misconception is that Live Items syndication means syndicating only published items. In fact, the Live Items syndication includes published items, expired items, and non-workflowed items (but not drafts, versions, and deleted items).

• When performing a first-time or large syndication it is recommended to turn off the JCR text search. Refer to Support Technote 1299498, "Initial large library syndication in WCM while indexing may cause failures," for details.

Reading the traces

Most activity occurs on the subscriber side because that's where the subscriber goes through the list of items to be saved, fetches these items from the syndicator database, adds them to the repository, and then saves the document in its own repository. Here is what occurs in very basic terms (this may not be technically accurate, but the idea is to provide a mental picture of the syndication functioning in the most common scenario):

1. On the syndicator side the itemchangedtaskdelay value determines when the system checks for updates to the event log. If it finds updated items it will contact the subscriber and provide it with the list of items to be updated.
2. The subscriber then starts processing each item in this list, skipping items that are already at the most current version (you'll see the statement "skipping item" in the logs) and adding the newer modified items to the items that need to be updated.
3. After doing that, the subscriber fetches these items from the syndicator server's database and then attempts to save the documents (items) on the subscriber JCRDB. Most syndication errors are seen during this time.
4. If the subscriber is able to process all the items sent by the syndicator, it will send the syndicator a 603 message code upon receipt. If, for some reason, there are items that failed to be processed by the subscriber, they will be processed the next time the itemgatherer queries the event log.

New items are saved first, then updates and, lastly, removals. Typically you see a message at the end of the syndication in the subscriber trace logs such as this, which provides the subscription summary:

[datetime] 00000084 PackageProces I ****************** Start of subscription update report ******************
[datetime] 00000084 PackageProces I Subscription update started: 2008-06-04T22:19:17.125
[datetime] 00000084 PackageProces I Subscription update finished: 2008-06-04T22:20:02.141 (45016 milliseconds)
[datetime] 00000084 PackageProces I Syndicator: syndicator6_to_subscriber4, URL=http://testsyndication1.raleigh.ibm.com:10038/wps/wcm/connect/?MOD=Synd
[datetime] 00000084 PackageProces I Subscriber: subscriber4_from_syndicator6, URL=http://testsyndication2.raleigh.ibm.com:10038/wps/wcm/connect/?MOD=Subs
[datetime] 00000084 PackageProces I Full update
[datetime] 00000084 PackageProces I Updates sent: 21
[datetime] 00000084 PackageProces I Removes sent: 0
[datetime] 00000084 PackageProces I Items up to date: 20
[datetime] 00000084 PackageProces I No errors were detected during this syndication.
[datetime] 00000084 PackageProces I

---

High Level Details

---

[datetime] 00000084 PackageProces I Start of library updates, time=2008-06-04T22:20:02.109, count=1
[datetime] 00000084 PackageProces I Total items: 1
[datetime] 00000084 PackageProces I End of library updates, time=2008-06-04T22:20:02.109, elapsed=0 milliseconds
[datetime] 00000084 PackageProces I Start of draft item removal, time=2008-06-04T22:20:02.109, count=0
[datetime] 00000084 PackageProces I Total items: 0
[datetime] 00000084 PackageProces I End of draft item removal, time=2008-06-04T22:20:02.109, elapsed=0 milliseconds
[datetime] 00000084 PackageProces I Start of draft item removal, time=2008-06-04T22:20:02.109, count=0
[datetime] 00000084 PackageProces I Total items: 0
[datetime] 00000084 PackageProces I End of draft item removal, time=2008-06-04T22:20:02.109, elapsed=0 milliseconds
[datetime] 00000084 PackageProces I Start of new published items, time=2008-06-04T22:20:02.109, count=0
[datetime] 00000084 PackageProces I Total items: 0
[datetime] 00000084 PackageProces I End of new published items, time=2008-06-04T22:20:02.109, elapsed=0 milliseconds
[datetime] 00000084 PackageProces I Start of update published items, time=2008-06-04T22:20:02.109, count=0
[datetime] 00000084 PackageProces I Total items: 0
[datetime] 00000084 PackageProces I End of update published items, time=2008-06-04T22:20:02.109, elapsed=0 milliseconds
[datetime] 00000084 PackageProces I Start of new draft items, time=2008-06-04T22:20:02.125, count=0
[datetime] 00000084 PackageProces I Total items: 0
[datetime] 00000084 PackageProces I End of new draft items, time=2008-06-04T22:20:02.125, elapsed=0 milliseconds
[datetime] 00000084 PackageProces I Start of update draft items, time=2008-06-04T22:20:02.125, count=0
[datetime] 00000084 PackageProces I Total items: 0
[datetime] 00000084 PackageProces I End of update draft items, time=2008-06-04T22:20:02.125, elapsed=0 milliseconds
[datetime] 00000084 PackageProces I Start of item removal, time=2008-06-04T22:20:02.125, count=0
[datetime] 00000084 PackageProces I Total items: 0
[datetime] 00000084 PackageProces I End of item removal, time=2008-06-04T22:20:02.125, elapsed=0 milliseconds
[datetime] 00000084 PackageProces I Start of deferred new published content links, time=2008-06-04T22:20:02.125, count=0
[datetime] 00000084 PackageProces I Total items: 0
[datetime] 00000084 PackageProces I End of deferred new published content links, time=2008-06-04T22:20:02.125, elapsed=0 milliseconds
[datetime] 00000084 PackageProces I Start of deferred new draft content links, time=2008-06-04T22:20:02.125, count=0
[datetime] 00000084 PackageProces I Total items: 0
[datetime] 00000084 PackageProces I End of deferred new draft content links, time=2008-06-04T22:20:02.125, elapsed=0 milliseconds
[datetime] 00000084 PackageProces I Start of resolving references, time=2008-06-04T22:20:02.125, count=0
[datetime] 00000084 PackageProces I Total items: 0
[datetime] 00000084 PackageProces I End of resolving references, time=2008-06-04T22:20:02.125, elapsed=0 milliseconds
[datetime] 00000084 PackageProces I Start of item renaming, time=2008-06-04T22:20:02.141, count=0
[datetime] 00000084 PackageProces I Total items: 0
[datetime] 00000084 PackageProces I End of item renaming, time=2008-06-04T22:20:02.141, elapsed=0 milliseconds
[datetime] 00000084 PackageProces I Start of create versions, time=2008-06-04T22:20:02.141, count=0
[datetime] 00000084 PackageProces I Total items: 0
[datetime] 00000084 PackageProces I End of create versions, time=2008-06-04T22:20:02.141, elapsed=0 milliseconds
[datetime] 00000084 PackageProces I Start of update versions, time=2008-06-04T22:20:02.141, count=0
[datetime] 00000084 PackageProces I Total items: 0
[datetime] 00000084 PackageProces I End of update versions, time=2008-06-04T22:20:02.141, elapsed=0 milliseconds
[datetime] 00000084 PackageProces I Start of version removal, time=2008-06-04T22:20:02.141, count=0
[datetime] 00000084 PackageProces I Total items: 0
[datetime] 00000084 PackageProces I End of version removal, time=2008-06-04T22:20:02.141, elapsed=0 milliseconds
[datetime] 00000084 PackageProces I Start of item status update, time=2008-06-04T22:20:02.141, count=0
[datetime] 00000084 PackageProces I Total items: 0
[datetime] 00000084 PackageProces I End of item status update, time=2008-06-04T22:20:02.141, elapsed=0 milliseconds
[datetime] 00000084 PackageProces I

---

Low Level Details

---

[datetime] 00000084 PackageProces I Start of library updates, time=2008-06-04T22:20:02.109, count=1 [DepRef(id:409ae28049c897ccab84bf55786cef3c type: com.ibm.workplace.wcm.services.library.Library nonDraft:true draft:false purged:false parentId:409ae28049c897ccab84bf55786cef3c timeStamp:1211200237696 stateUpdate: false versions:null)]
[datetime] 00000084 PackageProces I Start of draft item removal, time=2008-06-04T22:20:02.109, count=0 []
[datetime] 00000084 PackageProces I Start of draft item removal, time=2008-06-04T22:20:02.109, count=0 []
[datetime] 00000084 PackageProces I Start of new published items, time=2008-06-04T22:20:02.109, count=0 []
[datetime] 00000084 PackageProces I Start of update published items, time=2008-06-04T22:20:02.109, count=0 []
[datetime] 00000084 PackageProces I Start of new draft items, time=2008-06-04T22:20:02.125, count=0 []
[datetime] 00000084 PackageProces I Start of update draft items, time=2008-06-04T22:20:02.125, count=0 []
[datetime] 00000084 PackageProces I Start of item removal, time=2008-06-04T22:20:02.125, count=0 []
[datetime] 00000084 PackageProces I Start of deferred new published content links, time=2008-06-04T22:20:02.125, count=0 []
[datetime] 00000084 PackageProces I Start of deferred new draft content links, time=2008-06-04T22:20:02.125, count=0 []
[datetime] 00000084 PackageProces I Start of resolving references, time=2008-06-04T22:20:02.125, count=0 []
[datetime] 00000084 PackageProces I Start of item renaming, time=2008-06-04T22:20:02.141, count=0 []
[datetime] 00000084 PackageProces I Start of create versions, time=2008-06-04T22:20:02.141, count=0 []
[datetime] 00000084 PackageProces I Start of update versions, time=2008-06-04T22:20:02.141, count=0 []
[datetime] 00000084 PackageProces I Start of version removal, time=2008-06-04T22:20:02.141, count=0 []
[datetime] 00000084 PackageProces I Start of item status update, time=2008-06-04T22:20:02.141, count=0 []
[datetime] 00000084 PackageProces I ******************* End of subscription update report *******************


Sometimes, instead of the 603 "no more confirmations to send" message, the subscriber responds to the syndicator with a 200 message code containing the term "OK". This indicates that the subscriber is still processing the updates the syndicator sent in earlier and does not want to process any new ones. This occurs when, say, the syndicator sends in a large number of updates, the subscriber is processing each one of them (and taking a long time), and meanwhile the itemchangedtaskdelay, which is set to 30 seconds, causes the syndicator to contact the subscriber again for event Log updates. Note that while the subscriber is processing the updates sent by the syndicator, the status of the subscriber will show as active, whereas the syndicator will show as idle, for most part.

To read the traces:

1. Start with the syndicator log to identify the syndication initiation point and correlate the corresponding message on the subscriber systemout.log
2. Once identified, use the systemout.log to determine where the error occurs (it's easier to scan through the systemout.log than the trace.log to find this).
3. Once identified, then find the corresponding entry in the trace.log to see if additional information is available.

Since the majority of the activity in the most common scenarios occurs on the subscriber side, the subscriber logs typically are larger than those for the syndicator, and any errors most probably show up in the subscriber logs.

Finally, edit the WCMConfigService.properties file, changing the deployment.enableReport setting to "true". This enables high-level reporting of syndication to the systemout.log on the subscriber server, providing a summary of items that were processed and which items failed, to help with troubleshooting issues.

Event log database and syndication

First, we must note that syndication changed dramatically from versions V5 to V6 and, although some parts use the same names (the item gatherers for example), they are completely different from what they were in previous releases.

Syndication begins at the library level. Each WCM library is associated with two item gatherers, the All Items gatherer and the Live Items gatherer. Unique to every library, these item gatherers provide the means to track items by status in specific libraries. Unlike in pre-V6 releases, the item gatherers do not contain any information about the items; rather, they are simply a label used to associate a syndicator object with the items it must syndicate.

When an item is created or modified, the state of the item is updated in the event log database. It is important to understand that the event log stores the state of items, not the actions that were performed on them. To illustrate, let's look at the following example:

When you create a new content document, an entry similar to the following is added to the event log:





where:

• ITEMID is the ID of the content document
• TYPENAME is the class type of the content document
• ITEMTYPE indicates whether the content document is published (P) or draft (D)
• STATE is a number representing the current state of the content document
• DELETED indicates whether the content document has been deleted (Y), purged (P), or moved (M)
• PARENTID is the ID of the content document's parent node (i.e. a sitearea id)
• TSTAMP is the timestamp of when the content document was last modified
• ITEMGATHERERID is the ID of the appropriate item gatherer (All or Live) associated with the content document's library

When the content document is modified, the entry in the event log is updated to reflect the updated state of the item. For example, syndication uses the information in the event log to determine what items it must send to a subscriber. When you create a syndicator object you select a library and a scope (All items or Live item), associating the syndicator object with a particular ITEMGATHERERID. When syndication occurs, the syndicator obtains all the items in the event log that match its associated item gatherer and sends them to the subscriber to be processed.

The full process uses the STATE, DELETED flag and other information in the event log to determine what action to perform on the subscriber. Basically the event log maintains a list of items in the system, by state and associated with an item gatherer id, and that syndication uses the event log to determine what items it must send to a subscriber.

When you run the wcm-reset-event-log task you are actually clearing all the entries from the event log database and telling WCM to rebuild the event log when the server restarts. This is possible because the event log database stores the current state of all the items in the system and not the actions that were performed on the items, so no information is lost when the event log is reset. WCM simply gets every item in the system and adds it to the event log, using its current state.

Ideally the event log database should always be kept in sync with all the items in the system and never need to be reset. In practice, however, sometimes an item is not added to the event log, or an entry in the event log is not updated due to a defect or error in the system. This can cause the item to not be syndicated; resetting the event log will correct the problem by re-adding all items in the system in their current state.

Troubleshooting the most common issues

In this section we discuss how to troubleshoot some common issues, focusing on what to verify from the existing information, the additional information to request (if needed), and actions to be taken.

PROBLEM: Automatic syndication is not occurring

In this case, the reported symptoms are that:

• syndication is not even initiating
• the syndicator/subscriber connect, but there is no data being transferred
• syndication starts up from the syndicator, but there's no response from the subscriber
• syndication updates don't show up on the subscriber at all

1. On the syndicator machine, check:

to make sure that inittask for syndication, connect.moduleconfig.syndication.inittasks, is set to true.

2. On the syndicator machine, check

to make sure that subscriberOnly is NOT set to true and that deployment.subscriberOnly=false.

3. If Steps 1 and 2 check out OK, then the issue can be due to one of the following:

a) Event log is empty
b) Persistence tables need to be cleared
c) EJBTimers are not firing

Check these by using the following steps:

1. Make sure the following message logged in the syndicator SystemOut.log at startup:

"SCHD0133I: Scheduler WPSTaskScheduler (wps/Scheduler) has acquired the lease and will run all tasks on this application server."

In a cluster environment, this message should log into either cluster member's SystemOut.log. If the message is not the SystemOut.log, then EJBTimer won't be able to fire the scheduled timers to send syndication updates to the subscriber. WebSphere Application Server support can help to configure the scheduler for cluster environment.

2. Make sure there are no SQL exceptions in the trace, for example, com.ibm.websphere.ce.cm.DuplicateKeyException. If an SQL exception is found in the trace, use the Clear_WCM_Persistence_Service_table.jsp to clear the WCM_OBJECTS table from persistence database. This tool is available in the IBM Support Tools Portlet.

3. Get the output of the following command and EJBTimers_nn.nn.log from the syndicator server:

Typically you should see something like this:



If not found, then try restarting WebSphere Portal and run the command again. If still not present, then run cancelEJBTimers and restart WebSphere Portal.

4. If there are many EJBTimer tasks scheduled, then run following command on the syndicator server:

NOTE: For EJBTimer issues, enable the following trace: com.ibm.workplace.wcm.util.scheduler.*=all

5. Make sure that the Event log is populated with entries and is not empty. To do this:

a) Run the displayEventLog tool from the IBM Support Tools portlet on the syndication server.
b) Review the output of the jsp to ensure that it is not empty.
c) If it is empty, make sure that you double-check that the subscriberOnly value is set to true.
d) Try restarting the WebSphere Portal server.

6. If the Event log is empty and the subscriberOnly value is set to false, and a server restart does not fix the issue, and everything else above checked out OK, then do the following;

a) Make sure additional traces are added on the syndicator server:

com.ibm.workplace.wcm.services.eventlog.*=finest:com.ibm.workplace.wcm.services.transaction.steps.eventlog.*=finest

(NOTE: As mentioned above, ensure that the traces in WebSphere Application Server are set and the number of historical files and file size of the trace files are adequate)

b) Reset the Event log
c) Restart WebSphere Portal
d) Run the displayEventLog tool and check if the Event log is now populated; if so, then re-attempt syndication. If not, then collect the trace logs and the jsp output for IBM Support.

PROBLEM: Syndication updates complete but with errors

These are some of the most typical errors seen in the logs:

"DuplicatePathException"

To resolve the error:

1. Remove the failing object(s) from the Subscriber instance.
2. Then re-save the failing object on the syndicator and do an "Update", or you can simply do a "Full Rebuild" of the syndication pair.

Refer to the Support Technote, "Troubleshooting Duplicate Path Exception errors in Web Content Management," for more information.

"ReferentialIntegrity Errors"
This would indicate that the references for an item being saved could not be found. Typically it means an item may be referencing another item in another library, and most likely that library is not getting syndicated over. Confirm that all the libraries that have items referencing each other are being syndicated.

"DuplicateKeyException"
com.ibm.websphere.ce.cm.DuplicateKeyException: [IBM][CLI Driver][DB2/AIX64] SQL0803N. One or more values in the INSERT statement, UPDATE statement, or foreign key update caused by a DELETE statement are not valid because the primary key, unique constraint, or unique index identified by "1" constrains table "JCR.EV_VERSION" from having duplicate rows for those columns.SQLSTATE=23505

Resetting the Event log should resolve this error. Also, you can use the Clear Persistence table jsp available in the IBM Support tools portlet.

"ItemNotFound"
javax.jcr.ItemNotFoundException: A node does not exist with UUID: 60b1f58044f3bee49956b9f420fa6210 at com.ibm.icm.jcr.WorkspaceImpl.getNodeByUuid(WorkspaceImpl.java:1117)

While retrieving an item from the syndicator database, the subscriber logs throw this error; it is unable to find the item in the JCRDB of the syndicator server. This could possibly indicate that the Event log is not in sync with the JCRDB for the syndicator. Reset the Event log, and then try the syndication again.

"Parent Not Found"
[5/6/09 16:32:24:577 EDT] 00000041 PackageProces W Could not save item with id DepRef(id:16c567804590753c8bd68f0b6a6ea584 type: com.aptrix.pluto.site.SiteArea nonDraft:true draft:false purged:false parentId:6babef80458422afb958b9fb02923ee3 timeStamp:1227863963980 stateUpdate: false versions:null moved: false) because it could not find its parent.

Typically you see this error as a symptom of the problem in which the parent item or its parent could not be saved. Use the parent UUID mentioned in the string above to check the logs to determine the reason for the parent failing. If a top-level site or a workflow fails to be saved, it will cause the saving of several items to fail, and the above error will display in the logs.

"InvalidClassException"
IWKSY1059X: detail: IWKPD1088X: Exception occurred while trying to retrieve document from item.IWKSY1060X:(Cause:java.io.InvalidClassException: com.aptrix.pluto.content.Content; local class incompatible: stream classdesc serialVersionUID = 1595762135600321749, local class serialVersionUID = -1089925949120930392)
at com.aptrix.deployment.subscriber.SerialisationTransformer.transformDocument(SerialisationTransformer.java:88)

This issue is addressed in Support Technote, "Error: 'InvalidClassException' occurs during WCM syndication".

From the versioninfo.log, verify that the fixes installed on both the syndicator and subscriber match. If the fix levels match and you still see the error, check for loose class files on the syndicator or subscriber system, which will impact the fix levels. These loose class files are located in:

PROBLEM: Syndication performance is poor

To address this issue:

1. Increase the default syndication frequency of 30 seconds for servers that don’t require frequent syndication. Be sure not to space the intervals too far apart; 10 to 30 minutes is a good range.
2. Don’t enable a syndicator on the production rendering server. Set the subscriber.Only property to “true” in WCMConfigServices.properties . This stops the monitoring task from running on the IWWCM instance that tracks object changes for later syndication to another server. Whenever the subscriber.Only setting is changed, you also must reset the EventLog (\config\WPSConfig wcm-reset-event-log) before restarting the server
3. Avoid having too many Subscribers (>5) linked to the one Syndicator. Create a tiered syndication structure, for example, in which the first syndicator syndicates to two servers and those servers further syndicate onto the rest. This improves performance because you are decreasing the load on the syndicator server.
4. Consider disabling Versioning for all items on both authoring and rendering servers. If storing previous versions of items isn’t required, then disabling this feature can improve performance of saving and syndication. Refer to the topic, "Web content authoring options," in the WebSphere Portal and the Web Content Management InfoCenters for details on how to achieve this.

Set the Version control options in the WCMConfigservice.properties file, as follows:

* versioningStrategy.AuthoringTemplate= never (or manual)
* versioningStrategy.Component= never (or manual)
* versioningStrategy.Content= never (or manual)
* versioningStrategy.PresentationTemplate= never (or manual)
* versioningStrategy.Site= never(or manual)
* versioningStrategy.Taxonomy= never (or manual)
* versioningStrategy.Workflow= never (or manual)
* versioningStrategy.Default= never (or manual)

PROBLEM: Syndication changes do not show up on all cluster members

Typically the scenario reported in this case is syndication from a standalone authoring server to a single node in a two-node horizontal cluster. The syndication completes successfully, but the changes are not reflected when accessing the cluster through the front-end HTTP server or when accessing the non-syndicator node directly through its internal HTTP transport port.

This is not a syndication issue but rather an issue with the WCM cache objects defined in the WebSphere Application Server dynacache. A simple test to verify this is to restart the non-syndicator node (the changes should reflect now).

To work around the issue, refer to the Support Technote, "Content not displayed on cluster members even after syndication completes in Web Content Management (WCM) v," If the issue persists even after following the Technote instructions, then this needs to be investigated as a possible WCM cache issue.

PROBLEM: Syndication of a particular item(s) is not occurring

Sometimes in an already-running syndication setup, when an existing content item on the syndicator is edited and re-saved, it does not show up on the subscriber side.

To resolve the issue, do the following:

1. Use the IBM Support Tools Portlet to determine the UUID for the item that is not being syndicated.
2. Use the displayEventlog from the portlet to get a screen capture of the output.
3. Re-edit the item and re-save it.
4. Use the displayEventlog tool and to get another screen capture of the output.
5. Compare the screen captures to see if the UUID was added to the event log; many times just re-saving the item corrects the issue.
6. If it has not been added, then there might be an issue with the event log, in which case you need to ask the following:

a) Does this happen for an item in a particular library?
b) If so, then does this happen for all items in that library?
c) If so, is it a migrated library?

To resolve the issue. do one of the following:

• Make sure additional traces are added on the syndicator:

com.ibm.workplace.wcm.services.eventlog.*=finest:com.ibm.workplace.wcm.services.transaction.steps.eventlog.*=finest

• Reset the event log.
• Restart WebSphere Portal (note that restarting WebSphere Portal after resetting the event log may take some time; it does not mean that it's hung).
• Use the displayEventLog tool to check if the event log is now populated with the UUID of the item; if so, then retry syndication. If not, then attempt the following steps for a new item with the above-mentioned traces in place:

1. Create a new item in the library being syndicated.
2. Use the IBM Support Tools portlet to determine the UUID for the item.
3. Use the displayEventlog tool to get a screen capture of the output; does the item get added to the event log?

• Collect the trace logs, review the traces and the jsp output and consult with IBM

NOTE: In scenarios such as this, always try the syndication with a new item to ensure its not a content/library-specific issue.

• If the item is added to the event log, then attempt the syndication with the MustGather traces enabled. Review the subscriber logs to see the cause of the failure. Typically the item would have failed to be processed by the subscriber due to the syndication errors already covered in the "Syndication updates complete but with errors" section above.

Syndication completes but not all items show up on the subscriber side

(or need to verify if all items on the syndicator match with all items on the subscriber)

In this scenario, consult IBM Support to obtain a LibraryExport.jsp and run it against the syndicator and subscriber nodes. This will generate a CSV file each on the syndicator and subscriber servers. IBM Support will compare the outputs of these and determine whether certain items are missing.

1. Use the troubleshooting techniques discussed thus far to determine why certain items did not make it through the syndication; that is, based on the differences, identify the missing UUIDs.
2. Review the logs to see the errors associated with the syndication of these UUIDs. If no errors are in the logs, then look at the event log to determine whether the item was added to the event log; if not, then enable traces to determine why.
3. Keep in mind that you can identify a large number of syndication issues by comparing the LibraryExport output from both the syndicator and subscriber, along with the displayEventLog output on the syndicator side.

Syndication and clustering

A clustered environment typically consists of several nodes and a load balancer, but it can include more complex relationships between several clusters with multiple levels of load distribution (see figure 2). It is therefore important to understand the topology of the environment before making any decisions about how to configure syndication.






The term "load balancer" is often used loosely to mean any system that distributes load to multiple sources. To prevent confusion, here we use the term load balancer to mean a system that distributes load across several nodes in a single cluster, and we use the term "IP sprayer" to mean a system that distributes load across several clusters as shown in figure 2.

Configuring syndication in a clustered environment

When configuring syndication it is best to keep the ultimate goal in mind, which is to update the JCR repository of a subscribing environment with changes from the JCR repository of a syndicating environment. This means that, for complex setups like the one shown above, in which there is more than one JCR repository to update in what is conceptually a single-subscription environment, you must treat each cluster as a separate environment and create multiple syndication pairs.

For that reason IP sprayers should never be used in configuring syndication. In other words, the syndicator and/or subscriber URLs should never point to a system that distributes load between multiple servers or clusters with different JCR repositories. The remainder of this section describes configuring syndication between the single authoring cluster and one cluster in the rendering environment. This process should be repeated to achieve syndication to the other rendering cluster.

You can create the syndicator and subscriber objects by accessing the interface via the load balancer or any node directly. Because all nodes in a cluster share the same repository, it doesn't matter which sends (for a syndicator) or receives and processes (for a subscriber) the syndication request.

This means that the syndication URLs can be configured to/from any single node in the cluster; however, it is recommended that it be configured to/from the load balancer to facilitate failover, should a node go down. The syndication URLs can be set to any specific node or the load balancer irrespective of how you accessed the system to create the objects.

Scheduling syndication in a clustered environment

Automatic syndication it initiated by the scheduler service that runs on each node in a clustered environment. Each scheduler competes for a lease that is granted only to one node in the cluster. The node that obtains the lease is then responsible for running all tasks in the system until the lease is released and granted to another node (see the "High availability" section in the developerWorks article, "WebSphere Enterprise Scheduler planning and administration guide," for more information).

This means that any node in the cluster could initiate automatic syndication, which might be confusing if you have configured the syndication URLs to a specific node in the cluster. The important thing to realize here is that the syndication URLs do not restrict which nodes are used to perform syndication but simply indicate how each environment should contact the other.

For example, the syndication pair may be configured between cluster 1, node 1 (C1N1), and cluster 2, node 3 (C2N3), but the scheduler may initiate syndication on cluster 1, node 2 (C1N2). C1N2 will contact the subscriber using the configured URL - C2N3, which will receive and process the initial packet. C2N3 will then use the configured syndicator URL - C1N1 to request updated items. Because all syndication operations occur individual HTTP request/responses, it is OK for the process to switch between nodes in this fashion. Interestingly, this is also why it is OK to configure syndication to a load balancer that may distribute requests for updated items across different nodes in the cluster.

Resources

Support Technote, "Troubleshoot: Web Content Management (WCM) v6 syndication known issues":
http://www-01.ibm.com/support/docview.wss?rs=1041&uid=swg21265396

Support Technote, "Learn more: A basic overview of Web Content Management (WCM) v6 syndication":
http://www-01.ibm.com/support/docview.wss?rs=1041&uid=swg21265394

developerWorks aricle, :Understanding syndication in IBM Workplace Web Content Management":
http://www.ibm.com/developerworks/lotus/library/wwcm-syndication/

Web Content Management 6.1 InfoCenter, Troubleshooting section for Syndication:
http://publib.boulder.ibm.com/infocenter/wcmdoc/v6r0/topic/com.ibm.lotus.wcm.doc/wcm/wcm_syndication_troubleshooting.html

IBM Support Tools portlet for Lotus Web Content Management:
https://greenhouse.lotus.com/plugins/plugincatalog.nsf/assetDetails.xsp?action=editDocument&documentId=AE2BB2412F20AA318525772E006F7014

About the author

Sunil Bhatnagar is an Advisory Software Engineer working with the Lotus Web Content Management team. He is also the team lead for WCM US L2 Support team since 2007 and has worked extensively on a variety of syndication issues. Sunil holds a Masters degree in Computer Science and is an IBM Certified Application Developer for IBM Workplace Web Content Management