This article provides hints and tips for using the start and exclude parameter files for the WebSphere Portal Document Manager migration tool. Refer to the information in this article to troubleshoot problems with the parameter files and to improve the efficiency of your migrations.
Note: The information in this article applies to migrations where IBM Tivoli Directory Integrator is used, for instance to migrate to Lotus Quickr or to migrate from WebSphere Portal V5.1.
Using reserved characters in start/exclude parameter files
When adding file paths to your start and exclude path remember that there are a number of reserved characters that may cause problems. These characters may include '[' or '+' which are valid characters in a Portal Document Manager file name and also is a reserved character in a Java regular expression. You can "escape" reserved characters by adding a backslash \ in front of it. e.g.:
The path below contains reserved characters:
/DM/Document Manager/folder[1]/hints+tips
To fix it we need to escape the characters:
/DM/Document Manager/folder\[1]/hints\+tips
Tips for Selectively Migrating Particular File Types
You can use regular expressions to selectively migrate specific file types. The following regular expression eliminates anything with .xxx at the end and anything without a "." in the path:
^((?!(\.xxx$|/[^\.]*$)).)*
/[^\.]*$ migrates the folder structure of your Portal Document Manager library.
\.xxx$ selects file types. You can set this value to any file type.
For example, if you want to migrate .PDF files from your library, use the following regular expression in the exclude file:
^((?!(\.pdf$|/[^\.]*$)).)*
This expression only migrates .PDF files and the folder structure.
You can specify multiple file types in the exclude file by inserting | symbol between file extensions. For example, the following migrates .pdf, .doc, and .ppt files:
^((?!(\.pdf$|\.doc$|\.ppt$|/[^\.]*$)).)*
Configuring the Start and Exclude files
- Ensure each line is a valid regular expression. If you want to use a character that is a reserved character in Java Regex, escape it using "\". For example if you wish to use the path:
/contentRoot/icm:libraries[7]/myfolder
You will need to escape the '[' character:
/contentRoot/icm:libraries\[7]/myfolder
- Make sure you have no whitespace before or after each line, e.g. check for spaces or tab characters before or after each line.
Start / Exclude tips
The start and exclude functionality is based on the Java regular expression engine and uses the same syntax. If a path contains a Java regex reserved character (e.g. '[' ) it must be escaped using the '\' character. e.g.
/DM/Document Manager/folder[1]/subfolder -> this will not work.
/DM/Document Manager/folder\[1]/subfolder -> this will!
Other useful pieces of syntax to implement are the wild card character and the multiplier:
. -> represents any character
-> represents “one or more” of the preceding character.
For example:
/DM/Document Manager/folder2/.*
This will indicate any sub folder and content of folder2. If you wished to exclude folder 1, folder 2, folder 3 etc..., add the following to the exclude file:
/DM/Document Manager/folder.*
It is important to save the start/exclude files with Unicode encoding. This is to provide support for international characters.
The start and exclude files can have as many entries as required. If you wish to exclude one folder from two different libraries you could use the following:
/DM/Document Manager/folder3
/DM/library2/folderB
Differences between start and exclude
To start a migration from a certain folder or subtree – you must first ensure that all the parent folders exist. For example. If you wanted to start the migration from:
/DM/Library1/folder1/folder2
You would need to make sure that both the library (Library1) and the parent folder (folder1) existed. This makes the start file a little more complex than the exclude:
/DM/Library1 -> migrate the library
/DM/Library1/ -> we want to migrate content of the library starting from root
/DM/Library1/folder1 -> and including the folder: folder1
/DM/Library1/folder1/folder2 -> and folder2
/DM/Library1/folder1/folder2/.* -> without this line no content or sub folders of folder2 would be migrated.
To illustrate a more complex example: Say we wanted to migrate some subtrees starting from the following paths
/DM/Sales/2009/March
/DM/Sales/2009/April
/DM/Financial Reports/2009/Q1
Our start file would have to look like the following:
/DM/Sales
/DM/Sales/
/DM/Sales/2009
/DM/Sales/2009/March
/DM/Sales/2009/March/.*
/DM/Sales/2009/April
/DM/Sales/2009/April/.*
/DM/Financial Reports
/DM/Financial Reports/
/DM/Financial Reports/2009
/DM/Financial Reports/2009/Q1
/DM/Financial Reports/2009/Q1/.*
Using reports to help generate start/exclude files
While the start/exclude functionality is useful for selectively migrating content. It has advantages when it comes to enhancing performance. Suppose there is a power outage during a migration. The migration only managed to successfully migrate about half of the content. This can be confirmed by examining the detailed migration report generated by the tool. We can use the exclude or start files to ignore content previously migrated (reducing the overall migration time as there is no need to migrate files already migrated) or to carry on migration from where it left off. Lets look at these using an example:
For clarity, only the “source library”, “source path”, “entity type” and “Migration status” columns are shown. The report used is from a GVT test and includes international characters.
We have two options:
1) exclude everything that has migrated successfully
2) start where the migration left off
1) By inspection we can see that all of /DM/Large data set has migrated successfully. This is shown by a set of “SUCCESS” messages and the fact that the tool has moved on to begin migrating /DM/Library1. We could therefore exclude /DM/Large data set. It is not enough to simple copy and paste all the completed paths into the exclude file unless we are certain that all sub-content has been migrated.
For example: If we copied “/” into the exclude file then no content would be migrated at all as we would be excluding everything. Similarly, if we had the library excluded – then no content in that library would be migrated.
To exclude content we need to intelligently decide which sub-trees do not need to be migrated. In the example above – we could exclude the libraries
/DM/Large data set
/DM/Document Manager
As they both completed without problems. However, if /DM/Library1/folder1 failed because of corrupted data – we could exclude it from the migration also. Our exclude file would then contain:
/DM/Large data set
/DM/Document Manager
/DM/Library1/folder1
2) Alternatively we can continue the migration from a specific point or range of points.
This involves knowledge of what data exists. If we know that the following libraries exist:
/DM/Document Manager
/DM/Large data set
/DM/Library1
/DM/Library2
And we can see from the report that Document Manager and Large data set migrated completely, then we can just carry on from Library1 and Library2. Our start file would then include:
/DM/Library1
/DM/Library1/.*
/DM/Library2
/DM/Library2/.*
Or perhaps more cleanly (especially if we had Library1....libary100):
/DM/Library.*
Consider also the we may already have migrated some content form Library1 (or perhaps folder1 is corrupt). We could use a combination of start and exclude:
startfile
/DM/Library1
/DM/Library1/.*
/DM/Library2
/DM/Library2/.*
excludefile
/DM/Library1/folder1
Tips for Selectively Migrating libraries in specific order
For example, if the libraries gets migrated in alphabetical order and if the migration stops in the middle and when you want to start migration again
but want to avoid migrating all the libraries that have been already migrated you can use the regular expression in exclude file
DM/[d-qD-Q].* - This will exclude all the libraries starting with character 'd' to 'q' both in lower case and upper case
http://java.sun.com/j2se/1.4.2/docs/api/java/util/regex/Pattern.html