Manipulating Design Elements with LotusScript (or Java)

Example applications for these techniques might be:

• Update view selection formula to change over time (to avoid use of @Today in a view).
• Perform automated quality checks on Notes/Domino designs, e.g.:
• find all editable fields that have no field help.
• determine which forms, subforms and pages contain a particular word in their static text (see Resources)
• automatically find formulas that could be optimized (because they use the same @DbLookup twice, for instance).
• Change the look of an application by automatically updating a stylesheet based on selections in a setup form.
• Change the look of an application by scanning forms, subforms and pages for text with certain formatting, and changing them to different formatting.
• Increase productivity by automatically creating design elements -- e.g.:
• Create a custom view based on user selection input in a dialog, copying predefined columns from a model view.
• Create a view that includes all the fields on a form, so that you just have to delete the ones you didn't want (see Resources)
• Create an "email survey" form based on user input of questions and answer lists (see Resources).
• Rescue design elements that have somehow become corrupted.
• ...

Anything that you can't do with LotusScript, you can do with the Notes C++ API or C API. However, LotusScript does a lot of the work for you in many cases, and it's easy to integrate into an application so that it can be invoked from the Notes client as needed.

General Considerations
Here are a few things you need to be aware of when working with design elements.

Caching Issues
The Notes client maintains a cache of design elements; the first time you access a design note, it's added to the cache, and as long as the application remains open, when you make use of that design element the cached copy will be used. When you edit a design element using Domino Designer, Designer sends a signal to the Notes client to dump its cache of that design element, so that you can see the changes immediately. However, when the changes are made by an automated process, in general there is no way to notify the client of the change, so the cached copy will continue to be used. This is an obstacle to certain things we would like to be able to automate, and it is to be hoped that at some point in the future a method will be available to refresh the cache programmatically.

Meanwhile, you can deal with this in the following ways:

• If you modify an Outline design element (in verion 8.0 or later), the method NotesUIWorkspace.OutlineReload can be used to update the cache and re-paint the outline if it is displayed on screen at that time.
• Design notes with file attachments are not cached. This is kind of a kludge, but if you attach a hidden file attachment to your design element, you should be able to update it and have users see the changes immediately. This works with forms, subforms, and pages, since they have a rich-text body where you can attach something. As it happens, though, every design element is a note, and any note can contain any items, so it's possible in theory to add a file attachment to anything. I don't know whether this would work to prevent caching of other design element types; if you try it, please add a comment!
• Rename the design element also, and refer to it by its new name or alias. This may require some refinement to allow reference by an invariant alias while changing some other alias that would force reloading. For instance, if it's a form, you might call it "User Survey|Surveyxxx|Survey" (it has two aliases, and in the first alias you replace xxx with a sequential number). Then you compose the form using the Surveyxxx alias (which is not in the cache, so the user sees the latest version), but when they save the form, the value of the Form item is "Survey".
• Tell the user to close and reopen the application. Note: The application is considered open if any window to it is still open, including in Domino Designer or Administrator. In Designer 8.5, you might have to close the Designer client entirely to make it release the database (it might be sufficient to close the tabs into the application and collapse it in the navigator).
• If you have other ingenious methods of dealing with this limitation, please add a comment! Thanks.

Certain design elements -- image resources for instance -- it doesn't matter whether they're signed because they can contain no executable code. There's no security risk if someone modifies them, so confirming their signatures would be a waste of time. In general, though, you should make sure any signatures in design elements are up to date whenever you modify them.

This is covered in more detail in sections below.

Private Design Elements
There are two types of private design elements. "Server Private" design elements are stored in the application NSF, just like any other design element, and you can access their designs the same way you can a regular, shared design element -- provided you have access to them. What distinguishes a server private design element from shared, is covered in more detail later.

"Desktop private" design elements are stored in the user's desktop file. These are difficult to automate, and in fact are outside the scope of this document. Someone might write a separate article about how to access design elements in the desktop file.

In theory, any design element type can be private, but generally, only views, folders and agents can easily be created as private by users.

Similar Design Elements Types
A subform is really nothing but a form that can't be used on its own. Anything we say about forms, can generally be applied to subforms also, with the exception of features that are different between them (window title formula, for instance). Likewise, a Page design element (not XPage) is nothing but a form that can't have fields and can't be used to edit a document.

Views and folders are likewise similar, and we'll often talk about views as shorthand for "views and folders."

There are several design elements that are essentially just containers for a file. Whatever we might say about File Resources, generally applies to Images, Stylesheets, Wiring Properties, Composite Applications, XPages and Themes. Some of these types only store XML files, but the format of the design element is not different.

The Design Element Classes
Because these classes are covered in the Domino Designer help files, we won't describe them in detail here -- just draw your attention to the ones that will be useful and add any caveats that you might not find in the help.

Generally speaking, there's not a lot you can do here that you can't do with DXL or in other ways; these classes are just a lot easier to use, if whatever you want to do is supported. So if we say you can't do something in one of the sections below, read that as, you can't do it with this class, not that it's necessarily impossible.

NotesDatabase
The database is not a design element, but it's the class you use as a starting point to access the different design elements. The members of the NotesDatabase class that are relevant to design programming:

• Forms property
• GetForm method: One drawback to this method is that a design will often contain multiple forms with the same name -- one for the Notes client and one for the web, for instance -- and there's no way to control which one you will get. You would have to iterate through the array returned by the Forms property to find the right one.
• Views property
• GetView method: As with GetForm, if there are duplicate names you can't predict which will be returned. However, iterating through the Views property is generally not a good idea, as will be explained in the section on NotesView, below.
• Agents property
• GetAgent method: See notes about GetForm, above. Duplicate agent names are rarer, since there's no real reason to have them.
• TemplateName and DesignTemplateName: These are unfortunately read-only.

One problem with NotesView is that when you create the object -- or maybe when you first try to access any of its properties or methods -- its index is automatically updated. For some views in some databases, this can take a long time. Sometimes you just want information about the view -- it's title, say, to see whether it's the view you're searching for in the NotesDatabase.Views array. In that case, building the index is a total waste of time. Not only is it slow, but if the view is seldom used, its index could well have expired, so the view indexing task on the server is not keeping it up to date. If you access the view and build its index, you're using up a chunk of storage on the server and adding work for the view indexer, which must now keep this view up to date for the next 45 days (assuming the view has the default indexing settings), even though nobody ever uses it.

That's why it's often better to use other techniques, described in later sections, to access design information about views. Unless you want to read view entries and documents from it, avoid causing the index to build.

The main contents of a view design are:

• Options: some are available through this class. The most useful is the SelectionFormula property, which lets you programmatically adjust the view selection. This is useful for automatically creating private views with customized selection formula, and for creating date-based views that don't use @Today -- instead, the date can be hardcoded into the selection formula and updated periodically by a scheduled agent, for better performance.
• Columns: There's a Columns property of NotesView that returns an array of NotesViewColumn objects. One useful thing you can do with these, is copy them to a different view using NotesView.CopyColumn method. This lets you put together customized views based on selections from among a list of pre-programmed columns.
• Actions: Not available through this class.
• Event code: Not available through this class.

There's no easy way to figure out which columns contribute to the ColumnValues array and which do not. You would have to parse the formula yourself to determine whether it returned a constant value.

NotesForm
The NotesForm class represents a form or subform. You can get an object of this class either using the GetForm method of NotesDatabase, or the Forms property.

Probably the most useful property is Fields, which returns you an array of the names of fields on the form. This class doesn't provide any way to get more information about the fields, however, such as their datatype or whether they're editable.

There doesn't appear to be any way to use a NotesForm to get access to a Page design element (though, as mentioned above, they are otherwise similar).

NotesAgent
You can automatically enable profiling for an agent, and you can retrieve the document with the profiling results after it runs. It would be nice to have a way to adjust the agent's schedule and selection criteria from here, but that's not supported.

NotesOutline and NotesOutlineEntry
The NotesOutline class represents an Outline design element. By using this and the NotesOutlineEntry class, it's possible to create or manipulate outlines.

Besides their use for navigation in applications, outlines are also used internally to represent toolbars and bookmarks. The outline entries in these special outlines use a special format, so you can't use some of the properties that you might think would be useful in these cases. For instance, you can't use NotesOutlineEntry.Formula to access the formula of a toolbar icon. However, you can copy outline entries, even these special ones, from one outline to another using the CreateEntryFrom method of NotesOutline.

Once you have updated an outline, if it's visible on the user's screen, you can force it to update using the method NotesUIDatabase.OutlineRefresh. This is the only design element for which we have the ability to update the client's design element cache.

Internals of Design Elements

To do more with design elements than the above classes will allow, you must understand how design elements are stored in a Notes application.

Every design element and every document in an NSF or NTF file is stored in the same way, as a "note." A note is a collection of "items" with some header information (e.g. when was it last modified). Each item has a name, datatype, value, and potentially some flags (for instance, one of the flags says whether the item is a Readers list).

With the exception of file attachments and OLE objects, all information about a design element is stored in the items of the design note and in the note header.

It's instructive to highlight a design element in the design list in Domino Designer and examine its properties, where you can see the values of the items. Figure 1 shows an example, where we can see the $Title item contains a two-element list whose values are "Response" and "Response". Prior to version 8.5, this information would be displayed in the "fields" tab of the design element infobox (Figure 2).

Figure 1: Properties of a design element, showing item values (from Designer version 8.5)


Figure 2: Properties of a design element, showing item values (pre-8.5)


Note Class
The "note class" is part of the note header. There are several different note classes, e.g. "document" (a regular document, profile document, or other special document), "form" (a Form design element or related design element), "filter" (an agent or similar), "view", and several others. The note class distinguishes a design element from a data note, and some types of design elements from others. Some types of design elements have the same note class (form and subform are both "form" class, for instance) and are told apart by their $Flags item (more about this later).

Unfortunately, the note class is not available through the NotesDocument properties.

Common Design Element Data Items
Each design element type stores information in items with specific names. Some item names are only used for specific types of design elements, but some apply to all design element types. Here's a list of some common items that apply to all or many design elements.

There are many other item names which are specific to one or a few types of design element. If you're unsure, it helps to manually edit the design element and see what effect this has on the item values, to find out where and how certain design information is stored.

In cases where it's not clear how a certain property is represented, you can always copy down information from the items, change that property, and then compare the items before and after. The file "stdnames.h" contains definitions of these names and sometimes comments about how they're used (see Resources)

The $Flags Item
Many properties of design elements are encoded in the $Flags item. For instance, a design element that's hidden from web users contains the character "w" in $Flags. This item is also used to distinguish between design elements of the same note class but different types. For instance, "U" means the design element is a subform (assuming its note class is "Form").

To find out the meanings of the different characters, refer to the file "stdnames.h" (see Resources)

A few of the more important characters that apply to all types of design elements:

Binary-Formatted Items
Much of the most useful information in design elements is stored in binary form within items. Generally speaking, you're not going to be able to make much sense of this data without the Notes APIs or client to help you. These items are generally formatted as "Composite Data Records," (CD for short). Each record begins with a few bytes that identify the code for the record type and a length. These are formatted so that even if you don't recognize the record type, there's a standard way to determine the length. This lets the client code skip over records of a type it doesn't support. This is helpful in providing backwards compatibility, allowing earlier versions of Notes to make use of design elements edited by later versions, while skipping over information having to do with new features.

However, there are some cases where you can do something useful with the binary-formatted data, without having to parse it. Sometimes it makes sense to delete an item from the design note. For instance, there's a tool to remove compiled LotusScript code from forms and views, which unnecessarily duplicates the code in shared actions. This is done by determining which actions are shared (by examining the DXL, about which more later) and removing the corresponding $$ScriptObj_x items from the note.

Sometimes, you may also want to copy items from one design note to another. If you can figure out which items represent the data on an action bar, for instance, you can copy an action bar and all the actions from one view to another automatically, together with all the action bar formatting options such as background color and button height.

The Form Body
The $Body item contains the body of a form design element. One useful fact about this item, is that it's a rich text item, just like the body of a document, and you can use the Notes rich text classes (NotesRichTextItem, NotesRichTextRange, etcetera) to read and update it. There are no entities in the rich text classes corresponding to fields, buttons, URL links, and many other things you can find on forms, but you can manipulate static text and text styles, which is often useful, and you can work with tables to a limited extent.

Like other binary-formatted items, rich text consists of CD records. Unlike those other binary formats, though, some interpretation of the contents is possible with the built-in classes.

LotusScript - Source and Object
LotusScript source code is stored as "straight text" in items of the design note. There may be several separate items of source code. For instance, on a form, there's an item that represents the source code for the (Globals) section, another for the form event, and for each field that has LotusScript event code (such as an Entering event).

For buttons and other action hotspots embedded in the rich text, there is no separate item for source code storage. The LotusScript code in this case is embedded in the CD records of the rich text.

The source code for actions is stored in the binary records $ACTIONS and $V5ACTIONS.

When you save a design element in Designer, the LotusScript source is compiled (not necessarily all of it, however -- modules you haven't edited won't be). The compiled "object" code is stored in items whose names end with "_O" (letter O). The exception is actions, whose object code is stored in items named $SCRIPTOBJ_# (with # replaced by a number corresponding to the numerical position of the action in the list of actions).

Signatures
If you modify a design element by accessing its items directly, and any of the items you change are included in the electronic signature, you've invalidated the signature. In that case, generally, you need to update the signature or the design element will be unusable. You can only sign design elements (or any note for that matter) using the ID that's running the code.

Pay attention to which items in a note are signed. You can see this in the design note properties (Figures 1 and 2). Certain items are not signed because we need to be able to modify them without invalidating the signature -- whether an agent is enabled to run, for instance. If you are only modifying unsigned items, it's probably better to not update the signature, unless you're using an ID you know is privileged in your end users' ECLs.

The Icon Note
Each application contains one special design note which is used to store the database icon and also many of the database-level properties. The $Flags item in this note encodes many of these properties; others are stored in their own items.

You can quickly locate the icon note by its special noteID ( db.GetDocumentByID("FFFF0010") ). This ID isn't officially documented by IBM that I know of, and I don't guarantee that it'll never change, but it seems unlikely at this point that it would, as some of IBM's own code depends on this. If you want to be extra safe, use the NotesNoteCollection.SelectIcon property to build a collection containing just that note; however, this is not as efficient. See Resources for documentation of special note IDs.

Private Design Elements
As noted above, a "server private" design element has a $Authors item that both designates the owner and gives them access to edit it. Additionally, the $Flags item contains the character "V".

I'm not sure whether, if you just remove the Authors flag from the $Authors item, you can produce a design element that's private but can't be edited by the owner. If you know the answer, please post a comment.

Stored Forms
There is very seldom a good reason to use stored forms.

Several of the items from the Form design note are copied into the document, including but not limited to $Body, $ACTIONS and $V5ACTIONS, and everything containing LotusScript object code. In effect., everything that makes up the form design, plus everything in the document.

If you want to convert a stored form document to a regular document, there's code to do this in the Domino Designer help document titled "Examples: RemoveItem method". This code is sufficient to make the document behave like a normal document; however, it might be leaving behind some $Items that you could delete to save storage. Check the document properties to see whether you missed anything.

Shared Actions
Shared actions are unlike other shared design elements in that they are all stored in a single design note. You only see multiple Shared Action design notes in multilingual applications, in which case they should contain the same set of actions in different languages.

Each shared action has a numeric ID which is assigned at the time the action is created. When you include the action in the action bar of form or view, the form or view contains a reference to the shared action, and also a copy of the action. At runtime, the client will first try to locate the action information in the shared action note using the reference number; if not found, the copy may be used.

When you copy a design element from one application to another, if that design element contains references to shared actions, the shared actions are not copied automatically along with the referencing design element. This can cause problems if the application into which the design element is pasted also contains shared actions. Since the reference IDs of these actions may duplicate those of the shared actions in the original application, the pasted design element will now be referencing different actions than those that it originally referenced.

Shared View Columns
Shared view columns aren't shared in the same sense as shared fields and actions. They aren't referenced at runtime as you're using the view; they are simply flagged with a shared column ID which is used during editing. When you save a shared column, Domino Designer finds the views that contain that column and update them.

One problem you might run into: views' copies of shared columns can only be updated if the person editing the shared column has access to those views. Since view access control lists apply to all users, even Managers, it's possible they may not be able to update all views.

Also, the function to update views with the new column design, is part of the Domino Designer user interface. If you modify a Shared Column in some other way, for instance by importing DXL or accessing its items directly through a NotesDocument, the referencing views will not be automatically updated. There's no easy way to automate this.

Accessing Design Elements Using NotesDocument
With the information in the above section, you should be able to use the properties and methods of NotesDocument, NotesItem, and NotesRichTextItem, to do some useful things with design elements. The basis of this technique is the fact that any note, including a design note, can be manipulated by accessing it with the NotesDocument class. To do this, you start with the note ID or UNID of the design note, and use NotesDatabase.GetDocumentByID or GetDocumentByUNID. To get the note ID or UNID, the following techniques may be useful:

• NotesNoteCollection class lets you build collections of note IDs of design elements. The SelectXxx properties of the class let you narrow down the type of design element you want to retrieve, and the SelectionFormula property is useful for homing in further on design elements of that type with specific properties, by testing the values of the items in the design note. For instance, to find design elements with a name or alias of "Report", you might use the selection formula $TITLE = "Report". Testing the $Flags item in the selection is also occasionally useful (use @Contains here).
• If the design element is a view or folder, the NotesView.UniversalID property lets you retrieve the UNID.
• Using a technique described in the Resources section, it's possible to programmatically modify a Notes view to display design elements instead of (or in addition to) documents. This lets you access the design elements directly from the view using GetNextDocument, or via the NotesViewEntry object.

If a future version updates NotesNoteCollection class to include specific logic to select some of these new design element types, some of the options in the SetSelection function may quit working. That's because many of them rely on the "SelectMisc..." properties, which return lists of design elements of a given note class, whose $Flags values don't correspond to a type that NotesNoteCollection recognizes. If the class is updated to recognize additional design element types, those might be removed from the subset of elements returned by "misc" collections.

Once you have the NotesDocument, you can access the items using the NotesItem and NotesRichTextItem classes (the latter for the $Body item of a form). You can retrieve items for other binary types besides rich text, and they may be returned either as a NotesItem or a NotesRichTextItem, but in either case, you aren't going to be able to get any useful information about the contents from that. Even with such items, though, it's sometimes useful to be able to:

• tell that they exist and how large they are.
• delete them (using NotesItem.Remove method, for example).
• copy them from one design note to another (using NotesItem.CopyItemToDocument method), after deleting any item with the same name that you want to replace.

Accessing Design Elements with DXL
The third major method of accessing design elements, is by way of DXL (Domino XML). Every design element (with some exceptions which IBM is treating as bugs) can be exported into an XML description, which you can then manipulate and re-import.

This technique is often useful in combination with other techniques described here. For instance, IBM has a tool used internally that uses DXL to export design elements to get information about their action bars, and then uses the NotesDocument technique of the previous section to remove $SCRIPTOBJ objects that are unneeded (there used to be a bug that failed to delete them when the actions were moved or deleted).

The output of the DXL exporter (NotesDXLExporter class) can be directed into an XML parser object, such as the NotesDOMParser, where it can be examined or translated into another format. The NotesDXLImporter can be connected to a source of XML data (such as the output of a NotesDOMParser) and used to create or update design elements.

"Note Format" versus "Descriptive Format"
Some design elements offer you a choice of how they are rendered into DXL. The descriptive format, which is the default, is relatively easy to interpret; when you look at the DXL description of an agent, for instance, you might see:

A person can look at this and stand a fair chance of figuring out what they would see in the properties infobox if they opened the design element in Domino Designer.

The "note format", on the other hand, just lists the items in the note and their contents. If the item is binary, it's included verbatim using base64 encoding. Someone reading this XML output doesn't get any assistance in figuring out when the agent runs -- that's part of a binary item and you can't even pick out the strings from it to see the server name. The contents of $Flags are just reported -- not explained. In the note format, only rich text CD records are interpreted into a readable form; everything else is "raw."

Some design elements don't support a descriptive format (mostly resource elements at this point). In some cases, though, it make make sense to use the note format deliberately by setting the property NotesDXLExporter.ForceNoteFormat. The problem with the descriptive format is that it doesn't necessarily report every option (at all, or correctly), so if you export and import it you may be losing information.

The note format, on the other hand, is round-trippable without loss of information (with the possible exception of some of the more esoteric/newer options of rich text entities). It's a better choice if you want to archive a checkpoint copy of the design element. If you want to manipulate the DXL and re-import it, however, the descriptive format is far easier to work with in most cases.

Signatures
Design elements created via DXL import are unsigned. This is deliberately done as a security measure. Your code must explicitly sign them using NotesDocument.Sign. Listing 1 shows how to import DXL, then find all the imported design elements and update their signatures.

Resources
The file stdnames.h, part of the Notes C API toolkit, contains symbolic names for all the $Flags characters and many other special values used in design elements. You can download this toolkit from the Lotus Downloads page of IBM developerWorks.

"Make a Notes view list design elements" wiki article.

Lotusphere 2008 - AD307 Session examples from Rocky Oliver, includes an application to patch together Form design elements containing surveys with questions created by end-users. It's an example of creating complex design elements, without any use of Domino Designer.

This sample database from The View contains a LotusScript agent to automatically create a view with all the fields from any form.

Here's a tool to search a database (or all databases) for design elements that meet specified search criteria, including strings in source code and static text on forms.

For a glimpse of the future of design element programming, the Secret Agent project on openntf shows how to access design information from a sidebar plugin.

Special note IDs to find the icon note and other design documents.

See comments for additional resources suggested by our readers!

Sample Code

Listing 1: Importing DXL and Signing the Imported Design Elements

...

Sub SetSelection(ncoll As NotesNoteCollection, Byval strDesignType As String)