IBM®
Skip to main content
    Country/region select      Terms of use
 
 
   
     Home      Products      Services & solutions      Support & downloads      My account     

developerWorks  >  Lotus  >  Forums & community  >  Best Practice Makes Perfect

Best Practice Makes Perfect

A collaboration with Domino developers about how to do it and how to get it right in Domino

I'm on the team working to improve DXL. Our goal is to make it round-trippable as has been occasionally requested by developers, but it's a big job.  So, we're trying to figure out what part of that we should attempt for version 8.5.1.

NOTE: As usual, this does not constitute a promise on IBM's part to provide any specific functionality in any specific version.

Please chime in with your prioritized list of the top five or so specific DXL issues. An explanation of how those fixes would affect you/what you would be able to do that you can't now, would be welcome also.

Also, please comment on what full fidelity means to you. Is it sufficient if the design element or document is identical in terms of function and appearance, or do you need it to be identical bit for bit (except for the timestamps and signatures, of course), and why?

Thanks.

Andre Guirard | 17 March 2009 01:31:00 PM ET | Home, Plymouth, MN, USA | Comments (31)


 Comments

1) What does roundtrip mean?
Grégory Engels | 3/17/2009 4:23:32 PM

I would measure it so - take the mail template.

1. Export the template to DXL.

2. Import the DXL into a new application template (aka .ntf).

3. repeat with pernames.ntf

expected outcome:

a) it should export and import neatly with out errors or warnings.

b) a new application that is created using the new template can actually be used in the same way as a application created from the original mail template.

2) DXL - design versioning
Martin Pradny | 3/17/2009 4:27:02 PM

I wanted to use DXL for db design versioning- e.g. dump DXL of database design and store it in SVN. For this to work seamlessly would be required that those elements imported back work 100% and if not touched after import back the next export would be the same (if we strip modification times and similar fields).

So it would not require 100% bit match beween Notes stored design elements, but match in exported files would be great, so SVN doen't version it over and over again.

Martin

3) Roundtripping the light fantastic
Nathan T. Freeman | 3/17/2009 4:43:31 PM

Top 3 items:

1) Representation for Embedded Editors

2) InViewEdit switch for view columns

3) Import of TrueType font settings for view columns

How to define full-fidelity: Take any design provided by a customer or business partner. Export it to DXL. Change all text color elements to blue. Re-import it. The design should work identically, except that all elements should be blue (instead of whatever color they already were.)

Why is that the metric? Because the point isn't to be able to successfully export & re-import to any external representation. We can already do that by using NoteFormat and exporting the Rich text in raw format. We do this for 100% reliable version control and rollback in Lotus 911's dev labs today.

The point is to get an external format that can be programmatically manipulated without platform-specific code. The only way to prove that out is to manipulate it in some way.

The process needs to work on every template that ships with the product AND every template published on OpenNTF.org. That should be the starting test set of templates.

4) Non-Crashing
Bill | 3/17/2009 4:48:01 PM

Weve worked in projects where we've exported the design of 36,000 templates and analysed those templates for code - Lotusscript, Formula and java - and then looked for code weaknesses.

1. DONT CRASH

Its very very easy to redbox a client or server whilst exporting design based DXL, or exporting funky rich-text content.

If it cant process - tell me it cant process. But dont crash!

Ironically, we found that 6.0. didn't crash as much as 6.5, or 7.x. No, that doesn't make sense to us either.

2. All design elements

So outlines, navigators, and those funky replication pages from the location document.

I know no-one uses navigators anymore -horrible things. Well, unless you look at most customer sites where they might have 10+ year old applications.

3. Full Fidelity

Sorry. Shooting for the moon here. If I export a design element, and then reimport it someplace else it should work.

4. Hotfixes

Look, if you have something that might work, send us hotfixes. Even private ones. We're possibly as enthusiastic as you to get this working. ;-)

We're actually starting this project up again soon and we can probably identify items that crash on DXL export design. Want these items sent back to you brainacs as test data ?

(Some folks export design elements as binary items. Sorry. We dont just want the binary representation of a design element. We could just take the original document and export all items. )

Hope this helps,

----* Bill

5) round trip
Alan Bell | 3/17/2009 6:12:39 PM

round tripping is the thing that would make DXL useful. I would say export a database to DXL, import it to a new database. Export it again. The two DXL files should be identical apart from timestamps and signatures. The two databases should be identical to use. Global changes of colours and fonts are the kind of things that DXL should be able to do but isn't safe to do at the moment.

6) The Holy grail, round trip
Kurt Higley | 3/17/2009 6:52:23 PM

I have an application that I wrote to be able to pick design elements(aka, dxl). and create databases. I had originally wrote it to act as a code library(i.e.., versioning, checkout, relationships, etc). Some of the things don't function as expected. Its been a while since I looked at it since the roundtrip wasn't 100%, but here are a couple off the top of my head:

1. Images. Depending on if the image was copied directly into the form or if insert as an image resource(can't remember if it was really because jpg vs gif).

2. Frames/Framesets dont retain their formatting(ie. border settings)

3. Database icon import

Now I haven't had the opportunity to revisit these on anything later then r6 so I apologize if these have been addressed.

Round trip, how sweet would that be!

7) Those are okay
Nathan T. Freeman | 3/17/2009 9:29:37 PM

@6 - From my experience over the last year, all those items work just fine in the 8.x codestream.

8) Non crashing - feature stable
Stephan H. Wissel | 3/17/2009 10:20:04 PM

- When exporting DXL it should yield the same DXL result all the time.

- It shouldn't crash

- Export DXL, create a new NSF by importing, export again: should be the same as the original DXL (minus the modified information of course). Changes in the binary stuff are ok.

- Whitespace neutral: XML requires that any whitespace doesn't constitute a difference. In DXL it does (linebreaks etc). Need to add markup elements for spaces/linebreaks

- Additional attributes (e.g. form's custom mime-type) should be attributes not just notes items.

- All attributes covered please.

9) My List.
Bernfried Geiger | 3/18/2009 2:00:23 AM

1) Finally commit once and forewever where you go (it would be not an excellent use to have it even perfect in 8.5.1 if we can't rely on it staying perfect forever - whatever perfect in this case means.

2) 'Somewhat perfect' and 'we are getting there' may sound good enough to management but is far away from good enough for risk free heavy investment on our side.

Please understand that there is huge potential in this area for many uses. I have lost already years in not using it because of waiting for the one more relase when there is both, commitment that it is complete and commitment that it stays complete. This is not only me, but a huge part of the community. The way you ask looks like 851 might again not going to be it.

My wishes:

- I would like to have a perfectly complete export (to replace Synopsis by free tools from the community, or we need both, a complete and efficient synopsis again able to do at least discussion and mailtemplate 100% complete all design elements at the same time).

=> Motivation is ressearch within an nsf for whatever I think I should do there. Both, Design Elements and documents. If something signed or encrypted would be a non documented block, this might be o.k. to some extend, but please include it.

- I would like to have a version of an export (in an ideal world the same of above) that will for 100% sure import again - as the way to change a color or a font is one use case, another one to change a formula in a view or a form or as one to add a design element - things like that.

- Importing the somewhat modified dxl should only change the time stamp and autor of a design element if the DXL area of this design element has changed (Give me a chance to see efficiently if I messed up something using Designer at the end of the process).

- I like to see the commitment that this three things will stay.

10) Web service design elements
Jaeger | 3/18/2009 3:19:04 AM

Haven't seen in 8.5 but before the code in webservices was not representable in DXL.

Would be great to create a default kit of webservices that could be easliy modified in xml editor and imported into any database.

11) put it in the QA process
Alan Bell | 3/18/2009 3:26:51 AM

also take a non-technical approach. No IBM template gets tested until exported to DXL and re-imported. No new feature gets added to Notes until it round trips with DXL. I could understand some rusty old feature not being implemented properly but things like colour columns came out several point releases prior to their DXL. That shouldn't happen.

12) DXL export improvements
Daniel Bjarsch | 3/18/2009 4:15:15 AM

1. Better error trapping - if it does´nt process - please tell me why (and dont crash :)

2. Export a view with both content and design (column headers)...

3. A style sheet for rich-text would be nice!

13) Thanks for asking and and excellent use of irony by a non brit !
Sean Cull | 3/18/2009 4:37:56 AM

1) Where there is a hiccup with a DXL process it should fail gracefully such that it can be caught in an error trap and must never ( within reason ) crash a server.

Why - beacuse we cannot use this mechanism for business processes if it only works sometimes or risks crashing customers servers

2) If there are specific known limitations with DXL then they should be documented and easily found by developers.

Why - there is too much Fear / Uncertainty / Doubt to take overcome the positive comments by excellant developers like John Head and Nathan.

3) A clear statement as to whether the dxl based ( ? ) search and replace in DDE is trustworthy

Why - customers entrust their applications to us, we must be sure that we have not corrupted them ( it has happened to me before with DXL pre DDE )

4) everything said by @3 Natahan and @4 Bill. Hotfixes are one of our aims too

14) DXL
Julian Buss | 3/18/2009 5:44:09 AM

all my wishes are named in the previous points. I just want to add this:

1.) We need DXL to do version control via SVN. And you (IBM) need this for doing version control via SVN, too.

The Domino Designer team is aware that many devs want to use SVN at least for version controlling LotusScript/Java code.

2.) Round-tripping definition: we take the most complex application we have (our CRM suite with lots of code and very complex forms), exporting it to DXL, creating a new DB out of this DXL and everything should work as before.

3.) I understand it's a huge amount of work. Therefore, my priorities are:

a) Code libraries (LotusScript, Java, JavaScript)

b) XPages (should be relativly easy since it's XML)

c) Forms

d) other design stuff

e) Notes documents

15) My priorities for DXL
Devin Olson | 3/18/2009 6:07:13 AM

1) Don't crash the server. Throw an error if you must, bail on the process, or whatever is necessary. But never, ever crash the server.

2) Gracefully respect the security model. If an element (or part of an element) should not be exported due to ACL / Readers (I'm referring to both design and data here) then don't export it -BUT make certain the rest of the requested elements do export properly, and that the DXL remains valid.

3) Full-Fidelity round trip for design elements. I should be able to take any template (to which I have appropriate rights -see #2) and export the design DXL. I should be able to then create a NEW blank template, import said DXL, and replace the design of of any database (that was using the original template design) with that of the new template. After replacing the design, the new database should operate and LOOK AND FEEL exactly the same as before. Being able to perform customizations (such as changing font colors or operational code) is a given; however unless I can reproduce the behavior and appearance of the original application EXACTLY there is no point in doing so. The reason why should be obvious -how can I know if my changes are working when other things change?

4) Hotfixes - Bill already covered this @4, but I want to reiterate his point. If you have something "almost" working give it to us to play with. We (collectively) are the best design lab resource you have; we are very, very good at what we do, we love this product and want to see it grow, and we want to help. LET US HELP YOU.

The last word of your post was "Thanks". You're welcome. More importantly though, THANK YOU for asking for input (and putting up with our bitching) on this "occasionally requested" topic. It means a lot to me personally, and proves that your level of microgivashits is very high.

16) Embedded views in funky forms
Jan Schulz | 3/18/2009 8:06:16 AM

I want my embedded views, which are in quite complex forms (tables in tables, ...) work. Finaly a tool which reliable can set my EVs for me.

17) Can we get some feedback on specific design elements
Dick Annicchiarico | 3/18/2009 8:19:56 AM

The feedback so far is tremendous and definitely keep it all coming. But I'd also like to request some more specific feedback, driven by a reality we must all accept -- no way we can get 100% fidelity in 8.51, which is a short-term window of only perhaps a couple of months of development time. So could we get some feedback on the specific 2 or 3 design elements, or even better, sub-features of said design elements, that would be a great start and a great help to you in 8.51?

18) DXL priorities
Erik Brooks | 3/18/2009 8:31:35 AM

Here is how I think DXL round-tripping should be tackled strategically:

#1: Don't crash - This is obvious.

#2: Support full round-trip fidelity, WITHOUT RICH TEXT MODIFICATION, of anything.

Forget about rich text for a moment. There are attributes (checkboxes, properties, etc.) of design elements that still, in 8.5, can't be programatically manipulated at all. This is a fairly sizable deficiency that needs to be addressed.

The biggest challenge for complete, *changeable* round-trip fidelity is (and has been) rich text. Yes, rich text is nasty, and it touches many areas of Notes/Domino.

For now, if the exporter can't figure an RT stream out (or it's using some rt feature that isn't yet round-trippable) then just export a <dirtyrichtext> tag that gets exported with a byte-stream inside of it that we couldn't (or shouldn't) touch. That way we can at least round-trip a design element/doc/db and toggle a few attributes and be on our way.

Get this one down and you've at least made round-tripping a reality, even though all rich text isn't yet modifiable. People will at least be able to modify their design element attributes. That seems to be a recurring theme here and I would guess is a 90% use-case for what people want out of DXL.

#3: Gradually provide greater and greater manipulative round-trip fidelity of rich text.

Once you've got #2 down (and you maintain it going forward), then you have some goals for point releases: gradually improve rich text export/import.

E.g. in 8.5.1, very simple rich text may export fine, but complex rich text may cause the exporter to go "hmm... better use the <dirtyrichtext> tag here." Over time you can refine its capability so that it is resorting to the <dirtyrichtext> tag in fewer and fewer cases.

Hotfixes / BP testing is a great way to facilitate this sort of incremental effort.

19) Might have <webservice> for you to kick around
Dick Annicchiarico | 3/18/2009 8:33:56 AM

Re @4 Look, if you have something that might work, send us hotfixes. Even private ones. We're possibly as enthusiastic as you to get this working. ;-)

Re @10 Haven't seen in 8.5 but before the code in webservices was not representable in DXL.

Re @15 Hotfixes - Bill already covered this @4, but I want to reiterate his point. If you have something "almost" working give it to us to play with.

Well, we potentially have some good news here. In late '06 - early '07 I did an initial implementation of <webservice> in DXL. It actually was a quite tricky thing to implement (never could have done it without the Web Services team - Steve Nikopoulos, John Grosjean, and Tom Carriker) and we didn't have the QE resources at the time to complete it. So it has sat, "roped off" in the code. We could, perhaps, dust that off and let you try it out in a first iteration. We'll need to weigh demand for it vs. demand for other features, of course.

If you're wondering, there isn't anything else like that that is implemented but roped off.

20) DXL part 2
Erik Brooks | 3/18/2009 8:44:51 AM

@17 - Dick, thanks for reading!

For a priority of design elements, I would marry (1) amount of usage with (2)the possibility for 100% or near-100% fidelity without richtext.

To me, this means:

#1: Views (everybody uses them, *should* be 99% round-trippable for nearly everything)

#2: Script Libraries (these should be trivial to accomplish 100% round-trip with EVERY attribute)

#3: Agents (same as script libraries)

#4: XPages (also should be trivial to 100% round-trip if not already, and you're about to have the whole world using them)

From there you can move down the rest of the list according to usage and possibility-of-high-fidelity. Web Services are likely an obvious next choice.

Forms and Pages, for example, would likely be trivial to get working 100% without modifiable rich text (e.g. my <dirtyrichtext> idea above).

21) DXL Round Trip Priorities
Kevin Pettitt | 3/18/2009 12:15:33 PM

1) If I understand correctly, there is some sort of byte-stream style of DXL export that guarantees full fidelity roundtripping, but of course with the compromise that such bits won't be manipulable. If so, flag every one of these areas (not just richtext) and isolate them. Then as @18 Erik Brooks says simply open up those areas bit by bit as the effort proceeds.

2) Don't crash. This probably involves among other things modifying all the DXL Lotusscript classes to be sensitive to whatever new tagging model is set up to isolate byte-streamed elements, so if I try to manipulate the "wrong" node because I'm an idiot, the worst thing that happens is that my error trapping logs the problem (ok, so if I don't do error trapping, the code craps out but does so gracefully).

3) Document every one of the "gaps" identified in #1 (echoing @13 Sean).

4) Element priorities for me are

- Agents, in all their permutations, particularly scheduled agents. This seems to work without a problem now in my testing but Nathan I think noted that problems occurred if you include a document selection as part of the agent. If I know that's the *only* part that fails I can simply avoid it when developing specific functionality and can control which agents are affected by my DXL code.

- Frameset border colors/fonts, action bar properties in all elements, embedded outline colors/fonts, image resources (that feed actions, action bars, and outline entries) including the ability to pull in new or replace existing images from the file system, and view fonts/colors (titles, alternating row colors, etc.) as these are the keys to easily "reskinning" a traditional Notes application such as the templates provided. The actions and views also represent a huge effort if modified manually so this is a big savings. If I missed anything somebody can chime in.

- View events (e.g. PostOpen, etc.) which might already be supported but I'll mention anyway since they are key to injecting standard code (e.g. prevent pasting) into all views in one go. Again, a big effort savings compared to same capability in a form.

- Form design - first the colors/fonts for text, then colors for tables, then other table attributes, then embedded stuff, then form events such that you can search/replace, detect whether code exists in an event or not, etc.

That's probably enough to fill the 8.5.1 development schedule. Thank you for asking, and good luck :-).

22) DXL thoughts
Brian Miller | 3/18/2009 1:08:32 PM

The only thing that hasn't been explicitly mentioned is that if you export DXL today, (at least in 7, I haven't looked to see if it's already different in 8), the exporter puts in line breaks pretty much wherever it feels like it, to "enhance human readability". This needs to not happen at all. At a minimum, we need to be able to turn it off. Also, to echo @8 Stephan, there needs to be entities or tags for line breaks as well.

As far as priority, I'm also dying to use SVN on Domino code. If I check out a block of DXL, I should be able to build that straight into a database, and start working on it.

So, for me it's:

1 - Include everything, even if deprecated. Use hexed binary for navigators and the like if you have to.

2 - Don't miss any details. If I have to program it when I write an application, it needs to get represented in the DXL *somehow*. Literally walk through every property box for every element, and make sure that all the things that you can set are reflected in the DXL somewhere.

3 - Nthing "Don't crash".

23) Send us your problem dbs
Dick Annicchiarico | 3/19/2009 9:43:15 AM

The above discussion includes at least one offer to send us your "problem" databases so we can reproduce and analyze the areas lacking full fidelity and work toward a solution. I'd like to take you up on that offer.

Send dbs to dannicch@notesdev.ibm.com

However...

We have to tread a bit carefully here to maintain proper business conduct guidelines. I cannot accept anything that might be copyrighted by your company or contains your company's intellectual property, etc. Thus, the best scenario is if you can reproduce a problem in a simple unencumbered sample. If not, I may have to get our legal dept involved and we may have to look into some sort of NDA. Would really like to avoid that hassle if possible.

Another possibility is a step-by-step cookbook to recreate the problem.

I can't promise to look at all of these, but I will do my best. Over time, this should allow us to create a list of "real-life" issues that are hindering progress. Ideally, as we move on with this effort, we'll have our own processes to determine everything that lacks full fidelity, but it makes great sense to attack the actual problems you're seeing first, in a priority order.

24) Not just design elements
Richard Schwartz | 3/19/2009 2:50:59 PM

While most of the comments here center on design elements, I need to emphasize that DXL needs full fidelity for documents.

For priorities:

1. No crashes, and no leaks -- in high volume applications using huge databases containing individually huge notes.

2. Everything exports; that includes encrypted content even if you don't have the key; and if something within a note (e.g., encrypted data) has to be exported as base64/binary, that's fine, but the rest of the note should still be exported as the normal structured tags.

Thoughts on how to validate:

- Take some huge, old, mailfiles and applications. Look for databases that contain a wide variety of notes with custom forms, stored forms, collapsible sections, signed & encrypted (both Notes PKI and S/MIME), ole objects and layout regions in rich text, etc.

- Export them as DXL

- Also use the new archiving API to export the notes in the databases as binary files.

- Import the DXL into new databases.

- Use the new archiving API to export the notes in the new databases as binary files.

- Diff the corresponding binary files.

- Re-export the new databases as DXL.

- Diff the corresponding DXL exports.

I do realize that the diffs would have to be intelligent about internal things that aren't expected to be preserved and don't matter with respect to fidelity; but something like this approach is probably the only way to automate fidelity testing of DXL.

25) Fonts
Mike Miller | 3/21/2009 8:11:44 AM

A while back I commented on one aspect of DXL brokenness: { Link } but here's the gist of it.

1. Similar to the HTML tags that the Domino HTTP engine generates, if you use the default fonts (Default Serif, Default Sans Serif, etc.) you won't see a node in the DXL.

2. If you use a font other than the default font, you will get a node along with a "name" attribute, but will only get a "size" attribute if use a size other than the default size, 10.

26) Signing
Jack Zoutendijk | 3/23/2009 6:34:19 PM

I am currently using DXL to create new design elements (Forms) in a database. The form has some buttons that hold Lotusscript code to perform certain actions.

Unfortunaltey, when a user creates a document with this form, when he/she clicks the button and ECL Alert comes up but with a signer name - No signature -. It would be nice to have the DXL importer to sign the complete form, including all elements, not just the form itself and leave the buttons/actions unsigned.

For the rest, I think everything I wish for has been mentioned already.

Thanks.

27) For Me It’s All Or Nothing
Peter Presnell | 3/26/2009 1:56:37 AM

I now part of the purpose for the original post was to get specific issues to prioritize but from where I sit it is either 100% or it is for all intents and purposes zero %. The value of DXL falls away very rapidly if even the smallest things dont roundtrip. I support one of the previous comments. Make it a requirement that EVERYTHING moving forward must be 100% DXL supported and then move to clear up as much as possible. At least the problem will not be getting worse. Stopping the server/client crashing should perhaps be the highest priority as that wastes a lot of time doing restarts.

28) My top list
Martijn de Jong | 3/26/2009 9:08:56 AM

Seeing that DXL for version control needs full fidelity and that this apparently is not possible for 8.5.1, please focus on these items (much the same as @20, just in a different order):

1. Agents

2. Views

3. Script libraries

4. Xpages

29) source code control
Nick Radov | 3/30/2009 4:31:34 PM

Our primary concern is with source code control integration. I want an easy way to treat each design element as a separate module in our source code control system. We happen to use CVS, but ideally it should work with any source control system that can integrate with Eclipse. That will make it practical for multiple developers to work simultaneously on one template.

The key thing for source code control is that it operates on a line basis. For branching, merging, and reporting changes it looks at what lines were changed, deleted, or added. So Domino DXL should generate lines of a reasonably short length, and a particular line should only be changed if the programmer has made a functional or appearance change.

30) my favorits
Oliver Schumann | 4/2/2009 4:03:31 PM

1. Agents (export, adjust schedule / server ..., import)

2. Agents (export, adjust schedule / server ..., import)

3. Agents (export, adjust schedule / server ..., import)

3. Views with all properties

4. Forms with all properties

 Add a Comment
Subject:
   
Name:
Comment:  (No HTML - Links will be converted if prefixed http://)
 
Remember Me?     Cancel

Search this blog 

Disclaimer 

    About IBM Privacy Contact