IBM®
Skip to main content
    Country/region select      Terms of use
 
 
   
     Home      Products      Services & solutions      Support & downloads      My account     

developerWorks  >  Lotus  >  Forums & community  >  Best Practice Makes Perfect

Best Practice Makes Perfect

A collaboration with Domino developers about how to do it and how to get it right in Domino

When a document is created, it's assigned a unique ID based on the creation time. However, the clock ticks over about every .01s, so if you create two documents in a very short time, it gives the second one a future creation time. Normally you don't notice this because this time is just .01s ahead, but if you create many, many documents in a short time, each has to have a unique creation time, so it pushes the time farther and farther forward, until it can be off by hours -- enough that users who create documents in the morning may find they're shown as being created early that afternoon, for instance.

If you modify an existing document, you aren't assigning a new ID, so I believe Notes doesn't find it necessary to advance the clock to prevent two documents having the same modified time. But if you create a document, and then modify a document (even if they aren't the same document), we want the creation time and the modified time to be in the right order. So if the creation time has been pushed ahead, this will also affect the modified time of documents you edit.

Mass document creation most often happens because of an agent that deletes all the documents overnight, and creates all new ones from some outside data source. Another effect of such an agent is to create many, many deletion stubs -- many times more than you have documents. There are many reasons you don't want to do this; weirding out your creation/modification times is one of the more minor ones. It's much better to synchronize documents with an outside data source by locating the corresponding old documents and only updating them if you need to.

The algorithm for doing such an update is shown in this download. It contains an agent named "5. Replicate Pirates," with a reusable replication engine that reads one record at a time from two LC LSX result sets and compares the keys and data values to decide whether or not there's a difference between the records, synchronizing the data one-way. This is possibly slower than just deleting and re-creating, but possibly not, since those deletion stubs tend to hurt performance.

Andre Guirard | 24 June 2008 11:16:38 AM ET | Home, Plymouth, MN, USA | Comments (7)


 Comments

1) Creating documents in the future
John Kingsley | 6/24/2008 3:38:31 PM

Likewise, if you control the incoming data set, it is much easier to indicate adds, changes, deletes in this data set. This information should be tracked by the sourcing application.

2) Creating documents in the future
Stephan H. Wissel | 6/25/2008 3:53:48 AM

Good summary,

actually the date creep applied to the update date as well. However you need quite some data to reproduce that:

- take a DB with 250k documents

- use a simple agent e.g. FIELD test := "1"

- use a fast machine

Updated date will flow into the future.

3) Creating documents in the future
Tripp Black | 6/25/2008 10:20:08 AM

We've used LSX for some years now.

The gotcha is always handling deletes, especially when you have to "replicate" both ways. On the DB2 side, we usually have a "delete table" where deleted records in the "main" table are added and then cleared after the deletes are replicated to Notes. On the Notes side, we do basically a "soft" delete (with a hidden field), and hard delete it after the delete is sent to the other side.

This works but is convoluted, a documented/sanctioned way of programatically getting a handle on a deletion stubs on the Notes side would be a smoother way.

4) Creating documents in the future
This "feature" has cost one of my customers lots of troubles | 6/26/2008 3:47:28 AM

This is really a very unfortunate "time creeping" feature, which has cost one of my customers lots of troubles.

Why on earth has this "unwise" way of creating UNID not been fixed long a go?

brgds Jesper Kiaer

5) Creating documents in the future
Andre Guirard | 6/26/2008 11:11:56 AM

> Why on earth has this "unwise" way of creating UNID not been fixed long a go?

Dear This,

I haven't talked to the the people who are looking at this, but I'd guess it's a tough problem. This UNID is the created date/time in the current architecture; I don't think we store the created date/time separately. We represent date/time values in a certain way which is based on .01 second intervals. There's no other really good way to generate values that have the desirable properties of being translatable to a date/time which is also the created date/time, being in order for a binary sort, and being guaranteed unique in that replica once we do the clock adjustment for creating multiple in the same tick (we go for uniqueness across replicas by combining it with the database ID, which isn't visible in the UI).

I think in large part, the problem is that the original designers of the Notes architecture, and of the Windows OS and hardware, didn't really take into account how fast computers were going to get. The clock on Windows PCs ticks over at a certain rate -- can't make it go in smaller increments. We've got gazillions of date/time values already stored. If we tried to store them at a greater precision, we'd need more bits to store them in, or we run out on the high end.

6) Creating documents in the future
Jesper Kiaer | 6/27/2008 7:00:33 AM

Hi Andre

Thanks for the response!

I agree it is very difficult to foresee what may happen in the future so sometimes decisions made, may some same odd or wrong later in time.

I do not think the is the case here.

A UNID sole purpose should be, to be unique. The creation of the UNID should not be dependant on "some information" should be extracted from it, like the creation date/time.

The idea that that only one document may be created in "a time tick", what ever that size may be, is wrong.

My customer experienced this the hard way on 90 servers worldwide. Setting a db property (unread marks I think) caused 100.000 of documents to be modified and time crept a week in the future completely messing up replications etc. Lotus Notes/Domino was not a popular platform the following weeks...

Hopefully it will be fixed in future a release.

brgds Jesper Kiaer (This)

{ Link }

7) Creating documents in the future
Joseph LeMay | 7/11/2008 4:19:25 PM

amen, delete and recreate is bad, much better to update existing documents. I just went through this, and, over a period of a few weeks I ended up with hundreds of thousands of deletions stubs, all of which need to replicate to all replicas. So I changed the replication cut-off from the default 90 days to 10 days to get rid of all of these stubs. Then, if someone hasn't replicated in 10 days, and they go to replicate, I get old documents creeping back into the database. Bad design on my part; I redid it to update the documents I already have.

 Add a Comment
Subject:
   
Name:
Comment:  (No HTML - Links will be converted if prefixed http://)
 
Remember Me?     Cancel

Search this blog 

Disclaimer 

    About IBM Privacy Contact