I have a system where we import huge amount of data using a CSV file. This CSV file data can vary from 500KB to 50MB. The importing also validation and rollups. The code was developed in R5 (LotusScript) and still being used. But the issue is that the import is taking too much of a time (approx 5 hours for a 40-50MB file). Also, because of the size of the file and the time taken to import, we have kept only one Agent Manager as haveing more than one, causes the import to time-out the session ot hang the server.
My question(s) to improve the performance are
1. Is there a better suited format other than CSV for importing the data? Will importing the data using XML help?
2. Are there any performance enhancing classes / code that can be used for the import ?
3. Is LotusScript best suited for importing data or should I be using Java?
Any idea for importing data and enhancing performance will be much appreciated.
Note: There are around 200 import files that need to be imported every month. So desperately need the performence to increase.
Feedback number WEBB959AAN created by Abhinav Rawat on 02/25/2013