Document search by integrating IBM Forms Experience Builder view responses with Ajax Solr

Introduction

IBM® Forms Experience Builder is an easy-to-use Web application for data collection. With its Web-driven user interface, it provides a secure, dynamic, and intelligent frontend to on-demand business processes. Apache Solr is the popular, ultra-fast, open-source enterprise search platform from the Lucene project.

By integrating the two products, we could obtain a highly reliable file-search mechanism that can be used for all data-collection Web applications, using Forms Experience Builder for the user interface.

This article provides a brief overview of Forms and Solr technologies and illustrated instructions on implementing a real business scenario. It then explains how to configure and install both products, how to integrate them to obtain an intelligent search of Forms Experience Builder view response documents, and how we can present it on the Web, using Ajax Solr (a JavaScriptTM framework for creating user interfaces to Apache Solr.)

Forms Experience Builder

IBM Forms Experience Builder provides the ability to build forms by dragging and dropping items from a palette of choices. IBM Forms provides a secure, dynamic, and intelligent frontend to on-demand business processes.

It is quite easy to create and deploy a form using Forms Experience Builder. Just install the IBM Forms Server - Forms Experience Builder and we can start building an interactive, easy-to-use application to collect data or gather user feedback, simply using drag and drop.

Apache Solr

Apache Solr is an open-source enterprise search server based on the Lucene Java search library, with XML/HTTP and JSON APIs. It consists of hit highlighting, faceted search, caching, replication, debugging and a Web administration interface. Apache Solr is a scalable, highly available, easy-to-maintain search solution.

It can be used with any Java Servlet Container, but it also has a built-in server called “jetty” that is easy to install and use, so we use it for our demo. Download and install Apache Solr version 3.5.0, to try out the demo.

To install it, you just need to extract the zipped folder and start using it. It has schema.xml, which describes how to use different Solr features, we need to add document data fields to the schema which are used for searching.

After installing IBM Forms we must create an application that lets us use a real-life retail-store business scenario in which we create a customer product purchase behavior feedback that is filled by retail shop owners.

Implementation

Creating Forms Experience Builder

1. First we create a form, Retail_Product_Owners, with the fields shown in figure 1.

Figure 1. Retail_Product_Owners form


2. After filling in all the fields, you should see that, at runtime, the backend View Response page of the application looks like that shown in figure 2.

Figure 2. View Response page

Using Experience Builder REST API to fetch response data from Forms Experience Builder

Now we use the Experience Builder REST API to fetch the response data from Forms Experience Builder, using cURL to fetch the response data. First, you need to install cURL on your system; on Windows, set up cURL as follows (see the attached Curl_setup folder):

Curl involves two further stages to access data:

(1) Log in to the form. You need to have access to the Forms Experience Builder application in order to fetch its view responses, and you need a security token before fetching any data. The security token are cookies that are provided after log in:

a) Extract the setup folder and go to the cURL directory.
b) Delete cookie.txt, if it already exists, and run the following command from directory

curl --insecure --cookie-jar cookie.txt –data "j_username=some_user_name&j_password=some_password" --output null.txt

where https://projectnitro.victoria.ibm.com/nitro is the Forms Nitro forms link where I have created my data collection form application.

c) Check the cURL installation folder; you'll see that a text file cookie.txt has been generated.

(2) List the records. To list data from a given application, the application ID and the form name must be known:

a) If you open the Forms Experience Builder application and check its link, it will be given as

(see figure 3).

Figure 3. URL link

 

where “449effa4-b9aa-4561-89d6-28cb2e06011c” is the application ID, and “F_Form1” is the form name.

b) We need to convert it into the following format, replacing “app” with data, and removing “form=”, as follows:

c) Finally, issue this cURL command, and at the end of command, add the format in which you want to retrieve the data; in this case, text/xml for xml output:

curl --insecure --cookie cookie.txt > output.xml

The above command fetches Forms Experience Builder content into XML format and stores it in output.xml, as shown in figure 4.

Figure 4. Output.xml file

Converting .xml output from Forms Experience Builder to input-data file format

Now we convert the .xml output from Experience Builder to an input-data file format for Solr, using a Document Object Model (DOM) or Simple API for XML (SAX) parser. To do this, we must take the cURL xml output and convert it into the Solr input format (a file containing number of documents):

<add>
<doc>
…..
</doc>
<doc>
…...
</doc>
</add>

We can use a Java program to do that. The program I used is in the attached parsing_java_program file. It gives us output as a file having a number of documents that we need to search later and will be used as the input file for Solr (see figure 5).

Figure 5. SolrInputFile.xml

Using Apache Solr to index the file and perform document search

To do this, use these steps:

1. Extract apache-solr folder and go to the apache-solr example folder:

C:\apache-solr-3.5.0\apache-solr-3.5.0\example

2. Run “java -jar start.jar” from a command prompt; this starts Solr at port 8983 on your system (see figure 6).

Figure 6. Solr Start command




3. Check the link, http://localhost:8983/solr/admin/; the page in figure 7 should display.

Figure 7. Solr Admin (example) page



If you see this page, then Solr has started properly, and we can put the Solr input file containing the documents shown in figure 5 into the Solr exampledocs folder and index it in Solr, using post.jar.

However, before doing that, we must configure schema.xml (which is in the C:\apache-solr-3.5.0\apache-solr-3.5.0\example\solr\conf directory) according to the data fields in documents, changing the schema in three places:

• Provide field type. Check whether the already-existing field types can be used, or define your own field type, if required.

• Provide field values. Add the fields to the schema, remembering that field names should be the same as defined in the Solr input file, as shown in figure 8.

Figure 8. Add fields to the schema

 

NOTE: We define a unique field as required=”true” and, if that field is missing in that file, Solr will show an error while indexing. Similarly, for whatever norms we are defining in this field definition, if they are not met, we will see errors while indexing. The errors are implicit, and therefore we can correct them by changing field properties.

• Provide uniqueKey field value and default field name and operator for query parsing. We can define these values as shown in figure 9.

Figure 9. Provide unique Key and default search field



We can index the file by issuing the command, “java -jar post.jar filename.xml”, from a command prompt in the Solr exampledoc directory, which will yield the output shown in figure 10.

Figure 10. Indexing command output

Performing basic Solr searches

We can now do basic Solr searches, using the Solr admin console. In this section, we look at a few examples.

• To check for documents created by a customer named “Tom”, we can do query search as http://localhost:8983/solr/select/?q=CustomerName:Tom, which yields the output shown in figure 11. The number of documents found is 1, and all relevant information regarding the number of orders and product lines ordered is fetched.

Figure 11. Search results for Tom

• To check which product line has orders between 0 to 35 within a month, assuming the data collected is for a specific month, we can do a query search, using the query http://localhost:8983/solr/select/?q=NumberOfOrders:[0 TO 35], as shown in figure 12.

Figure 12. Query results for orders between 0 and 35

 

• To find which product line has the maximum number of orders or sales from a single customer, we can do a query search such as http://localhost:8983/solr/select/?q=*:*&sort=NumberOfOrders desc&rows=1 (see figure 13).

Figure 13. Query results for maximum orders

• To find the number of customers ordering various products for a particular product line, use the facet search query, http://localhost:8983/solr/select/?q=*:*&facet=true&facet.field=ProductLineID (figure 14).

Figure 14. Number of customers ordering products for a product line

• To find customer Client ID's or Client Code purchasing a particular product, that is, to track a particular customer purchase behavior, use http://localhost:8983/solr/select/?q=CustomerID:224L (see figure 15).

Figure 15. Track customer purchase behavior

Using Solr document search with Ajax

Here we use Solr document search with Ajax to perform real-time dynamic document search over the Web and supporting data mining for various business scenarios.

The Web is the largest collection of electronically accessible documents, which makes it the richest source of information in the world. The problem with the Web is that this information is not well structured and organized, so it can be difficult to retrieve.

Search engines help in accessing Web documents by keywords, but this is still far from what we need. To use the knowledge available on the Web effectively, we can achieve this only by using data mining techniques.

We have various data-mining business scenarios, for example, the following departments:

• Sales. Sales departments need a data-mining technique to analyze customer buying behavior; for example: How does Amazon.com determine which products to present to you in the hope of up-selling or cross-selling them through mining customer purchase data that can be gathered and analyzed on the Web dynamically?

• Production. Customer project plan files and documents need to be tracked and mined, to extract various plan changes at various stages of product development so that we can decide on resource efficiency, limitations, and qualitative effects on customers.

• Accounting. The Accounting department needs to provide numbers on the services billed, discounts given, the amounts actually collected, and receivables aging statistics (the amount receivables change their patterns over time).

• Marketing. In marketing we need to predict the likelihood of a customer reacting to a shopping catalogue or a special discount in a direct mailing. Again, data mining and document search is used to predict that.

• Legal. Law enforcement uses "real" data mining for many different purposes, such as crime pattern recognition to solve the crime sprees, predicting how to deploy the limited resources to optimally prevent crime, and the emerging field of counter-terrorism or using data mining for Intelligence-related activities to prevent terrorism.

We can actually use Web-based data collection via Experience Builder and can further use Ajax Solr to do runtime data mining. To use Solr with Ajax, we must write a simple jquery ajax application, using the AJAX Solr JavaScript library. For more information, refer to “evolvingweb / ajax-solr”, and for the architectural concepts of APIs refer to “Architectural overview.”

The attached sample project demo, Ajax_Solr_Web_Demo_Application, basically uses Ajax JavaScript API's to fetch search results from Solr. This API uses managers, parameters, widgets, etc. to implement Solr search. There is a sample demo project mentioned on the github Web site I have used a part in this project, modifying it per our requirements and defining my own query to search:

“Manager.store.addByValue('q', 'NumberOfOrders:[0 TO 35]');“

I used an Ajax Solr API to make a sample demo Web site application that does document search and data mining dynamically for a fixed query search. I can also enhance it by changing the query at runtime.

Suppose that, in the sample retail business scenario described earlier, we need to analyze the product purchase behavior over a year by analyzing purchasing trends by each month. We can sample each month's behavior and include more retail shops to plot the analysis graph and can analyze product positioning in the market. This application is not only a document application but also a real data-mining scenario for marketing and sales.

Figure 16 shows the output of a real-time, market-analyses demo site, in which we plotted a graph with data fetched at runtime.

Figure 16. Output of market-analyses demo site

Conclusion

We have discussed how to integrate Forms Experience Builder Web application data response with the Solr search engine, to do document search dynamically. We just need to install cURL and Solr on the server side, where we fetch the results, and clients need not install anything.

These two technologies are well established and stabilized. If we can integrate the two, then we can come up with a highly reliable file search mechanism that can be used for all data collection Web applications that use Forms Experience Builder as their user interface. It can be used for many business scenarios, and we don't require a data warehouse system or data-mining tools to perform data analyses.

Tell us what you think

Please visit this link to take a one-question survey about this article:

Resources

IBM Forms wiki Learning Center Video: “Creating and deploying a form using IBM Forms Experience Builder”:
http://www-10.lotus.com/ldd/lfwiki.nsf/dx/Creating_and_deploying_a_form_using_IBM_Forms_Nitro

Solr tutorial:
http://lucene.apache.org/solr/api/doc-files/tutorial.html

github evolvingweb/ajax-solr:
https://github.com/evolvingweb/ajax-solr/wiki

developerWorks® Java technology article, “Search smarter with Apache Solr, Part 1: Essential features and the Solr schema:”
http://www.ibm.com/developerworks/java/library/j-solr1/

ezine.com article, “Data Mining For Professional Service Firms - The Marketing Mother Lode May Already Be in Your Files:”
http://ezinearticles.com/?Data-Mining-For-Professional-Service-Firms---The-Marketing-Mother-Lode-May-Already-Be-in-Your-Files&id=4607430

developerWorks Java technology article, “What's new with Apache Solr:”
http://www.ibm.com/developerworks/java/library/j-solr-update

About the author

Romil Mehla is a Software Engineer for IBM Software Labs, India, working on WebSphere Service Integration Bus (SIBus) for the past 1.6 years. She holds a Masters degree in Information Technology and, apart from regular work, she likes to devote time to learning new and emerging technologies. You can reach Romil at romehla1@in.ibm.com.