It is common to see online retailers give customers who enter a coupon code during checkout a special offer. Some examples are:

• Buy 3 goldfish, get the 4th free.
• Buy “camera model A” and get “tripod model B” for free
• 10% off your order cost when you enter offer code X
• All batteries 20% off when using promo code X
• $10 off any order totaling over $100
• Buy any 3 car care products, get a free microfiber towel.
• Free shipping using code X

Many offers of this type are valid for only a short period of time – a week, a weekend, or even a single day (Black Friday offers).

This functionality calls for using a Rule Engine.

A rule engine could also serve as a “recommendation engine” – it can be used to recommend or suggest additional products to the customer based on their past orders, their user profile, etc. For instance, if you purchased car wax, the system might recommend their bestselling microfiber towel or something specific to your car’s year/make/model.

Modifications to JPetstore

Domain Model
Below is the modified domain model from JPetstore.

I added fields to Cart and CartItem to receive the changes from the business rules. For example, CartItem’s calculatedPrice is the modified list price of Item based on the rules. Cart’s createDate is when the shoppingcart was created. It is used in the rules to verify the given coupon has not expired.

It is important to have a rich domain model containing any field you think you’ll use. If it is not in the model, it can’t be used in the rules. Here, Product does not have a weight field. Having a weight field would have been useful for calculating a shipping cost. Product length/width/height could be used to determine if an additional shipping fee is required.

Screen Flow
Previously, the application flow went from viewing the shoppingcart directly to collecting the payment information. I added an intermediary step. Now clicking “Proceed to checkout” on the “view cart” page runs the rule service then displays the modified cart. At that point, the old functionality to collect the payment information is a click away.

View Cart

View Cart after running rules

Moving to the Spring MVC controllers, I created CheckoutController to perform the checkout step instead of having ViewCartController perform both actions. CheckoutController was configured to require authentication in petstore-servlet.xml to ensure an Account object (containing city, state, zip, etc) would be available to the rules.

The noteworthy thing about CheckoutController is that it calls the injected RuleService with the customers account and shoppingcart.

JPetstore Rules
A few examples of the rules I wrote for JPetstore are shown below.

rule "10 dollars off any order over 100 with 10off100 code"
	when
		$cart : Cart(coupon == "10off100",
			subTotal > 100,
			now >= "25-Nov-2011",
			now 	then
		$cart.setDiscount(10.00);
end

rule "buy 2 iguana, get 5 dollars off"
	when
		$cart : Cart(now <= "27-Nov-2011")
		$ci : CartItem(item.product.productId == "RP-LI-02", 
			quantity >= 2)        
	then
		$cart.setDiscount(5.00);
end

rule "20% off price of any bird"
dialect "mvel"
    when
        #conditions
        $ci : CartItem($item : item,
        	item.product.categoryId == "BIRDS")
    then
        #actions
        $ci.setCalculatedPrice($ci.item.listPrice * 0.8);
end

As you can see with the three rules shown, it is basically pattern matching on object types and field values. You can store matched field values in variables that can later be used in either the LHS or RHS.

Rule Service
RuleServiceImpl is where the rule engine is invoked with the user’s account and shoppingcart objects. The rules operate on objects inserted into the KnowledgeSession. Here you see the Cart, CartItems, and Account inserted into the session so the rules can operate on them. Changes to the objects made made by the triggered rules are visible when fireAllRules returns.

public class RuleServiceImpl implements RuleService {
    final Logger logger = LoggerFactory.getLogger(RuleServiceImpl.class);
    private ItemDao itemDao;
    private KnowledgeAgent kagent;
	
    public RuleServiceImpl() {
        kagent = KnowledgeAgentFactory.newKnowledgeAgent( "MyAgent" );
        kagent.applyChangeSet(ResourceFactory.newClassPathResource(
                "droolsChangeSet.xml"));

        ResourceFactory.getResourceChangeNotifierService().start();
        ResourceFactory.getResourceChangeScannerService().start();
		
        KnowledgeBase kbase = kagent.getKnowledgeBase();
    }
	
    public void execute(Cart cart, UserSession user) {
        KnowledgeBase kbase = kagent.getKnowledgeBase();
        StatefulKnowledgeSession knowledgeSession = 
                                    kbase.newStatefulKnowledgeSession();
        JPetRuleLogger myLogger = new JPetRuleLoggerImpl(logger);
		
        knowledgeSession.setGlobal("myLogger", myLogger);
        knowledgeSession.setGlobal("itemDao", itemDao);
		
        knowledgeSession.insert(cart);
        for (Iterator i = cart.getAllCartItems(); i.hasNext(); ) {
            CartItem ci = (CartItem) i.next();
            knowledgeSession.insert(ci);
        }
		
        Account acct = user.getAccount();
        knowledgeSession.insert(acct);
		
        //run the rules
        knowledgeSession.fireAllRules();
		
        knowledgeSession.dispose();
    }
...

Separating the rule lifecyle from the application lifecyle
One of the advantages of using a rules engine is you can separate the business rule lifecyle from the application lifecyle. An easy way to achieve this separation with Drools is to use the combination of a KnowledgeAgent and a changeset.xml file as shown above in RuleServiceImpl.

<?xml version="1.0" encoding="UTF-8"?>
<change-set xmlns='http://drools.org/drools-5.0/change-set'
	xmlns:xs='http://www.w3.org/2001/XMLSchema-instance'
	xs:schemaLocation='http://drools.org/drools-5.0/change-set 
	http://anonsvn.jboss.org/repos/labs/labs/jbossrules/trunk/drools-api/src/main/resources/change-set-1.0.0.xsd'>
	<add>
	<resource source='http://127.0.0.1:9090/jpetstoreApplication.drl'
		type='DRL' />
	<resource source='http://127.0.0.1:9090/jpetstoreDSL.dsl'
		type='DSL' />
	<resource source='http://127.0.0.1:9090/jpetstoreDSLR.dslr'
		type='DSLR' />  
	</add>
</change-set>

Another benefit of KnowledgeAgent is that it caches the knowledgebase for you. With Drools, it is an expensive operation to create a KB so you do not want to create them with every request. KnowledgeSession, which are obtained from KB are not so costly, so each user request gets their own.

Being able to change the rule resources outside a full application deployment is not helpful unless the application can detect the rule resource was changed and thereafter use the updated resource.
That is the purpose of these 2 lines in RuleServiceImpl:

		ResourceFactory.getResourceChangeNotifierService().start();
		ResourceFactory.getResourceChangeScannerService().start();

Conclusion
As you can see, the Drools rule engine makes offering this type of functionality very easy. A rule engine is a great tool to have in your toolset.

If you would like to get this running on your machine, you can use Eclipse with the SVN plugin and the m2eclipse Maven plugin to create the jpetstore project from Spring’s SVN repository. JPetstore is a Maven project so you’ll just need to add the Drools dependencies to the pom.

Benefits of using a Rule Engine:
Separates the Business Rule lifecyle from the Application lifecyle
In many applications, the business rules change more often than the application code.  Moving your rules to a Rule Engine allows them to be updated independently of the application source code.  This would allow rule changes to occur much more rapidly than the alternative of a full production release of the application.

Simplifies creating/modifying business rules
Instead of having nested “if..else” statements littered throughout your codebase, you have a known set of rule files.  Rules expressed in the Drools Rule Language (DRL), a Domain Specific Language, or a Decision Table are easier to understand because the format is clearer compared to nested “if..else” statements in the source code.  It is possible for business analysts to create and modify rules without needing a developer’s assistance depending on the tools available to the business analyst.

Fosters reuse and creates a central point of knowledge
The benefit of business rules being easier to modify/update is multiplied if you use the same rules in multiple applications.  The centralization of rules in a Rule Engine forms a central point of knowledge.  For example, if 5 applications use many of the same rules, then a single change is needed to update all 5 applications.  Contrast this with updating a different implementation of the rules in each of the 5 applications.  This approach to updating the rules is less prone to errors and is much quicker to complete.

Increases the accessibility of rules so they can be verified and audited by experts
When not using a Rule Engine, developers implement the rules expressed in a requirements document into source code.  This translation step can introduce errors.  Further, requirements documents can become outdated.  Externalizing rules allows them to be quickly retrieved and reviewed by the business experts.  The technical skill level required to review them is reduced if the rules are expressed in the form of a Decision Table or Domain Specific Language.

Rule Formats using Drools 5.2

Drools Rule Language
Here is an example rule written in the drools rule language.  The file extension is .drl

rule "10 dollars off any order over 100 with 10off100 code"
   when
      $cart : Cart(coupon == "10off100",
              subTotal > 100,
              now >= "25-Nov-2011",
              now <= "28-Nov-2011" )
   then
      $cart.setDiscount(10.00);
end

Each rule has a name.  There is a left hand side (when) and a RHS (then).  When the LHS matches what you inserted into working memory, the RHS is executed.  As you can see, the LHS is basically pattern matching on object types and on the value of their attributes.  You can capture a match and store it in a variable (shown here as $cart).  This variable can be used to restrict the later patterns in the LHS or used as needed in the RHS.

Decision Table
A decision table is a specially formatted spreadsheet that is translated by Drools into the drools rule language.  Expressing rules in this format makes it easier to see missed scenarios.  Below is an example of a drools decision table.

In this example, Drools creates a rule in the drools rule language behind the scenes for you for each row of the table starting from row 9.  The columns are either conditions of the rule (LHS) or the action to take (RHS) when there is a match.  The colors shown in the spreadsheet are cosmetic.

The orange cells of the spreadsheet represent the rule domain objects and their attributes used in the rules.  The lower cells in white area are the data values for those attributes.

The rule for Row 9 says “if there is a driver having an age 18 to 24 years old with 0 prior claims and is applying for comprehensive insurance then give the driver a 1% discount on their policy.

Instead of using the drools rule engine and the populated decision table, you could implement this table in Java using normal “if..else” statements.  Lets look at one such implementation.

if (age >= 18 && age <= 24 && priorClaims == 0) {
    if (type.equals("COMPREHENSIVE")) {
        policy.applyDiscount(1);
    }
    else if (type.equals("FIRE_THEFT")) {
        policy.applyDiscount(2);
    }
} else if (age >= 25 && age <= 30 && priorClaims == 1 
        && type.equals("COMPREHENSIVE")) {
    policy.applyDiscount(5);
}

The small rule table with 3 conditions is starting to look very busy when translated to Java “if..else” statements with only a few rows implemented.  Imagine if each rule had 10 conditions.  The decision table would still be readable, however the corresponding Java code would be menacing.

Implementing the rules in this manner would no doubt involve much copy and paste.   This may lead to shortcuts to reduce the amount of code by nesting some of the “if..else” statements.  It may work fine, but maintainability would suffer.  What would happen if a condition changed and required some of the nesting to be eliminated?  For example, imagine a new rule was added for 18-24 wanting comprehensive with 1 prior claim.  There would be a ripple effect. Imagine if an a new condition needed to be added to the rules.  Using a decision table, this would be an extra column with the associated values.  Changing Java “if..else” statements would be a nightmare.

Domain Specific Language (DSL)
Another means available in Drools for expressing rules in a format friendly to non-technical users is to use a DSL.  A DSL empowers business experts to write rules using a set of phrases you developed collaboratively with them.  This enables them to express the rules in the language natural to their business domain in the form of a sentence.  Instead of cryptic code, it looks like a requirements document.  Here is an example DSL and rules written using it.

expander jpetstore.dsl
rule "10off50 coupon between nov 23 to 29 2011"
when
A customer prepares to place an order
- using coupon code "10off50"
- totaling at least 50 dollars
- starting on "23-Nov-2011"
- ending on "29-Nov-2011"
then
discount the order total by 10 dollars
end

DSL rule (jpetstore.dslr)

This format is easy to understand and allows the creation of multiple rules of this type quickly using GUI tools.  Further, not all of the restrictions on the order are required.  For example, the line specifying the coupon code could be eliminated which would then allow any order over $50 to obtain the $10 discount.

Now lets take a look at the technical side, the DSL definition, which would be created by a developer.

[when]A customer prepares to place an order=cart:Cart(subTotal > 0)
[when]- using coupon code "{coupon}"=coupon == "{coupon}"
[when]- totaling at least {value} dollars=subTotal >= {value}
[when]- starting on "{start}"=now >= "{start}"
[when]- ending on "{end}"=now <= "{end}"
[then]discount the order total by {amount} dollars=cart.setDiscount({amount});

DSL definition (jpetstore.dsl)

Basically, the phrase on the left is converted to the code on the right with the appropriate values used in the placeholders. Using the Drools IDE in Eclipse, you can see how rules written using a DSL (the .dslr file) are transformed into the drools rule language by clicking on the DRL viewer table in the Eclipse editor. 

Besides an increase in readability, you also gain an increase in flexibility when you use a DSL because the phrases are independent of the domain model.  You can change the domain model without impacting the rules that are written in the DSL.  Similarly, you can change the wording of the phrases independently of the domain model.  All that is required is a modification to the DSL definition.

General steps needed to use Drools in your application
Here is an overview of how to use Drools in your application:

• Create a POJO (Plain Old Java Object) domain model complete with attributes. It should model all the concepts needed by your rules
• Author your rules using your domain model in your desired format. The rules be written in the drools rule language, a decision table, or a domain specific language
• Build a KnowledgeBase using one or more rule resources.
• Create a KnowledgeSession from the KnowledgeBase
• Insert populated POJOs into the KnowledgeSession
• Run the rules which then modify the appropriate POJOs
• Use the modified POJOs in your application

Conclusion
I’ve covered the benefits of using a Rule Engine, the different formats rules can be expressed using the Drools Rule Engine, and a general overview of the steps required to use Drools in your application.  In a later post, I will show how a shoppingcart can leverage a rules engine to implement some of the features we commonly see offered by online retailers.

To learn more about Drools and related products like Guvnor, you can visit the community project’s homepage. JBoss Rules, the enterprise version of Drools supported by Red Hat, also has its documentation available online.

Many applications like to save uploads where they can later be accessed/served to other users. One location to use is the exploded WAR file directory of your application. If you are using Tomcat, this would be $CATALINA_HOME/webapps/YourWarFile.

Saving your uploaded files in this location and not restricting the types of files a user can upload is very dangerous because a user could upload and execute a JSP file. Once the user accesses their JSP on your server, the application server would execute any code contained in it with the privileges of the system user running the application server.

What code could this JSP execute? The JSP would have access to any Java library available in the application and the standard Java libraries, including java.lang.Runtime. As you may recall, with Runtime you can execute and display the output of local system commands from Java with code along the lines of:

<%
Runtime runtime = Runtime.getRuntime();
Process process = runtime.exec(command);
%>

An example command that can be run on Windows is “cmd /c dir” and a similar example on Linux is “ls –l”. Imagine the possibilities if the application server was running as root!

It is a good idea to limit the types of files you application accepts and do not accept JSP files. Also, carefully consider where uploaded files are ultimately stored. Finally, run your application server with a low privileged user that has the minimal system access required to perform the job.

So how do you prevent users from uploading PDF files, MS Word documents, etc to your image sharing service?

One approach is to maintain a list of allowed file extensions such as .gif, .jpg, and .png. This type of list which contains only what is allowed is referred to as a whitelist. In other words, it is a list of what we accept and we reject everything else that is not in the list. (The opposite is called a blacklist – a list of what we reject and we accept everything else.) With this whitelist of file extensions, we can check the filename in the Controller/Action class handling the upload to see if it is allowed by the system by using a simple regex. If it is not valid, the application can notify the user.
The problem with this approach is the file name has nothing to do with the actual file content. A user can simply rename test.pdf to test.jpg and the system would accept it.

A better approach is to check the file’s content for its “magic number”. Wikipedia has a good explanation of magic numbers, but basically many file types have a distinct signature that can be used to identify it. For example, the content of a PDF file starts with %PDF. It even includes the version like %PDF-1.3 to indicate a PDF file version 1.3 or %PDF-1.6 to indicate PDF version 1.6. Likewise, the content of a GIF image starts with GIF89a or GIF87a. So if someone is uploading a GIF image and it does not contain the correct magic number, then you know it is not a GIF.

One way to quickly implement this idea in your application is to leverage Apache Tika. Tika is a subproject of Lucene and is used in Solr. From the Tika website “Apache Tika is a toolkit for detecting and extracting metadata and structured text content from various documents using existing parser libraries”.

FileInputStream fis = ...//uploaded file
Tika tika = new Tika();
String result = tika.detect(fis);

From this, you’ll get a String such as image/png, image/jpeg, text/plain, application/pdf. For a GIF image, you’ll get image/gif.

Are Magic numbers a magic solution to validating the file type of an uploaded file? Could a magic number checker be tricked? It is best to follow a defense in depth approach and check both the file extension and the content type any time you allow users to upload files to your system.

These are often described as CRUD (Create, Read, Update, Delete) applications. An example would be a simple web registration form which would request from the user their first name, last name, email address, password, and mailing list preferences in order to create a profile on the site. In this scenario, we’ll have a one to one mapping of the user table to the form on a web page. An example of each CRUD operation would be:

• Create – registering on the site
• Read – logging in and viewing your profile
• Update – changing your mailing list preferences
• Delete – requesting your account be closed. Many times data is not actually deleted from the database, but instead an update is executed to set the status to “inactive”.

In this user profile example, only one person is updating the data (since only one person knows the correct password needed to access the record) so the data in the database is the same between the Read and Update steps. However, there are some web applications where multiple users can modify the same data set concurrently. These applications can suffer from the “lost update” problem.

Lost Update Problem
To illustrate the lost update problem, consider a web application which allows salespeople to maintain a list of clients. This web application allows any salesperson to view a list of clients and then click on any one client to update that client’s information. The user must click the “Save” button to persist their changes to the database.

Using this web application, Joe (Salesperson) wants update Bob’s (Client) record by adding Bob’s new cell phone number. Joe performs a search, the application displays a matching list of client records, and Joe clicks on Bob’s record. At this time, the data currently stored in the database for Bob is retrieved and displayed on a page in Joe’s web browser. Joe decides to get a cup of coffee from the break room before making his change to Bob’s record.

While Joe gets his coffee, Fred (Salesperson) wants to update the address on Bob’s record. Like Joe, Fred accesses Bob’s record via a search and is shown the data currently stored in the database for Bob. At this point, both Fred and Joe have the same data displayed in their web browser for Bob. Fred changes Bob’s address in the form, clicks the “Save” button, and goes to Lunch.

Joe comes back with his coffee. At this point, Joe’s view of Bob’s data is not the current data for Bob in the database. Fred changed Bob’s address and that change is not shown in Joe’s web browser. Most web applications operate in this manner – they only send data to the user (Joe) when the user requests it. Now Joe adds Bob’s cell phone number and clicks the “Save” button. This causes Bob’s new phone number as well as the outdated address that was shown in his form to be saved in the database. Fred’s update is now lost.

Fred gets back from lunch, searches for and views Bob’s record again, and sees Bob’s old address along with a new phone number. Fred is angered and fires off an email to his boss telling him the cool new sales web application is broken and he wants to use the old application.

As you can imagine, this is not good for the application development team or the salespeople. The diagram below illustrates the sequence of events leading up to the lost update.

As is often the case, each time the user clicks “Save” on the page in their web browser, all the data in their web form gets sent to the application server and written to the database. The flaw in this case is the application assumes the data submitted by the user is the current state of the data in the database. As you can see with the above example with Joe and Fred, this is not necessarily true.

To handle the lost update problem, you can use either optimistic concurrency control or pessimistic concurrency control.

Optimistic Concurrency Control
Using optimistic concurrency control, an additional value is sent along with the user’s web form data when the user clicks the “Save” button. This additional value is then used to determine if someone else changed the data in the database after this user last read it. If the data in the database was not changed, the user’s change succeeds otherwise it fails and the user gets a friendly error message along the lines of “Another user has changed the data since your last request. Please try again”.

At this point, some implementations might try to help the user merge their changes with the current state in the database. Others implementations send the user the latest state of the data in the database and force them to re-enter their changes. As you can imagine, optimistic concurrency control can get very frustrating if you have a large web form and get many “please try again” messages.

A common implementation of optimistic concurrency control is to have a timestamp column in your table that is automatically updated by the database to match the time the row was last modified. When the user reads the record from the database to populate their web form, the timestamp is included and becomes a hidden form field that is sent back to the database with all the other form data when the users clicks the “Save” button. The “where” clause of the update SQL statement is modified to now include the timestamp:
update … where id = ? and timestamp = ?

Going back to our example, if it used optimistic concurrency control both Fred and Joe would get the same timestamp value from the database in their initial read. Fred’s phone number change would succeed because the timestamp matches the current timestamp in the row for Bob. Besides changing the phone number, Fred’s change causes the database to update Bob’s timestamp column to match the time Fred submitted his change.

Now Joe sends his address change with the old timestamp and the application would reject it because the where clause, which now includes the timestamp, does not succeed. The application would give Joe an error message along with the current state of Bob’s record, including the new timestamp. Joe now has the option to redo his address change.

Pessimistic Concurrency Control
Using pessimistic concurrency control, the concept of locking data to prevent others from attempting to modify it is introduced. A user can only change data if the user has its lock. The lock on the data is obtained when the data is initially read and is released when the user sends their changes to the server. If a user attempts to obtain the lock when another user already has it, it will fail and the user would get an error message.

Locks can create a number of issues. The classic example is one user needing to change a record locked by another user who is on vacation. With pessimistic concurrency, some things to consider are how locks are acquired and released, if locks automatically expire after a period of time, and if a user can forcefully unlock data locked by another user.

A common implementation of pessimistic concurrency control is to use a “select … for update nowait” SQL statement when reading data with the intent to modify it. Once that statement executes, that user has the lock. Another user who later attempts to execute the same statement before the user with the lock either does a commit or rollback would get an error. For Oracle, the error is:
ORA-00054: resource busy and acquire with NOWAIT specified

If the example sales application used pessimistic concurrency control, Joe’s request to read Bob’s data from the database would give Joe the lock on Bob’s record. Later, when Fred would read Bob’s record with the intent to modify it, Fred would be given a “This record is currently locked. Please try again later” application error since Joe already has the lock. Once Joe clicks the “Save” or “Cancel” button, the lock is released.

Conclusion
Of the two strategies, optimistic concurrency control is widely used in web applications. Pessimistic concurrency control is not often seen with web applications, however it is seen with client/server (2-tier) applications.

To use pessimistic concurrency control, the client needs to maintain the same database connection for their entire interaction with the database (think SQL*Plus). This is often described as a stateful connection. Web applications commonly use a database connection pool in the application server for performance and scalability reasons (a relatively small number of db connections can be recycled and serve a large number of web clients). As a result state is not retained because a web user gets a random db connection from the pool with each request.

Going back to the sequence diagram at the beginning, each time Joe or Fred submits their request to the application server, a random db connection from the database connection pool is extracted, executes the SQL statements on their behalf, and is returned to the pool for the next user request.

Another valid option to handling the lost update problem is to design your application so multiple users do not have write access to the same set of records. In my example, if each client is assigned a salesperson, then only the assigned salesperson can change the record. Joe is assigned to Bob and only Joe can change Bob’s data. Fred would never see Bob in the list of users he can edit.

It is also worth noting that loosing an update may not be a problem for some applications as they may have a history table or an audit table that allows users to view past changes inside the application. These applications follow what is a called a “Last update wins” or “Last commit wins” strategy. The sales web application as described at the beginning of this post uses the “Last update wins” strategy. This strategy is very easy to implement because it is what you get if you do nothing.

The “lost update” problem is not a very difficult problem to solve once you are made aware of it.

Let’s use Google as an example. By default, you are shown the first 10 best results matching your query. At the bottom of the page, there are numbered links representing the corresponding pages of the search results. Clicking on “3″ gives you the 3rd page of results. With a page size of 10, the results shown are 21 -30.

Searching for “lucene” at Google, it responds with
“Results 1 – 10 of about 2,440,000 for lucene. (0.33 seconds)”

Imagine if Google did not use pagination and instead returned everything to you at once on a single page! That would be a very large page and take a very long time to load.

If you use Lucene, you too can present your search results in a paginated manner even though Lucene does not provide a direct way to do it in their API (2.4.1). Thanks to how Lucene is designed, it is very easy to implement.

In Lucene, when you perform a search you do not retrieve the actual documents immediately, instead you get an array of ranked pointers to the document matches. Specifically, the TopDocs object returned contains an array of ScoreDoc objects which contain the document IDs of the matching documents. They are ordered by decreasing relevancy to the user supplied search term, so the best matches are at the front. It is the call to the IndexSearcher object, specifically IndexSearcher.doc(docID), which pulls the document into memory. Lucene’s lazy loading of documents is very beneficial for pagination.

As we can see in the Lucene FAQ, the Lucene developers recommended approach to pagination is re-executing the search and navigating to the correct position in the ScoreDoc array.

As a result, pagination becomes a problem of finding the correct start position and end position in the ScoreDoc array.
Here is a paginated search:

Query query = qp.parse(searchTerm);
TopDocs hits = searcher.search(query, maxNumberOfResults);
ArrayLocation arrayLocation = paginator.calculateArrayLocation(hits.scoreDocs.length, pageNumber, pageSize);

for (int i = arrayLocation.getStart() - 1; i < arrayLocation.getEnd(); i++) {
	int docId = hits.scoreDocs[i].doc;

	//load the document
	Document doc = searcher.doc(docId);
	String filename = doc.get("filename");
	String contents =  doc.get(searchField);

}

Here are the details of calculating the locations in the ScoreDocs array:

public class Paginator {

	public ArrayLocation calculateArrayLocation(int totalHits, int pageNumber,
		int pageSize) {
		ArrayLocation al = new ArrayLocation();

		if (totalHits < 1 || pageNumber < 1 || pageSize < 1) {
			al.setStart(0);
			al.setEnd(0);
			return al;
		}

		int start= 1 + (pageNumber -1) * pageSize;
		int end = Math.min(pageNumber * pageSize, totalHits);
		if (start > end) {
			start = Math.max(1, end - pageSize);
		}

		al.setStart(start);
		al.setEnd(end);
		return al;
	}
}

The majority of the time, your users will find what they need on the first page of search results since the matches most relevant to their search are returned at the top of the list. I think most people don’t look beyond the first few pages and instead revise their search terms if they don’t immediately see what they need.

The benefits of pagination include:

• lower response time since fewer documents are processed on the server side
• lower memory use since fewer documents are loaded
• lower network use since the amount of data sent over the network to the client is reduced

The end result is a better user experience.

Here are some numbers to give you a general idea of the impact of pagination. It shows the same search being performed with and without pagination and its effect on response time. MaxResults is the argument passed to the IndexSearcher.search(query, maxresults) method which bounds the number of results returned.

public void updateDocument(Term term, Document doc)
                    throws CorruptIndexException, IOException

Lucene’s updateDocument operation is basically delete and insert wrapped into a single function. All documents matching the Term parameter are deleted from the Lucene index and the supplied Document instance is then inserted into the index. While Lucene allows multiple copies of the same document to exist in the index, the behavior of the update operation does not insert a copy of the supplied document for every match. In other words, if your Term matches 5 documents in the index then 5 documents are deleted and a single document is inserted in its place.

As you can see, it is a very good idea for you to design your documents so they have a field that uniquely identifies them in the entire index. In the database world, this is called a primary key field.

At times, it is helpful to think of the Lucene index as a database having a single table and the Documents as rows. It is a good analogy when you frame it in terms of searching. Boolean Queries seem to fit this concept nicely.

However, there are many differences between a Lucene index and a database.

• Lucene does not provide a way to enforce field uniqueness. It is up to you to achieve the concept.
• Lucene does not require a predefined document schema for the documents in the index. This means all documents in the index do not need to have the same number of fields or use the same field names. As an example, some documents can have the fields (id, url, contents) and other documents can have the fields (productid, manufacturer, summary, review).
• Fields can be repeated in a document. For example, a document can have 3 product review fields (productid, manufacturer, summary, review, review, review). We will revist this later in the code example.

Lucene’s updateDocument method overwrites the document(s) matching the given Term with the given Document. This is a problem if you only want to update a few fields and keep the remainder.

In the scenario pictured below, you can uniquely identify a document in the index whose author field you would like to update. So you then call updateDocument and pass in the Term and a Document instance populated with the new author field value. The result is an updated author field and the loss of the 3 other fields previously stored – the title, publisher, and contents fields.

What do you do when you need to update a subset of the fields in a document but cannot re-create the remaining fields? There can be many reasons for this dilemma. Perhaps you are unable to re-create the fields because the original text is not available or perhaps the operations to re-create the fields are very costly.

One approach to resolve this dilemma is to search for the current document in the index, change the desired fields, and use the modified document as the input to the updateDocument call. This idea is illustrated below. UpdateUtil.java contains the full source.

int docId = hits.scoreDocs[0].doc;
			
//retrieve the old document
Document doc = searcher.doc(docId);

List<Field> replacementFields = updateDoc.getFields();
for (Field field : replacementFields) {
	String name = field.name();
	String currentValue = doc.get(name);
	if (currentValue != null) {
		//replacement field value
		
		//remove all occurrences of the old field
		doc.removeFields(name);

		//insert the replacement
		doc.add(field);
	} else {
		//new field
		doc.add(field);
	}
}

//write the old document to the index with the modifications
writer.updateDocument(term, doc);

Here we pass in a Document that can have both replacement fields and additional fields for the document identified by a search using the term parameter as the basis for a TermQuery.. First we obtain the list of Fields from the document parameter. If the matched document already has a field by that name, it is considered a replacement otherwise it is a new field to be added to the document.

Notice the method first removes all fields in the Document having the same name as the replacement prior to inserting the replacement field. As mentioned earlier, a Lucene document can have multiple fields with the same name.

Without the remove call, you would be adding another value for the field instead of replacing the existing value.

A great tool to view what is actually in your Lucene index is Luke, the Lucene Index Toolbox. It is very helpful tool to answer “what if” questions when you read the Lucene API.

Out of the box, Lucene does not provide a way to update the individual fields of a document in the index. However, it is relatively easy to achieve this functionality by grouping together the available API calls.

You can browse the full source at google code and download a copy of the entire project via svn.
svn checkout http://hrycan-blog.googlecode.com/svn/trunk/lucene-highlight/

You can browse the full source online or you can use subversion to checkout the code for a local copy. As an example, to checkout the lucene code at the command line:

svn checkout http://hrycan-blog.googlecode.com/svn/trunk/lucene-highlight/

Consider what happens when you search for ‘Apache’ at Google. Your results would include the Apache server, the Apache Software Foundation, the Apache Helicopter, and Apache County. The contextual fragments displayed with each search result helps you judge if a search result is an appropriate match and if you need to add additional search terms to narrow the search result space. Search would not be as user friendly as it is today without these fragments.

This post covers version 2.4.1 of Apache Lucene, the popular open source search engine library written in Java. It may not be widely known, but Lucene provides a way to generate these contextual fragments so your system can display them with each search result. The functionality is not found in lucene-core-2.4.1.jar but in the contrib library lucene-highlighter-2.4.1.jar. The contrib libraries are included with the Lucene download and are located in the contrib folder once the download is unzipped.

If you are not familiar with Lucene, you can think of it as a library which provides

• a way to create a search index from multiple text items
• a way to quickly search the index and return the best matches.

A more thorough explanation of Lucene can be found at the Apache Lucene FAQ.

As an example of what the Lucene Highlighter can do, here is what appears when I search for ‘queue’ in an index of PDF documents.

e14510.pdf
Oracle Coherence Getting Started Guide
of the ways that Coherence can eliminate bottlenecks is to queue up transactions that have occurred…
duration of an item within the queue is configurable, and is referred to as the Write-Behind Delay. When data changes, it is added to the write-behind queue (if it is not already in the queue), and the queue entry is set to ripen after the configured Write-Behind Delay has passed…

The Steps
First, before you can display highlighted fragments with each search result, the text to highlight must be available. Shown below is a snippet of indexing code. We are storing the text that will be used to generate the fragment in the contents field.

Document doc = new Document();
doc.add(new Field("contents", contents, Field.Store.COMPRESS, 
    Field.Index.ANALYZED, Field.TermVector.WITH_POSITIONS_OFFSETS));
doc.add(new Field("title", bookTitle, Field.Store.YES, 
    Field.Index.NOT_ANALYZED));
doc.add(new Field("filepath", f.getCanonicalPath(), Field.Store.YES, 
    Field.Index.NOT_ANALYZED));
doc.add(new Field("filename", f.getName(), Field.Store.YES, 
    Field.Index.NOT_ANALYZED));
writer.addDocument(doc);  

The values Field.Store.COMPRESS or Field.Store.Yes tell Lucene to store the the field in the index for later retrieval with a doc.get() invocation.

Field.Store.COMPRESS causes Lucene to store the contents field in a compressed form in the index. Lucene automatically uncompresses it when it is retrieved.

Field.Index.ANALYZED indicates the field is searchable and an Analyzer will be applied to its contents. An example of an Analyzer is StandardAnalyzer. One of the things done by StandardAnalyzer is to remove stopwords (a, as, it, the, to, …) from the text being indexed.

Note: You should use the same analyzer type (like StandardAnalyzer) for your indexing and searching operations otherwise you will not get the results you are seeking.

Last part of the indexing side is the TermVectors. From the Lucene Javadocs:
“A term vector is a list of the document’s terms and their number of occurrences in that document.”

For the Highlighter, TermVectors need to be available and you have a choice of either computing and storing them with the index at index time or computing them as you need them when the search is performed. Above, Field.TermVector.WITH_POSITIONS_OFFSETS indicates were are computing and storing them in the index at index time.

With the index ready for presenting contextual fragments, lets move on to generating them while processing a search request. Below is a typical “Hello World” type search block.

QueryParser qp = new QueryParser(“contents”, analyzer);
Query query = qp.parse(searchInput);
TopDocs hits = searcher.search(query, 10);

for (int i = 0; i < hits.scoreDocs.length; i++) {
	int docId = hits.scoreDocs[i].doc;
	Document doc = searcher.doc(docId);
	String filename = doc.get("filename");
	String contents =  doc.get(“contents”);
	
	String[] fragments = hlu.getFragmentsWithHighlightedTerms(analyzer, 
                   query, “contents”, contents, 5, 100);
}

Starting at the top, we create a query based on the user supplied search string, searchInput, using the QueryParser. Lucene supports a sophisticated query language and QueryParser simplifies transforming the supplied string to a query object. Next, we get the top 10 results matching the query. This is pretty standard so far, but now in the loop we come to the getFragmentsWithHighlightedTerms call.

Here is the code to generate the fragments:

TokenStream stream = TokenSources.getTokenStream(fieldName, fieldContents, 
                      analyzer);
SpanScorer scorer = new SpanScorer(query, fieldName,
				new CachingTokenFilter(stream));
Fragmenter fragmenter = new SimpleSpanFragmenter(scorer, 100);
		
Highlighter highlighter = new Highlighter(scorer);
highlighter.setTextFragmenter(fragmenter);		
String[] fragments = highlighter.getBestFragments(stream, fieldContents, 5);

First we obtain the TokenStream. The call shown above assumes term vectors were not stored in the index at index time.

Next is the SpanScorer and SimpleSpanFragmenter. These work to break the contents into 100 character fragments and rank them by relevancy. You can use SpanScorer and SimpleSpanFragmenter or QueryScorer and SimpleFragmenter. The full details can be found in the Javadocs.

Note: when indexing large files, like the full contents of PDF manuals, you might need to tell the Highlighter object to look at the full text by calling the setMaxDocCharsToAnalyze method with Integer.MAX_VALUE or a more appropriate value. In my case, the default value specified by Lucene was too small, thus Highlighter did not look at the full text to generate the fragments. This was not good because the match I was seeking was near the end of the contents.

Finally, we tell the Highlighter to return the best 5 fragments.

The full code for this example can be downloaded from my Google Code project. The source file that makes use of the Highlighter is HighligherUtil.java

You can also find examples of using Highlighter in the Lucene SVN Repository, specifically HighlighterTest.java

As you can see, returning search results with contextual fragments containing your search terms is very easy with the Lucene Highlighter contrib library once you know the steps to follow.

<?xml version="1.0" encoding="UTF-8"?>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
	<title/>
</head>
<body>
	<div>
		<p>Paragraph 1</p>
	</div>
	<div>
		<p>Paragraph 2</p>
	</div>
	<div>
		<p>Paragraph 3</p>
	</div>
</body>
</html>

You would think the XPATH to extract the contents of each paragraph would be

String xpathExpr = "//div/p";

If you try it out, you will see there are no matches. The gotcha is the namespace

xmlns="http://www.w3.org/1999/xhtml"

If that namspace was not specified, then the above XPATH expression would work. The solution is to use an alternate set of API calls than what is shown in the standard XPATH examples in the dom4j documentation.

package com.hrycan.blog.xml;

import java.util.List;
import java.util.Map;
import org.dom4j.Document;
import org.dom4j.DocumentException;
import org.dom4j.DocumentHelper;
import org.dom4j.Node;
import org.dom4j.XPath;

public class XPATHProcessor {

	private Map<String, String>  namespaceURIMap;
	
	public List<Node> extract(String content, String xpathExpr) throws DocumentException {
		Document document = DocumentHelper.parseText(content);		
		XPath path = DocumentHelper.createXPath(xpathExpr);
		path.setNamespaceURIs(namespaceURIMap);
		
		List<Node> list = path.selectNodes(document.getRootElement());
		return list;
	}

	public void setNamespaceURIMap(Map<String, String> namespaceURIMap) {
		this.namespaceURIMap = namespaceURIMap;
	}
}

Here is the JUnit test showing how it would be used with the content of Listing1 and the corresponding XPATH expression using the namespace.

package com.hrycan.blog.xml;

import java.util.HashMap;
import java.util.List;
import java.util.Map;

import org.dom4j.DocumentException;
import org.dom4j.Node;
import org.junit.Test;
import static org.junit.Assert.assertTrue;

public class XPATHProcessorTest {
	private XPATHProcessor xpathProcessor;
	
	@Test
	public void testExtract() throws DocumentException {
		xpathProcessor = new XPATHProcessor();
		Map<String, String>  namespaceURIMap = new HashMap<String, String> ();
		namespaceURIMap.put("html", "http://www.w3.org/1999/xhtml");
		xpathProcessor.setNamespaceURIMap(namespaceURIMap);
		
		String content = //content of Listing1 shown above		
		String xpathExpr = "//html:div/html:p";
		
		List<Node> list = xpathProcessor.extract(content, xpathExpr);
		assertTrue(list.size() == 3);
	}	
}