DCCT4090 - IR Term Project

Well, there was some difficulty from the subtlety of adding postings to a word which already exists in the lexicon. The crux of our problem is this: words parsed from article bodies either exist or don't exist in the lexicon. If they do exist, then we must check if they exist for the current document id, otherwise we must create a new posting. The code which solves this problem is seen in the InvertedIndex implementation of the Index interface, under the function, addLexicalTerms:


public void addLexicalTerms(Document d)
{
Pattern p = Pattern.compile("\\w+");
String[] temp = d.getBodyText().split("\\ "); // split on spaces
//d.addBody("");
for(int j=0;j<temp.length;j++)
{
 Matcher m = p.matcher(temp[j]);
 if(!temp[j].equals("")&&!(temp[j].length()==1)&&m.matches())
 {
  if(!StopWordList.getInstance().containsStopWord(temp[j].toLowerCase()))
  {
   if(lexicon.containsKey(temp[j]))
   {
    boolean exitFlag = false;
    for(int k=0;k<lexicon.get(temp[j]).size();k++)
    {
     if(lexicon.get(temp[j]).get(k).getDocID().equals(d.getDocumentID()))
     {
      lexicon.get(temp[j]).get(k).increaseFrequency();
      
     }
     else if(k==(lexicon.get(temp[j]).size()-1))
     {
      exitFlag=true;
     }
    }
    if(exitFlag)
    {
     Posting newPosting = new Posting(d.getDocumentID());
     newPosting.increaseFrequency();
     lexicon.get(temp[j]).add(newPosting);
     postingCount++;
    }
   }
   else if(!lexicon.containsKey(temp[j]))
   {
    /*
     * We should create a new lexical entry
     */
    Posting newPosting = new Posting(d.getDocumentID());
    newPosting.increaseFrequency();
    LinkedList<Posting> tempList = new LinkedList<Posting>();
    tempList.add(newPosting);
    lexicon.put(temp[j], tempList);
    postingCount++;
   }
  }
 }
}
}

because not many people like reading others code, here's a better explanation

To date, we have a very simple implementation of a Query Processor. All it does right now is strip stop words from the input query, both when using the pre-packaged Google GSearchForm and when using the custom search input. While a seemingly small accomplishment, it is nevertheless a head first introduction to AJAX. The choice to use AJAX was a natural answer to the question: "how does Javascript communicate with a Servlet?"

let's take a look at the callback function which intercepts the GSearchControl execute function. The function intercept() is provided as a callback functor to the GSearchControl's method, "setSearchStartingCallback". This function tells the GSearchControl to call the intercept() function before submitting the query to Google. The intercept() function does some interesting things, starting with walking the DOM tree to get the value of the text the user entered into the GSearchControl. This is done because Google provides no direct way to access this value through a method call.

Next, we get ready to create an instance of the XMLHttpResponder by determining whether the current browser is IE or other. Internet explorer implments the XMLHttpResponder as an ActiveX control, whereas other browsers do not. After creating the appropriate instance of an XMLHttpReponder object, we open a connection to the servlet, providing the message in the GET string. The second parameter, a boolean value of true, to the open method, indicates communication with the servlet is asynchronous. Next, we set the callback method to handle the servlet response, and finally we send the request to the specified Servlet.


function intercept()
{
 var srchInput = document.getElementsByTagName('input');
 
 for(var i=0;i<srchInput;i++)
 {
  if(srchInput[i].type=='text')
  {
   srchStr = srchInput[i].value;
  }
 }
 
 if(window.ActiveXObject)
 {
  xmlHttp = new ActiveXObject("Microsoft.XMLHTTP");
 }
 else if(window.XMLHttpRequest)
 {
  xmlHttp = new XMLHttpRequest();
 }

 xmlHttp.open("GET","http://localhost:8080/Phase1/servlet/QueryProcessor?query="+srchStr,true);
 xmlHttp.onreadystatechange=responseHandler;
 xmlHttp.send(null);
}

Now, let's take a look into the servlet code, QueryProcessor.java.


/**
* * Copyright (c) 2008, Michael Kuredjian
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*     * Redistributions of source code must retain the above copyright
*       notice, this list of conditions and the following disclaimer.
*     * Redistributions in binary form must reproduce the above copyright
*       notice, this list of conditions and the following disclaimer in the
*       documentation and/or other materials provided with the distribution.
*     * Neither the name of the University of Guelph-Humber nor the
*       names of its contributors may be used to endorse or promote products
*       derived from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY Michael Kuredjian ``AS IS'' AND ANY
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
* WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
* DISCLAIMED. IN NO EVENT SHALL Michael Kuredjian BE LIABLE FOR ANY
* DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
* (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
* ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */
package krooj.servlet.search;

import java.io.*;
import javax.servlet.http.*;
import javax.servlet.*;

/**
 * Query processor is an AJAX-active servlet that will eventually have much more
 * functionality, but right now it just takes in the user's query, makes a call to 
 * the StopWordStripper and returns the cleaned query string to the browser.
 * @author michael
 *
 */
public class QueryProcessor extends HttpServlet
{
 public QueryProcessor()
 {
  
 }
 
 public void doGet(HttpServletRequest req, HttpServletResponse res) throws ServletException, IOException
 {
  
  /*
   * The query is supplied in the request URL to this servlet. This line gets the query string.
   */
  String search = req.getParameter("query");
  /*
   * Strip any stop words from the query string
   */
  search = new StopWordStripper(search).getCorrectedQuery();
  System.out.println(search); //DEBUG - replace with call to logger
  /*
   * Set the response type to text/xml - this is AJAX, after all.
   */
  res.setContentType("text/xml");
  res.setHeader("Cache-Control", "no-cache");
  /*
   * Send the expected tag and value combination.
   */
        res.getWriter().write(""+search+"");
        /*
         * Flush the buffer. Always flush the buffer!!
         */
  res.flushBuffer();
 }
 
 public void doPost(HttpServletRequest request, HttpServletResponse  response) throws IOException, ServletException 
 {
  doGet(request,response);
 }
 
}

Right now its very obvious the QueryProcessor isn't doing much besides removing stop words. In the future, this will change significantly; the QueryProcessor will act as an entry point to our customized search. Stay tuned.


/**
* * Copyright (c) 2008, Michael Kuredjian
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*     * Redistributions of source code must retain the above copyright
*       notice, this list of conditions and the following disclaimer.
*     * Redistributions in binary form must reproduce the above copyright
*       notice, this list of conditions and the following disclaimer in the
*       documentation and/or other materials provided with the distribution.
*     * Neither the name of the University of Guelph-Humber nor the
*       names of its contributors may be used to endorse or promote products
*       derived from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY Michael Kuredjian ``AS IS'' AND ANY
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
* WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
* DISCLAIMED. IN NO EVENT SHALL Michael Kuredjian BE LIABLE FOR ANY
* DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
* (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
* ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */

package krooj.servlet.search;

/**
 * 
 * @author michael
 * Class that will be instantiated when the user enters a search.
 * This class will strip the query of insignificant stop words.
 */
public class StopWordStripper 
{
 private String finalQuery;
 
 /**
  * Constructor that will use the singleton StopWordList class to help clean the 
  * supplied query.
  * @param query Un-stopped query
  */
 public StopWordStripper(String query)
 {
  finalQuery = new String();
  String temp[] = query.split("\\ "); //Split on spaces
  for(int i=0;i<temp.length;i++)
  {
   if(!StopWordList.getInstance().containsStopWord(temp[i]))
   {
    finalQuery+=" "+temp[i];
   }
  }
  
 }
 
 /**
  * Returns the stopped query
  * @return Stopped query string that can be supplied to the search engine
  */
 public String getCorrectedQuery()
 {
  return finalQuery;
 }

}


/**
* * Copyright (c) 2008, Michael Kuredjian
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*     * Redistributions of source code must retain the above copyright
*       notice, this list of conditions and the following disclaimer.
*     * Redistributions in binary form must reproduce the above copyright
*       notice, this list of conditions and the following disclaimer in the
*       documentation and/or other materials provided with the distribution.
*     * Neither the name of the University of Guelph-Humber nor the
*       names of its contributors may be used to endorse or promote products
*       derived from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY Michael Kuredjian ``AS IS'' AND ANY
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
* WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
* DISCLAIMED. IN NO EVENT SHALL Michael Kuredjian BE LIABLE FOR ANY
* DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
* (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
* ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */
package krooj.servlet.search;

import krooj.commons.propertybroker.*;
import org.jboss.logging.Logger;
import java.util.*;
import java.io.*;
/**
 * 
 * @author michael
 *This class will load the stop word list into memory as an array list for - slightly - faster access
 *than via secondary storage. Improvements should be made to the data structure for faster access.
 */
public class StopWordList 
{
 private static final Logger log = Logger.getLogger(StopWordList.class.getName());
 private static StopWordList swl;
 private List swList;
 private File swFile;
 
 /**
  * Singleton contructor that will init the class.
  */
 private StopWordList()
 {
  try
  {
   log.debug("Tyring to load the stopword list");
   System.out.println(PropertyBrokerImpl.getInstance().getProperty("stopwords.location"));
   swFile = new File(PropertyBrokerImpl.getInstance().getProperty("stopwords.location"));
   loadStopList(swFile);
   log.debug("Loaded the stopword list");
  }
  catch(Exception e)
  {
   System.out.println(e.getMessage());
  }
 }
 
 /**
  * This function will perform the physical file read into the array list for the stop word list.
  * Note: the array list instantiation is performed here.
  * @param f File object for the stop word list.
  * @throws Exception
  */
 private void loadStopList(File f) throws Exception
 {
  String lne;
  swList = new ArrayList<String>();
  BufferedReader br = new BufferedReader(new FileReader(f));
  while((lne=br.readLine())!=null)
  {
   swList.add(lne);
  }
 }
 
 /**
  * Checks whether the supplied stop word is contained in the stop word list.
  * @param s Stop word to look for
  * @return Either true (the stop word was found) or false (not found)
  */
 public boolean containsStopWord(String s)
 {
  boolean contains = false;
  if(swList.contains(s))
  {
   contains=true;
  }
  return contains;
 }
 
 /**
  * Singleton getInstance. Static classes for snazzy mofos.
  * @return An instance of this class. 
  */
 public static StopWordList getInstance()
 {
  if(swl==null)
  {
   swl = new StopWordList();
  }
  return swl;
 }
}

DCCT4090 - IR Term Project

Tuesday, July 1, 2008

Adding Synonym Search

Monday, June 30, 2008

Challenges Faced When Adding to The Lexicon

What Search Model & Why?

Monday, June 2, 2008

Odds & Ends

Query Processor

Interface Design

Friday, May 30, 2008

Meeting Minutes