INDEXER:
A Learning Companion For Document Browsing.

Abstract

Definitions:

KEYWORD:

It is a word which the system evaluates to be on a high priority category
using position, frequency and other factors.

INTERESTING WORD:

It is a narrower category of words and contains the words which the system
after training, will present as a clickable index in the slave window.

A Real World Analogy Of The Problem:


X is taking a course in Networking and wants to read about the OSI layering. He goes to the library and gets the results of the subject search "Networks". He now has a long list of call numbers and it could take a long time for him to scan through all the books and get a narrower list of books which actually contain a detailed explanation of the OSI layers. It would have been easier if he could more easily get to the level where he has the list of books containing information on the specific "interesting word". It is a similar situation in internet search. A keyword search of a document can be too broad for a user and to get to the documents which will be of immediate interest to him could be hectic.
To alleviate the above situation, this is a proposal to discuss an intelligent agent, hereafter called the INDEXER.

Architectural Components:

• Netscape 1.1N
  
• Perl
  
• A custom proxy server: It acts as an intermediate server which provides a path through which all browser requests pass.
  
• An INDEXER program
  
• SUMMARIZER program by Joe Felder will be included in the package.
  
• A master client(Netscape window in which document will be displayed).
  
• An INDEXER slave client(Netscape window with clickable keywords).
  
• A SUMMARIZER slave client(Netscape window in which TOC will be displayed).
  

Solution Approach:

The basis will be using the vector space information retrieval paradigm. Word weights will be found using a specific schema and a "interestingness" value will be assigned to each of the words. Learning algoritm will be supplied with number of keywords and number of documents. The document will be represented as vector of D, with each element being a keyword. The document also has a vector V, where each element is the weight of that word in the document. Word weights will be found using an appropriate weighting formulae. Now the words are presented on the menu screen to the user so that can be weighted on a scale of -5 to +5. The system trains itself using an appropriate Learning algorithm to identify the appropriate words.



The Blue Print For Indexing:

>


 1.   Obtain training document (Display in master window).

 2.   Identify individual text words.

 3.   Use stop list to delete common words.

 4.   Use suffix stripping algorithms.

 5.   Identify the retrieved words as relevant(score=1) 
      and non-relevant(score=0) to the user.

 6.   Compute term weights of relevant words using prescribed formula.

 7.   Place words and weights in user preferences file.

 8.   Obtain new document and repeat steps 2-4. 

 9.   Place the keywords in a document vector.

10.   Find similarity coefficient of u.p.f and document vector.

11.   Find weights of newly found terms.

12.   Reformulate contents of u.p.f. using relevance feedback formula.

13.   Create a clickable index of "interesting words" which are
      the contents of u.p.f. (Display in slave window)

14.   Return to step 8





Learning and Relevance Feedback:


The assumption is that the user has a specific subject in mind and already has a broad list of documents containing references to the subject. So all documents are related to the subject. The user has the choice ofintroducing one or more training documents, else the default will be the contents of the user preferences file.
In case the user chooses to provide the training document which is preferably the document with the highest hit rate in the keyword search, it is opened in the master window. The INDEXER identifies the key-words from this document. It presents this list to the user, who in turn evaluates the words with scores of 0(not-relevant) or 1(relevant). This is done till the user is satisfied with the training data which is stored in a user preferences file along with the corresponding weights computed by the INDEXER using the following formula

     w(i,j) = t(i,j)*log(N/d(i))  


          where w(i,j) = weight of ith word in jth document
                t(i,j) = ith term frequency in jth document
                N      = number of documents evaluated
                d(i)   = number of documents in which word i appears

The preference vector is one whose elements are the weights of the relevant words placed in order.  
              P = w(i,j) for all i, j
          
Once the system is trained to satisfaction, a new document is retrieved and its keywords are determined. The weights of the new words, d(i,j), found using the above formula are placed in a document vector.
              D =  d(i,j) for all i,j

The similarity coefficient which gives 

placed
that the  and the system adjusts its parameters and makes a more precise list of "interesting" words. The system undergoes training till it can identify the "interesting" words on its own,  for the rest of the documents.  It then displays a clickable index of "interesting" words for each of the documents.

Future Work

1. Integrating Learning capacity to INDEXER
2. "Stemming of words" will be included as part of INDEXER and will be based on the heuristics used by the 'SMART' system.
3. It would be nice to have a pop up window to set up selection heuristics and display learning But for now it will be done transparently.
4. The implementation of more advanced Visual techniques to choose keywords (TAU system, Swaminathan 1993) can be added as a further development but may not be included in this project.
5. "Synonym analysis" can also be added as a feature of the system.