What is Automatic Indexing?

Automatic indexing, a groundbreaking advancement in information retrieval, has revolutionized how we organize and access vast data repositories. Creating indices for large volumes of documents has traditionally been labor-intensive and time-consuming, requiring human experts to analyze and categorize each piece of information carefully. However, with the advent of automatic indexing techniques, sophisticated algorithms and machine learning models take center stage, efficiently processing massive amounts of text and automatically generating relevant and accurate indexes. This paradigm shift has not only saved countless hours of human effort. Still, it has also significantly enhanced the precision and recall of search results, opening up new possibilities in various domains, from digital libraries and databases to web search engines and enterprise knowledge management systems.

1.1 What is Automatic Indexing?

Automatic indexing refers to the process of generating indexes for large collections of documents or data automatically, without the need for human intervention. In traditional manual indexing, human indexers carefully analyze each document and assign appropriate keywords or descriptors to represent the document’s content. However, this process can be time-consuming and resource-intensive, especially when dealing with massive volumes of information.

Automatic indexing leverages advanced technologies, such as natural language processing (NLP) and machine learning, to extract relevant terms or keywords from the documents automatically and create an index. The system scans the text, identifies important words or phrases, and assigns them as index entries, which can be used for efficient information retrieval. Several techniques are employed in automatic indexing, including statistical analysis of word frequencies, linguistic analysis, and machine learning algorithms. These methods aim to identify the most significant terms in the document that best represent its subject matter. Some automatic indexing systems may also employ thesauri or controlled vocabularies to ensure consistent and accurate indexing.

Automatic indexing offers several benefits, such as increased efficiency, scalability, and consistency in generating indexes. It significantly reduces the manual effort required in traditional indexing, making it feasible to process large collections of documents quickly. Moreover, automatic indexing can improve the precision and recall of search results, enhancing the overall user experience in information retrieval systems. This technology finds applications in various fields, including digital libraries, databases, search engines, content management systems, and other information organization and retrieval platforms. As technology advances, automatic indexing continues to evolve, offering more sophisticated and accurate ways to index and access information efficiently.

1.2 Manual Indexing vs. Automatic Indexing:

Manual indexing and automatic indexing are two distinct approaches to organizing and categorizing information, each with its advantages and limitations. Here’s a comparison between manual indexing and automatic indexing:

1.3 Methods of Computerised Indexing:

A. Keyword Indexing: An indexing system without controlling the vocabulary may be referred to as ‘Natural Language Indexing’ or sometimes as ‘Free Text Indexing.’ Keyword indexing is also known as Natural Language or Free Text Indexing. ‘Keyword’ means a catchword or significant word or subject denoting a word taken mainly from the titles and/or sometimes from the abstract or text of the document for indexing. Thus, keyword indexing is based on the natural language of the documents to generate index entries, and no controlled vocabulary is required for this indexing system. Keyword indexing is not new. It existed in the nineteenth century when it was called a ‘catchword indexing’. Computers began to be used to aid information retrieval systems in the 1950s. The Central Intelligence Agency (CIA) of the USA is said to be the first organization to use the machine-produced keywords index from Title since 1952. H.P Luhn and his associates produced and distributed copies of machine-produced permuted title indexes at the International Conference of Scientific Information held at Washington in 1958, which he named the Keyword-In-Context (KWIC) index and reported the method of generation of the KWIC index in a paper. American Chemical Society established the value of KWIC after its adoption in 1961 for its publication ‘Chemical Titles’:

KWIC (Keyword-In-Context) Index:

As mentioned earlier, H.P. Luhn is credited for developing the KWIC index. This index was based on the keywords in the title of a paper and was produced with the help of computers. Each entry in the KWIC index consists of the following three parts:

a) Keywords: Significant or subject-denoting words which serve as approach terms;

b) Context: Keywords selected also specify the particular context of the document (i.e., usually the rest of the terms of the title).

c) Identification or Location Code: Code used (usually the serial numbers of the entries in the main part) to provide the document’s address where the full bibliographic description will be available.

The operational stages of KWIC indexing consist of the following:

a) Mark the significant words or prepare the ‘stop list’ and keep it on the computer. The ‘stop list’ refers to a list of words considered to have no value for indexing/retrieval. These may include insignificant words like articles (a, an, the), prepositions, conjunctions, pronouns, and auxiliary verbs together with such general words as ‘aspect,’ ‘different,’ ‘very,’ etc. Each major search system has defined its own ‘stop list’ ;

b) Selection of keywords from the title and/or abstract and/or full text of the document, excluding the stop words;

c) KWIC routine rotates the title to make it accessible from each significant term. Given this, manipulate the title or title-like phrase in such a way that each keyword serves as the approach term and comes in the beginning (or in the middle) by rotation, followed by the rest of the title. d) Separate the last word and first word of the title by using a symbol, say, stroke [ / ] (sometimes an asterisk “*” is used) in an entry. Keywords are usually printed in bold typeface; e) Put the identification/location code at the right end of each entry; and finally

f) Arrange the entries alphabetically by keywords.

Let us take the title ‘control of damages of rice by insets’ to demonstrate the index entries generated through the KWIC principle:

Control of damages of rice by insets 118

Damages of rice by insets / Control of 118

Insets / Control of damages of rice by 118

Rice by insets / Control of damages of 118

The keywords can also be positioned at the center in the computer-generated index.

Variations of KWIC:
Two important other versions of the keyword index are KWOC and KWAC, which are discussed below:

KWOC (key-word out-of-context) Index:

The KWOC is a variant of the KWIC index. Here, each keyword is taken out and printed separately in the left-hand margin, with the complete title in its normal order printed to the right. For examples,

Control

Control of damages of rice by insets 118

Damages

Control of damages of rice by insets 118

Insets

Control of damages of rice by insets 118

Rice

Control of damages of rice by insets 118

Sometimes, the keyword is printed as a heading, and the title is printed in the next line instead of the same line as shown above. For examples,

Control

Control of damages of rice by insets  118

Damages

Control of damages of rice by insets  118

Insets

Control of damages of rice by insets 118

Rice

Control of damages of rice by insets 118

KWAC (key-word Augmented-in-context) Index:

KWAC also stands for ‘key-word-and-context.’ In many cases, the title cannot always represent the thought content of the document co-extensively. KWIC and KWOC could not solve the problem of retrieving irrelevant documents. To solve the problem of false drops, KWAC enriches the keywords of the title with additional keywords taken either from the abstract or from the original text of the document and inserted into the title or added at the end to give further index entries. KWAC is also called enriched KWIC or KWOC. CBAC (Chemical Biological Activities) of BIOSIS uses the KWAC index, where the title is enriched by another title-like phrase formulated by the indexer.

Other Versions:

A number of varieties of keyword indexes are noticed in the literature, and they differ only in terms of their formats, but indexing techniques and principles remain more or less the same. They are

i) KWWC (Key-Word-With-Context) Index, where only the part of the title (instead of a full title) relevant to the keyword is considered as an entry term.

ii) KEYTALPHA (Key-Term Alphabetical) Index. It is a permuted subject index that lists only keywords assigned to each abstract. Keytalpha index is being used in the ‘Oceanic Abstract.’

iii) WADEX (Word and Author Index). It is an improved version of the KWIC index where the authors’ names are also treated as keywords in addition to the significant subject term and thus facilitates a satisfied author approach to the documents. It is used in ‘Applied Mechanics Review’. AKWIC (Author and keyword in context) index is another version of WADEX.

iv) DKWTC (Double KWIC) Index. It is another improved version of the KWIC index.

v) KLIC (Key-Letter-In-Context) Index. This system allows truncation of the word ( instead of a complete word), either at the beginning (i.e., left truncation) or at the end (i.e., right truncation), where a fragment (i.e., key letters) can be specified, and the computer will pick up any term containing that fragment. The Chemical Society (London) published a KLIC index as a guide to truncation. The KLIC index indicates which terms any particular word fragment will capture.

Uses of Keyword Index:

A number of indexing and abstracting services prepare their subject indexes by using keyword indexing techniques. They are nothing but variations of keyword indexing apart from those mentioned above. Some notable examples are:

• • • Chemical Titles;
    • BASIC (Biological Abstracts Subject In Context);
    • Keyword Index of Chemical Abstracts;
    • CBAC (Chemical Biological Activities);
    • KWIT (Keyword-In-Title) of Laurence Burkeley Laboratory;
    • SWIFT (Selected Words in Full Titles); and
    • SAPIR (System of Automatic Processing and Indexing of Reports).

1.4 What are the possible advantages of Automatic Indexing?

Automatic indexing offers several advantages, making it a valuable tool for managing and accessing large volumes of information. Some of the possible advantages of automatic indexing include:

1. Automatic indexing is a highly efficient process compared to manual indexing. It can handle large volumes of documents and data much faster than human indexers. This efficiency is crucial when dealing with vast collections, such as digital archives, research databases, or web content, where manual indexing would be impractical and time-consuming.
2. One of the key strengths of automatic indexing is its scalability. As the data size grows, the indexing process can scale up without a significant increase in resources. This makes it particularly suitable for handling ever-expanding datasets and accommodating the continuous influx of new information.
3. Manual indexing is susceptible to variations and inconsistencies in index terms due to the subjective interpretation of human indexers. Automatic indexing, however, follows predefined rules and algorithms, ensuring higher consistency in the generated indexes. This consistency is crucial for accurate information retrieval and maintaining uniformity in indexing.
4. While the initial development and setup of an automatic indexing system may require investment, the long-term benefits of reduced human involvement make it cost-effective. The system’s ability to handle large volumes of data without additional human resources results in substantial cost savings over time.
5. Automatic indexing can index documents at a remarkable speed, especially compared to manual indexing. This rapid indexing process ensures that information is available for search and retrieval almost instantaneously, improving user experience and productivity.
6. Automatic indexing relies on algorithms and predefined rules, reducing the impact of human subjectivity and biases. This objectivity enhances the quality and consistency of index terms, leading to more reliable and accurate information retrieval.
7. Automatic indexing systems can effectively manage extensive vocabularies of terms and concepts, which may be challenging or time-consuming for human indexers. The ability to handle large vocabularies ensures comprehensive coverage and accuracy in representing the document content.
8. Advanced automatic indexing systems can process documents in multiple languages, making them suitable for multilingual content and international databases. This capability facilitates information retrieval across diverse linguistic contexts.
9. Some automatic indexing systems incorporate machine learning techniques, allowing them to adapt to language, terminology, and context changes. As the system processes more data and learns from user interactions, its indexing accuracy can improve over time.
10. Well-implemented automatic indexing can significantly enhance the precision and recall of search results. The system’s ability to identify relevant index terms based on statistical analysis and linguistic patterns leads to more targeted and accurate information retrieval.
11. Automatic indexing systems can be designed to update the index dynamically as new data becomes available or when existing documents are modified or added. This ensures the index stays current and reflects the most up-to-date information, providing users with the latest content.
12. Automatic indexing excels at processing unstructured data, such as text from web pages, documents, or social media content. It converts this unstructured information into structured and organized data, making it easily accessible and retrievable.

The advantages of automatic indexing contribute to streamlining information management, enhancing user experience, and facilitating knowledge discovery in various domains, from digital libraries and content management systems to search engines and enterprise knowledge bases.

Reference Article:

1. Unit-4 Indexing Systems and Techniques. (2017). Retrieved from http://egyankosh.ac.in/handle/123456789/11150