Looking for a Haystack: Selecting Data Sources in a Distributed Retrieval System

Loading...
Thumbnail Image
Date
2010-06-01
Journal Title
Journal ISSN
Volume Title
Publisher
[Bloomington, Ind.] : Indiana University
Abstract
The Internet contains billions of documents and thousands of systems for searching over these documents. Searching for a useful document can be as difficult as the proverbial search for a needle in a haystack. Each search engine provides access to a different collection of documents. Collections may be large or small, focused or comprehensive. Focused collections may be centered on any possible topic, and comprehensive collections typically have particular topical areas with higher concentrations of documents. Some of these collections overlap, but many documents are available from only a single collection. To find the most needles, one must first select the best haystacks. This dissertation develops a framework for automatic selection of search engines. In this framework, the collection underlying each search engine is examined to determine how properties such as central topic, size, and degree of focus affect retrieval performance. When measured with appropriate techniques, these properties may be used to predict performance. A new distributed retrieval algorithm that takes advantage of this knowledge is presented and compared to existing retrieval algorithms.
Description
Thesis (PhD) - Indiana University, Computer Sciences, 2006
Keywords
distributed information retrieval, collection selection, search engine, metasearch, database selection, information retrieval
Citation
DOI
Link(s) to data and video for this item
Relation
Rights
Type
Doctoral Dissertation