A Multi-level Approach to Intelligent Information Filtering: Model, System, and Evaluation

Abstract

To conduct efficient information filtering, uncertanties occurring at multiple levels must be managed. Uncertainties can occur due to changing document space as well as stochasticity and non-stationarity of the user. In this paper, a filtering model is proposed that decomposes the overall task into subsystem functionalities and highlights the need for multiple adaptation techniques to cope with uncertainties. A filtering system, named SIFTER, has been implemented based on the model, using established techniques in information retrieval and artificial intelligence. These techniques include document representation using vector-space model, document classification by unsupervised learning, and user modeling by reinforcement learning. The system can filter information based on content and user's specific interests. The user's interest is automatically learned with only limited user intervention in the form of optional relevance feedbacks for documents. We also describe extensive experimental studies conducted with SIFTER to filter computer and information science documents collected from the Internet and commercial database services. The experimental results demonstrate that the system performs very well in filtering documents in a realistic problem setting.