Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Analysis of feature selection measures for text categorisation

Analysis of feature selection measures for text categorisation The curse of dimensionality has made dimension reduction an essential step in text categorisation. Feature selection is an approach for dimension reduction. In this paper an analysis on feature selection measures for text categorisation is performed. Under the unsupervised approach document frequency and under the supervised approach chi-square, odds ratio, mutual information, and information gain are considered for analysis. They are considered here because they are the widely used and effective measures. Analysis of these measures is performed using the 20 newsgroups dataset. Twenty newsgroups dataset consists of closely related categories as well as highly unrelated categories. Certain categories of 20 newsgroups dataset are selected and organised into three groups of overlapping (highly related) classes, non-overlapping (highly unrelated) classes and combination of overlapping and non-overlapping classes. Feature selection and subsequent classification is applied to the three groups separately and the classification performance is studied based on the feature selection measures. The noticeable behaviour was with odds ratio measure in that it performed well for non-overlapping group and overlapping groups considered separately and was poorer in performance for the group containing both overlapping and non-overlapping categories. Remaining measures showed consistent behaviour for all the three groups. Classification was achieved http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png International Journal of Enterprise Network Management Inderscience Publishers

Analysis of feature selection measures for text categorisation

Download PDF
16 pages

Loading next page...
 
/lp/inderscience-publishers/analysis-of-feature-selection-measures-for-text-categorisation-v0KFyXM2PY
Publisher
Inderscience Publishers
Copyright
Copyright © 2017 Inderscience Enterprises Ltd.
ISSN
1748-1252
eISSN
1748-1260
DOI
10.1504/IJENM.2017.083606
Publisher site
See Article on Publisher Site

Abstract

The curse of dimensionality has made dimension reduction an essential step in text categorisation. Feature selection is an approach for dimension reduction. In this paper an analysis on feature selection measures for text categorisation is performed. Under the unsupervised approach document frequency and under the supervised approach chi-square, odds ratio, mutual information, and information gain are considered for analysis. They are considered here because they are the widely used and effective measures. Analysis of these measures is performed using the 20 newsgroups dataset. Twenty newsgroups dataset consists of closely related categories as well as highly unrelated categories. Certain categories of 20 newsgroups dataset are selected and organised into three groups of overlapping (highly related) classes, non-overlapping (highly unrelated) classes and combination of overlapping and non-overlapping classes. Feature selection and subsequent classification is applied to the three groups separately and the classification performance is studied based on the feature selection measures. The noticeable behaviour was with odds ratio measure in that it performed well for non-overlapping group and overlapping groups considered separately and was poorer in performance for the group containing both overlapping and non-overlapping categories. Remaining measures showed consistent behaviour for all the three groups. Classification was achieved

Journal

International Journal of Enterprise Network Management – Inderscience Publishers

Published: Jan 1, 2017

There are no references for this article.