Mining Low-support Discriminative Patterns from Dense and High-dimensional Data (Tech Report)


Gang Fang, Gaurav Pandey, Wen Wang, Manish Gupta,  Michael Steinbach and Vipin Kumar

To appear (link) in IEEE Transaction on Knowledge and Data Engineering (TKDE).

Abstract

Discriminative patterns can provide valuable insights into datasets with class labels, that may not be available from the individual features or predictive models built using them. Most existing approaches work efficiently for sparse or low-dimensional datasets. However, for dense and highdimensional datasets, they have to use high thresholds to produce the complete results within limited time, and thus, may miss interesting low-support patterns. In this paper, we address the necessity of trading off the completeness of discriminative pattern discovery with the efficient discovery of lowsupport discriminative patterns from such datasets. We propose a family of anti-monotonic measures named SupMaxK that organize the set of discriminative patterns into nested layers of subsets, which are progressively more complete in their coverage, but require increasingly more computation. In particular, the member of SupMaxK with K = 2, named SupMaxPair, is suitable for dense and high-dimensional datasets. Several experiments on a cancer gene expression dataset demonstrate that there are low-support patterns that can be discovered using SupMaxPair, but not by existing approaches, and that these patterns are statistically significant and biologically relevant. This illustrates the complementarity of SupMaxPair to existing approaches for discriminative pattern discovery. The codes and dataset for this paper are available at http://vk.cs.umn.edu/SMP/.

The pre-processed Gene Expression Data can be downloaded from here: Download datasets.

Source Codes for SMP (Download SMP) : A bug is corrected on September 14, 2011. Currently Linux version only, requires Matlab. A version that is independent on Matlab will be released soon.)

(1) Link of the original gene expression dataset (van de Vijer et al. (2002)) : Link

(2) Instructions on how to run the codes are provided in the file "readme.txt" in both the source codes folders

(3) Parameters details are provided in the paper.

(4) Appendix A: Experiments to show how well SupMaxK estimates DiffSup when K increases.

If any questions, please email : Gang Fang, gangfang cs umn edu