FSelector: a Ruby gem for feature selection

algorithm                        shortcut    algo_type  applicability  feature_type
--------------------------------------------------------------------------------------------------
Accuracy                         Acc         weighting  multi-class    discrete
AccuracyBalanced                 Acc2        weighting  multi-class    discrete
BiNormalSeparation               BNS         weighting  multi-class    discrete
CFS_d                            CFS_d       searching  multi-class    discrete
ChiSquaredTest                   CHI         weighting  multi-class    discrete
CorrelationCoefficient           CC          weighting  multi-class    discrete
DocumentFrequency                DF          weighting  multi-class    discrete
F1Measure                        F1          weighting  multi-class    discrete
FishersExactTest                 FET         weighting  multi-class    discrete
FastCorrelationBasedFilter       FCBF        searching  multi-class    discrete
GiniIndex                        GI          weighting  multi-class    discrete
GMean                            GM          weighting  multi-class    discrete
GSSCoefficient                   GSS         weighting  multi-class    discrete
InformationGain                  IG          weighting  multi-class    discrete
INTERACT                         INTERACT    searching  multi-class    discrete
JMeasure                         JM          weighting  multi-class    discrete
KLDivergence                     KLD         weighting  multi-class    discrete
MatthewsCorrelationCoefficient   MCC, PHI    weighting  multi-class    discrete
McNemarsTest                     MNT         weighting  multi-class    discrete
OddsRatio                        OR          weighting  multi-class    discrete
OddsRatioNumerator               ORN         weighting  multi-class    discrete
PhiCoefficient                   PHI         weighting  multi-class    discrete
Power                            Power       weighting  multi-class    discrete
Precision                        Precision   weighting  multi-class    discrete
ProbabilityRatio                 PR          weighting  multi-class    discrete
Recall                           Recall      weighting  multi-class    discrete
Relief_d                         Relief_d    weighting  two-class      discrete
ReliefF_d                        ReliefF_d   weighting  multi-class    discrete
Sensitivity                      SN, Recall  weighting  multi-class    discrete
Specificity                      SP          weighting  multi-class    discrete
SymmetricalUncertainty           SU          weighting  multi-class    discrete
BetweenWithinClassesSumOfSquare  BSS_WSS     weighting  multi-class    continuous
CFS_c                            CFS_c       searching  multi-class    continuous
FTest                            FT          weighting  multi-class    continuous
KS_CCBF                          KS_CCBF     searching  multi-class    continuous
KSTest                           KST         weighting  two-class      continuous
PMetric                          PM          weighting  two-class      continuous
Relief_c                         Relief_c    weighting  two-class      continuous
ReliefF_c                        ReliefF_c   weighting  multi-class    continuous
TScore                           TS          weighting  two-class      continuous
WilcoxonRankSum                  WRS         weighting  two-class      continuous
LasVegasFilter                   LVF         searching  multi-class    discrete, continuous, mixed
LasVegasIncremental              LVI         searching  multi-class    discrete, continuous, mixed
Random                           Rand        weighting  multi-class    discrete, continuous, mixed
RandomSubset                     RandS       searching  multi-class    discrete, continuous, mixed

algorithm                         note
---------------------------------------------------------------------------------------
normalize_by_log!                 normalize by logarithmic transformation
normalize_by_min_max!             normalize by scaling into [min, max]
normalize_by_zscore!              normalize by converting into zscore
discretize_by_equal_width!        discretize by equal width among intervals
discretize_by_equal_frequency!    discretize by equal frequency among intervals
discretize_by_ChiMerge!           discretize by ChiMerge algorithm
discretize_by_Chi2!               discretize by Chi2 algorithm
discretize_by_MID!                discretize by Multi-Interval Discretization algorithm
discretize_by_TID!                discretize by Three-Interval Discretization algorithm

algorithm                         note                                   feature_type                     
---------------------------------------------------------------------------------------------
replace_by_fixed_value!           replace by a fixed value               discrete, continuous
replace_by_mean_value!            replace by mean feature value          continuous
replace_by_median_value!          replace by median feature value        continuous
replace_by_knn_value!             replace by weighted knn feature value  continuous
replace_by_most_seen_value!       replace by most seen feature value     discrete

$ gem install fselector

require 'fselector'

# use InformationGain (IG) as a feature selection algorithm
r1 = FSelector::IG.new

# read from random data (or csv, libsvm, weka ARFF file)
# no. of samples: 100
# no. of classes: 2
# no. of features: 15
# no. of possible values for each feature: 3
# allow missing values: true
r1.data_from_random(100, 2, 15, 3, true)

# number of features before feature selection
puts "  # features (before): "+ r1.get_features.size.to_s

# select the top-ranked features with scores >0.01
r1.select_feature_by_score!('>0.01')

# number of features after feature selection
puts "  # features (after): "+ r1.get_features.size.to_s

# you can also use a second alogirithm for further feature selection
# e.g. use the ChiSquaredTest (CHI) with Yates' continuity correction
# initialize from r1's data
r2 = FSelector::CHI.new(:yates, r1.get_data)

# number of features before feature selection
puts "  # features (before): "+ r2.get_features.size.to_s

# select the top-ranked 3 features
r2.select_feature_by_rank!('<=3')

# number of features after feature selection
puts "  # features (after): "+ r2.get_features.size.to_s

# save data to standard ouput as a weka ARFF file (sparse format)
# with selected features only
r2.data_to_weka(:stdout, :sparse)

require 'fselector'

# example 1
#


# creating an ensemble of feature selectors by using 
# a single feature selection algorithm (INTERACT) 
# by instance perturbation (e.g. random sampling)

# test for the type of feature subset selection algorithms
r = FSelector::INTERACT.new(0.0001)

# an ensemble of 40 feature selectors with 90% data by random sampling
re = FSelector::EnsembleSingle.new(r, 40, 0.90, :random_sampling)

# read SPECT data set (under the test/ directory)
re.data_from_csv('test/SPECT_train.csv')

# number of features before feature selection
puts '  # features (before): ' + re.get_features.size.to_s

# only features with above average count among ensemble are selected
re.select_feature!

# number of features after feature selection
puts '  # features before (after): ' + re.get_features.size.to_s


# example 2
#


# creating an ensemble of feature selectors by using 
# two feature selection algorithms: InformationGain (IG) and Relief_d. 
# note: can be 2+ algorithms, as long as they are of the same type, 
# either filter_by_feature_weighting or filter_by_feature_searching

# test for the type of feature weighting algorithms 
r1 = FSelector::IG.new
r2 = FSelector::Relief_d.new(10)

# an ensemble of two feature selectors
re = FSelector::EnsembleMultiple.new(r1, r2)

# read random discrete data (containing missing value)
re.data_from_random(100, 2, 15, 3, true)

# replace missing value because Relief_d 
# does not allow missing value
re.replace_by_most_seen_value!

# number of features before feature selection
puts '  # features (before): ' + re.get_features.size.to_s

# based on the max feature score (z-score standardized) among
# an ensemble of feature selectors
re.ensemble_by_score(:by_max, :by_zscore)

# select the top-ranked 3 features
re.select_feature_by_rank!('<=3')

# number of features after feature selection
puts '  # features (after): ' + re.get_features.size.to_s

require 'fselector'

# the Information Gain (IG) algorithm requires data with discrete feature
r = FSelector::IG.new

# but the Iris data set contains continuous features
r.data_from_url('http://repository.seasr.org/Datasets/UCI/arff/iris.arff', :weka)
    
# let's first discretize it by ChiMerge algorithm at alpha=0.10
# then perform feature selection as usual
r.discretize_by_ChiMerge!(0.10)

# number of features before feature selection
puts '  # features (before): ' + r.get_features.size.to_s

# select the top-ranked feature
r.select_feature_by_rank!('<=1')

# number of features after feature selection
puts '  # features (after): ' + r.get_features.size.to_s