meka.core

Class StatUtils

java.lang.Object

meka.core.StatUtils

public abstract class StatUtils
extends Object

StatUtils - Helpful statistical functions.

Version:

March 2013 - Multi-target Compatible

Author:

Jesse Read (jesse@tsc.uc3m.es)

Field Summary

Fields

Modifier and Type	Field and Description
static double[]	CRITICAL Critical value used for Chi^2 test.

Constructor Summary

Constructors

Constructor and Description
StatUtils()

Method Summary

Modifier and Type	Method and Description
static double[][]	chi2(double[][][] M, double[][][] Exp) Chi^2 - Chi-squared test.
static double[][]	chi2(weka.core.Instances D) Chi^2 - Do the chi-squared test on all pairs of labels.
static double	chi2(weka.core.Instances Y, int j, int k) Chi^2 - Do the chi-squared test on the j-th and k-th labels in Y.
static double[][]	condDepMatrix(weka.core.Instances D, Result result) CondDepMatrix - Get a Conditional Dependency Matrix.
static double[][]	F(weka.core.Instances D) F - Relative frequency matrix (between p(j),p(k) and p(j,k)) in dataset D.
static int[][]	getApproxC(weka.core.Instances D) GetApproxP - A fast version of getC(D), based on frequent sets.
static double[][]	getApproxP(weka.core.Instances D) GetApproxP - A fast version of getP(D), based on frequent sets.
static int[][]	getC(weka.core.Instances D) GetC - Get pairwise co-ocurrence counts from the training data D.
static double[][]	getP(weka.core.Instances D) GetP - Get a pairwise empirical joint-probability matrix P[][] from dataset D.
static double[][]	getP(int[][] C, int N)
static double[][]	H(weka.core.Instances D) H - Get a Conditional Entropy Matrix.
static double[][]	H(int[][] C, int N) H - Get a Conditional Entropy Matrix.
static double	H(int[][] C, int j, int k, int Ncount) H - Conditional Entropy H(y_j|y_k).
static double[][]	I(double[][] P) I - Mutual Information -- fast version, must calcualte P[][] = getP(D) first.
static double	I(double[][] P, int j, int k) I - Mutual Information.
static double[][]	I(weka.core.Instances D, int L) I - Get an Unconditional Depndency Matrix.
static double	I(weka.core.Instances D, int j, int k) I - Mutual Information.
static double[][]	I(int[][] C, int N) I - Get a Mutual Information Matrix.
static double	I(int[][] C, int j, int k, int Ncount) I - Mutual Information I(y_j;y_k).
static double[][]	jPMF(weka.core.Instances D, int j, int k) jPMF - Joint PMF.
static double[][][]	jPMF(weka.core.Instances D, int j, int k, int l) Joint Distribution.
static double[][]	LEAD(weka.core.Instances D, weka.classifiers.Classifier h, Random r) LEAD - Performs LEAD on dataset 'D', using BR with base classifier 'h', under random seed 'r'.
static double[][]	LEAD(weka.core.Instances D, weka.classifiers.Classifier h, Random r, String MDType)
static double[][]	LEAD(weka.core.Instances D, Result result)
static double[][]	LEAD(weka.core.Instances D, Result R, String MDType) LEAD - Performs LEAD on dataset 'D', with corresponding gresult 'R', and dependency measurement type 'MDType'.
static double[][]	LEAD2(weka.core.Instances D, Result result) LEAD.
static void	main(String[] args) Main - do some tests.
static double[][]	margDepMatrix(weka.core.Instances D, String op) MargDepMatrix - Get an Unconditional Depndency Matrix.
static double[]	P(double[][] Y, int[] x) P - Empirical prior.
static double[]	P(double[][] Y, int[] j, int[] x) P - Empirical prior.
static double	p(double[][] Y, int j, int k) p - Empirical prior.
static double	P(double[][] Y, int j, int v, int k, int w) P - Empirical joint.
static double	P(weka.core.Instances D, int[] j, int[] v) P - Empirical joint.
static double	p(weka.core.Instances D, int j, int j_) p - Empirical prior.
static double	P(weka.core.Instances D, int j, int v, int k, int w) p - Empirical joint.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CRITICAL

public static final double[] CRITICAL

Critical value used for Chi^2 test.

Constructor Detail

StatUtils

public StatUtils()

Method Detail

P

public static double[] P(double[][] Y,
                         int[] x)

P - Empirical prior.

Parameters:

Y - label matrix

x - label values

Returns:

[P(Y_1==x[1]), P(Y_2==x[2]), ..., P(Y_L==x[L])]

P

public static double[] P(double[][] Y,
                         int[] j,
                         int[] x)

P - Empirical prior.

Parameters:

Y - label matrix

x - label values

j - label indices

Returns:

[P(Y_j[1]==x[1]), P(Y_j[2]==x[2]), ..., P(Y_j[L]==x[L])]

p

public static double p(double[][] Y,
                       int j,
                       int k)

p - Empirical prior. In the multi-label case, k in {0,1}

Parameters:

Y - label matrix

j - label index

k - label value

Returns:

P(Y_j==k) in Y.

p

public static double p(weka.core.Instances D,
                       int j,
                       int j_)

p - Empirical prior. In the multi-label case, k in {0,1}

Parameters:

D - Instances

j - label index

j_ - label value

Returns:

P(Y_j==j_) in D.

P

public static double P(double[][] Y,
                       int j,
                       int v,
                       int k,
                       int w)

P - Empirical joint. Multi-target friendly.

Parameters:

Y - label matrix

j - 1st label index

v - 1st label value

k - 2nd label index

w - 2nd label value

Returns:

P(Y_j = v, Y_k = w) in Y.

P

public static double P(weka.core.Instances D,
                       int j,
                       int v,
                       int k,
                       int w)

p - Empirical joint. Multi-target friendly.

Parameters:

D - Instances

j - 1st label index

v - 1st label value

k - 2nd label index

w - 2nd label value

Returns:

P(Y_j = v, Y_k = w) in D.

P

public static double P(weka.core.Instances D,
                       int[] j,
                       int[] v)

P - Empirical joint. Multi-target friendly.

Parameters:

D - Instances

j - label indices, e.g., 1,2,3

v - label values, e.g., 0,0,1

Returns:

P(x_1,x_2,x_3 = v_1,v_2,v_3) for j = 1,2,3 in D

jPMF

public static double[][] jPMF(weka.core.Instances D,
                              int j,
                              int k)

jPMF - Joint PMF.

Returns:

the joint PMF of the j-th and k-th labels in D.

jPMF

public static double[][][] jPMF(weka.core.Instances D,
                                int j,
                                int k,
                                int l)

Joint Distribution.

Returns:

the joint PMF of the j-th and k-th and lthlabels in D.

getP

public static double[][] getP(weka.core.Instances D)

GetP - Get a pairwise empirical joint-probability matrix P[][] from dataset D.
NOTE multi-label only

getApproxC

public static int[][] getApproxC(weka.core.Instances D)

GetApproxP - A fast version of getC(D), based on frequent sets. Actually, if we don't prune, this is not even approximate -- it is the real empirical P.

getApproxP

public static double[][] getApproxP(weka.core.Instances D)

GetApproxP - A fast version of getP(D), based on frequent sets. Actually, if we don't prune, this is not even approximate -- it is the real empirical P.

getP

public static double[][] getP(int[][] C,
                              int N)

getC

public static int[][] getC(weka.core.Instances D)

GetC - Get pairwise co-ocurrence counts from the training data D.
NOTE multi-label only

Returns:

C[][] where C[j][k] is the number of times where Y[i][j] = 1 and y[i][k] = 1 over all i = 1,...,N

I

public static double I(int[][] C,
                       int j,
                       int k,
                       int Ncount)

I - Mutual Information I(y_j;y_k). multi-label only -- count version

Parameters:

C - count matrix

j - j-th label index

k - k-th label index

Ncount - number of instances in the training set

Returns:

H(Y_j|Y_k)

H

public static double H(int[][] C,
                       int j,
                       int k,
                       int Ncount)

H - Conditional Entropy H(y_j|y_k). multi-label only

Parameters:

C - count matrix

j - j-th label index

k - k-th label index

Ncount - number of instances in the training set

Returns:

H(Y_j|Y_k)

I

public static double[][] I(double[][] P)

I - Mutual Information -- fast version, must calcualte P[][] = getP(D) first.

Returns:

I[][]

See Also:

I(double[][], int, int)

I

public static double I(double[][] P,
                       int j,
                       int k)

I - Mutual Information.
NOTE binary only

Returns:

I(Y_j;Y_k) in dataset D.

I

public static double I(weka.core.Instances D,
                       int j,
                       int k)

I - Mutual Information.
NOTE Multi-target friendly (does not assume binary labels).
NOTE a bit slow

Returns:

I(Y_j;Y_k) in dataset D.

I

public static double[][] I(weka.core.Instances D,
                           int L)

I - Get an Unconditional Depndency Matrix. (Works for both ML and MT data).

Parameters:

D - dataset

L - number of labels

Returns:

a L*L matrix representing Unconditional Depndence.

chi2

public static double chi2(weka.core.Instances Y,
                          int j,
                          int k)

Chi^2 - Do the chi-squared test on the j-th and k-th labels in Y.
NOTE multi-label only! @TODO Use enumerateValues() !!!
If they are correlated, this means unconditional dependence!

Returns:

The chi-square statistic for labels j and k in Y.

chi2

public static double[][] chi2(weka.core.Instances D)

Chi^2 - Do the chi-squared test on all pairs of labels.

Parameters:

D - dataset

Returns:

The chi-square statistic matrix X

See Also:

chi2(Instances, int, int)

chi2

public static double[][] chi2(double[][][] M,
                              double[][][] Exp)

Chi^2 - Chi-squared test. If they are correlated, this means unconditional dependence!

Parameters:

M - measured joint P(Y_1,Y_2)

Exp - expect joint P(Y_1)P(Y_2) given null hypothesis

Returns:

The chi-square statistic for labels j and k in Y; normalized by critical value.

margDepMatrix

public static double[][] margDepMatrix(weka.core.Instances D,
                                       String op)

MargDepMatrix - Get an Unconditional Depndency Matrix.

Parameters:

D - dataset

op - how we will measure the dependency

Returns:

a L*L matrix representing Unconditional Depndence.

I

public static double[][] I(int[][] C,
                           int N)

I - Get a Mutual Information Matrix.

H

public static double[][] H(int[][] C,
                           int N)

H - Get a Conditional Entropy Matrix.

H

public static double[][] H(weka.core.Instances D)

H - Get a Conditional Entropy Matrix.

F

public static double[][] F(weka.core.Instances D)

F - Relative frequency matrix (between p(j),p(k) and p(j,k)) in dataset D.

condDepMatrix

public static double[][] condDepMatrix(weka.core.Instances D,
                                       Result result)

CondDepMatrix - Get a Conditional Dependency Matrix. Based on Zhang's 'LEAD' approach, where
the probability of labels j and k both getting errors on the same instance is error(j)*error(k) if the actual co-occurence is otherwise.

Parameters:

D - dataset

Returns:

a L*L matrix of Unconditional Depndence.

LEAD2

public static double[][] LEAD2(weka.core.Instances D,
                               Result result)

LEAD. Do the chi-squared LEAD test on all labels in D. We would expect the 3 kinds of error to be uncorrelacted. However, if they are significantly correlated, this means that there is conditional dependence!

LEAD

public static double[][] LEAD(weka.core.Instances D,
                              Result R,
                              String MDType)

LEAD - Performs LEAD on dataset 'D', with corresponding gresult 'R', and dependency measurement type 'MDType'.

LEAD

public static double[][] LEAD(weka.core.Instances D,
                              Result result)

LEAD

public static double[][] LEAD(weka.core.Instances D,
                              weka.classifiers.Classifier h,
                              Random r)
                       throws Exception

LEAD - Performs LEAD on dataset 'D', using BR with base classifier 'h', under random seed 'r'.
WARNING: changing this method will affect the perfomance of e.g., BCC -- on the other hand the original BCC paper did not use LEAD, so don't worry.

Throws:

Exception

LEAD

public static double[][] LEAD(weka.core.Instances D,
                              weka.classifiers.Classifier h,
                              Random r,
                              String MDType)
                       throws Exception

Throws:

Exception

main

public static void main(String[] args)
                 throws Exception

Main - do some tests.

Throws:

Exception