meka.core

Class Metrics

java.lang.Object

meka.core.Metrics

public abstract class Metrics
extends java.lang.Object

Metrics.java - Evaluation Metrics.

L_ are loss/error measures (less is better)

P_ are payoff/accuracy measures (higher is better).

For more on the evaluation and threshold selection implemented here, see
Jesse Read, Scalable Multi-label Classification. PhD Thesis, University of Waikato, Hamilton, New Zealand (2010).

Version:

Feb 2013

Author:

Jesse Read (jesse@tsc.uc3m.es)

Constructor Summary

Constructors

Constructor and Description
Metrics()

Method Summary

Methods

Modifier and Type	Method and Description
static double	F1(int[] s1, int[] s2) F1 - the F1 measure for two sets.
static double	L_Hamming(int[][] Y, int[][] Ypred) Hamming loss.
static double	L_Hamming(int[] y, int[] ypred) Hamming loss.
static double	L_JaccardDist(int[][] Y, int[][] Ypred) Jaccard Distance -- the loss version of Jaccard Index
static double	L_LogLoss(double y, double rpred, double C) L_LogLoss - the log loss between real-valued confidence rpred and true prediction y.
static double	L_LogLoss(int[][] Y, double[][] Rpred, double C) L_LogLoss - the log loss between real-valued confidences Rpred and true predictions Y with a maximum penalty C [Important Note: Earlier versions of Meka only normalised by N, and not N*L as here].
static double	L_LogLossD(int[][] Y, double[][] Rpred) L_LogLoss - the log loss between real-valued confidences Rpred and true predictions Y with a maximum penalty based on the number of examples D [Important Note: Earlier versions of Meka only normalised by N, and not N*L as here].
static double	L_LogLossL(int[][] Y, double[][] Rpred) L_LogLoss - the log loss between real-valued confidences Rpred and true predictions Y with a maximum penalty based on the number of labels L [Important Note: Earlier versions of Meka only normalised by N, and not N*L as here].
double	L_MAE(int[] y, double[] p) MAE
double	L_MSE(int[] y, double[] p) MSE
static double	L_OneError(int[][] Y, double[][] Rpred) OneError -
static double	L_RankLoss(int[][] Y, double[][] Rpred)
static double	L_RankLoss(int[] y, double[] rpred)
static double	L_RankLoss(int[] y, int[] r) Rank Loss - the average fraction of labels which are not correctly ordered.
static double	L_ZeroOne(int[][] Y, int[][] Ypred) 0/1 Loss.
static double	L_ZeroOne(int[] y, int[] ypred) 0/1 Loss.
static void	main(java.lang.String[] args) Do some tests.
static double	P_Accuracy(int[][] Y, int[][] Ypred) Jaccard Index -- often simply called multi-label 'accuracy'.
static double	P_Accuracy(int[] y, int[] ypred) Jaccard Index -- often simply called multi-label 'accuracy'.
static double	P_AveragePrecision(int[][] Y, double[][] Rpred)
static double	P_AveragePrecision(int[] y, double[] rpred)
static double	P_AveragePrecision(int[] y, int[] r) Average Precision - computes for each relevant label the percentage of relevant labels among all labels that are ranked before it.
static double	P_ExactMatch(int[][] Y, int[][] Ypred) Exact Match, i.e., 1 - [0/1 Loss].
static double	P_FalseNegatives(int[] y, int[] ypred) P_FalseNegatives - 0 but supposed to be 1 (the length of ypred \ y).
static double	P_FalsePositives(int[] y, int[] ypred) P_FalsePositives - 1 but supposed to be 0 (the length of y \ ypred).
static double	P_FmacroAvgD(int[][] Y, int[][] Ypred) F-Measure Macro Averaged by D - The F-measure macro averaged by example.
static double	P_FmacroAvgL(int[][] Y, int[][] Ypred) F-Measure Macro Averaged by L - The 'standard' macro average.
static double	P_FmicroAvg(int[][] Y, int[][] Ypred) P_FmicroAvg - Micro Averaged F-measure (F1, as if all labels in the dataset formed a single vector)
static double	P_Hamming(int[][] Y, int[][] Ypred) Hamming score aka label accuracy.
static double	P_Hamming(int[][] Y, int[][] Ypred, int j) Hamming score aka label accuracy.
static double	P_Harmonic(int[][] Y, int[][] Ypred) Harmonic Accuracy -- average over all labels.
static double	P_Harmonic(int[][] Y, int[][] Ypred, int j) Harmonic Accuracy -- for the j-th label.
static double	P_Harmonic(int[] y, int[] ypred) Harmonic Accuracy.
static double	P_JaccardIndex(int[][] Y, int[][] Ypred) Jaccard Index -- often simply called multi-label 'accuracy'.
double	P_LogLikelihood(int[] y, double[] p) Log Likelihood
static double	P_Precision(int[][] Y, int[][] Ypred, int j) P_Precision - (retrieved AND relevant) / retrieved
static double	P_Precision(int[] y, int[] ypred) P_Precision - (retrieved AND relevant) / retrieved
static double	P_Recall(int[][] Y, int[][] Ypred, int j) P_Recall - (retrieved AND relevant) / relevant
static double	P_Recall(int[] y, int[] ypred) P_Recall - (retrieved AND relevant) / relevant
static double	P_TrueNegatives(int[] y, int[] ypred) P_TrueNegatives - 0 and supposed to be 0.
static double	P_TruePositives(int[] y, int[] ypred) P_TruePositives - 1 and supposed to be 1 (the intersection, i.e., logical AND).

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Metrics

public Metrics()

Method Detail

P_ExactMatch

public static double P_ExactMatch(int[][] Y,
                  int[][] Ypred)

Exact Match, i.e., 1 - [0/1 Loss].

L_ZeroOne

public static double L_ZeroOne(int[] y,
               int[] ypred)

0/1 Loss.

L_ZeroOne

public static double L_ZeroOne(int[][] Y,
               int[][] Ypred)

0/1 Loss.

L_Hamming

public static double L_Hamming(int[] y,
               int[] ypred)

Hamming loss.

L_Hamming

public static double L_Hamming(int[][] Y,
               int[][] Ypred)

Hamming loss.

P_Hamming

public static double P_Hamming(int[][] Y,
               int[][] Ypred)

Hamming score aka label accuracy.

P_Hamming

public static double P_Hamming(int[][] Y,
               int[][] Ypred,
               int j)

Hamming score aka label accuracy.

P_Harmonic

public static double P_Harmonic(int[] y,
                int[] ypred)

Harmonic Accuracy. Multi-label only.

P_Harmonic

public static double P_Harmonic(int[][] Y,
                int[][] Ypred,
                int j)

Harmonic Accuracy -- for the j-th label. Multi-label only.

P_Harmonic

public static double P_Harmonic(int[][] Y,
                int[][] Ypred)

Harmonic Accuracy -- average over all labels. Multi-label only.

P_Accuracy

public static double P_Accuracy(int[] y,
                int[] ypred)

Jaccard Index -- often simply called multi-label 'accuracy'. Multi-label only.

P_Accuracy

public static double P_Accuracy(int[][] Y,
                int[][] Ypred)

Jaccard Index -- often simply called multi-label 'accuracy'. Multi-label only.

P_JaccardIndex

public static double P_JaccardIndex(int[][] Y,
                    int[][] Ypred)

Jaccard Index -- often simply called multi-label 'accuracy'. Multi-label only.

L_JaccardDist

public static double L_JaccardDist(int[][] Y,
                   int[][] Ypred)

Jaccard Distance -- the loss version of Jaccard Index

L_LogLoss

public static double L_LogLoss(double y,
               double rpred,
               double C)

L_LogLoss - the log loss between real-valued confidence rpred and true prediction y.

Parameters:

y - label

rpred - prediction (confidence)

C - limit (maximum loss of log(C))

Returns:

Log loss

L_LogLossL

public static double L_LogLossL(int[][] Y,
                double[][] Rpred)

L_LogLoss - the log loss between real-valued confidences Rpred and true predictions Y with a maximum penalty based on the number of labels L [Important Note: Earlier versions of Meka only normalised by N, and not N*L as here].

L_LogLossD

public static double L_LogLossD(int[][] Y,
                double[][] Rpred)

L_LogLoss - the log loss between real-valued confidences Rpred and true predictions Y with a maximum penalty based on the number of examples D [Important Note: Earlier versions of Meka only normalised by N, and not N*L as here].

L_LogLoss

public static double L_LogLoss(int[][] Y,
               double[][] Rpred,
               double C)

L_LogLoss - the log loss between real-valued confidences Rpred and true predictions Y with a maximum penalty C [Important Note: Earlier versions of Meka only normalised by N, and not N*L as here].

P_TruePositives

public static double P_TruePositives(int[] y,
                     int[] ypred)

P_TruePositives - 1 and supposed to be 1 (the intersection, i.e., logical AND).

P_FalsePositives

public static double P_FalsePositives(int[] y,
                      int[] ypred)

P_FalsePositives - 1 but supposed to be 0 (the length of y \ ypred).

P_TrueNegatives

public static double P_TrueNegatives(int[] y,
                     int[] ypred)

P_TrueNegatives - 0 and supposed to be 0.

P_FalseNegatives

public static double P_FalseNegatives(int[] y,
                      int[] ypred)

P_FalseNegatives - 0 but supposed to be 1 (the length of ypred \ y).

P_Precision

public static double P_Precision(int[] y,
                 int[] ypred)

P_Precision - (retrieved AND relevant) / retrieved

P_Recall

public static double P_Recall(int[] y,
              int[] ypred)

P_Recall - (retrieved AND relevant) / relevant

F1

public static double F1(int[] s1,
        int[] s2)

F1 - the F1 measure for two sets.

P_Precision

public static double P_Precision(int[][] Y,
                 int[][] Ypred,
                 int j)

P_Precision - (retrieved AND relevant) / retrieved

P_Recall

public static double P_Recall(int[][] Y,
              int[][] Ypred,
              int j)

P_Recall - (retrieved AND relevant) / relevant

P_FmicroAvg

public static double P_FmicroAvg(int[][] Y,
                 int[][] Ypred)

P_FmicroAvg - Micro Averaged F-measure (F1, as if all labels in the dataset formed a single vector)

P_FmacroAvgL

public static double P_FmacroAvgL(int[][] Y,
                  int[][] Ypred)

F-Measure Macro Averaged by L - The 'standard' macro average.

P_FmacroAvgD

public static double P_FmacroAvgD(int[][] Y,
                  int[][] Ypred)

F-Measure Macro Averaged by D - The F-measure macro averaged by example. The Jaccard index is also averaged this way.

L_OneError

public static double L_OneError(int[][] Y,
                double[][] Rpred)

OneError -

P_AveragePrecision

public static double P_AveragePrecision(int[][] Y,
                        double[][] Rpred)

P_AveragePrecision

public static double P_AveragePrecision(int[] y,
                        double[] rpred)

P_AveragePrecision

public static double P_AveragePrecision(int[] y,
                        int[] r)

Average Precision - computes for each relevant label the percentage of relevant labels among all labels that are ranked before it.

Parameters:

y - 0/1 labels [0, 0, 1 ] (true labels)

r - ranking position [1, 2, 0 ]

Returns:

Average Precision

L_RankLoss

public static double L_RankLoss(int[][] Y,
                double[][] Rpred)

L_RankLoss

public static double L_RankLoss(int[] y,
                double[] rpred)

L_RankLoss

public static double L_RankLoss(int[] y,
                int[] r)

Rank Loss - the average fraction of labels which are not correctly ordered. Thanks to Noureddine Yacine NAIR BENREKIA for providing bug fix for this.

Parameters:

y - 0/1 labels [0, 0, 1 ]

r - ranking position [1, 2, 0 ]

Returns:

Ranking Loss

P_LogLikelihood

public double P_LogLikelihood(int[] y,
                     double[] p)

Log Likelihood

L_MSE

public double L_MSE(int[] y,
           double[] p)

MSE

L_MAE

public double L_MAE(int[] y,
           double[] p)

MAE

main

public static void main(java.lang.String[] args)

Do some tests.