Metrics (meka 1.9.1 API)

java.lang.Object
- meka.core.Metrics

```
public abstract class Metrics
extends Object
```
Metrics.java - Evaluation Metrics.
L_ are loss/error measures (less is better)

P_ are payoff/accuracy measures (higher is better).
For more on the evaluation and threshold selection implemented here, see
Jesse Read, Scalable Multi-label Classification. PhD Thesis, University of Waikato, Hamilton, New Zealand (2010).

Version:

Feb 2013

Author:

Jesse Read (jesse@tsc.uc3m.es)

Constructor Summary

Constructors
Constructor and Description

Metrics()

Constructors
Constructor and Description
`Metrics()`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`static double[][]`	`align(int[] real, double[] pred)` Helper function for missing values in the labels and missing predictions (i.e., from abstaining classifiers).
`static int[][]`	`align(int[] real, int[] pred)` Helper function for missing values in the labels and missing predictions (i.e., from abstaining classifiers).
`static boolean`	`allMissing(int[] real)` Helper function for missing values in the labels.
`static weka.core.Instances[]`	`curveData(int[][] Y, double[][] P)` Get Data for Plotting PR and ROC curves.
`static weka.core.Instances`	`curveData(int[] y, double[] p)` Get Data for Plotting PR and ROC curves.
`static weka.core.Instances`	`curveDataMacroAveraged(int[][] Y, double[][] P)` Get Data for Plotting PR and ROC curves.
`static weka.core.Instances`	`curveDataMicroAveraged(int[][] Y, double[][] P)` Get Data for Plotting PR and ROC curves.
`static double`	`F1(int[] s1, int[] s2)` F1 - the F1 measure for two sets.
`static double`	`L_Hamming(int[][] Y, int[][] Ypred)` Hamming loss.
`static double`	`L_Hamming(int[] y, int[] ypred)` Hamming loss.
`static double`	`L_JaccardDist(int[][] Y, int[][] Ypred)` Jaccard Distance -- the loss version of Jaccard Index
`static double`	`L_LevenshteinDistance(int[][] Y, int[][] P)` Levenshtein Distance.
`static double`	`L_LevenshteinDistance(int[] y, int[] p)` Levenshtein Distance divided by the number of labels.
`static double`	`L_LogLoss(double y, double rpred, double C)` L_LogLoss - the log loss between real-valued confidence rpred and true prediction y.
`static double`	`L_LogLoss(int[][] Y, double[][] Rpred, double C)` L_LogLoss - the log loss between real-valued confidences Rpred and true predictions Y with a maximum penalty C [Important Note: Earlier versions of Meka only normalised by N, and not N*L as here].
`static double`	`L_LogLossD(int[][] Y, double[][] Rpred)` L_LogLoss - the log loss between real-valued confidences Rpred and true predictions Y with a maximum penalty based on the number of examples D [Important Note: Earlier versions of Meka only normalised by N, and not N*L as here].
`static double`	`L_LogLossL(int[][] Y, double[][] Rpred)` L_LogLoss - the log loss between real-valued confidences Rpred and true predictions Y with a maximum penalty based on the number of labels L [Important Note: Earlier versions of Meka only normalised by N, and not N*L as here].
`double`	`L_MAE(int[] y, double[] p)` MAE
`double`	`L_MSE(int[] y, double[] p)` MSE
`static double`	`L_OneError(int[][] Y, double[][] Rpred)` OneError -
`static double`	`L_RankLoss(int[][] Y, double[][] Rpred)`
`static double`	`L_RankLoss(int[] y, double[] rpred)`
`static double`	`L_RankLoss(int[] y, int[] r)` Rank Loss - the average fraction of labels which are not correctly ordered.
`static double`	`L_ZeroOne(int[][] Y, int[][] Ypred)` 0/1 Loss.
`static double`	`L_ZeroOne(int[] y, int[] ypred)` 0/1 Loss.
`static void`	`main(String[] args)` Do some tests.
`static int`	`numberOfMissingLabels(int[] real)` Helper function for missing values in the labels.
`static double`	`P_Accuracy(int[][] Y, int[][] Ypred)` Jaccard Index -- often simply called multi-label 'accuracy'.
`static double`	`P_Accuracy(int[] y, int[] ypred)` Jaccard Index -- often simply called multi-label 'accuracy'.
`static double`	`P_AveragePrecision(int[][] Y, double[][] Rpred)`
`static double`	`P_AveragePrecision(int[] y, double[] rpred)` Converts confidences in prediction array to ranking array, and continues with `P_AveragePrecision(using ranking array)`.
`static double`	`P_AveragePrecision(int[] y, int[] r)` Average Precision - computes for each relevant label the percentage of relevant labels among all labels that are ranked before it.
`static double`	`P_ExactMatch(int[][] Y, int[][] Ypred)` Exact Match, i.e., 1 - [0/1 Loss].
`static double`	`P_FalseNegatives(int[] y, int[] ypred)` P_FalseNegatives - 0 but supposed to be 1 (the length of ypred \ y).
`static double`	`P_FalsePositives(int[] y, int[] ypred)` P_FalsePositives - 1 but supposed to be 0 (the length of y \ ypred).
`static double`	`P_FmacroAvgD(int[][] Y, int[][] Ypred)` F-Measure Averaged by D - The F-measure macro averaged by example.
`static double`	`P_FmacroAvgL(int[][] Y, int[][] Ypred)` F-Measure Macro Averaged by L - The 'standard' macro average.
`static double`	`P_FmicroAvg(int[][] Y, int[][] Ypred)` P_FmicroAvg - Micro Averaged F-measure (F1, as if all labels in the dataset formed a single vector)
`static double`	`P_Hamming(int[][] Y, int[][] Ypred)` Hamming score aka label accuracy.
`static double`	`P_Hamming(int[][] Y, int[][] Ypred, int j)` Hamming score aka label accuracy.
`static double`	`P_Harmonic(int[][] Y, int[][] Ypred)` Harmonic Accuracy -- average over all labels.
`static double`	`P_Harmonic(int[][] Y, int[][] Ypred, int j)` Harmonic Accuracy -- for the j-th label.
`static double`	`P_Harmonic(int[] y, int[] ypred)` Harmonic Accuracy.
`static double`	`P_JaccardIndex(int[][] Y, int[][] Ypred)` Jaccard Index -- often simply called multi-label 'accuracy'.
`double`	`P_LogLikelihood(int[] y, double[] p)` Log Likelihood
`static double`	`P_macroAUPRC(int[][] Y, double[][] P)` Calculate AUPRC: Area Under the Precision-Recall curve.
`static double`	`P_macroAUROC(int[][] Y, double[][] P)` Calculate AUROC: Area Under the ROC curve.
`static double`	`P_Precision(int[][] Y, int[][] Ypred, int j)` P_Precision - (retrieved AND relevant) / retrieved
`static double`	`P_Precision(int[] y, int[] ypred)` P_Precision - (retrieved AND relevant) / retrieved
`static double`	`P_PrecisionMacro(int[][] Y, int[][] Ypred)` P_Precision - (retrieved AND relevant) / retrieved
`static double`	`P_PrecisionMicro(int[][] Y, int[][] Ypred)` P_Precision - (retrieved AND relevant) / retrieved
`static double`	`P_Recall(int[][] Y, int[][] Ypred, int j)` P_Recall - (retrieved AND relevant) / relevant
`static double`	`P_Recall(int[] y, int[] ypred)` P_Recall - (retrieved AND relevant) / relevant
`static double`	`P_RecallMacro(int[][] Y, int[][] Ypred)` P_Recall - (retrieved AND relevant) / relevant
`static double`	`P_RecallMicro(int[][] Y, int[][] Ypred)` P_Recall - (retrieved AND relevant) / relevant
`static double`	`P_TrueNegatives(int[] y, int[] ypred)` P_TrueNegatives - 0 and supposed to be 0.
`static double`	`P_TruePositives(int[] y, int[] ypred)` P_TruePositives - 1 and supposed to be 1 (the intersection, i.e., logical AND).
`static int[]`	`toIntArray(double[] doubles)` Helper function for missing values in the labels.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail
- Metrics
```
public Metrics()
```

Method Detail

toIntArray
```
public static int[] toIntArray(double[] doubles)
```
Helper function for missing values in the labels. Transforms a double array to an int array.

Returns:

the new array

allMissing
```
public static boolean allMissing(int[] real)
```
Helper function for missing values in the labels. Simply checks if all real labels are missing.

Returns:

If all labels are missing

numberOfMissingLabels
```
public static int numberOfMissingLabels(int[] real)
```
Helper function for missing values in the labels. Simply returns number of real missing labels.

Returns:

Number of missing labels.

align
```
public static int[][] align(int[] real,
                            int[] pred)
```
Helper function for missing values in the labels and missing predictions (i.e., from abstaining classifiers). Aligns the predictions with the real labels, discarding labels and predictions that are missing.

Parameters:

real - The real values from the data

pred - The predicted values from the classifiers

Returns:

Aligned predicted and real labels.

align
```
public static double[][] align(int[] real,
                               double[] pred)
```
Helper function for missing values in the labels and missing predictions (i.e., from abstaining classifiers). Aligns the predictions with the real labels, discarding labels and predictions that are missing.

Parameters:

real - The real values from the data

pred - The predicted values from the classifiers

Returns:

Aligned predicted and real labels.

P_ExactMatch

public static double P_ExactMatch(int[][] Y,
                                  int[][] Ypred)

Exact Match, i.e., 1 - [0/1 Loss].

L_ZeroOne

public static double L_ZeroOne(int[] y,
                               int[] ypred)

0/1 Loss.

L_ZeroOne

public static double L_ZeroOne(int[][] Y,
                               int[][] Ypred)

0/1 Loss.

L_Hamming

public static double L_Hamming(int[] y,
                               int[] ypred)

Hamming loss.

L_Hamming

public static double L_Hamming(int[][] Y,
                               int[][] Ypred)

Hamming loss.

P_Hamming

public static double P_Hamming(int[][] Y,
                               int[][] Ypred)

Hamming score aka label accuracy.

P_Hamming

public static double P_Hamming(int[][] Y,
                               int[][] Ypred,
                               int j)

Hamming score aka label accuracy.

P_Harmonic

public static double P_Harmonic(int[] y,
                                int[] ypred)

Harmonic Accuracy. Multi-label only.

P_Harmonic

public static double P_Harmonic(int[][] Y,
                                int[][] Ypred,
                                int j)

Harmonic Accuracy -- for the j-th label. Multi-label only.

P_Harmonic

public static double P_Harmonic(int[][] Y,
                                int[][] Ypred)

Harmonic Accuracy -- average over all labels. Multi-label only.

P_Accuracy

public static double P_Accuracy(int[] y,
                                int[] ypred)

Jaccard Index -- often simply called multi-label 'accuracy'. Multi-label only.

P_Accuracy

public static double P_Accuracy(int[][] Y,
                                int[][] Ypred)

Jaccard Index -- often simply called multi-label 'accuracy'. Multi-label only.

P_JaccardIndex

public static double P_JaccardIndex(int[][] Y,
                                    int[][] Ypred)

Jaccard Index -- often simply called multi-label 'accuracy'. Multi-label only.

L_JaccardDist

public static double L_JaccardDist(int[][] Y,
                                   int[][] Ypred)

Jaccard Distance -- the loss version of Jaccard Index

L_LogLoss
```
public static double L_LogLoss(double y,
                               double rpred,
                               double C)
```
L_LogLoss - the log loss between real-valued confidence rpred and true prediction y.

Parameters:

y - label

rpred - prediction (confidence)

C - limit (maximum loss of log(C))

Returns:

Log loss

L_LogLossL
```
public static double L_LogLossL(int[][] Y,
                                double[][] Rpred)
```
L_LogLoss - the log loss between real-valued confidences Rpred and true predictions Y with a maximum penalty based on the number of labels L [Important Note: Earlier versions of Meka only normalised by N, and not N*L as here].

L_LogLossD
```
public static double L_LogLossD(int[][] Y,
                                double[][] Rpred)
```
L_LogLoss - the log loss between real-valued confidences Rpred and true predictions Y with a maximum penalty based on the number of examples D [Important Note: Earlier versions of Meka only normalised by N, and not N*L as here].

L_LogLoss
```
public static double L_LogLoss(int[][] Y,
                               double[][] Rpred,
                               double C)
```
L_LogLoss - the log loss between real-valued confidences Rpred and true predictions Y with a maximum penalty C [Important Note: Earlier versions of Meka only normalised by N, and not N*L as here].

P_TruePositives

public static double P_TruePositives(int[] y,
                                     int[] ypred)

P_TruePositives - 1 and supposed to be 1 (the intersection, i.e., logical AND).

P_FalsePositives

public static double P_FalsePositives(int[] y,
                                      int[] ypred)

P_FalsePositives - 1 but supposed to be 0 (the length of y \ ypred).

P_TrueNegatives

public static double P_TrueNegatives(int[] y,
                                     int[] ypred)

P_TrueNegatives - 0 and supposed to be 0.

P_FalseNegatives

public static double P_FalseNegatives(int[] y,
                                      int[] ypred)

P_FalseNegatives - 0 but supposed to be 1 (the length of ypred \ y).

P_Precision

public static double P_Precision(int[] y,
                                 int[] ypred)

P_Precision - (retrieved AND relevant) / retrieved

P_Recall

public static double P_Recall(int[] y,
                              int[] ypred)

P_Recall - (retrieved AND relevant) / relevant

F1

public static double F1(int[] s1,
                        int[] s2)

F1 - the F1 measure for two sets.

P_PrecisionMacro

public static double P_PrecisionMacro(int[][] Y,
                                      int[][] Ypred)

P_Precision - (retrieved AND relevant) / retrieved

P_RecallMacro

public static double P_RecallMacro(int[][] Y,
                                   int[][] Ypred)

P_Recall - (retrieved AND relevant) / relevant

P_PrecisionMicro

public static double P_PrecisionMicro(int[][] Y,
                                      int[][] Ypred)

P_Precision - (retrieved AND relevant) / retrieved

P_RecallMicro

public static double P_RecallMicro(int[][] Y,
                                   int[][] Ypred)

P_Recall - (retrieved AND relevant) / relevant

P_Precision

public static double P_Precision(int[][] Y,
                                 int[][] Ypred,
                                 int j)

P_Precision - (retrieved AND relevant) / retrieved

P_Recall

public static double P_Recall(int[][] Y,
                              int[][] Ypred,
                              int j)

P_Recall - (retrieved AND relevant) / relevant

P_FmicroAvg
```
public static double P_FmicroAvg(int[][] Y,
                                 int[][] Ypred)
```
P_FmicroAvg - Micro Averaged F-measure (F1, as if all labels in the dataset formed a single vector)

P_FmacroAvgL

public static double P_FmacroAvgL(int[][] Y,
                                  int[][] Ypred)

F-Measure Macro Averaged by L - The 'standard' macro average.

P_FmacroAvgD
```
public static double P_FmacroAvgD(int[][] Y,
                                  int[][] Ypred)
```
F-Measure Averaged by D - The F-measure macro averaged by example. The Jaccard index is also averaged this way.

L_OneError

public static double L_OneError(int[][] Y,
                                double[][] Rpred)

OneError -

P_AveragePrecision

public static double P_AveragePrecision(int[][] Y,
                                        double[][] Rpred)

P_AveragePrecision
```
public static double P_AveragePrecision(int[] y,
                                        double[] rpred)
```
Converts confidences in prediction array to ranking array, and continues with P_AveragePrecision(using ranking array).

Parameters:

y - The real label values of an instance.

rpred - The predicted confidences for the labels.

Returns:

the calculated average precision for an instance.

P_AveragePrecision
```
public static double P_AveragePrecision(int[] y,
                                        int[] r)
```
Average Precision - computes for each relevant label the percentage of relevant labels among all labels that are ranked before it.

Parameters:

y - 0/1 labels [0, 0, 1 ] (true labels)

r - ranking position [1, 2, 0 ]

Returns:

Average Precision

L_RankLoss

public static double L_RankLoss(int[][] Y,
                                double[][] Rpred)

L_RankLoss

public static double L_RankLoss(int[] y,
                                double[] rpred)

L_RankLoss
```
public static double L_RankLoss(int[] y,
                                int[] r)
```
Rank Loss - the average fraction of labels which are not correctly ordered. Thanks to Noureddine Yacine NAIR BENREKIA for providing bug fix for this.

Parameters:

y - 0/1 labels [0, 0, 1 ]

r - ranking position [1, 2, 0 ]

Returns:

Ranking Loss

P_macroAUPRC

public static double P_macroAUPRC(int[][] Y,
                                  double[][] P)

Calculate AUPRC: Area Under the Precision-Recall curve.

P_macroAUROC

public static double P_macroAUROC(int[][] Y,
                                  double[][] P)

Calculate AUROC: Area Under the ROC curve.

curveDataMicroAveraged

public static weka.core.Instances curveDataMicroAveraged(int[][] Y,
                                                         double[][] P)

Get Data for Plotting PR and ROC curves.

curveDataMacroAveraged

public static weka.core.Instances curveDataMacroAveraged(int[][] Y,
                                                         double[][] P)

Get Data for Plotting PR and ROC curves.

curveData

public static weka.core.Instances curveData(int[] y,
                                            double[] p)

Get Data for Plotting PR and ROC curves.

curveData

public static weka.core.Instances[] curveData(int[][] Y,
                                              double[][] P)

Get Data for Plotting PR and ROC curves.

L_LevenshteinDistance

public static double L_LevenshteinDistance(int[][] Y,
                                           int[][] P)

Levenshtein Distance. Multi-target compatible

L_LevenshteinDistance

public static double L_LevenshteinDistance(int[] y,
                                           int[] p)

Levenshtein Distance divided by the number of labels. Multi-target compatible

P_LogLikelihood

public double P_LogLikelihood(int[] y,
                              double[] p)

Log Likelihood

L_MSE

public double L_MSE(int[] y,
                    double[] p)

MSE

L_MAE

public double L_MAE(int[] y,
                    double[] p)

MAE

main

public static void main(String[] args)

Do some tests.

Class Metrics

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

Metrics

Method Detail

toIntArray

allMissing

numberOfMissingLabels

align

align

P_ExactMatch

L_ZeroOne

L_ZeroOne

L_Hamming

L_Hamming

P_Hamming

P_Hamming

P_Harmonic

P_Harmonic

P_Harmonic

P_Accuracy

P_Accuracy

P_JaccardIndex

L_JaccardDist

L_LogLoss

L_LogLossL

L_LogLossD

L_LogLoss

P_TruePositives

P_FalsePositives

P_TrueNegatives

P_FalseNegatives

P_Precision

P_Recall

F1

P_PrecisionMacro

P_RecallMacro

P_PrecisionMicro

P_RecallMicro

P_Precision

P_Recall

P_FmicroAvg

P_FmacroAvgL

P_FmacroAvgD

L_OneError

P_AveragePrecision

P_AveragePrecision

P_AveragePrecision

L_RankLoss

L_RankLoss

L_RankLoss

P_macroAUPRC

P_macroAUROC

curveDataMicroAveraged

curveDataMacroAveraged

curveData

curveData

L_LevenshteinDistance

L_LevenshteinDistance

P_LogLikelihood

L_MSE

L_MAE

main