Synopsis
This operator delivers as a
Description
This performance evaluator operator should be used for classification tasks, i.e. in cases where the label attribute has a (poly-)nominal value type. <p> This operator expects a test ExampleSet as input, containing one attribute with the role label and one with the role prediction. See the Set Role operator for more details. On the basis of this two attributes a PerformanceVector is calculated, containing the values of the performance criteria. If a PerformanceVector was fed into performance input, it's values are kept if it does not already contain the new criteria. Otherwise the values are averaged over the old and the new values.
All of the performance criteria can be switched on using boolean parameters. Their values can be queried by a Log operator using the same names. The main criterion is used for comparisons and need to be specified only for processes where performance vectors are compared, e.g. attribute selection or other meta optimization process setups. If no main criterion was selected, the first criterion in the resulting performance vector will be assumed to be the main criterion.
Input
- labelled data: expects: ExampleSet, expects: ExampleSetMetaData: #examples: = 0; #attributes: 0
- performance: optional: PerformanceVector
Output
- performance:
- example set:
Parameters
- main criterion: The criterion used for comparing performance vectors.
- accuracy: Relative number of correctly classified examples
- classification error: Relative number of misclassified examples
- kappa: The kappa statistics for the classification
- weighted mean recall: The weighted mean of all per class recall measurements.
- weighted mean precision: The weighted mean of all per class precision measurements.
- spearman rho: The rank correlation between the actual and predicted labels, using Spearman's rho.
- kendall tau: The rank correlation between the actual and predicted labels, using Kendall's tau-b.
- absolute error: Average absolute deviation of the prediction from the actual value
- relative error: Average relative error (average of absolute deviation of the prediction from the actual value divided by actual value)
- relative error lenient: Average lenient relative error (average of absolute deviation of the prediction from the actual value divided by maximum of the actual value and the prediction)
- relative error strict: Average strict relative error (average of absolute deviation of the prediction from the actual value divided by minimum of the actual value and the prediction)
- normalized absolute error: The absolute error divided by the error made if the average would have been predicted.
- root mean squared error: Averaged root-mean-squared error
- root relative squared error: Averaged root-relative-squared error
- squared error: Averaged squared error
- correlation: Returns the correlation coefficient between the label and predicted label.
- squared correlation: Returns the squared correlation coefficient between the label and predicted label.
- cross-entropy: The cross-entropy of a classifier, defined as the sum over the logarithms of the true label's confidences divided by the number of examples
- margin: The margin of a classifier, defined as the minimal confidence for the correct label.
- soft margin loss: The average soft margin loss of a classifier, defined as the average of all 1 - confidences for the correct label.
- logistic loss: The logistic loss of a classifier, defined as the average of ln(1 + exp(- [confidence of the correct class]))
- skip undefined labels: If set to true, examples with undefined labels are skipped.
- comparator class: Fully qualified classname of the PerformanceComparator implementation.
- use example weights: Indicated if example weights should be used for performance calculations if possible.
- class weights: The weights for all classes (first column: class name, second column: weight), empty: using 1 for all classes.