Synopsis
This operator delivers as output a list of performance values according to a list of selected performance criteria (for binominal classification tasks).
Description
This performance evaluator operator should be used for classification tasks, i.e. in cases where the label attribute has a binominal value type. Other polynominal classification tasks, i.e. tasks with more than two classes can be handled by the PolynominalClassificationPerformanceEvaluator operator. This operator expects a test ExampleSet as input, whose elements have both true and predicted labels, and delivers as output a list of performance values according to a list of performance criteria that it calculates. If an input performance vector was already given, this is used for keeping the performance values.
All of the performance criteria can be switched on using boolean parameters. Their values can be queried by a ProcessLogOperator using the same names. The main criterion is used for comparisons and need to be specified only for processes where performance vectors are compared, e.g. feature selection or other meta optimization process setups. If no other main criterion was selected, the first criterion in the resulting performance vector will be assumed to be the main criterion.
The resulting performance vectors are usually compared with a standard performance comparator which only compares the fitness values of the main criterion. Other implementations than this simple comparator can be specified using the parameter comparator_class. This may for instance be useful if you want to compare performance vectors according to the weighted sum of the individual criteria. In order to implement your own comparator, simply subclass PerformanceComparator. Please note that for true multi-objective optimization usually another selection scheme is used instead of simply replacing the performance comparator.
Input
- labelled data: expects: ExampleSet, expects: ExampleSetMetaData: #examples: = 0; #attributes: 0
- performance: optional: PerformanceVector
Output
- performance:
- example set:
Parameters
- main criterion: The criterion used for comparing performance vectors.
- accuracy: Relative number of correctly classified examples
- classification error: Relative number of misclassified examples
- kappa: The kappa statistics for the classification
- AUC (optimistic): The area under a ROC curve. Given example weights are also considered. Please note that the second class is considered to be positive.
- AUC: The area under a ROC curve. Given example weights are also considered. Please note that the second class is considered to be positive.
- AUC (pessimistic): The area under a ROC curve. Given example weights are also considered. Please note that the second class is considered to be positive.
- precision: Relative number of correctly as positive classified examples among all examples classified as positive
- recall: Relative number of correctly as positive classified examples among all positive examples
- lift: The lift of the positive class
- fallout: Relative number of incorrectly as positive classified examples among all negative examples
- f measure: Combination of precision and recall: f=2pr/(p+r)
- false positive: Absolute number of incorrectly as positive classified examples
- false negative: Absolute number of incorrectly as negative classified examples
- true positive: Absolute number of correctly as positive classified examples
- true negative: Absolute number of correctly as negative classified examples
- sensitivity: Relative number of correctly as positive classified examples among all positive examples (like recall)
- specificity: Relative number of correctly as negative classified examples among all negative examples
- youden: The sum of sensitivity and specificity minus 1
- positive predictive value: Relative number of correctly as positive classified examples among all examples classified as positive (same as precision)
- negative predictive value: Relative number of correctly as negative classified examples among all examples classified as negative
- psep: The sum of the positive predicitve value and the negative predictive value minus 1
- skip undefined labels: If set to true, examples with undefined labels are skipped.
- comparator class: Fully qualified classname of the PerformanceComparator implementation.
- use example weights: Indicated if example weights should be used for performance calculations if possible.