Please cite us if you use the software

Example-3 (Activation threshold)

Binary classification

In [1]:
from pycm import ConfusionMatrix

def activation1(i):
    if i<0.75:
        return 1
    else:
        return 0
In [2]:
y_actual = [1,1,0,1,1,0]
y_pred = [0.65,0.34,0.80,0.54,0.32,0.12]
In [3]:
cm = ConfusionMatrix(y_actual,y_pred,threshold=activation1)
cm.classes
Out[3]:
[0, 1]
In [4]:
print(cm)
Predict 0       1       
Actual
0       1       1       

1       0       4       





Overall Statistics : 

95% CI                                                            (0.53513,1.13154)
ACC Macro                                                         0.83333
ARI                                                               0.34783
AUNP                                                              0.75
AUNU                                                              0.75
Bangdiwala B                                                      0.77273
Bennett S                                                         0.66667
CBA                                                               0.65
CSI                                                               0.65
Chi-Squared                                                       2.4
Chi-Squared DF                                                    1
Conditional Entropy                                               0.33333
Cramer V                                                          0.63246
Cross Entropy                                                     1.03701
F1 Macro                                                          0.77778
F1 Micro                                                          0.83333
FNR Macro                                                         0.25
FNR Micro                                                         0.16667
FPR Macro                                                         0.25
FPR Micro                                                         0.16667
Gwet AC1                                                          0.73333
Hamming Loss                                                      0.16667
Joint Entropy                                                     1.25163
KL Divergence                                                     0.11871
Kappa                                                             0.57143
Kappa 95% CI                                                      (-0.19538,1.33824)
Kappa No Prevalence                                               0.66667
Kappa Standard Error                                              0.39123
Kappa Unbiased                                                    0.55556
Krippendorff Alpha                                                0.59259
Lambda A                                                          0.5
Lambda B                                                          0.0
Mutual Information                                                0.31669
NIR                                                               0.66667
Overall ACC                                                       0.83333
Overall CEN                                                       0.39624
Overall J                                                         (1.3,0.65)
Overall MCC                                                       0.63246
Overall MCEN                                                      0.27683
Overall RACC                                                      0.61111
Overall RACCU                                                     0.625
P-Value                                                           0.35117
PPV Macro                                                         0.9
PPV Micro                                                         0.83333
Pearson C                                                         0.53452
Phi-Squared                                                       0.4
RCI                                                               0.34487
RR                                                                3.0
Reference Entropy                                                 0.9183
Response Entropy                                                  0.65002
SOA1(Landis & Koch)                                               Moderate
SOA2(Fleiss)                                                      Intermediate to Good
SOA3(Altman)                                                      Moderate
SOA4(Cicchetti)                                                   Fair
SOA5(Cramer)                                                      Strong
SOA6(Matthews)                                                    Moderate
Scott PI                                                          0.55556
Standard Error                                                    0.15215
TNR Macro                                                         0.75
TNR Micro                                                         0.83333
TPR Macro                                                         0.75
TPR Micro                                                         0.83333
Zero-one Loss                                                     1

Class Statistics :

Classes                                                           0             1             
ACC(Accuracy)                                                     0.83333       0.83333       
AGF(Adjusted F-score)                                             0.68041       0.89087       
AGM(Adjusted geometric mean)                                      0.82426       0.65533       
AM(Difference between automatic and manual classification)        -1            1             
AUC(Area under the ROC curve)                                     0.75          0.75          
AUCI(AUC value interpretation)                                    Good          Good          
AUPR(Area under the PR curve)                                     0.75          0.9           
BCD(Bray-Curtis dissimilarity)                                    0.08333       0.08333       
BM(Informedness or bookmaker informedness)                        0.5           0.5           
CEN(Confusion entropy)                                            0.52832       0.35221       
DOR(Diagnostic odds ratio)                                        None          None          
DP(Discriminant power)                                            None          None          
DPI(Discriminant power interpretation)                            None          None          
ERR(Error rate)                                                   0.16667       0.16667       
F0.5(F0.5 score)                                                  0.83333       0.83333       
F1(F1 score - harmonic mean of precision and sensitivity)         0.66667       0.88889       
F2(F2 score)                                                      0.55556       0.95238       
FDR(False discovery rate)                                         0.0           0.2           
FN(False negative/miss/type 2 error)                              1             0             
FNR(Miss rate or false negative rate)                             0.5           0.0           
FOR(False omission rate)                                          0.2           0.0           
FP(False positive/type 1 error/false alarm)                       0             1             
FPR(Fall-out or false positive rate)                              0.0           0.5           
G(G-measure geometric mean of precision and sensitivity)          0.70711       0.89443       
GI(Gini index)                                                    0.5           0.5           
GM(G-mean geometric mean of specificity and sensitivity)          0.70711       0.70711       
IBA(Index of balanced accuracy)                                   0.25          0.75          
ICSI(Individual classification success index)                     0.5           0.8           
IS(Information score)                                             1.58496       0.26303       
J(Jaccard index)                                                  0.5           0.8           
LS(Lift score)                                                    3.0           1.2           
MCC(Matthews correlation coefficient)                             0.63246       0.63246       
MCCI(Matthews correlation coefficient interpretation)             Moderate      Moderate      
MCEN(Modified confusion entropy)                                  0.5           0.46439       
MK(Markedness)                                                    0.8           0.8           
N(Condition negative)                                             4             2             
NLR(Negative likelihood ratio)                                    0.5           0.0           
NLRI(Negative likelihood ratio interpretation)                    Negligible    Good          
NPV(Negative predictive value)                                    0.8           1.0           
OC(Overlap coefficient)                                           1.0           1.0           
OOC(Otsuka-Ochiai coefficient)                                    0.70711       0.89443       
OP(Optimized precision)                                           0.5           0.5           
P(Condition positive or support)                                  2             4             
PLR(Positive likelihood ratio)                                    None          2.0           
PLRI(Positive likelihood ratio interpretation)                    None          Poor          
POP(Population)                                                   6             6             
PPV(Precision or positive predictive value)                       1.0           0.8           
PRE(Prevalence)                                                   0.33333       0.66667       
Q(Yule Q - coefficient of colligation)                            None          None          
QI(Yule Q interpretation)                                         None          None          
RACC(Random accuracy)                                             0.05556       0.55556       
RACCU(Random accuracy unbiased)                                   0.0625        0.5625        
TN(True negative/correct rejection)                               4             1             
TNR(Specificity or true negative rate)                            1.0           0.5           
TON(Test outcome negative)                                        5             1             
TOP(Test outcome positive)                                        1             5             
TP(True positive/hit)                                             1             4             
TPR(Sensitivity, recall, hit rate, or true positive rate)         0.5           1.0           
Y(Youden index)                                                   0.5           0.5           
dInd(Distance index)                                              0.5           0.5           
sInd(Similarity index)                                            0.64645       0.64645       

Multiclass classification

In [5]:
def activation2(i):
    ref = ["cat","dog","ship"]
    return ref[i.index(max(i))]
In [6]:
y_actual = ["ship","cat","dog","cat","dog","ship"]
y_pred = [[90,433,600],[345,100,2],[100,90,10],[432,102,80],[156,402,2],[73,20,532]]
In [7]:
cm2 = ConfusionMatrix(y_actual,y_pred,threshold=activation2)
cm2.classes
Out[7]:
['cat', 'dog', 'ship']
In [8]:
print(cm2)
Predict    cat        dog        ship       
Actual
cat        2          0          0          

dog        1          1          0          

ship       0          0          2          





Overall Statistics : 

95% CI                                                            (0.53513,1.13154)
ACC Macro                                                         0.88889
ARI                                                               0.44444
AUNP                                                              0.875
AUNU                                                              0.875
Bangdiwala B                                                      0.75
Bennett S                                                         0.75
CBA                                                               0.72222
CSI                                                               0.72222
Chi-Squared                                                       8.0
Chi-Squared DF                                                    4
Conditional Entropy                                               0.33333
Cramer V                                                          0.8165
Cross Entropy                                                     1.72331
F1 Macro                                                          0.82222
F1 Micro                                                          0.83333
FNR Macro                                                         0.16667
FNR Micro                                                         0.16667
FPR Macro                                                         0.08333
FPR Micro                                                         0.08333
Gwet AC1                                                          0.75258
Hamming Loss                                                      0.16667
Joint Entropy                                                     1.9183
KL Divergence                                                     0.13835
Kappa                                                             0.75
Kappa 95% CI                                                      (0.30269,1.19731)
Kappa No Prevalence                                               0.66667
Kappa Standard Error                                              0.22822
Kappa Unbiased                                                    0.74468
Krippendorff Alpha                                                0.76596
Lambda A                                                          0.75
Lambda B                                                          0.66667
Mutual Information                                                1.12581
NIR                                                               0.33333
Overall ACC                                                       0.83333
Overall CEN                                                       0.16279
Overall J                                                         (2.16667,0.72222)
Overall MCC                                                       0.78335
Overall MCEN                                                      0.18464
Overall RACC                                                      0.33333
Overall RACCU                                                     0.34722
P-Value                                                           0.01783
PPV Macro                                                         0.88889
PPV Micro                                                         0.83333
Pearson C                                                         0.75593
Phi-Squared                                                       1.33333
RCI                                                               0.71031
RR                                                                2.0
Reference Entropy                                                 1.58496
Response Entropy                                                  1.45915
SOA1(Landis & Koch)                                               Substantial
SOA2(Fleiss)                                                      Intermediate to Good
SOA3(Altman)                                                      Good
SOA4(Cicchetti)                                                   Excellent
SOA5(Cramer)                                                      Very Strong
SOA6(Matthews)                                                    Strong
Scott PI                                                          0.74468
Standard Error                                                    0.15215
TNR Macro                                                         0.91667
TNR Micro                                                         0.91667
TPR Macro                                                         0.83333
TPR Micro                                                         0.83333
Zero-one Loss                                                     1

Class Statistics :

Classes                                                           cat           dog           ship          
ACC(Accuracy)                                                     0.83333       0.83333       1.0           
AGF(Adjusted F-score)                                             0.92319       0.68041       1.0           
AGM(Adjusted geometric mean)                                      0.81962       0.82426       1.0           
AM(Difference between automatic and manual classification)        1             -1            0             
AUC(Area under the ROC curve)                                     0.875         0.75          1.0           
AUCI(AUC value interpretation)                                    Very Good     Good          Excellent     
AUPR(Area under the PR curve)                                     0.83333       0.75          1.0           
BCD(Bray-Curtis dissimilarity)                                    0.08333       0.08333       0.0           
BM(Informedness or bookmaker informedness)                        0.75          0.5           1.0           
CEN(Confusion entropy)                                            0.23219       0.26416       0             
DOR(Diagnostic odds ratio)                                        None          None          None          
DP(Discriminant power)                                            None          None          None          
DPI(Discriminant power interpretation)                            None          None          None          
ERR(Error rate)                                                   0.16667       0.16667       0.0           
F0.5(F0.5 score)                                                  0.71429       0.83333       1.0           
F1(F1 score - harmonic mean of precision and sensitivity)         0.8           0.66667       1.0           
F2(F2 score)                                                      0.90909       0.55556       1.0           
FDR(False discovery rate)                                         0.33333       0.0           0.0           
FN(False negative/miss/type 2 error)                              0             1             0             
FNR(Miss rate or false negative rate)                             0.0           0.5           0.0           
FOR(False omission rate)                                          0.0           0.2           0.0           
FP(False positive/type 1 error/false alarm)                       1             0             0             
FPR(Fall-out or false positive rate)                              0.25          0.0           0.0           
G(G-measure geometric mean of precision and sensitivity)          0.8165        0.70711       1.0           
GI(Gini index)                                                    0.75          0.5           1.0           
GM(G-mean geometric mean of specificity and sensitivity)          0.86603       0.70711       1.0           
IBA(Index of balanced accuracy)                                   0.9375        0.25          1.0           
ICSI(Individual classification success index)                     0.66667       0.5           1.0           
IS(Information score)                                             1.0           1.58496       1.58496       
J(Jaccard index)                                                  0.66667       0.5           1.0           
LS(Lift score)                                                    2.0           3.0           3.0           
MCC(Matthews correlation coefficient)                             0.70711       0.63246       1.0           
MCCI(Matthews correlation coefficient interpretation)             Strong        Moderate      Very Strong   
MCEN(Modified confusion entropy)                                  0.26416       0.25          0             
MK(Markedness)                                                    0.66667       0.8           1.0           
N(Condition negative)                                             4             4             4             
NLR(Negative likelihood ratio)                                    0.0           0.5           0.0           
NLRI(Negative likelihood ratio interpretation)                    Good          Negligible    Good          
NPV(Negative predictive value)                                    1.0           0.8           1.0           
OC(Overlap coefficient)                                           1.0           1.0           1.0           
OOC(Otsuka-Ochiai coefficient)                                    0.8165        0.70711       1.0           
OP(Optimized precision)                                           0.69048       0.5           1.0           
P(Condition positive or support)                                  2             2             2             
PLR(Positive likelihood ratio)                                    4.0           None          None          
PLRI(Positive likelihood ratio interpretation)                    Poor          None          None          
POP(Population)                                                   6             6             6             
PPV(Precision or positive predictive value)                       0.66667       1.0           1.0           
PRE(Prevalence)                                                   0.33333       0.33333       0.33333       
Q(Yule Q - coefficient of colligation)                            None          None          None          
QI(Yule Q interpretation)                                         None          None          None          
RACC(Random accuracy)                                             0.16667       0.05556       0.11111       
RACCU(Random accuracy unbiased)                                   0.17361       0.0625        0.11111       
TN(True negative/correct rejection)                               3             4             4             
TNR(Specificity or true negative rate)                            0.75          1.0           1.0           
TON(Test outcome negative)                                        3             5             4             
TOP(Test outcome positive)                                        3             1             2             
TP(True positive/hit)                                             2             1             2             
TPR(Sensitivity, recall, hit rate, or true positive rate)         1.0           0.5           1.0           
Y(Youden index)                                                   0.75          0.5           1.0           
dInd(Distance index)                                              0.25          0.5           0.0           
sInd(Similarity index)                                            0.82322       0.64645       1.0