Example-5 (Sample weights)

Without weights

In [1]:
from pycm import ConfusionMatrix
y_test = [2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2]
y_pred = [0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2]
In [2]:
cm1=ConfusionMatrix(y_test, y_pred)
cm1
Out[2]:
pycm.ConfusionMatrix(classes: [0, 1, 2])
In [3]:
print(cm1)
Predict 0       1       2       
Actual
0       3       0       0       

1       0       1       2       

2       2       1       3       





Overall Statistics : 

95% CI                                                            (0.30439,0.86228)
ACC Macro                                                         0.72222
AUNP                                                              0.66667
AUNU                                                              0.69444
Bennett S                                                         0.375
CBA                                                               0.47778
Chi-Squared                                                       6.6
Chi-Squared DF                                                    4
Conditional Entropy                                               0.95915
Cramer V                                                          0.5244
Cross Entropy                                                     1.59352
F1 Macro                                                          0.56515
F1 Micro                                                          0.58333
Gwet AC1                                                          0.38931
Hamming Loss                                                      0.41667
Joint Entropy                                                     2.45915
KL Divergence                                                     0.09352
Kappa                                                             0.35484
Kappa 95% CI                                                      (-0.07708,0.78675)
Kappa No Prevalence                                               0.16667
Kappa Standard Error                                              0.22036
Kappa Unbiased                                                    0.34426
Lambda A                                                          0.16667
Lambda B                                                          0.42857
Mutual Information                                                0.52421
NIR                                                               0.5
Overall ACC                                                       0.58333
Overall CEN                                                       0.46381
Overall J                                                         (1.225,0.40833)
Overall MCC                                                       0.36667
Overall MCEN                                                      0.51894
Overall RACC                                                      0.35417
Overall RACCU                                                     0.36458
P-Value                                                           0.38721
PPV Macro                                                         0.56667
PPV Micro                                                         0.58333
Pearson C                                                         0.59568
Phi-Squared                                                       0.55
RCI                                                               0.34947
RR                                                                4.0
Reference Entropy                                                 1.5
Response Entropy                                                  1.48336
SOA1(Landis & Koch)                                               Fair
SOA2(Fleiss)                                                      Poor
SOA3(Altman)                                                      Fair
SOA4(Cicchetti)                                                   Poor
SOA5(Cramer)                                                      Relatively Strong
SOA6(Matthews)                                                    Weak
Scott PI                                                          0.34426
Standard Error                                                    0.14232
TPR Macro                                                         0.61111
TPR Micro                                                         0.58333
Zero-one Loss                                                     5

Class Statistics :

Classes                                                           0             1             2             
ACC(Accuracy)                                                     0.83333       0.75          0.58333       
AGF(Adjusted F-score)                                             0.9136        0.53995       0.5516        
AGM(Adjusted geometric mean)                                      0.83729       0.692         0.60712       
AM(Difference between automatic and manual classification)        2             -1            -1            
AUC(Area under the ROC curve)                                     0.88889       0.61111       0.58333       
AUCI(AUC value interpretation)                                    Very Good     Fair          Poor          
AUPR(Area under the PR curve)                                     0.8           0.41667       0.55          
BCD(Bray-Curtis dissimilarity)                                    0.08333       0.04167       0.04167       
BM(Informedness or bookmaker informedness)                        0.77778       0.22222       0.16667       
CEN(Confusion entropy)                                            0.25          0.49658       0.60442       
DOR(Diagnostic odds ratio)                                        None          4.0           2.0           
DP(Discriminant power)                                            None          0.33193       0.16597       
DPI(Discriminant power interpretation)                            None          Poor          Poor          
ERR(Error rate)                                                   0.16667       0.25          0.41667       
F0.5(F0.5 score)                                                  0.65217       0.45455       0.57692       
F1(F1 score - harmonic mean of precision and sensitivity)         0.75          0.4           0.54545       
F2(F2 score)                                                      0.88235       0.35714       0.51724       
FDR(False discovery rate)                                         0.4           0.5           0.4           
FN(False negative/miss/type 2 error)                              0             2             3             
FNR(Miss rate or false negative rate)                             0.0           0.66667       0.5           
FOR(False omission rate)                                          0.0           0.2           0.42857       
FP(False positive/type 1 error/false alarm)                       2             1             2             
FPR(Fall-out or false positive rate)                              0.22222       0.11111       0.33333       
G(G-measure geometric mean of precision and sensitivity)          0.7746        0.40825       0.54772       
GI(Gini index)                                                    0.77778       0.22222       0.16667       
GM(G-mean geometric mean of specificity and sensitivity)          0.88192       0.54433       0.57735       
IBA(Index of balanced accuracy)                                   0.95062       0.13169       0.27778       
IS(Information score)                                             1.26303       1.0           0.26303       
J(Jaccard index)                                                  0.6           0.25          0.375         
LS(Lift score)                                                    2.4           2.0           1.2           
MCC(Matthews correlation coefficient)                             0.68313       0.2582        0.16903       
MCCI(Matthews correlation coefficient interpretation)             Moderate      Negligible    Negligible    
MCEN(Modified confusion entropy)                                  0.26439       0.5           0.6875        
MK(Markedness)                                                    0.6           0.3           0.17143       
N(Condition negative)                                             9             9             6             
NLR(Negative likelihood ratio)                                    0.0           0.75          0.75          
NLRI(Negative likelihood ratio interpretation)                    Good          Negligible    Negligible    
NPV(Negative predictive value)                                    1.0           0.8           0.57143       
OC(Overlap coefficient)                                           1.0           0.5           0.6           
OOC(Otsuka-Ochiai coefficient)                                    0.7746        0.40825       0.54772       
OP(Optimized precision)                                           0.70833       0.29545       0.44048       
P(Condition positive or support)                                  3             3             6             
PLR(Positive likelihood ratio)                                    4.5           3.0           1.5           
PLRI(Positive likelihood ratio interpretation)                    Poor          Poor          Poor          
POP(Population)                                                   12            12            12            
PPV(Precision or positive predictive value)                       0.6           0.5           0.6           
PRE(Prevalence)                                                   0.25          0.25          0.5           
Q(Yule Q - coefficient of colligation)                            None          0.6           0.33333       
RACC(Random accuracy)                                             0.10417       0.04167       0.20833       
RACCU(Random accuracy unbiased)                                   0.11111       0.0434        0.21007       
TN(True negative/correct rejection)                               7             8             4             
TNR(Specificity or true negative rate)                            0.77778       0.88889       0.66667       
TON(Test outcome negative)                                        7             10            7             
TOP(Test outcome positive)                                        5             2             5             
TP(True positive/hit)                                             3             1             3             
TPR(Sensitivity, recall, hit rate, or true positive rate)         1.0           0.33333       0.5           
Y(Youden index)                                                   0.77778       0.22222       0.16667       
dInd(Distance index)                                              0.22222       0.67586       0.60093       
sInd(Similarity index)                                            0.84287       0.52209       0.57508       

With random weights

In [4]:
from random import randint,seed
seed(100)
weights = [randint(1,10) for i in range(len(y_test))]
weights[2]*=9
In [5]:
cm2=ConfusionMatrix(y_test, y_pred, sample_weight = weights)
cm2
Out[5]:
pycm.ConfusionMatrix(classes: [0, 1, 2])
In [6]:
print(cm2)
Predict  0        1        2        
Actual
0        24       0        0        

1        0        7        8        

2        12       3        76       





Overall Statistics : 

95% CI                                                            (0.75748,0.88868)
ACC Macro                                                         0.88205
AUNP                                                              0.82779
AUNU                                                              0.82623
Bennett S                                                         0.73462
CBA                                                               0.65617
Chi-Squared                                                       110.9678
Chi-Squared DF                                                    4
Conditional Entropy                                               0.65034
Cramer V                                                          0.6533
Cross Entropy                                                     1.21
F1 Macro                                                          0.74286
F1 Micro                                                          0.82308
Gwet AC1                                                          0.76652
Hamming Loss                                                      0.17692
Joint Entropy                                                     1.82
KL Divergence                                                     0.04034
Kappa                                                             0.63722
Kappa 95% CI                                                      (0.50272,0.77173)
Kappa No Prevalence                                               0.64615
Kappa Standard Error                                              0.06863
Kappa Unbiased                                                    0.63481
Lambda A                                                          0.41026
Lambda B                                                          0.52174
Mutual Information                                                0.5544
NIR                                                               0.7
Overall ACC                                                       0.82308
Overall CEN                                                       0.28807
Overall J                                                         (1.82323,0.60774)
Overall MCC                                                       0.64625
Overall MCEN                                                      0.38181
Overall RACC                                                      0.51231
Overall RACCU                                                     0.51553
P-Value                                                           0.00097
PPV Macro                                                         0.75714
PPV Micro                                                         0.82308
Pearson C                                                         0.67861
Phi-Squared                                                       0.8536
RCI                                                               0.47399
RR                                                                43.33333
Reference Entropy                                                 1.16966
Response Entropy                                                  1.20474
SOA1(Landis & Koch)                                               Substantial
SOA2(Fleiss)                                                      Intermediate to Good
SOA3(Altman)                                                      Good
SOA4(Cicchetti)                                                   Good
SOA5(Cramer)                                                      Strong
SOA6(Matthews)                                                    Moderate
Scott PI                                                          0.63481
Standard Error                                                    0.03347
TPR Macro                                                         0.76728
TPR Micro                                                         0.82308
Zero-one Loss                                                     23

Class Statistics :

Classes                                                           0             1             2             
ACC(Accuracy)                                                     0.90769       0.91538       0.82308       
AGF(Adjusted F-score)                                             0.94152       0.68599       0.76783       
AGM(Adjusted geometric mean)                                      0.91704       0.81486       0.81018       
AM(Difference between automatic and manual classification)        12            -5            -7            
AUC(Area under the ROC curve)                                     0.9434        0.72029       0.81502       
AUCI(AUC value interpretation)                                    Excellent     Good          Very Good     
AUPR(Area under the PR curve)                                     0.83333       0.58333       0.86996       
BCD(Bray-Curtis dissimilarity)                                    0.04615       0.01923       0.02692       
BM(Informedness or bookmaker informedness)                        0.88679       0.44058       0.63004       
CEN(Confusion entropy)                                            0.23219       0.44655       0.28458       
DOR(Diagnostic odds ratio)                                        None          32.66667      19.63333      
DP(Discriminant power)                                            None          0.83477       0.71286       
DPI(Discriminant power interpretation)                            None          Poor          Poor          
ERR(Error rate)                                                   0.09231       0.08462       0.17692       
F0.5(F0.5 score)                                                  0.71429       0.63636       0.88993       
F1(F1 score - harmonic mean of precision and sensitivity)         0.8           0.56          0.86857       
F2(F2 score)                                                      0.90909       0.5           0.84821       
FDR(False discovery rate)                                         0.33333       0.3           0.09524       
FN(False negative/miss/type 2 error)                              0             8             15            
FNR(Miss rate or false negative rate)                             0.0           0.53333       0.16484       
FOR(False omission rate)                                          0.0           0.06667       0.32609       
FP(False positive/type 1 error/false alarm)                       12            3             8             
FPR(Fall-out or false positive rate)                              0.11321       0.02609       0.20513       
G(G-measure geometric mean of precision and sensitivity)          0.8165        0.57155       0.86927       
GI(Gini index)                                                    0.88679       0.44058       0.63004       
GM(G-mean geometric mean of specificity and sensitivity)          0.9417        0.67416       0.81477       
IBA(Index of balanced accuracy)                                   0.98718       0.22395       0.6906        
IS(Information score)                                             1.85244       2.6009        0.37018       
J(Jaccard index)                                                  0.66667       0.38889       0.76768       
LS(Lift score)                                                    3.61111       6.06667       1.29252       
MCC(Matthews correlation coefficient)                             0.76889       0.52824       0.60381       
MCCI(Matthews correlation coefficient interpretation)             Strong        Moderate      Moderate      
MCEN(Modified confusion entropy)                                  0.26416       0.4754        0.40758       
MK(Markedness)                                                    0.66667       0.63333       0.57867       
N(Condition negative)                                             106           115           39            
NLR(Negative likelihood ratio)                                    0.0           0.54762       0.20737       
NLRI(Negative likelihood ratio interpretation)                    Good          Negligible    Poor          
NPV(Negative predictive value)                                    1.0           0.93333       0.67391       
OC(Overlap coefficient)                                           1.0           0.7           0.90476       
OOC(Otsuka-Ochiai coefficient)                                    0.8165        0.57155       0.86927       
OP(Optimized precision)                                           0.84769       0.56327       0.79836       
P(Condition positive or support)                                  24            15            91            
PLR(Positive likelihood ratio)                                    8.83333       17.88889      4.07143       
PLRI(Positive likelihood ratio interpretation)                    Fair          Good          Poor          
POP(Population)                                                   130           130           130           
PPV(Precision or positive predictive value)                       0.66667       0.7           0.90476       
PRE(Prevalence)                                                   0.18462       0.11538       0.7           
Q(Yule Q - coefficient of colligation)                            None          0.94059       0.90307       
RACC(Random accuracy)                                             0.05112       0.00888       0.45231       
RACCU(Random accuracy unbiased)                                   0.05325       0.00925       0.45303       
TN(True negative/correct rejection)                               94            112           31            
TNR(Specificity or true negative rate)                            0.88679       0.97391       0.79487       
TON(Test outcome negative)                                        94            120           46            
TOP(Test outcome positive)                                        36            10            84            
TP(True positive/hit)                                             24            7             76            
TPR(Sensitivity, recall, hit rate, or true positive rate)         1.0           0.46667       0.83516       
Y(Youden index)                                                   0.88679       0.44058       0.63004       
dInd(Distance index)                                              0.11321       0.53397       0.26315       
sInd(Similarity index)                                            0.91995       0.62243       0.81392