Precision, recall, F1 and ROC

Accuracy lies with imbalanced classes: 1 % ill → «everyone is healthy» gives 99 %.

The confusion matrix: TP (true positive), FP (false alarm), FN (missed), TN.

• Precision = TP / (TP + FP): of the ones we raised the alarm about, how many were right?
• Recall = TP / (TP + FN): of all the ill people, how many did we find?
• F1 = the harmonic mean of the two.

Cancer screening: miss nobody → high recall. A spam filter: do not throw away important post → high precision. The metric should follow the cost of the error.

from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, roc_auc_score

y_true = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0]
y_pred = [1, 1, 1, 0, 1, 0, 0, 0, 0, 0]
print(confusion_matrix(y_true, y_pred))    # [[5 1] [1 3]] → TN=5 FP=1 FN=1 TP=3
print(precision_score(y_true, y_pred))     # 0.75
print(recall_score(y_true, y_pred))        # 0.75
print(f1_score(y_true, y_pred))            # 0.75

p = [0.9, 0.8, 0.7, 0.4, 0.6, 0.3, 0.2, 0.1, 0.05, 0.35]   # probabilities
print(roc_auc_score(y_true, p))            # 0.83 — independent of the threshold

The ROC curve plots recall against the false-positive rate for every threshold. AUC = 0.5 is chance, 1.0 is perfect. Under strong imbalance: look at the precision–recall curve instead.