Logistic regression and decision trees

Logistic regression is linear regression squeezed through a sigmoid so that the output becomes a probability between 0 and 1:

p = σ(w·x + b), where σ(z) = 1/(1 + e⁻ᶻ)

The boundary between the classes becomes a straight line (or a plane). Fast, stable, and the coefficients can be interpreted.

A decision tree asks yes/no questions in sequence: «age > 30?» → «income > 40k?» → the class. The boundary becomes staircase-shaped. It handles non-linear patterns and needs no scaling — but it overfits easily if it is allowed to grow freely.

A rule of thumb: start with logistic regression as a baseline. If the pattern cannot be separated with a line, try a tree or a random forest (many trees voting).

from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.metrics import confusion_matrix, classification_report

models = {
    "logreg": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
    "tree":   DecisionTreeClassifier(max_depth=4, random_state=0),
    "forest": RandomForestClassifier(n_estimators=200, random_state=0),
}
for name, m in models.items():
    m.fit(X_tr, y_tr)
    print(name, round(m.score(X_va, y_va), 3))

print(confusion_matrix(y_va, models["logreg"].predict(X_va)))
# [[85  7]     row = truth, column = guess
#  [12 46]]    → 7 false positives, 12 false negatives
print(classification_report(y_va, models["logreg"].predict(X_va)))

The threshold is a choice. predict() uses 0.5, but predict_proba() gives the probability — lower the threshold if missed positives are more expensive than false alarms.