3 Aug 2021

Classifying data with Python

Using scikit-learn and some Python features to set up a machine learning pipeline to quantify the efficiency of different classifers for classifying the movement of a stock price.

This post walks through a solution to problem 10 from Chapter 4 of An Introduction to Statistical Learning. It demonstrates how to use Python features to build a practical machine learning pipeline from first principles.

The first part of this post specifies the problem, a binary classification problem, and introduces the fundamental differences between the classifiers considered. Subsequent sections introduce the dataset and perform exploratory data analysis to identify significant variables. I finish by introducing a simple machine learning pipeline built using standard Python libaries that applies combinations of preprocessing scripts and classifiers to the dataset.

The problem

I’m tackling Q10 from Chapter 4 of ISL that can be summarised as follows:

Compare the performance of logistic regression, linear discriminant analysis, quadratic discriminant analysis and k nearest neighbours on predicting whether a stock price will go up or down in the Weekly dataset.

The Weekly dataset records stock market returns for years and can be downloaded from this location.

Maths

The Weekly dataset exemplifies a binary classification problem: given observations , predict the binary response . We model the probability that belongs to a particular category. The equation below calculates the probability that belongs to category .

Lets describe the key differences between each classification method. Logistic regression models using the logistic function:

The coefficients and are fitted using the Maximum Likelihood method (see this explanation). This method selects and to make predicted probabilities match observed outcomes as closely as possible.

Linear discriminant analysis (LDA) models the distribution of predictors given , using Bayes’ theorem to estimate . LDA outperforms logistic regression when classes are well-separated, sample sizes are small, or predictors follow a normal distribution. LDA also handles more than two response classes effectively.

LDA assumes observations within each class follow a multivariate Gaussian distribution with a class-specific mean and a covariance matrix shared across all classes. Quadratic Discriminant Analysis (QDA) relaxes this assumption: each class has its own covariance matrix.

The k Nearest Neighbour algorithm estimates the conditional distribution of given and then classifies an observation to the class with the highest estimated probability. Given a positive integer and test observation the k-NN first identifies the K points in the training data which are closest to , represented by . The classifier then estimates the conditional probability for class as the fraction of points in whose response values equal .

The dataset

The Weekly dataset contains the Direction column as the response variable (up or down) and uses the Volume column plus five lag variables as predictors. Here’s the code to load the data:

import pyreadr
import pandas as pd

def get_dataset():

    # include a try/except clause to load data if it doesn't exist locally
    url = "https://github.com/cran/ISLR/blob/master/data/Weekly.rda?raw=true"
    dst_path = "Weekly.rda"
    if Path(dst_path).exists():
        Weekly = pyreadr.read_r(dst_path)
    else:
        Weekly = pyreadr.read_r(pyreadr.download_file(url, dst_path), dst_path)
    df_weekly = Weekly['Weekly']
    df_weekly['Direction_text'] = df_weekly['Direction']
	# convert categorical Direction to numeric (0/1)
    df_weekly['Direction'] = pd.get_dummies(df_weekly['Direction'])
    return df_weekly

df_weekly = get_dataset()
print(df_weekly.describe().to_markdown())

Returns the table below. The lag variables share identical minimum and maximum values because they derive from the lagged Today column. Notice that columns are not normalised—values don’t fall between 0 and 1. This matters: distance-based classifiers like k-NN are sensitive to the original data scale.

	Year	Lag1	Lag2	Lag3	Lag4	Lag5	Volume	Today	Direction
count	1089	1089	1089	1089	1089	1089	1089	1089	1089
mean	2000.05	0.150585	0.151079	0.147205	0.145818	0.139893	1.57462	0.149899	0.444444
std	6.03318	2.35701	2.35725	2.3605	2.36028	2.36128	1.68664	2.35693	0.497132
min	1990	-18.195	-18.195	-18.195	-18.195	-18.195	0.087465	-18.195	0
25%	1995	-1.154	-1.154	-1.158	-1.158	-1.166	0.332022	-1.154	0
50%	2000	0.241	0.241	0.241	0.238	0.234	1.00268	0.241	0
75%	2005	1.405	1.409	1.409	1.409	1.405	2.05373	1.405	1
max	2010	12.026	12.026	12.026	12.026	12.026	9.32821	12.026	1

Exploratory analysis

To identify which variables correlate significantly with Direction, I use logistic regression. I adapted the approach from the statsmodels documentation.

Here’s the logistic regression model. The null hypothesis, , posits no relationship between the predictors (Lag1 through Lag5, Volume) and stock price Direction. The chained methods format output directly as markdown, avoiding manual copying.

import statsmodels.api as sm

X = df_weekly[['Lag1', 'Lag2', 'Lag3', 'Lag4', 'Lag5', 'Volume']]
y = df_weekly['Direction']

# building the model and fitting the data
log_reg = sm.Logit(y, X).fit()
log_reg_summary = log_reg.summary()
results_as_html = log_reg_summary.tables[1].as_html()
print(pd.read_html(results_as_html, header=0, index_col=0)[0].to_markdown())

The logistic regression identifies only Lag2 and Volume as significant predictors: both have p-values below 0.05. These are the only variables worth using as classifier inputs ¹.

	coef	std err	z		0.025	0.975
Lag1	0.0327	0.026	1.250	0.211	-0.019	0.084
Lag2	-0.0682	0.027	-2.556	0.011	-0.12	-0.016
Lag3	0.0081	0.026	0.306	0.759	-0.044	0.060
Lag4	0.0194	0.026	0.740	0.459	-0.032	0.071
Lag5	0.0069	0.026	0.261	0.794	-0.045	0.058
Volume	-0.0569	0.027	-2.125	0.034	-0.109	-0.004

Applying classifiers

The task requires comparing four classifiers on the dataset:

Logistic regression
Linear discriminant analysis
Quadratic discriminant analysis
k Nearest neighbours

Though scikit-learn Pipeline exists, building a custom pipeline explores how core Python libraries work together.

I started by defining classes to describe the Preprocessing stage of the pipeline and the Classifier using the dataclasses module. The Preprocessing class holds the name and a function which performs a set of transforms on the dataset to create the test and training datasets. The Classifier holds the name, a scikit-learn classifier and optionally the classifier score and Preprocessing class. I’ve added optional type hints as these are a massive help when understanding code.

The run_classifier function takes a list of Preprocessing and Classifier classes and uses itertools.product to create an list of pipelines to be run on the Weekly dataset. itertools.product is similar to np.meshgrid but for Python objects rather than np.arrays.

Instead of filtering the results dataframe, I use attrgetter from the operator module to find the classifier with the highest score directly.

# create budget pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis, QuadraticDiscriminantAnalysis
from sklearn.metrics import accuracy_score
from sklearn.base import BaseEstimator
from typing import List, Callable, Tuple, Optional
from dataclasses import dataclass
from operator import attrgetter
import itertools

pd.options.display.float_format = '${:,.5f}'.format

@dataclass
class Preprocess:
    """
    Class to hold information on the preprocessing used on a dataset.

    """
    name: str
    transform: Callable


@dataclass
class Classifier:
    """
    Class to hold information on a Classifier.

    """
    name: str
    model: BaseEstimator
    preprocess: Optional[Preprocess] = None
    score: float = 0.0


def lag2_preprocess(df: pd.DataFrame) -> Tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame, pd.DataFrame]:
    """This is the default test training split required by problem."""
    idx_train = df['Year'] < 2009.0
    idx_test = df['Year'] > 2008.0
    all_cols = ['Lag1', 'Lag2', 'Lag3', 'Lag4', 'Lag5', 'Volume']
    X_train = df['Lag2'][idx_train].values[:, None] # need to pass a 2d array to .fix(X, y)
    y_train = df['Direction'][idx_train]
    X_test = df['Lag2'][idx_test].values[:, None]
    y_test = df['Direction'][idx_test]
    return X_train, y_train, X_test, y_test


def lag2_volume_preprocess(df: pd.DataFrame) -> Tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame, pd.DataFrame]:
    """This is the default test training split required by problem."""
    idx_train = df['Year'] < 2009.0
    idx_test = df['Year'] > 2008.0
    all_cols = ['Lag1', 'Lag2', 'Lag3', 'Lag4', 'Lag5', 'Volume']
    X_train = df[['Lag2', 'Volume']][idx_train]
    y_train = df['Direction'][idx_train]
    X_test = df[['Lag2', 'Volume']][idx_test]
    y_test = df['Direction'][idx_test]
    return X_train, y_train, X_test, y_test


def apply_classifier(model: Classifier, dataset: pd.DataFrame) -> Classifier:
    """
    Apply classifier to dataset given function to create training and test datasets.

    """
    X_train, y_train, X_test, y_test = model.preprocess.transform(dataset)
    model.model = model.model.fit(X_train, y_train)
    y_predict = model.model.predict(X_test)
    model.score = accuracy_score(y_predict, y_test)
    return model


def run_classifiers():
    # define preprocessing functions
    preprocessors = [
        Preprocess("Lag 2", lag2_preprocess),
        Preprocess("Lag 2 and Volume", lag2_volume_preprocess),
    ]

    # define classifier models
    classifiers = [
        Classifier("Logistic Regression", LogisticRegression(random_state=0)),
        Classifier("Linear Discriminant Analysis", LinearDiscriminantAnalysis()),
        Classifier("Quadratic Discriminant Analysis", QuadraticDiscriminantAnalysis()),
        Classifier("kNN, k=1", KNeighborsClassifier(n_neighbors=1)),
        Classifier("kNN, k=2", KNeighborsClassifier(n_neighbors=2))
    ]

    # associate preprocessor function with classifiers for analysis
    prepped_classifiers = []
    for preprocessor, classifier in itertools.product(preprocessors, classifiers):
        prepped_classifier = Classifier(classifier.name, classifier.model, preprocessor)
        prepped_classifiers.append(prepped_classifier)

    # load data
    df_weekly = get_dataset()

    # train classifiers on datasets
    trained_classifiers = [apply_classifier(classifier, df_weekly) for classifier in prepped_classifiers]

    # create results dataframe
    df = pd.DataFrame({"Classifier name": [classifier.name for classifier in trained_classifiers],
                       "Preprocess name": [classifier.preprocess.name for classifier in trained_classifiers],
                       "Score": [classifier.score for classifier in trained_classifiers]})

    with pd.option_context('display.float_format', '{:,.5f}'.format):
        print(df.to_html())

    # get classifiers with maximum scores
    # returns the first classifier with the highest score in the event of a tie
    best_classifier = max(trained_classifiers, key=attrgetter('score'))
    print("The best classifier is {0} using the {1} preprocessor with a score of {2:.4f}".format(best_classifier.name, best_classifier.preprocess.name, best_classifier.score))


run_classifiers()

Returns:

The best classifier is Logistic Regression using the Lag 2 preprocessor with a score of 0.6250

Logistic Regression and Linear Discriminant Analysis achieve the best performance across both preprocessing methods and produce identical results. The difference lies in how they fit coefficients: logistic regression uses maximum likelihood, while LDA derives coefficients from an estimated normal distribution. When data is approximately Gaussian, both methods converge. Linear Discriminant Analysis outperforms logistic regression when data is truly Gaussian; logistic regression wins otherwise.

	Classifier name	Preprocess name	Score
0	Logistic Regression	Lag 2	0.62500
1	Linear Discriminant Analysis	Lag 2	0.62500
2	Quadratic Discriminant Analysis	Lag 2	0.58654
3	kNN, k=1	Lag 2	0.50000
4	kNN, k=2	Lag 2	0.58654
5	Logistic Regression	Lag 2 and Volume	0.53846
6	Linear Discriminant Analysis	Lag 2 and Volume	0.53846
7	Quadratic Discriminant Analysis	Lag 2 and Volume	0.47115
8	kNN, k=1	Lag 2 and Volume	0.55769
9	kNN, k=2	Lag 2 and Volume	0.61538

Review

This post demonstrates how core Python libraries combine to build a machine learning pipeline for binary classification. Four classifiers were applied to the Weekly dataset to predict stock price direction. The custom pipeline implementation showcased Python’s dataclasses, itertools, and operator modules. Extending this work could involve applying different scaling methods to improve classifier performance.

Typical p-value cutoffs are or . Choosing which p-value to use and even whether they should be used in this way is an area of academic debate. I enjoyed this tweet on the subject. ↩

Classifying data with Python

The problem

Maths

The dataset

Exploratory analysis

Applying classifiers

Review

Footnotes

Related Posts

Learning about data science

How to reduce the memory used by Pandas DataFrames

Calculating covariance matrices with Python

Demonstrating handling Python errors with reproducible scripts