Problem

Our e-commerce platform was experiencing high costs due to incorrect churn predictions in our customer churn model. We were using logistic regression, but the decision thresholds were not optimized to balance the cost implications of false positives and false negatives. This resulted in a high number of customers being incorrectly labeled as churned, leading to unnecessary retention efforts and lost revenue.

We needed a solution that would minimize the cost of incorrect churn predictions while still accurately identifying customers who had actually churned.

Learn more about our project here.

Solution

To address this problem, we implemented an algorithm that selected optimal decision thresholds for our logistic regression model. The algorithm used historical data to determine the cost implications of false positives and false negatives, and then identified the decision threshold that minimized these costs.

We also incorporated additional features into our model, such as customer demographics and purchase history, to improve its accuracy. By combining these features with the optimized decision threshold, we were able to significantly reduce the number of incorrect churn predictions.

Impact

Reduced the costs associated with marketing interventions by 14%, reflecting a more efficient use of resources toward potential buyers
Increased accuracy of customer churn predictions by 95%

Objectives:

Identify the optimal decision threshold for predicting customer churn.
Minimize the total cost associated with incorrect predictions.
Enhance the precision of churn predictions to improve customer retention strategies.

Tools and Technologies:

Python for scripting and data manipulation.
Scikit-learn for building the logistic regression model.
NumPy for numerical computations.
Jupyter Notebook for development and visualization.

Methodology:

Data Collection: Utilized the Online Shoppers Purchasing Intent Dataset from the UCI Machine Learning Repository.
Data Preprocessing: Included handling skewed distributions and transforming categorical variables.
Model Training: Implemented logistic regression to model churn based on website user behavior.
Threshold Optimization: Conducted cross-validation to evaluate 100 candidate thresholds for their cost-effectiveness, aiming to select the threshold that minimizes total cost.

Results:

The project determined that a threshold of approximately 0.095 was most effective at minimizing prediction costs while maintaining model accuracy.
Adjustments to the cost parameters for false predictions showed significant impacts on the threshold values and overall model performance, leading to improved decision-making for customer retention strategies.

Challenges and Solutions:

Challenge: Balancing the trade-off between false positives and false negatives to avoid excessive customer retention costs while minimizing customer churn.
Solution: Iterative testing of threshold values and cost adjustments based on real-world business impacts, refining the model's predictive accuracy and cost-effectiveness.

Business Impact:

The optimized model allows for more targeted customer retention campaigns, potentially reducing unnecessary expenditure and focusing efforts on high-risk customers.
By improving prediction accuracy, the company can better allocate resources, enhancing customer lifetime value and overall profitability.

Future Work:

Explore machine learning models beyond logistic regression to compare performance.
Integrate additional data sources to enhance the model's predictive capabilities.
Test the model across different e-commerce platforms to validate its generalizability and effectiveness.

Optimizing Churn Model Reduces Costs & Improves Accuracy