Table of Content

Problem Statement

The main objective is to build a predictive model, which could help in predicting the views of the videos uploaded on the TEDx website.

Objective

The main objective is to build a predictive model, which could help in predicting the views of the videos uploaded on the TEDx website.

Dataset

We have the data of previous TED talk events , which contains data points such as the length (duration ) of the talk, topics , speaker occupation and textual features such as Transcript , Title , and Description And most importantly , the target variable : the view of the video The Data is available for 4005 TED talks.

Number of records: 4,005
Number of attributes: 19

Data Pipeline

Know Your Data: The first step in this project was to examine the various features of the dataset, understand the structure of the data and identify any patterns or trends. We looked at the shape of the data, the data types of each feature, and a statistical summary.
Exploratory Data Analysis: We conducted an exploratory analysis of the data to identify patterns and dependencies, and to draw conclusions that would be useful for further processing.
Data Cleaning: We checked for duplicated values in the dataset and then addressed any null values and outliers by imputing empty strings and dropping some of the null rows.
Textual Data Preprocessing: We used techniques such as stop word removal, punctuation removal, conversion to lowercase, stemming, tokenization, and word vectorization to prepare the textual data for clustering. We also used Principal Component Analysis (PCA) to handle the curse of dimensionality.
Cluster Implementation: We used K-Means and Agglomerative Hierarchical clustering algorithms to cluster the movies and determine the optimal number of clusters.
Content-Based Recommendation System: We built a content-based recommendation system using the similarity matrix obtained from cosine similarity, which will provide the user with 10 recommendations based on the type of movie/show they have watched.

Project Workflow

Algorithms used for ML Model implementation

Linear Regression
Ridge Regression(L2)
Lasso Regression(L1)
DecisionTreeRegressor
RandomForestRegressor
AdaBoostRegressor
GradientBoostingRegressor
XGBRegressor
VotingRegressor
StackingRegressor

Best Models to Achieve our Objective:-

1. RandomForestRegressor with hyperparameter tuning

Training data R2 and Adjusted R2 Score

R2 score 0.9108
Adjusted R2 score 0.9106

Testing data R2 and Adjusted R2 Score

R2 score 0.8977
Adjusted R2 score 0.8968

Cross-validation score

0.8974

The performance metrics

MAE 0.2613
MSE 0.1055
RMSE 0.3249

2. VotingRegressor with hyperparameter tuning

Training data R2 and Adjusted R2 Score

R2 score 0.9109
Adjusted R2 score 0.9107

Testing data R2 and Adjusted R2 Score

R2 score 0.8981
Adjusted R2 score 0.8972

Cross-validation score

0.8977

The performance metrics

MAE 0.2615
MSE 0.1051
RMSE 0.3243

Code (github)

Notebook Link

Conclusion

After evaluating multiple regression models on the dataset, Random Forest Regressor perform better than other models. They have higher R2 scores, lower error metrics, and can generalize well on unseen data.
Linear Regressor and Lasso Regressor have slightly lower performance metrics compared to Random Forest Regressor and Gradient Boosting Regressor.
Decision Tree Regressor has a lower R2 score, higher error metrics, and little bit overfits the data comparing to the other best models, indicating it's not the best model to use.
Gradient Boosting Regressor and XGBoost Regressor are high-performing algorithms that achieved impressive results in our evaluation. However, even with their strong performance, these boosting models were not able to outperform the Random Forest algorithm on this particular dataset. In other words, Random Forest Regressor still achieved the best overall performance among all the models we tested.
Voting Regressor and Stacking Regressor are ensemble techniques that combine multiple models to make predictions. The Voting Regressor model showed excellent performance, outperforming all other models. Although the difference in performance between this two models voting regressor and random forest regressor was very small, we ultimately chose to use the random forest regressor in our scenario. This decision was based on the fact that the random forest model is generally more interpretable and easier to explain to stakeholders, compared to the voting regressor which aggregates predictions from multiple models. Additionally, the random forest algorithm has been shown to perform well in a variety of regression tasks and can handle non-linear relationships between features and the target variable, making it a more robust choice for our use case.

Therefore, based on the evaluation results, the Random Forest Regressor was chosen as the best model to achieve our objective. Also in future we can try implementing some other optimising techniques to wind up with better results.

Predicting TED Talk Views with Machine Learning