Hybrid Movie Recommendation System

A full end-to-end hybrid movie recommender built on MovieLens 32M and TMDb metadata, combining collaborative filtering, content-based filtering, and a LightGBM learning-to-rank re-ranker.

1. Project Overview

This project builds a three-stage hybrid movie recommendation pipeline:

Stage 1 — Candidate Generation
Item-based Collaborative Filtering (Item-CF) and Content-Based Filtering retrieve a broad pool of candidate movies for each user. Item-CF covers 43,884 rated movies while ALS covers 36,470; the union provides maximum recall.

Stage 2 — Scoring
Three complementary models score every candidate:

SVD (Surprise) — predicts exact star ratings, optimising RMSE
ALS (Implicit) — ranks items by user preference, optimising NDCG and Hit Rate
Item-CF — weighted average of user's rated movies' similarity scores

Stage 3 — Re-Ranking
A LightGBM LambdaRank model takes 12 hand-crafted features (collaborative scores, content scores, interaction terms, popularity, average rating, genre overlap) and produces the final ranked top-K list.

2. System Architecture

┌─────────────────────────────────────────────────────────┐
│                      Data Sources                       │
│  ┌─────────────────────┐    ┌──────────────────────┐    │
│  │   MovieLens 32M     │    │      TMDb API        │    │
│  │  · ratings.csv      │    │  · 86,234 Movies     │    │
│  │  · movies.csv       │    │  · 20 Attributes     │    │
│  │  · links.csv        │    └──────────────────────┘    │
│  └─────────────────────┘                                │
└─────────────────────────────────────────────────────────┘
              │                          │
              ▼                          ▼
    ┌─────────────────┐       ┌──────────────────────────┐
    │  DATA CLEANING  │       │  PREPROCESSING           │
    │  Deduplication  │──────▶│  Temporal Split         │
    │  Filtering      │       │  Train 80% / Val 10%     │
    │  Standardisation│       │  Test 10%                │
    └─────────────────┘       └──────────────────────────┘
              │                          │
    ┌──────────────────────┐   ┌──────────────────────────┐
    │  FEATURE ENGINEERING │   │  USER-ITEM MATRIX        │
    │  Sentence Transformer│   │  170,282 × 36,470        │
    │  all-mpnet-base-v2   │   │  Density 0.41%           │
    │  Split PCA (256 dim) │   └──────────────────────────┘
    │  Genre · Director    │              │
    │  Numericals          │              │
    └──────────────────────┘             │
              │                          │
    ┌───────────────────────┐   ┌───────────────────────────┐
    │  CONTENT-BASED        │   │  COLLABORATIVE FILTERING  │
    │  FAISS IndexFlatIP    │   │  SVD  — RMSE 0.8059       │
    │  User Content Profiles│   │  ALS  — HitRate@10 0.777  │
    │  Nearest Neighbour    │   │  Item-CF — Similarity Dict│
    └───────────────────────┘   └───────────────────────────┘
              │                          │
              └──────────┬───────────────┘
                         ▼
              ┌──────────────────────┐
              │  HYBRID RE-RANKER    │
              │  LightGBM LambdaRank │
              │  12 Features         │
              │  300 Rounds          │
              │  num_leaves=127      │
              └──────────────────────┘
                         │
                         ▼
              ┌──────────────────────┐
              │  FINAL EVALUATION    │
              │  RMSE     0.8154     │
              │  NDCG@20  0.2811     │
              │  HitRate@20 0.5624   │
              └──────────────────────┘

3. Dataset

MovieLens 32M

File	Description	Size
`ratings.csv`	User–movie ratings (0.5–5.0 stars, half-star increments)	~32M rows
`movies.csv`	Movie titles and pipe-separated genre strings	~86K rows
`links.csv`	Mapping between MovieLens IDs and TMDb/IMDb IDs	~86K rows

Key statistics (after cleaning):

Metric	Value
Total ratings	~32,000,000
Unique users	~170,282
Unique movies rated	~36,470
Average rating	3.54 / 5.0
Rating scale	0.5 – 5.0 (half-star)
Matrix sparsity	99.59% sparse
Date range	January 1996 → March 2023

TMDb Metadata

Fetched via the TMDb API for all 86,234 movies in the catalog.

Field	Coverage
overview	~99.7%
genres	~99.9%
keywords	~95%+
director	~95%+
top_cast	~95%+
runtime	~93% (bad values clipped)
vote_average	~99%+
popularity	~99%+
original_language	100%

Language distribution in full catalog: English (62.7%), French (6.0%), Italian (3.9%), Japanese (3.4%), Spanish (3.3%), German (2.6%), other (18.2%)

Top genres: Drama (34,175), Comedy (23,124), Thriller (11,823), Romance (10,369), Action (9,668), Documentary (9,363)

4. Repository Structure

TIH-july2026-movie-reco/
│
├── Notebooks/
│   ├── 01_data_cleaning.ipynb         # Clean ratings, movies, TMDb CSVs
│   ├── 02_preproccessing.ipynb        # Join datasets, temporal split, sparse matrix
│   ├── 03_feature_engineering.ipynb   # Sentence embeddings, genre, director, PCA
│   ├── 04_collaborative_filtering.ipynb  # SVD, ALS, Item-CF
│   ├── 05_content_based.ipynb         # Split PCA, FAISS index, content profiles
│   ├── 06_hybrid_reranker.ipynb       # LightGBM LambdaRank — final model
│   ├── 07_data_demographics.ipynb     # Dataset statistics and visualisations
│   └── 08_model_analysis.ipynb        # SHAP, error analysis, language bias
│
├── Images/
│   ├── architecture/
│   │   └── system_architecture.png    # Full pipeline diagram
│   ├── analysis/
│   │   ├── analysis_shap_bar.png      # SHAP feature importance (bar)
│   │   ├── analysis_shap_beeswarm.png # SHAP beeswarm (direction + magnitude)
│   │   ├── analysis_error_analysis.png # 5-star movies vs recommendation rank
│   │   ├── analysis_language_bias.png  # Language distribution pie charts
│   │   ├── analysis_language_bias_bar.png  # Language bias bar chart
│   │   ├── analysis_overview_distribution.png # Overview word count stats
│   │   ├── feature_importance_content_dims.png
│   │   └── feature_importance_lgbm.png
│   └── demographics/
│       ├── demographics_rating_distribution.png
│       ├── demographics_ratings_per_user.png
│       ├── demographics_ratings_per_movie.png
│       ├── demographics_activity_over_time.png
│       ├── demographics_genre_distribution.png
│       ├── demographics_release_year.png
│       ├── demographics_runtime.png
│       ├── demographics_language.png
│       ├── demographics_metadata_coverage.png
│       └── demographics_sparsity.png
│
└── LICENSE.md
└── README.md

5. Notebook Walkthrough

NB 1 — Data Cleaning

File: 01_data_cleaning.ipynb
Run on: Local machine
Inputs: ratings.csv, movies.csv, links.csv, tmdb_features.csv
Outputs: ratings_clean.csv, movies_clean.csv, tmdb_clean.csv

What it does:

ratings.csv:

Removes duplicate ratings for the same user–movie pair (keeps most recent by timestamp)
Remaps secondary duplicate movieIds to their canonical (lower) ID, then resolves post-remap duplicates
Filters users with fewer than 5 ratings (MIN_USER_RATINGS = 5)
Filters movies with fewer than 5 ratings (MIN_MOVIE_RATINGS = 5)

movies.csv:

Merges duplicate movieId rows by taking the union of their genre strings
Extracts release_year from the title string using a trailing (YYYY) regex pattern and cleans the title
Fills missing genres with "(no genres listed)"
Keeps only movies present in the cleaned ratings

tmdb_features.csv:

Fills text fields (overview, keywords, genres, director, top_cast) with empty strings
Fills original_language with "unknown" where missing
Flags bad runtimes (< 10 min or > 300 min) as NaN instead of dropping rows
Replaces zero budget/revenue values with NaN (TMDb convention for unknown)
Parses release_date and extracts release_year
Retains all 86K TMDb rows — no filtering by rated movies (needed for cold-start content-based recommendations)

Config thresholds:

Parameter	Value	Description
`MIN_USER_RATINGS`	5	Drop users below this
`MIN_MOVIE_RATINGS`	5	Drop movies below this
`MIN_RUNTIME`	10 min	Bad data floor
`MAX_RUNTIME`	300 min	Bad data ceiling

NB 2 — Preprocessing

File: 02_preproccessing.ipynb
Run on: Kaggle / Google Colab (requires > 16 GB RAM for the full join)
Inputs: ratings_clean.csv, movies_clean.csv, tmdb_clean.csv, links.csv
Outputs: master_df.parquet, train.parquet, val.parquet, test.parquet, val_loo.parquet, test_loo.parquet, user_item_matrix.npz, id_maps.pkl

What it does:

Joins all three datasets into a single master_df. To avoid MemoryError on limited-RAM machines, only a slim subset of TMDb columns (9 fields) is merged into the 32M-row DataFrame. The full TMDb metadata is left for notebook 3.

Temporal train/val/test split (no leakage):

All ratings sorted by timestamp
├── Train  80%  (oldest ratings)
├── Val    10%  (middle)
└── Test   10%  (most recent)

An assertion verifies that val_min_timestamp >= train_max_timestamp and test_min_timestamp >= val_max_timestamp. This is a stricter split than random sampling — it simulates a real deployment where the model is trained on historical data and evaluated on future interactions.

Leave-one-out ground truth is generated for val and test: for each user, their single most recently rated movie is extracted as the positive item for ranking evaluation.

Sparse user-item matrix:

Shape: 170,282 users × 36,470 movies (CSR format, float32)
Non-zero cells: ~32M
Density: 0.41%
Contiguous ID mappings (user2idx, movie2idx, idx2user, idx2movie) are saved as id_maps.pkl and reused in every subsequent notebook

NB 3 — Feature Engineering

File: 03_feature_engineering.ipynb
Run on: Kaggle with T4 GPU (CPU fallback ~2–3 hours)
Inputs: tmdb_clean.csv, movies_clean.csv, links.csv, id_maps.pkl
Outputs: embeddings.npy, overview_embeddings.npy, tag_embeddings.npy, content_features.npz, content_movie_ids.npy, feature_meta.pkl

What it does:

This notebook constructs the full content feature vector per movie across 86K items. There are four feature blocks:

Block 1 — Semantic Embeddings (768 dims)
Uses sentence-transformers/all-mpnet-base-v2 to encode a concatenated text string per movie:

text = f"{overview} {keywords} {keywords} {genres} {genres}"

Keywords and genres are repeated twice to give them extra weight relative to the longer free-text overview. Embeddings are L2-normalised. On a T4 GPU with fp16 and batch size 256, this takes ~15 minutes for 86K movies.

Split embeddings (new in v2):
Two additional embedding sets are generated separately:

overview_embeddings.npy — overview text only (768 dims)
tag_embeddings.npy — keywords + genres only (768 dims)

These are used in notebook 5 to apply differential weighting (70% overview, 30% tags) in the PCA step.

Block 2 — Genre Multi-Hot (~20 dims)
Pipe-separated TMDb genre strings are parsed into lists and binarised with MultiLabelBinarizer.

Block 3 — Director Encoding (N dims, sparse)
Directors appearing in ≥ 3 movies (MIN_DIRECTOR_FREQ = 3) are one-hot encoded as a sparse matrix. Rarer directors are discarded.

Block 4 — Numerical Features (4 dims)
runtime, vote_average, popularity, release_year — missing values imputed with column median, then scaled with MinMaxScaler to [0, 1].

Cast encoding was tested and removed:
An ablation study showed that 47,025 cast-encoding dimensions contributed only 2.2% LightGBM gain but bloated the content vector from ~8,636 to ~55,661 dimensions. Cast was removed to keep the vector tractable.

Final content vector shape: (86,234 movies) × (~8,636 dims), stored as a CSR sparse matrix.

NB 4 — Collaborative Filtering

File: 04_collaborative_filtering.ipynb
Run on: Kaggle CPU (ALS GPU extension unavailable on standard Kaggle environments)
Inputs: train.parquet, val.parquet, test.parquet, val_loo.parquet, test_loo.parquet, user_item_matrix.npz, id_maps.pkl
Outputs: svd_model.pkl, als_model.pkl, user_embeddings.npy, item_embeddings.npy, item_sim_dict.pkl, item_cf_model.pkl, cf_eval_results.json

Three complementary collaborative filtering models are trained:

Model A — SVD (Surprise)

Optimises RMSE on explicit 0.5–5.0 star ratings. The Netflix Prize baseline is 0.8572.

Hyperparameter	Value
`n_factors`	150
`n_epochs`	30
`lr_all`	0.005
`reg_all`	0.02

Evaluation: RMSE and MAE on the val testset (users/movies seen in training).

Model B — ALS (Implicit)

Optimises ranking metrics (NDCG@10, Hit Rate@10) via leave-one-out evaluation with 99 random negatives per user. Explicit ratings are converted to confidence values: c_ui = alpha × rating where alpha = 40.

Hyperparameter	Value
`factors`	256
`iterations`	30
`regularization`	0.01
`alpha`	40

User and item embedding matrices are extracted and saved for use in the hybrid re-ranker (notebook 6) and for dot-product feature computation.

Model C — Item-CF (Item-Item Cosine Similarity)

Computes pairwise cosine similarity between all 36,470 item vectors in the rating matrix. For each movie, the top 50 most similar movies are stored in item_sim_dict (keyed by movie index). At inference time, a user's predicted rating for a candidate movie is the weighted average of their ratings for similar movies:

predicted_rating = Σ(sim(i, j) × r_uj) / Σ|sim(i, j)|

Item-CF is critical because it covers all 43,884 rated movies in the catalog, whereas ALS only covers 36,470 movies in the training matrix. The extra coverage improves recall in the candidate generation stage.

NB 5 — Content-Based Filtering

File: 05_content_based.ipynb
Run on: Kaggle CPU/GPU
Inputs: content_features.npz, overview_embeddings.npy, tag_embeddings.npy, train.parquet, val.parquet, val_loo.parquet, id_maps.pkl, user_item_matrix.npz
Outputs: pca_model.pkl, content_features_pca.npy, faiss_index.bin, content_movie_ids.npy, cb_eval_results.json

Split PCA (new in v2):

Two separate PCA models reduce the 768-dim overview and tag embeddings into a combined 256-dim content vector:

Overview PCA: 768 → 180 dims (70% of the final vector)
Tag PCA: 768 → 76 dims (30% of the final vector)
Combined: [overview_pca | tag_pca] = 256 dims

This gives semantic plot meaning 2.3× more weight than genre/keyword tags, which reflects human intuition: two movies are more similar if their plots are alike than if they share a genre.

Both blocks are L2-normalised after concatenation.

FAISS Index:

A faiss.IndexFlatIP (exact inner-product search) is built over all 86K normalised 256-dim vectors. Since vectors are L2-normalised, inner product equals cosine similarity. The index enables sub-millisecond approximate-nearest-neighbour search across the full catalog.

User content profiles:

A user's content profile is computed as a weighted average of the content vectors of movies they have rated ≥ 3.5 stars (falling back to all rated movies if no high ratings exist). The weight for each movie is max(0.1, rating - 3.5 + 1), so 5-star movies contribute more than 4-star movies.

Evaluation is run on a sample of 3,000 val users, measuring Precision@10, Recall@10, Hit Rate@10, and NDCG@10.

NB 6 — Hybrid Re-Ranker (Final Model)

File: 06_hybrid_reranker.ipynb
Run on: Kaggle CPU/GPU
Inputs: All outputs from notebooks 2–5
Outputs: hybrid_model.pkl, final_eval_results.json

This is the production model. It combines all upstream signals into a single learning-to-rank model.

Candidate generation (two sources):

Item-CF candidates: for each movie the user has rated highly, retrieve its top-K similar movies from item_sim_dict and collect the union.

Content-Based candidates: query the FAISS index with the user's content profile to retrieve the nearest 200 movies.

The two candidate sets are merged into a single pool, typically 500–1,500 candidates per user.

12 features for LightGBM:

Feature	Description
`als_score`	ALS recommendation score for this user–movie pair
`cb_score`	Content-based cosine similarity (user profile vs movie vector)
`svd_score`	SVD predicted rating (0.5–5.0 scale)
`icf_score`	Item-CF weighted predicted rating
`als_dot`	Direct dot product of ALS user and item embeddings
`popularity`	Number of ratings for the movie in the training set
`avg_rating`	Historical average rating for the movie
`genre_overlap`	Cosine similarity between user genre profile and movie genre vector
`als_cb_interaction`	`als_score × cb_score`
`svd_avg_interaction`	`svd_score × avg_rating`
`icf_als_interaction`	`icf_score × als_score`
`icf_cb_interaction`	`icf_score × cb_score`

LightGBM hyperparameters:

Parameter	Value
`objective`	`lambdarank`
`num_leaves`	127
`learning_rate`	0.03
`num_boost_round`	300
`metric`	`ndcg`
`ndcg_eval_at`	[10, 20]

Training data construction:
For each training user, candidates are labelled 1 (positive) if the movie appears in the user's val liked set (rating ≥ 3.5) and 0 otherwise. Positive-to-negative ratio is typically 1:50–1:100. query_group sizes (number of candidates per user) are passed to LightGBM's listwise LambdaRank objective.

Final evaluation is run on the held-out test set across a sample of users, reporting RMSE (from SVD), NDCG@20, Precision@20, Recall@20, and Hit Rate@20.

NB 7 — Data Demographics

File: 07_data_demographics.ipynb
Run on: Local machine (no GPU required)
Inputs: ratings_clean.csv, movies_clean.csv, tmdb_clean.csv, train.parquet, val.parquet, test.parquet, user_item_matrix.npz

Generates 10 statistical charts profiling the dataset. All charts are saved to the Images/demographics/ folder.

Chart	What it shows
`demographics_rating_distribution.png`	Frequency per star value + cumulative distribution; mean 3.54, peak at 4.0 (8.4M ratings)
`demographics_ratings_per_user.png`	Power-law distribution; most users have 20–100 ratings
`demographics_ratings_per_movie.png`	Extreme long tail; most movies have very few ratings
`demographics_activity_over_time.png`	Rating activity by year, showing growth and recent plateau
`demographics_genre_distribution.png`	Drama dominates (34,175), followed by Comedy (23,124)
`demographics_release_year.png`	Catalog spans ~1888–2024; volume peaks in 2000–2020
`demographics_runtime.png`	Median ~95 min; bimodal (short films vs features)
`demographics_language.png`	English 62.7% in catalog; non-English well-represented
`demographics_metadata_coverage.png`	All key TMDb fields above 90% coverage
`demographics_sparsity.png`	Visual spy plot of the 500×500 matrix sub-sample

NB 8 — Model Analysis

File: 08_model_analysis.ipynb
Run on: Local machine (no GPU required)
Inputs: tmdb_clean.csv, ratings_clean.csv, test.parquet, train.parquet, hybrid_model.pkl, id_maps.pkl, user_item_matrix.npz, all embedding/index files

Four deep-dive analyses are run after the model is trained:

Analysis 1 — Overview Attribute Distribution
Most movies have 30–100 word overviews (median 40 words, mean 49 words). Only 248 movies are missing overviews entirely. Overview coverage is ≥ 99.5% across every genre. Interestingly, extremely long overviews (> 100 words) correlate with slightly lower average TMDb vote scores, suggesting verbose descriptions may reflect lower-quality entries.

Analysis 2 — Error Analysis
A sample of 500 user–movie pairs where the user gave 5 stars in the test set is analysed to see where the model placed that movie in its ranked list. Key findings: most 5-star movies that are missed (rank > 20) are low-popularity/niche titles. The SVD predicted rating for missed movies does not clearly separate from hits, confirming that popularity bias — not prediction error — is the primary source of misses.

Analysis 3 — Language Bias
Compares the language distribution in the full catalog, test ground truth, and model recommendations. The catalog is 62.7% English, but ground truth is 90.4% English (reflecting user behaviour). Recommendations mirror ground truth at 90.3% English — the model is not adding additional language bias beyond what is already present in users' rating histories.

Analysis 4 — SHAP Feature Attribution
shap.TreeExplainer is applied to the LightGBM model on a sample of 5,000 candidate (user, movie) pairs. The SHAP bar and beeswarm plots reveal the following feature importance ranking:

Rank	Feature	Interpretation
1	`icf_als_interaction`	CF interaction term dominates
2	`svd_avg_interaction`	SVD × historical rating
3	`als_dot`	Raw ALS dot product
4	`als_score`	ALS score
5	`icf_score`	Item-CF score
6	`genre_overlap`	Genre similarity
7	`svd_score`	SVD predicted rating
8	`avg_rating`	Popularity signal
9	`popularity`	Log popularity
10	`cb_score`	Content-based similarity
11	`icf_cb_interaction`	Negligible
12	`als_cb_interaction`	Negligible

Key finding: Collaborative signals (ranks 1–5) overwhelmingly dominate content-based signals (rank 10). cb_score alone contributes < 5% of total SHAP impact. This confirms that for users with sufficient rating history, collaborative filtering is the stronger signal.

6. Results & Metrics

Rating Prediction (SVD on test set)

Metric	Value	Benchmark
RMSE	0.8154	< 0.8572 (Netflix Prize)
MAE	—	—

Ranking (Hybrid Model on test set)

Metric	Value	Target
NDCG@20	0.2811	> 0.70
Hit Rate@20	0.5624	> 0.80
Precision@20	—	> 0.30
Recall@20	—	> 0.40

Intermediate model metrics (ALS, val set)

Metric	Value
ALS Hit Rate@10	0.777
SVD Val RMSE	0.8059

Note: RMSE beats the Netflix Prize benchmark (0.8572). NDCG@20 and Hit Rate@20 are below the original stretch targets, which is expected given the extreme matrix sparsity (99.59%) and the cold-start nature of many test users.

7. Key Design Decisions

Temporal split over random split
A time-based 80/10/10 split is strictly enforced. This prevents future leakage and simulates real deployment: the model is always trained on older interactions and evaluated on newer ones.

Split embeddings (v2 innovation)
Generating separate PCA projections for overview text (180 dims, 70%) and tag text (76 dims, 30%) gives the system fine-grained control over the semantic vs structural trade-off in content similarity. A single combined embedding would blur this distinction.

Cast encoding removed after ablation
Including all frequent cast members added 47,025 feature dimensions to the content vector but only improved LightGBM gain by 2.2%. The dimension blow-up worsened FAISS indexing time and user profile quality. Cast was removed in feature engineering v2.

Item-CF for broader candidate coverage
ALS operates only on movies present in the training matrix (36,470 movies). Item-CF uses the full normalized rating matrix and covers 43,884 movies, including some that ALS skips due to minimum ratings thresholds. Using Item-CF as the primary candidate generator improves recall, especially for less-popular films.

LightGBM LambdaRank as the final layer
A listwise learning-to-rank objective (LambdaRank) is better suited than pointwise regression for recommendation tasks. The hybrid model integrates all signals (CF + CB + popularity + genre overlap + interaction terms) in a single optimised pass.

FAISS flat index for exact search
IndexFlatIP performs exact inner-product (= cosine) search rather than approximate search. With 86K 256-dim vectors, exact search is fast enough (~10–50ms per query) and avoids the accuracy trade-off of ANN methods.

8. Environment Setup

Dependencies

pip install pandas numpy scipy scikit-learn matplotlib
pip install sentence-transformers==2.7.0 transformers==4.40.0 tokenizers accelerate
pip install "numpy<2" scikit-surprise implicit
pip install faiss-cpu lightgbm shap

⚠️ NumPy version note: scikit-surprise requires NumPy < 2. Install it before importing surprise:
pip install "numpy<2" scikit-surprise -q

Hardware requirements

Notebook	Recommended	Minimum
NB 1 (Data Cleaning)	Any CPU, 8 GB RAM	4 GB RAM
NB 2 (Preprocessing)	16+ GB RAM	Kaggle / Colab (free tier)
NB 3 (Feature Engineering)	T4 GPU (Kaggle free tier)	CPU (2–3 hours)
NB 4 (Collaborative Filtering)	CPU, 16 GB RAM	8 GB RAM
NB 5 (Content-Based)	CPU, 8 GB RAM	4 GB RAM
NB 6 (Hybrid Re-Ranker)	CPU/GPU, 16 GB RAM	8 GB RAM
NB 7 & 8 (Analysis)	Local CPU	Any machine

9. How to Run

The notebooks must be run in order. Each notebook's outputs become the next notebook's inputs.

Step 1 — Download raw data

Download MovieLens 32M from grouplens.org/datasets/movielens/ and fetch TMDb metadata using the TMDb API (API key required). Place files at the paths defined in each notebook's config section.

Step 2 — Run NB 1 locally

notebooks/01_data_cleaning.ipynb

Outputs: ratings_clean.csv, movies_clean.csv, tmdb_clean.csv

Step 3 — Upload cleaned CSVs to Kaggle / Colab, run NB 2

notebooks/02_preproccessing.ipynb

Outputs: master_df.parquet, train.parquet, val.parquet, test.parquet, user_item_matrix.npz, id_maps.pkl

Step 4 — Run NB 3 on Kaggle with GPU enabled

Go to Settings → Accelerator → T4 GPU before running.

notebooks/03_feature_engineering.ipynb

Outputs: embeddings.npy, overview_embeddings.npy, tag_embeddings.npy, content_features.npz, content_movie_ids.npy, feature_meta.pkl

Step 5 — Run NB 4 on Kaggle CPU

notebooks/04_collaborative_filtering.ipynb

Outputs: svd_model.pkl, als_model.pkl, user_embeddings.npy, item_embeddings.npy, item_sim_dict.pkl, cf_eval_results.json

Step 6 — Run NB 5 on Kaggle

notebooks/05_content_based.ipynb

Outputs: pca_model.pkl, content_features_pca.npy, faiss_index.bin, cb_eval_results.json

Step 7 — Run NB 6 on Kaggle (final model)

notebooks/06_hybrid_reranker.ipynb

Outputs: hybrid_model.pkl, final_eval_results.json

Step 8 — Run NB 7 & NB 8 locally for analysis

notebooks/07_data_demographics.ipynb
notebooks/08_model_analysis.ipynb

Place all model artifacts in the same directory and update the file path config at the top of each notebook.

10. Artifacts & Output Files

File	Generated by	Description
`ratings_clean.csv`	NB 1	Deduplicated, filtered ratings
`movies_clean.csv`	NB 1	Cleaned movie titles and genres
`tmdb_clean.csv`	NB 1	Cleaned TMDb metadata (all 86K movies)
`master_df.parquet`	NB 2	Full joined dataset
`train.parquet`	NB 2	Training split (80%)
`val.parquet`	NB 2	Validation split (10%)
`test.parquet`	NB 2	Test split (10%)
`val_loo.parquet`	NB 2	Leave-one-out ground truth (val)
`test_loo.parquet`	NB 2	Leave-one-out ground truth (test)
`user_item_matrix.npz`	NB 2	Sparse CSR rating matrix (170K × 36K)
`id_maps.pkl`	NB 2	user2idx, movie2idx, idx2user, idx2movie
`embeddings.npy`	NB 3	Combined semantic embeddings (86K × 768)
`overview_embeddings.npy`	NB 3	Overview-only embeddings (86K × 768)
`tag_embeddings.npy`	NB 3	Tag-only embeddings (86K × 768)
`content_features.npz`	NB 3	Full content feature matrix (86K × ~8636, sparse)
`content_movie_ids.npy`	NB 3	movieId order for content_features rows
`feature_meta.pkl`	NB 3	Encoders and scalers for inference
`svd_model.pkl`	NB 4	Trained SVD model (Surprise)
`als_model.pkl`	NB 4	Trained ALS model (Implicit)
`user_embeddings.npy`	NB 4	ALS user latent factors (170K × 256)
`item_embeddings.npy`	NB 4	ALS item latent factors (36K × 256)
`item_sim_dict.pkl`	NB 4	Item-CF similarity dict {item_idx: [(sim_idx, score), ...]}
`cf_eval_results.json`	NB 4	SVD, ALS, Item-CF evaluation metrics
`pca_model.pkl`	NB 5	Split PCA models (overview + tag)
`content_features_pca.npy`	NB 5	Compressed content vectors (86K × 256)
`faiss_index.bin`	NB 5	FAISS IndexFlatIP for cosine search
`cb_eval_results.json`	NB 5	Content-based precision/recall metrics
`hybrid_model.pkl`	NB 6	Trained LightGBM LambdaRank model
`final_eval_results.json`	NB 6	Final test set evaluation results

11. Visualisations

All charts are pre-generated and committed to the Images/ folder.

System Architecture

SHAP Feature Importance

The SHAP beeswarm confirms that collaborative signals (icf_als_interaction, svd_avg_interaction, als_dot) drive recommendations, while content signals (cb_score) play a minor supporting role.

Genre Distribution

Rating Distribution

The mean rating is 3.54 with a pronounced mode at 4.0 (8.4M ratings). Ratings below 2.5 are rare, confirming that users selectively rate movies they mostly liked (positivity bias).

Language Bias Analysis

The model's recommendations match the language distribution of ground truth (90.3% vs 90.4% English), confirming no additional language bias is introduced by the algorithm.

Error Analysis

Most 5-star movies that are missed by the top-20 list are low-popularity titles (log popularity ≤ 7). This is a known limitation of popularity-biased collaborative filtering.

Acknowledgements

GroupLens Research for the MovieLens 32M dataset
The Movie Database (TMDb) for movie metadata
HuggingFace sentence-transformers for all-mpnet-base-v2
Implicit for the ALS implementation
FAISS for efficient similarity search
LightGBM for the LambdaRank implementation
SHAP for model explainability

Name		Name	Last commit message	Last commit date
Images		Images
Notebooks		Notebooks
LICENSE		LICENSE
README.md		README.md

Latest commit

History

Folders and files

Repository files navigation

Hybrid Movie Recommendation System

Table of Contents

1. Project Overview

2. System Architecture

3. Dataset

MovieLens 32M

TMDb Metadata

4. Repository Structure

5. Notebook Walkthrough

NB 1 — Data Cleaning

NB 2 — Preprocessing

NB 3 — Feature Engineering

NB 4 — Collaborative Filtering

NB 5 — Content-Based Filtering

NB 6 — Hybrid Re-Ranker (Final Model)

NB 7 — Data Demographics

NB 8 — Model Analysis

6. Results & Metrics

Rating Prediction (SVD on test set)

Ranking (Hybrid Model on test set)

Intermediate model metrics (ALS, val set)

7. Key Design Decisions

8. Environment Setup

Dependencies

Hardware requirements

9. How to Run

10. Artifacts & Output Files

11. Visualisations

System Architecture

SHAP Feature Importance

Genre Distribution

Rating Distribution

Language Bias Analysis

Error Analysis

Acknowledgements

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages