Co-Clustering for Visual Category Recognition

Research implementation of Information-Theoretic Co-Clustering applied to visual object category recognition. Co-clusters the image–feature matrix to discover semantically related groups of visual words and images simultaneously, yielding structured feature representations that outperform standard Bag-of-Features on VOC and Caltech benchmarks.

📄 Full paper: visual category recognition using information-theoretic co-clustering.pdf

---

Key Idea

Standard Bag-of-Features (BOF) represents each image as an unstructured histogram of visual words. This ignores relationships between visual words.

Co-clustering partitions both images (rows) and visual words (columns) simultaneously:

Standard k-means:                  Co-clustering (biclustering):

Images → [clusters]                Images   × Visual Words
                                       ↓              ↓
                                   [r image    × [c word
                                    clusters]    clusters]

The key insight: visual words that co-occur in similar images should be grouped together. This structure, discovered by information-theoretic (Bregman) co-clustering, is then used to build a more discriminative feature representation.

---

Pipeline

flowchart LR
    subgraph Extract["Feature Extraction (ccsift/)"]
        IMG["Images"] --> SIFT["Dense SIFT\n128-d descriptors"]
        SIFT --> CB["Codebook\nK-means, K=1000"]
        CB --> BOF["BOF histogram\nper image"]
    end

    subgraph Topic["Topic Building (bagoftopics/)"]
        BOF --> IWM["Image–Word Matrix\n(N × K)"]
        IWM --> CC["Co-clustering\nBregman / Spectral Bi"]
        CC --> TOPICS["Topic Dictionary\n(word clusters)"]
    end

    subgraph Classify["Classification (cocluster/)"]
        TOPICS --> FEAT["Co-clustered\nfeature vector"]
        FEAT --> SVM["RBF SVM\nor KNN"]
        SVM --> ACC["F-score / mAP"]
    end

    subgraph Subspace["Subspace Variant (cocluster/subspace_cocluster.py)"]
        BOF --> DICT["Sparse Dictionary\n(K atoms)"]
        DICT --> COEFF["Sparse Coefficients\nLasso / OMP"]
        COEFF --> CC2["Co-cluster Coefficients\nSpectral Biclustering"]
        CC2 --> BLOCK["Block-aggregated\nfeatures (r × c)"]
        BLOCK --> SVM
    end

Loading

---

Two Approaches

Approach	Input to co-clustering	Output feature	Python	MATLAB
Word co-clustering	Image × visual-word matrix	Topic-cluster histogram	cocluster/ccWord.py, ccTopic.py	calcCoClustTopic.m
Subspace co-clustering	Image × dictionary-atom matrix (sparse codes)	Block-aggregated activations	cocluster/subspace_cocluster.py ★	calcCoClustSubspace.m

★ New Python port — replaces MATLAB-only implementation

---

Repository Layout

co-clustering-visual-categorization/
├── ccsift/
│   └── ccsiftVOC06.py              # SIFT extraction + BOF encoding for VOC2006
├── ccbof/
│   └── ccbofVOC06.py               # Baseline BOF classification (no co-clustering)
├── bagoftopics/                    # Dictionary and topic building
│   ├── universalWordDictionary.py  # K-means codebook from all SIFT features
│   ├── universalTopicDictionary.py # Topic dictionary from co-clustering
│   ├── imageWord.py                # Image-to-word-histogram mapping
│   ├── uniWord2Topic.py            # Word → topic aggregation
│   └── ...
├── cocluster/                      # Core co-clustering and classification
│   ├── subspace_cocluster.py       # ★ NEW: Python port of 4 MATLAB files
│   ├── cocluster.py                # Co-clustering wrapper
│   ├── ccWord.py                   # Co-cluster on word features
│   ├── ccTopic.py                  # Co-cluster on topic features
│   ├── classify.py                 # SVM / KNN classifier
│   ├── coclusterTopic.py           # Experiment runner
│   ├── plotresult.py               # Result visualisation
│   └── VOC2006/, Scene15/, ...     # Result PNGs
├── matlab/                         # Original MATLAB implementation
│   ├── calcCoClustTopic.m          # Topic co-clustering
│   ├── calcCoClustSubspace.m       # Subspace co-clustering (now ported)
│   ├── coclustsubspaceCoeff.m      # Coefficient encoding (now ported)
│   ├── coclustSubspaceDictionary.m # Dictionary from subspace (now ported)
│   ├── coclustSubspaceDict.m       # Dictionary variant (now ported)
│   └── ccssprojcoeff.m             # Projection step (now ported)
├── paper/
│   ├── visual category recognition using information-theoretic co-clustering.pdf
│   └── visapp2012ppt.pdf           # Conference presentation slides
└── requirements.txt

---

Quick Start

pip install -r requirements.txt

Word co-clustering

from cocluster.cocluster import CoCluster
from cocluster.classify import classify_svm

# Load pre-computed BOF histogram matrix (N_images × N_words)
import numpy as np
X = np.load("bof_matrix.npy")
y = np.load("labels.npy")

# Co-cluster: 20 image clusters × 50 word clusters
cc = CoCluster(n_row_clusters=20, n_col_clusters=50)
X_coclust = cc.fit_transform(X)

# Classify
scores = classify_svm(X_coclust, y, n_folds=10)
print(f"Mean F-score: {scores.mean():.3f}")

Subspace co-clustering (new Python port)

from cocluster.subspace_cocluster import CoClusterSubspace

cc = CoClusterSubspace(
    n_components=128,       # dictionary atoms
    n_row_clusters=20,      # image clusters
    n_col_clusters=50,      # atom clusters
    alpha=0.1,              # Lasso regularisation
)

X_train, X_test = ...       # BOF feature matrices
cc.fit(X_train)
Z_train = cc.transform(X_train)   # (N_train, 20 × 50) block features
Z_test  = cc.transform(X_test)

Or from the command line:

python cocluster/subspace_cocluster.py features.txt \
    --n-components 128 --n-row-clusters 20 --n-col-clusters 50

---

Datasets

Dataset	Classes	Size	Download
Pascal VOC 2006	10	~5K images	VOC Challenge
Pascal VOC 2007	20	~10K images	VOC Challenge
Pascal VOC 2010	20	~20K images	VOC Challenge
Caltech-101	101	~9K images	Caltech Vision Lab
Caltech-256	256	~30K images	Caltech Vision Lab
Scene-15	15	~4.5K images	Scene Understanding

Path configuration: All scripts originally used /vol/vssp/diplecs/ash/Data/ (university server). Update rootDir at the top of each script to point to your local data directory.

---

MATLAB vs Python

MATLAB file	Python equivalent	Status
calcCoClustSubspace.m	cocluster/subspace_cocluster.py → CoClusterSubspace.fit_cocluster()	✅ Ported
coclustsubspaceCoeff.m	cocluster/subspace_cocluster.py → CoClusterSubspace.encode()	✅ Ported
coclustSubspaceDictionary.m	cocluster/subspace_cocluster.py → CoClusterSubspace.build_dictionary()	✅ Ported
coclustSubspaceDict.m	cocluster/subspace_cocluster.py → CoClusterSubspace.project()	✅ Ported
ccssprojcoeff.m	cocluster/subspace_cocluster.py → CoClusterSubspace.project()	✅ Ported
calcCoClustTopic.m	cocluster/ccTopic.py	Already in Python
callCalcCCSSClassPerf.m	cocluster/classify.py	Already in Python

Co-clustering engine: Original MATLAB/Python called a compiled cocluster-linux binary (Bregman co-clustering, Dhillon et al. 2003). The Python port uses sklearn.cluster.SpectralBiclustering which provides equivalent row+column joint clustering with SVD-based spectral methods.

---

References

• Dhillon, I., Mallela, S., Modha, D. (2003). Information-theoretic co-clustering. KDD.
• Paper: paper/visual category recognition using information-theoretic co-clustering.pdf
• Conference slides: paper/visapp2012ppt.pdf

---

License

MIT — see LICENSE.