Skip to content

Commit 3e65be0

Browse files
yogita-choudharyyogita-choudhary
committed
tags
1 parent f491c4f commit 3e65be0

89 files changed

Lines changed: 2222 additions & 91 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎assets/exercises.html‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3795,6 +3795,56 @@
37953795
of preferred outputs. This makes DPO simpler and more stable, while RLHF
37963796
is more flexible but computationally more complex.</p></li>
37973797
</ol></section></section>
3798+
<section><h2>Lecture32 - Vision-Language Models</h2><section><h1>Exercises</h1>
3799+
<ol>
3800+
<li><p>Why are VLMs modeled using left-to-right probability?
3801+
<strong>Answer:</strong> Because they extend LLMs to multimodal
3802+
settings. <strong>Explanation:</strong> VLMs follow the same
3803+
autoregressive principle as LLMs, where tokens are generated
3804+
sequentially. The difference is that the conditioning context includes
3805+
both previous text tokens and visual tokens extracted from
3806+
images.</p></li>
3807+
<li><p>What is the main difference between CLIP and generative VLMs?
3808+
<strong>Answer:</strong> CLIP is contrastive, while generative VLMs are
3809+
autoregressive. <strong>Explanation:</strong> CLIP learns aligned
3810+
embeddings for images and text using a contrastive loss, but does not
3811+
generate text. Generative VLMs explicitly model <span class="math inline">\(p(y \mid x)\)</span> and produce outputs token by
3812+
token.</p></li>
3813+
<li><p>Why is multimodal fusion difficult? <strong>Answer:</strong>
3814+
Because modalities are heterogeneous and often misaligned.
3815+
<strong>Explanation:</strong> Images and text differ in structure,
3816+
scale, and representation. Additionally, corresponding information may
3817+
be incomplete, noisy, or temporally misaligned, making it difficult to
3818+
learn a shared representation.</p></li>
3819+
<li><p>Why are ViT-based encoders preferred in modern VLMs?
3820+
<strong>Answer:</strong> Because they produce token-based
3821+
representations. <strong>Explanation:</strong> Vision Transformers
3822+
convert images into sequences of tokens, which align naturally with
3823+
Transformer-based language models. This simplifies multimodal fusion
3824+
compared to CNN-based feature maps.</p></li>
3825+
<li><p>What makes evaluation harder in VLMs than in LLMs?
3826+
<strong>Answer:</strong> Evaluation must consider both language quality
3827+
and visual grounding. <strong>Explanation:</strong> Unlike LLMs, VLMs
3828+
must ensure that outputs are not only fluent but also consistent with
3829+
the visual input. This introduces additional challenges such as
3830+
grounding, hallucination detection, and multimodal alignment.</p></li>
3831+
<li><p>What is the difference between early fusion and late fusion?
3832+
<strong>Answer:</strong> Early fusion combines modalities at the token
3833+
level, while late fusion combines them after separate encoding.
3834+
<strong>Explanation:</strong> In early fusion, visual and textual tokens
3835+
are processed together in a shared sequence model, enabling rich
3836+
cross-modal interactions from the start. In late fusion, each modality
3837+
is encoded separately and combined later through mechanisms such as
3838+
cross-attention, which improves modularity but may limit early
3839+
interaction.</p></li>
3840+
<li><p>What is the purpose of alignment tuning in VLMs?
3841+
<strong>Answer:</strong> To ensure outputs are helpful, safe, and
3842+
aligned with human preferences. <strong>Explanation:</strong> Alignment
3843+
tuning uses human feedback or preference data to refine model behavior
3844+
beyond instruction following. Methods such as RLHF and DPO help ensure
3845+
that responses are not only correct and grounded in visual input but
3846+
also appropriate and aligned with user expectations.</p></li>
3847+
</ol></section></section>
37983848
</main>
37993849
</body>
38003850
</html>

‎assets/exercises/Lecture32_Vision-Language-Models.html‎

Lines changed: 50 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,56 @@
2424
</head>
2525
<body>
2626
<main>
27-
<p>No exercises posted for this lecture yet.</p>
27+
<section><h1>Exercises</h1>
28+
<ol>
29+
<li><p>Why are VLMs modeled using left-to-right probability?
30+
<strong>Answer:</strong> Because they extend LLMs to multimodal
31+
settings. <strong>Explanation:</strong> VLMs follow the same
32+
autoregressive principle as LLMs, where tokens are generated
33+
sequentially. The difference is that the conditioning context includes
34+
both previous text tokens and visual tokens extracted from
35+
images.</p></li>
36+
<li><p>What is the main difference between CLIP and generative VLMs?
37+
<strong>Answer:</strong> CLIP is contrastive, while generative VLMs are
38+
autoregressive. <strong>Explanation:</strong> CLIP learns aligned
39+
embeddings for images and text using a contrastive loss, but does not
40+
generate text. Generative VLMs explicitly model <span class="math inline">\(p(y \mid x)\)</span> and produce outputs token by
41+
token.</p></li>
42+
<li><p>Why is multimodal fusion difficult? <strong>Answer:</strong>
43+
Because modalities are heterogeneous and often misaligned.
44+
<strong>Explanation:</strong> Images and text differ in structure,
45+
scale, and representation. Additionally, corresponding information may
46+
be incomplete, noisy, or temporally misaligned, making it difficult to
47+
learn a shared representation.</p></li>
48+
<li><p>Why are ViT-based encoders preferred in modern VLMs?
49+
<strong>Answer:</strong> Because they produce token-based
50+
representations. <strong>Explanation:</strong> Vision Transformers
51+
convert images into sequences of tokens, which align naturally with
52+
Transformer-based language models. This simplifies multimodal fusion
53+
compared to CNN-based feature maps.</p></li>
54+
<li><p>What makes evaluation harder in VLMs than in LLMs?
55+
<strong>Answer:</strong> Evaluation must consider both language quality
56+
and visual grounding. <strong>Explanation:</strong> Unlike LLMs, VLMs
57+
must ensure that outputs are not only fluent but also consistent with
58+
the visual input. This introduces additional challenges such as
59+
grounding, hallucination detection, and multimodal alignment.</p></li>
60+
<li><p>What is the difference between early fusion and late fusion?
61+
<strong>Answer:</strong> Early fusion combines modalities at the token
62+
level, while late fusion combines them after separate encoding.
63+
<strong>Explanation:</strong> In early fusion, visual and textual tokens
64+
are processed together in a shared sequence model, enabling rich
65+
cross-modal interactions from the start. In late fusion, each modality
66+
is encoded separately and combined later through mechanisms such as
67+
cross-attention, which improves modularity but may limit early
68+
interaction.</p></li>
69+
<li><p>What is the purpose of alignment tuning in VLMs?
70+
<strong>Answer:</strong> To ensure outputs are helpful, safe, and
71+
aligned with human preferences. <strong>Explanation:</strong> Alignment
72+
tuning uses human feedback or preference data to refine model behavior
73+
beyond instruction following. Methods such as RLHF and DPO help ensure
74+
that responses are not only correct and grounded in visual input but
75+
also appropriate and aligned with user expectations.</p></li>
76+
</ol></section>
2877
</main>
2978
</body>
3079
</html>

‎assets/jupyterlite/api/contents/all.json‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,11 @@
22
"content": [
33
{
44
"content": null,
5-
"created": "2026-04-26T04:55:20.710466Z",
5+
"created": "2026-05-01T08:40:00.919186Z",
66
"format": null,
77
"hash": null,
88
"hash_algorithm": null,
9-
"last_modified": "2026-04-26T04:55:20.710466Z",
9+
"last_modified": "2026-05-01T08:40:00.919186Z",
1010
"mimetype": null,
1111
"name": "notebooks",
1212
"path": "notebooks",
@@ -15,11 +15,11 @@
1515
"writable": true
1616
}
1717
],
18-
"created": "2026-04-26T04:55:20.697292Z",
18+
"created": "2026-05-01T08:40:00.903185Z",
1919
"format": "json",
2020
"hash": null,
2121
"hash_algorithm": null,
22-
"last_modified": "2026-04-26T04:55:20.697292Z",
22+
"last_modified": "2026-05-01T08:40:00.903185Z",
2323
"mimetype": null,
2424
"name": "",
2525
"path": "",

0 commit comments

Comments
 (0)