Skip to content

Commit 8f85e16

Browse files
SrameoSrameo
committed
[ADD] Let's prepare for MIPI@2024
1 parent e052f27 commit 8f85e16

28 files changed

Lines changed: 1758 additions & 3 deletions

‎.assets/logo-mipi.svg‎

Lines changed: 3 additions & 0 deletions
Loading

‎README.md‎

Lines changed: 29 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
1+
# <div align="center"> Let's Prepare for <a href="https://mipi-challenge.org/MIPI2024/">MIPI@2024</a>! \[<a href="/tools/mipi_starting_kit">Starting-Kit</a>\] </div>
2+
13
<p align="center">
2-
<img src='.assets/logo.svg' alt='ICCV23_LED_LOGO' width='200px'/><br/>
4+
<img src='.assets/logo-mipi.svg' alt='ICCV23_LED_LOGO' width='200px'/><br/>
35
</p>
46

57
## <div align="center"><a href="https://srameo.github.io/projects/led-extension/">Homepage</a> | <a href="https://arxiv.org/abs/2308.03448v2">Paper</a> | <a href="https://drive.google.com/drive/folders/11MYkjzbPIZ7mJbu9vrgaVC-OwGcOFKsM?usp=sharing">Google Drive</a> | <a href="https://pan.baidu.com/s/17rA_8GvfNPZJY5Zl9dyILw?pwd=iay5">Baidu Cloud</a> | <a href="https://zhuanlan.zhihu.com/p/648242095">知乎</a> | <a href="https://github.com/Srameo/LED/files/12733867/iccv23_poster.pdf">Poster</a> | <a href="https://srameo.github.io/projects/led-iccv23/assets/slides/iccv23_slides_en.pdf">Slides</a> | <a href="https://youtu.be/Jo8OTAnUYkU">Video</a> </div>
@@ -55,15 +57,16 @@ Achieveing <b style='font-size: large'>SOTA performance</b> in <b style='font-si
5557
> Future work can be found in [todo.md](docs/todo.md).
5658
5759
<ul>
60+
<li><b>Jan 13, 2024</b>: Release the <a href="/tools/mipi_starting_kit">starting-kit</a> for <a href="https://mipi-challenge.org/MIPI2024/">MIPI@2024</a>. Additionally, we release the pre-trained parameters of Restormer and NAFNet.</li>
5861
<li><b>Dec 27, 2023</b>: Update an extension version of our ICCV 23 paper (<a href="https://srameo.github.io/projects/led-extension/">Project Page</a>/<a href="https://arxiv.org/abs/2308.03448v2">Paper</a>).</li>
5962
<li><b>Dec 1-5, 2023</b>: Add the related code/doc[<a href='https://github.com/Srameo/LED/blob/main/docs/calib_en.md'>EN</a>/<a href='https://github.com/Srameo/LED/blob/main/docs/calib_cn.md'>CN</a>] from <a href="https://github.com/Srameo/LED/pull/14">PR#14</a>/<a href="https://github.com/Srameo/LED/pull/16">PR#16</a>, thanks to @<a href="https://github.com/HYX20011209">HYX20011209</a></li>
6063
<li><b>Sep 27, 2023</b>: Add the urls to our <a href="https://github.com/Srameo/LED/files/12733867/iccv23_poster.pdf">Poster</a>, <a href="https://srameo.github.io/projects/led-iccv23/assets/slides/iccv23_slides_en.pdf">Slides</a>, and <a href="https://youtu.be/Jo8OTAnUYkU">Video</a>.</li>
6164
<li><b>Aug 19, 2023</b>: Release relevent files on <a href="https://pan.baidu.com/s/17rA_8GvfNPZJY5Zl9dyILw?pwd=iay5">Baidu Clould</a>(pwd: iay5).</li>
62-
<li><b>Aug 15, 2023</b>: For faster benchmark, we released the relevant files in commit <a href="https://github.com/Srameo/LED/commit/fadffc7282b02ab2fcc7fbade65f87217b642588"><code>fadffc7</code></a>.</li>
6365
</ul>
6466
<details>
6567
<summary>History</summary>
6668
<ul>
69+
<li><b>Aug 15, 2023</b>: For faster benchmark, we released the relevant files in commit <a href="https://github.com/Srameo/LED/commit/fadffc7282b02ab2fcc7fbade65f87217b642588"><code>fadffc7</code></a>.</li>
6770
<li><b>Aug, 2023</b>: We released a Chinese explanation of our paper on <a href="https://zhuanlan.zhihu.com/p/648242095">知乎</a>.</li>
6871
<li><b>Aug, 2023</b>: Our code is publicly available!</li>
6972
<li><b>July, 2023</b>: Our paper "Lighting Every Darkness in Two Pairs: A Calibration-Free Pipeline for RAW Denoising" has been accepted by ICCV 2023.</li>
@@ -146,6 +149,30 @@ Or you can just use the following pretrained LED module for custumizing on your
146149
<th> [<a href="https://drive.google.com/file/d/1IzOkJuHWQVXmkzFJzQ9-gkPXBlrutO2p/view?usp=drive_link">Google Drive</a>] </th>
147150
<th> [<a href="/options/LED/pretrain/CVPR20_ELD_Setting_Ratio1-200.yaml">options/LED/pretrain/CVPR20_ELD_Setting_Ratio1-200.yaml</a>] </th>
148151
</tr>
152+
<tr>
153+
<td>LED</td>
154+
<th> ELD (5 Virtual Cameras) </th>
155+
<th> Pretrain </th>
156+
<th> Restormer </th>
157+
<th> ELD </th>
158+
<th> 100-300 </th>
159+
<th> - </th>
160+
<th> - </th>
161+
<th> [<a href="https://drive.google.com/file/d/1iKNLaNRH5UejstaZbuq83yAdYxLaPa4x/view?usp=drive_link">Google Drive</a>] </th>
162+
<th> [<a href="/options/LED/other_arch/Restormer/LED+Restormer_Pretrain.yaml">options/LED/other_arch/Restormer/LED+Restormer_Pretrain.yaml</a>] </th>
163+
</tr>
164+
<tr>
165+
<td>LED</td>
166+
<th> ELD (5 Virtual Cameras) </th>
167+
<th> Pretrain </th>
168+
<th> NAFNet </th>
169+
<th> ELD </th>
170+
<th> 100-300 </th>
171+
<th> - </th>
172+
<th> - </th>
173+
<th> [<a href="https://drive.google.com/file/d/1FmqGv_YICLX4Gc-aWzvcTl8aWgeJ5cEB/view?usp=drive_link">Google Drive</a>] </th>
174+
<th> [<a href="/options/LED/other_arch/NAFNet/LED+NAFNet_Pretrain.yaml">options/LED/other_arch/NAFNet/LED+NAFNet_Pretrain.yaml</a>] </th>
175+
</tr>
149176
</table>
150177

151178
## :camera: Quick Demo

‎VERSION‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
0.0.1
1+
0.1.1

‎docs/pretrained-models.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,30 @@ Or you can just use the following pretrained LED module for custumizing on your
6565
<th> [<a href="https://drive.google.com/file/d/1IzOkJuHWQVXmkzFJzQ9-gkPXBlrutO2p/view?usp=drive_link">Google Drive</a>] </th>
6666
<th> [<a href="/options/LED/pretrain/CVPR20_ELD_Setting_Ratio1-200.yaml">options/LED/pretrain/CVPR20_ELD_Setting_Ratio1-200.yaml</a>] </th>
6767
</tr>
68+
<tr>
69+
<td>LED</td>
70+
<th> ELD (5 Virtual Cameras) </th>
71+
<th> Pretrain </th>
72+
<th> Restormer </th>
73+
<th> ELD </th>
74+
<th> 100-300 </th>
75+
<th> - </th>
76+
<th> - </th>
77+
<th> [<a href="https://drive.google.com/file/d/1iKNLaNRH5UejstaZbuq83yAdYxLaPa4x/view?usp=drive_link">Google Drive</a>] </th>
78+
<th> [<a href="/options/LED/other_arch/Restormer/LED+Restormer_Pretrain.yaml">options/LED/other_arch/Restormer/LED+Restormer_Pretrain.yaml</a>] </th>
79+
</tr>
80+
<tr>
81+
<td>LED</td>
82+
<th> ELD (5 Virtual Cameras) </th>
83+
<th> Pretrain </th>
84+
<th> NAFNet </th>
85+
<th> ELD </th>
86+
<th> 100-300 </th>
87+
<th> - </th>
88+
<th> - </th>
89+
<th> [<a href="https://drive.google.com/file/d/1FmqGv_YICLX4Gc-aWzvcTl8aWgeJ5cEB/view?usp=drive_link">Google Drive</a>] </th>
90+
<th> [<a href="/options/LED/other_arch/NAFNet/LED+NAFNet_Pretrain.yaml">options/LED/other_arch/NAFNet/LED+NAFNet_Pretrain.yaml</a>] </th>
91+
</tr>
6892
</table>
6993

7094
## Network for Benchmark
@@ -151,6 +175,30 @@ Or you can just use the following pretrained LED module for custumizing on your
151175
<th> [<a href="https://drive.google.com/file/d/1vmeJBdXSjecnbTXLrMHOIhLgJbMFRaA1/view?usp=drive_link">Google Drive</a>] </th>
152176
<th> [<a href="/options/LED/finetune/SID_SonyA7S2_CVPR20_ELD_Setting.yaml">options/LED/finetune/SID_SonyA7S2_CVPR20_ELD_Setting.yaml</a>] </th>
153177
</tr>
178+
<tr>
179+
<td>LED</td>
180+
<th> Real Noise (6 Pairs on SID SonyA7S2) </th>
181+
<th> Finetune / Deploy </th>
182+
<th> Restormer </th>
183+
<th> ELD* </th>
184+
<th> 100-300 </th>
185+
<th> SonyA7S2 </th>
186+
<th> SID SonyA7S2 </th>
187+
<th> [<a href="https://drive.google.com/file/d/1KqY15BeXwjlwXGU5mEywBv3A2nbso7UL/view?usp=drive_link">Google Drive</a>] </th>
188+
<th> [<a href="/options/LED/other_arch/Restormer/LED+Restormer_Finetune.yaml">options/LED/other_arch/Restormer/LED+Restormer_Finetune.yaml</a>] </th>
189+
</tr>
190+
<tr>
191+
<td>LED</td>
192+
<th> Real Noise (6 Pairs on SID SonyA7S2) </th>
193+
<th> Finetune / Deploy </th>
194+
<th> NAFNet </th>
195+
<th> ELD* </th>
196+
<th> 100-300 </th>
197+
<th> SonyA7S2 </th>
198+
<th> SID SonyA7S2 </th>
199+
<th> [<a href="https://drive.google.com/file/d/11bgv3cD02ea0SU8mkL_W1UCNaH_FDbSS/view?usp=drive_link">Google Drive</a>] </th>
200+
<th> [<a href="/options/LED/other_arch/NAFNet/LED+NAFNet_Finetune.yaml">options/LED/other_arch/NAFNet/LED+NAFNet_Finetune.yaml</a>] </th>
201+
</tr>
154202
<tr>
155203
<td>LED</td>
156204
<th> Real Noise (24 Pairs on ELD SonyA7S2) </th>

‎led/archs/nafnet_arch.py‎

Lines changed: 199 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,199 @@
1+
import torch
2+
import torch.nn as nn
3+
import torch.nn.functional as F
4+
from led.utils.registry import ARCH_REGISTRY
5+
6+
class LayerNormFunction(torch.autograd.Function):
7+
@staticmethod
8+
def forward(ctx, x, weight, bias, eps):
9+
ctx.eps = eps
10+
N, C, H, W = x.size()
11+
mu = x.mean(1, keepdim=True)
12+
var = (x - mu).pow(2).mean(1, keepdim=True)
13+
y = (x - mu) / (var + eps).sqrt()
14+
ctx.save_for_backward(y, var, weight)
15+
y = weight.view(1, C, 1, 1) * y + bias.view(1, C, 1, 1)
16+
return y
17+
18+
@staticmethod
19+
def backward(ctx, grad_output):
20+
eps = ctx.eps
21+
22+
N, C, H, W = grad_output.size()
23+
y, var, weight = ctx.saved_variables
24+
g = grad_output * weight.view(1, C, 1, 1)
25+
mean_g = g.mean(dim=1, keepdim=True)
26+
27+
mean_gy = (g * y).mean(dim=1, keepdim=True)
28+
gx = 1. / torch.sqrt(var + eps) * (g - y * mean_gy - mean_g)
29+
return gx, (grad_output * y).sum(dim=3).sum(dim=2).sum(dim=0), grad_output.sum(dim=3).sum(dim=2).sum(
30+
dim=0), None
31+
32+
class LayerNorm2d(nn.Module):
33+
def __init__(self, channels, eps=1e-6):
34+
super(LayerNorm2d, self).__init__()
35+
self.register_parameter('weight', nn.Parameter(torch.ones(channels)))
36+
self.register_parameter('bias', nn.Parameter(torch.zeros(channels)))
37+
self.eps = eps
38+
39+
def forward(self, x):
40+
return LayerNormFunction.apply(x, self.weight, self.bias, self.eps)
41+
42+
class SimpleGate(nn.Module):
43+
def forward(self, x):
44+
x1, x2 = x.chunk(2, dim=1)
45+
return x1 * x2
46+
47+
class NAFBlock(nn.Module):
48+
def __init__(self, c, DW_Expand=2, FFN_Expand=2, drop_out_rate=0.):
49+
super().__init__()
50+
dw_channel = c * DW_Expand
51+
self.conv1 = nn.Conv2d(in_channels=c, out_channels=dw_channel, kernel_size=1, padding=0, stride=1, groups=1, bias=True)
52+
self.dwconv2 = nn.Conv2d(in_channels=dw_channel, out_channels=dw_channel, kernel_size=3, padding=1, stride=1, groups=dw_channel,
53+
bias=True)
54+
self.conv3 = nn.Conv2d(in_channels=dw_channel // 2, out_channels=c, kernel_size=1, padding=0, stride=1, groups=1, bias=True)
55+
56+
# Simplified Channel Attention
57+
self.sca = nn.Sequential(
58+
nn.AdaptiveAvgPool2d(1),
59+
nn.Conv2d(in_channels=dw_channel // 2, out_channels=dw_channel // 2, kernel_size=1, padding=0, stride=1,
60+
groups=1, bias=True),
61+
)
62+
63+
# SimpleGate
64+
self.sg = SimpleGate()
65+
66+
ffn_channel = FFN_Expand * c
67+
self.conv4 = nn.Conv2d(in_channels=c, out_channels=ffn_channel, kernel_size=1, padding=0, stride=1, groups=1, bias=True)
68+
self.conv5 = nn.Conv2d(in_channels=ffn_channel // 2, out_channels=c, kernel_size=1, padding=0, stride=1, groups=1, bias=True)
69+
70+
self.norm1 = LayerNorm2d(c)
71+
self.norm2 = LayerNorm2d(c)
72+
73+
self.dropout1 = nn.Dropout(drop_out_rate) if drop_out_rate > 0. else nn.Identity()
74+
self.dropout2 = nn.Dropout(drop_out_rate) if drop_out_rate > 0. else nn.Identity()
75+
76+
self.beta = nn.Parameter(torch.zeros((1, c, 1, 1)), requires_grad=True)
77+
self.gamma = nn.Parameter(torch.zeros((1, c, 1, 1)), requires_grad=True)
78+
79+
def forward(self, inp):
80+
x = inp
81+
82+
x = self.norm1(x)
83+
84+
x = self.conv1(x)
85+
x = self.dwconv2(x)
86+
x = self.sg(x)
87+
x = x * self.sca(x)
88+
x = self.conv3(x)
89+
90+
x = self.dropout1(x)
91+
92+
y = inp + x * self.beta
93+
94+
x = self.conv4(self.norm2(y))
95+
x = self.sg(x)
96+
x = self.conv5(x)
97+
98+
x = self.dropout2(x)
99+
100+
return y + x * self.gamma
101+
102+
103+
class DownSample(nn.Module):
104+
def __init__(self, chan) -> None:
105+
super().__init__()
106+
self.downconv = nn.Conv2d(chan, 2*chan, 2, 2)
107+
108+
def forward(self, x):
109+
return self.downconv(x)
110+
111+
112+
class UpSample(nn.Module):
113+
def __init__(self, chan) -> None:
114+
super().__init__()
115+
self.upconv = nn.Conv2d(chan, chan * 2, 1, bias=False)
116+
self.pixel_shuffle = nn.PixelShuffle(2)
117+
118+
def forward(self, x):
119+
return self.pixel_shuffle(self.upconv(x))
120+
121+
122+
123+
@ARCH_REGISTRY.register()
124+
class NAFNet(nn.Module):
125+
def __init__(self, img_channel=3, width=16, middle_blk_num=1, enc_blk_nums=[], dec_blk_nums=[]):
126+
super().__init__()
127+
128+
self.intro = nn.Conv2d(in_channels=img_channel, out_channels=width, kernel_size=3, padding=1, stride=1, groups=1,
129+
bias=True)
130+
self.ending = nn.Conv2d(in_channels=width, out_channels=img_channel, kernel_size=3, padding=1, stride=1, groups=1,
131+
bias=True)
132+
133+
self.encoders = nn.ModuleList()
134+
self.decoders = nn.ModuleList()
135+
self.middle_blks = nn.ModuleList()
136+
self.ups = nn.ModuleList()
137+
self.downs = nn.ModuleList()
138+
139+
chan = width
140+
for num in enc_blk_nums:
141+
self.encoders.append(
142+
nn.Sequential(
143+
*[NAFBlock(chan) for _ in range(num)]
144+
)
145+
)
146+
self.downs.append(
147+
DownSample(chan)
148+
)
149+
chan = chan * 2
150+
151+
self.middle_blks = \
152+
nn.Sequential(
153+
*[NAFBlock(chan) for _ in range(middle_blk_num)]
154+
)
155+
156+
for num in dec_blk_nums:
157+
self.ups.append(
158+
UpSample(chan)
159+
)
160+
chan = chan // 2
161+
self.decoders.append(
162+
nn.Sequential(
163+
*[NAFBlock(chan) for _ in range(num)]
164+
)
165+
)
166+
167+
self.padder_size = 2 ** len(self.encoders)
168+
169+
def forward(self, inp):
170+
B, C, H, W = inp.shape
171+
inp = self.check_image_size(inp)
172+
173+
x = self.intro(inp)
174+
175+
encs = []
176+
177+
for encoder, down in zip(self.encoders, self.downs):
178+
x = encoder(x)
179+
encs.append(x)
180+
x = down(x)
181+
182+
x = self.middle_blks(x)
183+
184+
for decoder, up, enc_skip in zip(self.decoders, self.ups, encs[::-1]):
185+
x = up(x)
186+
x = x + enc_skip
187+
x = decoder(x)
188+
189+
x = self.ending(x)
190+
x = x + inp
191+
192+
return x[:, :, :H, :W]
193+
194+
def check_image_size(self, x):
195+
_, _, h, w = x.size()
196+
mod_pad_h = (self.padder_size - h % self.padder_size) % self.padder_size
197+
mod_pad_w = (self.padder_size - w % self.padder_size) % self.padder_size
198+
x = F.pad(x, (0, mod_pad_w, 0, mod_pad_h))
199+
return x

0 commit comments

Comments
 (0)