Question About Target Computation in lbm_model.py

[whikwon]

LBM/src/lbm/models/lbm/lbm_model.py

Line 168 in b41e144

target = z_source - z

Could you kindly explain why this form was not used target = (z - noisy_sample) / (1 - t) for the target computation? I would greatly appreciate any insight you can provide.

[Qiuhao-Wu]

分母可除可不除 模型有这个能力 ， z_source - z 实际上是与下面对上了，因为网络实际上预测的是 - v

[whikwon]

Thank you for your explanation.

Regarding why we predict z_0 - z_1 instead of z_1 - z_t: may I interpret this as substituting equation (5) from the paper in place of z_t and then simply dropping the noise term—since the network is ultimately learning the expectation anyway?

I have one more question about the sampling code

LBM/src/lbm/models/lbm/lbm_model.py

Lines 441 to 447 in b41e144

	sigmas = self._get_sigmas(
	self.sampling_noise_scheduler, timestep, n_dim=4, device=z.device
	)
	sample = sample + self.bridge_noise_sigma * (
	sigmas * (1.0 - sigmas)
	) ** 0.5 * torch.randn_like(sample)
	sample = sample.to(z.dtype)

At the very end, the factor \sqrt{\Delta t} that normally multiplies dW appears to be missing. Is there a specific reason for omitting it? I noticed that in the stochastic-interpolant implementation (https://github.com/malbergo/stochastic-interpolants/blob/c9bc956ec5a51c907bcbfb114f96d3539acb70bb/interflow/stochastic_interpolant.py#L324-L327)
torch.sqrt(self.dt) is likewise excluded.

[ZhouQianang]

@whikwon Hi, I have the same question here. Do you understand now? I am still confused.

@Qiuhao-Wu Hi, I thought the question was that, at every timestamp t, the sample on the trajectory is z_t = t*z_0 + (1-t)*z_1 + w*z_noise, so the correct velocity that the network should predict is v=z_1-z_t, but now the code is v=z_1-z_0, there is a little difference caused by z_noise, I don't know why, maybe because the self.bridge_noise_sigma is small enough.

And, if I need a higher-level noise (bigger self.bridge_noise_sigma), should I use v=z_1-z_t?

[whikwon]

@ZhouQianang Hi, reading the paper (https://arxiv.org/abs/2405.05953) helped me gain a deeper mathematical understanding. In the LBM code, looking at the pixel_loss (rather than the latent_loss) suggests that it’s learning z_1 - z_t.

As for sampling, it appears to approximate the equations from the paper, and I suspect that because it’s only trained on a small number of steps, it still works without issue.

[ZhouQianang]

@whikwon Thank you for your patient explanation. I think you are right, the drift and diffusion terms are just computed separately, and both of them supervise the prediction.

[ZhouQianang]

@whikwon Hi, are you still working on this project? After some attempts, I have encountered a few issues and would greatly appreciate the opportunity to discuss them with you:

1. Noise may not be necessary? According to the results in the paper, appropriate noise only slightly increases the FID.
2. LBM seems unable to handle high-weight noise. In my flow matching model, excessive noise leads to poor results, where the generated images/videos are not only inconsistent with the input but also deviate from natural images.

[whikwon]

@ZhouQianang
In my case, I am addressing an interpolation problem in medical imaging. I found that the Brownian bridge (LBM, https://arxiv.org/abs/2405.05953) did not provide substantial benefits, and therefore I shifted to a Flow Matching framework.

When you refer to noise, are you specifically pointing to the Wiener process in the Brownian bridge setting? May I also ask which domain of data you are focusing on in your experiments?

[yoon28]

Two things are related to the equation target = z_source - z.
First, the LBM's objective is E{(z_1 - z_t)/(1-t) - v(z_t,t)}^2 then the network tried to learn (z_1 - z_t)/(1-t), the target equation.
Now, subtitute z_t with eq 5 in the paper and simplify it, then you get z_1 - z_0 - b / (1-t), where b= sigma * sqrt(t(1-t)) * eps.
Second, the code's noise and time schedulers are reversed, meaning that the t in eq 5 changes from 1 to 0.
So we need to change the target equation by changing t with k=1-t, which is z_0 - z_1 - b / k, since in the sampling step its time schedule is reversed as well, see the code sample = self.sampling_noise_scheduler.step( in the sample method of lbm_model.py.
And b / k term is became 0 because of the expectation operation.