How to avoid weight close to zero?

[shanshuo]

Thanks for your elegant code. I've got a question about the output weight. Though sigmoid function could output a non-zero vector.

Meta-Weight-Net_Code-Optimization/model.py

Line 108 in 1d3cab0

return torch.sigmoid(x)

But the weight vector's elements could be all close to zero, leading to a small loss.

Meta-Weight-Net_Code-Optimization/main.py

Line 129 in 1d3cab0

loss = torch.mean(weight * loss_vector_reshape)

How could we avoid this?