Convergence criteria differ from GSI and JEDI

[SamuelDegelia-NOAA]

Description

Creating an issue to track our discussion/investigation on this point.

We currently set the JEDI convergence criterion to gradient norm reduction: 1e-3 which may be too loose. Most of our 3DVar runs stop before reaching 50 inner loops, although they still appear to converge in most cases (see this slide from @delippi).

We should either tighten this threshold so the solver uses the full 50 inner loops, or set it to a value consistent with GSI.

[delippi]

Thanks for making this @SamuelDegelia-NOAA. I have just a few thoughts I wanted to write down.

I guess the main question is how much does the convergence criteria make a difference in the case where we break out of the minimization as a result of this threshold? For example, if each outer loop has ~20 iterations because it reaches gnorm threshold of 1e-3, do we even gain anything from tightening the threshold to 1e-10? I don't think we gain much other than increased run times. My guess is the gnorm reduction threshold doesn't matter for full suite of obs because we are likely hitting the declared max number of inner loop iterations (50) and not breaking out due to convergence. I believe most of our tests so far use less than 50 because we're mostly testing small subsets of obs.

A second question is what would be the equivalent for JEDI? Is 50 iterations in JEDI the same as 50 iterations in GSI? Probably.