Median nightly delays

[aewallwi]

Tools to better ensure that redcal degeneracies are relatively constant in time. Depends on #772. Depends on #798 #815 . Coverage gaps arise from gaps that will be addressed in some shape or form in #798

[codecov]

Codecov Report

Merging #795 (beb3c23) into main (d974d1a) will increase coverage by 0.03%.
The diff coverage is 100.00%.

❗ Current head beb3c23 differs from pull request most recent head 77a85fe. Consider uploading reports for the commit 77a85fe to get more accurate results

@@            Coverage Diff             @@
##             main     #795      +/-   ##
==========================================
+ Coverage   97.04%   97.07%   +0.03%     
==========================================
  Files          18       18              
  Lines        8294     8389      +95     
==========================================
+ Hits         8049     8144      +95     
  Misses        245      245              

Impacted Files	Coverage Δ
hera_cal/frf.py	97.98% <ø> (ø)
hera_cal/reflections.py	92.68% <ø> (ø)
hera_cal/apply_cal.py	94.54% <100.00%> (ø)
hera_cal/io.py	98.16% <100.00%> (+<0.01%)	⬆️
hera_cal/redcal.py	99.30% <100.00%> (+0.06%)	⬆️
hera_cal/smooth_cal.py	96.66% <100.00%> (+0.01%)	⬆️
hera_cal/vis_clean.py	97.76% <100.00%> (+0.01%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d974d1a...77a85fe. Read the comment docs.

[aewallwi]

The problem that I am trying to fix arises from transient misidentified antenna polarities. I found that using remove_degen_gains with a median delay and polarity over the night does not seem to fix the problem. What does seem to fix the problem is just mulitplying each antenna gain by (-1) ** abs(median_polarity - per_time_polarity). Here are some examples:

Original phases

Using remove_degen_gains where degen_gains is set to the median delay and polarity.

Here is just multiplying by (-1) ** abs(median_polarity - per_time_polarity)

And finally, here is using remove_degen_gains only to fix delays and then using a separate step of (-1) ** abs(median_polarity - per_time_polarity) to fix the polarities

[jsdillon]

I'm surprised that last step does literally nothing... Have you checked multiple antennas? What's your reference antenna?

We saw jumps like these in H1C IDR2 before there was ever any polarity flipping, e.g. figure 6 of https://arxiv.org/pdf/2108.02263.pdf

[aewallwi]

I'm surprised that last step does literally nothing... Have you checked multiple antennas? What's your reference antenna?

We saw jumps like these in H1C IDR2 before there was ever any polarity flipping, e.g. figure 6 of https://arxiv.org/pdf/2108.02263.pdf

The last plot had a bug. This is actually what happens when you try to fix the delays without paying attention to polarity flips and then flip polarities as a separate step.

[aewallwi]

I've updated this PR so that the default recipe for updating first-cal is to both switch the polarities and to update the offsets and delays in the degenerate subspace. I found that the polarity flips fixes the polarity misidentification. Updating the degeneracies, while it has minimal impact on the 2459140 data is functionality that might be desirable in the future to guarantee that we have constant first-cal parameters.

[jsdillon]

Looking pretty good. Just a few minor tweaks.

Also, not that the PR is almost done, can you add some detail to the PR description for posterity's sake and show some plots of this procedure in action on real data?

[jsdillon]

Please document the other kwargs, even though they are pretty obvious.

[jsdillon]

write them out with that median. is unclear. What you're actually doing is making new redcal meta files with median-ed info.

Also, this function is misleadingly named, since it also edits offsets (missing from the docstring) and polarities. Perhaps just call it median_nightly_firstcal_metadata or similar?

[jsdillon]

I would have used a list comprehension

[jsdillon]

If all the polarity flips are np.nan for a given antenna, this will turn into True though I think you want it to be False by default. Probably doesn't matter but you might want an a check on not np.any(np.isfinite())

[jsdillon]

I think these comments got misplaced

[jsdillon]

If you have this option, it's weird that old_redcal_meta_file is a required parameter. Perhaps you want to put a way to allow the user to pass None for old_redcal_meta_file if this is False?

[jsdillon]

ntimes isn't used

[jsdillon]

a bit weird to call it a dly_factor when it includes an offset (which can be summed inside the exponent)

[jsdillon]

lol, thanks

[jsdillon]

Is there any other test we can do? Like, does doing this make the calibration solutions smoother in time? If so, can we quantify that and make an assertion about it?

[aewallwi]

I think it's overkill to show whether or not the calibration is smoother. Anecdotally, I find that the differences in the solutions after replacing the degeneracies are quite small. I will document this above.

[jsdillon]

Fair point. Mostly I'm interested in the question of whether polarities are handled appropriately.

[aewallwi]

I am unable to reproduce the macOS unittest failures on a mac. I have tried to create a fresh conda environment and still no luck. Should we remove the macOS tests?

[jsdillon]

Maybe @bhazelton or @plaplant have ideas about how to debug testing failures that appear only to happen in the CI but can't be reproduced locally...

[plaplant]

@aewallwi @jsdillon I ran the tests locally on my (M1-based) Mac in a fresh environment, and I'm getting a test failure too, though on a different test than the CI failures. I've posted the full traceback at the bottom of my comment.

As a way to debug testing failures in CI that don't appear locally, I tend to ssh into the machine running the job, and poke around on the command line there. This way you're using the exact same environment and hardware that produced the error, so it's easier to figure out what's going wrong. I would highly (can't stress this enough) recommend against ignoring test failures or tuning off CI checks. It's better in the long run to get to the bottom of failing tests, and understand what's going wrong.

My local test failure:

__________________________________________________________________ Test_ReflectionFitter_XTalk.test_misc_svd_funcs ___________________________________________________________________

self = <hera_cal.tests.test_reflections.Test_ReflectionFitter_XTalk object at 0x12fb78640>

    def test_misc_svd_funcs(self):
        # setup RF object
        RF = reflections.ReflectionFitter(self.uvd)
        # add noise
        np.random.seed(0)
        Namp = 3e0
        for k in RF.data:
            RF.data += stats.norm.rvs(0, Namp, RF.Ntimes * RF.Nfreqs).reshape(RF.Ntimes, RF.Nfreqs) + 1j * stats.norm.rvs(0, Namp, RF.Ntimes * RF.Nfreqs).reshape(RF.Ntimes, RF.Nfreqs
)
        bl = (23, 24, 'ee')

        # fft data
        RF.fft_data(data=RF.data, window='blackmanharris', overwrite=True)

        # sparse sv decomp
        svd_wgts = RF.svd_weights(RF.dfft, RF.delays, min_dly=150, max_dly=500, side='both')
        RF.sv_decomp(RF.dfft, wgts=svd_wgts, keys=[bl], overwrite=True, Nkeep=None, sparse_svd=True)
        assert RF.umodes[bl].shape == (100, 98)
        assert RF.vmodes[bl].shape == (98, 128)

>       RF.sv_decomp(RF.dfft, wgts=svd_wgts, keys=[bl], overwrite=True, Nkeep=10, sparse_svd=True)

hera_cal/tests/test_reflections.py:429:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
hera_cal/reflections.py:589: in sv_decomp
    u, svals, v = sparse.linalg.svds(dfft[k] * wgts[k], k=Nk, which='LM')
../../../../miniforge3/envs/hera_cal_tests/lib/python3.10/site-packages/scipy/sparse/linalg/_eigen/_svds.py:350: in svds
    eigvals, eigvec = eigsh(XH_X, k=k, tol=tol ** 2, maxiter=maxiter,
../../../../miniforge3/envs/hera_cal_tests/lib/python3.10/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py:1566: in eigsh
    ret = eigs(A, k, M=M, sigma=sigma, which=which, v0=v0,
../../../../miniforge3/envs/hera_cal_tests/lib/python3.10/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py:1345: in eigs
    params.iterate()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <scipy.sparse.linalg._eigen.arpack.arpack._UnsymmetricArpackParams object at 0x2da3c35b0>

    def iterate(self):
        if self.tp in 'fd':
            self.ido, self.tol, self.resid, self.v, self.iparam, self.ipntr, self.info =\
                self._arpack_solver(self.ido, self.bmat, self.which, self.k,
                                    self.tol, self.resid, self.v, self.iparam,
                                    self.ipntr, self.workd, self.workl,
                                    self.info)
        else:
            self.ido, self.tol, self.resid, self.v, self.iparam, self.ipntr, self.info =\
                self._arpack_solver(self.ido, self.bmat, self.which, self.k,
                                    self.tol, self.resid, self.v, self.iparam,
                                    self.ipntr, self.workd, self.workl,
                                    self.rwork, self.info)

        xslice = slice(self.ipntr[0] - 1, self.ipntr[0] - 1 + self.n)
        yslice = slice(self.ipntr[1] - 1, self.ipntr[1] - 1 + self.n)
        if self.ido == -1:
            # initialization
            self.workd[yslice] = self.OP(self.workd[xslice])
        elif self.ido == 1:
            # compute y = Op*x
            if self.mode in (1, 2):
                self.workd[yslice] = self.OP(self.workd[xslice])
            else:
                Bxslice = slice(self.ipntr[2] - 1, self.ipntr[2] - 1 + self.n)
                self.workd[yslice] = self.OPa(self.workd[Bxslice])
        elif self.ido == 2:
            self.workd[yslice] = self.B(self.workd[xslice])
        elif self.ido == 3:
            raise ValueError("ARPACK requested user shifts.  Assure ISHIFT==0")
        else:
            self.converged = True

            if self.info == 0:
                pass
            elif self.info == 1:
                self._raise_no_convergence()
            else:
>               raise ArpackError(self.info, infodict=self.iterate_infodict)
E               scipy.sparse.linalg._eigen.arpack.arpack.ArpackError: ARPACK error 3: No shifts could be applied during a cycle of the Implicitly restarted Arnoldi iteration. One possibility is to increase the size of NCV relative to NEV.

../../../../miniforge3/envs/hera_cal_tests/lib/python3.10/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py:757: ArpackError

[jsdillon]

Thanks @plaplant. How do you ssh into github actions?

[aewallwi]

@jsdillon it looks like there isn't a built-in way to do it for Github Actions (unlike in CircleCI). I found this: https://github.com/marketplace/actions/debugging-with-ssh. Maybe that will work?

The tests pass for linux which is the OS that is going to be used for our science runs. I think this raises an interesting question. How much time do we want to be investing in order to fully support multiple operating systems when Linux is the standard for our production analysis and the error here is not reproducible on other macOS machines and could potentially be a bug on githubs end?