MNT: Run multithreaded tests weekly

[Remi-Gau]

• relates to nibabel thread safe? #1454

[Remi-Gau]

dirty script used to automatically mark as thread_unsafe tests parallel failiing in CI

mostly putting that here in case I nuke my local repo

from pathlib import Path
from rich import print

dry_run = False

with (Path(__file__).parent / "tmp2.txt").open() as f:
    content = f.readlines()

test_files = {}

is_summary = False
for l in content:
    if "short test summary info" in l:
        is_summary = True
        continue

    if is_summary:
        if "PARALLEL FAILED" not in l:
            continue

        file_fn_msg = l.split("PARALLEL FAILED\x1b[0m ", maxsplit=1)[1]
        file_fn = file_fn_msg.split("\x1b[0m - ", maxsplit=1)[0]
        file_fn = file_fn.replace("\n", "")

        tmp = file_fn.split("::")

        file = tmp[0]
        if file not in test_files:
            test_files[file] = {"fn":[], "cls": {}}

        if len(tmp) == 2:
            fn = tmp[1].replace("\x1b[1m", "")
            if fn not in test_files[file]["fn"]:
                test_files[file]["fn"].append(fn)
            del fn

        elif len(tmp) == 3:
            cls = tmp[1].replace("\x1b[1m", "")
            meth = tmp[2].replace("\x1b[1m", "")
            if cls not in test_files[file]["cls"]:
                test_files[file]["cls"][cls] = []
            if meth not in test_files[file]["cls"][cls]:
                test_files[file]["cls"][cls].append(meth)
            del cls
            del meth

root_dir = Path(__file__).parent / ".."

for x in test_files:
    with (root_dir / x).open("r") as f:
        content = f.readlines()
    
    with (root_dir / x).open("w") as f:

        in_class = False
        current_class = None

        skip = False
        for l in content:

            if "pytest.mark.thread_unsafe" in l:
                skip = True
                f.write(l)
                continue

            for cls in test_files[x]["cls"]:
                if l.startswith(f"class {cls}"):
                    in_class = True
                    current_class = cls
                    break

            fn_to_remove = None
            for fn in test_files[x]["fn"]:
                if  l.startswith(f"def {fn}"):
                    in_class = False
                    current_class = None
                    fn_to_remove = fn
                    print(f"marking {x}::{fn}")
                    if not skip and not dry_run:
                        f.write("@pytest.mark.thread_unsafe\n")
                        skip = False
                    break
            if fn_to_remove is not None:
                test_files[x]["fn"].remove(fn_to_remove)

            if in_class:
                meth_to_remove = None
                for meth in test_files[x]["cls"][current_class]:
                    if  f"def {meth}" in l:
                        meth_to_remove = meth
                        print(f"marking {x}::{cls}::{meth}")
                        if not skip and not dry_run:
                            f.write("    @pytest.mark.thread_unsafe\n")
                            skip = False
                        break
                if meth_to_remove is not None:
                    test_files[x]["cls"][current_class].remove(meth_to_remove)

            f.write(l)

print(test_files)

[effigies]

Do you want CI to run? LMK make you a member of the org...

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.44%. Comparing base (f541273) to head (0404a52).
⚠️ Report is 110 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1455      +/-   ##
==========================================
+ Coverage   95.41%   95.44%   +0.02%     
==========================================
  Files         209      209              
  Lines       29822    29981     +159     
  Branches     4483     4483              
==========================================
+ Hits        28455    28614     +159     
  Misses        931      931              
  Partials      436      436              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Remi-Gau]

Do you want CI to run?

getting it to run on my fork is mostly fine for now

[Remi-Gau]

@effigies
things seem green but from my nilearn experience things may be a bit flaky with parallel thread testing, so I would not be surprised if some things came out red once in a while

Next step: do you want to try to integrate this into tox.ini and use the pre-existing CI workflow instead of adding a new one?

[effigies]

do you want to try to integrate this into tox.ini and use the pre-existing CI workflow instead of adding a new one?

I think so.

[Remi-Gau]

• move this in the same workflow that is schedule to run tests on pre

[effigies]

Looks like we discovered a numpy bug:

     @freesurfer_test
      def test_label():
          """Test IO of .label"""
          label_path = pjoin(data_path, 'label', 'lh.cortex.label')
  >       label = read_label(label_path)
                  ^^^^^^^^^^^^^^^^^^^^^^
  
  label_path = '/home/runner/work/nibabel/nibabel/nibabel-data/nitest-freesurfer/fsaverage/label/lh.cortex.label'
  
  /home/runner/work/nibabel/nibabel/nibabel/freesurfer/tests/test_io.py:349: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  /home/runner/work/nibabel/nibabel/nibabel/freesurfer/io.py:587: in read_label
      label_array = np.loadtxt(filepath, dtype=int, skiprows=2, usecols=[0])
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          filepath   = '/home/runner/work/nibabel/nibabel/nibabel-data/nitest-freesurfer/fsaverage/label/lh.cortex.label'
          read_scalars = False
  /home/runner/work/nibabel/nibabel/.tox/free_threaded/lib/python3.14t/site-packages/numpy/lib/_npyio_impl.py:1384: in loadtxt
      arr = _read(fname, dtype=dtype, comment=comment, delimiter=delimiter,
          comment    = ['#']
          comments   = '#'
          converters = None
          delimiter  = None
          dtype      = <class 'int'>
          encoding   = None
          fname      = '/home/runner/work/nibabel/nibabel/nibabel-data/nitest-freesurfer/fsaverage/label/lh.cortex.label'
          like       = None
          max_rows   = None
          ndmin      = 0
          quotechar  = None
          skiprows   = 2
          unpack     = False
          usecols    = [0]
  /home/runner/work/nibabel/nibabel/.tox/free_threaded/lib/python3.14t/site-packages/numpy/lib/_npyio_impl.py:1011: in _read
      fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          byte_converters = False
          comment    = '#'
          comments   = None
          converters = None
          delimiter  = None
          dtype      = dtype('int64')
          encoding   = None
          fh_closing_ctx = <contextlib.nullcontext object at 0x36acb300060>
          filelike   = False
          fname      = '/home/runner/work/nibabel/nibabel/nibabel-data/nitest-freesurfer/fsaverage/label/lh.cortex.label'
          imaginary_unit = 'j'
          max_rows   = -1
          ndmin      = 0
          quote      = None
          read_dtype_via_object_chunks = None
          skiplines  = 2
          unpack     = False
          usecols    = [0]
  /home/runner/work/nibabel/nibabel/.tox/free_threaded/lib/python3.14t/site-packages/numpy/lib/_datasource.py:191: in open
      ds = DataSource(destpath)
           ^^^^^^^^^^^^^^^^^^^^
          destpath   = '.'
          encoding   = None
          mode       = 'rt'
          newline    = None
          path       = '/home/runner/work/nibabel/nibabel/nibabel-data/nitest-freesurfer/fsaverage/label/lh.cortex.label'
  /home/runner/work/nibabel/nibabel/.tox/free_threaded/lib/python3.14t/site-packages/numpy/lib/_datasource.py:248: in __init__
      self._destpath = os.path.abspath(destpath)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
          destpath   = '.'
          self       = <numpy.lib.npyio.DataSource object at 0x36acb2c0190>
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  
  path = '.'
  
      def abspath(path):
          """Return an absolute path."""
          path = os.fspath(path)
          if isinstance(path, bytes):
              if not path.startswith(b'/'):
                  path = join(os.getcwdb(), path)
          else:
              if not path.startswith('/'):
  >               path = join(os.getcwd(), path)
                              ^^^^^^^^^^^
  E               FileNotFoundError: [Errno 2] No such file or directory
  
  path       = '.'
  
  <frozen posixpath>:381: FileNotFoundError

numpy.lib._datasource.open uses destpath='.' by default, which triggers os.getcwd(), which can fail if the CWD is deleted.

[effigies]

chdir() is failing for a lot of thread-unsafe tests, which either means that they are being run in parallel with each other if not with multiple runs of themselves, or that the earlier tests that broke os.getcwd() made all chdir contexts fail because the CWD is never set back to an existing file.

[Remi-Gau]

Looks like we discovered a numpy bug

probably worth reporting upstream?

[effigies]

Yes, trying to create a reproduction.