"import rapidfuzz" causes "illegal hardware instruction"

[workflowsguy]

While I had used rapidfuzz previously without issues in my Python scripts, now every script that uses it crashes with an error such as illegal hardware instruction (Console) or Terminated due to signal: ILLEGAL INSTRUCTION (4) (Development environment)

The simplest script to demonstrate this just consists of

import rapidfuzz

Versions:
rapidfuzz 3.9.0 (installed with pip install rapidfuzz)
Python 3.11.9

After I reverted to rapidfuzz 3.8.1, the issue did not occur anymore

Thanks!

[maxbachmann]

strange nothing really changed between these two versions that should affect the C++ version. Can you provide some context:

• what is the CPU you are using?
• what is the OS?

[workflowsguy]

Machine information:

macOS 10.13.6
Intel(R) Core(TM) i7 CPU 870 @ 2.93GHz

[maxbachmann]

I think I have an idea what this could be. I did build the wheels using the macos-latest image. Apparently github moved this from macOS 12 x86 to macOS 14 arm in between the builds. Possibly this raised e.g. the min macOS version.

I will make a test build using macOS 13 x86 and link you the wheel here for a test install once it's finished for you to test.

[maxbachmann]

Can you test this wheel?
artifact-build_wheels_macos-3.zip

[workflowsguy]

I installed the wheel with pip install ~/rapidfuzz-3.9.0-cp311-cp311-macosx_10_9_x86_64.whl

After that, running the test script still causes illegal hardware instructionand Python crashes.
The first lines of the crash report follow. Maybe this can give you a clue...

Process:               Python [3585]
Path:                  /opt/local/Library/Frameworks/Python.framework/Versions/3.11/Resources/Python.app/Contents/MacOS/Python
Identifier:            Python
Version:               3.11.9 (3.11.9)
Code Type:             X86-64 (Native)
Parent Process:        zsh [750]
Responsible:           Python [3585]
User ID:               501

Date/Time:             2024-05-18 11:52:12.219 +0200
OS Version:            Mac OS X 10.13.6 (17G14042)
Report Version:        12
Anonymous UUID:        A41421B4-F1A6-23F0-F507-8C741C7D2643

Sleep/Wake UUID:       269CD014-9694-4B86-8B91-B0410B57CBA8

Time Awake Since Boot: 18000 seconds
Time Since Wake:       3700 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_INSTRUCTION (SIGILL)
Exception Codes:       0x0000000000000001, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Illegal instruction: 4
Termination Reason:    Namespace SIGNAL, Code 0x4
Terminating Process:   exc handler [0]

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   metrics_cpp_avx2.cpython-311-darwin.so	0x000000010abc1a52 __pyx_pymod_exec_metrics_cpp_avx2(_object*) + 8242
1   org.python.python             	0x000000010a064de2 PyModule_ExecDef + 114
2   org.python.python             	0x000000010a16873e _imp_exec_dynamic + 12
3   org.python.python             	0x000000010a062bf8 cfunction_vectorcall_O + 104
4   org.python.python             	0x000000010a11b35e _PyEval_EvalFrameDefault + 62106
5   org.python.python             	0x000000010a10adce PyEval_EvalCode + 281
6   org.python.python             	0x000000010a1058b5 builtin_exec + 491
7   org.python.python             	0x000000010a062a53 cfunction_vectorcall_FASTCALL_KEYWORDS + 95
8   org.python.python             	0x000000010a11b35e _PyEval_EvalFrameDefault + 62106
9   org.python.python             	0x000000010a11dd6e _PyEval_Vector + 96
10  org.python.python             	0x000000010a00f9bf object_vacall + 296
11  org.python.python             	0x000000010a00f808 PyObject_CallMethodObjArgs + 227
12  org.python.python             	0x000000010a163150 PyImport_ImportModuleLevelObject + 2957
13  org.python.python             	0x000000010a103044 builtin___import__ + 199
14  org.python.python             	0x000000010a062a53 cfunction_vectorcall_FASTCALL_KEYWORDS + 95
15  org.python.python             	0x000000010a11b35e _PyEval_EvalFrameDefault + 62106
16  org.python.python             	0x000000010a11dd6e _PyEval_Vector + 96
17  org.python.python             	0x000000010a00f9bf object_vacall + 296
18  org.python.python             	0x000000010a00f808 PyObject_CallMethodObjArgs + 227
19  org.python.python             	0x000000010a162878 PyImport_ImportModuleLevelObject + 693
20  org.python.python             	0x000000010a114a89 _PyEval_EvalFrameDefault + 35269
21  org.python.python             	0x000000010a10adce PyEval_EvalCode + 281
22  org.python.python             	0x000000010a1058b5 builtin_exec + 491
23  org.python.python             	0x000000010a062a53 cfunction_vectorcall_FASTCALL_KEYWORDS + 95
24  org.python.python             	0x000000010a11b35e _PyEval_EvalFrameDefault + 62106
25  org.python.python             	0x000000010a11dd6e _PyEval_Vector + 96
26  org.python.python             	0x000000010a00f9bf object_vacall + 296
27  org.python.python             	0x000000010a00f808 PyObject_CallMethodObjArgs + 227
28  org.python.python             	0x000000010a163150 PyImport_ImportModuleLevelObject + 2957
29  org.python.python             	0x000000010a103044 builtin___import__ + 199
30  org.python.python             	0x000000010a062a53 cfunction_vectorcall_FASTCALL_KEYWORDS + 95
31  org.python.python             	0x000000010a11b35e _PyEval_EvalFrameDefault + 62106
32  org.python.python             	0x000000010a11dd6e _PyEval_Vector + 96
33  org.python.python             	0x000000010a00f9bf object_vacall + 296
34  org.python.python             	0x000000010a00f808 PyObject_CallMethodObjArgs + 227
35  org.python.python             	0x000000010a162878 PyImport_ImportModuleLevelObject + 693
36  org.python.python             	0x000000010a114a89 _PyEval_EvalFrameDefault + 35269
37  org.python.python             	0x000000010a10adce PyEval_EvalCode + 281
38  org.python.python             	0x000000010a1058b5 builtin_exec + 491
39  org.python.python             	0x000000010a062a53 cfunction_vectorcall_FASTCALL_KEYWORDS + 95
40  org.python.python             	0x000000010a11b35e _PyEval_EvalFrameDefault + 62106
41  org.python.python             	0x000000010a11dd6e _PyEval_Vector + 96
42  org.python.python             	0x000000010a00f9bf object_vacall + 296
43  org.python.python             	0x000000010a00f808 PyObject_CallMethodObjArgs + 227
44  org.python.python             	0x000000010a163150 PyImport_ImportModuleLevelObject + 2957
45  org.python.python             	0x000000010a114a89 _PyEval_EvalFrameDefault + 35269
46  org.python.python             	0x000000010a10adce PyEval_EvalCode + 281
47  org.python.python             	0x000000010a18c9e1 run_eval_code_obj + 78
48  org.python.python             	0x000000010a18ce31 run_mod + 96
49  org.python.python             	0x000000010a18cdaa pyrun_file + 133
50  org.python.python             	0x000000010a18c3dd _PyRun_SimpleFileObject + 558
51  org.python.python             	0x000000010a18b8b6 _PyRun_AnyFileObject + 136
52  org.python.python             	0x000000010a1b2e48 Py_RunMain + 2222
53  org.python.python             	0x000000010a1b3af0 pymain_main + 482
54  org.python.python             	0x000000010a1b3c74 Py_BytesMain + 42
55  libdyld.dylib                 	0x00007fff71a44015 start + 1

Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x000000010ae3571f  rbx: 0x000000010ae444eb  rcx: 0x000000010ae4b1a8  rdx: 0x000000010ae4ae68
  rdi: 0x000000010ab1a6d0  rsi: 0x000000010ab50170  rbp: 0x00007ffee5c7c220  rsp: 0x00007ffee5c7b000
   r8: 0x3891b6418558226a   r9: 0x10b1d3cd09b79304  r10: 0x000000010ab501b8  r11: 0x000000010ab501a0
  r12: 0x000000010ab1a480  r13: 0x000000010aba53e8  r14: 0x000000010ae35420  r15: 0x0000000000000000
  rip: 0x000000010abc1a52  rfl: 0x0000000000010206  cr2: 0x000000010abc1000
  
Logical CPU:     6
Error Code:      0x00000000
Trap Number:     6

[maxbachmann]

Thanks. Can you test the following package as well?
artifact-build_wheels_macos-0.zip

This is version 3.8.1 built with the current toolchains since I suspect this is a build issue and not an issue with the library itself.

[workflowsguy]

This also causes the error message illegal hardware instruction in the console, followed by a cash of Python:

Process:               Python [61921]
Path:                  /opt/local/Library/Frameworks/Python.framework/Versions/3.11/Resources/Python.app/Contents/MacOS/Python
Identifier:            Python
Version:               3.11.9 (3.11.9)
Code Type:             X86-64 (Native)
Parent Process:        zsh [779]
Responsible:           Python [61921]
User ID:               501

Date/Time:             2024-05-19 10:40:48.508 +0200
OS Version:            Mac OS X 10.13.6 (17G14042)
Report Version:        12
Anonymous UUID:        A41421B4-F1A6-23F0-F507-8C741C7D2643


Time Awake Since Boot: 16000 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_INSTRUCTION (SIGILL)
Exception Codes:       0x0000000000000001, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Illegal instruction: 4
Termination Reason:    Namespace SIGNAL, Code 0x4
Terminating Process:   exc handler [0]

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   metrics_cpp_avx2.cpython-311-darwin.so	0x0000000103d76543 __pyx_pymod_exec_metrics_cpp_avx2(_object*) + 8179
1   org.python.python             	0x0000000103219de2 PyModule_ExecDef + 114
2   org.python.python             	0x000000010331d73e _imp_exec_dynamic + 12
3   org.python.python             	0x0000000103217bf8 cfunction_vectorcall_O + 104
4   org.python.python             	0x00000001032d035e _PyEval_EvalFrameDefault + 62106
5   org.python.python             	0x00000001032bfdce PyEval_EvalCode + 281
6   org.python.python             	0x00000001032ba8b5 builtin_exec + 491
7   org.python.python             	0x0000000103217a53 cfunction_vectorcall_FASTCALL_KEYWORDS + 95
8   org.python.python             	0x00000001032d035e _PyEval_EvalFrameDefault + 62106
9   org.python.python             	0x00000001032d2d6e _PyEval_Vector + 96
10  org.python.python             	0x00000001031c49bf object_vacall + 296
11  org.python.python             	0x00000001031c4808 PyObject_CallMethodObjArgs + 227
12  org.python.python             	0x0000000103318150 PyImport_ImportModuleLevelObject + 2957
13  org.python.python             	0x00000001032b8044 builtin___import__ + 199
14  org.python.python             	0x0000000103217a53 cfunction_vectorcall_FASTCALL_KEYWORDS + 95
15  org.python.python             	0x00000001032d035e _PyEval_EvalFrameDefault + 62106
16  org.python.python             	0x00000001032d2d6e _PyEval_Vector + 96
17  org.python.python             	0x00000001031c49bf object_vacall + 296
18  org.python.python             	0x00000001031c4808 PyObject_CallMethodObjArgs + 227
19  org.python.python             	0x0000000103317878 PyImport_ImportModuleLevelObject + 693
20  org.python.python             	0x00000001032c9a89 _PyEval_EvalFrameDefault + 35269
21  org.python.python             	0x00000001032bfdce PyEval_EvalCode + 281
22  org.python.python             	0x00000001032ba8b5 builtin_exec + 491
23  org.python.python             	0x0000000103217a53 cfunction_vectorcall_FASTCALL_KEYWORDS + 95
24  org.python.python             	0x00000001032d035e _PyEval_EvalFrameDefault + 62106
25  org.python.python             	0x00000001032d2d6e _PyEval_Vector + 96
26  org.python.python             	0x00000001031c49bf object_vacall + 296
27  org.python.python             	0x00000001031c4808 PyObject_CallMethodObjArgs + 227
28  org.python.python             	0x0000000103318150 PyImport_ImportModuleLevelObject + 2957
29  org.python.python             	0x00000001032b8044 builtin___import__ + 199
30  org.python.python             	0x0000000103217a53 cfunction_vectorcall_FASTCALL_KEYWORDS + 95
31  org.python.python             	0x00000001032d035e _PyEval_EvalFrameDefault + 62106
32  org.python.python             	0x00000001032d2d6e _PyEval_Vector + 96
33  org.python.python             	0x00000001031c49bf object_vacall + 296
34  org.python.python             	0x00000001031c4808 PyObject_CallMethodObjArgs + 227
35  org.python.python             	0x0000000103317878 PyImport_ImportModuleLevelObject + 693
36  org.python.python             	0x00000001032c9a89 _PyEval_EvalFrameDefault + 35269
37  org.python.python             	0x00000001032bfdce PyEval_EvalCode + 281
38  org.python.python             	0x00000001032ba8b5 builtin_exec + 491
39  org.python.python             	0x0000000103217a53 cfunction_vectorcall_FASTCALL_KEYWORDS + 95
40  org.python.python             	0x00000001032d035e _PyEval_EvalFrameDefault + 62106
41  org.python.python             	0x00000001032d2d6e _PyEval_Vector + 96
42  org.python.python             	0x00000001031c49bf object_vacall + 296
43  org.python.python             	0x00000001031c4808 PyObject_CallMethodObjArgs + 227
44  org.python.python             	0x0000000103318150 PyImport_ImportModuleLevelObject + 2957
45  org.python.python             	0x00000001032c9a89 _PyEval_EvalFrameDefault + 35269
46  org.python.python             	0x00000001032bfdce PyEval_EvalCode + 281
47  org.python.python             	0x00000001033419e1 run_eval_code_obj + 78
48  org.python.python             	0x0000000103341e31 run_mod + 96
49  org.python.python             	0x0000000103341daa pyrun_file + 133
50  org.python.python             	0x00000001033413dd _PyRun_SimpleFileObject + 558
51  org.python.python             	0x00000001033408b6 _PyRun_AnyFileObject + 136
52  org.python.python             	0x0000000103367e48 Py_RunMain + 2222
53  org.python.python             	0x0000000103368af0 pymain_main + 482
54  org.python.python             	0x0000000103368c74 Py_BytesMain + 42
55  libdyld.dylib                 	0x00007fff557b1015 start + 1

Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x0000000103fea72f  rbx: 0x0000000103ff94eb  rcx: 0x00000001040001a0  rdx: 0x0000000103fffe68
  rdi: 0x0000000103cce860  rsi: 0x0000000103d00cb0  rbp: 0x00007ffeecac8220  rsp: 0x00007ffeecac7020
   r8: 0xc0b04e9999526c62   r9: 0x01c27fe65c0f7904  r10: 0x0000000103d00cf8  r11: 0x0000000103d00ce0
  r12: 0x0000000103cce610  r13: 0x0000000103d5a3e8  r14: 0x0000000103fea440  r15: 0x0000000000000000
  rip: 0x0000000103d76543  rfl: 0x0000000000010206  cr2: 0x000000010bd2a000
  
Logical CPU:     5
Error Code:      0x00000000
Trap Number:     6

[maxbachmann]

@henryiii can you have a quick look at this? You are usually more familiar with ecosystem changes in Python build tools.

To give a short recap of what is happening here:

1. mac os build from 07.04.2024 is working as expected https://github.com/rapidfuzz/RapidFuzz/actions/runs/8591457629/job/23540277660
2. mac os build of the exact same source code from 18.05.2024 is causing illegal instruction error on old versions of mac os https://github.com/rapidfuzz/RapidFuzz/actions/runs/9140181829/job/25133302598

So this sounds very much like something broke due to implicit updates of build tooling. Looking at the build log I noticed a couple of differences:

• macos-latest changed from macOs 12 x86 to macOs 14 arm. I already tried macOs 13 x86 which leads to the same error. I couldn't get macOs 12 to work anymore. When building with it cibuildwheels says that I didn't select any targets https://github.com/rapidfuzz/RapidFuzz/actions/runs/9140101349/job/25133150472
• apple clang 14 vs 15
• both builds set CMAKE_OSX_DEPLOYMENT_TARGET:STRING=10.9 so they are still supposed to work for macOS 10.13.6

I don't see any other differences between the two builds, but maybe I am missing something. Do you have any idea what could go wrong here?

[maxbachmann]

Can you give the following build a try:
artifact-build_wheels_macos-0.zip

This is a build without the avx2 versions

[henryiii]

As a first step, I'd recommend using macos-X and macos-14, where X < 14, building only native packages on each. (I have no idea why 12 isn't working, that was the default for a long time and cibuildwheel should be fine on it, certainly shouldn't have an issue!) If you want a universal wheel too (pip won't ever download it with the native packages present), then you can fuse the native packages into a universal2 package. (I think it's not too hard, but don't remember where I saw it done).

The only similar issue I've seen so far was that either uv or ruff builds with an old macOS because there was an issue using the newer Xcode when targeting older macOS. I haven't checked yet to see exactly what the issue is, they are building on macos-11 for now. (maturin-action and Rust, not cibuildwheel)

I technically do have access to a macOS <11 machine (10.12 IIRC - it's an iMac with an NVIDIA graphics card, it's that old!), but it's in my old office and I won't be going there probably at least for a week. (Currently at PyCon, so def. don't have access to it now!)

[workflowsguy]

Can you give the following build a try: artifact-build_wheels_macos-0.zip

This is a build without the avx2 versions

No crash from the test script 👍

[henryiii]

On a branch macOS-12 worked. But sounds like this is the correct fix. Does avx make it noticeably faster? You could make a higher minimum version wheel too with avx if it was important.

[maxbachmann]

On a branch macOS-12 worked.

I don't see anything directly obvious on what I might have done wrong in 177663a that would lead to no build targets being selected

But sounds like this is the correct fix.

Which fix do you mean? Disabling avx2 using macOs-12 or something else?

Does avx make it noticeably faster?

When performing many x many comparisons it actually yields close to a 2x performance improvement for some metrics, since it allows me to compare twice as many strings in parallel. So I would prefer to keep it. Currently I include the binary twice and just load the correct one based on CPU feature flags.

You could make a higher minimum version wheel too with avx if it was important.

Do you mean publishing both an 10.9+ version using only sse2 and a version targeting a later version which guarantees AVX2? If so which one does actually guarantee this? I mean the cpu in question here (i7 870) already supports AVX2.

[henryiii]

I don't see anything directly obvious

I don't either, but #384 passes.

Disabling avx2 using macOs-12 or something else?

I was thinking avx2, but it was previously working, so to me this sounds like a bug. What's odd though the 13+ (probably Xcode is the important part, the previous macOS always updates Xcode once to next macOS's Xcode) does seem to work except on older macOS. It would be interesting to list the binary symbols, I think.

I think since macos-12 does work (in #384), the simplest thing might be to use that (assuming it fixes the problem) until GHA drops the image. I think the macos-11 image is still usable but deprecated, so it's probably ~1 year unless there are hardware reasons to drop it sooner.

Do you mean publishing both an 10.9+ version using only sse2 and a version targeting a later version which guarantees AVX2?

Yes, but I don't know what version this would be, somewhere between 10.13 and 13; since it only happens on the actual macOS versions, it would be really hard to pin down.

I rather expect this is a bug and it might get ironed out in later releases, if we could pin it down a bit more possibly could even report it.