[BUG]: Failing test in 1.26.1: e2e::issue_1745

[kpcyrd]

Pre-submission checks

• I am not filing a feature request. These should be filed via the feature request form instead.
• I have checked the Troubleshooting Guide for my problem.
• I have looked through both the open and closed issues for a duplicate report.
• I affirm that I have followed the AI policy and that this report constitutes my own words and understanding.

zizmor version

1.26.1

Expected behavior

cargo test works

Actual behavior

---- e2e::issue_1745 stdout ----
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Snapshot Summary ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Snapshot: issue_1745
Source: crates/zizmor/tests/integration/e2e.rs:640
────────────────────────────────────────────────────────────────────────────────
Expression: zizmor()
    .offline(true)
    .no_config(true)
    .working_dir(input_under_test("issue-1745-repro"))
    .input(".github")
    .args(["--format=github"])
    .run()?
────────────────────────────────────────────────────────────────────────────────
-old snapshot
+new results
────────────┬───────────────────────────────────────────────────────────────────
    1       │-::warning file=@@WORKING_DIR@@/@@TEST_PREFIX@@/issue-1745-repro/@@INPUT@@/workflows/test.yml,line=10,title=artipacked::test.yml:10: credential persistence through GitHub Actions artifacts: does not set persist-credentials: false
    2       │-::error file=@@WORKING_DIR@@/@@TEST_PREFIX@@/issue-1745-repro/@@INPUT@@/workflows/test.yml,line=10,title=unpinned-uses::test.yml:10: unpinned action reference: action is not pinned to a hash (required by blanket policy)
          1 │+::warning file=workflows/test.yml,line=10,title=artipacked::test.yml:10: credential persistence through GitHub Actions artifacts: does not set persist-credentials: false
          2 │+::error file=workflows/test.yml,line=10,title=unpinned-uses::test.yml:10: unpinned action reference: action is not pinned to a hash (required by blanket policy)
────────────┴───────────────────────────────────────────────────────────────────
To update snapshots run `cargo insta review`
Stopped on the first failure. Run `cargo insta test` to run all snapshots.

thread 'e2e::issue_1745' (47673) panicked at /build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/insta-1.47.2/src/runtime.rs:719:13:
snapshot assertion for 'issue_1745' failed in line 640
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Reproduction steps

1. cargo test --frozen

Logs

Additional context

No response

[woodruffw]

Hmm, this test isn't failing in CI, although I did recently change those snapshots as part of a refactor.

How are you running the unit tests? Are you running them in a different working directory than a "plain" clone would?

[kpcyrd]

This is executed on the Arch Linux build server (inside nspawn, that shouldn't matter though). The instructions used are (the rustc --print is outdated and hasn't been updated yet):

cd "${pkgname}-${pkgver}"
cargo fetch --locked --target "$(rustc --print host-tuple)"
cargo build --frozen --release
cargo test --frozen

The first cd is switching into the folder contained in this archive:

https://github.com/zizmorcore/zizmor/archive/v1.26.1/zizmor-1.26.1.tar.gz

There are also some environment variables set:

SHELL=/bin/bash
DEBUG_FFLAGS=-g -ffile-prefix-map=/build/zizmor/src=/usr/src/debug/zizmor
BUILDTOOL=devtools
SUDO_GID=0
PYTHONHASHSEED=0
TEXTDOMAINDIR=/usr/share/locale
SUDO_COMMAND=/bin/bash -c bash -c cd\ /startdir;\ makepkg\ "$@" -bash --syncdeps --noconfirm --log --holdver --skipinteg --install
SUDO_USER=root
GIT_CONFIG_GLOBAL=/dev/null
SUDO_TTY=/dev/pts/0
FFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt         -Wp,-D_FORTIFY_SOURCE=3 -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/zizmor/src=/usr/src/debug/zizmor
PWD=/build/zizmor/src
SOURCE_DATE_EPOCH=1782056600
LOGNAME=builduser
CXXFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -Wp,-D_GLIBCXX_ASSERTIONS -g -ffile-prefix-map=/build/zizmor/src=/usr/src/debug/zizmor
SUDO_HOME=/root
DEBUG_RUSTFLAGS=-C debuginfo=2 --remap-path-prefix=/build/zizmor/src=/usr/src/debug/zizmor
COMMAND_MODE=legacy
LDFLAGS=-Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now          -Wl,-z,pack-relative-relocs
HOME=/build
LANG=C.UTF-8
GIT_CONFIG_SYSTEM=/etc/makepkg.d/gitconfig
RUSTFLAGS=-C force-frame-pointers=yes -C debuginfo=2 --remap-path-prefix=/build/zizmor/src=/usr/src/debug/zizmor
FCFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt         -Wp,-D_FORTIFY_SOURCE=3 -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/zizmor/src=/usr/src/debug/zizmor
MAKEFLAGS=-j96
TERM=xterm-256color
USER=builduser
SHLVL=1
DEBUGINFOD_URLS=https://debuginfod.archlinux.org 
CHOST=x86_64-pc-linux-gnu
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl
CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/zizmor/src=/usr/src/debug/zizmor
SUDO_UID=0
MAIL=/var/mail/builduser
BUILDTOOLVER=1:1.5.0-1-any
OLDPWD=/startdir
TEXTDOMAIN=pacman-scripts
_=/usr/bin/env

[woodruffw]

Gotcha, thanks for the extra information!

[woodruffw]

Hmm, looking more closely at the diff, there's something suspicious there: the paths on the positive half of the diff are wrong as well.

I'm guessing what's happening here is that zizmor is trying to make the paths relative to the Git repo root (this is new behavior in 1.26), but since you're running the tests from an archive of the tag there's no Git root. Consequently, zizmor falls back to the old behavior of using the user-supplied prefix, which matches what the diffs have for you.

Does that match up with your understanding? If so, I think the easiest fix is probably for the Arch build to use a Git checkout of the tag instead of GitHub's source archive for the same tag.

[kpcyrd]

That does indeed sound like the root cause. 👍 Switching to a Git clone would be possible for Arch Linux, but the Alpine package is going to run into the same problem, and if I recall correctly abuild can't pull from git.

[woodruffw]

Hmm, gotcha. I think creating a small stub-only .git directory would also "trick" zizmor in that case, but that seems like a pretty ugly hack 🙂

[kastl-ars]

For the record, the openSUSE package is hitting the same failing test...

[woodruffw]

This was giving me déjà vu, so I looked: #596

Last time there was an easy workaround in zizmor itself, but this time the change is unfortunately pretty fundamental to how zizmor works (and the test change is actually coverage-bearing, unlike last time). I think the best thing for downstreams to do here is probably to use a Git checkout directly.

(Other options would be to ignore the test, or do the .git stub hack mentioned above.)

[shaanmajid]

One alternative could be to make the test more self-contained and construct the condition it needs (rather than assuming the source tree is a Git checkout and depending on the zizmor repo's own .git/ dir). The test could instead copy the test fixture into a repo-shaped temp dir -- this could be a real git repo but probably simplest to just use a .git/ stub -- and execute zizmor from there.

Synthetic test setups always give me a bit of pause, but I think it's warranted here. The current root-discovery contract is the presence of a.git/ directory. This marker of course can't be provided directly in a fixture since it is necessarily untracked and reserved, so scaffolding that in the test seems preferable to relying on the packaging/source tree layout.