fix(windows): set UTF-8 environment values through wide CRT - #2290
knewstimek wants to merge 2 commits into
Conversation
Signed-off-by: News <knewstimek@users.noreply.github.com>
|
Thanks for opening this — it has been seen, and it is queued. This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence. Current review status: working through a backlog. What that means for this PR, concretely:
Things that will genuinely speed it up whenever review does happen:
If this fixes a bug, a reproduction we can run is worth more than a description of the symptom. Thanks for contributing, and sorry in advance for the wait. |
|
|
DeusData
left a comment
There was a problem hiding this comment.
Thank you for splitting #2268 as asked; this piece is small and focused, which made it easy to check where the values go afterwards.
One problem before it merges. With _wputenv_s, a narrow getenv returns the value in the ANSI code page rather than UTF-8, and several in-tree callers read variables set through cbm_setenv with a raw getenv and treat the result as UTF-8:
internal/cbm/cbm.creadsCBM_INDEX_MARKER_FILEandCBM_INDEX_QUARANTINE_FILEwithgetenvand opens them withcbm_fopen; the supervisor sets both inworker_set_local_env.src/cli/cli.csavesgetenv("CBM_CACHE_DIR")aftermain.chas set it, then restores it throughcbm_setenv, so ANSI bytes get written back as if they were UTF-8.
With a non-ASCII cache path, crash recovery's marker and quarantine files and the CLI cache-dir restore would break, where today they work whenever _putenv_s accepts the bytes.
What would land:
- Switch those readers to
cbm_safe_getenv, which returns UTF-8 on Windows. - Add a Windows test that covers a caller: for example, set a non-ASCII marker path through the supervisor's setter and show
cbm_fopenopens it, or show the CLI cache-dir save and restore round-trips.
The red test-diag job is pipeline_python_cross_module_call, a known flake on our side, and your diff is entirely inside #ifdef _WIN32. Thank you again.
Signed-off-by: News <knewstimek@users.noreply.github.com>
DeusData
left a comment
There was a problem hiding this comment.
Thank you, @knewstimek. This is exactly what we hoped for. All three readers now go through cbm_safe_getenv (cbm.c marker and quarantine readers, cli.c cache-dir save/restore), and the new extraction test drives the real journal writer with a Δ/丁 path. So it checks the path the supervisor actually uses during crash recovery, not just the setter, and that's what lets us merge this with confidence. Thanks as well for the control build against the previous head for the installer ACL failures. It saved us from chasing them.
We checked every reader of the four variables cbm_setenv now writes. The only raw getenv left is CBM_INDEX_SINGLE_THREAD in pipeline.c, which only ever holds "1", so ASCII and ANSI agree and it's fine. Children are spawned with the inherited wide environment block, so they see the same values.
Approved. Our PR CI doesn't run the Windows unit suites, so before merging we'll run extraction and platform on our Windows VM, including a revert check of the marker reader, to see the new test catch the bug.
Three small notes, none blocking:
- The 4 KiB buffer in
cli.cmeans an over-longCBM_CACHE_DIRis now treated as absent, so it gets unset on restore rather than restored. That's fine in practice; it's just worth knowing about. - The quarantine reader has the same shape as the marker reader but no test of its own. A twin test would be welcome in a follow-up.
- The narrow-
getenvassertion intest_platform.cpins UCRT's own conversion behaviour. That holds on our toolchains, but would change meaning under a UTF-8activeCodePagemanifest.
Thank you again for splitting #2268 so cleanly!
On Windows, set UTF-8 environment values through
_wputenv_sand keepSetEnvironmentVariableWfor the process environment. Narrowgetenvthen returns ANSI-code-page bytes, so the worker marker/quarantine readers and the CLI cache-directory save now usecbm_safe_getenvto keep their paths in UTF-8.The Windows platform test covers a UTF-8 value rejected by the old setter under a legacy ANSI code page and verifies the wide, UTF-8, and narrow representations. A new extraction regression sets a non-ASCII marker path, calls the actual journal writer, and verifies both start and completion records in that file. A focused before/after journal harness fails with the previous reader and passes with this change.
Local Windows x64 validation with LLVM-MinGW 20260908 / clang 23.1.1: the platform and extraction suites pass (400 passed, 4 platform skips). The CLI suite has 289 passed, 7 platform skips, and four installer failures caused by inherited ACL checks; a control build of the previous PR head reproduces the same four failures at the same assertions. Changed sources compile with
-Werror.Split from #2268.