fix: Non-ASCII characters (e.g. accented letters) in a data contract … - #1650
Merged
Merged
Conversation
…file were garbled on Windows
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
The non_utf8_default_encoding fixture now mirrors open()'s signature (a positional buffering argument no longer lands in encoding) and also patches io.open, so Path.open and importlib.resources reads default to cp1252 too. New tests check that fetch_schema decodes both the bundled and a local schema file as UTF-8.
…s UTF-8 Importers (avro, bigquery, dbml, dcs, iceberg, jsonschema, protobuf, sql, unity), datacontract init, the config file and the saved API response of datacontract test used the platform default encoding. On Windows, non-ASCII characters in these files were garbled, or a protobuf source crashed with UnicodeDecodeError.
jschoedl
requested changes
Sep 23, 2026
jschoedl
left a comment
jschoedl
left a comment
Collaborator
There was a problem hiding this comment.
Thanks!
I pushed two commits on top:
- small fix to Tests: the
non_utf8_default_encodingfixture now takes its arguments in the same order asopen(), so a call likeopen(p, "r", -1)no longer puts-1intoencoding. It also patchesio.open, which lets it reachPath.open/importlib.resourcesreads. That way thefetch_schemachange is covered too; the new tests are intests/test_lint_schema.py. - More files: I used the opportunity to fix the same problem in the other places that read or write files with the default encoding
The accented-characters test in test_export_excel.py is redundant — the core encoding fix is already well-covered by test_lint_files.py, which exercises the same code path under a cp1252 default encoding via the non_utf8_default_encoding fixture. There is no Excel-specific behavior to test here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Non-ASCII characters (e.g. accented letters) in a data contract file were garbled on Windows
uv run pytest)uv run ruff check --fix && uv run ruff format)