Problem
UTF-8 BOM is easy to handle, but legacy source encodings need a deliberate policy. Decoding every non-UTF-8 file as CP949 can turn valid GBK, Big5, Shift-JIS or Latin-1 byte pairs into the wrong text. Rejecting bytes that are neither UTF-8 nor CP949 would also change current indexing behavior for files with a single non-UTF-8 comment byte.
Proposed direction
Allow an explicit source encoding setting, with a clear default and per-project or per-file precedence. Please decide how an unconfigured file with invalid UTF-8 should be handled: retain current indexing behavior, replace invalid bytes, or skip with a visible diagnostic. UTF-8 BOM handling can be separate.
Cross-platform conversion also needs a build decision. macOS requires linking iconv explicitly; static Linux builds may need converter availability verified at runtime. A small fixture for each chosen encoding should run against the shipped binary, not only a dynamically linked test build.
Related: #511 (invalid UTF-8 in MCP output) and #2268 (closed after splitting bug fixes). This issue is about source decoding policy during indexing.
Problem
UTF-8 BOM is easy to handle, but legacy source encodings need a deliberate policy. Decoding every non-UTF-8 file as CP949 can turn valid GBK, Big5, Shift-JIS or Latin-1 byte pairs into the wrong text. Rejecting bytes that are neither UTF-8 nor CP949 would also change current indexing behavior for files with a single non-UTF-8 comment byte.
Proposed direction
Allow an explicit source encoding setting, with a clear default and per-project or per-file precedence. Please decide how an unconfigured file with invalid UTF-8 should be handled: retain current indexing behavior, replace invalid bytes, or skip with a visible diagnostic. UTF-8 BOM handling can be separate.
Cross-platform conversion also needs a build decision. macOS requires linking iconv explicitly; static Linux builds may need converter availability verified at runtime. A small fixture for each chosen encoding should run against the shipped binary, not only a dynamically linked test build.
Related: #511 (invalid UTF-8 in MCP output) and #2268 (closed after splitting bug fixes). This issue is about source decoding policy during indexing.