Fix the tlbparser ctypes mapping for VT_I1.

@haoxuanjng-lang

Summary

tools.tlbparser maps the COM type VT_I1 to ctypes.c_char.
This mapping has been used since the early days of the project.

However, VT_I1 means a signed 8-bit integer (see MSLearn). ctypes.c_char represents one byte of character data.
So, the current mapping does not match the meaning of the COM type.

#962 by @haoxuanjng-lang proposes mapping VT_I1 to ctypes.c_byte.
This change may help handle numeric data correctly and fix problems with NumPy support.

However, this mapping has been used for a long time.
Changing it may affect existing Python code.
We need to check both the need for this fix and its impact on users.

Proposed Changes

The generated types for VT_I1 will change:

For VT_I1, c_char will become c_byte.
For pointers to VT_I1, STRING will become POINTER(c_byte).
- Other types that generate STRING, such as VT_LPSTR, are not affected.

This change may also affect arrays and other types that contain VT_I1, such as pointers, SAFEARRAYs, and C arrays.

Important

How this package marshals automation.VARIANT values is outside the scope of this issue. This issue is limited to the type library parser and code generator.

Problems

1. NumPy support

VT_I1 is a signed 8-bit integer. Its matching NumPy data type is numpy.int8.

However, when VT_I1 is mapped to ctypes.c_char, SAFEARRAY elements may be treated as character data. This causes problems when creating or converting NumPy arrays.
Negative values or array shapes also not work as expected.

Mapping VT_I1 to the correct signed integer type may fix these problems.

2. The relation between COM types and ctypes types

COM type information defines data types, such as method arguments and return values. comtypes reads this information with tlbparser and creates Python modules with codegenerator.

In these modules, ctypes types are used for COM method arguments and return values, as well as structure and union fields. These types must match the meaning of the data defined by COM type information.

VT_I1 means a signed 8-bit integer, so ctypes.c_byte matches its meaning. ctypes.c_char also uses one byte, but it is for character data. The two types do not have the same meaning.

Therefore, the mapping in tlbparser affects the types of method arguments and return values, as well as structure and union fields in generated modules.
We need the correct mapping so that these types match the COM type information.

3. Long-standing behavior and backward compatibility

The mapping from VT_I1 to ctypes.c_char has been used since the early days of the project.
Existing Python code may depend on this behavior.

For example, users may pass bytes to generated methods or expect return values to be bytes.
Some code may also depend on structure or union fields using c_char.

#962 improves the handling of numeric values. However, it also affects some existing calls.

Long-standing behavior is not always intended behavior. However, fixing a bug and ignoring its impact on existing users are two different things.

If the impact of the change is very small, or users can easily fix their calling code, this is a strong reason to make the change. However, if important libraries or common use cases are affected, and users cannot easily fix the problem, this may block the change.

We need to understand not only why the fix is needed, but also which code is affected, how users can fix the problem, and how much work is required.
Based on this information, we can decide how to change the mapping and how to handle backward compatibility.

Completion criteria

VT_I1 is mapped to the correct ctypes type for a signed 8-bit integer.
Generated method arguments and return values, as well as structure and union fields, match the COM type information.
NumPy SAFEARRAY support gives the expected values, signs, dtype, and shape.
The impact on existing code and the work needed to fix it are clear.
Important affected libraries and use cases have been checked, and a plan for backward compatibility has been decided.
The required tests and documentation are ready.

@haoxuanjng-lang

@haoxuanjng-lang
I am also thinking about showing a warning when a type library uses VT_I1 during code generation (more exactly, when parsing the type library).
This would help users check how your proposed changes in #962 may affect them.

Please compare the pros and cons and let me know what you think.

I regenerated the four installed Microsoft type libraries twice, with the legacy and proposed tlbparser mappings in separate Python processes and separate empty generation directories. I inspected the generated declarations and constructed local records; this is not an end-to-end test of a speech engine, portable device, CLR object or MSVidCtl event source.

Library/use case	Observed old/new behavior and migration impact
SAPI `SPPHRASERULE.Confidence`, `SPPHRASEPROPERTY.Confidence`, `SPPHRASEELEMENT.RequiredConfidence`/`ActualConfidence`	The Windows SDK 10.0.22621.0 `sapi.idl` declares signed-byte confidence and `SP_LOW_CONFIDENCE = -1`. In each freshly generated record, writing native `0xff` gives `b'\xff'` with the old mapping and `-1` with the proposed one; assigning `-1` raises `TypeError` before and works after. This supports the numeric fix. Code expecting bytes or using `ord()` must change; numeric comparison with the low-confidence constant becomes natural.
PortableDevice generated VARIANT/PROPVARIANT union fields `cVal`/`pcVal`, and `tagCAC.pElems`	Generated `cVal` changes from byte data to signed integer. `pcVal` and `pElems` change from `STRING` to `POINTER(c_byte)`. For a local `CAC` with `cElems=3` and data `(-1, 0, 1)`, old `pElems` yields only `b'\xff'` because of the embedded NUL; new indexed access yields all three signed numbers. Callers must honor `cElems` and retain the backing allocation. This concerns these generated records, not changes to `automation.VARIANT` marshaling.
mscorlib `SByte.m_value`, `IConvertible.ToSByte`, `IFormatterConverter.ToSByte`	Fresh `SByte.m_value` reads native `0xff` as bytes before and `-1` after. Both `ToSByte` generated retval parameters change from `STRING` to `POINTER(c_byte)`, matching their documented signed numeric return. This is declaration-level evidence for the real interfaces; I have not invoked those CLR methods, so I am not claiming their runtime exception is fixed.
`_IMSVidCtlEvents.Error.CancelDisplay`	SDK `msvidctl.idl` declares `boolean*`, a mutable flag rather than string content. Fresh generation changes its declaration from `STRING` to `POINTER(c_byte)`. This is a potentially important event-handler compatibility case: code treating a value as bytes, indexing it, or writing through it may need changes. I have not exercised the IDispatch event path, and its interaction with VARIANT marshaling needs separate validation.

The matched generated record sizes and field offsets were equal across the two mappings on this Windows x64 installation. That supports layout compatibility for these records; it does not establish Python calling compatibility or the prevalence of affected applications. In particular, bytes and integer truth tests differ (b'\0' versus 0), so migrations must check logic as well as argument types. The native test fixture already demonstrates bytes inputs and writable character buffers becoming ArgumentError; the revised tests assert those exact exceptions.

Sources: Microsoft SAPI rule confidence, SAPI element confidence, PROPVARIANT and counted arrays, IConvertible.ToSByte, and IFormatterConverter.ToSByte; installed SDK sapi.idl and msvidctl.idl, and the regenerated libraries listed above.

For the parsing-time warning suggested here, I reviewed a409eea:

Benefits: the LIBID/version and specific field/parameter identify what to audit; a transition release could warn while retaining the old mapping, giving users time to migrate; parsing can detect numeric records and arrays as well as plain pointers.
Limits/costs: parsing a library does not prove the application uses every affected member; one warning per field can be noisy; cached wrappers bypass parsing, so absence of a warning does not mean absence of impact; warnings can break consumers running with -W error.
Recommendation: use an explicit, filterable warning category and keep the issue link and library/member details. Consider a per-library summary with opt-in per-member diagnostics. Document the cache limitation and an explicit regeneration/audit procedure. Test pointer/SAFEARRAY/C-array/alias traversal, unaffected LPSTR/UI1, suppression/filtering and warning-as-error behavior before treating it as migration coverage. The prototype's _uses_vt_i1 does not follow VT_USERDEFINED, so alias/reference cases deserve a check. I would prefer a warning in a transition release before changing the default rather than treating a simultaneous warning as a compatibility solution.

These findings support the need for the signed mapping, but I cannot conclude that the real-world impact is small or that the MSVidCtl event path is safe. I am keeping #962 as a draft pending the compatibility/transition decision. I have not copied the warning prototype into the PR.

AI assistance: prepared and validated with OpenAI Codex.

Thank you for your investigation and for sharing your thoughts.

Based on your findings and the SDK documentation, I think this change could solve not only a technical inconsistency but also real problems for users.

At the same time, changing the mapping could break existing codebases. Some users may have added ad hoc workarounds for the current behavior. After the change, their code might raise an exception or return a completely different value, causing their business logic to fail.

For this reason, I think we need to handle this migration carefully.

The matched generated record sizes and field offsets were equal across the two mappings on this Windows x64 installation. That supports layout compatibility for these records; it does not establish Python calling compatibility or the prevalence of affected applications.

Record sizes and field offsets are important for technologies based on ctypes, because they help us avoid breaking low-level memory layouts.
Since both char_type and schar_type produce the same record sizes and field offsets in this case, I do not think we need to be too concerned about this particular risk. We should still investigate Python calling compatibility separately.

Parsing a library does not prove the application uses every affected member.

We have to rely on the type information available to us. We cannot know from the type library alone whether an application actually uses every affected member, or what values a VARIANT will return in practice, because that depends on the implementation.

Nevertheless, I think parsing the type library is the most practical starting point for detecting potential compatibility issues. A warning at this point would tell users that they may need to investigate their applications before changing the mapping.

One warning per field can be noisy.

I have a small hope that the noise could encourage users to pay attention to the issue and investigate its impact on their applications.
The purpose of this warning is not simply to notify users about a technical change. It is to help them find out whether the change affects their own code.
Some noise may be acceptable if it helps users discover problems before upgrading.

Cached wrappers bypass parsing, so absence of a warning does not mean absence of impact.

From another perspective, warnings are not emitted on every import if the generated module is reused. This can help prevent unnecessary noise.

comtypes has a version checker for generated modules.
When the version of comtypes changes and a cached module under the gen directory was generated by an incompatible version, an ImportError is raised. The client catches this error, parses the library, emits warnings and regenerates the module.

Warnings can break consumers running with -W error.

I understand this concern. However, users who run Python with -W error have explicitly chosen to treat warnings as errors.
They can use warnings.filterwarnings() or an equivalent mechanism to filter this specific warning if they decide that it should not stop their application.

Use an explicit, filterable warning category and keep the issue link and library/member details.

If the built-in UserWarning category is too broad, we may need a custom warning class that inherits from a built-in warning category but adds no methods or attributes.

Consider a per-library summary with opt-in per-member diagnostics.

In practice, I am considering a feature flag that lets users choose which mapping they want to use. This would make the behavior consistent across the type libraries being parsed.
If users need different mappings for individual fields or arguments, they may have to maintain their own record or interface definitions in their project.
We should discuss whether such libraries and use cases actually exist, and whether supporting them would be reasonable.

I would prefer a warning in a transition release before changing the default rather than treating a simultaneous warning as a compatibility solution.

I agree that a warning alone is not a complete compatibility solution. In addition to warnings, I am considering introducing a feature flag to make the migration more manageable.
Here is what I found about the import process.

Importing comtypes does not indirectly import comtypes.tools.
In a typical use case, comtypes.tools is initialized when the user imports comtypes.client.
For example, the following code shows the additional modules loaded by import comtypes.client:

>>> import sys
>>> import pprint
>>> import comtypes
>>> pre_importing_client = {k for k in sys.modules if k.startswith('comtypes')}
>>> pprint.pprint(pre_importing_client)
{'comtypes',
 'comtypes.GUID',
 'comtypes._comobject',
 'comtypes._memberspec',
 'comtypes._meta',
 'comtypes._npsupport',
 'comtypes._post_coinit',
 'comtypes._post_coinit._cominterface_meta_patcher',
 'comtypes._post_coinit.activeobj',
 'comtypes._post_coinit.bstr',
 'comtypes._post_coinit.instancemethod',
 'comtypes._post_coinit.misc',
 'comtypes._post_coinit.unknwn',
 'comtypes._safearray',
 'comtypes._tlib_version_checker',
 'comtypes._vtbl',
 'comtypes.automation',
 'comtypes.errorinfo',
 'comtypes.hresult',
 'comtypes.patcher',
 'comtypes.safearray',
 'comtypes.typeinfo'}
>>> import comtypes.client
>>> post_importing_client = {k for k in sys.modules if k.startswith('comtypes')}
>>> pprint.pprint(post_importing_client - pre_importing_client)
{'comtypes.client',
 'comtypes.client._activeobj',
 'comtypes.client._code_cache',
 'comtypes.client._constants',
 'comtypes.client._create',
 'comtypes.client._events',
 'comtypes.client._generate',
 'comtypes.client._managing',
 'comtypes.client.dynamic',
 'comtypes.client.lazybind',
 'comtypes.connectionpoints',
 'comtypes.gen',
 'comtypes.tools',
 'comtypes.tools.codegenerator',
 'comtypes.tools.codegenerator.codegenerator',
 'comtypes.tools.codegenerator.heads',
 'comtypes.tools.codegenerator.helpers',
 'comtypes.tools.codegenerator.modulenamer',
 'comtypes.tools.codegenerator.namespaces',
 'comtypes.tools.codegenerator.packing',
 'comtypes.tools.codegenerator.typeannotator',
 'comtypes.tools.tlbparser',
 'comtypes.tools.typedesc',
 'comtypes.tools.typedesc_base'}

Based on this import flow, I am considering defining a flag such as _PARSE_VT_I1_AS_SCHAR_ in a place accessible after the initial import comtypes.
When comtypes.client is imported, the parser could read the flag and select the mapping that will remain in effect for the lifetime of the Python interpreter.

For example:

# comtypes/__init__.py
_PARSE_VT_I1_AS_SCHAR_ = False

The mapping in tlbparser.py would then look something like this:

COMTYPES = {
    ...
    # Signed-byte mapping and compatibility:
    # https://github.com/enthought/comtypes/issues/969
    automation.VT_I1: (
        schar_type if comtypes._PARSE_VT_I1_AS_SCHAR_ else char_type
    ),
    ...
}

This is only a proposal, and I would like to hear your thoughts on it.

The prototype's _uses_vt_i1 does not follow VT_USERDEFINED, so alias/reference cases deserve a check.

You are right. I overlooked this case.

Overall, I think the mapping change is technically justified, but we should make the compatibility risks visible and give users a practical way to control the transition.
I would like to discuss the warning and feature-flag approaches further before deciding on the migration strategy.

Thank you!

I agree with retaining the current default and making an opt-in transition a separate step. #962 now keeps COMTYPES[VT_I1] unchanged and records the current failures and handwritten proposed behavior; its migration page has been removed.

An interpreter-wide flag selected before importing comtypes.client sounds easier to explain and test than per-field choices. I do not have evidence of an application that needs different mappings for individual fields, so I would not add that mechanism based on the library survey alone.

The generated-module cache needs an explicit policy alongside the flag. In the current source, GetModule returns _get_existing_module(tlib) before invoking the generator, and generated names depend on library identity/version, without a mapping selector. Selecting a new parser mapping therefore does not itself change an existing generated module. Even a fresh interpreter can load wrappers written under the other setting. We should either isolate/identify caches by the selected mapping or document and test a regeneration procedure, including imported dependent libraries; once wrappers are loaded, changing the setting should not imply they have changed. A later test could generate the same library under each setting in separate processes with empty and reused caches, then assert actual declarations and calls.

Your point about -W error is fair: that is the user's explicit policy, and a dedicated warning category gives them a narrow filter. My concern is documenting the behavior and allowing a targeted filter, not overriding that policy. A summary versus detailed member reporting can remain a separate diagnostics choice; the mapping flag controls semantics.

The previous real-library survey remains declaration/local-record evidence, not an end-to-end SAPI/PortableDevice/CLR/event test. The bytes-versus-integer field values and counted-buffer truncation show why preserving defaults during this discussion is useful, but do not establish prevalence or prove a complete user operation is fixed.

This is feedback on the proposal, not an implementation or validation of a flag. AI assistance: OpenAI Codex.

Thank you so much for your deep research and insights.
I would like to share some thoughts and ideas from a maintainer's point of view.

The generated-module cache needs an explicit policy alongside the flag.

We should either isolate/identify caches by the selected mapping or document

If we try to completely separate caches for each mapping, it will make the code too complex to maintain. Compared to choices like the No-GIL versus normal versions of CPython (where the massive benefits justify the complexity), this kind of opt-in feature for the cache doesn't seem to offer enough benefit to justify the extra complexity.

Instead, as a way to identify them, we think it would be good enough if the code generator just adds a small comment in the generated files, like this:

# This code was generated with comtypes=={comtypes.__version__}, _PARSE_VT_I1_AS_SCHAR_=={comtypes._PARSE_VT_I1_AS_SCHAR_}

For the regeneration guide, we can explain that users should do the following steps in the first __init__.py file when their app starts up:

Clear the cache (doing something like comtypes.clear_cache)
import comtypes and set the flag (for example, comtypes._PARSE_VT_I1_AS_SCHAR_ = ...)
import comtypes.client and call client.GetModule(...) as needed

However, comtypes.clear_cache is designed for command-line use rather than being called from scripts, so we would need to refactor it if applications want to call it from their code. Also, because it imports client internally, they need to set the mapping flag after importing comtypes and before calling clear_cache.
If a project uses a package like pywinauto (which needs changes to sys.coinit_flags to work), the application side will need to handle the correct startup order.
Example:

# myapp/__init__.py
# import sys
# com_init_mode = 0  # COINIT_MULTITHREADED = 0x0
# sys.coinit_flags = com_init_mode
import comtypes

comtypes._PARSE_VT_I1_AS_SCHAR_ = ...
# comtypes.clear_cache.clear()  # calling the refactored function
import comtypes.client
comtypes.client.GetModule(...)
# import pkg_requiring_multithread

and test a regeneration procedure, including imported dependent libraries

A later test could generate the same library under each setting in separate processes with empty and reused caches, then assert actual declarations and calls.

Even if we only support an interpreter-wide flag instead of flags for each library or parameter, this test is still possible.
For the tests, we think the test helper patch_gen_dir (which we already made in test_client_regenerate_modules.py) will be very helpful.
In a similar way, we can temporarily remove tools and gen from the import system, patch the flag, and call GetModule to test the flag changes safely in separate processes.
However, since import system helpers can get pretty complicated, we should add good comments explaining how and why it works, along with clear assert checks to make sure it behaves as expected.

...a dedicated warning category gives them a narrow filter.

Do you have any ideas for what built-in type to use as the base for the custom warning, or what name we should give it?

...not an end-to-end SAPI/PortableDevice/CLR/event test.

If there are any technical blockers that stop us from running E2E tests for SAPI, PortableDevice, CLR, or events on CI, could you please let us know?

Navigation Menu

Repository navigation

Fix the `tlbparser` ctypes mapping for `VT_I1`. #969

Summary

Proposed Changes

Problems

1. NumPy support

2. The relation between COM types and ctypes types

3. Long-standing behavior and backward compatibility

Completion criteria

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Fix the tlbparser ctypes mapping for VT_I1. #969

Description

Summary

Proposed Changes

Problems

1. NumPy support

2. The relation between COM types and ctypes types

3. Long-standing behavior and backward compatibility

Completion criteria

Activity

junkmd commented on Oct 17, 2026

haoxuanjng-lang commented on Oct 17, 2026

junkmd commented on Oct 17, 2026

haoxuanjng-lang commented on Oct 18, 2026

junkmd commented on Oct 19, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Fix the `tlbparser` ctypes mapping for `VT_I1`. #969