So far we have tested various models from well-known projects: torchvision, torchaudio, transformers, etc.
Exporting such models often leads us to discover and fix serialization issues. Still, we have not tested against more than a few hundred architectures. The goal of this effort is to gain a more statistical view of how many architectures pass export and which have specific issues, thus prioritizing the bugs to fix.
To gain this knowledge, a large zoo of different model architectures needs to be collected and tested. It seems the PyTorch maintainers have such a collected dataset, as they refer to it in some GitHub issues.
This would not be part of the test suite but rather a standalone tool to collect that information.
So far we have tested various models from well-known projects: torchvision, torchaudio, transformers, etc.
Exporting such models often leads us to discover and fix serialization issues. Still, we have not tested against more than a few hundred architectures. The goal of this effort is to gain a more statistical view of how many architectures pass export and which have specific issues, thus prioritizing the bugs to fix.
To gain this knowledge, a large zoo of different model architectures needs to be collected and tested. It seems the PyTorch maintainers have such a collected dataset, as they refer to it in some GitHub issues.
This would not be part of the test suite but rather a standalone tool to collect that information.