Tests - Add test cases for pre-process modules in GPT-like models to numerical tests#76
Open
yzygitzh wants to merge 1 commit into
Open
Tests - Add test cases for pre-process modules in GPT-like models to numerical tests#76yzygitzh wants to merge 1 commit into
yzygitzh wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR adds numerical test coverage for the embedding layer component of GPT-like models by introducing a new test file for the LanguageModelEmbedding module. The test validates the embedding layer's forward and backward passes through multiple training steps.
- Adds comprehensive test case for LanguageModelEmbedding with parameterized configurations
- Tests embedding layer with RoPE position embeddings and various model configurations
- Implements multi-step training loop to verify gradient computation and parameter updates
Comments suppressed due to low confidence (2)
tests/numerical_tests/modules/test_embedding.py:27
- The test only covers a single attention head configuration. Consider adding test cases with multiple attention heads to ensure the embedding layer works correctly with different model architectures.
'num_attention_heads': 1,
tests/numerical_tests/modules/test_embedding.py:35
- The test only covers RoPE position embeddings. Consider adding test cases for other position embedding types (e.g., 'learned', 'alibi') to ensure comprehensive coverage of the embedding layer's position encoding functionality.
'position_embedding_type': 'rope',
|
Change the title "Tests - Add test cases for embedding layer in transformer-based models." |
cp5555
reviewed
Jul 28, 2025
| ) | ||
| output = model(*inputs) | ||
| loss = output.mean() | ||
| loss.backward() |
There was a problem hiding this comment.
Will parameters of embedding layer be changed? I think it should be no?
abuccts
approved these changes
Jul 28, 2025
| from tests.numerical_tests.modules.test_module import TestModule | ||
|
|
||
|
|
||
| class TestLanguageModelEmbedding(TestModule): |
Member
There was a problem hiding this comment.
seems this test is not enabled in current pipeline?
|
Marking as stale. No activity in 60 days. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds test cases for pre-process modules in GPT-like models to numerical tests. Including: