Have different names for testsuites in junit.#318
Open
rhopp wants to merge 1 commit into
Open
Conversation
📝 WalkthroughWalkthroughThis PR enhances the TSSC E2E testing framework by injecting Playwright JUnit metadata environment variables into Tekton task steps, introducing a custom JUnit reporter that embeds project and context information into XML output, and updating test suite descriptions for clarity. Changes
Sequence Diagram(s)sequenceDiagram
actor Tekton
participant TestTask as Tekton Task<br/>(tssc-e2e.yaml)
participant PlaywrightRunner as Playwright<br/>Test Runner
participant CustomReporter as JUnitWithProject<br/>Reporter
participant FileSystem as File System
Tekton->>TestTask: Inject PLAYWRIGHT_JUNIT_SUITE_NAME<br/>PLAYWRIGHT_JUNIT_SUITE_ID<br/>(from pipelineRun/taskRun labels)
TestTask->>PlaywrightRunner: Execute with env vars
PlaywrightRunner->>PlaywrightRunner: Run tests
PlaywrightRunner->>CustomReporter: onStart(config, suite)
PlaywrightRunner->>PlaywrightRunner: Process test results
PlaywrightRunner->>CustomReporter: onEnd()
CustomReporter->>CustomReporter: Aggregate per-project suites<br/>Build testcase entries<br/>Classify failures/errors<br/>Serialize to JUnit XML
CustomReporter->>FileSystem: Write XML to junit.xml
FileSystem-->>CustomReporter: Success
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Review Summary by QodoImplement custom JUnit reporter with unique suite names WalkthroughsDescription• Implement custom JUnit reporter using describe block names • Combine top-level describe names with project names for unique suite identifiers • Rename test describe blocks for cleaner, more meaningful suite names • Add Kubernetes Downward API environment variables to Tekton tasks Diagramflowchart LR
A["Playwright Config"] -->|uses| B["Custom JUnit Reporter"]
B -->|reads| C["Describe Block Names"]
B -->|reads| D["Project Names"]
B -->|reads| E["Env Variables"]
C -->|combines with| D
D -->|creates| F["Unique Suite Names"]
E -->|populates| G["Root Testsuites Element"]
H["Tekton Tasks"] -->|inject| E
F -->|generates| I["JUnit XML"]
G -->|generates| I
File Changes1. src/reporters/junit-with-project.ts
|
Code Review by Qodo
1. Totals not reset
|
Collaborator
Author
|
/retest |
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
Comment on lines
+37
to
+54
| private totalTests = 0; | ||
| private totalFailures = 0; | ||
| private totalSkipped = 0; | ||
| private totalErrors = 0; | ||
| private stripANSIControlSequences = false; | ||
| private includeProjectInTestName = false; | ||
|
|
||
| constructor(options: JUnitOptions = {}) { | ||
| this.outputFile = options.outputFile || 'test-results/junit.xml'; | ||
| this.stripANSIControlSequences = !!options.stripANSIControlSequences; | ||
| this.includeProjectInTestName = !!options.includeProjectInTestName; | ||
| } | ||
|
|
||
| onBegin(config: FullConfig, suite: Suite): void { | ||
| this.config = config; | ||
| this.suite = suite; | ||
| this.timestamp = new Date(); | ||
| } |
There was a problem hiding this comment.
1. Totals not reset 🐞 Bug ⛯ Reliability
The reporter’s aggregate counters are instance state that never reset in onBegin(), so repeated runs in the same process (e.g., Playwright UI mode) can inflate the root <testsuites> counts. This can lead CI/JUnit consumers to display incorrect totals.
Agent Prompt
### Issue description
The custom JUnit reporter keeps aggregate counters as instance fields and increments them during a run, but does not reset them in `onBegin()`. If Playwright reuses the reporter instance across multiple runs (e.g., UI mode), the root `<testsuites>` counts will accumulate and become incorrect.
### Issue Context
Counters are declared as fields and used to populate the root element attributes.
### Fix Focus Areas
- src/reporters/junit-with-project.ts[37-54]
- src/reporters/junit-with-project.ts[108-112]
- src/reporters/junit-with-project.ts[64-74]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
|
@rhopp: The following test has Failed, say /retest to rerun failed tests.
Inspecting Test ArtifactsTo inspect your test artifacts, follow these steps:
mkdir -p oras-artifacts
cd oras-artifacts
oras pull quay.io/konflux-test-storage/rhtap-team/rhtap-cli:e2e-4.20-wn5lxTest results analysis<not enabled> OCI Artifact Browser URL<not enabled> |
jkopriva
approved these changes
Mar 10, 2026
BohdanMar
approved these changes
Mar 10, 2026
…ect names Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
rhopp
changed the title
coderabbitai
Bot
added
enhancement
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/reporters/junit-with-project.ts`:
- Around line 58-60: The reporter currently groups tests per file by iterating
projectSuite.suites (fileSuite) and calling _buildTestSuite once per file, which
names the testsuite using suite.suites[0].title and causes multiple top-level
describe() blocks in the same file to be merged under the first name; change the
logic to split suites by top-level describe blocks: for each projectSuite
(project) iterate each fileSuite and then iterate fileSuite.suites (each
top-level describe) and call _buildTestSuite for each top-level describe suite
(passing the correct titles/parents) so each top-level describe becomes its own
<testsuite>; apply the same change to the similar loop/logic referenced in the
95-120 region to ensure consistent splitting and accurate suite names.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: f6092633-a02b-4e28-a918-2d07619e05bf
📒 Files selected for processing (7)
integration-tests/tasks/tssc-e2e.yamlintegration-tests/tasks/tssc-ui.yamlplaywright.config.tssrc/reporters/junit-with-project.tstests/tssc/full_workflow.test.tstests/ui/component.test.tstests/ui/gitopsResource.test.ts
Comment on lines
+58
to
+60
| for (const projectSuite of this.suite.suites) { | ||
| for (const fileSuite of projectSuite.suites) | ||
| children.push(await this._buildTestSuite(projectSuite.title, fileSuite)); |
There was a problem hiding this comment.
Split JUnit suites by top-level describe, not by file.
Right now the reporter builds one <testsuite> per file and then names it from suite.suites[0].title. If a file has multiple top-level test.describe() blocks, every testcase in that file is reported under the first block’s name, so suite names become inaccurate and can still collide in CI.
Suggested fix
async onEnd(result: FullResult): Promise<void> {
const children: XMLEntry[] = [];
for (const projectSuite of this.suite.suites) {
- for (const fileSuite of projectSuite.suites)
- children.push(await this._buildTestSuite(projectSuite.title, fileSuite));
+ for (const fileSuite of projectSuite.suites) {
+ const suitesToReport = fileSuite.suites.length ? fileSuite.suites : [fileSuite];
+ for (const topLevelSuite of suitesToReport)
+ children.push(await this._buildTestSuite(projectSuite.title, topLevelSuite));
+ }
}
@@
- const topLevelDescribe = suite.suites.length > 0 ? suite.suites[0].title : '';
- const suiteLabel = topLevelDescribe || suite.title;
+ const suiteLabel = suite.title;
const suiteName = projectName
? `${suiteLabel} - ${projectName}`
: suiteLabel;Also applies to: 95-120
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/reporters/junit-with-project.ts` around lines 58 - 60, The reporter
currently groups tests per file by iterating projectSuite.suites (fileSuite) and
calling _buildTestSuite once per file, which names the testsuite using
suite.suites[0].title and causes multiple top-level describe() blocks in the
same file to be merged under the first name; change the logic to split suites by
top-level describe blocks: for each projectSuite (project) iterate each
fileSuite and then iterate fileSuite.suites (each top-level describe) and call
_buildTestSuite for each top-level describe suite (passing the correct
titles/parents) so each top-level describe becomes its own <testsuite>; apply
the same change to the similar loop/logic referenced in the 95-120 region to
ensure consistent splitting and accurate suite names.
|
Scenario: pr-e2e-tests
Inspecting Test ArtifactsTo inspect your test artifacts, follow these steps:
mkdir -p oras-artifacts
cd oras-artifacts
oras pull quay.io/konflux-test-storage/rhtap-team/rhtap-cli:e2e-4.20-pdqwvTest results analysis<not enabled> OCI Artifact Browser URL<not enabled> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Playwright's built-in JUnit reporter names test suites after file paths, which results in duplicate suite names in the generated XML. This causes problems when CI tools try to aggregate or display test results — suites with the same name get merged or become indistinguishable.
Changes
Summary by CodeRabbit
New Features
Tests