Skip to content

Latest commit

ย 

History

History
714 lines (574 loc) ยท 26.5 KB

File metadata and controls

714 lines (574 loc) ยท 26.5 KB

๐Ÿ” Black Trigram (ํ‘๊ด˜) - End-to-End Test Plan

๐Ÿ” ISMS Alignment: This E2E test plan implements Hack23 Secure Development Policy Section 4.3.3 - End-to-End Testing Requirements.

๐Ÿ“‹ Executive Summary

This End-to-End (E2E) Test Plan provides comprehensive testing coverage for the Black Trigram Korean martial arts combat simulator, ensuring all critical user journeys and game mechanics function correctly across different browsers, screen sizes, and performance scenarios.

ISMS Compliance Requirements

Per Hack23 AB's Secure Development Policy, this project maintains:

๐ŸŽฏ Requirement ๐Ÿ“Š Implementation โœ… Status ๐Ÿ“‹ ISMS Reference
Critical Path Coverage Complete game flow testing โœ… Implemented Section 4.3.3.1
Browser Testing Chrome, Firefox, Edge โœ… Validated Section 4.3.3.2
Automated Execution Every PR via Cypress โœ… Active Section 4.3.3.3
Public Reporting Mochawesome reports โœ… Published Section 4.3.3.4
Performance Assertions Load time & FPS monitoring โœ… Tracked Section 4.3.3.5

Evidence Links:

See Also:


๐ŸŽฏ Testing Objectives

  • ๐Ÿฅ‹ Validate Critical Game Workflows: Ensure all Korean martial arts gameplay paths work correctly
  • ๐ŸŽฎ Verify Combat Mechanics: Test trigram system, vital point targeting, and combat calculations
  • ๐Ÿ”„ Confirm State Management: Validate game state persistence and transitions
  • ๐ŸŒ Cross-Browser Compatibility: Test across Chrome, Firefox, and Edge browsers
  • ๐Ÿ“ฑ Responsive Design Validation: Ensure proper functionality on desktop, tablet, and mobile viewports
  • โšก Performance Verification: Confirm 60fps rendering and acceptable load times
  • ๐ŸŽจ PixiJS Integration: Validate WebGL rendering and canvas interactions
  • ๐ŸŽต Audio System: Test Korean traditional music and combat sound effects

๐Ÿงฉ Test Categories

1. ๐Ÿ›๏ธ Core Game Journey Workflows

Test Coverage:

  • Complete navigation cycle: Intro โ†’ Training โ†’ Combat โ†’ Intro
  • All 8 trigram stances in training mode
  • Combat mechanics with movement and techniques
  • State management and transitions
  • Performance under load

Critical User Journeys:

Journey Priority Test File Status
Intro โ†’ Training โ†’ Intro โœ… High game-journey.cy.ts Implemented
Intro โ†’ Combat โ†’ Intro โœ… High game-journey.cy.ts Implemented
Training - All Stances โœ… High training-flow.cy.ts Implemented
Combat - Movement โœ… High combat-flow.cy.ts Implemented
Combat - Trigram Techniques โœ… High combat-system-integration.cy.ts Implemented
Combat - Vital Points โœ… High pixi-korean-martial-arts.cy.ts Implemented

2. โš”๏ธ Combat System Integration Workflows

Combat Mechanics Tested:

  • Movement system (WASD + Arrow keys)
  • Trigram stance selection (1-8 keys)
  • Technique execution (Space bar)
  • Vital point targeting (Mouse clicks)
  • Damage calculations (VitalPointSystem)
  • Health/Ki resource management
  • Combat feedback (visual + audio)
  • AI opponent behavior

3. ๐ŸŽจ PixiJS Rendering and Performance

Performance Assertions:

Metric Target Test Method
Initial Load Time < 3 seconds Measure time from visit() to canvas ready
Screen Transition < 500ms Time between ESC and new screen render
Input Response < 100ms Key press to visual feedback
Combat Action < 200ms Technique execution to effect render
Frame Rate 60fps ยฑ5% Monitor FPS during intense combat
Memory Usage < 512MB Check heap size after 5 minutes

4. ๐Ÿ“ฑ Responsive Design and Browser Compatibility

Browser Testing Matrix:

Browser Version Platform WebGL Support Three.js Compatibility Priority Status
Chrome Latest Stable Windows/Mac/Linux โœ… Excellent (SwiftShader) โœ… Excellent โœ… High Tested
Firefox Latest Stable Windows/Mac/Linux โœ… Excellent โœ… Excellent โœ… High Tested
Edge Latest Stable Windows โœ… Excellent (Chromium) โœ… Excellent โœ… High Tested
Safari Latest Stable Mac โœ… Good โœ… Good โš ๏ธ Medium Planned

Chrome Configuration for Three.js (See CHROME_CYPRESS_THREEJS_CONFIG.md for details):

  • SwiftShader software rendering for WebGL in headless mode
  • Optimized Chrome flags for Three.js performance
  • 60fps frame rate capping for consistency
  • 4GB Node.js heap for memory-intensive Three.js scenes

Viewport Testing Matrix:

Device Type Width x Height Orientation Priority
Desktop HD 1280 x 720 Landscape โœ… High
Desktop FHD 1920 x 1080 Landscape โœ… High
Tablet 768 x 1024 Portrait โš ๏ธ Medium
Mobile 375 x 667 Portrait ๐Ÿ“ฑ Low

5. ๐ŸŽต Audio System Integration

Audio Features Tested:

  • Traditional Korean music in Training Mode
  • Cyberpunk combat music in Combat Mode
  • Impact sounds for technique execution
  • Bone crack audio for critical hits
  • Audio synchronization with visuals

๐Ÿงช Test Implementation with Cypress

Test Structure

cypress/
โ”œโ”€โ”€ e2e/
โ”‚   โ”œโ”€โ”€ game-journey.cy.ts           # Complete game flow tests
โ”‚   โ”œโ”€โ”€ core-features.cy.ts          # Essential gameplay features
โ”‚   โ”œโ”€โ”€ combat-system-integration.cy.ts  # Combat mechanics
โ”‚   โ”œโ”€โ”€ training-system-integration.cy.ts # Training mode
โ”‚   โ”œโ”€โ”€ combat-flow.cy.ts            # Combat-specific workflows
โ”‚   โ”œโ”€โ”€ training-flow.cy.ts          # Training-specific workflows
โ”‚   โ”œโ”€โ”€ combat-screen-layout.cy.ts   # Combat UI validation
โ”‚   โ”œโ”€โ”€ combat-mode.cy.ts            # Combat mode entry/exit
โ”‚   โ”œโ”€โ”€ training-mode.cy.ts          # Training mode entry/exit
โ”‚   โ”œโ”€โ”€ pixi-korean-martial-arts.cy.ts # PixiJS integration (legacy)
โ”‚   โ”œโ”€โ”€ performance-threejs.cy.ts    # Three.js FPS and performance
โ”‚   โ”œโ”€โ”€ webgl-verification.cy.ts     # WebGL context and rendering verification โœจ NEW
โ”‚   โ”œโ”€โ”€ app.cy.ts                    # Application smoke tests
โ”‚   โ””โ”€โ”€ game-flow.cy.ts              # Navigation flows
โ”œโ”€โ”€ fixtures/
โ”‚   โ””โ”€โ”€ test-data.json               # Test data fixtures
โ”œโ”€โ”€ support/
โ”‚   โ”œโ”€โ”€ e2e.ts                       # E2E support file
โ”‚   โ”œโ”€โ”€ component.ts                 # Component support
โ”‚   โ”œโ”€โ”€ commands.ts                  # Custom Cypress commands
โ”‚   โ”œโ”€โ”€ performance.ts               # Performance monitoring utilities
โ”‚   โ”œโ”€โ”€ test-isolation.ts            # Test isolation utilities
โ”‚   โ””โ”€โ”€ resource-monitoring.ts       # Resource leak detection

WebGL Verification Tests โœจ NEW

Test File: webgl-verification.cy.ts

Comprehensive WebGL and Three.js verification tests to ensure proper rendering in headless CI:

Coverage:

  1. WebGL Context Verification

    • Verify WebGL/WebGL2 context creation
    • Check renderer, version, and capabilities
    • Validate texture size and viewport limits
    • Test WebGL extensions support
  2. Frame Rate Performance

    • Measure average FPS (target: 30-60fps)
    • Monitor frame timing consistency
    • Detect frame drops or stuttering
    • Validate software rendering performance
  3. Rendering Quality

    • Verify active canvas rendering (pixel changes)
    • Check for blank/black canvas issues
    • Validate visual content is present
    • Test canvas state changes
  4. Memory and Resource Management

    • Detect WebGL context leaks
    • Monitor GPU memory usage
    • Track memory growth during gameplay
    • Verify resource cleanup
  5. Mode-Specific Testing

    • Combat mode WebGL performance
    • Training mode WebGL performance
    • Screen transition stability
    • Context preservation across scenes

Running WebGL Tests:

# Full WebGL verification
npx cypress run --spec "cypress/e2e/webgl-verification.cy.ts"

# With memory optimization
NODE_OPTIONS="--max-old-space-size=4096" \
  npx cypress run --spec "cypress/e2e/webgl-verification.cy.ts"

For detailed configuration and troubleshooting, see CHROME_CYPRESS_THREEJS_CONFIG.md.

Custom Cypress Commands

Available Commands:

  • cy.visitWithWebGLMock() - Visit page with WebGL mocking for headless
  • cy.waitForCanvasReady() - Wait for PixiJS canvas initialization
  • cy.gameActions() - Execute game inputs with proper timing
  • cy.annotate() - Add test annotations for reporting
  • cy.enterTrainingMode() - Navigate to training mode
  • cy.enterCombatMode() - Navigate to combat mode
  • cy.returnToIntro() - Return to intro screen
  • cy.practiceStance() - Practice specific trigram stance

๐Ÿ“Š Test Data Management

Game State Test Data

Test fixtures are defined in cypress/fixtures/test-data.json:

Player Archetypes:

  • ๋ฌด์‚ฌ (Musa) - Traditional Warrior
  • ์•”์‚ด์ž (Amsalja) - Shadow Assassin
  • ํ•ด์ปค (Hacker) - Cyber Warrior
  • ์ •๋ณด์š”์› (Jeongbo Yowon) - Intelligence Operative
  • ์กฐ์งํญ๋ ฅ๋ฐฐ (Jojik Pokryeokbae) - Organized Crime

Trigram Stances:

  • ๊ฑด (Geon) - Heaven
  • ํƒœ (Tae) - Lake
  • ๋ฆฌ (Li) - Fire
  • ์ง„ (Jin) - Thunder
  • ์† (Son) - Wind
  • ๊ฐ (Gam) - Water
  • ๊ฐ„ (Gan) - Mountain
  • ๊ณค (Gon) - Earth

๐Ÿ“Š Test Reporting with Mochawesome

Report Generation Process

1. Test Execution:

npm run test:e2e        # Local execution with Chrome
npm run test:e2e:ci     # CI execution (headless)

2. Report Merging:

npm run test:e2ereportmerge    # Merge JSON reports
npm run test:e2ereporthtmlall  # Generate HTML report

3. Report Location (Local):

  • Individual JSON: docs/cypress/mochawesome/*.json
  • Merged JSON: docs/cypress/mochawesome-all.json
  • HTML Report: docs/cypress/mochawesome/index.html
  • JUnit XML: docs/cypress/junit/results-*.xml

4. CI Artifacts: CI uploads artifacts from the following locations (configured in cypress.config.ts to write to docs/cypress/ but uploaded from standard Cypress paths):

  • Videos: cypress/videos/ (failures only) โ†’ uploaded to docs/cypress/videos/
  • Screenshots: cypress/screenshots/ (failures only) โ†’ uploaded to docs/cypress/screenshots/
  • Results: cypress/results/ โ†’ uploaded to docs/cypress/results/
  • Reports: Available as downloadable artifacts from GitHub Actions workflow runs

Mochawesome Report Contents

  • โœ… Test pass/fail summary with percentages
  • ๐Ÿ“Š Interactive charts showing test distribution
  • ๐Ÿ–ผ๏ธ Embedded screenshots for failed tests
  • ๐ŸŽฌ Video playback for failed test runs
  • โฑ๏ธ Execution time for each test suite
  • ๐Ÿ” Detailed error messages and stack traces

๐Ÿ”„ CI/CD Integration

GitHub Actions Workflow

E2E Test Job:

e2e-tests:
  needs: [prepare, build-validation]
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v5
    - uses: actions/setup-node@v6
      with:
        node-version-file: '.nvmrc'
    - name: Install dependencies
      run: npm ci
    - name: Start app and run Cypress tests
      run: |
        xvfb-run --auto-servernum --server-args="-screen 0 1280x720x24" npm run test:e2e
    - name: Upload Cypress results
      if: always()
      uses: actions/upload-artifact@v5
      with:
        name: cypress-results
        path: |
          cypress/videos
          cypress/screenshots
          cypress/results

Trigger Conditions:

  • โœ… Every push to main branch
  • โœ… Every pull request to main branch
  • โš™๏ธ Manual workflow dispatch (optional)

Environment:

  • OS: Ubuntu Latest (Linux)
  • Node: from .nvmrc
  • Display: Xvfb (virtual framebuffer for headless)
  • Resolution: 1280x720x24 bit color depth
  • Browser: Chrome (headless)

Dependencies:

  • Korean font support: fonts-noto-cjk, fonts-noto-cjk-extra
  • System libraries: libgtk-3-0, libgbm-dev, libnss3, etc.
  • google-chrome-stable

๐Ÿ“‹ Test Coverage and Success Criteria

Critical User Journeys Coverage

Journey Coverage Tests Status
Intro โ†’ Training โ†’ Intro 100% game-journey.cy.ts โœ…
Intro โ†’ Combat โ†’ Intro 100% game-journey.cy.ts โœ…
Training Mode - All Stances 100% training-flow.cy.ts โœ…
Combat - Movement 100% combat-flow.cy.ts โœ…
Combat - Trigram Stances 100% combat-system-integration.cy.ts โœ…
Combat - Vital Points 100% pixi-korean-martial-arts.cy.ts โœ…
Responsive Design 100% core-features.cy.ts โœ…
Cross-Browser 100% All tests โœ…

Test Execution KPIs

Metric Target Current (Enhanced) Previous Status
Test Coverage >90% of user journeys ~95% ~95% โœ… Excellent
Pass Rate >95% 100% (10/10) ~98% โœ… Excellent
Execution Time 10-12 minutes ~10-12 min (optimized) ~20 min โœ… Target Met
Browser Compatibility 100% Chrome/Firefox/Edge 100% 100% โœ… Complete
Performance Compliance <3s load, <500ms interaction Measured Measured โœ… Met
Flaky Test Rate 0% 0% (enhanced monitoring) <1% โœ… Zero Flakes
Retry Count โ‰ค1 1 (runMode), 0 (openMode) 2 (runMode), 1 (openMode) โœ… Improved
Resource Leaks 0 detected Monitored (10-30MB growth) Not monitored โœ… Tracked
Test Isolation Complete โœ… Implemented Partial โœ… Complete

Enhanced Reliability Features (Nov 2025):

  • โœ… Test Isolation: Complete state reset between tests (localStorage, sessionStorage, game state)
  • โœ… Resource Monitoring: Tracks audio elements, canvas elements, memory usage, event listeners
  • โœ… Global Cleanup Hooks: Automatic cleanup in beforeEach/afterEach for all tests
  • โœ… Flakiness Detection: CI runs tests 3x on PRs to detect inconsistent behavior
  • โœ… Reliability Reporting: Automated HTML report generation with flaky test detection
  • โœ… Reduced Retries: From 2โ†’1 (runMode), 1โ†’0 (openMode) to catch flakes faster

Note: Execution time optimized from ~20 minutes to 10-12 minutes through configuration tuning, wait time reduction, and performance monitoring. Test reliability improvements (Nov 2025) added comprehensive isolation and monitoring to achieve 0% flaky rate. See E2E_OPTIMIZATION_RESULTS.md for details.

Feature Coverage Matrix

Feature Unit Tests E2E Tests Status
Trigram System โœ… TrigramCalculator.test.ts โœ… combat-system-integration.cy.ts Complete
Vital Points โœ… VitalPointSystem.test.ts โœ… pixi-korean-martial-arts.cy.ts Complete
Combat Mechanics โœ… GameIntegration.test.ts โœ… combat-flow.cy.ts Complete
Player Utils โœ… playerUtils.test.ts โœ… game-journey.cy.ts Complete
UI Components โœ… CombatStatsPanel.test.tsx โœ… combat-screen-layout.cy.ts Complete
State Management โœ… Integration tests โœ… game-journey.cy.ts Complete

๐ŸŽฎ Game Screen Test Scenarios

IntroScreen (์ธํŠธ๋กœ ํ™”๋ฉด)

Test Coverage:

  • โœ… Korean logo rendering (ํ‘๊ด˜)
  • โœ… Menu navigation (Training/Combat buttons)
  • โœ… Bilingual text display (Korean/English)
  • โœ… Background visual effects
  • โœ… Audio system initialization
  • โœ… Keyboard navigation (1, 2 keys)

Test Files: app.cy.ts, core-features.cy.ts, game-journey.cy.ts

TrainingScreen (ํ›ˆ๋ จ ๋ชจ๋“œ)

Test Coverage:

  • โœ… All 8 trigram stance practice
  • โœ… Korean technique names display
  • โœ… Visual technique feedback
  • โœ… Training progression tracking
  • โœ… Return to intro (ESC)

Test Files: training-mode.cy.ts, training-flow.cy.ts, training-system-integration.cy.ts

CombatScreen (์ „ํˆฌ ํ™”๋ฉด)

Test Coverage:

  • โœ… Combat HUD rendering
  • โœ… Health bar display (player + opponent)
  • โœ… Ki resource management
  • โœ… Movement controls (WASD, arrows)
  • โœ… Trigram stance selection (1-8)
  • โœ… Technique execution (Space)
  • โœ… Vital point targeting (Mouse)
  • โœ… Combat log updates
  • โœ… Damage calculations
  • โœ… Visual effects (blood, impacts)
  • โœ… Audio feedback (bone impacts)

Test Files: combat-mode.cy.ts, combat-flow.cy.ts, combat-system-integration.cy.ts, combat-screen-layout.cy.ts, pixi-korean-martial-arts.cy.ts

๐Ÿ”„ Test Maintenance Strategy

Test Review Schedule

  • ๐Ÿ“… Weekly: Review failing tests and identify flaky tests
  • ๐Ÿ”„ Bi-weekly: Update test scenarios based on new features
  • ๐Ÿ“Š Monthly: Comprehensive test coverage analysis
  • ๐ŸŽฏ Quarterly: Test framework and strategy evaluation

Flaky Test Handling

  • ๐Ÿ” Retries: Configured in cypress.config.ts
    • runMode: 2 - CI environment retries
    • openMode: 1 - Local development retries
  • โฑ๏ธ Timeouts: Optimized for game loading
    • defaultCommandTimeout: 6000ms
    • pageLoadTimeout: 15000ms

Test Optimization

  • ๐Ÿš€ Memory Management: experimentalMemoryManagement: true
  • ๐Ÿ’พ Test Retention: numTestsKeptInMemory: 5
  • ๐ŸŽฌ Video Optimization: videoUploadOnPasses: false (failures only)
  • ๐Ÿ“ธ Screenshots: Only on failure
  • ๐Ÿงน Cleanup: trashAssetsBeforeRuns: true

๐Ÿ” Security Testing Integration

Security Test Coverage

  • ๐Ÿ”’ XSS Prevention: Input validation tests
  • ๐Ÿ›ก๏ธ CSP Compliance: Content Security Policy validation
  • ๐Ÿ” Dependency Scanning: Automated via npm audit
  • ๐Ÿ“Š OWASP ZAP: Periodic security scans
  • ๐ŸŽฏ CodeQL: Security vulnerability scanning

Security Test Files:

  • ZAP scan: .github/workflows/zap-scan.yml
  • CodeQL: .github/workflows/codeql.yml
  • Scorecard: .github/workflows/scorecards.yml

๐Ÿ“– Test Execution Guide

Local Development

# Install Cypress
npm run cypress:install
npm run cypress:verify

# Run all E2E tests
npm run test:e2e

# Open Cypress Test Runner (interactive)
npx cypress open

# Run specific test file
npx cypress run --spec "cypress/e2e/game-journey.cy.ts"

# Run tests in specific browser
npx cypress run --browser firefox

CI Environment

# Tests run automatically on:
# - Push to main branch
# - Pull requests to main branch

# View test results:
# https://github.com/Hack23/blacktrigram/actions

Troubleshooting

Issue: Tests fail in headless mode

xvfb-run --auto-servernum --server-args="-screen 0 1280x720x24" npm run test:e2e

Issue: Korean fonts not rendering

sudo apt-get install fonts-noto-cjk fonts-noto-cjk-extra

๐ŸŽฏ Compliance Mapping

ISO 27001 Alignment

Control Requirement Implementation
A.8.9 Configuration Management Version-controlled test plans
A.14.2 Testing in Development Automated E2E testing
A.14.3 Test Data Protection Sanitized test data only

NIST CSF Alignment

Function Category Implementation
PR.IP-1 Baseline Configuration Cypress config management
PR.IP-2 SDLC Integration E2E tests in CI/CD
DE.CM-4 Malicious Code Detection Dependency scanning

CIS Controls Alignment

Control Description Implementation
2.1 Software Inventory Package.json tracking
16.9 Automated Testing E2E test automation
16.10 Test Environment Segregation Isolated test runs

๐Ÿ“š Additional Resources

Documentation

Related Plans

โšก Performance Optimization

Execution Time Improvement

The E2E test suite has been optimized to reduce execution time from ~20 minutes to the target of 10-12 minutes while maintaining full test coverage and reliability.

Key Optimizations:

  1. Configuration Tuning (2-3 min savings)

    • Video compression optimized for faster encoding
    • Memory management improved
    • Timeout settings reduced appropriately
  2. Code Optimization (3-4 min savings)

    • Fixed wait times replaced with assertions
    • Command delays reduced
    • Test transitions optimized
  3. Performance Monitoring (observability)

    • Per-test timing metrics
    • Slowest test identification
    • CI duration reporting

Test Suite Options:

  • Smoke Tests (npm run test:e2e:smoke): 5-6 minutes - Critical path validation
  • Full Suite (npm run test:e2e): 10-12 minutes - Comprehensive testing
  • Reliability Check (npm run test:reliability): <1 minute - Analyze flaky tests

For detailed optimization analysis and metrics, see E2E_OPTIMIZATION_RESULTS.md.


๐Ÿ›ก๏ธ Test Reliability & Stability (Nov 2025)

Test Isolation Infrastructure

Global Cleanup Hooks (cypress/support/e2e.ts):

beforeEach(() => {
  // Clear browser storage
  cy.clearLocalStorage();
  cy.clearCookies();
  
  // Reset viewport
  cy.viewport(1280, 720);
  
  // Clear game state
  // Start resource monitoring
  cy.startResourceMonitoring();
});

afterEach(() => {
  // Detect resource leaks
  cy.detectResourceLeaks();
  
  // Log test metrics
  // Force cleanup resources
});

Test Isolation Utilities (cypress/support/test-isolation.ts):

  • cy.isolateTest() - Reset to clean state before test
  • cy.cleanupTest() - Clean up after test completion
  • cy.captureState() - Capture current state for restoration
  • cy.restoreState() - Restore previously captured state
  • State management: localStorage, sessionStorage, game state, timers, event listeners

Resource Monitoring (cypress/support/resource-monitoring.ts):

  • cy.startResourceMonitoring() - Begin tracking resources
  • cy.detectResourceLeaks() - Check for leaks after test
  • cy.logResourceReport() - Log detailed resource usage
  • cy.forceResourceCleanup() - Force cleanup all resources
  • Tracks: audio elements, canvas elements, memory usage, event listeners

Flakiness Detection (CI)

Pull Request Testing (.github/workflows/test-and-report.yml):

  • Runs smoke tests 3 times on every PR
  • Detects inconsistent behavior (0 < failures < 3 = flaky)
  • Reports consistent failures (all 3 runs fail)
  • Uploads results as artifacts for analysis

Reliability Reporting (scripts/generate-reliability-report.cjs):

npm run test:reliability

Generates:

  • HTML report with metrics and flaky test details
  • Console summary with pass rates and failures
  • Identifies flaky tests with pass rate percentages
  • Lists consistently failing tests
  • Calculates average test duration

Configuration Changes

Cypress Retries (cypress.config.ts):

  • runMode: Reduced from 2 to 1 (better flaky detection)
  • openMode: Reduced from 1 to 0 (immediate feedback)
  • Encourages fixing tests rather than masking flakiness

Timeout Settings (optimized for reliability):

  • defaultCommandTimeout: 5000ms (fast failure detection)
  • pageLoadTimeout: 12000ms (sufficient for app load)
  • requestTimeout: 6000ms (reasonable for API calls)
  • responseTimeout: 6000ms (matches request timeout)

Best Practices for Test Authors

Using Test Isolation:

describe('My Feature Tests', () => {
  beforeEach(() => {
    cy.isolateTest(); // Optional: explicit isolation
    cy.visit('/');
  });
  
  afterEach(() => {
    cy.cleanupTest(); // Optional: explicit cleanup
  });
  
  it('should work reliably', () => {
    // Test implementation
    // Global hooks handle most cleanup automatically
  });
});

Resource Monitoring:

it('should not leak resources', () => {
  cy.startResourceMonitoring(); // Optional: already in global hook
  
  // Perform test actions
  cy.get('[data-testid="button"]').click();
  
  cy.detectResourceLeaks(); // Optional: already in global hook
  // Logs warnings if leaks detected
});

Checking for Flakiness Locally:

# Run tests multiple times
for i in {1..5}; do npm run test:e2e:smoke; done

# Generate reliability report
npm run test:reliability

Metrics & Achievements

Current Status (as of Nov 2025):

  • โœ… 0% Flaky Test Rate (target achieved)
  • โœ… 100% Pass Rate (10/10 smoke tests)
  • โœ… Resource Monitoring Active (detects 10-30MB memory growth)
  • โœ… Complete Test Isolation (state reset between tests)
  • โœ… Reduced Retries (1 or 0, down from 2)
  • โœ… CI Flakiness Detection (3x runs on PRs)

Before Improvements:

  • Flaky Test Rate: <1% (acceptable but not ideal)
  • Retries: 2 (could mask flaky tests)
  • Resource Leaks: Not monitored
  • Test Isolation: Partial

After Improvements:

  • Flaky Test Rate: 0% (zero tolerance)
  • Retries: 1 (runMode), 0 (openMode)
  • Resource Leaks: Monitored and logged
  • Test Isolation: Complete with global hooks

๐Ÿ“‹ Document Metadata:
โœ… Approved by: Development Team
๐Ÿ“ค Distribution: Development Team, QA Team, Security Team
๐Ÿ”„ Review Cycle: Monthly
โฐ Last Updated: 2025-11-17
๐Ÿ“ Version: 1.2.0 (Test Reliability Enhanced)
๐Ÿ‘ค Maintained by: Test Specialist Agent


๐Ÿฅ‹ ํ‘๊ด˜์˜ ํ’ˆ์งˆ์„ ์ง€ํ‚ค๋ผ - Protect the Quality of Black Trigram