Skip to content

testing.md

Latest commit

 

History

History
256 lines (166 loc) · 14.9 KB

testing.md

File metadata and controls

256 lines (166 loc) · 14.9 KB
Raw

Testing

This document provides an overview of the testing strategies employed in the Core Service. Our comprehensive testing approach ensures high-quality software delivery, encompassing unit tests, integration tests, end-to-end (E2E) tests, mutation tests, load tests, and more.

Table of Contents

Prerequisites

Before executing tests, run make setup-test-db. This command will create a separate database for testing purposes.

Unit Testing

Unit tests are the foundation of our testing strategy, focusing on small, isolated parts of the application to ensure they behave as expected. We write unit tests for individual classes and methods, mocking dependencies to isolate the unit of work.

Our Unit and Integration test coverage is 100%. You can check this GitHub CI workflow to see the results of the latest Unit and Integration test execution.

Location:

Tests are organized under the /tests/Unit directory, with further categorization, similar to folders in src.

Tools:

We primarily use PHPUnit for unit testing, along with mock objects for dependencies.

Best Practices:

  • Test One Thing at a Time: Each test should focus on a single behavior or aspect of the component under the test. This approach makes it easier to identify what is broken when a test fails and ensures that each test is only for one purpose.
  • Use Descriptive Test Names: Test method names should clearly describe what they are testing. A well-named test can serve as documentation for what a piece of code is supposed to do. Descriptive names make it easier to understand the purpose of the test without having to dive into the implementation details.
  • Keep Tests Independent: Tests should not rely on the state produced by other tests. Each test should set up its data and not depend on the order of test execution. This practice ensures that tests can be run in any order and that the outcome of a test is not affected by the preceding tests.
  • Mock External Dependencies: Use mocks for any external service or database interaction to isolate the component under test. Mocking external dependencies allows us to test the internal logic of a component without worrying about the setup and behavior of external systems.

Execution:

Run make unit-tests to execute unit tests.

Integration Testing

Our Unit and Integration test coverage is 100%. You can check this GitHub CI workflow to see the results of the latest Unit and Integration test execution.

Integration tests assess the interaction between different parts of the application, such as database access and external services, to ensure they work together as expected. These tests are crucial for identifying issues that may not be visible through unit testing alone.

Location:

Integration tests are located in the /tests/Integration directory. This organization mirrors the structure of the src directory to make it easier to find the tests relevant to specific components or functionalities.

Tools:

For integration testing, we use PHPUnit in conjunction with real database connections (MongoDB) and external services. This approach allows us to test the application in an environment that closely resembles production.

Best Practices:

  • Test Real Interactions: Unlike unit tests, integration tests should use real instances of classes and services to ensure that their interactions are tested accurately.
  • Use Database Cleanup: To maintain a consistent state and avoid polluting the database, clean up data after each test. This ensures that each test starts in a clean state.
  • Focus on Critical Paths: Given the potentially large scope of integration testing, focus on critical paths through the application. This includes customer creation, update flows, and data processing tasks.
  • Monitor Performance: Integration tests can be slower than unit tests due to their reliance on external services and databases. Monitor performance to ensure that the test suite remains efficient and manageable.

Execution:

Run make integration-tests to execute the integration tests. This command ensures that all dependencies are correctly set up and that the tests are run against the configured test database and external services.

Memory-Safety Testing

FrankenPHP worker mode keeps the Symfony kernel and container alive across requests, so the repository now includes a dedicated same-kernel memory-safety layer under tests/Memory.

What This Covers

  • Repeated same-kernel REST requests for:
    • /api/health
    • /api/customers
    • /api/customer_statuses
    • /api/customer_types
  • Repeated same-kernel GraphQL requests for the committed /api/graphql operations for customers, customer statuses, and customer types.
  • A reset-aware observability check that proves test-only metric objects do not accumulate across repeated requests.

How It Works

  • The suite uses shipmonk/memory-scanner as the primary retained-object detector for the FrankenPHP migration track.
  • Tests use BrowserKit clients with disableReboot() so multiple requests reuse the same kernel instead of rebuilding the full container on every request.
  • A test-only request subscriber records the Symfony main Request object for each handled request. The suite verifies that the previous request object is deallocated before or during the next same-kernel request cycle.
  • BusinessMetricsEmitterSpy now implements ResetInterface, so Symfony resets it through kernel.reset when same-kernel tests call disableReboot().

Current Limitations

  • The application now runs through FrankenPHP worker mode in the default Docker stack, but the PHPUnit suite itself remains the primary retained-object detector because it can pinpoint specific leaked objects and reset failures.
  • The current PHPUnit stack is 10.5, while the upstream ObjectDeallocationCheckerKernelTestCaseTrait from shipmonk/memory-scanner targets PHPUnit 11+. The repository therefore uses a local manual bridge around ObjectDeallocationChecker until PHPUnit is upgraded.
  • The committed repository still lacks a real authenticated route and firewall configuration, so authenticated same-kernel memory coverage remains blocked on a follow-up implementation.

Execution

Run the dedicated memory suite locally or through the standalone GitHub Actions workflow:

make memory-tests
make ci

GitHub Actions runs this suite through .github/workflows/memory-tests.yml as separate Memory leak tests (dev env), Memory leak tests (test env), and Memory leak tests (prod env) checks. The workflow boots the FrankenPHP worker stack for each environment using only make entrypoints, executes the object-level PHPUnit memory suite in the test-environment job, reruns the K6 smoke suite against the live worker in every environment, and fails if the memory-support helper coverage drops below 100% or if the worker memory guardrail detects sustained growth across the soak loop. The guardrail smooths each checkpoint with multiple docker stats reads so borderline failures are driven by real sustained growth instead of a single noisy sample, and it allows the workflow to require extra warmup passes before the measured baseline in environments that still finish lazy dev-only initialization after the first endpoint sweep.

The development-environment row intentionally uses APP_ENV=dev with APP_DEBUG=0 and disables the hot_reload and watch helper snippets during the leak gate. Symfony allows APP_ENV and APP_DEBUG to vary independently, and this keeps the gate focused on the long-running worker contract instead of profiler/debug collectors or live-reload watchers. The workflow still covers the dev kernel and full endpoint inventory, but it does so without the development-only diagnostics that intentionally retain extra request metadata.

API Contract Validation

We validate the generated OpenAPI specification against the live application with Schemathesis. This gives us a contract-level regression check that runs real HTTP traffic against the Dockerized service instead of validating examples in isolation.

Workflow

Run make start first so the application and its supporting services boot through Docker, then execute make schemathesis-validate.

The Schemathesis target:

  • seeds deterministic reference data before the run,
  • exercises the examples, coverage, and fuzzing phases by default,
  • auto-enables the stateful phase only when the generated OpenAPI specification contains links,
  • keeps the conformance and server-error checks enabled while skipping the two acceptance heuristics that currently over-report on undocumented validation constraints and ignored extra query parameters,
  • writes JUnit, HAR, NDJSON, and coverage reports under /tmp/core-service-schemathesis-report/.

Execution

Use the following commands for local validation:

make start
make schemathesis-validate

In GitHub Actions, the dedicated Schemathesis workflow follows the same Docker-first path by starting the application with make start before invoking the validation target.

Mutation Testing

Mutation testing is a rigorous approach to testing that involves making small, deliberate modifications to our code (mutants) to verify that our tests can detect these changes. This method helps in evaluating the quality and effectiveness of our test suites.

We have 0 escaped and uncovered mutants in Core Service. You can check this GitHub CI workflow to see the results of the latest Mutation test execution.

Tools

We utilize Infection, a powerful PHP mutation testing framework. Infection automatically generates mutants by altering our codebase in small ways, then runs our test suite against each mutant. The goal is to have our tests fail for each mutant, indicating that our tests are effectively covering our code.

Execution

Run make infection, to run mutation testing and see a comprehensive report about the quality of our test.

Load Testing

Load testing is a critical aspect of our application's development lifecycle, designed to ensure that our application can handle expected traffic volumes and maintain performance under stress. This process helps us identify bottlenecks and optimize the application's scalability.

We test each available endpoint of our service with multiple loads. For each endpoint, we have Smoke, Average, Stress, and Spike load tests. You can check this link for more information about them.

Also, you can check this GitHub CI workflow to see the results of the latest Load test execution.

Location

Our load testing scripts are organized within the /tests/Load directory. These scripts are crafted to simulate various realistic usage scenarios that our application might face in production. By doing so, we can accurately assess how our system behaves under different levels of demand.

Tools

For scripting and executing our load tests, we rely on k6, a powerful and modern load-testing tool. k6 allows us to script complex user behavior and supports running these tests at scale. The scripts cover a wide range of API endpoints and user actions, ensuring comprehensive coverage of our application's functionality.

Configuration

The tests/Load/config.json.dist file serves as an example of load tests configuration. You can copy tests/Load/config.json.dist to tests/Load/config.json and customize variables for local development.

There is a wide range of customizable options, starting with a global setting for all load test scripts. Here is an example of them:

{
  "apiHost": "localhost",
  "apiPort": "443",
  "apiScheme": "https",
  "delayBetweenScenarios": 60,
  "batchSize": 50,
  "resultsDirectory": "results",
  "customersFileName": "customers.json",
  "customersFileLocation": "/loadTests/"
}

Note: Update apiHost with your actual domain when running load tests against production or staging environments. Local FrankenPHP runs use HTTPS with the self-signed development certificate, so the bundled K6 scenarios keep TLS verification disabled for that local-only path.

Also, you can customize plenty of options for each separate script, and even for each load type. Here is an example:

{
  "endpoints": {
    "health": {
      "setupTimeoutInMinutes": 30,
      "teardownTimeoutInMinutes": 30,
      "smoke": {
        "threshold": 9000,
        "rps": 10,
        "vus": 5,
        "duration": 10
      },
      "average": {
        "threshold": 200,
        "rps": 25,
        "vus": 25,
        "duration": {
          "rise": 3,
          "plateau": 10,
          "fall": 3
        }
      }
    }
  }
}

Execution

Run make load-tests to execute load tests for each endpoint with all load scenarios. Also, you can test only specific load types by using one of the following commands:

    make smoke-load-tests
    make average-load-tests
    make stress-load-tests
    make spike-load-tests

After the load test execution, you'll find results under tests/Load/results/.

End-to-End (E2E) Testing

End-to-end (E2E) testing is a critical phase in the application development lifecycle, aimed at simulating real user scenarios to ensure the system meets external requirements and behaves as expected across the entire application. This type of testing validates the integrated components of the application in a production-like environment, from the user interface down to the database operations and network communications.

We cover each possible response with a separate test. You can check this GitHub CI workflow to see the results of the latest E2E test execution.

Location:

Testing scenarios, written in BDD style, are located in the /features folder.

Code, responsible for processing the scenarios, is located in the /tests/Behat folder, with further categorization for each scenario.

Tools:

For E2E testing, our project utilizes Behat, a PHP framework for auto-testing your business expectations. Behat allows us to write tests in a human-readable format, using the Gherkin language, which makes it accessible not only to developers but also to non-technical stakeholders.

Best Practices for writing scenarios:

Check this link to learn how to write features for Behat.

Execution:

Run make behat to execute end-to-end tests.

Learn more about Advanced Configuration Guide.