Skip to content

Empty failure tags on iOS JUnit XML report #999

@msdeleyto

Description

@msdeleyto

Describe the bug
When running iOS tests we have detected failed tests with no content in the failure tag:

    ...
    <testcase name="testAccessibility" time="13.387" classname="TestTarget.Tests">
      <failure></failure>
    </testcase>
    ...

To Reproduce
Steps to reproduce the behaviour:

  1. Full configuration
# Check Marathon documentation in:
# https://docs.marathonlabs.io/intro/configure
# https://github.com/MarathonLabs
name: "E2E tests"
outputDir: "./reports"
screenRecordingPolicy: "ON_FAILURE"
deviceInitializationTimeoutMillis: 480000
debug: true
uncompletedTestRetryQuota: 0
batchingStrategy:
  type: "fixed-size"
  size: 10
shardingStrategy:
  type: "count"
  count: 1
executionStrategy:
  mode: ALL_SUCCESS
  fast: true
vendorConfiguration:
  type: "iOS"
  bundle:
    application: "./App.app"
    testApplication: "./Tests.xctest"
    testType: "xcuitest"
  xcresult:
    pull: true
    remoteClean: true
  xcodebuildTestArgs:
    "-test-timeouts-enabled": "YES"
    "-maximum-test-execution-time-allowance": "180"
  testParserConfiguration:
    type: "xctest"
  screenRecordConfiguration:
    preferableRecorderType: "screenshot"
  compactOutput: true
  permissions:
    grant:
      - all
  devices: "./Marathondevices"
outputConfiguration:
  maxPath: 1024
filteringConfiguration:
  allowlist:
    - type: "method"
      values:
        - "testAccessibility"
  1. Any additional information about the device provider (devices that were stuck, etc.)
    • The error is because the log from the xcrun does not indicate the class throwing / causing the test to fail, and it just prints <unknown>:0 instead of class/path/:line format.
    • It seems to happen consistently when an a11y test fails.

Expected behavior
The error message displayed on the logs and the xcresult instead of an empty tag.

Logs and reports

...
<DebugLogPrinter> <unknown>:0: error: -[TestTarget.Tests testAccessibility] : Element has no description
<DebugLogPrinter>     t =     4.79s Find the Any (Descendant with identity binding to AX element pid: 62723, elementOrHash.elementID: 105553170498336.190)
<DebugLogPrinter>     t =     4.80s Checking existence of `"empty state view" StaticText`
<DebugLogPrinter>     t =     4.81s Find the "empty state view" StaticText
...

Devices (please complete the following information):

  • Device: iPhone 16
  • OS: iOS 18

Additional context
PR with an initial proposal to fix the issue: #1000

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions