Suggestion: Multi-dimensional scenario outlines

[willclarktech]

I have a really long, multi-step form, which gets validated at various points along the way, and certain answers can block submission. I'd like to end-to-end test that the various permissible routes through the form all allow the form to be submitted at the end. (In almost all cases the validity of a field's value is independent of the values of other fields.)

I believe my current options are:

1. Just write separate scenarios for each possible way of going through the form.
2. Write one big scenario outline using a gigantic table of examples with one row for every possible way of going through the form.
3. Several scenario outlines each with a large table of examples with one row for every possible way of going through the form given the scope of the individual scenario outline.
4. Maybe something to do with inline tables? Although as I understand it, that would mean just one scenario with each step passed a selection of values, which sounds really difficult to turn into isolated browser-based tests.
5. Accept that comprehensive scenario coverage isn't going to happen and only write tests for a subset of the possible routes through the form.

Ideally I'd be able to define a set of example values for each of several fields, and a separate scenario would run for each combination.

For example with something like this in a feature file (I have no opinions about syntax btw):

Scenario Outline Multidimensional:
    Given an [animal]
    When I [action] the animal
    Then [result] happens
    And I feel [feeling]

    Examples: Animals
        | animal |
        | zebra  |
        | fox    |
        | eagle  |

    Examples: Actions
        | action   |
        | talk to  |
        | befriend |

    Examples: Consequences
        | result            | feeling       |
        | nothing much      | nonplussed    |
        | something amazing | over the moon |

yadda would run 3 x 2 x 2 = 12 scenarios.

[cressie176]

That's an interesting idea I'd never thought of. I think it would be relatively easy to change the feature parser to support multiple example tables then generate the scenarios accordingly. The feature parser is pluggable so you could give it a try if you felt inclined.

I'm not sure I'd go down this route though as I don't think the specification would really add any clarity / value. Instead what about generating the test in code? After the spec is parsed it's just an array of strings, which would be far easier to generate with a set of nested for loops and a template.

var scenarios = []
['zebra', 'fox', 'eagle'].forEach((animal) => {
  ['talk to', 'befriend'].forEach((action) => {
    [{ result: 'nothing much', feeling: 'nonplussed' }, { result: 'something amazing',  feeling: 'over the moon' }].forEach((consequence) => {
      var title = `${animal} ${action} ${consequence.result} ${consequence.feeling}`
      var steps = render(template, { animal: animal, action: action, consequence: consequence }).split('\n')
      scenarios.push({ title: title, steps: steps })
    })
  })
})

template

Given an {{animal}}
When I {{action}} the animal
Then {{consequence.result}} happens
And I feel {{consequence.feeling}}

[willclarktech]

This is super helpful, thank you! I'm going to start out with the code-generated tests, and maybe also generate a feature file for non-developers to read (but not edit obviously). If I end up looking at the parser I'll let you know!

[cressie176]

OK to close?

[willclarktech]

Yep, thanks again.

[lolmaus]

@willclarktech Any luck implementing this?

I believe this should be available out of the box. Creating a single Examples table with all possible combinations is tedious, bulky, hard to read and prone to errors.

[lolmaus]

@cressie176 Please reopen. This should at least be available as one of examples/ (pun not intended).

[lolmaus]

In support of @willclarktech's feature requst, please compare these two sets of examples.

Proposed:

Examples: Animals
    | animal |
    | zebra  |
    | fox    |
    | eagle  |

Examples: Actions
    | action   |
    | talk to  |
    | befriend |

Examples: Consequences
    | result            | feeling       |
    | nothing much      | nonplussed    |
    | something amazing | over the moon |

Currently available equivalent:

Examples:
    | Animal | Action   | Result            | Feeling       |
    | zebra  | talk to  | nothing much      | nonplussed    |
    | zebra  | talk to  | something amazing | over the moon |
    | zebra  | befriend | nothing much      | nonplussed    |
    | zebra  | befriend | something amazing | over the moon |
    | fox    | talk to  | nothing much      | nonplussed    |
    | fox    | talk to  | something amazing | over the moon |
    | fox    | befriend | nothing much      | nonplussed    |
    | fox    | befriend | something amazing | over the moon |
    | eagle  | talk to  | nothing much      | nonplussed    |
    | eagle  | talk to  | something amazing | over the moon |
    | zebra  | befriend | nothing much      | nonplussed    |
    | eagle  | befriend | something amazing | over the moon |

[willclarktech]

@lolmaus I didn't end up pursuing this as a colleague convinced me that end-to-end tests should focus on scenarios which efficiently provide confidence in your application, rather than aiming to be exhaustive.

I did end up writing this library to help with writing exhaustive (e.g. unit) tests with Yadda/Gherkin-like steps in Mocha: https://github.com/LiskHQ/mocha-bdd. It supports nesting rather than multi-dimensional templates per se but that's more easily generalisable. It's obviously not as human-readable as Yadda feature files, but I'm now doubtful whether Yadda-style feature files should even cater for anything approaching exhaustive test suites.

[cressie176]

Re-opening for the purpose of discussion. On review I'm not keen to implement multiple example tables as described above - as mentioned above I think they fail from a readability perspective. What might be worth considering is extending the example syntax to support a named data feed, e.g.

Examples: "animals"

The feed could be an array of arrays or function that yields an array of arrays, that Yadda would use for examples

{ 
  animals: [
    ['Animal', 'Action', 'Result', 'Feeling'],
    ['zebra', 'talk to', 'nothing much', 'nonplussed'],
    ['zebra', 'talk to', 'something amazing', 'over the moon'],
    ['zebra', 'befriend', 'nothing much', 'nonplussed'],
    ['zebra', 'befriend', 'something amazing', 'over the moon'],
    ['fox', 'talk to', 'nothing much', 'nonplussed'],
    ['fox', 'talk to', 'something amazing', 'over the moon'],
    ['fox', 'befriend', 'nothing much', 'nonplussed'],
    ['fox', 'befriend', 'something amazing', 'over the moon'],
    ['eagle', 'talk to', 'nothing much', 'nonplussed'],
    ['eagle', 'talk to', 'something amazing', 'over the moon'],
    ['zebra', 'befriend', 'nothing much', 'nonplussed'],
    ['eagle', 'befriend', 'something amazing', 'over the moon']
  ]
}

Creating the array / implementing the function would be the responsibility of the program using Yadda, rather than Yadda itself. Then you could generate data in any which way you wanted.