FI-3412 | Map client bulk data requirements to existing tests

[carrils]

Summary

This adds references to the requirements fulfilled by the tests in bulk data v2.
Where each requirement is tested, a reference to the requirement_ID has been made.
The requirements_coverage csv file has also been updated with requirements:generate_coverage.

Testing Guidance

The test list map (excel workbook on sharepoint for bulk data v2) is a spreadsheet that lists the requirement, the location of where it is tested, and whether it has been added to the codebase with the verifies_requirement inferno DSL.

The bulk-data-test-kit_requirements_coverage.csv file has short ID's and Full ID's.

Can pull down branch and try running the rake requirements:generate_coverage command to generate the csv with short and full ID's.

[edeyoung]

not quite finished, but some actionable things for you to take a look at

[edeyoung]

thanks for your updates! similarly wasn't able t finish everything but have some actionable items for you.

[edeyoung]

Please see the latest comments.

Thoughts: in the excel doc, filter for the conformance verb SHALL and the Actor Server (or Server/Client). That will reduce your requirements pool considerably. Do you have an idea what requirements coverage looks like at this point?

Thank you!

[edeyoung]

it looks like the 230+ requirements are specific to output types: group / patient / system. Is there a place where these can be moved to?

[carrils]

So 230 specifically is _outputFormat: The server SHALL support Newline Delimited JSON. I left it here in v2.0.0/bulk_data_outputFormat_param_test.rb because of its location in the IG.
I have reviewed the operation-specific requirements and have moved these:
223, 224, 225, 229, 249, and 272.

[elsaperelli]

I believe this is correct but leaving it open for confirmation.

[edeyoung]

230 should be deprecated in favor of 49, which is how the rest of the output type specific requirements are.
230 should be removed from here.

Deprecating in the requirements sheet now.

[edeyoung]

Please

• regenerate the entire requirements file
• regenerate the coverage file

then

• check if your coverage file is showing 47, 49, 51, 52 as uncovered
• if so, trouble shoot why the coverage file is showing 47,49,51,52 (and maybe others?) as uncovered when they are referenced in lib/bulk_data_test_kit/v2.0.0/bulk_data_outputFormat_param_test.rb

Please also confirm that you reviewed the uncovered requirements and double checked that they couldn't be covered in the existing code.

[edeyoung]

230 should be deprecated in favor of 49, which is how the rest of the output type specific requirements are.
230 should be removed from here.

Deprecating in the requirements sheet now.

[elsaperelli]

Please

• regenerate the entire requirements file
• regenerate the coverage file

then

• check if your coverage file is showing 47, 49, 51, 52 as uncovered
• if so, trouble shoot why the coverage file is showing 47,49,51,52 (and maybe others?) as uncovered when they are referenced in lib/bulk_data_test_kit/v2.0.0/bulk_data_outputFormat_param_test.rb

Please also confirm that you reviewed the uncovered requirements and double checked that they couldn't be covered in the existing code.

• regenerated entire requirements file
• regenerated the coverage file
• removed req 230
• figured out why covered reqs were hidden- moved annotations 47, 49, 51, 52, 53
• Reviewed uncovered requirements- have some notes on ones that may be covered below

Remaining Questions:

• I think that the _since parameter for Patient-level export should actually be a SHALL requirement instead of a MAY
• The following are requirements that may be tested but I am not totally sure:
  • 15, 16, 18 - covered by bulk_data_client_wait_test.rb?
  • 32 - perform_export_kickoff_request ?
  • 148 - bulk_status_check_test.rb?
  • 201 - is this test correctly placed?

[edeyoung]

• I think that the _since parameter for Patient-level export should actually be a SHALL requirement instead of a MAY

Is this @53 "_since: Optionality for Server: required"?

• The following are requirements that may be tested but I am not totally sure:

  • 15, 16, 18 - covered by bulk_data_client_wait_test.rb?

bulk_data_client_wait_test.rb doesn't look like it actually tests anything - it just waits for the request to then send the data to other tests.
Maybe bulk_data_client_token_verification_test.rb for 15/16 (need to check the innards of the smart test)
18 would be an attestation - not erroring out when sent bulk data tests.

• 32 - perform_export_kickoff_request ?

I don't think so - i don't see anything checking comma delimited stuff here

• 148 - bulk_status_check_test.rb?

this is an optional test, so not here.
i think this needs to be an attestation. the requirement is whether or not the files are accessible to the client, which I don't think inferno does (or can) test.

• 201 - is this test correctly placed?

I think it's in the right place, but i'm confused about how it's tested when the expected response in ndjson_download_requiresAccessToken_check is assert_response_status([400, 401])