TASK 38260338: Updated the pipelines to produce Linux builds capable of complying with the FIPS 140-3 standard.

[al-msft]

Microsoft's version of Go provides the FIPS capability. The FIPS-capable builds are produced alongside the existing builds, which don't use Microsoft's version of Go.

Description

The CSI driver image is built by another team at Microsoft. That image needs to comply with the FIPS 140-3 standard by June 30th. A FIPS-compliance checker scans all binaries in the image; AzCopy is one of those binaries. To pass the FIPS-compliance check, each binary must be built with the Microsoft build of Go in the particular way the check expects and pass a run-time check that considers the host OS. The CSI driver image downloads azcopy as an RPM package on Azure Linux 3.0. To reduce the risk of migrating AzCopy from the official build of Go to the Microsoft build of Go, it was decided to release FIPS-capable builds of AzCopy - only for Linux - alongside the existing builds, and not to release Docker container images containing FIPS-capable builds. All existing Linux builds will now have a FIPS-capable analog, other than the Mariner 2 builds, because Mariner 2 has reached end of life.

These changes are intended to match the approach that was taken in previous pipeline updates, especially regarding the assurances and costs of testing. The changes are expected to increase the run-time of the affected pipelines; to mitigate this, parallelism was used (via the "matrix" strategy) where it seemed unlikely to lead to conflicts in the parallel operations, the tests in run_scenarios.yml were not updated to test the FIPS-capable builds (there seemed to be little value in doing so), and only the FIPS-capable build for the AMD64 architecture was tested (following the approach that was taken for the non-FIPS-capable builds). The "unit" tests, end-to-end tests, and Python tests were updated to test the FIPS-capable build, because each of those types of tests includes tests at the integration level or higher. If we later update the set of "unit" tests to ensure they are all true unit tests, we should update the pipeline to stop running those tests on the FIPS-capable build.

Cut for Time/Risk:

• FIPS-capable builds for any OS other than Linux
• FIPS-capable builds in Docker container images
• Updating GitHub Actions
• Logging a message when AzCopy is invoked that states whether FIPS is enabled
• Updating the performance-test pipeline to test the FIPS-capable builds
• Addressing technical debt in the pipelines
• Updating the pool agent that runs Azure Linux 3.0 to use the same minor version of Python 3 as the other pool agents to run the Python-based AzCopy tests
• Optimizing the run-time, parallelism, amount of duplication, and the use of parameters and variables in the pipelines
• Scanning the FIPS-capable binaries for FIPS compliance in the pipeline(s)
• Validating the details of the DEB and RPM packages in the release pipeline (probably in verify_linux.xml) - especially the descriptions and conflicts this PR changed, but also the pre-existing details, like version, maintainer, and URL

Items to Discuss with the Product Manager before This PR Is Marked Ready for Review:

• The package name and description of the FIPS-capable AzCopy packages (.deb and .rpm files)
• The names of the FIPS-capable artifacts, including the package files and archives
• Whether to publish the FIPS-capable artifacts to GitHub
• The potential need to update documentation

Related Links:

• TASK 38260338
• BlobFuse FIPS Teams [chat](https://teams.microsoft.com/l/chat/19:4d86c8df4c5d4e2fa324a5064f0852f0@thread.v2/conversations?context=%7B%22contextType%22%3A%22chat%22%7D
• AZCopy FIPS Compliance Teams [chat](https://teams.microsoft.com/l/chat/19:96584494811d4c0dbeb7f4eb07bc789a@thread.v2/conversations?context=%7B%22contextType%22%3A%22chat%22%7D
• FIPS Support in the Microsoft build of Go (Warning: This is misleading in some places)

Type of Change

• Bug fix
• New feature
• Documentation update required
• Code quality improvement
• Other (describe):

How Has This Been Tested?

• See the description for general information.
• The test pipeline (which runs on PRs, including this one) has been updated to additionally test the FIPS-capable AMD64 Linux build in two distinct scenarios: with FIPS enabled and with FIPS disabled. That pipeline has been manually run on this branch several times, and although there are still issues specific to the agent being used to test with FIPS enabled, some tests pass, and the same tests ultimately fail each time. For example, see the test results for https://dev.azure.com/azstorage/AzCopy-NextGen/_build/results?buildId=30978&view=results.
• The release pipeline has been run on this branch without actually publishing the artifacts to packages.microsoft.com or GitHub.
• The artifacts the updated release pipeline creates have been scanned by manually running a FIPS-compliance checker we were instructed to use and that is part of the same tool suite as the FIPS-compliance checker used to scan the CSI driver image and the binaries within it.
• The package info for the FIPS and non-FIPS .deb and .rpm packages the updated release pipeline creates have been manually reviewed. The conflict indicators in the AMD64 FIPS and non-FIPS .deb and .rpm packages (other than the SE Linux packages and the untouched .rpm packages for Mariner 2) have been tested on FIPS-disabled Ubuntu (with the .deb packages) and FIPS-enabled Azure Linux 3.0 (with the .rpm packages) in various combinations of the FIPS and non-FIPS packages the updated release pipeline creates and previous azcopy packages that lack a conflict indicator.
• The AMD64 FIPS and non-FIPS .deb and .rpm packages the updated release pipeline creates (other than the SE Linux packages and the untouched .rpm packages for Mariner 2) have been installed on FIPS-disabled Ubuntu (with the .deb packages) and FIPS-enabled Azure Linux 3.0 (with the .rpm packages). After installing each package, azcopy was invoked with and without GOFIPS-1, and (as intended) it only panicked when the FIPS .deb package was installed on Ubuntu and azcopy was invoked with GOFIPS-1.

[al-msft]

@al-msft please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree company="Microsoft"