steps/RustSetupSteps.yml: Use step template to download artifacts (#233)

Downloads the Cargo tools from an Azure pipeline using a YAML step
template.

This resolves a problem in commit 69f6e96 that was not found until
after check-in due to the nature of the issue. It is summarized
below.

In the original check-in (69f6e96), the following succeeds and fails:

- Downloading and Caching as an Artifact:
  - ✓ Use GitHub REST API to download release and extract the binaries
  - ✓ Publish the binaries as a pipeline artifact in the projectmu org
- Retrieving the Artifact
  - Note: In all cases, try the `DownloadPipelineArtifact@2` and
    `DownloadBuildArtifacts@1` tasks
  - ✓ Access in a manually triggered pipeline
  - ✓ Access in a PR triggered pipeline from a branch in the microsoft
    org repo (i.e. not a fork)
  - ✗ Access in a PR triggered pipeline from a branch outside the
    microsoft org repo (i.e. a fork)

To allow testing using pre-existing PR check pipelines (which have
access to branches on the microsoft org, not forks), the last case
was unexpected and not encountered until the PRs completed.

Since the information is readily available using the Azure Pipelines
REST API without authentication, this change replaces the built-in
tasks with calls to download the binaries from the REST API.

There are a few quirks with pipelines that are accounted for:

1. The Python code is inline so it can directly be used as a
   template in other YAML files that are used as a repository resource.
   If in a separate Python file, the mu_devops repo would need to be
   checked out to use access the file. Checking out the repo would
   increase build times and complicate pre-existing logic.
2. The Azure Pipeline `InvokeRESTAPI@1` task is not used as it is
   limited to agentless jobs.
3. Conditions are not allowed on templates. Therefore the OS condition
   is moved to a string parameter that is compared in the `condition`
   on the task in the template. By default, the task will run on all
   operating systems if not specified.

As a follow up, the `Cache@2` task will be explored which might simplify
some logic but require changes elsewhere as an alternative to cache
the binaries in the pipelines.

Signed-off-by: Michael Kubacki <michael.kubacki@microsoft.com>

Author: makubacki

Steps/DownloadAzurePipelineArtifact.yml

@@ -0,0 +1,119 @@
+## @file
+# Azure Pipelines step template to download Azure Pipeline artifacts.
+#
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+
+parameters:
+- name: artifact_name
+  displayName: Artifact Name
+  type: string
+  default: 'Binaries'
+- name: azure_org_name
+  displayName: Azure Org Name
+  type: string
+  default: 'projectmu'
+- name: azure_proj_name
+  displayName: Azure Project Name
+  type: string
+  default: 'mu'
+- name: azure_pipeline_def_id
+  displayName: Azure Pipeline Definition ID
+  type: string
+  default: '0'
+- name: file_pattern
+  displayName: File Pattern
+  type: string
+  default: '*'
+- name: target_dir
+  displayName: Target Directory
+  type: string
+  default: ''
+- name: target_os
+  displayName: Target OS For Task to Run
+  type: string
+  default: 'Windows_NT,Darwin,Linux'
+- name: task_display_name
+  displayName: Task Display Name
+  type: string
+  default: 'Download Pipeline Artifact'
+- name: work_dir
+  displayName: Work Directory
+  type: string
+  default: ''
+
+steps:
+
+- task: PythonScript@0
+  displayName: ${{ parameters.task_display_name }}
+  env:
+    ARTIFACT_NAME: ${{ parameters.artifact_name }}
+    AZURE_ORG_NAME: ${{ parameters.azure_org_name }}
+    AZURE_PROJ_NAME: ${{ parameters.azure_proj_name }}
+    AZURE_PIPELINE_DEF_ID: ${{ parameters.azure_pipeline_def_id }}
+    FILE_PATTERN: ${{ parameters.file_pattern }}
+    TARGET_DIR: ${{ parameters.target_dir }}
+    WORK_DIR: ${{ parameters.work_dir }}
+  inputs:
+    scriptSource: inline
+    workingDirectory: $(Agent.BuildDirectory)
+    script: |
+        import os
+        import requests
+        import shutil
+        import zipfile
+        from pathlib import Path
+
+        ARTIFACT_NAME = os.environ["ARTIFACT_NAME"]
+        AZURE_ORG_NAME = os.environ["AZURE_ORG_NAME"]
+        AZURE_PROJ_NAME = os.environ["AZURE_PROJ_NAME"]
+        AZURE_PIPELINE_DEF_ID = os.environ["AZURE_PIPELINE_DEF_ID"]
+        FILE_PATTERN = os.environ["FILE_PATTERN"]
+        TARGET_DIR = Path(os.environ["TARGET_DIR"])
+        WORK_DIR = os.environ["WORK_DIR"]
+
+        build_id_url = f"https://dev.azure.com/{AZURE_ORG_NAME}/{AZURE_PROJ_NAME}/_apis/build/builds?definitions={AZURE_PIPELINE_DEF_ID}&$top=1&api-version=6.0"
+
+        # Fetch the list of assets from the GitHub releases
+        response = requests.get(build_id_url)
+        response.raise_for_status()
+        latest_build_id = response.json()["value"][0]["id"]
+
+        artifact_url = f"https://dev.azure.com/{AZURE_ORG_NAME}/{AZURE_PROJ_NAME}/_apis/build/builds/{latest_build_id}/artifacts?artifactName={ARTIFACT_NAME}&api-version=6.0"
+        response = requests.get(artifact_url)
+        response.raise_for_status()
+        download_url = response.json()["resource"]["downloadUrl"]
+
+        print(f"Latest Build ID: {latest_build_id}")
+        print(f"Artifact Download URL: {download_url}")
+
+        download_path = Path(WORK_DIR, "artifact_download", ARTIFACT_NAME).with_suffix(".zip")
+        download_path.parent.mkdir(parents=True)
+        with requests.get(download_url, stream=True) as r:
+            with download_path.open('wb') as f:
+                for chunk in r.iter_content(chunk_size=8192):
+                    f.write(chunk)
+
+        with zipfile.ZipFile(download_path, 'r') as zip_ref:
+            zip_ref.extractall(download_path.parent)
+
+        unzip_path = download_path.parent / ARTIFACT_NAME
+
+
+        def flatten_copy(src: Path, dst: Path, pattern: str):
+            if not dst.exists():
+                dst.mkdir(parents=True)
+
+            for item in src.rglob(pattern):
+                print(f"Current item is {item}")
+                if item.is_dir():
+                    flatten_copy(item, dst, pattern)
+                else:
+                    shutil.copy2(item, dst)
+
+
+        TARGET_DIR.mkdir(parents=True, exist_ok=True)
+        flatten_copy(unzip_path, TARGET_DIR, FILE_PATTERN)
+        shutil.rmtree(download_path.parent)
+  condition: and(succeeded(), contains('${{ parameters.target_os}}', variables['Agent.OS']))

Steps/RustSetupSteps.yml

@@ -48,53 +48,53 @@ steps:
   displayName: Install Rust 1.71.1 (Windows)
   condition: eq(variables['Agent.OS'], 'Windows_NT')
 
-- task: DownloadPipelineArtifact@2
-  displayName: Download Cargo Make (Windows)
-  inputs:
-    buildType: specific
-    project: mu
-    definition: 166
-    targetPath: '$(cargoBinPath)\'
-    itemPattern: '**/cargo-make.exe'
-    artifactName: Binaries
-  condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'))
+- script: pip install requests --upgrade
+  displayName: Install and Upgrade requests PIP Module
+  condition: succeeded()
 
-- task: DownloadPipelineArtifact@2
-  displayName: Download Cargo Make (Linux)
-  inputs:
-    buildType: specific
-    project: mu
-    definition: 166
-    targetPath: '$(Agent.TempDirectory)'
-    itemPattern: '**/cargo-make'
-    artifactName: Binaries
-  condition: eq(variables['Agent.OS'], 'Linux')
+- template: DownloadAzurePipelineArtifact.yml
+  parameters:
+    task_display_name: Download Cargo Make (Windows)
+    artifact_name: Binaries
+    azure_pipeline_def_id: 166
+    file_pattern: "**/cargo-make.exe"
+    target_dir: "$(cargoBinPath)"
+    target_os: "Windows_NT"
+    work_dir: "$(Agent.TempDirectory)"
+
+- template: DownloadAzurePipelineArtifact.yml
+  parameters:
+    task_display_name: Download Cargo Make (Linux)
+    artifact_name: Binaries
+    azure_pipeline_def_id: 166
+    file_pattern: "**/cargo-make"
+    target_dir: "$(Agent.TempDirectory)"
+    target_os: "Linux"
+    work_dir: "$(Agent.TempDirectory)"
 - script: |
     cp $AGENT_TEMPDIRECTORY/cargo-make /.cargo/bin
   displayName: Copy cargo-make
   condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))
 
-- task: DownloadPipelineArtifact@2
-  displayName: Download Cargo Tarpaulin (Windows)
-  inputs:
-    buildType: specific
-    project: mu
-    definition: 167
-    targetPath: '$(cargoBinPath)\'
-    itemPattern: '**/cargo-tarpaulin.exe'
-    artifactName: Binaries
-  condition: and(succeeded(), eq(variables['Agent.OS'], 'Windows_NT'))
+- template: DownloadAzurePipelineArtifact.yml
+  parameters:
+    task_display_name: Download Cargo Tarpaulin (Windows)
+    artifact_name: Binaries
+    azure_pipeline_def_id: 167
+    file_pattern: "**/cargo-tarpaulin.exe"
+    target_dir: "$(cargoBinPath)"
+    target_os: "Windows_NT"
+    work_dir: "$(Agent.TempDirectory)"
 
-- task: DownloadPipelineArtifact@2
-  displayName: Download Cargo Tarpaulin (Linux)
-  inputs:
-    buildType: specific
-    project: mu
-    definition: 167
-    targetPath: '$(Agent.TempDirectory)'
-    itemPattern: '**/cargo-tarpaulin'
-    artifactName: Binaries
-  condition: eq(variables['Agent.OS'], 'Linux')
+- template: DownloadAzurePipelineArtifact.yml
+  parameters:
+    task_display_name: Download Cargo Tarpaulin (Linux)
+    artifact_name: Binaries
+    azure_pipeline_def_id: 167
+    file_pattern: "**/cargo-tarpaulin"
+    target_dir: "$(Agent.TempDirectory)"
+    target_os: "Linux"
+    work_dir: "$(Agent.TempDirectory)"
 - script: |
     cp $AGENT_TEMPDIRECTORY/cargo-tarpaulin /.cargo/bin
   displayName: Copy cargo-tarpaulin