Skip to content

Commit 83c3e6b

Browse files
makubackimakubacki
authored
.sync: Add CodeQL GitHub workflow (#133)
Adds a new workflow that is synced to Mu repos that are currently expected to run against CodeQL. This workflow has the following features to support maintainability across the repos it is synced to: - The packages are auto discovered and a dynamic matrix is generated for each package build. This allows the same file to work as-is in each repo that performs CI builds (packages are in the repo root directory). - The Mu Basecore plugin directory is auto discovered in the workspace based on the presence of the CodeQL plugin being present in the directory. - The operations supported by the Stuart CI script are dynamically discovered. - CodeQL is only run on Windows agents. There is a known issue when building edk2-style code on Linux so this avoids encountering that issue. See: github/codeql-action#1338 - The Windows CodeQL CLI package is about 260MB at this time. The GitHub Action cache is used by this workflow to cache the CLI after it is initially pulled down in the Stuart ext dep update. - The CLI ext dep directory name and version used for caching are read from the ext_dep YAML file to reduce maintenance needed in the workflow if the file changes in the future. Note that the SARIF file for each run is uploaded as a per-package artifact. These can be downloaded and opened in VS Code with the SARIF Viewer extension to view issues locally with the ability to click to issue locations in files. Signed-off-by: Michael Kubacki <michael.kubacki@microsoft.com>
1 parent 4b3b857 commit 83c3e6b

2 files changed

Lines changed: 302 additions & 0 deletions

File tree

‎.sync/Files.yml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -369,6 +369,19 @@ group:
369369
microsoft/mu_tiano_platforms
370370
microsoft/mu_tiano_plus
371371
372+
# Leaf Workflow - CodeQL
373+
- files:
374+
- source: .sync/workflows/leaf/codeql.yml
375+
dest: .github/workflows/codeql.yml
376+
repos: |
377+
microsoft/mu_basecore
378+
microsoft/mu_common_intel_min_platform
379+
microsoft/mu_feature_dfci
380+
microsoft/mu_feature_mm_supv
381+
microsoft/mu_plus
382+
microsoft/mu_silicon_intel_tiano
383+
microsoft/mu_tiano_plus
384+
372385
# Leaf Workflow - Issue Triage
373386
- files:
374387
- source: .sync/workflows/leaf/triage-issues.yml

‎.sync/workflows/leaf/codeql.yml‎

Lines changed: 289 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,289 @@
1+
# This workflow runs CodeQL against the repository.
2+
#
3+
# Results are uploaded to GitHub Code Scanning.
4+
#
5+
# Note: Important: This file currently only works with "CI" builds. "Platform" builds can
6+
# be supported without much effort but that will be done in the future.
7+
#
8+
# Note: This workflow only supports Windows as CodeQL CLI has confirmed issues running
9+
# against edk2-style codebases on Linux (only tested on Ubuntu). Therefore, this
10+
# workflow is written only for Windows but could easily be adapted to run on Linux
11+
# in the future if needed (e.g. swap out "windows" with agent OS var value, etc.)
12+
#
13+
# NOTE: This file is automatically synchronized from Mu DevOps. Update the original file there
14+
# instead of the file in this repo.
15+
#
16+
# - Mu DevOps Repo: https://github.com/microsoft/mu_devops
17+
# - File Sync Settings: https://github.com/microsoft/mu_devops/blob/main/.sync/Files.yml
18+
#
19+
# Copyright (c) Microsoft Corporation.
20+
# SPDX-License-Identifier: BSD-2-Clause-Patent
21+
22+
name: "CodeQL"
23+
24+
on:
25+
push:
26+
branches:
27+
- release/*
28+
pull_request_target:
29+
branches:
30+
- release/*
31+
paths-ignore:
32+
- '!**.c'
33+
- '!**.h'
34+
35+
jobs:
36+
gather_packages:
37+
name: Gather Repo Packages
38+
runs-on: ubuntu-latest
39+
outputs:
40+
packages: ${{ steps.generate_matrix.outputs.packages }}
41+
42+
steps:
43+
- name: Checkout repository
44+
uses: actions/checkout@v3
45+
46+
- name: Install Python
47+
uses: actions/setup-python@v4
48+
with:
49+
python-version: '>=3.11'
50+
51+
- name: Generate Package Matrix
52+
id: generate_matrix
53+
shell: python
54+
run: |
55+
import os
56+
import json
57+
58+
packages = [d for d in os.listdir() if d.strip().lower().endswith('pkg')]
59+
60+
# Ensure the package can actually be built
61+
for package in packages:
62+
if not any(file.endswith('.dsc') for file in os.listdir(package)):
63+
packages.remove(package)
64+
65+
packages.sort()
66+
67+
with open(os.environ['GITHUB_OUTPUT'], 'a') as fh:
68+
print(f'packages={json.dumps(packages)}', file=fh)
69+
70+
analyze:
71+
name: Analyze
72+
runs-on: windows-2022
73+
needs:
74+
- gather_packages
75+
permissions:
76+
actions: read
77+
contents: read
78+
security-events: write
79+
80+
strategy:
81+
fail-fast: false
82+
matrix:
83+
package: ${{ fromJson(needs.gather_packages.outputs.packages) }}
84+
include:
85+
- archs: IA32,X64
86+
- tool_chain_tag: VS2022
87+
88+
steps:
89+
- name: Checkout repository
90+
uses: actions/checkout@v3
91+
92+
- name: Install Python
93+
uses: actions/setup-python@v4
94+
with:
95+
python-version: '>=3.11'
96+
97+
- name: Install/Upgrade pip Modules
98+
run: pip install -r pip-requirements.txt --upgrade
99+
100+
- name: Determine CI Settings File Supported Operations
101+
id: get_ci_file_operations
102+
shell: python
103+
run: |
104+
import importlib
105+
import os
106+
import sys
107+
from pathlib import Path
108+
from edk2toolext.invocables.edk2_ci_setup import CiSetupSettingsManager
109+
from edk2toolext.invocables.edk2_setup import SetupSettingsManager
110+
111+
# Find the CI Settings file (usually in .pytool/CISettings.py)
112+
ci_settings_file = list(Path(os.environ['GITHUB_WORKSPACE']).rglob('.pytool/CISettings.py'))
113+
114+
# Note: At this point, submodules have not been pulled, only one CI Settings file should exist
115+
if len(ci_settings_file) != 1 or not ci_settings_file[0].is_file():
116+
print("::error title=Workspace Error!::Failed to find CI Settings file!")
117+
sys.exit(1)
118+
119+
ci_settings_file = ci_settings_file[0]
120+
121+
# Try Finding the Settings class in the file
122+
module_name = 'ci_settings'
123+
124+
spec = importlib.util.spec_from_file_location(module_name, ci_settings_file)
125+
module = importlib.util.module_from_spec(spec)
126+
spec.loader.exec_module(module)
127+
128+
try:
129+
settings = getattr(module, 'Settings')
130+
except AttributeError:
131+
print("::error title=Workspace Error!::Failed to find Settings class in CI Settings file!")
132+
sys.exit(1)
133+
134+
# Determine Which Operations Are Supported by the Settings Class
135+
ci_setup_supported = issubclass(settings, CiSetupSettingsManager)
136+
setup_supported = issubclass(settings, SetupSettingsManager)
137+
138+
with open(os.environ['GITHUB_OUTPUT'], 'a') as fh:
139+
print(f'ci_setup_supported={str(ci_setup_supported).lower()}', file=fh)
140+
print(f'setup_supported={str(setup_supported).lower()}', file=fh)
141+
142+
- name: Setup
143+
if: steps.get_ci_file_operations.outputs.setup_supported == 'true'
144+
run: stuart_setup -c .pytool/CISettings.py -t DEBUG -a ${{ matrix.archs }} TOOL_CHAIN_TAG=${{ matrix.tool_chain_tag }}
145+
146+
- name: CI Setup
147+
if: steps.get_ci_file_operations.outputs.ci_setup_supported == 'true'
148+
run: stuart_ci_setup -c .pytool/CISettings.py -t DEBUG -a ${{ matrix.archs }} TOOL_CHAIN_TAG=${{ matrix.tool_chain_tag }}
149+
150+
- name: Update
151+
run: stuart_update -c .pytool/CISettings.py -t DEBUG -a ${{ matrix.archs }} TOOL_CHAIN_TAG=${{ matrix.tool_chain_tag }}
152+
153+
- name: Find CodeQL Plugin Directory
154+
id: find_dir
155+
shell: python
156+
run: |
157+
import os
158+
import sys
159+
from pathlib import Path
160+
161+
# Find the plugin directory that contains the CodeQL plugin
162+
plugin_dir = list(Path(os.environ['GITHUB_WORKSPACE']).rglob('.pytool/Plugin/CodeQL'))
163+
164+
# This should only be found once
165+
if len(plugin_dir) == 1:
166+
plugin_dir = str(plugin_dir[0])
167+
168+
with open(os.environ['GITHUB_OUTPUT'], 'a') as fh:
169+
print(f'codeql_plugin_dir={plugin_dir}', file=fh)
170+
else:
171+
print("::error title=Workspace Error!::Failed to find Mu Basecore plugin directory!")
172+
sys.exit(1)
173+
174+
- name: Get CodeQL CLI Cache Data
175+
id: cache_key_gen
176+
env:
177+
CODEQL_PLUGIN_DIR: ${{ steps.find_dir.outputs.codeql_plugin_dir }}
178+
shell: python
179+
run: |
180+
import os
181+
import yaml
182+
183+
codeql_cli_ext_dep_name = 'codeqlcli_windows_ext_dep'
184+
codeql_plugin_file = os.path.join(os.environ['CODEQL_PLUGIN_DIR'], codeql_cli_ext_dep_name + '.yaml')
185+
186+
with open (codeql_plugin_file) as pf:
187+
codeql_cli_ext_dep = yaml.safe_load(pf)
188+
189+
cache_key_name = codeql_cli_ext_dep['name']
190+
cache_key_version = codeql_cli_ext_dep['version']
191+
cache_key = f'{cache_key_name}-{cache_key_version}'
192+
193+
codeql_plugin_cli_ext_dep_dir = os.path.join(os.environ['CODEQL_PLUGIN_DIR'], codeql_cli_ext_dep['name'].strip() + '_extdep')
194+
195+
with open(os.environ['GITHUB_OUTPUT'], 'a') as fh:
196+
print(f'codeql_cli_cache_key={cache_key}', file=fh)
197+
print(f'codeql_cli_ext_dep_dir={codeql_plugin_cli_ext_dep_dir}', file=fh)
198+
199+
- name: Attempt to Load CodeQL CLI From Cache
200+
id: codeqlcli_cache
201+
uses: actions/cache@v3
202+
with:
203+
path: ${{ steps.cache_key_gen.outputs.codeql_cli_ext_dep_dir }}
204+
key: ${{ steps.cache_key_gen.outputs.codeql_cli_cache_key }}
205+
206+
- name: Download CodeQL CLI
207+
if: steps.codeqlcli_cache.outputs.cache-hit != 'true'
208+
run: stuart_update -c .pytool/CISettings.py -t DEBUG -a ${{ matrix.archs }} TOOL_CHAIN_TAG=${{ matrix.tool_chain_tag }} --codeql
209+
210+
- name: Remove CI Plugins Irrelevant to CodeQL
211+
shell: python
212+
env:
213+
CODEQL_PLUGIN_DIR: ${{ steps.find_dir.outputs.codeql_plugin_dir }}
214+
run: |
215+
import os
216+
import shutil
217+
from pathlib import Path
218+
219+
# Only these two plugins are needed for CodeQL
220+
plugins_to_keep = ['CodeQL', 'CompilerPlugin']
221+
222+
plugin_dir = Path(os.environ['CODEQL_PLUGIN_DIR']).parent.absolute()
223+
if plugin_dir.is_dir():
224+
for dir in plugin_dir.iterdir():
225+
if str(dir.stem) not in plugins_to_keep:
226+
shutil.rmtree(str(dir.absolute()), ignore_errors=True)
227+
228+
- name: CI Build
229+
env:
230+
STUART_CODEQL_PATH: ${{ steps.cache_key_gen.outputs.codeql_cli_ext_dep_dir }}
231+
run: stuart_ci_build -c .pytool/CISettings.py -t DEBUG -p ${{ matrix.package }} -a ${{ matrix.archs }} TOOL_CHAIN_TAG=${{ matrix.tool_chain_tag }} --codeql
232+
233+
- name: Prepare Env Data for CodeQL Upload
234+
id: env_data
235+
env:
236+
PACKAGE_NAME: ${{ matrix.package }}
237+
shell: python
238+
run: |
239+
import os
240+
241+
package = os.environ['PACKAGE_NAME'].strip().lower()
242+
directory_name = 'codeql-analysis-' + package + '-debug'
243+
file_name = 'codeql-db-' + package + '-debug-0.sarif'
244+
sarif_path = os.path.join('Build', directory_name, file_name)
245+
246+
with open(os.environ['GITHUB_OUTPUT'], 'a') as fh:
247+
print(f'sarif_file_path={sarif_path}', file=fh)
248+
249+
- name: Upload Setup and Update Logs As An Artifact
250+
uses: actions/upload-artifact@v3
251+
with:
252+
name: ${{ matrix.package }}-Setup-Update-Logs
253+
path: |
254+
**/OVERRIDELOG.TXT
255+
CISETUP.txt
256+
SETUPLOG.txt
257+
UPDATE_LOG.txt
258+
retention-days: 3
259+
if-no-files-found: ignore
260+
261+
- name: Upload Build Log As An Artifact
262+
uses: actions/upload-artifact@v3
263+
with:
264+
name: ${{ matrix.package }}-Build-Logs
265+
path: |
266+
**/BUILD_REPORT.TXT
267+
BUILDLOG_*.md
268+
BUILDLOG_*.txt
269+
CI_*.md
270+
CI_*.txt
271+
retention-days: 7
272+
if-no-files-found: ignore
273+
274+
- name: Upload CodeQL Results (SARIF) As An Artifact
275+
uses: actions/upload-artifact@v3
276+
with:
277+
name: ${{ matrix.package }}-CodeQL-SARIF
278+
path: ${{ steps.env_data.outputs.sarif_file_path }}
279+
retention-days: 14
280+
if-no-files-found: warn
281+
282+
- name: Upload CodeQL Results (SARIF) To GitHub Code Scanning
283+
uses: github/codeql-action/upload-sarif@v2
284+
with:
285+
# Path to SARIF file relative to the root of the repository.
286+
sarif_file: ${{ steps.env_data.outputs.sarif_file_path }}
287+
# Optional category for the results. Used to differentiate multiple results for one commit.
288+
# Each package is a separate category.
289+
category: ${{ matrix.package }}

0 commit comments

Comments
 (0)