Skip to content

Commit f4a7942

Browse files
Jian WuCopilot
Jian Wu
and
Copilot
committed
Add E2E test pipeline for azure.ai.agents extension
MCP-tool-driven workflow for running cli-interactive-tester scenarios. Manual dispatch only (workflow_dispatch) with tier selection (0, 0+1, 0+1+2). Key design: - All scenarios executed via cli-interactive-tester MCP tool (not shell parsing) - Tool installed via git clone + pip install -e from coreai-microsoft repo - Checkout hardcoded to trangevi/test-scenarios (until PR Azure#8524 merges) - ubuntu-22.04 runner (consistent with existing pipelines) - profile.local.yaml generated from GitHub secrets at runtime - Tier 2 includes always-run teardown step for resource cleanup - Results uploaded as artifacts Blocking: python -m auto_test_tool.runner batch mode needs to be confirmed or implemented. Without it, scenarios cannot run headlessly in CI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 3672e81 commit f4a7942

1 file changed

Lines changed: 143 additions & 0 deletions

File tree

‎.github/workflows/e2e-ext-azure-ai-agents.yml‎

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
name: "ext-azure-ai-agents-e2e"
2+
3+
# E2E tests for the azure.ai.agents CLI extension.
4+
# Driven by cli-interactive-tester MCP tool (tmux-based scenario execution).
5+
#
6+
# Manual trigger only — switch to PR-trigger once pipeline is stable.
7+
# Scenarios live on branch trangevi/test-scenarios until PR #8524 merges.
8+
9+
on:
10+
workflow_dispatch:
11+
inputs:
12+
tier:
13+
description: "Which tiers to run"
14+
type: choice
15+
options:
16+
- "0"
17+
- "0+1"
18+
- "0+1+2"
19+
default: "0"
20+
confirm_tier2_cost:
21+
description: "Confirm Tier 2 Azure costs (~$2-5)"
22+
type: boolean
23+
default: false
24+
25+
concurrency:
26+
group: ${{ github.workflow }}-${{ github.run_id }}
27+
cancel-in-progress: true
28+
29+
permissions:
30+
contents: read
31+
32+
env:
33+
SCENARIOS_DIR: cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios
34+
AZD_AGENTS_FIXTURES: cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/fixtures
35+
# TODO: change to 'main' after PR #8524 merges
36+
TARGET_BRANCH: trangevi/test-scenarios
37+
38+
jobs:
39+
e2e-test:
40+
name: "E2E scenarios (Tier ${{ inputs.tier }})"
41+
runs-on: ubuntu-22.04
42+
timeout-minutes: 90
43+
steps:
44+
- uses: actions/checkout@v6
45+
with:
46+
# TODO: remove ref after PR #8524 merges to main
47+
ref: ${{ env.TARGET_BRANCH }}
48+
49+
- uses: actions/setup-go@v6
50+
with:
51+
go-version-file: "cli/azd/go.mod"
52+
53+
- uses: actions/setup-python@v5
54+
with:
55+
python-version: "3.12"
56+
57+
- name: Build azd + install extension
58+
working-directory: cli/azd
59+
run: |
60+
go build -o ./azd .
61+
export PATH="$PWD:$PATH"
62+
azd extension install azure.ai.agents --source ./extensions/azure.ai.agents
63+
64+
- name: Install cli-interactive-tester
65+
run: |
66+
sudo apt-get install -y tmux
67+
# TODO: confirm repo visibility. If private, need PAT:
68+
# git clone https://x-access-token:${{ secrets.GH_TOKEN }}@github.com/coreai-microsoft/cli-interactive-tester.git
69+
# TODO: confirm preferred distribution (git clone vs pip install from PyPI/registry)
70+
git clone https://github.com/coreai-microsoft/cli-interactive-tester.git /tmp/cli-interactive-tester
71+
cd /tmp/cli-interactive-tester
72+
pip install -e .
73+
python -c "from auto_test_tool.mcp_server import main; print('MCP tool ready')"
74+
75+
- name: Install uv (for Tier 2 run-local scenario)
76+
if: contains(inputs.tier, '2')
77+
uses: astral-sh/setup-uv@v6
78+
79+
- name: Azure Login
80+
uses: azure/login@v2
81+
with:
82+
client-id: ${{ secrets.AZURE_CLIENT_ID }}
83+
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
84+
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
85+
86+
- name: GitHub CLI Auth
87+
run: echo "${{ secrets.GH_TOKEN }}" | gh auth login --with-token
88+
89+
- name: Create test profile
90+
working-directory: ${{ env.SCENARIOS_DIR }}
91+
run: |
92+
cat > profile.local.yaml <<EOF
93+
prefix: "ci-${{ github.run_number }}"
94+
subscription: "${{ secrets.AZURE_SUBSCRIPTION_ID }}"
95+
region: "North Central US"
96+
foundry_project_endpoint: "${{ secrets.FOUNDRY_PROJECT_ENDPOINT }}"
97+
EOF
98+
99+
- name: Run scenarios via MCP tool
100+
working-directory: ${{ env.SCENARIOS_DIR }}
101+
env:
102+
AZD_AGENTS_FIXTURES: ${{ github.workspace }}/${{ env.AZD_AGENTS_FIXTURES }}
103+
UV_HTTP_TIMEOUT: "300"
104+
run: |
105+
export PATH="${{ github.workspace }}/cli/azd:$PATH"
106+
# Drive all scenarios through cli-interactive-tester.
107+
# Each scenario is: load_scenario → run_pre_hooks → start_session
108+
# → [driver accomplishes goals] → finish_session → run_post_hooks
109+
#
110+
# TODO: confirm batch runner CLI exists. If not, need to implement
111+
# a wrapper that calls auto_test_tool internal APIs directly.
112+
python -m auto_test_tool.runner \
113+
--scenarios-dir . \
114+
--profile profile.yaml \
115+
--profile-local profile.local.yaml \
116+
--tier ${{ inputs.tier }} \
117+
--output results.json
118+
119+
- name: Upload results
120+
if: always()
121+
uses: actions/upload-artifact@v4
122+
with:
123+
name: e2e-results-tier-${{ inputs.tier }}
124+
path: |
125+
${{ env.SCENARIOS_DIR }}/results.json
126+
${{ env.SCENARIOS_DIR }}/reports/
127+
retention-days: 30
128+
129+
- name: Teardown (Tier 2 always cleanup)
130+
if: always() && contains(inputs.tier, '2')
131+
working-directory: ${{ env.SCENARIOS_DIR }}
132+
env:
133+
AZD_AGENTS_FIXTURES: ${{ github.workspace }}/${{ env.AZD_AGENTS_FIXTURES }}
134+
run: |
135+
export PATH="${{ github.workspace }}/cli/azd:$PATH"
136+
# Ensure Azure resources are cleaned up even on failure
137+
python -m auto_test_tool.runner \
138+
--scenarios-dir . \
139+
--scenario 2Z-teardown-down.yaml \
140+
--profile profile.yaml \
141+
--profile-local profile.local.yaml || true
142+
143+

0 commit comments

Comments
 (0)