Skip to main content
AI/MLjmagly

uat-execute

Execute a UAT plan against live MCP connections, tracking pass/fail per test and filing issues on failure

Stars
141
Source
jmagly/aiwg
Updated
2026-05-31
Slug
jmagly--aiwg--uat-execute
View on GitHubRaw SKILL.md

// install — copy + paste into any project

mkdir -p .claude/skills && curl -fsSL https://raw.githubusercontent.com/jmagly/aiwg/HEAD/agentic/code/addons/uat-mcp/skills/uat-execute/SKILL.md -o .claude/skills/uat-execute.md

Drops the SKILL.md into .claude/skills/uat-execute.md. Works with Claude Code, Cursor, and any agent that loads SKILL.md files from .claude/skills/.

UAT Execute

Execute a UAT plan against live MCP connections. Runs each phase sequentially, tracks pass/fail per test, files issues for failures, and always runs cleanup.

Usage

# Execute a UAT plan
/uat-execute .aiwg/testing/uat/plan-gitea-20260227.md

# Dry run — show what would be executed without making calls
/uat-execute .aiwg/testing/uat/plan.md --dry-run

# Skip issue filing
/uat-execute .aiwg/testing/uat/plan.md --no-issues

# Resume from a specific phase (after fixing a blocker)
/uat-execute .aiwg/testing/uat/plan.md --resume-from 3

# Override execution mode (run only quick smoke subset)
/uat-execute .aiwg/testing/uat/plan.md --execution-mode quick

Parameters

plan_path (required)

Path to the UAT plan document generated by /uat-generate.

--mode (default: mcp)

Test mode. Must match the plan's mode. Currently only mcp is supported.

--execution-mode (optional)

Override the plan's execution mode. Useful for running a quick smoke test from a full plan.

--dry-run (optional)

Parse the plan and show what would be executed without making any MCP calls.

--no-issues (optional)

Disable automatic issue filing for failures. Results are still recorded.

--resume-from (optional)

Resume execution from a specific phase number. Useful after fixing a blocker found mid-execution. Previous phase results are loaded from the checkpoint file.

Workflow

Step 1: Parse Plan

Read the UAT plan and extract:

  • Phase list with prerequisites
  • Test cases per phase with MCP call specs
  • Variable wiring requirements
  • Negative test isolation flags
Parsing UAT Plan...
  Plan: plan-gitea-20260227.md
  Phases: 12
  Test cases: 165
  Negative tests: 32 (will be isolated)
  Variables: 18 cross-phase references

Step 2: Preflight Check

Before executing tests, verify:

  • MCP server(s) are reachable
  • Required tools are available
  • Authentication is valid
Preflight Check...
  Server: gitea — Connected
  Tools available: 78/78
  Auth: Valid (user: roctibot)
  Ready to execute.

Step 3: Execute Phases

For each phase in order:

═══════════════════════════════════════
Phase 3: Issue Tracking (12 tests)
═══════════════════════════════════════

  P03-001: Create Issue (Happy Path)
    Tool: mcp__gitea__create_issue
    Parameters: {owner: "test-org", repo: "test-repo", title: "UAT Test Issue", body: "..."}
    Response: {number: 1, title: "UAT Test Issue", state: "open"}
    ✅ Criterion 1: Returns issue with number field — PASS
    ✅ Criterion 2: title matches — PASS
    ✅ Criterion 3: state is "open" — PASS
    Stored: ISSUE_INDEX = 1
    Result: PASS (3/3)

  P03-002: Get Issue by Index
    Tool: mcp__gitea__get_issue_by_index
    Parameters: {owner: "test-org", repo: "test-repo", index: 1}
    Response: {number: 1, title: "UAT Test Issue", ...}
    ✅ Criterion 1: Returns issue matching ISSUE_INDEX — PASS
    ✅ Criterion 2: title is "UAT Test Issue" — PASS
    Result: PASS (2/2)

  P03-003: Create Issue — Missing Title (Negative)
    [Isolation: single call]
    Tool: mcp__gitea__create_issue
    Parameters: {owner: "test-org", repo: "test-repo", body: "No title"}
    Response: {error: "validation failed"}
    ✅ Criterion 1: Returns error — PASS
    ✅ Criterion 2: Error indicates missing title — PASS
    Result: PASS (2/2)

Phase 3 Summary: 12/12 passed (0 failed, 0 skipped)
Duration: 45 seconds

Step 4: Handle Failures

On test failure:

  1. Record the failure with full details
  2. File an issue (unless --no-issues)
  3. Continue to next test (don't stop)
  4. If a stored variable was expected but not captured, mark dependent tests as error
  P04-007: Update Repository Description
    Tool: mcp__gitea__update_repo
    Parameters: {owner: "test-org", repo: "test-repo", description: "Updated"}
    Response: {error: "not found", message: "repository not found"}
    ❌ Criterion 1: Returns updated repository — FAIL
    Result: FAIL (0/1)
    Issue filed: #415 "UAT: P04-007 — Update Repository returns not found"

Step 5: Cleanup Phase

The cleanup phase ALWAYS runs, even if earlier phases failed:

═══════════════════════════════════════
Phase 11: Cleanup (always runs)
═══════════════════════════════════════

  P11-001: Delete Test Repository
    Tool: mcp__gitea__delete_repo  (if available)
    ...
  P11-002: Delete Test Organization
    ...

Cleanup complete. 3/3 test entities removed.

Step 6: Save Results

Write results to .aiwg/testing/uat/results/ following the uat-result.yaml schema:

Results saved:
  File: .aiwg/testing/uat/results/results-gitea-20260227-143022.yaml
  Checkpoint: .aiwg/testing/uat/results/checkpoint-gitea-20260227.yaml

Generate report with:
  /uat-report .aiwg/testing/uat/results/results-gitea-20260227-143022.yaml

Step 7: Summary

═══════════════════════════════════════════
UAT Execution Complete
═══════════════════════════════════════════

Plan: plan-gitea-20260227.md
Duration: 8 minutes 32 seconds

Results:
  Total tests: 165
  Passed: 159 (96.4%)
  Failed: 4 (2.4%)
  Skipped: 1 (0.6%)
  Errors: 1 (0.6%)

Issues filed: 4
  #415: P04-007 — Update Repository returns not found
  #416: P06-003 — Wiki page creation timeout
  #417: P08-002 — Release tag not created
  #418: P09-011 — Search returns stale results

Coverage: 76/78 tools tested (97.4%)
  Untested: mcp__gitea__cancel_repo_action_run (no active runs)
            mcp__gitea__stop_stopwatch (no active stopwatch)

═══════════════════════════════════════════

Error Handling

Plan Not Found

Error: UAT plan not found at {path}

Generate a plan first:
  /uat-generate --mode mcp

MCP Server Unavailable

Error: MCP server {name} is not reachable.

The UAT plan requires this server. Check your MCP configuration.
Retry with: /uat-execute {plan_path}

Checkpoint Recovery

If execution crashes mid-run, results are checkpointed per-phase:

Checkpoint detected: .aiwg/testing/uat/results/checkpoint-gitea-20260227.yaml
  Phases completed: 0-5
  Phases remaining: 6-11

Resume with: /uat-execute {plan_path} --resume-from 6

References

  • Agent: @$AIWG_ROOT/agentic/code/addons/uat-mcp/agents/uat-executor.md
  • Schema: @$AIWG_ROOT/agentic/code/addons/uat-mcp/schemas/uat-result.yaml
  • Template: @$AIWG_ROOT/agentic/code/addons/uat-mcp/templates/uat-executor-guide.md