Run Judges Skill

Purpose

Execute specialized judge agents in parallel to evaluate implementation plan quality (16 judges, 4 batches), code quality (11 judges, 3 batches), PRD quality (5 judges, 2 batches), or Feature quality (3 judges, 1 batch). All batches respect the Task tool's 4-concurrent-agent limit. Aggregates results into $CLOSEDLOOP_WORKDIR/plan-judges.json (plan), $CLOSEDLOOP_WORKDIR/code-judges.json (code), $CLOSEDLOOP_WORKDIR/prd-judges.json (prd), or $CLOSEDLOOP_WORKDIR/feature-judges.json (feature) with validated output format.

Parameters

--workdir: Path to the working directory containing judge artifacts (optional)

• Resolved in order: --workdir argument → $CLOSEDLOOP_WORKDIR environment variable → .closedloop-ai/judges (default, relative to current working directory)
• The directory is created automatically if it does not exist
• All output files (plan-judges.json, code-judges.json, prd-judges.json, judge-input.json, perf.jsonl, etc.) are written to this resolved directory

--artifact-type: Artifact category to evaluate (plan | code | prd | feature), default: plan

• plan (default): Evaluate implementation plan with 16 judges, 4 batches, output to plan-judges.json
• code: Evaluate implemented code with 11 judges, 3 batches, output to code-judges.json
• prd: Evaluate PRD document with 5 judges across 2 sequential batches (3 + 2, max 4 concurrent per batch), output to prd-judges.json
• feature: Evaluate Feature artifact with 3 judges, 1 batch, output to feature-judges.json

Judge Input Contract (judge-input.json)

The judge input contract is maintained in:

skills/run-judges/references/judge-input-contract.md (resolve to an absolute path at runtime via Glob)

This keeps orchestration flow readable while preserving a single source of truth for contract fields and semantics.

run-judges is the producer chokepoint for judge-input.json. After mode-specific context preparation and before launching any judge agent, invoke the deterministic mapper:

uv run "${CLAUDE_PLUGIN_ROOT}/skills/run-judges/scripts/judge_input_mapping.py" \
  --workdir "$CLOSEDLOOP_WORKDIR" \
  --artifact-type "$ARTIFACT_TYPE" \
  --schema "${CLAUDE_PLUGIN_ROOT}/schemas/judge-input.schema.json"

The mapper builds from the runtime workdir contract: primary artifacts under <runDir>, supporting context under <runDir>/.closedloop-ai/context, and attachments under <runDir>/.closedloop-ai/work/attachments. It validates the generated envelope against schemas/judge-input.schema.json before judge launch. If mapping fails, emit a clear warning and use the documented one-run legacy fallback paths (prd.md, plan.md, or existing compatibility artifacts) only for that run.

Task Context

You are orchestrating quality evaluation for a ClosedLoop artifact (implementation plan, code, or PRD). Your responsibilities:

For plan artifacts (default):

1. Launch context-manager-for-judges agent to prepare compressed plan context
2. Build judge-input.json with plan task/context mapping
3. Launch all 16 judge agents in parallel batches
4. Aggregate their CaseScore outputs into a valid EvaluationReport
5. Write the report to $CLOSEDLOOP_WORKDIR/plan-judges.json
6. Validate output structure and completeness

For code artifacts (--artifact-type code):

1. Launch context-manager-for-judges agent to prepare compressed context
2. Build judge-input.json with code task/context mapping
3. Launch 11 judge agents in parallel batches
4. Aggregate their CaseScore outputs into a valid EvaluationReport
5. Write the report to $CLOSEDLOOP_WORKDIR/code-judges.json
6. Validate output structure and completeness

For PRD artifacts (--artifact-type prd):

1. Check $CLOSEDLOOP_WORKDIR/prd.md exists (graceful exit if missing)
2. Build and schema-validate judge-input.json by invoking scripts/judge_input_mapping.py
3. Launch the 5 PRD judges in 2 sequential batches (3 + 2, max 4 concurrent per batch)
4. Aggregate all 5 CaseScores into a valid EvaluationReport
5. Write the report to $CLOSEDLOOP_WORKDIR/prd-judges.json
6. Validate output structure and completeness

For Feature artifacts (--artifact-type feature):

1. Check $CLOSEDLOOP_WORKDIR/feature.md exists, or $CLOSEDLOOP_WORKDIR/prd.md exists for legacy Feature inputs (graceful exit code 0 if both are missing)
2. Build and schema-validate judge-input.json by invoking scripts/judge_input_mapping.py
3. Launch 3 judges in 1 batch (feature-completeness-judge + prd-testability-judge + prd-dependency-judge)
4. Aggregate 3 CaseScores into a valid EvaluationReport
5. Write the report to $CLOSEDLOOP_WORKDIR/feature-judges.json
6. Validate output structure and completeness

Feature mode judge selection rationale:

• prd-auditor is excluded because it assumes US-###/AC-#.# numbering and multi-story traceability, which Feature artifacts do not follow
• prd-scope-judge is excluded because it assumes In/Out-of-Scope sections that are not present in Feature artifacts

Feature mode preamble: Feature mode uses the dedicated feature_preamble.md so judges receive a Feature-shaped contract (evaluation_type=feature, lightweight structure, no PRD-only sections). Do NOT substitute prd_preamble.md — it would frame the input as a full PRD and contradict the envelope's evaluation_type.

Success criteria:

• All judges executed (or error CaseScores generated for failures)
• Valid JSON written to appropriate output file
• Validation script passes with zero errors

Threshold Overrides

The run-judges skill supports per-artifact-type threshold customization via JSON configuration files. This allows you to adjust evaluation strictness for different artifact types (e.g., applying a lower threshold for test-judge when evaluating code vs plan).

Configuration Schema

Threshold overrides are defined in a JSON file with the following structure:

{
  "overrides": {
    "artifact_type:judge_name": <threshold_float>
  }
}

Where:

• Key format: "artifact_type:judge_name" (e.g., "code:test-judge", "plan:technical-accuracy-judge")
• Value: Threshold as a float in range [0.0, 1.0]

Example configuration:

{
  "overrides": {
    "code:test-judge": 0.75,
    "plan:technical-accuracy-judge": 0.85
  }
}

Loading Precedence

The skill checks the following locations in order, using the first valid configuration found:

1. Run-specific overrides (highest precedence):

   • Path: $CLOSEDLOOP_WORKDIR/.closedloop-ai/settings/threshold-overrides.json
   • Use case: Override thresholds for a specific ClosedLoop run

2. Repo-level defaults (fallback):

   • Path: <project-root>/.closedloop-ai/settings/threshold-overrides.json
   • Use case: Set project-wide threshold defaults

3. Hardcoded defaults (graceful degradation):

   • If no configuration file exists at any location, use built-in defaults
   • No error is raised for missing configuration files

Default Overrides

The following default overrides apply when evaluating code artifacts:

All other judges use the same threshold (typically 0.8) across artifact types.

Validation and Error Handling

When loading threshold overrides, the skill applies the following validation rules:

Schema Validation:

• Configuration must contain an "overrides" key
• Each key must match the pattern artifact_type:judge_name
• Each value must be a float in range [0.0, 1.0]
• Keys must reference valid artifact types (plan, code, prd) and judge names

Error Behavior:

• Malformed JSON: Log warning and continue with hardcoded defaults

  Warning: Invalid threshold-overrides.json, skipping overrides: {error}
  

• Invalid schema: Log warning and continue with hardcoded defaults
• File not found: Silently use defaults (no warning logged)

Error recovery ensures the skill always completes judge execution, even if threshold configuration is incorrect.

Integration with Judge Execution

When executing judges:

1. Before launching judge batches: Load threshold overrides from the precedence chain
2. Merge with defaults: Loaded overrides take precedence over hardcoded defaults
3. Apply per-judge: Each judge receives its artifact-type-specific threshold via the evaluation context
4. CaseScore validation: Thresholds are used to determine final_status (pass/fail) based on metric scores

When artifact type is code:

• Load threshold overrides before executing judge batches
• Apply code-specific thresholds to each judge's evaluation criteria
• Merge loaded overrides with defaults (loaded values take precedence)

Performance Instrumentation (Mandatory)

You MUST emit a pipeline_step event to $CLOSEDLOOP_WORKDIR/perf.jsonl at the end of each phase below. This keeps perf telemetry in the canonical schema and adds nested metadata for judge/sub-agent work.

Context: CLOSEDLOOP_WORKDIR, CLOSEDLOOP_RUN_ID, and CLOSEDLOOP_ITERATION are set by the run-loop. CLOSEDLOOP_PARENT_STEP and CLOSEDLOOP_PARENT_STEP_NAME are set as env vars on the claude invocation by run-loop; they are inherited by all Bash tool calls — no sourcing needed. Use sub_step as numeric phase order and optional sub_step_name to capture the judge/sub-agent name when applicable (for batch-level phases where many judges run, use the batch label).

Sub-step numbering:

Start of phase (run Bash once at the beginning of each phase): Set the two sub-step variables at the top for the current phase, then run the block. It writes start time to a temp file so the end-of-phase Bash can compute duration. CLOSEDLOOP_PARENT_STEP and CLOSEDLOOP_PARENT_STEP_NAME are already in the environment (set by run-loop on the claude invocation).

# Set these two values for the current phase:
SUB_STEP_NUM=0
SUB_STEP_LABEL="context_manager"   # context_manager | batch_1 … | aggregate | validate

mkdir -p "$CLOSEDLOOP_WORKDIR/.closedloop-ai"
{
  echo "SUB_STEP=${SUB_STEP_NUM}"
  echo "SUB_STEP_NAME=${SUB_STEP_LABEL}"
  echo "PARENT_STEP=${CLOSEDLOOP_PARENT_STEP:-0}"
  echo "PARENT_STEP_NAME=${CLOSEDLOOP_PARENT_STEP_NAME:-unknown}"
  echo "STARTED_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
  echo "START_EPOCH=$(date +%s)"
} > "$CLOSEDLOOP_WORKDIR/.closedloop-ai/perf-substep-start.env"

End of phase (run Bash once at the end of each phase, after the phase work is done): Read start time, compute duration, append one line to perf.jsonl, then remove the temp file.

source "$CLOSEDLOOP_WORKDIR/.closedloop-ai/perf-substep-start.env"
END_EPOCH=$(date +%s)
ENDED_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
DURATION=$((END_EPOCH - START_EPOCH))
jq -n -c \
  --arg event "pipeline_step" \
  --arg run_id "${CLOSEDLOOP_RUN_ID:-unknown}" \
  --argjson iteration "${CLOSEDLOOP_ITERATION:-0}" \
  --argjson step "$PARENT_STEP" \
  --arg step_name "$PARENT_STEP_NAME" \
  --argjson sub_step "$SUB_STEP" \
  --arg sub_step_name "$SUB_STEP_NAME" \
  --arg started_at "$STARTED_AT" \
  --arg ended_at "$ENDED_AT" \
  --argjson duration_s "$DURATION" \
  --argjson exit_code 0 \
  --argjson skipped false \
  '{event:$event,run_id:$run_id,iteration:$iteration,step:$step,step_name:$step_name,sub_step:$sub_step,sub_step_name:$sub_step_name,started_at:$started_at,ended_at:$ended_at,duration_s:$duration_s,exit_code:$exit_code,skipped:$skipped}' >> "$CLOSEDLOOP_WORKDIR/perf.jsonl"
rm -f "$CLOSEDLOOP_WORKDIR/.closedloop-ai/perf-substep-start.env"

Order of operations per phase: Run the "start of phase" Bash first (set SUB_STEP_NUM and SUB_STEP_LABEL at the top, then run the block), then perform the phase work, then run the "end of phase" Bash.

Execution Workflow

Working Directory Resolution

Before any other step, resolve the working directory and export it as CLOSEDLOOP_WORKDIR:

# Resolve working directory (precedence: --workdir arg > env var > default)
if [ -n "$ARG_WORKDIR" ]; then
  WORKDIR="$ARG_WORKDIR"
elif [ -n "$CLOSEDLOOP_WORKDIR" ]; then
  WORKDIR="$CLOSEDLOOP_WORKDIR"
else
  WORKDIR="$(pwd)/.closedloop-ai/judges"
fi

mkdir -p "$WORKDIR"
export CLOSEDLOOP_WORKDIR="$WORKDIR"

Where $ARG_WORKDIR is the value passed via --workdir in the invocation prompt. All subsequent references to $CLOSEDLOOP_WORKDIR use this resolved value.

Agents Snapshot (Pre-Step)

Before any judge execution, ensure a snapshot of judge agent definitions exists in $CLOSEDLOOP_WORKDIR/agents-snapshot/. This preserves the exact agent versions used for each evaluation run.

Action: Run the snapshot script via Bash:

bash "${CLAUDE_PLUGIN_ROOT}/skills/run-judges/scripts/ensure_agents_snapshot.sh" "$CLOSEDLOOP_WORKDIR"

The script is idempotent — it skips if manifest.json already exists.

Error handling: If the script fails or is not found, log a warning and continue — snapshot failure must not block judge execution.

Agent Registry Validation (Pre-Flight Check)

Before any judge execution, validate the agent registry to ensure all judge agents required for the current artifact type are resolvable. This prevents launching batches only to discover agents are missing mid-run.

Action: Run validate_agent_registry.py via Bash:

uv run "${CLAUDE_PLUGIN_ROOT}/tools/python/validate_agent_registry.py" \
  --artifact-type "$ARTIFACT_TYPE" \
  --workdir "$CLOSEDLOOP_WORKDIR"

Exit behavior:

• Exit code 0 — all required agents are registered; proceed with judge execution
• Exit code non-zero — one or more required agents are missing or unresolvable; abort immediately and do NOT proceed to judge batches

On failure:

• Log the validation error output in full
• Exit the skill with a non-zero status code
• Do NOT generate partial error CaseScores for this failure mode (the workflow should not proceed at all)

Step 0: Mandatory Contract Pre-Read

Before any prerequisite checks or judge launches:

1. Resolve the contract file path using Glob with:
   • **/skills/run-judges/references/judge-input-contract.md
2. Read the resolved judge-input-contract.md file in full.
3. Apply the contract requirements when constructing $CLOSEDLOOP_WORKDIR/judge-input.json.
4. If the file is missing, ambiguous (multiple matches), or unreadable, fail fast with a clear error (do not proceed with judge execution).

Prerequisites Check

Performance: At the start of this phase run the "start of phase" Bash with SUB_STEP_NUM=0 and SUB_STEP_LABEL=context_manager for both plan and code modes. For prd and feature modes, emit sub_step=0 with SUB_STEP_LABEL=context_prep and skipped=true immediately (no context manager runs). At the end of the phase run the "end of phase" Bash.

Before starting, verify required inputs exist:

For plan artifacts (default):

# Validate input files exist
if [ ! -f "$CLOSEDLOOP_WORKDIR/prd.md" ]; then
  echo "WARNING: $CLOSEDLOOP_WORKDIR/prd.md not found. Skipping judges."
  exit 0  # Graceful skip - do not fail workflow
fi

if [ ! -f "$CLOSEDLOOP_WORKDIR/plan.json" ]; then
  echo "WARNING: $CLOSEDLOOP_WORKDIR/plan.json not found. Skipping judges."
  exit 0
fi

Investigation log resolution (plan mode):

After validating prd.md and plan.json, resolve supporting context for plan judges:

1. Use existing file first

   • If $CLOSEDLOOP_WORKDIR/investigation-log.md exists, use it as-is.

2. Check @code:pre-explorer availability before invoking

   • Perform an explicit capability probe for @code:pre-explorer in the active Claude/plugin environment.
   • Treat "unknown agent", "agent not found", or plugin resolution errors as pre-explorer unavailable.
   • Recommended probe pattern:
     • Attempt a minimal Task() call targeting @code:pre-explorer.
     • If the platform rejects the agent type before execution, classify as unavailable and continue to internal fallback.

3. If available, invoke pre-explorer

   • Launch @code:pre-explorer with WORKDIR=$CLOSEDLOOP_WORKDIR to generate missing pre-exploration artifacts.
   • Re-check for $CLOSEDLOOP_WORKDIR/investigation-log.md after completion.

4. If unavailable or invocation failed, run internal fallback

   • Generate investigation-log.md with a lightweight local-only investigation.
   • Keep it fast and deterministic (no external web research).
   • Internal fallback should:
     • Read prd.md and extract top entities/actions as search seeds.
     • Run targeted Glob/Grep against the local repository for likely implementation files.
     • Record top relevant files and short rationale under Files Discovered / Key Findings.
     • Add requirement-to-code evidence links under Requirements Mapping.
   • Use the canonical sections:
     • ## Search Strategy
     • ## Files Discovered
     • ## Key Findings
     • ## Requirements Mapping
     • ## Uncertainties

5. Never block plan context preparation on investigation context

   • If log generation still fails, emit a warning and continue.

6. Prepare plan-context.json via context-manager-for-judges

   • Launch @judges:context-manager-for-judges with artifact_type=plan.
   • Verify $CLOSEDLOOP_WORKDIR/plan-context.json exists.
   • If missing after invocation, log warning and activate compatibility mode for this run:
     • Compatibility mode allows one emergency fallback to raw plan.json + prd.md.
     • Use compatibility mode only when context generation fails.

7. Plan-mode source-of-truth policy

   • Normal mode: plan-context.json is primary and required.
   • Compatibility mode: plan.json + prd.md may be used for this run only.

8. Build plan-mode judge-input.json

   • Invoke scripts/judge_input_mapping.py with --artifact-type plan.
   • The mapper sets evaluation_type, task, primary_artifact, supporting_artifacts, source_of_truth, fallback_mode, and metadata from the runtime workdir contract.
   • In compatibility mode, allow the mapper to produce a schema-valid fallback envelope for existing plan compatibility artifacts and include prd.md as supporting evidence when available.
   • If the mapper exits non-zero, log the error and use the one-run legacy fallback only if prd.md plus plan.md or the existing compatibility artifact is readable.

For code artifacts (--artifact-type code):

# Resolve investigation context for code judges (best effort)
if [ ! -f "$CLOSEDLOOP_WORKDIR/investigation-log.md" ]; then
  echo "INFO: investigation-log.md missing. Attempting best-effort generation via @code:pre-explorer..."
  # Launch @code:pre-explorer with WORKDIR=$CLOSEDLOOP_WORKDIR
  # If unavailable/fails, continue with warning (non-blocking for code judges)
fi

# Launch context-manager-for-judges agent to prepare compressed context
# This agent reads code artifacts (git diff, changed-files.json, etc.)
# and produces .closedloop-ai/context/code-context.json with token-budgeted compression

# investigation-log.md is optional secondary context for code judging
if [ ! -f "$CLOSEDLOOP_WORKDIR/investigation-log.md" ]; then
  echo "WARNING: investigation-log.md unavailable. Continuing code judges with canonical code context only."
fi

# Verify canonical code context exists after context manager completes. The root
# code-context.json path is fallback-only for old runs.
if [ ! -f "$CLOSEDLOOP_WORKDIR/.closedloop-ai/context/code-context.json" ] && [ ! -f "$CLOSEDLOOP_WORKDIR/code-context.json" ]; then
  echo "ERROR: Context preparation failed - .closedloop-ai/context/code-context.json not found"
  # Abort with error CaseScore for all judges
  # Generate error report with final_status=3, justification="Context preparation failed"
  exit 1
fi

# Build and validate code-mode judge-input.json with scripts/judge_input_mapping.py.
# The mapper prefers .closedloop-ai/context/code-context.json as primary and
# preserves root code-context.json as a one-run legacy fallback when needed.

For PRD artifacts (--artifact-type prd):

PRD mode does NOT use context-manager-for-judges. Context preparation is lightweight: verify the PRD document exists, then build judge-input.json directly from it.

# PRD mode context prep: check prd.md exists
if [ ! -f "$CLOSEDLOOP_WORKDIR/prd.md" ]; then
  echo "WARNING: $CLOSEDLOOP_WORKDIR/prd.md not found. Skipping PRD judges."
  exit 0  # Graceful exit — do not fail parent workflow
fi

# Build and validate prd-mode judge-input.json with scripts/judge_input_mapping.py.
# The mapper sets primary_artifact to primary_prd and includes mapped context,
# prompt, repo metadata, prior summaries, and attachments in source_of_truth order.

PRD context prep notes:

• Missing prd.md results in a WARNING and graceful exit (code 0), not an error
• No context manager is launched; judge-input.json is built by scripts/judge_input_mapping.py and validated against schemas/judge-input.schema.json
• Performance: emit sub_step=0 (context_prep, skipped=true) perf event immediately, then proceed to sub_step=1 (batch_1) and sub_step=2 (batch_2)

For Feature artifacts (--artifact-type feature):

Feature mode does NOT use context-manager-for-judges. Context preparation is lightweight: verify feature.md exists, or prd.md exists for legacy Feature inputs, then build judge-input.json from the mapper.

# Feature mode context prep: check feature.md or legacy prd.md exists
if [ ! -f "$CLOSEDLOOP_WORKDIR/feature.md" ] && [ ! -f "$CLOSEDLOOP_WORKDIR/prd.md" ]; then
  echo "WARNING: neither $CLOSEDLOOP_WORKDIR/feature.md nor legacy $CLOSEDLOOP_WORKDIR/prd.md found. Skipping Feature judges."
  exit 0  # Graceful exit — do not fail parent workflow
fi

# Build and validate feature-mode judge-input.json with scripts/judge_input_mapping.py.
# The mapper prefers feature.md and marks fallback_mode.active=true when it must
# use the legacy prd.md Feature path.

Feature context prep notes:

• Missing both feature.md and legacy prd.md results in a WARNING and graceful exit (code 0), not an error
• No context manager is launched; judge-input.json is built by scripts/judge_input_mapping.py with evaluation_type="feature"
• Performance: emit sub_step=0 (context_prep, skipped=true) perf event immediately, then proceed to sub_step=1 (batch_1), sub_step=2 (aggregate), sub_step=3 (validate)
• Preamble: use feature_preamble.md for all 3 feature judges

If required files are missing:

• Plan mode: Exit gracefully with code 0 (do not fail parent workflow)
• Code mode: Exit with error if context preparation fails
• PRD mode: Exit gracefully with code 0 if prd.md is not found
• Feature mode: Exit gracefully with code 0 if prd.md is not found

Artifact Type Configuration

The run-judges skill supports three artifact types with different judge configurations:

Plan Artifacts (Default)

• Judges: 16 total
• Batches: 4 sequential batches (max 4 concurrent per batch)
• Output: plan-judges.json
• Report ID: {RUN_ID}-plan-judges
• Validation: --category plan (16 judges expected)

Code Artifacts (--artifact-type code)

• Judges: 11 total (excludes goal-alignment-judge, verbosity-judge)
• Batches: 3 sequential batches (max 4 concurrent per batch)
• Output: code-judges.json
• Report ID: {RUN_ID}-code-judges
• Validation: --category code (11 judges expected)

Code Judge Batches:

Batch 1: Core Principles (4 judges)

• judges:dry-judge
• judges:ssot-judge
• judges:kiss-judge
• judges:code-organization-judge

Batch 2: Best Practices + SOLID Principles (4 judges)

• judges:custom-best-practices-judge
• judges:readability-judge
• judges:solid-isp-dip-judge
• judges:solid-liskov-substitution-judge

Batch 3: Technical Quality + Testing (3 judges)

• judges:solid-open-closed-judge
• judges:technical-accuracy-judge
• judges:test-judge

PRD Artifacts (--artifact-type prd)

• Judges: 5 total
• Batches: 2 sequential batches (max 4 concurrent per batch)
• Output: prd-judges.json
• Report ID: {RUN_ID}-prd-judges
• Validation: --category prd (5 judges expected)
• Canonical input: $CLOSEDLOOP_WORKDIR/judge-input.json produced by scripts/judge_input_mapping.py, with primary_prd normally pointing to $CLOSEDLOOP_WORKDIR/prd.md

Feature Artifacts (--artifact-type feature)

• Judges: 3 total (feature-completeness-judge, prd-testability-judge, prd-dependency-judge)
• Batches: 1 batch (max 4 concurrent per batch)
• Output: feature-judges.json
• Report ID: {RUN_ID}-feature-judges
• Validation: --category feature (3 judges expected)
• Canonical input: $CLOSEDLOOP_WORKDIR/judge-input.json produced by scripts/judge_input_mapping.py, with primary_feature normally pointing to feature.md and legacy fallback to prd.md
• Preamble: use feature_preamble.md (Feature-shaped contract; do NOT substitute prd_preamble.md)

Feature Mode Execution:

Batch 1: Feature Quality (sub_step=1)

• judges:feature-completeness-judge — evaluates Feature request completeness and clarity
• judges:prd-testability-judge — evaluates requirement testability
• judges:prd-dependency-judge — evaluates dependency clarity and completeness

PRD Mode Execution:

Batch 1: Structure & Completeness (sub_step=1)

• judges:feature-completeness-judge — evaluates Feature request completeness and clarity
• judges:prd-auditor — structural completeness audit of the PRD
• judges:prd-scope-judge — evaluates scope definition and boundary clarity

Batch 2: Quality Gates (sub_step=2)

• judges:prd-dependency-judge — evaluates dependency clarity and completeness
• judges:prd-testability-judge — evaluates requirement testability

Step 1: Launch Judge Agents in Parallel

Performance: For each batch/phase, run "start of phase" Bash before launching the batch and "end of phase" Bash after the batch completes. Plan: batch_1=sub_step 1, batch_2=sub_step 2, batch_3=sub_step 3, batch_4=sub_step 4. Code: batch_1=sub_step 1, batch_2=sub_step 2, batch_3=sub_step 3. PRD: batch_1=sub_step 1, batch_2=sub_step 2. Feature: batch_1=sub_step 1.

Constraint: The Task tool supports maximum 4 concurrent agents per batch.

Action: Launch judges in sequential batches based on artifact type.

WORKDIR=$CLOSEDLOOP_WORKDIR. Read $CLOSEDLOOP_WORKDIR/judge-input.json first.
Evaluate according to `task` and `source_of_truth` ordering.
Treat the envelope's `primary_artifact` as authoritative.
If `fallback_mode.active=true`, use fallback artifacts specified in the envelope.

WORKDIR=$CLOSEDLOOP_WORKDIR. Read $CLOSEDLOOP_WORKDIR/judge-input.json first.
Evaluate according to `task` and `source_of_truth` ordering.
Treat the envelope's `primary_artifact` as authoritative.
Apply your {judge_name} criteria to assess code quality.

WORKDIR=$CLOSEDLOOP_WORKDIR. Read $CLOSEDLOOP_WORKDIR/judge-input.json first.
Evaluate according to `task` and `source_of_truth` ordering.
Treat the envelope's `primary_artifact` as the authoritative PRD document and load supporting descriptors as source-of-truth evidence.
Apply your {judge_name} criteria to assess PRD quality.

WORKDIR=$CLOSEDLOOP_WORKDIR. Read $CLOSEDLOOP_WORKDIR/judge-input.json first.
Evaluate according to `task` and `source_of_truth` ordering.
Treat the envelope's `primary_artifact` as the authoritative Feature document and load supporting descriptors as source-of-truth evidence.
Apply your {judge_name} criteria to assess Feature quality.

Expected Output Format

{
  "type": "case_score",
  "case_id": "dry-judge",
  "final_status": 1,
  "metrics": [
    {
      "metric_name": "dry_score",
      "threshold": 0.8,
      "score": 0.85,
      "justification": "Plan follows DRY principles..."
    }
  ]
}

Error Handling Protocol

{
  "type": "case_score",
  "case_id": "{judge-name}",
  "final_status": 3,
  "error_reason": "Brief human-readable description of what failed",
  "metrics": [
    {
      "metric_name": "{metric}_score",
      "threshold": 0.8,
      "score": 0.0,
      "justification": "Judge execution failed: {error message}"
    }
  ]
}

Summary Table Formatting

When displaying the evaluation results summary (e.g., in the final output or any human-readable report), follow these conventions for errored scores:

Errored score display:

• Use the ERR marker in place of a numeric score for any judge whose CaseScore has final_status=3. error_reason, when present, can be displayed in a hover/tooltip or separate column but does not control whether ERR is shown.

Example summary table:

Average annotation:

• When some judges are excluded due to errors, annotate the aggregate average as "avg of N/M judges", where N is the number of non-errored judges and M is the total number of judges.
• Example: avg of 14/16 judges

Footer line:

• When one or more judges are excluded, add a footer line to the summary:

  X of Y judges excluded due to errors
  

  where X is the count of errored judges and Y is the total expected judge count.

• Example: 2 of 16 judges excluded due to errors

When ALL judges errored:

• Display ERR for every judge row
• Display N/A (not a number) for the aggregate average — do not attempt to compute or display an average
• Footer: Y of Y judges excluded due to errors

Step 2: Aggregate Results into EvaluationReport

Performance: Run "start of phase" with sub_step 5 (plan), 4 (code), 3 (prd), or 2 (feature), sub_step_name=aggregate. Emit 'end of phase' after the aggregation step regardless of file write outcome.

Task: Collect all CaseScore outputs and structure them into an EvaluationReport.

if artifact_type == 'code':
    report_filename = 'code-judges.json'
    report_id = f'{RUN_ID}-code-judges'
elif artifact_type == 'prd':
    report_filename = 'prd-judges.json'
    report_id = f'{RUN_ID}-prd-judges'
elif artifact_type == 'feature':
    report_filename = 'feature-judges.json'
    report_id = f'{RUN_ID}-feature-judges'
else:
    report_filename = 'plan-judges.json'
    report_id = f'{RUN_ID}-plan-judges'
output_path = $CLOSEDLOOP_WORKDIR / report_filename

{
  "report_id": "{RUN_ID}-plan-judges",
  "timestamp": "2024-02-03T15:45:30Z",
  "stats": [
    { /* CaseScore from dry-judge */ },
    { /* CaseScore from ssot-judge */ },
    { /* CaseScore from kiss-judge */ },
    { /* CaseScore from code-organization-judge */ },
    { /* CaseScore from custom-best-practices-judge */ },
    { /* CaseScore from goal-alignment-judge */ },
    { /* CaseScore from readability-judge */ },
    { /* CaseScore from verbosity-judge */ },
    { /* CaseScore from solid-isp-dip-judge */ },
    { /* CaseScore from solid-liskov-substitution-judge */ },
    { /* CaseScore from solid-open-closed-judge */ },
    { /* CaseScore from technical-accuracy-judge */ },
    { /* CaseScore from test-judge */ },
    { /* CaseScore from brownfield-accuracy-judge */ },
    { /* CaseScore from codebase-grounding-judge */ },
    { /* CaseScore from convention-adherence-judge */ }
  ]
}

{
  "report_id": "{RUN_ID}-code-judges",
  "timestamp": "2024-02-03T15:45:30Z",
  "stats": [
    { /* CaseScore from dry-judge */ },
    { /* CaseScore from ssot-judge */ },
    { /* CaseScore from kiss-judge */ },
    { /* CaseScore from code-organization-judge */ },
    { /* CaseScore from custom-best-practices-judge */ },
    { /* CaseScore from readability-judge */ },
    { /* CaseScore from solid-isp-dip-judge */ },
    { /* CaseScore from solid-liskov-substitution-judge */ },
    { /* CaseScore from solid-open-closed-judge */ },
    { /* CaseScore from technical-accuracy-judge */ },
    { /* CaseScore from test-judge */ }
  ]
}

{
  "report_id": "{RUN_ID}-prd-judges",
  "timestamp": "2024-02-03T15:45:30Z",
  "stats": [
    { /* CaseScore from feature-completeness-judge */ },
    { /* CaseScore from prd-auditor */ },
    { /* CaseScore from prd-dependency-judge */ },
    { /* CaseScore from prd-testability-judge */ },
    { /* CaseScore from prd-scope-judge */ }
  ]
}

{
  "report_id": "{RUN_ID}-feature-judges",
  "timestamp": "2024-02-03T15:45:30Z",
  "stats": [
    { /* CaseScore from feature-completeness-judge */ },
    { /* CaseScore from prd-testability-judge */ },
    { /* CaseScore from prd-dependency-judge */ }
  ]
}

Step 3: Validate Output (MANDATORY)

Performance: Run "start of phase" with sub_step 6 (plan), 5 (code), 4 (prd), or 3 (feature), sub_step_name=validate. Emit 'end of phase' after each validation attempt regardless of exit code, then apply failure recovery logic.

CRITICAL: You MUST run the validation script after writing the judge report. Do not consider the task complete until validation passes.

SCRIPT_PATH="scripts/validate_judge_report.py"

if ! command -v uv &> /dev/null; then
  # Install uv — alternatives: brew install uv, pip install uv
  curl -LsSf https://astral.sh/uv/install.sh | sh
fi

# CRITICAL: Run from script's directory so uv can find inline dependencies
cd "$(dirname "$SCRIPT_PATH")"

# Determine category based on artifact type
CATEGORY="plan"  # default
if [ "$ARTIFACT_TYPE" = "code" ]; then
  CATEGORY="code"
elif [ "$ARTIFACT_TYPE" = "prd" ]; then
  CATEGORY="prd"
elif [ "$ARTIFACT_TYPE" = "feature" ]; then
  CATEGORY="feature"
fi

# Run validation with appropriate category
uv run "$SCRIPT_PATH" --workdir "$CLOSEDLOOP_WORKDIR" --category "$CATEGORY"

Validation Checks

brownfield-accuracy-judge
code-organization-judge
codebase-grounding-judge
convention-adherence-judge
custom-best-practices-judge
dry-judge
goal-alignment-judge
kiss-judge
readability-judge
solid-isp-dip-judge
solid-liskov-substitution-judge
solid-open-closed-judge
ssot-judge
technical-accuracy-judge
test-judge
verbosity-judge

code-organization-judge
custom-best-practices-judge
dry-judge
kiss-judge
readability-judge
solid-isp-dip-judge
solid-liskov-substitution-judge
solid-open-closed-judge
ssot-judge
technical-accuracy-judge
test-judge

feature-completeness-judge
prd-auditor
prd-dependency-judge
prd-testability-judge
prd-scope-judge

feature-completeness-judge
prd-dependency-judge
prd-testability-judge

Validation Exit Codes

If Validation Fails

Reference: Pydantic Models

class MetricStatistics(BaseModel):
    """A single metric evaluation result."""
    metric_name: str
    threshold: Optional[float] = None
    score: float
    justification: str

class CaseScore(BaseModel):
    """Score for a single judge evaluation."""
    type: Optional[str] = "case_score"
    case_id: str
    final_status: int  # 1=pass, 2=fail, 3=error
    metrics: List[MetricStatistics]
    error_reason: Optional[str] = None  # set when final_status=3; excluded from aggregation averages

class EvaluationReport(BaseModel):
    """Top-level report containing all judge evaluations."""
    report_id: str
    timestamp: str
    stats: List[CaseScore]

Success Checklist

Troubleshooting Guide

Error Handling Requirements

Invalid Artifact Type

If --artifact-type value is not 'plan', 'code', 'prd', or 'feature':

• Fail immediately with clear error message
• Do not attempt judge execution
• Exit with non-zero status

Context Manager Timeout (Code Mode)

If context-manager-for-judges agent exceeds 5 minutes:

• Abort judge execution
• Generate error CaseScores for all 11 judges
• Each error CaseScore: final_status=3, error_reason="Timeout: context preparation exceeded 5 minutes", justification="Context preparation timeout" (see error_reason guidance above)
• Write complete report with all error CaseScores

Context Manager Timeout (Plan Mode)

If context-manager-for-judges agent exceeds 5 minutes in plan mode:

• Attempt one emergency compatibility fallback to raw plan.json + prd.md
• If fallback files are unavailable, abort plan judge execution and emit clear error

Individual Judge Failures

If a single judge Task call fails during execution:

• Do not abort the entire workflow
• Generate error CaseScore for that judge only, with final_status=3 and a populated error_reason describing the specific failure (e.g. "Task tool error: agent not found", "Parse error: response was not valid JSON") per the error_reason guidance above
• Continue with remaining judges in batch and subsequent batches
• Include error CaseScore in final aggregated report

Plan Mode Execution Flow

When --artifact-type is not specified or equals 'plan':

• Execute standard 16-judge plan logic
• Launch 4 batches with existing judge assignments
• Write to plan-judges.json (not code-judges.json)
• Launch context-manager-for-judges for plan context preparation
• Use plan-context.json as primary input; use one-run compatibility fallback only if context preparation fails
• Build and pass judge-input.json envelope to judges
• Prepend preambles to judge prompts
• Use default validation with --category plan

This is the standard plan mode flow; orchestrators must support context-manager launch, judge-input.json construction, and preamble injection. The compatibility fallback (raw plan.json + prd.md) activates only when context preparation fails (e.g., context-manager timeout), not for orchestrators that have not been updated.

PRD Mode Execution Flow

When --artifact-type prd is specified:

• Check $CLOSEDLOOP_WORKDIR/prd.md exists; emit WARNING and exit gracefully (code 0) if missing
• Do NOT launch context-manager-for-judges
• Build and schema-validate judge-input.json with scripts/judge_input_mapping.py --artifact-type prd
• Launch the 5 PRD judges in 2 sequential batches (sub_step=1: feature-completeness-judge + prd-auditor + prd-scope-judge; sub_step=2: prd-dependency-judge + prd-testability-judge) to respect the 4-concurrent-agent Task limit
• Aggregate all 5 CaseScores (sub_step=3) and write to prd-judges.json
• Validate with --category prd (sub_step=4)

Feature Mode Execution Flow

When --artifact-type feature is specified:

• Check $CLOSEDLOOP_WORKDIR/feature.md exists, or legacy $CLOSEDLOOP_WORKDIR/prd.md exists; emit sub_step=0 (context_prep, skipped=true) perf event, emit WARNING, and exit gracefully (code 0) if both are missing
• Do NOT launch context-manager-for-judges
• Build and schema-validate judge-input.json with scripts/judge_input_mapping.py --artifact-type feature
• Use feature_preamble.md for all 3 feature judges (Feature-shaped contract; do NOT substitute prd_preamble.md)
• Launch the 3 feature judges in 1 batch (sub_step=1: feature-completeness-judge + prd-testability-judge + prd-dependency-judge) to respect the 4-concurrent-agent Task limit
• Aggregate all 3 CaseScores (sub_step=2) and write to feature-judges.json
• Validate with --category feature (sub_step=3)