Research Quality Audit

Audit the research corpus for shallow stubs, incomplete documentation, and missing source files. Detects analysis docs written from abstracts alone (the root cause of the 88-stub incident) and reports doc depth metrics across the corpus.

Triggers

"audit research quality"
"check for stubs"
"find shallow docs"
"research quality audit"
"how deep are the analysis docs?"
/research-quality-audit

Parameters

`--range REF-XXX:YYY` (optional)

Audit a specific range of REF identifiers. Default: entire corpus.

`--fix` (optional)

Auto-dispatch expansion agents to deepen stubs. Each stub gets a focused agent that reads the full PDF/source and rewrites the analysis doc.

`--threshold N` (optional)

Minimum line count for a doc to be considered non-stub. Default: 80.

`--format` (optional)

Output format: full (default), summary, or json.

`--pdf-check` (optional)

Also verify that each REF has an actual PDF or source file, not just metadata.

Execution Flow

Phase 1: Corpus Scan

Glob all finding docs: .aiwg/research/findings/REF-*.md (and/or documentation/references/REF-*.md depending on corpus layout)
For each doc, collect:
- Line count (total lines)
- Content lines (non-empty, non-frontmatter, non-heading lines)
- Section count (number of ## headings)
- Key quote count (blockquotes or inline quotes)
- Source availability — does the PDF exist at the referenced pdf_location?
- Full text available — does sources/text/REF-XXX.txt exist?
- Frontmatter completeness — required fields present?

Phase 2: Classification

Classify each doc into quality tiers:

Tier	Content Lines	Sections	Quotes	Verdict
Full	>= 150	>= 8	>= 3	Comprehensive analysis
Adequate	80-149	>= 5	>= 1	Meets minimum depth
Stub	40-79	>= 3	0	Written from abstract — needs expansion
Skeleton	< 40	any	0	Placeholder only — needs full rewrite

Additional flags:

No PDF: analysis exists but source PDF is missing
No full text: PDF exists but text extraction was not run
Abstract-only indicators: doc mentions "abstract" but no methodology/results sections

Phase 3: Report

Research Quality Audit
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Corpus: 372 documents
Threshold: 80 content lines

Quality Distribution:
  Full (150+):      124 (33%)  ████████████████░░░░░░░░░░
  Adequate (80-149): 89 (24%)  ████████████░░░░░░░░░░░░░░
  Stub (40-79):      98 (26%)  █████████████░░░░░░░░░░░░░
  Skeleton (<40):    61 (16%)  ████████░░░░░░░░░░░░░░░░░░

Statistics:
  Mean content lines:  112
  Median:              94
  Min:                 12 (REF-299)
  Max:                 591 (REF-018)

Source Availability:
  PDF present:         348 / 372 (94%)
  Full text extracted:  201 / 372 (54%)
  Missing PDF:          24 papers
  Missing text:        171 papers

Stubs Requiring Expansion (159):
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  REF-253  22 lines  skeleton  No PDF    "Agentic Design Patterns"
  REF-254  35 lines  skeleton  Has PDF   "Multi-Agent Debate"
  REF-255  45 lines  stub      Has PDF   "Language Agent Tree Search"
  REF-256  48 lines  stub      No text   "ReAct: Synergizing Reasoning"
  ...

Top 10 Shallowest (candidates for immediate expansion):
  1. REF-299  12 lines  skeleton  "Toolformer: Language Models Can..."
  2. REF-312  15 lines  skeleton  "WebArena: A Realistic Web..."
  3. REF-253  22 lines  skeleton  "Agentic Design Patterns..."
  ...

Phase 4: Auto-Fix (if --fix)

When --fix is specified:

Filter fixable stubs — only expand docs that have a PDF or full text available
Batch by priority — shallowest docs first, batch into groups of 10
Dispatch expansion agents — each agent:
- Reads the full PDF/extracted text for the source
- Rewrites the analysis doc with comprehensive content
- Target: 150+ content lines with methodology, findings, limitations, key quotes
Re-audit after expansion — run Phase 1-3 again to verify improvements
Report — docs expanded, mean line improvement, remaining stubs

Auto-Fix Results:
  Dispatched: 10 expansion agents (batch 1 of 16)
  Expanded: 10 / 10
  Mean improvement: 77 → 161 lines (+109%)
  Remaining stubs: 149

  Run again with --fix to process next batch.

Integration Points

Component	Relationship
`induct-research`	Quality audit should auto-run after batch induction
`corpus-snapshot`	Gates on stub rate > 10% (#814)
`research-lint`	`ref-frontmatter` rule catches incomplete metadata; quality-audit catches shallow content
`research-status`	Doc depth is a component of corpus health scoring
`research-acquire`	For stubs with missing PDFs, triggers acquisition before expansion

Distinction from Other Tools

Tool	What it checks
`research-lint`	Structural — frontmatter fields, naming, references resolve
`research-quality-audit`	Depth — is the content substantive? Was the source actually read?
`research-quality`	Evidence — GRADE assessment of the source's research quality
`corpus-health`	Aggregate — overall corpus metrics including depth, structure, coverage

Examples

# Full corpus audit
/research-quality-audit

# Audit specific range
/research-quality-audit --range REF-253:372

# Auto-expand stubs (batch of 10)
/research-quality-audit --fix

# Strict threshold (120 lines minimum)
/research-quality-audit --threshold 120

# Check source file availability
/research-quality-audit --pdf-check

# JSON for programmatic use
/research-quality-audit --format json

References

@$AIWG_ROOT/agentic/code/frameworks/research-complete/skills/induct-research/SKILL.md — Source of stubs when acquisition is skipped
@$AIWG_ROOT/agentic/code/frameworks/research-complete/skills/research-acquire/SKILL.md — Acquires PDFs for stub expansion
@$AIWG_ROOT/agentic/code/frameworks/research-complete/skills/research-lint/SKILL.md — Structural validation (complementary)
@$AIWG_ROOT/agentic/code/frameworks/research-complete/skills/research-quality/SKILL.md — GRADE evidence assessment (complementary)
@$AIWG_ROOT/agentic/code/frameworks/research-complete/skills/research-status/SKILL.md — Health scoring includes depth metrics