prompt-tuner

Feeds evaluation signals back into the system for continuous reinforcement. Reads state/log/evals.ndjson (the result of every skill run), buckets by skill/shape using recent 200 rows, identifies struggling and winning performers, and outputs a markdown digest with concrete suggestions for prompt updates.

The digest is consumed by follow-up agents or Robert who decide which prompt fragments in state/bin/head-screen/server.ts need iteration.

Steps

1. Read state/log/evals.ndjson, parse NDJSON tail (most recent 200 rows).
2. Extract skill (or verb) and score from each row.
3. Bucket by skill, compute mean score per bucket.
4. Flag struggling buckets (mean < 0.5, count >= min_bucket).
5. Flag winning buckets (mean > 0.85, count >= min_bucket).
6. For each bucket, count rows and extract recent primary_issue patterns.
7. Write to state/log/prompt-tuner-digest.md with:

• Summary stats (total rows scanned, buckets)
• Struggling section with mean score, row count, and top issues
• Winning section with mean score and row count
• Suggestions for each struggling skill (add examples, tighten triggers, remove distractors)
• Patterns observed in top issue keywords

Shape

Output markdown at state/log/prompt-tuner-digest.md:

# prompt-tuner digest

Generated: <ISO timestamp>

## Overview
- Total rows scanned: 200
- Unique skills: 43
- Struggling buckets (mean < 0.5, n >= 5): 7
- Winning buckets (mean > 0.85, n >= 5): 12

## Struggling (mean < 0.5, n >= 5)

**skill-name** mean=0.45 n=12
- Top issues: stale-snapshot (3), missing-data (2), format-error (1)
- Suggestion: Add a concrete GOOD/BAD example to the system prompt. The model may be missing context on when to trigger this skill.

**another-skill** mean=0.38 n=8
- Top issues: hallucination (4), out-of-scope (2)
- Suggestion: Tighten the trigger keywords in the system prompt. Remove alternative paths that are confusing the model.

## Winning (mean > 0.85, n >= 5)

**best-skill** mean=0.94 n=15
- No issues observed. Keep the current prompt and examples.

**another-winner** mean=0.88 n=10
- No issues observed.

## Patterns

Most common failure modes across all buckets:
1. `stale-snapshot` (7 occurrences) — suggests caching issues or refresh-window miscalculations
2. `hallucination` (5 occurrences) — suggests vague triggers or missing guardrails
3. `format-error` (3 occurrences) — suggests output schema needs clarification

## Next Steps

For each struggling skill:
1. Read the SKILL.md and AGENTS.md to understand the intent
2. Locate the system prompt fragment in server.ts or state/lib/<skill>.ts
3. Add a concrete GOOD example of when to emit, and a BAD example of when NOT to
4. Re-run the skill and observe if mean score climbs above 0.7
5. Archive this digest as prompt-tuner-digest-YYYY-MM-DD.md and generate a fresh one

Eval

Actor: the bucketing + statistics logic. Auditor: verify output file exists, contains expected sections (Overview, Struggling, Winning, Patterns).

const source_rows = parsed_evals.length;
const has_overview = output.includes("## Overview");
const has_struggling = output.includes("## Struggling");
const has_winning = output.includes("## Winning");
const has_patterns = output.includes("## Patterns");
const bucket_count = (output.match(/^## [A-Z]/gm) || []).length;

score("prompt-tuner", run_id, {
  score:
    source_rows === 0 ? 0.0 :
    has_overview && has_struggling && has_winning && has_patterns ? 1.0 :
    has_overview && has_struggling && has_winning ? 0.75 :
    source_rows > 0 && bucket_count >= 3 ? 0.5 :
    0.0,
  rows_scanned: source_rows,
  sections_found: [has_overview, has_struggling, has_winning, has_patterns].filter(Boolean).length,
  bucket_count,
  primary_issue:
    source_rows === 0 ? "no-eval-rows" :
    !has_overview ? "missing-overview" :
    !has_struggling ? "missing-struggling" :
    !has_winning ? "missing-winning" :
    !has_patterns ? "missing-patterns" :
    null,
});

Digest must include Overview, Struggling, Winning, and Patterns sections. Empty eval logs score 0.0.

Known Pitfalls

• Eval rows use skill (preferred) or verb (legacy). Normalize both to the same bucket key.
• primary_issue is nullable; skip rows where it's null when counting top issues.
• Mean score edge case: a bucket with n=5 all at 0.49 is still flagged as struggling (< 0.5). That's correct - the threshold is exact.
• Archive old digests before re-running. The script will overwrite state/log/prompt-tuner-digest.md on each run.

Rubric

criteria:
  - name: output_file_generated
    kind: deterministic
    check: "The skill must generate a markdown file at state/log/prompt-tuner-digest.md."
  - name: eval_log_parsing
    kind: deterministic
    check: "The script must parse state/log/evals.ndjson correctly, extracting skill/verb and score from each row. No silent drops."
  - name: bucketing_accuracy
    kind: deterministic
    check: "Buckets must group rows by skill/verb. Mean score for each bucket must be correct (sum of scores / count)."
  - name: struggling_flagging
    kind: deterministic
    check: "A bucket is flagged as struggling if and only if mean < 0.5 and count >= min_bucket. Output must list struggling buckets with their means and row counts."
  - name: winning_flagging
    kind: deterministic
    check: "A bucket is flagged as winning if and only if mean > 0.85 and count >= min_bucket. Output must list winning buckets."
  - name: issues_extraction
    kind: deterministic
    check: "For each struggling bucket, extract and count primary_issue values. Top issues must be listed with occurrence counts."
  - name: sections_present
    kind: deterministic
    check: "Output must include all four sections: Overview, Struggling, Winning, Patterns. No sections may be empty when data exists."

prompt-tuner - loader

Per-turn rules for the prompt-tuner skill. Full reference: state/skills/prompt-tuner/SKILL.md. Do not skip these.

Critical Rules

1. Input is append-only. Read state/log/evals.ndjson with tail semantics - take the most recent 200 rows (or window size requested). Do NOT rescan the entire file on every run; the tail prevents double-counting across weekly invocations.

1. Buckets are skill/verb, not individual runs. Each row has either skill (preferred) or legacy verb field. Normalize both to the same bucket key. A single skill may have 5+ runs in the tail window; all rows for that skill go into one bucket.

1. Mean score calculation must be accurate. For each bucket: mean = sum(scores) / count. Round to 2 decimals for display. Do NOT use median or mode; mean is the signal.

1. Struggling threshold is exact: mean < 0.5 AND count >= min_bucket. A bucket with mean=0.49 is struggling; mean=0.50 is NOT. The min_bucket guard (default 5) prevents single-outlier buckets from dominating the report.

1. Winning threshold is exact: mean > 0.85 AND count >= min_bucket. Same discipline. mean=0.85 is NOT winning; mean=0.86 is.

1. primary_issue extraction is case-sensitive. Some rows have null, some have strings. Skip nulls. Count occurrences of each non-null issue string, report top 3 per bucket. Do NOT fabricate issues.

1. Output file is always at state/log/prompt-tuner-digest.md. Overwrite on each run - do NOT append. The digest is meant to be fresh every week. Archive old digests manually if needed (e.g., cp prompt-tuner-digest.md prompt-tuner-digest-2026-04-30.md).

1. FORBIDDEN: do not edit server.ts or state/lib files directly. The digest is the artifact; humans or follow-up agents act on the suggestions. Your job is to feed data back, not to prescribe fixes into the runtime.

1. Overview section must show total rows, unique skills, bucket counts (struggling/winning). This is the first thing Robert reads to understand the scale of the analysis.

1. Patterns section must synthesize failure modes across all struggling buckets. If 7 buckets mention "stale-snapshot", that's a signal worth elevating - the prompt-tuner detected a systemic failure mode that deserves prompt-level fixes.

Commands

| ui dashboard | state/skills/prompt-tuner/resources/ui.openui |

Input Format

state/log/evals.ndjson - one JSON object per line:

{"ts":"2026-04-30T00:04:05.252Z","skill":"my-skill","score":0.8,"primary_issue":null}
{"ts":"2026-04-30T00:05:35.188Z","verb":"my-verb","score":0.5,"primary_issue":"stale-snapshot"}
{"ts":"2026-04-30T00:09:09.131Z","skill":"another-skill","score":1.0,"primary_issue":null}

Fields that matter:

• ts - timestamp (for sorting, take newest 200)
• skill OR verb - bucket key (normalize both to same field)
• score - numeric, 0.0 to 1.0 (sum and divide for mean)
• primary_issue - nullable string (extract for top-3 per bucket)

Output Format

Markdown at state/log/prompt-tuner-digest.md:

# prompt-tuner digest

Generated: 2026-04-30T15:22:00Z

## Overview
- Total rows scanned: 200
- Unique skills: 43
- Struggling buckets (mean < 0.5, n >= 5): 7
- Winning buckets (mean > 0.85, n >= 5): 12

## Struggling (mean < 0.5, n >= 5)

**skill-name** mean=0.45 n=12
- Top issues: stale-snapshot (3), missing-data (2), format-error (1)
- Suggestion: ...

## Winning (mean > 0.85, n >= 5)

**best-skill** mean=0.94 n=15
- No issues observed.

## Patterns

Most common failure modes:
1. `stale-snapshot` (7 occurrences)
2. `hallucination` (5 occurrences)
3. ...

## Next Steps

For each struggling skill:
1. Read the SKILL.md and AGENTS.md
2. Locate the system prompt fragment
3. Add concrete GOOD/BAD examples
4. Re-run the skill and check if score improves

OpenUI Resource

• Skill-owned OpenUI Lang resource: state/skills/prompt-tuner/resources/ui.openui. Read it before rendering or editing this skill's generated component surface.
• Treat this resource as a first-class artifact of the skill, not a generic chat response. Improve it when the skill's user-facing output needs to become richer.
• System resources compose OpenUI primitives and inherit SnappyChat tokens. Use ui_contract: branded in SKILL.md only for deliberate platform or client visuals.

Known Pitfalls

• Eval log may be very large (>10MB). Use tail semantics to read only the last N lines, not the entire file. The script should stream-read or use tail -<window_size> piped to jq.
• Bucket with only nulls in primary_issue. That's valid (no issues for those runs). The bucket still counts as struggling or winning if the mean satisfies the threshold; it just has an empty "Top issues" line.
• Score type inconsistency. Some rows may have score: null or missing score field. Treat missing/null score as 0.0 for mean calculation, not as "skip this row" - it still counts toward the bucket's sample size.
• Archive old digests before re-running. The script unconditionally overwrites the output file. If Robert wants to compare week-over-week, he must cp the old file first.
• Normalize skill vs verb. Use whichever is present; if both exist, prefer skill. Create a single merged bucket, not two separate ones.

Self-Test

An agent reading this should correctly:

1. [ ] Understand tail semantics: last 200 rows (or window size), not entire file?
2. [ ] Know bucket key normalization: skill OR verb, merged into one key per skill?
3. [ ] Calculate mean accurately: sum(scores) / count, not median or mode?
4. [ ] Apply thresholds exactly: < 0.5 for struggling, > 0.85 for winning, AND count >= min_bucket?
5. [ ] Extract primary_issue correctly: skip nulls, count top 3 per bucket?
6. [ ] Write only to output file, never modify server.ts or lib files?
7. [ ] Include all four sections: Overview, Struggling, Winning, Patterns?
8. [ ] Use tail-read or streaming to handle large eval logs?

Self-report

If this loader fell short, append a line:

echo "[$(date -u +%FT%TZ)] prompt-tuner: <what was missing or fixed> [FIXED|LOGGED] action_kind=<kind>" >> state/log/loader-feedback.log

• FIXED = you patched this loader inline (P-fix).
• LOGGED = too large for inline; the PostToolUse enqueue + Stop-hook drain will rewrite the loader from scratch on next session-end.
• action_kind: shape-ok | skill-ran | loader-rewritten | pattern-elevated

Do not skip this. Every agent run must leave the system better than it found it.