audit14-eval-snapshotter

Every 30 minutes, count the distinct skills in evals from the last hour and log the metric.

Steps

1. Read the last 1000 eval rows from state/log/evals.ndjson (most recent first)
2. Filter to rows timestamped within the last 60 minutes (check the ts field)
3. Extract the skill field from each matching row
4. Count distinct skill values
5. Append an eval row to state/log/evals.ndjson with:

• ts: current ISO timestamp
• skill: "audit14-eval-snapshotter"
• verb: "snapshot"
• score: the count (as a number between 0 and 1, normalized: count / 180)
• run_id: a fresh UUID
• primary_issue: null

Example eval row to append:

{"ts":"2026-05-02T11:30:00.000Z","skill":"audit14-eval-snapshotter","verb":"snapshot","score":0.45,"run_id":"uuid-here","primary_issue":null}

audit14-eval-snapshotter - loader

Per-turn rules for audit14-eval-snapshotter. Full reference: state/skills/audit14-eval-snapshotter/SKILL.md. Runs every 30 min (scheduled dispatch).

Critical Rules

1. One eval row per run. Append to state/log/evals.ndjson. Score = distinct_skill_count / 180 (normalized 0-1). Never two rows for one invocation.
2. Safe NDJSON tail. Use tail -1000 on the evals file then awk-filter by timestamp; do not use jq unless installed (not guaranteed).
3. macOS date flag. date -u -v-1H (macOS) vs date -u --date="-1 hour" (Linux). Use the 2>/dev/null || ... fallback pattern to handle both.
4. Score can exceed 1.0 if >180 skills ran. Clamp: SCORE=$(echo "scale=4; if ($DISTINCT_COUNT > 180) 1.0 else $DISTINCT_COUNT / 180" | bc).
5. Concurrent-safe. Lock via /tmp/claude-cron-audit14-eval-snapshotter.lock if running from cron; fine to skip when invoked manually.

Commands

Implementation

#!/bin/bash
EVALS_FILE="$HOME/projects/snappy-os/state/log/evals.ndjson"
HOUR_AGO=$(date -u -v-1H +%Y-%m-%dT%H:%M:%S 2>/dev/null || date -u --date="-1 hour" +%Y-%m-%dT%H:%M:%S)
NOW=$(date -u +%Y-%m-%dT%H:%M:%S.000Z)
RUN_ID=$(uuidgen | tr '[:upper:]' '[:lower:]')

DISTINCT_COUNT=$(tail -1000 "$EVALS_FILE" | \
  awk -v hour_ago="$HOUR_AGO" -F'"' '
    BEGIN { count = 0 }
    {
      for (i = 1; i <= NF; i++) {
        if ($i ~ /^ts$/) { ts_val=$(i+2); in_window=(ts_val >= hour_ago) }
        if ($i ~ /^skill$/ && in_window && !($(i+2) in seen)) { seen[$(i+2)]=1; count++ }
      }
    }
    END { print count }
  ')

SCORE=$(echo "scale=4; if ($DISTINCT_COUNT > 180) 1.0000 else $DISTINCT_COUNT / 180" | bc)
printf '{"ts":"%s","skill":"audit14-eval-snapshotter","verb":"snapshot","score":%s,"run_id":"%s","primary_issue":null}\n' \
  "$NOW" "$SCORE" "$RUN_ID" >> "$EVALS_FILE"
echo "audit14-eval-snapshotter: $DISTINCT_COUNT distinct skills in last hour (score=$SCORE)"

Self-Test

• [ ] Normalize score as distinct_count / 180 (clamped at 1.0)?
• [ ] Use macOS/Linux portable date fallback?
• [ ] Append exactly one eval row per invocation?
• [ ] Verify with tail -1 evals.ndjson | grep audit14-eval-snapshotter after run?

<!-- kernel-ok: historical attribution for the PID-loader pattern. -->

Self-correcting loader (PID feedback)

This is the rule that makes loaders learn. Ported from snappy-kernel, where it demonstrably worked: the snappy-dom-cartographer skill mapped Skool admin UI once, every later ship-agent inherited the full map for free.

Before you finish, do two things:

1. Fix gaps (P - proportional). MANDATORY. If this loader didn't cover your case - if you had to read another file, run an undocumented command, work around a wrong selector, or discover a quirk - you MUST attempt an Edit to this AGENTS.md before you log.

• Default action: edit this file. Surgical. Add the missing line, table

row, or code example where it belongs. No restructuring.

• Only LOGGED is allowed when: the fix needs >10 lines, spans multiple

files, or requires a structural rewrite. In that case the state/regen/drain.sh queue picks it up asynchronously.

• "I didn't have time" / "it's minor" / "the next agent will figure it out"

are NOT valid reasons. Minor gaps compound - that's the whole point of this loop.

• Concrete example of a valid P-fix: discover agent-browser screenshot

takes path positionally not as --output <path>. One line in a table. Edit it in. Do not just log it.

• The goal: the next agent never has to leave the loader.

2. Log the result.

echo "[$(date -u +%FT%TZ)] audit14-eval-snapshotter: <what was missing or fixed> [FIXED|LOGGED] action_kind=<kind>" >> state/log/loader-feedback.log

• FIXED = you patched this loader inline (P-fix).
• LOGGED = too large for inline; the PostToolUse enqueue + Stop-hook drain

will rewrite the loader from scratch on next session-end.

• action_kind: shape-ok | skill-ran | loader-rewritten | pattern-elevated

Do not skip this. Every agent run must leave the system better than it found it. The loader is the setpoint; you are the sensor; the gap is the error signal; closing the gap is the correction.