content-mine

Reads the raw substrate (transcripts in sources/transcripts/, commits via git log, inbox via sweep snapshot), extracts candidate atoms, and filters through Robert-signal (what Robert demos, brings up unprompted, or returns to) plus citation gate (requireCitations() from state/lib/voice.ts).

Why Robert-signal: (see feedback_mining_robert_signal.md) the mining should extract what Robert actually demos, not generic patterns. A generic "AI productivity" atom is useless; a "Robert showed Cal how the agent rewrote its own skill on Tuesday" atom is the gold.

Steps

0. Pre-flight: check substrate

Before extracting, verify sources/transcripts/ contains at least one non-.gitkeep file. If the directory is empty or has only .gitkeep:

const files = fs.readdirSync("sources/transcripts/").filter(f => f !== ".gitkeep");
if (files.length === 0) {
  score("content-mine", run_id, {
    score: 0.5,
    atom_count_raw: 0,
    atom_count_cited: 0,
    atom_count_kept: 0,
    primary_issue: "no-transcript-substrate",
  });
  console.warn("content-mine: sources/transcripts/ is empty — drop transcripts in before re-running");
  return; // skip gracefully, do not hard-fail
}

1. Extract candidates

const atoms = await extract(source, since); // raw [{text, source_ref, ts}]

2. Citation gate

const cited = atoms.filter(a => requireCitations(a.text, [a.source_ref]).pass);

Any atom without a verbatim source_ref is dropped. No paraphrase-only atoms.

3. Robert-signal filter

Dispatch gemini (via OpenRouter) with the prompt: "score each atom 0-1 by whether it represents something Robert specifically demoed, defended, or returned to. Generic industry observations score 0. Specific 'Robert did X' score 1." Keep atoms with score ≥ 0.6.

What qualifies as Robert-signal: the atom must name or clearly imply Robert as the actor - e.g. "Robert showed Cal how the agent rewrote its own skill on Tuesday" or "Robert demoed inbox-zero in 90 seconds." Generic observations like "AI agents can automate workflows" do not qualify even if they appear in a Robert transcript.

If OpenRouter returns HTTP 402 (credits exhausted), catch the error and score partial rather than hard-failing:

catch (err) {
  if (err.status === 402) {
    score("content-mine", run_id, {
      score: 0.5,
      atom_count_raw: atoms.length,
      atom_count_cited: cited.length,
      atom_count_kept: 0,
      primary_issue: "openrouter-credits-exhausted",
    });
    console.warn("content-mine: OpenRouter 402 — top up credits at https://openrouter.ai/settings/credits");
    return;
  }
  throw err; // re-throw unexpected errors
}

If no Robert-signal atoms survive the ≥ 0.6 filter, score partial (not 0):

if (kept.length === 0) {
  score("content-mine", run_id, {
    score: 0.5,
    atom_count_raw: atoms.length,
    atom_count_cited: cited.length,
    atom_count_kept: 0,
    primary_issue: "no-robert-signal",
  });
  return;
}

4. Log + eval

score("content-mine", run_id, {
  score: kept.length / Math.max(atoms.length, 1),
  atom_count_raw: atoms.length,
  atom_count_cited: cited.length,
  atom_count_kept: kept.length,
  primary_issue: null,
});

Eval

Actor: extractor + gemini Robert-signal scorer. Auditor: the deterministic requireCitations() gate runs BEFORE the signal filter, so paraphrase drift can't leak through even if the model hallucinates.

Gotchas

• Never paraphrase. (feedback_mining_pods_must_cite.md) Atoms are

verbatim stitches from source material. If the extractor summarizes, the citation gate catches it.

• Raw signal ≠ action plan. (feedback_mining_signals_are_raw.md) The

kept atoms go into content_atoms - any downstream "let's draft a client email from this" must re-filter through positioning.

Graduation

Stays prose through Wave 2. Sidecar lands when the signal scorer prompt is stable across two port runs.

Rubric

criteria:
  - name: transcript_substrate_exists
    kind: deterministic
    check: "The 'sources/transcripts/' directory contains at least one non-'.gitkeep' file."
  - name: citation_gate_applied
    kind: deterministic
    check: "The 'requireCitations()' function from 'state/lib/voice.ts' successfully filtered atoms, dropping any without verbatim 'source_ref'."
  - name: robert_signal_filtering
    kind: judge
    check: "The extracted 'kept' atoms demonstrably represent specific actions or observations by Robert, not generic industry commentary, as intended by the Robert-signal filter."
  - name: no_paraphrased_atoms
    kind: judge
    check: "All 'kept' atoms are verbatim extractions from the source material and do not contain paraphrased content."
  - name: script_graceful_partial_fail
    kind: deterministic
    check: "The script handles OpenRouter 402 errors or zero Robert-signal atoms by logging a partial score and warning, rather than hard-failing."

content-mine - loader

Per-turn rules for the content-mine skill. Full reference: state/skills/content-mine/SKILL.md. Do not skip these.

Critical Rules

• NEVER paraphrase. Atoms must be verbatim stitches from source material with source_ref. requireCitations() runs BEFORE the signal filter - paraphrase drift can't leak even if the model hallucinates. (feedback_mining_pods_must_cite.md)
• ALWAYS filter for Robert-signal (≥ 0.6 score). Generic "AI productivity" atoms are noise; "Robert showed Cal how the agent rewrote its own skill on Tuesday" is the gold. (feedback_mining_robert_signal.md)
• NEVER treat mining "next moves" as an action plan. Raw atoms must re-filter through positioning before any client-facing draft. (feedback_mining_signals_are_raw.md)

Commands

| ui dashboard | state/skills/content-mine/resources/ui.openui | |invoke: prose skill - call extract(source, since) then chain requireCitations() then dispatch gemini for Robert-signal score |invoke (graduated sidecar): run npx tsx state/bin/content-mine/tick.ts from repo root |source: current sidecar mines git commits since the last content-mine eval row; it does not read sources/transcripts/ yet |artifacts: writes an eval row on every run, appends a content-mine row to state/log/chain.ndjson, and writes state/log/content_atoms-<run_id>.ndjson only when kept > 0 |stdout: the sidecar prints JSON {run_id,since,raw,cited,kept,signal_mode}; use run_id to verify the exact eval row and atoms artifact you just produced |verify: if raw > 0, confirm every kept atom has a source_ref; if raw === 0, confirm the eval note records "no new commits since last run" as an empty-with-reason pass |eval log: state/log/evals.ndjson (auto eval - skill: "content-mine")

OpenUI Resource

• Skill-owned OpenUI Lang resource: state/skills/content-mine/resources/ui.openui. Read it before rendering or editing this skill's generated component surface.
• Treat this resource as a first-class artifact of the skill, not a generic chat response. Improve it when the skill's user-facing output needs to become richer.
• System resources compose OpenUI primitives and inherit SnappyChat tokens. Use ui_contract: branded in SKILL.md only for deliberate platform or client visuals.

Known Pitfalls

• Citation gate (state/lib/voice.ts requireCitations()) is BEFORE the signal scorer - order matters; reversing it lets paraphrase through
• If raw > 0 and kept.length === 0, score partial with primary_issue: "no-robert-signal"; do not confuse that with the current sidecar's valid raw === 0 empty-with-reason pass when no new commits landed since the last eval

Self-Test

An agent reading this should correctly:

1. [ ] Refuse to ship a paraphrased atom even if it scores high on Robert-signal
2. [ ] Apply the citation gate before the signal filter
3. [ ] Re-filter through positioning before drafting a client email from kept atoms

Self-report

If this loader fell short, append a line:

echo "[$(date -u +%FT%TZ)] content-mine: <what was missing>" >> state/log/loader-feedback.log

<!-- kernel-ok: historical attribution for the PID-loader pattern. -->

Self-correcting loader (PID feedback)

This is the rule that makes loaders learn. Ported from snappy-kernel, where it demonstrably worked: the snappy-dom-cartographer skill mapped Skool admin UI once, every later ship-agent inherited the full map for free.

Before you finish, do two things:

1. Fix gaps (P - proportional). MANDATORY. If this loader didn't cover your case - if you had to read another file, run an undocumented command, work around a wrong selector, or discover a quirk - you MUST attempt an Edit to this AGENTS.md before you log.

• Default action: edit this file. Surgical. Add the missing line, table

row, or code example where it belongs. No restructuring.

• Only LOGGED is allowed when: the fix needs >10 lines, spans multiple

files, or requires a structural rewrite. In that case the state/regen/drain.sh queue picks it up asynchronously.

• "I didn't have time" / "it's minor" / "the next agent will figure it out"

are NOT valid reasons. Minor gaps compound - that's the whole point of this loop.

• Concrete example of a valid P-fix: discover agent-browser screenshot

takes path positionally not as --output <path>. One line in a table. Edit it in. Do not just log it.

• The goal: the next agent never has to leave the loader.

2. Log the result.

echo "[$(date -u +%FT%TZ)] content-mine: <what was missing or fixed> [FIXED|LOGGED]" >> state/log/loader-feedback.log

• FIXED = you patched this loader inline (P-fix).
• LOGGED = too large for inline; the PostToolUse enqueue + Stop-hook drain

will rewrite the loader from scratch on next session-end.

Do not skip this. Every agent run must leave the system better than it found it. The loader is the setpoint; you are the sensor; the gap is the error signal; closing the gap is the correction.