.md file to compare - side-by-side diff against dogfood-loop
dogfood-loop
What it does for you
Has your assistant test your app like a real user and fix what falls short.
What it produces
A recent result, so you can see the kind of work it returns.
loading…
How to get it
These run inside the Snappy workspace. Want this working in your business? I set skills like this up with you, in one focused week.
For developers how this skill is built, graded, and how it runs
at a glance- the short version
what's inside - the parts that make up a skill 3/4 present
A skill is just a few plain-text files. Only the main one is required. The rest are optional, added as the work needs them. This is what the skill is made of; how it runs is just below.
state/skills/dogfood-loop/SKILL.md
present
state/lib/dogfood-loop.ts
present
state/bin/dogfood-loop/
not present
state/skills/dogfood-loop/AGENTS.md
present
how it's graded - what counts as a good run 4 criteria · 3 deterministic · 1 judge
Each row is one thing a good run has to get right. deterministic means a quick check decides, pass or fail. judge means the AI reads the result and rates it. Grading each piece on its own (instead of one overall score) shows exactly where a run fell short, so the fix is obvious.
how it runs - the shared frame every skill uses 4/5 present
Every skill runs the same way. One part does the work, a separate part checks it, and a short loader hands the AI exactly what it needs for the job. Anything this skill doesn't use shows a one-line note saying why, on purpose, not by accident.
This skill doesn't fix its own gaps yet.
state/log/evals.ndjson what it has learned - fixes written back in over time sample
When a run hits something this skill didn't handle, the fix gets written back into the skill so it doesn't happen again. FIXED means it was corrected on the spot. LOGGED means it's queued for a bigger rewrite. Either way, the skill gets a little better and never makes the same mistake twice.
- Loading feedback rows…
how the work flows- who makes it, who checks it
SKILL.md- the skill, written out in plain English
dogfood-loop
The dogfood loop is how Snappy OS gets sharper between releases. A subagent is BOTH a user (drives the app via the chat-drive queue) AND a developer (commits fixes to Snappy OS when a rendered card falls below world-class). Closing this loop is what unlocks Robert's generative-UI leverage.
The deliverable IS the playbook below. There is no backing lib - the prose is the executable. Producer skills (agent-recap, bundle-snapshot, chat-drive) supply the primitives; this skill stitches them into a cadence.
The primitives stack
[ subagent in any context ]
│
▼
state/lib/chat-drive.ts ─ dispatchInChatUI(text) ──┐
resetChatUI() │
▼
state/lib/desktop.ts ─ captureScreen(path)
captureWindow(app, path)
listWindows(app?)
Snappy OS polls 127.0.0.1:3147/chat-inject-pop every 500ms. When dispatchInChatUI pushes text to /chat-inject-push, the React polling effect injects via the React-controlled-component setter dance and fires submit. resetChatUI pushes a control message that flips setHasThread(false) + setChatKey(k+1), returning the welcome surface for the next scenario.
Steps
The 7-step cadence below is automated by the sidecar at state/bin/dogfood-loop/run.ts. Drive a full pass with:
npx tsx state/bin/dogfood-loop/run.ts \
--scenarios state/skills/dogfood-loop/scenarios/default.json \
--out /tmp/dogfood-loop-report.md
The sidecar handles steps 1-4 + 7 (reset, pre-shot, push, post-shot, append to report) for every scenario, then exits. Steps 5 and 6 (visual grading, fix-or-log) stay manual - actor (the sidecar) captures; auditor (the agent that runs it) reads each /tmp/loop-N-post.png against the eval checklist below and decides commit-vs-backlog.
For each scenario:
- Reset -
await resetChatUI({waitMs: 1500}). Welcome surface should
remount.
- Pre-shot -
await captureScreen('/tmp/loop-N-pre.png'). Confirms
reset succeeded.
- Push intent - `await dispatchInChatUI(intent, {waitForFirstFrame:
14000})`.
- Post-shot -
await captureScreen('/tmp/loop-N-post.png'). - Read both shots with the Read tool. Evaluate against the eval
checklist below.
- Fix or log - if anything is below world-class, edit
web/src/styles.css or the relevant component, redeploy via bash scripts/build-app.sh --install, re-run the scenario to verify. If the fix needs >25 LOC or spans 3+ files, stop and append to ~/projects/snappy-os/state/log/snappy-os-backlog.md instead.
- Append to report -
/tmp/dogfood-loop-report.mdper the format at
the bottom of this doc.
Eval checklist - is this card world-class?
For every rendered scenario, check all of these:
Render integrity
- [ ] Welcome surface unmounted, thread state visible
- [ ] No console errors (open DevTools via menubar, or check WKWebView log
if available)
- [ ] User pill right-anchored at the actual right edge of the message
column
- [ ] User pill bg =
--surface-elevated, border = subtle, radius ~12px,
max-width ≤540px, content fit
- [ ] Assistant content visible (text or generative-UI card)
Card rendering (when a generative-UI shape fires)
- [ ] Card chrome present: 1px border, soft shadow, 14px radius, no
side-stripe
- [ ] Channel hairline color present (3-4px gradient strip on top edge)
- [ ] Header row: verb chip (mono 11px) + channel chip (mono 10.5px) +
count if applicable
- [ ] Body text 14px sans, line-height 1.6, color =
--text(not
secondary)
- [ ] Action row only present if the shape uses one (Approve/Edit/Reject
for DispatchCard)
- [ ] No clipped content, no overflow
Layout quality
- [ ] Composer at bottom with Queue pill (copper-filled, "Queue" label)
- [ ] Mode chips inline on left of action-bar, model right-pushed
- [ ] Mic button is a quiet outline (not copper-filled) when idle
- [ ] Right panel headers in sentence-case sans, NOT uppercase mono
- [ ] Sidebar agent rows compact, sentence-case section labels
Behavior
- [ ] Dispatch completed (no "Unknown dispatch error" or stuck spinner)
- [ ] Streaming actually streamed (not all-at-once dump)
- [ ] Approve/Edit/Reject buttons on DispatchCard are clickable
If ALL pass → scenario is pass. If any fail → scenario is fail - <reason>. Don't half-pass.
Standard scenario set (ranked by render-richness)
Run as many as fit in budget. Each tests a different generative-UI shape; each is also a contract test for the producer skill that owns it:
compose a short project update about today's snappy-os build session→
DispatchCard with a message channel, counter fold, Approve/Edit/Reject. Producer: action-item-draft.
show me what the agents did today→ ProgressList. Producer:
what is the snappy-os bundle context→ ContextPanel with connectors.
Producer: bundle-snapshot.
open the file state/skills/desktop/SKILL.md→ WorkingFolder with file
pill. Producer: desktop.
give me feedback on the welcome screen design→ FeedbackForm if
plumbed; else text.
are you sure you want to fire snappy-verify?→ ConfirmDialog if plumbed;
else text.
DO NOT run anything that side-effects externally - no real LinkedIn posts, no emails, no Slack sends. The default backend is openrouter / gemini so most replies are text-only unless the prompt explicitly invokes a generative shape via the dispatcher.
Failure modes - what to do when
| Symptom | Cause | Action | ||
|---|---|---|---|---|
dispatchInChatUI succeeds but composer never receives text | Polling stopped or React un-mounted | Restart snappy-chat: `pgrep -af /Applications/SnappyChat.app/Contents/MacOS/SnappyChat \ | awk '{print $1}' \ | xargs kill; sleep 1; open /Applications/SnappyChat.app` |
| Welcome surface stays after dispatch | React controlled-component setter didn't fire | Confirm injectIntoComposer ran by reading App.tsx; check if textarea is display:none (offsetParent null on both variants) | ||
| Dispatch returns "Unknown dispatch error" | /dispatch/chat 5xx, often dispatch.json mis-routed | curl -XPOST 127.0.0.1:3147/dispatch-config -d '{"chat":{"backend":"openrouter","model":"gemini"}}' to reset axis | ||
Build fails on bash scripts/build-app.sh --install | TS error or stale Resources | Read the error; never bypass with --no-verify | ||
| Screenshot shows blank/white | peekaboo targeted the wrong window | Use captureScreen (full display) instead of captureWindow | ||
| Stuck process at /private/tmp/.../Snappy OS | Old build still running | pkill -f /private/tmp.*SnappyChat; sleep 0.5; open /Applications/SnappyChat.app | ||
Multiple npx tsx processes piling up | Subagent's previous calls leaked | pkill -f "npx tsx" is safe - the desktop helpers are stateless |
What goes where
- Quick fix (≤25 LOC, ≤2 files, in snappy-os) → commit immediately
with chat: prefix and explicit pathspec.
- Bigger fix (>25 LOC OR ≥3 files OR cross-repo) → append to
~/projects/snappy-os/state/log/snappy-chat-backlog.md as - [ ] <title> - <one-line scope>. Don't commit. Robert routes the larger work.
- Reference observation (e.g. "the welcome state's spark is rendering
blurry at 22px") → also goes to backlog. Cosmetic findings batch better.
- New skill needed (e.g. discovered we need a
notification-toast
generative-UI shape) → write a finding in the backlog AND mention it in the loop report's "next" section.
Report format
/tmp/dogfood-loop-report.md:
## Scenario N: <intent>
- Pre: /tmp/loop-N-pre.png
- Post: /tmp/loop-N-post.png
- Card rendered: <name | "text-only" | "none">
- Render quality: pass | fail - <reason>
- Action: <commit SHA> | "no fix needed" | "logged to backlog"
- Notes: <one or two sentences>
## Summary
- Scenarios run: N
- Clean passes: M
- Commits: <SHAs>
- Findings to backlog: <count>
- Next biggest gap: <one sentence>
Hard constraints
- PRODUCT.md tone is absolute. No emojis. No em dashes. No exclamation
points in user-visible copy.
- DESIGN.md tokens are the only color source. Every styles.css value
pulls from existing OKLCH tokens.
- Don't modify snappy-os skills/libs in a dogfood pass. That's a
separate axis.
- Don't side-effect externally. No real outbound messages.
- Don't run more than 6 minutes per scenario. If it eats more, log a
finding and move on.
Reference shots
Claude Desktop "Cowork" reference shots live in refs/cd-{01..23}.png - Robert's design target. See refs/README.md for the per-shot index. cd-01..06 = welcome state, cd-13/14/21 = thread state, cd-16..20 = customize / skills directory, cd-08..12 = projects + scheduled, cd-22..23 = artifacts.
Eval
Actor: the dogfood-running subagent. Captures screenshots, evaluates against the checklist, commits fixes, appends to report. Auditor: the next agent (or Robert) reads /tmp/dogfood-loop-report.md and verifies that scenarios actually passed by re-reading the post-shots. Distinct context = actor ≠ auditor.
| Outcome | Score |
|---|---|
| Report present, ≥3 scenarios, every fail has a fix-or-backlog action | 1.0 |
| Report present but a scenario flagged fail with no follow-through | 0.5 |
| No report or report missing scenario shots / actions | 0.0 |
Rubric
criteria:
- name: report_present
kind: deterministic
check: "/tmp/dogfood-loop-report.md exists and contains a Summary section."
- name: scenarios_min_three
kind: deterministic
check: "Report contains at least three Scenario blocks, each with Pre, Post, Card rendered, Render quality, Action, Notes lines."
- name: every_fail_has_action
kind: judge
check: "Every scenario marked 'fail' carries either a commit SHA in Action OR an entry appended to state/log/snappy-chat-backlog.md - no orphan failures."
- name: no_external_sideeffects
kind: deterministic
check: "No scenario triggered a real outbound send (no LinkedIn URN, no Gmail draft, no Slack post). The dispatch backend stays text-only or generative-shape only."AGENTS.md- what the AI loads when this skill comes up
dogfood-loop loader
Compact runtime loader for Snappy OS QA. The full contract lives in state/skills/dogfood-loop/SKILL.md; do not inject historical feedback logs or consolidated issue inventories into chat dispatch.
When invoked
Use the real /Applications/SnappyChat.app, not only /dispatch/chat. Drive it through POST 127.0.0.1:3147/chat-inject-push, reset with POST /chat-inject-control {"action":"reset"}, and capture screenshots from the running app. Screenshots are evidence; curl 200s are not enough.
Cadence
- Flush/reset the queue, then confirm the UI is ready.
- Push one concrete user intent.
- Wait for render completion or a visible failure.
- Capture the app screenshot.
- Grade: render integrity, card/canvas quality, layout quality, behavior.
- Fix narrow issues directly, redeploy, and re-run the same intent.
- If the fix is large or cross-repo, record a precise backlog item instead.
Five exchanges are required for a full dogfood pass when time allows. In an emergency, run the smallest loop that proves the reported failure is fixed.
Non-negotiables
- Actor is not auditor: do not treat the thing that generated output as the
final grade. Independently inspect the screenshot and logs.
- No blank right rail, empty canvas, placeholder, dead button, or silent
provider failure is acceptable demo behavior.
- Do not perform real outbound side effects such as posting, sending email, or
firing production agents unless the prompt explicitly asks and apply:true is present.
- Keep snappy-os fixes narrow. Build/install before claiming the UI is live:
bash ~/projects/snappy-os-app/apps/snappy-os/scripts/build-app.sh --install.
Useful commands
curl -sS -XPOST 127.0.0.1:3147/chat-inject-flush
curl -sS -XPOST 127.0.0.1:3147/chat-inject-control -H 'content-type: application/json' -d '{"action":"reset"}'
curl -sS -XPOST 127.0.0.1:3147/chat-inject-push -H 'content-type: application/json' -d '{"text":"<intent>","newThread":true}'
pgrep -af "/Applications/SnappyChat.app/Contents/MacOS/SnappyChat"
bash ~/projects/snappy-os-app/apps/snappy-os/scripts/build-app.sh --install
Report shape
Write the result plainly: intent, screenshot path, card/canvas rendered, pass or fail with reason, changed files or backlog item, and the next highest-risk gap.
api.ts- the code it can call
/**
* snappy-dogfood-loop/api.ts -- typed scenario API for the dogfood loop.
*
* Today the dogfood-loop sidecar lives at state/bin/dogfood-loop/run.ts
* and reads scenarios from a JSON file. This lib is the typed surface a
* subagent (or any caller) should adopt instead of shell-driving scenarios.
*
* The loop runs autonomously — the curriculum from
* feedback_loop_graduation is about NOT auto-promoting to cron until
* trust is established, not about "the operator runs each tick by hand."
* This lib provides the TYPED contract for any caller (subagent, cron,
* test, ad-hoc).
*
* import { runScenario, type Scenario } from "../lib/dogfood-loop.ts";
* const result = await runScenario({
* name: "linkedin-schedule",
* prompt: "show me linkedin schedule",
* waitMs: 14000,
* successPatterns: ["linkedin", "schedule"],
* });
*/
export interface Scenario {
/** Short name for the scenario; used in screenshot filenames + reports. */
name: string;
/** Prompt to push into the cockpit chat. */
prompt: string;
/** Wait after dispatch before the post-shot. Default 14000ms. */
waitMs?: number;
/** Substrings to look for in the captured surface. */
successPatterns?: string[];
/** Substrings whose presence means failure (e.g., a stale-canvas marker). */
failPatterns?: string[];
}
export interface ScenarioResult {
scenario: Scenario;
passed: boolean;
preShot: string | null;
postShot: string | null;
/** Substrings actually found in the post-shot OCR / DOM dump. */
matchedPatterns: string[];
/** Substrings expected but not found. */
missingPatterns: string[];
durationMs: number;
error?: string;
}
export interface RunOptions {
/** Bridge endpoint for chat dispatch. Default 127.0.0.1:3147/chat-inject-push. */
injectUrl?: string;
/** Bridge endpoint for chat reset. Default 127.0.0.1:3147/chat-inject-pop (reset variant). */
resetUrl?: string;
/** Capture function. Returns path to captured PNG, or null on failure. */
capture?: (label: string) => Promise<string | null>;
/** OCR function. Reads a PNG path; returns text content. Default no-op. */
readShot?: (path: string) => Promise<string>;
}
/**
* Run one scenario. Pure plumbing — reset → pre-shot → push → wait → post-shot
* → eval. Producers (chat-drive bridge, screencapture, OCR) are passed in;
* the lib doesn't import them so this stays portable to a remote runner.
*/
export async function runScenario(scenario: Scenario, opts: RunOptions = {}): Promise<ScenarioResult> {
const start = Date.now();
const waitMs = scenario.waitMs ?? 14_000;
const inject = opts.injectUrl ?? "http://127.0.0.1:3147/chat-inject-push";
const capture = opts.capture ?? (async () => null);
const readShot = opts.readShot ?? (async () => "");
let preShot: string | null = null;
let postShot: string | null = null;
const matchedPatterns: string[] = [];
const missingPatterns: string[] = [];
try {
preShot = await capture(`${scenario.name}-pre`);
const r = await fetch(inject, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ text: scenario.prompt }),
});
if (!r.ok) {
return {
scenario,
passed: false,
preShot,
postShot: null,
matchedPatterns: [],
missingPatterns: scenario.successPatterns ?? [],
durationMs: Date.now() - start,
error: `inject HTTP ${r.status}`,
};
}
await new Promise((resolve) => setTimeout(resolve, waitMs));
postShot = await capture(`${scenario.name}-post`);
const text = postShot ? (await readShot(postShot)).toLowerCase() : "";
for (const p of scenario.successPatterns ?? []) {
if (text.includes(p.toLowerCase())) matchedPatterns.push(p);
else missingPatterns.push(p);
}
let passed = (scenario.successPatterns ?? []).length === 0 || missingPatterns.length === 0;
for (const p of scenario.failPatterns ?? []) {
if (text.includes(p.toLowerCase())) {
passed = false;
missingPatterns.push(`(unexpected) ${p}`);
}
}
return {
scenario,
passed,
preShot,
postShot,
matchedPatterns,
missingPatterns,
durationMs: Date.now() - start,
};
} catch (e) {
return {
scenario,
passed: false,
preShot,
postShot,
matchedPatterns,
missingPatterns,
durationMs: Date.now() - start,
error: String(e),
};
}
}
/**
* Run a list of scenarios sequentially. Returns one result per scenario.
*/
export async function runAll(scenarios: Scenario[], opts: RunOptions = {}): Promise<ScenarioResult[]> {
const out: ScenarioResult[] = [];
for (const s of scenarios) {
out.push(await runScenario(s, opts));
}
return out;
}
scripts- helper scripts it can run
prose-only skill - 4 inline code blocks live in SKILL.md above (no state/bin/ sidecar yet).
how we check it- the checks, plus the last 10 runs
no recent runs logged - the eval contract is declared but nothing has been graded yet