snappy-recover

Disaster recovery skill. Wraps state/bin/sync/snapshot.sh, restore.sh, and rollback.sh with a uniform contract. Snapshot writes a named checkpoint to s3://robert-storage/snappy-os-snapshots/. Restore moves bytes-at-timestamp to a holding dir without touching live. Rollback is the explicit, label-confirmed full reversion. Failure modes prevented: accidental overwrite-by-restore, ambiguous rollback ("which snapshot did I mean?"), and silent partial restores leaking via auto-push.

Purpose

Make every change reversible within 30 days (DO Spaces version retention) or indefinitely via named snapshots. Robert remains in the loop on every destructive recovery - the tool refuses to overwrite live without explicit confirmation.

Inputs

Steps

Sub-flow: snapshot

1. Compute timestamp ts = ISO-8601 now.
2. Validate label is non-empty and matches ^[a-z0-9-]{3,64}$.
3. Walk the local canonical, build a manifest.
4. Worker POST /_snapshot with manifest + bodies → writes to

s3://robert-storage/snappy-os-snapshots/<ts>-<label>/.

1. Append row to state/log/snapshots.ndjson:

{ts, label, files, bytes, snapshot_id}.

Sub-flow: restore

1. Validate snapshot_id exists in snapshots.ndjson (or in DO bucket

versioning index for an ISO timestamp without a label).

1. Validate target is a single skill name (no wildcards).
2. Fetch versioned bytes via Worker GET /_restore?id=<snapshot_id>&target=<name>.
3. Write to ~/projects/snappy-os/_restore/<target>-<ts>/. NEVER

overwrites live state.

1. Print diff against current canonical for operator review.

Sub-flow: rollback

1. Refuse if apply !== true.
2. Refuse if label !== <snapshot's recorded label>. The label match

is intentional friction.

1. Acquire upload lock.
2. Worker POST /_rollback?id=<snapshot_id> → atomic prefix swap on DO.
3. KV invalidation: Worker clears SKILLS_STORE cache.
4. Log row to state/log/snapshots.ndjson with

action: "rollback-applied".

1. Recommend caller run snappy-os pull --force --scope all on every

machine.

Log + eval

append("chain", { run_id, skill: "snappy-recover", action,
                  snapshot_id, label, target, files, bytes });
score("snappy-recover", run_id, {
  score: ok ? 1.0 : 0.0,
  primary_issue: ok ? null : reason,
});

Eval

Actor: state/bin/sync/{snapshot,restore,rollback}.sh calling Worker over HTTPS. Auditor: independent fetch from a different session reads the snapshot prefix listing or the restored bytes and asserts the manifest matches what the script reported writing.

Failure modes

• _restore/ MUST be in SYNC_DENY and gitignored. Otherwise

partial restores auto-push to DO and corrupt the canonical.

• Bucket versioning retention is 30 days. Older recoveries depend

on named snapshots; restoring a 60-day-old skill without a snapshot is impossible.

• Cross-region backup fills the gap when primary bucket is lost.

See disaster-recovery.md for the weekly tor1 → nyc3 cold backup.

• Label mismatch on rollback is intentional friction. Do not soften

this gate; the friction prevents "wrong snapshot" foot-guns.

• Atomic prefix swap on DO is best-effort. A mid-flight failure

may leave canonical pointing at a half-rolled-back state. The rollback script's auditor checks the post-swap manifest; mismatch flags rollback-partial and surfaces an alert.

Rubric

criteria:
  - name: correct_action_outcome
    kind: judge
    check: "The outcome of the 'action' (snapshot, restore, or rollback) matches the skill's documented behavior and expected side effects without data integrity issues."
  - name: rollback_safety_checks_enforced
    kind: deterministic
    check: "When action is 'rollback', the skill explicitly refused to proceed if 'apply !== true' or if 'label' did not match the snapshot's recorded label, returning a specific refusal message."
  - name: snapshot_metadata_recorded
    kind: deterministic
    check: "If the action was 'snapshot', a corresponding row was appended to 'state/log/snapshots.ndjson' with 'ts', 'label', 'files', 'bytes', and 'snapshot_id'."
  - name: restore_nondestructive_output
    kind: deterministic
    check: "If the action was 'restore', the target skill's bytes were written exclusively to '~/projects/snappy-os/_restore/<target>-<ts>/' and no live state was overwritten."

snappy-recover - loader

Per-turn rules for the snappy-recover skill. Full reference: state/skills/snappy-recover/SKILL.md. Do not skip these.

Critical Rules

• RESTORE is non-destructive - ALWAYS writes to ~/projects/snappy-os/_restore/<target>-<ts>/, NEVER overwrites live state; print diff for operator review
• ROLLBACK refuses unless apply === true AND label === <snapshot's recorded label> - both gates are intentional friction; do NOT soften them
• _restore/ MUST be in SYNC_DENY and gitignored - otherwise partial restores auto-push to DO and corrupt the canonical
• SNAPSHOT label must match ^[a-z0-9-]{3,64}$ - empty or fancy-cased labels are rejected
• Bucket versioning retention is 30 DAYS - older recoveries depend on named snapshots; restoring a 60-day-old skill without a snapshot is impossible
• After rollback, recommend caller run snappy-os pull --force --scope all on every machine - KV invalidation alone does not refresh local working copies
• Atomic prefix swap on DO is best-effort; mid-flight failure may leave canonical pointing at half-rolled-back state - auditor checks post-swap manifest, mismatch flags rollback-partial

Commands

| ui dashboard | state/skills/snappy-recover/resources/ui.openui | |invoke: state/bin/sync/{snapshot,restore,rollback}.sh |snapshot dest: s3://robert-storage/snappy-os-snapshots/<ts>-<label>/ |restore holding dir: ~/projects/snappy-os/_restore/<target>-<ts>/ |catalog: state/log/snapshots.ndjson |eval log: state/log/evals.ndjson (skill: "snappy-recover")

OpenUI Resource

• Skill-owned OpenUI Lang resource: state/skills/snappy-recover/resources/ui.openui. Read it before rendering or editing this skill's generated component surface.
• Treat this resource as a first-class artifact of the skill, not a generic chat response. Improve it when the skill's user-facing output needs to become richer.
• System resources compose OpenUI primitives and inherit SnappyChat tokens. Use ui_contract: branded in SKILL.md only for deliberate platform or client visuals.

Known Pitfalls

• Forgetting to gitignore _restore/ AND add it to SYNC_DENY causes the next sync to push the holding dir back to DO and clobber canonical
• Calling rollback without --apply true and getting a refusal is NOT a bug - it is the safety gate working correctly; do not auto-retry with apply=true to "fix" the refusal
• Cross-region backup (weekly tor1 → nyc3 cold backup) is the fallback when primary bucket is lost - see disaster-recovery.md

Self-Test

An agent reading this should correctly:

1. [ ] Refuse to let restore overwrite live state, regardless of any flag?
2. [ ] Treat a rollback refusal due to missing apply or label mismatch as a successful safety gate (score 1.0), not a failure?
3. [ ] Print the diff between the restore holding dir and live canonical before any operator confirmation?
4. [ ] Recommend cross-machine snappy-os pull --force after a rollback applies?

Self-report

If this loader fell short, append a line:

echo "[$(date -u +%FT%TZ)] snappy-recover: <what was missing>" >> state/log/loader-feedback.log

<!-- kernel-ok: historical attribution for the PID-loader pattern. -->

Self-correcting loader (PID feedback)

This is the rule that makes loaders learn. Ported from snappy-kernel, where it demonstrably worked: the snappy-dom-cartographer skill mapped Skool admin UI once, every later ship-agent inherited the full map for free.

Before you finish, do two things:

1. Fix gaps (P - proportional). MANDATORY. If this loader didn't cover your case - if you had to read another file, run an undocumented command, work around a wrong selector, or discover a quirk - you MUST attempt an Edit to this AGENTS.md before you log.

• Default action: edit this file. Surgical. Add the missing line, table

row, or code example where it belongs. No restructuring.

• Only LOGGED is allowed when: the fix needs >10 lines, spans multiple

files, or requires a structural rewrite. In that case the state/regen/drain.sh queue picks it up asynchronously.

• "I didn't have time" / "it's minor" / "the next agent will figure it out"

are NOT valid reasons. Minor gaps compound - that's the whole point of this loop.

• Concrete example of a valid P-fix: discover agent-browser screenshot

takes path positionally not as --output <path>. One line in a table. Edit it in. Do not just log it.

• The goal: the next agent never has to leave the loader.

2. Log the result.

echo "[$(date -u +%FT%TZ)] snappy-recover: <what was missing or fixed> [FIXED|LOGGED]" >> state/log/loader-feedback.log

• FIXED = you patched this loader inline (P-fix).
• LOGGED = too large for inline; the PostToolUse enqueue + Stop-hook drain

will rewrite the loader from scratch on next session-end.

Do not skip this. Every agent run must leave the system better than it found it. The loader is the setpoint; you are the sensor; the gap is the error signal; closing the gap is the correction.