hermes-history-ingest

Hermes History Ingest — Conversation & Memory Mining

You are extracting knowledge from the user's Hermes agent history and distilling it into the Obsidian wiki. Hermes stores both free-form memories and structured session transcripts — focus on durable knowledge, not operational telemetry.

This skill can be invoked directly or via the wiki-history-ingest router (/wiki-history-ingest hermes).

Before You Start

1. Resolve config — follow the Config Resolution Protocol in llm-wiki/SKILL.md (inline @name override → walk up CWD for .env → ~/.obsidian-wiki/config → prompt setup). This gives OBSIDIAN_VAULT_PATH and HERMES_HISTORY_PATH (defaults to ~/.hermes)
2. Read .manifest.json at the vault root to check what has already been ingested
3. Read index.md at the vault root to understand what the wiki already contains

Ingest Modes

Append Mode (default)

Check .manifest.json for each source file. Only process:

• Files not in the manifest (new memory files, new session logs)
• Files whose modification time is newer than ingested_at in the manifest

Use this mode for regular syncs.

Full Mode

Process everything regardless of manifest. Use after wiki-rebuild or if the user explicitly asks for a full re-ingest.

Hermes Data Layout

Hermes stores all local artifacts under ~/.hermes/ (or $HERMES_HOME for non-default profiles).

~/.hermes/
├── memories/                          # Persistent agent memories (markdown or JSON)
│   └── *.md / *.json
├── skills/                            # Installed skills (read-only for ingest purposes)
│   └── <skill-name>/SKILL.md
├── sessions/                          # Session transcripts (if session logging is enabled)
│   └── YYYY-MM-DD/
│       └── <session-id>.jsonl
├── config.yaml                        # User config (model, theme, paths)
└── .hub/                              # Skills Hub state (lock.json, audit.log, quarantine/)

Key data sources ranked by value

1. memories/*.md / memories/*.json — highest signal; curated persistent knowledge the agent accumulated
2. sessions/**/*.jsonl — structured turn-by-turn transcripts; rich but noisy
3. config.yaml — metadata only (model preferences, paths); rarely worth ingesting

Skip .hub/ internals (audit/quarantine state) and the skills/ directory (source material, not user knowledge).

Step 1: Survey and Compute Delta

Scan HERMES_HISTORY_PATH and compare against .manifest.json:

• ~/.hermes/memories/
• ~/.hermes/sessions/**/ (if present)

Classify each file:

• New — not in manifest
• Modified — in manifest but file is newer than ingested_at
• Unchanged — already ingested and unchanged

Report a concise delta summary before deep parsing.

Step 2: Parse Memories First

Memories are the highest-value source. Hermes writes them as either:

• Markdown — structured prose with optional frontmatter; ingest directly
• JSON — {"content": "...", "created_at": "...", "tags": [...]} records

For each memory:

• Extract the core knowledge claim
• Note any tags Hermes attached (they often map to wiki categories)
• Merge into the appropriate wiki page rather than creating one memory = one page

Step 3: Parse Session JSONL Safely

Each session JSONL line is an event envelope. Common shapes:

{"role": "user", "content": "..."

Extraction rules

• Prioritize assistant responses that state conclusions, patterns, or decisions
• Extract user intent from high-signal turns; skip low-information follow-ups
• Treat tool_use / tool_result pairs as context, not primary content
• Skip token accounting, internal plumbing, and repeated plan echoes

Critical privacy filter

Session logs can include injected instructions, tool payloads, and sensitive text. Do not ingest verbatim.

• Remove API keys, tokens, passwords, credentials
• Redact private identifiers unless relevant and user-approved
• Summarize; do not quote raw transcripts verbatim

Step 4: Cluster by Topic

Do not create one wiki page per memory or session.

• Group memories by stable topic (concept, tool, project, technique)
• Split mixed sessions into separate themes
• Merge recurring patterns across dates and projects
• Use file paths or session cwd metadata to infer project scope when available

Step 5: Distill into Wiki Pages

Route extracted knowledge using existing wiki conventions:

• Project-specific architecture/process → projects/<name>/...
• General concepts → concepts/
• Recurring techniques/debug playbooks → skills/
• Tools/services/frameworks → entities/
• Cross-session patterns → synthesis/

For each impacted project, create/update projects/<name>/<name>.md.

Writing rules

• Distill knowledge, not chronology
• Avoid "on date X we discussed..." unless date context is essential
• Add summary: frontmatter on each new/updated page (1–2 sentences, ≤ 200 chars)
• Add confidence and lifecycle fields to every new page:

  base_confidence: 0.42
  lifecycle: draft
  lifecycle_changed: <ISO date today>
  Leave lifecycle unchanged on update.
• Add provenance markers:
  • ^[extracted] when directly grounded in explicit memory/session content
  • ^[inferred] when synthesizing patterns across multiple memories
  • ^[ambiguous] when memories conflict
• Add/update provenance: frontmatter mix for each changed page

Step 6: Update Manifest, Log, and Index

Update .manifest.json

For each processed source file:

• ingested_at, size_bytes, modified_at
• source_type: hermes_memory | hermes_session
• project: inferred project name (when applicable)
• pages_created, pages_updated

Add/update a top-level summary block:

{
  "hermes": {
    "source_path": "~/.hermes/",
    "last_ingested": "TIMESTAMP",
    "memories_ingested": 42,
    "sessions_ingested": 7,
    "pages_created": 5,
    "pages_updated": 12
  }
}

Update special files

Update index.md and log.md:

- [TIMESTAMP] HERMES_HISTORY_INGEST memories=N sessions=M pages_updated=X pages_created=Y mode=append|full

hot.md — Read $OBSIDIAN_VAULT_PATH/hot.md (create from the template in wiki-ingest if missing). Update Recent Activity with a one-line summary — e.g. "Ingested 42 Hermes memories and 7 sessions; dominant themes: reasoning strategies, tool use patterns." Keep the last 3 operations. Update updated timestamp.

Privacy and Compliance

• Distill and synthesize; avoid raw memory or transcript dumps
• Default to redaction for anything that looks sensitive
• Ask the user before storing personal or sensitive details
• Keep references to other people minimal and purpose-bound

Reference

See references/hermes-data-format.md for field-level notes and extraction guidance.

QMD Refresh After Vault Writes

QMD is a search index, not the source of truth. If $QMD_WIKI_COLLECTION is empty or unset, skip this step. Run it only after this skill has written or rewritten vault markdown. If QMD refresh fails, do not roll back the vault changes; report the QMD status separately.

Use $QMD_CLI if set; otherwise use qmd.

${QMD_CLI:-qmd} update

If the output says vectors are needed or embeddings may be stale, run:

${QMD_CLI:-qmd} embed

Verify the collection with either:

${QMD_CLI:-qmd} ls "$QMD_WIKI_COLLECTION"

or, when a specific page path is known:

${QMD_CLI:-qmd} get "qmd://$QMD_WIKI_COLLECTION/<page>.md" -l 5

Record one of:

• QMD refreshed: update + embed + verified
• QMD refreshed: update only + verified
• QMD skipped: QMD_WIKI_COLLECTION unset
• QMD skipped: qmd CLI unavailable
• QMD failed: <short error summary>