library-rag

Library RAG

Semantic search over ~/.hermes/library/ using Nemotron-3-Embed-1B embeddings (via NVIDIA NIM) stored in sqlite-vec. Enables meaning-based retrieval across any text corpus — books, documents, reference works — in any language.

Architecture

NVIDIA NIM API (nemotron-3-embed-1b, 2048-dim)
        │
        ▼
~/.hermes/library/rag_index.db (sqlite-vec)
  ├── chunks table      — text + metadata (source, book, chapter, section)
  ├── vec_chunks table  — L2-normalized vector embeddings
  └── indexed_files     — SHA-256 hash tracking for incremental updates

MCP Server (mcp_server.py)
  ├── search(query, top_k, source_type)  — semantic search, auto-available
  ├── stats()                             — index statistics
  └── add_book(file_path, ...)            — EPUB/PDF → md → index in one call

Onboarding workflow (run this the first time a user sets up the skill)

When a user wants to start using Library RAG, walk them through this sequence. Use AskUserQuestion for the decisions marked ASK — don't assume paths or silently create directories outside the home folder.

Step 1 — Check prerequisites

• If sqlite-vec import fails → run pip install -r requirements.txt (add --break-system-packages on externally-managed Python). See README "Installation".
• If NO API KEY → help the user create a free NVIDIA NIM key at https://build.nvidia.com/nvidia/nemotron-3-embed-1b, then store it: echo 'NVIDIA_API_KEY=nvapi-...' >> ~/.hermes/.env. (Legacy OpenRouter keys still work: if only OPENROUTER_API_KEY is set, load_api_key() auto-switches to the bge-m3 OpenRouter endpoint.)

Step 2 — Define directories (ASK)

Three locations drive everything. Confirm them with the user and export the env vars (persist in their shell profile). Defaults in parentheses:

Recommended layout (keep raw/text out of LIBRARY_ROOT — see pitfall below):

Pitfall — avoid double-indexing: discover_files() indexes every .md and .txt under LIBRARY_ROOT. If both the extracted .txt and the converted .md of the same book live under LIBRARY_ROOT, the content is embedded twice. Keep only one representation (prefer the structured .md) under LIBRARY_ROOT; keep raw + text staging in BOOKS_DIR outside it.

Create the chosen dirs:

Step 3 — Add the first content & build the index

• EPUB/PDF via MCP (if the server is configured): call add_book(...) — it converts and indexes in one step.
• Manual: drop source files in $BOOKS_DIR/raw, convert to structured markdown under $LIBRARY_ROOT/<source_type>/ (see convert_epub_library.py and references/epub-conversion.md), then:

Step 4 — Automated scanning/conversion/indexing (ASK)

Ask the user whether they want new files indexed automatically or manually:

• Manual (default, recommended to start): they run python3 scripts/rag_index.py after adding files. Incremental + SHA-256 tracking means re-runs are cheap and only touch new/changed files. No setup needed.

• Automated (cron): schedule a periodic incremental index. Only offer this once Step 3 works end-to-end. Example — index nightly at 02:00 and log output:

  # crontab -e   (adjust the repo path and env vars)
  0 2 * * * NVIDIA_API_KEY=nvapi-... LIBRARY_ROOT=$HOME/.hermes/library \
    /usr/bin/python3 $HOME/library-rag/scripts/rag_index.py >> $HOME/.hermes/rag_index.log 2>&1

  If they also want EPUBs auto-converted before indexing, chain a conversion step ahead of rag_index.py in the same cron line (or a small wrapper script). On macOS, launchd / a launchd plist is the more reliable equivalent of cron.

  Cron cautions to mention:

  • Cron has a minimal environment — set NVIDIA_API_KEY, LIBRARY_ROOT, and use an absolute python3 path (or activate the venv inside a wrapper script).
  • Never overlap two --rebuild runs on the same DB (see Pitfalls). A nightly incremental run is safe; a full --rebuild should stay manual.
  • NIM embeddings are free on the NVIDIA free trial tier; the log still shows token usage.

After onboarding, summarize for the user: the three paths chosen, where the DB lives, and whether indexing is manual or scheduled.

MCP Tools (auto-available in every conversation)

Once configured in config.yaml, these tools are available as mcp_library_rag_search, mcp_library_rag_stats, mcp_library_rag_add_book.

Config

# ~/.hermes/config.yaml
mcp_servers:
  library_rag:
    command: "python3"
    args: ["~/.hermes/skills/research/library-rag/scripts/mcp_server.py"]
    timeout: 60

Restart the gateway after adding.

search

mcp_library_rag_search(query="your search query", top_k=10)
mcp_library_rag_search(query="search terms", source_type="my-source-type")

Returns JSON array of results with similarity, source_type, book, chapter, section_title, text.

stats

mcp_library_rag_stats()
→ {"total_chunks": 78000, "files_indexed": 1383, "by_source": {...}}

add_book

mcp_library_rag_add_book(
    file_path="/path/to/book.epub",   # or /path/to/doc.pdf
    book_slug="my-new-book",
    author="Author Name",
    title="Book Title",
    year="2024"
)

Converts EPUB → chapters or PDF → pages, then indexes them. One-call workflow for adding new books to the library.

Scripts (for batch operations)

rag_index.py — Indexer

# Index all new/changed files (incremental — SHA-256 hash tracking)
python3 scripts/rag_index.py
 
# Full rebuild (drop everything, re-index from scratch)
python3 scripts/rag_index.py --rebuild
 
# Dry run (show what would be indexed, no API calls)
python3 scripts/rag_index.py --dry-run
 
# Index only one source type
python3 scripts/rag_index.py --source my-source-type
 
# Prune DB rows for files deleted/moved out of the library, then shrink the file
python3 scripts/rag_index.py --prune-missing --vacuum

When to use:

• After adding new files to the library → run incremental (default)
• After changing chunking strategy → run --rebuild
• To estimate cost/time → run --dry-run first
• After deleting/moving files out of the library → run --prune-missing (removes orphaned chunks + vectors so the DB doesn't keep dead rows; safe alongside incremental indexing — only files absent from disk are retired)
• After a --rebuild or a large --prune-missing → run --vacuum to shrink the DB file (SQLite reuses freed pages but doesn't shrink the file otherwise)

rag_query.py — CLI Query

python3 scripts/rag_query.py "your search query"
python3 scripts/rag_query.py "search terms" --top-k 5
python3 scripts/rag_query.py "query" --source my-source-type --verbose
python3 scripts/rag_query.py --stats

Can also be imported:

from rag_query import search
results = search("query text", top_k=10)

Conversion Tools

EPUB → Markdown (scripts/convert_epub_library.py)

Extracts text from EPUB files and splits into per-chapter Markdown with YAML frontmatter. Runnable as a CLI or importable as convert_epub() — the same implementation backs the MCP add_book tool, so there is one conversion code path.

python3 scripts/convert_epub_library.py \
    --epub book.epub --slug my-book --title "Book Title" --author "Author" --year 2024 \
    --md-root ~/.hermes/library/books/markdown   # under LIBRARY_ROOT so it gets indexed

Raw EPUB + extracted text stage under $BOOKS_DIR (outside LIBRARY_ROOT); only the structured markdown is written to --md-root, so the raw text is never double-indexed.

Chapter detection tries several styles (CHAPTER 12 / CHAPTER One / PART IV / 12. Title / bare 12\nTitle), picking the first that confidently applies, and guards against date/page-number false positives. Before trusting a conversion, eyeball convert_epub_library.audit_chapter_markers(text) — every entry should be a real chapter, not a date or page number.

Key pitfalls (see references/epub-conversion.md for full details):

1. Multiple TOC ghosts: Large EPUBs produce multiple table-of-contents blocks. Skip all when finding chapter headings in body text.
2. CHAPTER N vs N. Title: EPUBs use different heading patterns in TOC vs body. Match the BODY pattern.
3. Title + subtitle leak into YAML: Use only the main title for the title: field.
4. Slug truncation: Truncate slugs to 60 chars (preserving the chNN- prefix).
5. Page-number blocks: Some EPUBs extract huge blocks of standalone page numbers. Detect and exclude.
6. Post-processing pass: Always verify every .md file has valid YAML frontmatter.

PDF → Markdown (scripts/convert_pdf_library.py)

Extracts text and tables with pdfplumber. Each page → a ## Page N section (so the markdown chunker yields per-page citations); tables → **[Table N]** blocks. CLI or convert_pdf(); it also backs add_book for .pdf inputs.

python3 scripts/convert_pdf_library.py \
    --pdf doc.pdf --slug my-doc --title "Doc" --author "Author" --year 2024 \
    --md-root ~/.hermes/library/books/markdown

The table block is display-only: clean_for_embedding strips **[Table N]** and rows ending in <br> from the embedding input (the page's narrative text already carries the table content linearly), so tables stay searchable without being embedded twice.

Built-in chunkers (rag_index.py)

For domain-specific formats (PDFs, XML, JSON), write a custom chunker and register it in discover_files(). See references/chunking-strategies.md for patterns.

Adding new books to the library

Quick path (EPUB/PDF via MCP)

Use mcp_library_rag_add_book — it dispatches on extension and handles EPUB/PDF → md → index in one call.

Manual path

1. Convert to structured Markdown with the converter CLI for the format. Both stage the raw file + extracted text under $BOOKS_DIR (outside LIBRARY_ROOT, so they're not indexed) and write markdown to --md-root:

   # EPUB → per-chapter markdown
   python3 scripts/convert_epub_library.py \
       --epub book.epub --slug <slug> --title "..." --author "..." --year 2024 \
       --md-root $LIBRARY_ROOT/books/markdown
    
   # PDF → per-page markdown (tables preserved)
   python3 scripts/convert_pdf_library.py \
       --pdf doc.pdf --slug <slug> --title "..." --author "..." --year 2024 \
       --md-root $LIBRARY_ROOT/books/markdown

   For other sources (XML/JSON, etc.), follow the extraction pattern in references/chunking-strategies.md and write the markdown under $LIBRARY_ROOT/<source_type>/ yourself.

2. Register the chunker (if new file type):

   • Add a chunker function in rag_index.py
   • Register it in discover_files() See references/chunking-strategies.md for patterns and pitfalls. See references/portable-rag-per-skill.md for the standalone per-skill RAG pattern. See references/rag-pipeline-review.md for a systematic audit checklist.

3. Index:

   python3 scripts/rag_index.py  # incremental — only new files

Cost

Default provider is NVIDIA NIM (nvidia/nemotron-3-embed-1b) — free on the NVIDIA API free trial tier. No per-token embedding cost for personal use.

Legacy OpenRouter path (baai/bge-m3, ~$0.01/M tokens) remains available as a fallback; see references/openrouter-embeddings.md.

Effectively free for personal use.

Portable Standalone RAG Instances

The same Nemotron-3-Embed-1B → sqlite-vec pattern can be deployed as a standalone, self-contained RAG inside any skill directory — no MCP registration, no dependency on ~/.hermes/library/. This makes the skill portable: zip the folder, drop on another machine, it works.

How to build a standalone instance

1. Copy rag_index.py and rag_query.py into the skill's scripts/ dir
2. Parameterize the DB path — replace the hardcoded LIBRARY_ROOT / DB_PATH with an env var or CLI flag, defaulting to the skill's own references/ directory
3. Write a domain-specific chunker — chunk by the document's natural structure (headings, sections). For PDF-extracted content, parse <!-- Page N --> markers via split_by_pages() for per-chunk page citations.
4. Do NOT register an MCP server — scripts import rag_query.py directly:

   from rag_query import search
   results = search("query text", top_k=5)
5. L2-normalize vectors at store + query time — add normalize_vec() to float_to_blob() (indexer) and get_embedding() (query). Makes cosine similarity exact, not an implicit assumption.
6. Add ~15% chunk overlap — split_long_text() should carry the tail of each chunk into the next for better recall.
7. Clean embedding input — clean_for_embedding() must strip <br>, **[Table N]**, and <!-- Page N --> in addition to markdown markers.
8. Wire in MIN_CHUNK_CHARS — call merge_tiny_chunks() after chunking to prevent near-empty chunks from diluting search results.
9. Shared dependency: the NVIDIA API key (Nemotron-3-Embed-1B via NIM). The embedding model and sqlite-vec extension are shared, not duplicated.

See references/portable-rag-per-skill.md for full code patterns and references/rag-pipeline-review.md for a systematic audit checklist.

When to use standalone vs. the main library

Pitfalls

• sqlite-vec must be loaded before DROP/CREATE of vec0 tables: The extension must be loaded first in init_db().
• file_hash must be added to chunks before storing: The indexer adds it after chunking but before embedding. Don't forget this when adding new chunkers.
• Single-newline wrapping: Some text sources use single \n for line wrapping, not paragraph breaks. Normalize to spaces before splitting on \n\n.
• MCP server timeout: The default 60s timeout is fine for search and stats. add_book may need more time for large EPUBs — increase timeout in config if needed.
• sqlite-vec source filtering: sqlite-vec's MATCH/k vector search cannot be combined with arbitrary WHERE clauses. source_type and source_book filters must be applied as post-filters after over-fetching results (fetch top_k * 5, then filter on chunk metadata). See references/portable-rag-per-skill.md for the correct pattern.
• Portable RAG instances: To create a standalone RAG for a different skill, copy rag_index.py and rag_query.py and parameterize: (1) resolve paths from __file__ not hardcoded LIBRARY_ROOT, (2) store DB inside the skill's references/ dir, (3) skip the MCP server — import search() directly instead, (4) write a domain-specific chunker, (5) parse <!-- Page N --> markers for per-chunk page citations, (6) L2-normalize vectors at store+query time, (7) add ~15% chunk overlap for better recall. See references/portable-rag-per-skill.md for the full pattern.
• Page citations require marker parsing: PDF extractors insert <!-- Page N --> comments, but chunkers must actively parse them — otherwise all chunks degrade to chapter-level citations. Use split_by_pages() in the chunker to split by markers and stamp each chunk with its page.
• L2 normalization required for exact similarity: 1 - dist²/2 is only exact cosine if both stored and query vectors are unit-normalized. Nemotron-3-Embed-1B (and the legacy bge-m3 path) may return near-unit vectors, but normalize_vec() is called at store time (float_to_blob) and query time (get_embedding) to guarantee this explicitly.
• clean_for_embedding must strip table/PDF artifacts: In addition to markdown markers, strip <br>, **[Table N]**, and <!-- Page N --> from embedding input. These are noise from PDF table extraction. Chunk text in the DB retains them for display readability.
• Never run two --rebuild processes on the same DB simultaneously: SQLite allows concurrent connections but --rebuild drops and recreates tables. If a foreground test and a background job overlap, the background process gets spurious "Failed after 3 retries" errors. Always let any foreground --rebuild test fully exit before starting the background job, or use --dry-run for quick tests (it doesn't touch the DB).
• Full library re-index takes ~50-90 min: Small files finish fast, large text collections dominate. NIM embeddings are free; legacy OpenRouter path was ~$0.01-$0.12 depending on corpus size. Batch size 32 with a short delay between batches remains a safe default.