obsidian-rag/docs/superpowers/specs/2026-04-10-obsidian-rag-plugin-design.md

Obsidian RAG Plugin for OpenClaw — Technical Design Document

Date: 2026-04-10 Status: Approved Author: Santhosh Janardhanan

Overview

An OpenClaw plugin enabling semantic search through Obsidian vault notes using RAG (Retrieval-Augmented Generation). The plugin allows OpenClaw to respond to natural language queries about personal journal entries, shopping lists, financial records, health data, podcast notes, and project ideas stored in an Obsidian vault.

Problem Statement

Personal knowledge is fragmented across 677+ markdown files in an Obsidian vault, organized by topic but not searchable by meaning. Questions like "How was my mental health in 2024?" or "How much do I owe Sreenivas?" require reading multiple files across directories and synthesizing the answer. The plugin provides semantic search to surface relevant context.

Architecture

Approach: Separate Indexer Service + Thin Plugin

KnowledgeVault → Python Indexer (CLI) → LanceDB (filesystem)
                                           ↑ query
OpenClaw → TS Plugin (tools) ─────────────┘

• Python Indexer: Handles vault scanning, markdown parsing, chunking, embedding generation via Ollama, and LanceDB storage. Runs as a CLI tool.
• TypeScript Plugin: Registers OpenClaw tools that query the pre-built LanceDB index. Thin wrapper that provides the agent interface.
• LanceDB: Embedded vector database stored on local filesystem. Data directory resolved cross-platform via os.path.expanduser (Linux/macOS: ~/.obsidian-rag/, Windows: %APPDATA%/obsidian-rag/). No server required.

Layered Protocol Model

The system is structured in four protocol layers, each with its own contract:

┌─────────────────────────────────────────┐
│  TRANSPORT LAYER                         │
│  Tool registration, error normalization,  │
│  request/response envelope                │
├─────────────────────────────────────────┤
│  SESSION LAYER                            │
│  Health state machine, vault watcher,     │
│  auto-sync scheduling                     │
├─────────────────────────────────────────┤
│  TOOL LAYER                               │
│  search, index, status, memory_store      │
│  with parameter/response schemas           │
├─────────────────────────────────────────┤
│  DATA LAYER                               │
│  LanceDB client, indexer bridge,           │
│  chunking pipeline, embedding             │
└─────────────────────────────────────────┘

Technology Choices

Component	Choice	Rationale
Embedding model	mxbai-embed-large (1024-dim) via Ollama	Local, free, meets 1024+ dimension requirement, SOTA accuracy
Vector store	LanceDB (embedded)	No server, file-based, Rust-based efficiency, zero-copy versioning for incremental updates
Indexer language	Python	Richer embedding/ML ecosystem, better markdown parsing libraries
Plugin language	TypeScript	Native OpenClaw ecosystem, type safety, SDK examples
Config	Separate .obsidian-rag/config.json	Keeps plugin config separate from OpenClaw config

Transport Layer

Tool Registration Flow

The plugin follows OpenClaw's standard plugin lifecycle:

1. INSTALL — Link plugin to ~/.openclaw/extensions/obsidian-rag or <workspace>/.openclaw/extensions/obsidian-rag. OpenClaw reads openclaw.plugin.json on discovery.
2. REGISTER — Plugin registers 4 tools with OpenClaw's Tool Registry: obsidian_rag_search, obsidian_rag_index, obsidian_rag_status, obsidian_rag_memory_store. Each tool declares: name, description, parameterSchema, requiredPermissions.
3. SERVE — OpenClaw agent calls tools based on user intent.
4. HEALTH CHECK — Plugin.onLoad() probes Ollama, LanceDB, and the vault, then reports healthy | degraded | unavailable.

Standardized Response Envelope

Every tool response uses the same envelope:

{
  "status": "healthy" | "degraded" | "unavailable",
  "data": T | null,
  "error": {
    "code": string,
    "message": string,
    "recoverable": boolean,
    "suggestion": string
  } | null,
  "meta": {
    "query_time_ms": number,
    "chunks_scanned": number,
    "index_version": string,
    "vault_mtime": string
  }
}

The three-state health model (healthy, degraded, unavailable) lets the agent adapt behavior:

• healthy: Full service, no caveats.
• degraded: Partial service (e.g., Ollama down — searches still work using pre-computed vector similarity on the query text via LancDB's full-text search fallback, but embedding-based semantic search is unavailable. Results are returned with a staleness warning).
• unavailable: Cannot serve requests (e.g., no index — suggests remediation).

Error Normalization Matrix

Failure	Status	Code	Recoverable
Ollama down	degraded	OLLAMA_UNREACHABLE	yes — retry after start
Empty vault / no index	unavailable	INDEX_NOT_FOUND	yes — run index first
LanceDB corrupted	unavailable	INDEX_CORRUPTED	yes — run reindex
Path traversal attempt	unavailable	SECURITY_VIOLATION	no
Sensitive content blocked	degraded	SENSITIVE_FILTERED	yes — with confirmation
Indexer CLI crash	unavailable	INDEXER_FAILED	yes — retry once

OpenClaw's agent never sees raw exceptions. All errors are normalized with a code, human-readable message, recoverable flag, and remediation suggestion.

Session Layer

Health State Machine

The plugin transitions through three health states:

HEALTHY ⇄ DEGRADED ⇄ UNAVAILABLE

Transitions:

• HEALTHY → DEGRADED: Ollama health check fails (probed every 30s)
• DEGRADED → HEALTHY: Ollama responds again
• HEALTHY → UNAVAILABLE: Index deleted or DB corrupted
• UNAVAILABLE → HEALTHY: Successful reindex after recovery
• DEGRADED → UNAVAILABLE: Index corrupted while Ollama was down
• UNAVAILABLE → DEGRADED: Index restored but Ollama still down

Vault Watcher: Auto-Sync Protocol

The TS plugin runs a chokidar file watcher on the vault path, respecting deny_dirs and allow_dirs configuration.

Detection → Batching → Sync → Result:

1. DETECTION: chokidar watches vault_path. Debounces 2s after last change event to avoid triggering sync for every keystroke.
2. BATCHING: Collector window of 5s (configurable). Groups add/update/delete events into a changeset — single sync call for multiple files changed at once.
3. SYNC TRIGGER: After debounce + collect, spawns indexer with obsidian-rag sync. Indexer processes only modified files (mtime comparison). On completion, writes status JSON to the data directory (sync-result.json).
4. SYNC RESULT: Plugin reads sync-result.json on next tool call. Updates health state: HEALTHY if indexed >0 docs, UNAVAILABLE if 0. If last sync is >1h old and vault files changed, next search returns status: "degraded" with a staleness warning — results use LanceDB full-text search fallback instead of semantic embedding.

Session Lifecycle

Phase	Action
ON LOAD	Read config → Probe Ollama → Probe LanceDB → Probe Vault → Set health state
REGISTER	Register 4 tools with OpenClaw → Start vault watcher
SERVE	Handle tool calls → Auto-sync on file changes → Health re-probe every 30s
ON SHUTDOWN	Stop vault watcher → Wait for in-flight sync → Clean exit

Tool Layer

obsidian_rag_search

Primary semantic search tool for OpenClaw agent.

Parameters:

• query (required, string): Natural language question
• max_results (optional, int, default 5, range 1-50): Max chunks to return
• directory_filter (optional, string[]): Limit to subdirectories (e.g., ["Journal"])
• date_range (optional, object): { "from": "2025-01-01", "to": "2025-12-31" }
• tags (optional, string[]): Filter by hashtags

Response data:

{
  "results": [
    {
      "chunk_text": "...",
      "score": 0.87,
      "source_file": "Journal/2024-01-15.md",
      "section": "#mentalhealth",
      "date": "2024-01-15",
      "tags": ["#mentalhealth", "#therapy"],
      "chunk_index": 3
    }
  ],
  "sensitive_detected": false
}

When sensitive_detected=true, the agent confirms with the user before displaying. A memory_suggestion object may also be included with a suggested key/value for the agent to offer storage.

obsidian_rag_index

Trigger indexing from within OpenClaw.

Parameters:

• mode (required, enum): "full" | "sync" | "reindex"

Response data:

{
  "indexed_files": 677,
  "total_chunks": 3421,
  "duration_ms": 45230,
  "errors": []
}

Long-running operation. Returns two responses:

1. Immediate response (on spawn): { "job_id": "abc-123", "status": "started" } — agent polls progress via obsidian_rag_status which includes active_job: { id, mode, progress }.
2. Completion data (available via obsidian_rag_status after job finishes): { "indexed_files": 677, "total_chunks": 3421, "duration_ms": 45230, "errors": [] }.

obsidian_rag_status

Check index health — doc count, last sync, unindexed files, Ollama status.

Parameters: (none)

Response data:

{
  "plugin_health": "healthy",
  "total_docs": 677,
  "total_chunks": 3421,
  "last_sync": "2026-04-10T14:30:00Z",
  "unindexed_files": 3,
  "ollama_status": "up",
  "active_job": null
}

obsidian_rag_memory_store

Commit important facts to OpenClaw's memory for faster future retrieval.

Parameters:

• key (required, string): Identifier for the fact
• value (required, string): The fact to remember
• source (required, string): Source file path in vault

Response data:

{
  "stored": true,
  "key": "debt_to_sreenivas"
}

Auto-suggest logic: When search results match financial/health/commitment patterns, the plugin flags sensitive_detected=true and includes a memory_suggestion with a suggested key/value. The agent decides whether to commit.

OpenClaw Interaction Protocol

Intent → Tool mapping:

User Intent Pattern	Tool	Agent Behavior
"What did I write about X?" / "How was my Y in Z?"	search	Query vault, synthesize answer from chunks
"Index my vault" / "Update the search index"	index	Choose mode (sync default, full on no index), poll status
"Is the index up to date?" / "How many docs indexed?"	status	Report health, suggest index if stale
"Remember this" / Auto-suggest after search	memory_store	Confirm with user, store key+value+source

Protocol sequence:

1. Agent receives user query with vault-related intent
2. Agent calls obsidian_rag_status (if first interaction or >5min since last check)
   • Skips if status already cached and healthy
3. If unavailable → agent suggests running obsidian_rag_index (mode: "full")
4. If degraded → agent warns user but proceeds with search
5. Agent calls obsidian_rag_search with appropriate filters
6. If sensitive_detected=true → agent confirms before displaying
7. Agent synthesizes answer from chunks + existing knowledge
8. If memory_suggestion present → agent asks if user wants to store
   • User confirms → obsidian_rag_memory_store

Data Layer

LanceDB Table Schema

Table: obsidian_chunks

Column	Type	Description
vector	FixedList float32[1024]	Embedding from mxbai-embed-large via Ollama
chunk_id	string	UUID v4, primary key
chunk_text	string	Raw markdown text of the chunk
source_file	string	Relative path from vault root (e.g., "Journal/2024-01-15.md")
source_directory	string	Top-level directory name (e.g., "Journal")
section	string | null	Section heading for structured notes (e.g., "#mentalhealth")
date	string | null	ISO 8601 date parsed from filename
tags	list<string>	All hashtags found in the chunk
chunk_index	int32	Position within source document (0-indexed)
total_chunks	int32	Total chunks for this source file
modified_at	string	File mtime, ISO 8601 (for incremental sync)
indexed_at	string	When this chunk was indexed, ISO 8601

Indexer Bridge: TS ↔ Python

The TypeScript plugin invokes the Python indexer as a subprocess:

Invocation:

child_process.spawn("obsidian-rag", [mode], { env: { ...config } })

Communication:

• stdin: Not used
• stdout: NDJSON progress lines (streamed to agent via status)
• stderr: Error output (logged, surfaced on non-zero exit)
• Exit codes: 0 = success, 1 = partial failure, 2 = fatal

Progress events (stdout NDJSON):

{"type":"progress","phase":"embedding","current":150,"total":677}
{"type":"progress","phase":"storing","current":150,"total":677}
{"type":"complete","indexed_files":677,"total_chunks":3421,"duration_ms":45230,"errors":[]}

Sync result file: Written to {data_dir}/sync-result.json on completion. Read by the plugin on next tool call to update health state. Also serves as the source for status tool responses. Concurrent access: The indexer writes to sync-result.json.tmp then atomically renames to sync-result.json. The plugin reads sync-result.json — the rename ensures the plugin never reads a partially-written file. On Windows, the rename is atomic within the same volume.

Chunking Pipeline

SCAN → PARSE → CHUNK → ENRICH → EMBED → STORE

1. SCAN: Walk vault directories (respect deny/allow lists) → collect *.md files
2. PARSE: Extract frontmatter, headings, tags, dates from filename/path
3. CHUNK:
   • Structured notes (Journal entries): Split by section headers (#mentalhealth, #finance, etc.). Each section becomes its own chunk.
   • Unstructured notes (shopping lists, project ideas, entertainment index): Sliding window chunking (500 tokens, 100 token overlap).
4. ENRICH: Attach metadata: source_file, section, date, tags, chunk_index
5. EMBED: Batch chunks → Ollama /api/embed (mxbai-embed-large, 1024-dim). Batch size: 64 chunks per request.
6. STORE: Write vectors + metadata to LanceDB (upsert by chunk_id).

Incremental Sync Logic

1. Read last_sync_mtime from sync-result.json
2. Walk vault, compare each file's mtime vs last_sync_mtime
3. New/modified files → full chunk pipeline
4. Deleted files → remove chunks by source_file from LanceDB
5. Unchanged files → skip entirely

Security & Privacy

Four Defense Layers

Layer 1 — Path Traversal Prevention (Boundary) All file reads restricted to vault_path. Reject:

• Any path containing .. components
• Absolute paths not under vault_path
• Symlinks resolving outside vault_path

Implementation: path.resolve(vault_path, input) must start with vault_path.

Layer 2 — Input Sanitization (Clean) All vault content treated as untrusted before embedding:

• Strip HTML tags (prevent XSS in chunk_text)
• Remove executable code blocks (```...```)
• Normalize whitespace (prevent injection via formatting)
• Cap chunk_text at 2000 chars (prevent oversized payloads)

Layer 3 — Sensitive Content Guard (Gate) Detect and gate sensitive content in search results:

• Health: #mentalhealth, #physicalhealth, medication, therapy
• Financial: owe, owed, debt, paid, $, spent, spend
• Relations: #Relations (configurable section list)

Action: Set sensitive_detected=true in response. Agent must confirm before displaying to user. Never transmit sensitive content to external APIs.

Layer 4 — Local-Only Enforcement (Network) All data stays on the local machine:

• Ollama: localhost:11434 only
• LanceDB: local filesystem only
• Config: "local_only": true enforced
• Network audit test: verify zero outbound requests

If an external embedding endpoint is configured: require explicit user confirmation, block if sensitive content detected in payload.

Directory Access Control

Deny list (default): .obsidian, .trash, zzz-Archive, .git, .logseq — always excluded.

Allow list (optional): If set, only these directories are indexed. If empty, all directories except deny list are indexed. Example: ["Journal", "Finance"] → only those two directories.

Filter logic (Python indexer):

1. If allow_dirs is non-empty → only walk those directories
2. Skip any path matching deny_dirs patterns
3. Skip hidden directories (starting with .)

Filter logic (TS plugin):

• directory_filter parameter validates against known directories
• Reject unknown directories → prevent vault structure probing

Testing Strategy

Python Unit Tests (pytest)

Test File	Coverage
test_chunker.py	Section splitting, sliding window, metadata extraction
test_embedder.py	Mocked Ollama, batch embedding, error handling
test_vector_store.py	LanceDB CRUD, upsert, delete by source
test_security.py	Path traversal, sanitization, sensitive detection
test_indexer.py	Full pipeline, incremental sync, config handling

TypeScript Unit Tests (vitest)

Test File	Coverage
search.test.ts	Parameter validation, filter logic, response shape
index.test.ts	Mode validation, subprocess spawn, progress parsing
memory.test.ts	Key/value storage, auto-suggest patterns
vault-watcher.test.ts	Chokidar events, debounce, batching
security-guard.test.ts	Directory filter validation, sensitive flag

Integration Tests

Test File	Coverage
full_pipeline.test.ts	Index vault → search → verify results
sync_cycle.test.ts	Modify file → auto-sync → search updated content
health_state.test.ts	Stop Ollama → degraded → restart → healthy
openclaw_protocol.test.ts	Agent calls tools, validates envelope

Security Test Suite

Test File	Coverage
path_traversal.test.ts	../, symlinks, absolute paths, encoded paths
xss_prevention.test.ts	HTML/script injection in chunk text
prompt_injection.test.ts	Malicious content in vault notes
network_audit.test.ts	Verify zero outbound when local_only=true
sensitive_content.test.ts	Pattern detection, flagging, blocking

Mock vs Real

Component	Unit Test	Integration Test
Ollama embedding	Mocked — fixed 1024-dim vectors	Real — requires Ollama running
LanceDB	Real — temp directory, cleaned up	Real — temp directory, cleaned up
Obsidian vault	Mocked — fixture markdown files	Real — temp vault with real files
Python CLI	Mocked subprocess	Real — actual CLI invocation
Chokidar watcher	Mocked events	Real — actual filesystem events
OpenClaw agent	N/A	Real — tool call envelope validation

Integration tests require Ollama but skip gracefully if unavailable.

CLI Commands (Python Indexer)

Command	Purpose
obsidian-rag index	Initial full index of the vault (first-time setup)
obsidian-rag sync	Incremental — only process files modified since last sync
obsidian-rag reindex	Force full reindex (nuke existing, start fresh)
obsidian-rag status	Show index health: total docs, last sync time, unindexed files

Configuration

Config file resolved cross-platform via os.path.expanduser (Linux/macOS: ~/.obsidian-rag/config.json, Windows: %APPDATA%/obsidian-rag/config.json). For development, the config is at ./obsidian-rag/config.json relative to the project root.

{
  "vault_path": "./KnowledgeVault/Default",
  "embedding": {
    "provider": "ollama",
    "model": "mxbai-embed-large",
    "base_url": "http://localhost:11434",
    "dimensions": 1024
  },
  "vector_store": {
    "type": "lancedb",
    "path": "./obsidian-rag/vectors.lance"
  },
  "indexing": {
    "chunk_size": 500,
    "chunk_overlap": 100,
    "file_patterns": ["*.md"],
    "deny_dirs": [".obsidian", ".trash", "zzz-Archive", ".git", ".logseq"],
    "allow_dirs": []
  },
  "security": {
    "require_confirmation_for": ["health", "financial_debt"],
    "sensitive_sections": ["#mentalhealth", "#physicalhealth", "#Relations"],
    "local_only": true
  },
  "memory": {
    "auto_suggest": true,
    "patterns": {
      "financial": ["owe", "owed", "debt", "paid", "$", "spent", "spend"],
      "health": ["#mentalhealth", "#physicalhealth", "medication", "therapy"],
      "commitments": ["shopping list", "costco", "amazon", "grocery"]
    }
  }
}

Project Structure

obsidian-rag-skill/
├── README.md
├── LICENSE
├── .gitignore
├── openclaw.plugin.json
├── package.json
├── tsconfig.json
├── src/
│   ├── index.ts
│   ├── tools/
│   │   ├── search.ts
│   │   ├── index.ts
│   │   ├── status.ts
│   │   └── memory.ts
│   ├── services/
│   │   ├── vault-watcher.ts
│   │   ├── indexer-bridge.ts
│   │   └── security-guard.ts
│   └── utils/
│       ├── config.ts
│       └── lancedb.ts
├── python/
│   ├── pyproject.toml
│   ├── obsidian_rag/
│   │   ├── __init__.py
│   │   ├── cli.py
│   │   ├── indexer.py
│   │   ├── chunker.py
│   │   ├── embedder.py
│   │   ├── vector_store.py
│   │   ├── security.py
│   │   └── config.py
│   └── tests/
│       ├── test_chunker.py
│       ├── test_security.py
│       ├── test_embedder.py
│       ├── test_vector_store.py
│       └── test_indexer.py
├── tests/
│   ├── tools/
│   │   ├── search.test.ts
│   │   ├── index.test.ts
│   │   └── memory.test.ts
│   ├── services/
│   │   ├── vault-watcher.test.ts
│   │   └── security-guard.test.ts
│   ├── integration/
│   │   ├── full_pipeline.test.ts
│   │   ├── sync_cycle.test.ts
│   │   ├── health_state.test.ts
│   │   └── openclaw_protocol.test.ts
│   └── security/
│       ├── path_traversal.test.ts
│       ├── xss_prevention.test.ts
│       ├── prompt_injection.test.ts
│       ├── network_audit.test.ts
│       └── sensitive_content.test.ts
└── docs/
    └── superpowers/
        └── specs/

Publishing

Published to ClawHub as both a skill (SKILL.md) and a plugin package:

clawhub skill publish ./skill --slug obsidian-rag --version 1.0.0
clawhub package publish santhosh/obsidian-rag

Install: Link plugin to ~/.openclaw/extensions/obsidian-rag or use openclaw plugins install --link /path/to/obsidian-rag

Windows Development Notes

The development environment is Windows 11. The following platform-specific considerations apply:

Development Vault

The project uses ./KnowledgeVault/Default as the sample Obsidian vault for development. This vault is located at the project root and contains real note directories including Journal, Daily Life Stuff, Podcast Scripts, Side Project, and others.

Path Handling

• All path operations use path.resolve() and path.join() with forward slashes for consistency. Node.js and Python both handle forward slashes correctly on Windows.
• Config paths (vault_path, vector_store.path) support relative paths resolved from the project root (e.g., ./KnowledgeVault/Default).
• Data directory for production: %APPDATA%/obsidian-rag/ on Windows, ~/.obsidian-rag/ on Linux/macOS. For development: ./obsidian-rag/ relative to project root.
• Atomic file writes use .tmp + rename pattern. On Windows, rename is atomic within the same volume (both temp and target must be on the same drive).

Filesystem Watcher (chokidar)

• chokidar uses fs.watch on Windows which has reliable recursive watch support.
• The 2s debounce and 5s collect window are sufficient for Windows file change events, which may fire multiple events for a single save operation (especially with editors that use atomic save: write to temp file → rename).

Python CLI on Windows

• The obsidian-rag CLI entry point is installed via pip install -e . from python/. On Windows, the console script is created in the Python Scripts directory.
• child_process.spawn on Windows requires shell: true if the command is a Python console script, or use python -m obsidian_rag.cli as the command instead.
• Recommended: use python -m obsidian_rag.cli as the spawn command for cross-platform compatibility.

Testing on Windows

• Filesystem-based integration tests use os.tmpdir() for temporary vault and LanceDB directories.
• Path traversal tests must include Windows-specific vectors: C:\Windows\System32, CON, NUL, ..\\ (backslash variant), UNC paths (\\server\share).
• The deny_dirs list includes .obsidian which is the actual Obsidian config directory name on all platforms.