obsidian-rag/docs/superpowers/specs/2026-04-10-obsidian-rag-plugin-design.md

# Obsidian RAG Plugin for OpenClaw — Technical Design Document

**Date:** 2026-04-10
**Status:** Approved
**Author:** Santhosh Janardhanan

## Overview

An OpenClaw plugin enabling semantic search through Obsidian vault notes using RAG (Retrieval-Augmented Generation). The plugin allows OpenClaw to respond to natural language queries about personal journal entries, shopping lists, financial records, health data, podcast notes, and project ideas stored in an Obsidian vault.

## Problem Statement

Personal knowledge is fragmented across 677+ markdown files in an Obsidian vault, organized by topic but not searchable by meaning. Questions like "How was my mental health in 2024?" or "How much do I owe Sreenivas?" require reading multiple files across directories and synthesizing the answer. The plugin provides semantic search to surface relevant context.

## Architecture

### Approach: Separate Indexer Service + Thin Plugin

```
KnowledgeVault → Python Indexer (CLI) → LanceDB (filesystem)
                                           ↑ query
OpenClaw → TS Plugin (tools) ─────────────┘
```

- **Python Indexer**: Handles vault scanning, markdown parsing, chunking, embedding generation via Ollama, and LanceDB storage. Runs as a CLI tool.
- **TypeScript Plugin**: Registers OpenClaw tools that query the pre-built LanceDB index. Thin wrapper that provides the agent interface.
- **LanceDB**: Embedded vector database stored on local filesystem. Data directory resolved cross-platform via `os.path.expanduser` (Linux/macOS: `~/.obsidian-rag/`, Windows: `%APPDATA%/obsidian-rag/`). No server required.

### Layered Protocol Model

The system is structured in four protocol layers, each with its own contract:

```
┌─────────────────────────────────────────┐
│  TRANSPORT LAYER                         │
│  Tool registration, error normalization,  │
│  request/response envelope                │
├─────────────────────────────────────────┤
│  SESSION LAYER                            │
│  Health state machine, vault watcher,     │
│  auto-sync scheduling                     │
├─────────────────────────────────────────┤
│  TOOL LAYER                               │
│  search, index, status, memory_store      │
│  with parameter/response schemas           │
├─────────────────────────────────────────┤
│  DATA LAYER                               │
│  LanceDB client, indexer bridge,           │
│  chunking pipeline, embedding             │
└─────────────────────────────────────────┘
```

## Technology Choices

| Component | Choice | Rationale |
|-----------|--------|-----------|
| Embedding model | `mxbai-embed-large` (1024-dim) via Ollama | Local, free, meets 1024+ dimension requirement, SOTA accuracy |
| Vector store | LanceDB (embedded) | No server, file-based, Rust-based efficiency, zero-copy versioning for incremental updates |
| Indexer language | Python | Richer embedding/ML ecosystem, better markdown parsing libraries |
| Plugin language | TypeScript | Native OpenClaw ecosystem, type safety, SDK examples |
| Config | Separate `.obsidian-rag/config.json` | Keeps plugin config separate from OpenClaw config |

## Transport Layer

### Tool Registration Flow

The plugin follows OpenClaw's standard plugin lifecycle:

1. **INSTALL** — Link plugin to `~/.openclaw/extensions/obsidian-rag` or `<workspace>/.openclaw/extensions/obsidian-rag`. OpenClaw reads `openclaw.plugin.json` on discovery.
2. **REGISTER** — Plugin registers 4 tools with OpenClaw's Tool Registry: `obsidian_rag_search`, `obsidian_rag_index`, `obsidian_rag_status`, `obsidian_rag_memory_store`. Each tool declares: name, description, parameterSchema, requiredPermissions.
3. **SERVE** — OpenClaw agent calls tools based on user intent.
4. **HEALTH CHECK** — `Plugin.onLoad()` probes Ollama, LanceDB, and the vault, then reports `healthy | degraded | unavailable`.

### Standardized Response Envelope

Every tool response uses the same envelope:

```json
{
  "status": "healthy" | "degraded" | "unavailable",
  "data": T | null,
  "error": {
    "code": string,
    "message": string,
    "recoverable": boolean,
    "suggestion": string
  } | null,
  "meta": {
    "query_time_ms": number,
    "chunks_scanned": number,
    "index_version": string,
    "vault_mtime": string
  }
}
```

The three-state health model (`healthy`, `degraded`, `unavailable`) lets the agent adapt behavior:
- **healthy**: Full service, no caveats.
- **degraded**: Partial service (e.g., Ollama down — searches still work using pre-computed vector similarity on the query text via LancDB's full-text search fallback, but embedding-based semantic search is unavailable. Results are returned with a staleness warning).
- **unavailable**: Cannot serve requests (e.g., no index — suggests remediation).

### Error Normalization Matrix

| Failure | Status | Code | Recoverable |
|---------|--------|------|--------------|
| Ollama down | degraded | `OLLAMA_UNREACHABLE` | yes — retry after start |
| Empty vault / no index | unavailable | `INDEX_NOT_FOUND` | yes — run index first |
| LanceDB corrupted | unavailable | `INDEX_CORRUPTED` | yes — run reindex |
| Path traversal attempt | unavailable | `SECURITY_VIOLATION` | no |
| Sensitive content blocked | degraded | `SENSITIVE_FILTERED` | yes — with confirmation |
| Indexer CLI crash | unavailable | `INDEXER_FAILED` | yes — retry once |

OpenClaw's agent never sees raw exceptions. All errors are normalized with a code, human-readable message, recoverable flag, and remediation suggestion.

## Session Layer

### Health State Machine

The plugin transitions through three health states:

```
HEALTHY ⇄ DEGRADED ⇄ UNAVAILABLE
```

Transitions:
- **HEALTHY → DEGRADED**: Ollama health check fails (probed every 30s)
- **DEGRADED → HEALTHY**: Ollama responds again
- **HEALTHY → UNAVAILABLE**: Index deleted or DB corrupted
- **UNAVAILABLE → HEALTHY**: Successful reindex after recovery
- **DEGRADED → UNAVAILABLE**: Index corrupted while Ollama was down
- **UNAVAILABLE → DEGRADED**: Index restored but Ollama still down

### Vault Watcher: Auto-Sync Protocol

The TS plugin runs a chokidar file watcher on the vault path, respecting `deny_dirs` and `allow_dirs` configuration.

**Detection → Batching → Sync → Result:**

1. **DETECTION**: chokidar watches vault_path. Debounces 2s after last change event to avoid triggering sync for every keystroke.
2. **BATCHING**: Collector window of 5s (configurable). Groups add/update/delete events into a changeset — single sync call for multiple files changed at once.
3. **SYNC TRIGGER**: After debounce + collect, spawns indexer with `obsidian-rag sync`. Indexer processes only modified files (mtime comparison). On completion, writes status JSON to the data directory (`sync-result.json`).
4. **SYNC RESULT**: Plugin reads `sync-result.json` on next tool call. Updates health state: HEALTHY if indexed >0 docs, UNAVAILABLE if 0. If last sync is >1h old and vault files changed, next search returns `status: "degraded"` with a staleness warning — results use LanceDB full-text search fallback instead of semantic embedding.

### Session Lifecycle

| Phase | Action |
|-------|--------|
| **ON LOAD** | Read config → Probe Ollama → Probe LanceDB → Probe Vault → Set health state |
| **REGISTER** | Register 4 tools with OpenClaw → Start vault watcher |
| **SERVE** | Handle tool calls → Auto-sync on file changes → Health re-probe every 30s |
| **ON SHUTDOWN** | Stop vault watcher → Wait for in-flight sync → Clean exit |

## Tool Layer

### obsidian_rag_search

Primary semantic search tool for OpenClaw agent.

**Parameters:**
- `query` (required, string): Natural language question
- `max_results` (optional, int, default 5, range 1-50): Max chunks to return
- `directory_filter` (optional, string[]): Limit to subdirectories (e.g., `["Journal"]`)
- `date_range` (optional, object): `{ "from": "2025-01-01", "to": "2025-12-31" }`
- `tags` (optional, string[]): Filter by hashtags

**Response `data`:**
```json
{
  "results": [
    {
      "chunk_text": "...",
      "score": 0.87,
      "source_file": "Journal/2024-01-15.md",
      "section": "#mentalhealth",
      "date": "2024-01-15",
      "tags": ["#mentalhealth", "#therapy"],
      "chunk_index": 3
    }
  ],
  "sensitive_detected": false
}
```

When `sensitive_detected=true`, the agent confirms with the user before displaying. A `memory_suggestion` object may also be included with a suggested key/value for the agent to offer storage.

### obsidian_rag_index

Trigger indexing from within OpenClaw.

**Parameters:**
- `mode` (required, enum): `"full"` | `"sync"` | `"reindex"`

**Response `data`:**
```json
{
  "indexed_files": 677,
  "total_chunks": 3421,
  "duration_ms": 45230,
  "errors": []
}
```

Long-running operation. Returns two responses:
1. **Immediate response** (on spawn): `{ "job_id": "abc-123", "status": "started" }` — agent polls progress via `obsidian_rag_status` which includes `active_job: { id, mode, progress }`.
2. **Completion data** (available via `obsidian_rag_status` after job finishes): `{ "indexed_files": 677, "total_chunks": 3421, "duration_ms": 45230, "errors": [] }`.

### obsidian_rag_status

Check index health — doc count, last sync, unindexed files, Ollama status.

**Parameters:** (none)

**Response `data`:**
```json
{
  "plugin_health": "healthy",
  "total_docs": 677,
  "total_chunks": 3421,
  "last_sync": "2026-04-10T14:30:00Z",
  "unindexed_files": 3,
  "ollama_status": "up",
  "active_job": null
}
```

### obsidian_rag_memory_store

Commit important facts to OpenClaw's memory for faster future retrieval.

**Parameters:**
- `key` (required, string): Identifier for the fact
- `value` (required, string): The fact to remember
- `source` (required, string): Source file path in vault

**Response `data`:**
```json
{
  "stored": true,
  "key": "debt_to_sreenivas"
}
```

Auto-suggest logic: When search results match financial/health/commitment patterns, the plugin flags `sensitive_detected=true` and includes a `memory_suggestion` with a suggested key/value. The agent decides whether to commit.

### OpenClaw Interaction Protocol

**Intent → Tool mapping:**

| User Intent Pattern | Tool | Agent Behavior |
|---------------------|------|----------------|
| "What did I write about X?" / "How was my Y in Z?" | search | Query vault, synthesize answer from chunks |
| "Index my vault" / "Update the search index" | index | Choose mode (sync default, full on no index), poll status |
| "Is the index up to date?" / "How many docs indexed?" | status | Report health, suggest index if stale |
| "Remember this" / Auto-suggest after search | memory_store | Confirm with user, store key+value+source |

**Protocol sequence:**

1. Agent receives user query with vault-related intent
2. Agent calls `obsidian_rag_status` (if first interaction or >5min since last check)
   - Skips if status already cached and healthy
3. If unavailable → agent suggests running `obsidian_rag_index` (mode: "full")
4. If degraded → agent warns user but proceeds with search
5. Agent calls `obsidian_rag_search` with appropriate filters
6. If `sensitive_detected=true` → agent confirms before displaying
7. Agent synthesizes answer from chunks + existing knowledge
8. If `memory_suggestion` present → agent asks if user wants to store
   - User confirms → `obsidian_rag_memory_store`

## Data Layer

### LanceDB Table Schema

Table: `obsidian_chunks`

| Column | Type | Description |
|--------|------|-------------|
| `vector` | FixedList float32[1024] | Embedding from mxbai-embed-large via Ollama |
| `chunk_id` | string | UUID v4, primary key |
| `chunk_text` | string | Raw markdown text of the chunk |
| `source_file` | string | Relative path from vault root (e.g., "Journal/2024-01-15.md") |
| `source_directory` | string | Top-level directory name (e.g., "Journal") |
| `section` | string \| null | Section heading for structured notes (e.g., "#mentalhealth") |
| `date` | string \| null | ISO 8601 date parsed from filename |
| `tags` | list\<string\> | All hashtags found in the chunk |
| `chunk_index` | int32 | Position within source document (0-indexed) |
| `total_chunks` | int32 | Total chunks for this source file |
| `modified_at` | string | File mtime, ISO 8601 (for incremental sync) |
| `indexed_at` | string | When this chunk was indexed, ISO 8601 |

### Indexer Bridge: TS ↔ Python

The TypeScript plugin invokes the Python indexer as a subprocess:

**Invocation:**
```
child_process.spawn("obsidian-rag", [mode], { env: { ...config } })
```

**Communication:**
- **stdin**: Not used
- **stdout**: NDJSON progress lines (streamed to agent via status)
- **stderr**: Error output (logged, surfaced on non-zero exit)
- **Exit codes**: 0 = success, 1 = partial failure, 2 = fatal

**Progress events (stdout NDJSON):**
```json
{"type":"progress","phase":"embedding","current":150,"total":677}
{"type":"progress","phase":"storing","current":150,"total":677}
{"type":"complete","indexed_files":677,"total_chunks":3421,"duration_ms":45230,"errors":[]}
```

**Sync result file:** Written to `{data_dir}/sync-result.json` on completion. Read by the plugin on next tool call to update health state. Also serves as the source for status tool responses. **Concurrent access:** The indexer writes to `sync-result.json.tmp` then atomically renames to `sync-result.json`. The plugin reads `sync-result.json` — the rename ensures the plugin never reads a partially-written file. On Windows, the rename is atomic within the same volume.

### Chunking Pipeline

```
SCAN → PARSE → CHUNK → ENRICH → EMBED → STORE
```

1. **SCAN**: Walk vault directories (respect deny/allow lists) → collect `*.md` files
2. **PARSE**: Extract frontmatter, headings, tags, dates from filename/path
3. **CHUNK**:
   - Structured notes (Journal entries): Split by section headers (`#mentalhealth`, `#finance`, etc.). Each section becomes its own chunk.
   - Unstructured notes (shopping lists, project ideas, entertainment index): Sliding window chunking (500 tokens, 100 token overlap).
4. **ENRICH**: Attach metadata: `source_file`, `section`, `date`, `tags`, `chunk_index`
5. **EMBED**: Batch chunks → Ollama `/api/embed` (mxbai-embed-large, 1024-dim). Batch size: 64 chunks per request.
6. **STORE**: Write vectors + metadata to LanceDB (upsert by `chunk_id`).

### Incremental Sync Logic

1. Read `last_sync_mtime` from `sync-result.json`
2. Walk vault, compare each file's mtime vs `last_sync_mtime`
3. New/modified files → full chunk pipeline
4. Deleted files → remove chunks by `source_file` from LanceDB
5. Unchanged files → skip entirely

## Security & Privacy

### Four Defense Layers

**Layer 1 — Path Traversal Prevention (Boundary)**
All file reads restricted to `vault_path`. Reject:
- Any path containing `..` components
- Absolute paths not under `vault_path`
- Symlinks resolving outside `vault_path`

Implementation: `path.resolve(vault_path, input)` must start with `vault_path`.

**Layer 2 — Input Sanitization (Clean)**
All vault content treated as untrusted before embedding:
- Strip HTML tags (prevent XSS in `chunk_text`)
- Remove executable code blocks (` ```...``` `)
- Normalize whitespace (prevent injection via formatting)
- Cap `chunk_text` at 2000 chars (prevent oversized payloads)

**Layer 3 — Sensitive Content Guard (Gate)**
Detect and gate sensitive content in search results:
- Health: `#mentalhealth`, `#physicalhealth`, medication, therapy
- Financial: owe, owed, debt, paid, $, spent, spend
- Relations: `#Relations` (configurable section list)

Action: Set `sensitive_detected=true` in response. Agent must confirm before displaying to user. Never transmit sensitive content to external APIs.

**Layer 4 — Local-Only Enforcement (Network)**
All data stays on the local machine:
- Ollama: localhost:11434 only
- LanceDB: local filesystem only
- Config: `"local_only": true` enforced
- Network audit test: verify zero outbound requests

If an external embedding endpoint is configured: require explicit user confirmation, block if sensitive content detected in payload.

### Directory Access Control

**Deny list (default):** `.obsidian`, `.trash`, `zzz-Archive`, `.git`, `.logseq` — always excluded.

**Allow list (optional):** If set, only these directories are indexed. If empty, all directories except deny list are indexed. Example: `["Journal", "Finance"]` → only those two directories.

**Filter logic (Python indexer):**
1. If `allow_dirs` is non-empty → only walk those directories
2. Skip any path matching `deny_dirs` patterns
3. Skip hidden directories (starting with `.`)

**Filter logic (TS plugin):**
- `directory_filter` parameter validates against known directories
- Reject unknown directories → prevent vault structure probing

## Testing Strategy

### Python Unit Tests (pytest)

| Test File | Coverage |
|-----------|----------|
| `test_chunker.py` | Section splitting, sliding window, metadata extraction |
| `test_embedder.py` | Mocked Ollama, batch embedding, error handling |
| `test_vector_store.py` | LanceDB CRUD, upsert, delete by source |
| `test_security.py` | Path traversal, sanitization, sensitive detection |
| `test_indexer.py` | Full pipeline, incremental sync, config handling |

### TypeScript Unit Tests (vitest)

| Test File | Coverage |
|-----------|----------|
| `search.test.ts` | Parameter validation, filter logic, response shape |
| `index.test.ts` | Mode validation, subprocess spawn, progress parsing |
| `memory.test.ts` | Key/value storage, auto-suggest patterns |
| `vault-watcher.test.ts` | Chokidar events, debounce, batching |
| `security-guard.test.ts` | Directory filter validation, sensitive flag |

### Integration Tests

| Test File | Coverage |
|-----------|----------|
| `full_pipeline.test.ts` | Index vault → search → verify results |
| `sync_cycle.test.ts` | Modify file → auto-sync → search updated content |
| `health_state.test.ts` | Stop Ollama → degraded → restart → healthy |
| `openclaw_protocol.test.ts` | Agent calls tools, validates envelope |

### Security Test Suite

| Test File | Coverage |
|-----------|----------|
| `path_traversal.test.ts` | `../`, symlinks, absolute paths, encoded paths |
| `xss_prevention.test.ts` | HTML/script injection in chunk text |
| `prompt_injection.test.ts` | Malicious content in vault notes |
| `network_audit.test.ts` | Verify zero outbound when `local_only=true` |
| `sensitive_content.test.ts` | Pattern detection, flagging, blocking |

### Mock vs Real

| Component | Unit Test | Integration Test |
|-----------|-----------|------------------|
| Ollama embedding | Mocked — fixed 1024-dim vectors | Real — requires Ollama running |
| LanceDB | Real — temp directory, cleaned up | Real — temp directory, cleaned up |
| Obsidian vault | Mocked — fixture markdown files | Real — temp vault with real files |
| Python CLI | Mocked subprocess | Real — actual CLI invocation |
| Chokidar watcher | Mocked events | Real — actual filesystem events |
| OpenClaw agent | N/A | Real — tool call envelope validation |

Integration tests require Ollama but skip gracefully if unavailable.

## CLI Commands (Python Indexer)

| Command | Purpose |
|---------|---------|
| `obsidian-rag index` | Initial full index of the vault (first-time setup) |
| `obsidian-rag sync` | Incremental — only process files modified since last sync |
| `obsidian-rag reindex` | Force full reindex (nuke existing, start fresh) |
| `obsidian-rag status` | Show index health: total docs, last sync time, unindexed files |

## Configuration

Config file resolved cross-platform via `os.path.expanduser` (Linux/macOS: `~/.obsidian-rag/config.json`, Windows: `%APPDATA%/obsidian-rag/config.json`). For development, the config is at `./obsidian-rag/config.json` relative to the project root.

```json
{
  "vault_path": "./KnowledgeVault/Default",
  "embedding": {
    "provider": "ollama",
    "model": "mxbai-embed-large",
    "base_url": "http://localhost:11434",
    "dimensions": 1024
  },
  "vector_store": {
    "type": "lancedb",
    "path": "./obsidian-rag/vectors.lance"
  },
  "indexing": {
    "chunk_size": 500,
    "chunk_overlap": 100,
    "file_patterns": ["*.md"],
    "deny_dirs": [".obsidian", ".trash", "zzz-Archive", ".git", ".logseq"],
    "allow_dirs": []
  },
  "security": {
    "require_confirmation_for": ["health", "financial_debt"],
    "sensitive_sections": ["#mentalhealth", "#physicalhealth", "#Relations"],
    "local_only": true
  },
  "memory": {
    "auto_suggest": true,
    "patterns": {
      "financial": ["owe", "owed", "debt", "paid", "$", "spent", "spend"],
      "health": ["#mentalhealth", "#physicalhealth", "medication", "therapy"],
      "commitments": ["shopping list", "costco", "amazon", "grocery"]
    }
  }
}
```

## Project Structure

```
obsidian-rag-skill/
├── README.md
├── LICENSE
├── .gitignore
├── openclaw.plugin.json
├── package.json
├── tsconfig.json
├── src/
│   ├── index.ts
│   ├── tools/
│   │   ├── search.ts
│   │   ├── index.ts
│   │   ├── status.ts
│   │   └── memory.ts
│   ├── services/
│   │   ├── vault-watcher.ts
│   │   ├── indexer-bridge.ts
│   │   └── security-guard.ts
│   └── utils/
│       ├── config.ts
│       └── lancedb.ts
├── python/
│   ├── pyproject.toml
│   ├── obsidian_rag/
│   │   ├── __init__.py
│   │   ├── cli.py
│   │   ├── indexer.py
│   │   ├── chunker.py
│   │   ├── embedder.py
│   │   ├── vector_store.py
│   │   ├── security.py
│   │   └── config.py
│   └── tests/
│       ├── test_chunker.py
│       ├── test_security.py
│       ├── test_embedder.py
│       ├── test_vector_store.py
│       └── test_indexer.py
├── tests/
│   ├── tools/
│   │   ├── search.test.ts
│   │   ├── index.test.ts
│   │   └── memory.test.ts
│   ├── services/
│   │   ├── vault-watcher.test.ts
│   │   └── security-guard.test.ts
│   ├── integration/
│   │   ├── full_pipeline.test.ts
│   │   ├── sync_cycle.test.ts
│   │   ├── health_state.test.ts
│   │   └── openclaw_protocol.test.ts
│   └── security/
│       ├── path_traversal.test.ts
│       ├── xss_prevention.test.ts
│       ├── prompt_injection.test.ts
│       ├── network_audit.test.ts
│       └── sensitive_content.test.ts
└── docs/
    └── superpowers/
        └── specs/
```

## Publishing

Published to ClawHub as both a skill (SKILL.md) and a plugin package:

```bash
clawhub skill publish ./skill --slug obsidian-rag --version 1.0.0
clawhub package publish santhosh/obsidian-rag
```

Install: Link plugin to `~/.openclaw/extensions/obsidian-rag` or use `openclaw plugins install --link /path/to/obsidian-rag`

## Windows Development Notes

The development environment is Windows 11. The following platform-specific considerations apply:

### Development Vault

The project uses `./KnowledgeVault/Default` as the sample Obsidian vault for development. This vault is located at the project root and contains real note directories including `Journal`, `Daily Life Stuff`, `Podcast Scripts`, `Side Project`, and others.

### Path Handling

- All path operations use `path.resolve()` and `path.join()` with forward slashes for consistency. Node.js and Python both handle forward slashes correctly on Windows.
- Config paths (`vault_path`, `vector_store.path`) support relative paths resolved from the project root (e.g., `./KnowledgeVault/Default`).
- Data directory for production: `%APPDATA%/obsidian-rag/` on Windows, `~/.obsidian-rag/` on Linux/macOS. For development: `./obsidian-rag/` relative to project root.
- Atomic file writes use `.tmp` + rename pattern. On Windows, rename is atomic within the same volume (both temp and target must be on the same drive).

### Filesystem Watcher (chokidar)

- chokidar uses `fs.watch` on Windows which has reliable recursive watch support.
- The 2s debounce and 5s collect window are sufficient for Windows file change events, which may fire multiple events for a single save operation (especially with editors that use atomic save: write to temp file → rename).

### Python CLI on Windows

- The `obsidian-rag` CLI entry point is installed via `pip install -e .` from `python/`. On Windows, the console script is created in the Python Scripts directory.
- `child_process.spawn` on Windows requires `shell: true` if the command is a Python console script, or use `python -m obsidian_rag.cli` as the command instead.
- Recommended: use `python -m obsidian_rag.cli` as the spawn command for cross-platform compatibility.

### Testing on Windows

- Filesystem-based integration tests use `os.tmpdir()` for temporary vault and LanceDB directories.
- Path traversal tests must include Windows-specific vectors: `C:\Windows\System32`, `CON`, `NUL`, `..\\` (backslash variant), UNC paths (`\\server\share`).
- The `deny_dirs` list includes `.obsidian` which is the actual Obsidian config directory name on all platforms.