santhoshj/obsidian-rag
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
28 Commits 3 Branches 0 Tags
model/minimax
Commit Graph

10 Commits

Author SHA1 Message Date
Santhosh Janardhanan
21b9704e21 fix(indexer): use upsert_chunks return value for chunk count 
Previously total_chunks counted from process_file return (num_chunks)
which could differ from actual stored count if upsert silently failed.
Now using stored count returned by upsert_chunks.

Also fixes cli._index to skip progress yields when building result.
 2026-04-12 02:16:19 -04:00
Santhosh Janardhanan
4ab504e87c fix(indexer): write sync-result.json in reindex() not just sync() 
reindex() was consuming the full_index() generator but never calling
_write_sync_result(), leaving sync-result.json stale while CLI output
showed correct indexed_files/total_chunks.
 2026-04-12 01:49:43 -04:00
Santhosh Janardhanan
34f3ce97f7 feat(indexer): hierarchical chunking for large sections 
- Section-split first for structured notes
- Large sections (>max_section_chars) broken via sliding-window
- Small sections stay intact with heading preserved
- Adds max_section_chars config (default 4000)
- 2 new TDD tests for hierarchical chunking
 2026-04-11 23:58:05 -04:00
Santhosh Janardhanan
d946cf34e1 fix(indexer): truncate chunks exceeding Ollama context window 2026-04-11 23:12:13 -04:00
Santhosh Janardhanan
406dbf24a9 fix: use cwd instead of __file__ for config path resolution 
Fixes config file not found error when installed via pip
 2026-04-11 21:02:47 -04:00
Santhosh Janardhanan
e15e4ff856 fix: add configSchema to openclaw.plugin.json, add search CLI command, fix total_docs stat 
- Add required configSchema to openclaw.plugin.json for OpenClaw plugin discovery
- Add search command to CLI with --limit, --dir, --from-date, --to-date, --tags filters
- Fix get_stats() to properly count unique docs (was returning 0 for non-null values)
- Remove hardcoded max_results default of 5; search now returns all results by default
- Update INSTALL.md and design docs with correct OpenClaw extension path instructions
 2026-04-11 20:01:09 -04:00
Santhosh Janardhanan
4e991c329e Security review fixes 2026-04-11 15:11:07 -04:00
Santhosh Janardhanan
208531d28d Sprint 0-2: TS plugin scaffolding, LanceDB utils, tooling updates 
- Add index-tool.ts command implementation
- Wire lancedb.ts vector search into plugin
- Update src/tools/index.ts exports
- Bump package deps (ts-jest, jest, typescript, lancedb)
- Add .claude/settings.local.json

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-11 13:24:26 -04:00
Santhosh Janardhanan
83a54b2af6 fix: stable chunk_id eliminates duplicate rows on re-index 
UUID-based chunk_ids caused merge_insert to treat same content as new rows
on each re-index run. Now uses SHA1(content_hash + index) for deterministic
chunk_ids — same section/text always produces same chunk_id.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-11 13:23:17 -04:00
Santhosh Janardhanan
5c281165c7 Sprint 0-1: Python indexer, TS plugin scaffolding, and test suite 
## What's new

**Python indexer (`python/obsidian_rag/`)** — full pipeline from scan to LanceDB:
- `config.py` — JSON config loader with cross-platform path resolution
- `security.py` — path traversal prevention, HTML stripping, sensitive content detection, dir allow/deny lists
- `chunker.py` — section-split for journal entries (date-named files), sliding-window for unstructured notes
- `embedder.py` — Ollama `/api/embeddings` client with batched requests and timeout/error handling
- `vector_store.py` — LanceDB schema, upsert (merge_insert), delete, search with filters, stats
- `indexer.py` — full/sync/reindex pipeline orchestrator with progress yields
- `cli.py` — `index | sync | reindex | status` CLI commands

**TypeScript plugin (`src/`)** — OpenClaw plugin scaffold:
- `utils/` — config loader, TypeScript types, response envelope factory, LanceDB client
- `services/` — health state machine (HEALTHY/DEGRADED/UNAVAILABLE), vault watcher with debounce/batching, indexer bridge (subprocess spawner)
- `tools/` — 4 tool stubs: search, index, status, memory_store (OpenClaw wiring pending)
- `index.ts` — plugin entry point with health probe + vault watcher startup

**Config** (`obsidian-rag/config.json`, `openclaw.plugin.json`):
- 627 files / 3764 chunks indexed in dev vault

**Tests: 76 passing**
- Python: 64 pytest tests (chunker, security, vector_store, config)
- TypeScript: 12 vitest tests (lancedb client, response envelope)

## Bugs fixed

- LanceDB `tags` column filter: `LIKE '%tag%'` → `list_contains(tags, 'tag')` (List<String> column)
- LanceDB JS `db.list_tables()` returns `ListTablesResponse` object, not plain array
- LanceDB JS result score field: `_score` → `_distance`
- TypeScript regex literal with unescaped `/` in path-resolve regex
- Python: `create_table_if_not_exists` identity check → name comparison

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-10 22:56:50 -04:00