diff --git a/PRD Drafts/ai-agent-knowledge-management.md b/PRD Drafts/ai-agent-knowledge-management.md new file mode 100644 index 0000000..c9f8471 --- /dev/null +++ b/PRD Drafts/ai-agent-knowledge-management.md @@ -0,0 +1,319 @@ +# AI Agent Knowledge Management System PRD + +**Status:** Draft | **Author:** Artemis (AI Foreman) | **Date:** 2026-06-04 + +--- + +## 1. Executive Summary + +Artemis (Hermes Agent) generates persistent memory (MEMORY.md, USER.md), skills (SKILL.md files), operational logs, and PRD drafts. These `.md` files currently live in `~/.hermes/` and `~/documentation/` on the Artemis host. Commander Bobby needs a self-hosted knowledge management system with a web UI to: + +1. **Review and organize** Artemis's memory/skill files outside the agent context +2. **Bidirectionally sync** edits from the UI back to the filesystem +3. **AI-assisted review** of memory files for optimization (compaction, deduplication, relevance scoring) +4. **Serve as the canonical knowledge base** for Iron Legion operational documentation + +This PRD compares four candidates—TriliumNext Notes, Joplin Server, Obsidian, and Google NotebookLM—against these requirements and recommends an architecture. + +--- + +## 2. Requirements Recap + +| ID | Requirement | Priority | +|---|---|---| +| R1 | Self-hosted (Docker, LXC, or VM on Proxmox) | **Hard** | +| R2 | Web-based UI for reading/editing notes | **Hard** | +| R3 | Bidirectional sync with filesystem Markdown files | **Hard** | +| R4 | AI-powered review/analysis of notes | **Medium** | +| R5 | Markdown-native or robust Markdown import/export | **Hard** | +| R6 | REST API for automated scripting/cron integration | **Medium** | +| R7 | Full-text search across all notes | **Medium** | +| R8 | Note versioning / revision history | **Low** | +| R9 | Team/collaborative access (Commander Bobby only for now) | **Low** | + +--- + +## 3. Hardware Investigation (Crucial Finding) + +Before evaluating candidates, a critical discovery about the Proxmox cluster hardware must be addressed: + +### 3.1 The Problem +- **MK33, MK34, MK39** (PVE hosts) run Intel **N100 CPUs** with **4 cores / 4 threads each** +- Proxmox already reports **CPU utilization**: MK33 at 18.50%, MK34 at 32.42%, MK39 at 19.89% +- A medium-weight Node.js app like TriliumNext would consume 10-20% of a single node's total capacity under load +- MK42 has an Intel **J4125** (4 cores, 2.0GHz) which is **even weaker** — marginal but usable with strict cgroup limits +- The bare-metal fleet (MK7, Artemis host) have stronger CPUs but are already purpose-assigned +- **TrueNAS SCALE 25.10.2** (Beelink mini PC): 4-core CPU, 11GB RAM (3.5GB available), load 0.09 — substantial headroom confirmed via live check + +### 3.2 The Resource Model: LXC vs VM + +**Critical correction:** Proxmox LXC containers use **cgroups v2**, not VM-style resource reservation. CPU is **opportunistic and shared** — an LXC only consumes CPU when actively processing, and the host kernel scheduler dynamically balances contention between containers. + +Key LXC resource controls: +| Control | Effect | +|---|---| +| `cores` | Number of visible CPU cores (scheduling masks) | +| `cpulimit` | **Hard ceiling** — the LXC cannot exceed this fraction of host CPU | +| `cpu.weight` | **Relative priority** under contention (default 100, lower = less priority) | +| `memory` | **Hard limit** — kernel OOM-kills processes if exceeded | + +This means an idle TriliumNext LXC consumes **near-zero host CPU**, and an active one respects its ceiling regardless of other workloads. The earlier concern about "adding to an already-loaded node pushes it to 50%+" is **mitigated** when using `cpulimit` caps. + +### 3.3 Risk Assessment (Revised) + +With proper Proxmox LXC cgroup limits (`cores: 2`, `cpulimit: 2`, `memory: 512M`, `cpu.weight: 128`): + +| Deployment Target | Risk | Notes | +|---|---|---| +| **TrueNAS SCALE Docker** | ✅ Low | 4-core, 3.5GB free RAM, load 0.09. Preferred target | +| **MK33 PVE LXC** | ⚠️ Medium | 18% baseline, N100. Manageable with cpulimit=1.5 | +| **MK34 PVE LXC** | ⚠️ Medium | 32% baseline, N100. Needs cpulimit=1.0 | +| **MK39 PVE LXC** | ⚠️ Medium | 20% baseline, N100. Manageable with cpulimit=1.5 | +| **MK42 PVE LXC** | ⚠️ High | J4125 2.0GHz, weakest CPU. cpulimit=0.5 minimum | +| **Artemis host** | ⚠️ Medium | Stronger CPU but already runs Hermes agent (latency-sensitive) | +| **MK7** | ❌ | Purpose-assigned: Paperclip + PostgreSQL, no spare capacity | + +### 3.4 Recommended Path +**Primary: TrueNAS SCALE 25.10.2 Docker Compose** (confirmed 4-core, 11GB RAM, 0.09 load — ample headroom). **Fallback: PVE LXC on MK34 or MK39** with strict cgroup limits (cpulimit=1.0, memory=512M). Both approaches are viable; TrueNAS is preferred for CPU headroom. + +--- + +## 4. Candidate Evaluation + +### 4.1 TriliumNext Notes + +**Current state:** Active community fork of the archived `zadam/trilium`. Latest release `v0.103.0` (2026-05-13). 36,300+ GitHub stars. Commits daily. + +| Criterion | Status | Details | +|---|---|---| +| **Self-hosted** | ✅ | Official Docker image `triliumnext/trilium:latest`, AMD64+ARM64 | +| **Web UI** | ✅ | Full WYSIWYG editor, tree-based hierarchy, relation maps, mind maps | +| **Bidirectional MD sync** | ⚠️ | Bulk Markdown import/export supported, but no live filesystem watcher. Requires cron + ETAPI scripting | +| **AI integration** | ⚠️ | No built-in AI. JavaScript scripting engine can call external APIs (Ollama). Community themes/scripts exist | +| **REST API** | ✅ | **ETAPI** (External REST API) — OpenAPI-documented, supports CRUD on notes, search, attributes, import/export. Authenticated via token | +| **Markdown support** | ✅ | Native Markdown import/export; WYSIWYG editor auto-formats Markdown | +| **Full-text search** | ✅ | Built-in | +| **Versioning** | ✅ | Per-note revision history | +| **Performance** | ⚠️ | Node.js app. Idle ~150MB RAM, moderate CPU. Scales to 100K+ notes. **May strain N100/J4125 CPUs** | +| **Maintenance risk** | ⚠️ | TriliumNext is a community fork; original author archived `zadam/trilium`. Sync version incompatibility with old instances | + +**Deployment:** Minimal Docker Compose: +```yaml +services: + trilium: + image: triliumnext/trilium:v0.103.0 + ports: + - "8080:8080" + volumes: + - ~/trilium-data:/home/node/trilium-data + environment: + - TRILIUM_DATA_DIR=/home/node/trilium-data +``` + +**Verdict:** Best fit from the candidate list. Self-hosted, API-driven, Markdown-capable. AI gap is bridgeable via scripting + Ollama. Performance is manageable with cgroup limits (LXC) or on TrueNAS Docker (preferred, 4-core/11GB/0.09 load). + +--- + +### 4.2 Joplin Server + +**Current state:** Actively maintained by `laurent22`. Stable. Docker image `joplin/server:latest`. Sync server for Joplin desktop/mobile clients. + +| Criterion | Status | Details | +|---|---|---| +| **Self-hosted** | ✅ | Official Docker image, PostgreSQL backend, well-documented | +| **Web UI** | ❌ | Joplin Server is a **sync backend only**. The web client is minimal (basic note listing). Primary editing requires Joplin desktop/mobile app | +| **Bidirectional MD sync** | ⚠️ | Joplin's sync protocol is its own format (not raw MD). Export to raw Markdown requires the desktop app. Server API can push/pull notes in Joplin's JSON format | +| **AI integration** | ❌ | No server-side AI. Desktop plugins exist (e.g., "Note Overview" with ChatGPT) but require running desktop client | +| **REST API** | ✅ | Joplin Server API (beta), supports note CRUD, but less feature-rich than Trilium's ETAPI | +| **Markdown support** | ✅ | Joplin notes are Markdown internally, but stored in a SQL database, not as raw `.md` files | +| **Full-text search** | ✅ | Via PostgreSQL FTS in Joplin Server 3.x | +| **Versioning** | ✅ | Note history API available | +| **Maintenance risk** | ✅ | Low. Actively maintained, stable releases. Backed by a sole developer but well-established | + +**Verdict:** Excellent sync server, **not a web-based knowledge manager**. If Bobby wants to use Joplin desktop/mobile as his primary interface and sync through the server, this works. But requirement R2 (web-based UI) is essentially unmet. No AI pathway without desktop client. + +--- + +### 4.3 Obsidian + +**Current state:** Proprietary Electron desktop app. Extremely popular (100K+ GitHub stars for plugin API). No server edition. + +| Criterion | Status | Details | +|---|---|---| +| **Self-hosted** | ❌ | **No.** Desktop app only. Obsidian Sync ($5/mo) and Publish ($10/mo) are proprietary cloud services. No official Docker image or server mode | +| **Web UI** | ❌ | No. Obsidian Publish creates read-only static websites, not editable | +| **Bidirectional MD sync** | ⚠️ | Vaults are plain `.md` folders on disk — trivially scriptable. But editing requires the desktop app open. Community `obsidian-local-rest-api` plugin provides REST API **but only when the desktop app is running** | +| **AI integration** | ⚠️ | Community plugins exist (Smart Connections, Copilot, etc.). Local LLM integration possible via plugins. But again: desktop app must be running | +| **REST API** | ⚠️ | Only via community `obsidian-local-rest-api` plugin + running desktop app. Not headless | +| **Markdown support** | ✅ | Gold standard. Native Markdown with extensive extended syntax | +| **Versioning** | ✅ | Via Obsidian Sync ($5/mo) or external git | +| **Maintenance risk** | ⚠️ | Proprietary. Future licensing changes could affect workflows. Community heavy but core is closed-source | + +**Verdict:** The best Markdown editor on the market. **Entirely wrong architecture for this use case.** Cannot run headless on a server. If Bobby wants to manually review/edit memory files on his desktop, Obsidian is excellent—but it does not meet the self-hosted web service requirement. + +--- + +### 4.4 Google NotebookLM + +**Current state:** Google cloud service. Upload documents → AI-powered Q&A, summaries, audio overviews. No self-hosted edition. + +| Criterion | Status | Details | +|---|---|---| +| **Self-hosted** | ❌ | **No.** Pure Google SaaS | +| **Web UI** | ✅ | Excellent web interface, but cloud-only | +| **AI support** | ✅ | **Best-in-class.** Native RAG-powered summarization, Q&A, audio overviews | +| **Markdown support** | ⚠️ | Uploads supported but NotebookLM is document-centric, not a Markdown editor | +| **REST API** | ❌ | No public API for NotebookLM | +| **Bidirectional sync** | ❌ | Upload only. No export back to filesystem | +| **Data sovereignty** | ❌ | All data lives on Google servers | + +**Verdict:** Wrong architecture for this use case. No self-hosting. No bidirectional sync. No API. + +**However:** NotebookLM's AI capabilities are exactly what Bobby wants for reviewing memory files. See Section 5 for open-source RAG alternatives. + +--- + +## 5. AI-Powered Review: Open-Source RAG Layer + +NotebookLM is unavailable for self-hosting, but its core functionality (upload documents → ask questions → get summaries) can be replicated with open-source tools. This would be a **separate service** that ingests Markdown exports from the knowledge base. + +### 5.1 Candidate RAG Tools + +| Tool | Stars | Deployment | Notes | +|---|---|---|---| +| **AnythingLLM** | 35K+★ | Docker, single binary | All-in-one: document ingestion, RAG chat, multi-model (Ollama, OpenAI, etc.). Agent support. Best fit for "upload and chat" workflow | +| **Dify** | 60K+★ | Docker Compose | Full AI application platform. RAG pipeline builder, chat UI, workflow automation. Overkill but powerful | +| **PrivateGPT** | 60K+★ | Docker, Python | Privacy-focused document Q&A. Simpler than Dify. Good for batch processing docs | +| **Open WebUI** | 65K+★ | Docker | Ollama-native chat UI with RAG and document upload. Already running on the fleet? | +| **Flowise** | 35K+★ | Docker, Node.js | Low-code LLM workflow builder. Drag-and-drop RAG pipeline. Good for custom ingestion chains | + +### 5.2 Recommended AI Layer: AnythingLLM + +**Why:** Single Docker container, natively supports Ollama (already in the fleet), built-in document ingestion with chunking + embedding, chat-based review, multi-user. Lightweight enough to co-reside with the knowledge base or run on a separate node. + +**Architecture:** TriliumNext exports `.md` files via cron → AnythingLLM ingests them → Bobby chats with his memory files via AnythingLLM's UI → Insights inform manual edits in TriliumNext. + +--- + +## 6. Comparative Matrix + +| Criterion | TriliumNext | Joplin Server | Obsidian | NotebookLM | +|---|---|---|---|---| +| Self-hosted Docker | ✅ | ✅ | ❌ | ❌ | +| Web UI (edit) | ✅ | ❌ | ❌ | ✅ (cloud) | +| Bidirectional MD sync | ⚠️ scriptable | ⚠️ via desktop | ❌ | ❌ | +| REST API | ✅ ETAPI | ✅ (beta) | ❌ | ❌ | +| AI integration | ⚠️ scriptable | ❌ | ⚠️ plugins | ✅ native | +| Markdown-native | ✅ import/export | ✅ internal | ✅ gold standard | ❌ | +| Full-text search | ✅ | ✅ | ✅ | ✅ | +| Versioning | ✅ | ✅ | ✅ | ✅ | +| Maintenance risk | ⚠️ community fork | ✅ stable | ⚠️ proprietary | ❌ Google SaaS | +| Hardware fit | ⚠️ Node.js on N100 | ⚠️ PostgreSQL needed | N/A | N/A | + +--- + +## 7. Recommended Architecture + +### 7.1 Primary Recommendation: TriliumNext on TrueNAS Docker + +**Deploy:** TriliumNext as a Custom Docker Compose app on TrueNAS SCALE 25.10.2 (hardware pre-check pending). + +**Rationale:** +- Only candidate from Bobby's list that meets all hard requirements (self-hosted web UI, Markdown support, REST API) +- Active community fork with daily commits +- Scripting engine bridges the AI gap +- TrueNAS has stronger hardware than PVE N100 nodes + +### 7.2 AI Layer (Phase 2): AnythingLLM alongside TriliumNext + +1. Cron job exports Trilium notes as `.md` to a shared volume +2. AnythingLLM watches the volume and ingests new/changed docs +3. Bobby uses AnythingLLM's chat UI for AI-powered memory review +4. Insights from AI review → manual edits in TriliumNext web UI +5. TriliumNext edits → cron syncs back to Artemis `~/.hermes/` filesystem + +### 7.3 Bidirectional Sync Flow + +``` +Artemis ~/.hermes/memory/ ──cron export──→ TriliumNext (web UI) + │ + Bobby reviews/edits + │ +TriliumNext ←──cron pull── Bobby's edits saved + │ + └──cron export──→ ~/.hermes/memory/ (Artemis reads updated files) +``` + +### 7.4 Fallback: Joplin Server + +If TriliumNext proves too heavy for available hardware: +- Joplin Server on a PVE node (or TrueNAS Docker) +- Bobby uses Joplin desktop app for editing (not web UI) +- Sync via Joplin's native protocol +- Markdown export via Joplin CLI → Artemis filesystem + +--- + +## 8. Deployment Plan (Proposed) + +### Phase 1: TrueNAS Hardware Verification +1. SSH to TrueNAS — check CPU load, available RAM, Docker compatibility +2. Verify TrueNAS SCALE 25.10.2 supports privileged Custom Docker Compose apps +3. Test-deploy TriliumNext with resource limits (`cpus: 1.0`, `memory: 512M`) +4. Measure idle CPU/RAM consumption + +### Phase 2: TriliumNext Deployment +1. Deploy TriliumNext via TrueNAS Docker Compose UI +2. Configure ETAPI token for automation +3. Import existing `~/.hermes/memory/` and `~/.hermes/skills/` as note trees +4. Set up cron for bidirectional `.md` export/import between Artemis host and TriliumNext + +### Phase 3: AI Integration +1. Deploy AnythingLLM as a companion Docker Compose app on TrueNAS +2. Configure Ollama backend (point to fleet Ollama instance) +3. Create ingestion pipeline: Trilium export → shared volume → AnythingLLM workspace +4. Test AI-assisted review workflow + +### Phase 4: Productionize +1. Document sync schedule and retention policy +2. Add healthcheck monitoring +3. (Stretch) Explore real-time sync via filesystem watcher instead of cron + +--- + +## 9. Open Questions + +1. **TrueNAS Docker suitability?** Can TrueNAS SCALE 25.10.2's Docker Compose app platform run a Node.js app with moderate CPU/RAM usage without impacting NAS performance? + +2. **Alternative deployment target?** If TrueNAS is unsuitable and PVE nodes are too constrained, is there a bare-metal node with spare capacity? MK7 is assigned to Paperclip + PostgreSQL. Artemis host could run TriliumNext but adds desktop workload to the agent's host. + +3. **NotebookLM "light"?** If AI review is the primary goal and web editing is secondary, would a minimal RAG setup (AnythingLLM + direct Markdown file ingestion, no knowledge base UI) suffice for Phase 1? + +4. **Sync frequency?** How often should Artemis export memory to TriliumNext, and how often should TriliumNext edits sync back? 5-minute cron? Hourly? Manual trigger? + +5. **Scope of "bidirectional"?** Can Artemis **read** TriliumNext as an authoritative source for memory, or is TriliumNext purely a review/edit sandbox where changes are manually promoted? + +--- + +## 10. Decision Required + +**Bobby to decide:** + +| Decision | Options | +|---|---| +| **Primary tool** | A) TriliumNext on TrueNAS (recommended) — B) Joplin Server + desktop app — C) Minimalist: plain Markdown + AnythingLLM for AI review only | +| **Deployment target** | A) TrueNAS SCALE Docker — B) PVE node (despite CPU concerns) — C) Artemis host — D) MK7 | +| **AI layer** | A) AnythingLLM (recommended) — B) No AI yet, manual review only — C) Open WebUI if already in fleet | +| **Sync direction** | A) Bidirectional (read + write) — B) Read-only archive for review — C) Artemis treats TriliumNext as source of truth | + +--- + +## Appendix A: Source References + +- TriliumNext GitHub: `https://github.com/TriliumNext/Trilium` (36.3K ★, active) +- TriliumNext Docker Hub: `triliumnext/trilium` (2.6M pulls) +- Joplin Server: `https://github.com/laurent22/joplin` (47K ★) +- Obsidian Local REST API: `https://github.com/coddingtonbear/obsidian-local-rest-api` (2.4K ★) +- AnythingLLM: `https://github.com/Mintplex-Labs/anything-llm` (35K+ ★) +- TrueNAS SCALE 25.10.2 release notes: Apps migrated to Docker Compose from Kubernetes +- Research conducted: 2026-06-04 via web scraping of official sites, GitHub APIs, and Docker Hub diff --git a/fleet/admin-cheat-sheet.md b/fleet/admin-cheat-sheet.md index 8ce0f03..3d32087 100644 --- a/fleet/admin-cheat-sheet.md +++ b/fleet/admin-cheat-sheet.md @@ -19,6 +19,15 @@ | Homepage | https://home.ai.home | Service Portal | | Prometheus | https://prometheus.ai.home | Metrics DB | | Authelia | https://auth.ai.home | SSO Portal | +| Trilium (ZimaOS) | https://trilium.nb.mslnath.me | Personal Knowledge Base | + +--- + +## Standalone Nodes (No Ansible) + +|| Hostname | LAN IP | Domain | Role | Beszel | +||----------|--------|--------|------|--------| +| igor (MK-38) | 192.168.26.130 | trilium.nb.mslnath.me | ZimaOS NAS, Trilium, ARR Media Stack | ✅ | --- @@ -51,7 +60,7 @@ | mark5.ai.home | 192.168.6.5 | TBD | TBD | | shield.ai.home | 192.168.10.15 | - | iVentoy PXE Server | | artemis.ai.home | 192.168.15.182 | 100.100.97.18 | Discord Gateway | -| igor.ai.home | 192.168.10.211 | TBD | ZimaOS NAS (Mark XXXVIII) | +| igor.ai.home | 192.168.26.130 | TBD | ZimaOS NAS, Trilium, ARR Media Stack, Beszel agent | > **Note:** `igor.ai.home` is a separate physical node (ZimaOS NAS). Do NOT confuse with any armor codename. @@ -184,7 +193,7 @@ All Proxmox auto-install ISOs are **remastered** with: | nebuchadnezzar.ai.home | 192.168.192.24 | | shield.ai.home | 192.168.10.15 | | artemis.ai.home | 192.168.15.182 | -| igor.ai.home | 192.168.10.211 | +| igor.ai.home | 192.168.26.130 | --- @@ -228,7 +237,7 @@ Portable Host (F.R.I.D.A.Y.) | NEO | Nebuchadnezzar | Nextcloud/Gitea | | SHIELD | - | PXE Server | -> **Note:** `Igor` is **MK-38** (ZimaOS NAS at 192.168.10.211). It is NOT MK-34. +> **Note:** `Igor` is **MK-38** (ZimaOS NAS at 192.168.26.130). It is NOT MK-34. ---