JOURNEY.md

kh_Assistant — Architecture & Journey Record

Origins

Local-first AI assistant for non-technical users. Privacy-focused, zero cloud dependency.

Phase 1 — Ollama

• Why: Quickest path to local LLM inference (REST API, no compilation).
• Difficulties: External server process to manage. ollama pull blocked the UI (subprocess.run). No quantization control — Ollama decided, not the app. Model discovery tied to Ollama registry.
• Architecture: REST client in ai_manager.py, background threads for UI, AGENTS.md mentor mode for non-technical UX.

Phase 2 — llama.cpp (llama-cpp-python, in-process)

• Why migrate: CPU performance. Explicit GGUF quantization. Self-contained app.
• Why in-process over server: Advantage — no external process, lower latency. Disadvantage — Llama() init blocks 1-3s (mitigated with loading states). Advantage — full control over threads, context, quantization.
• Difficulties: No ollama pull equivalent → built model_downloader.py for HF. GGUF filenames are inconsistent → heuristic parser for quant/params. Model loading time → loading indicator.
• Architecture: Scanning ./models/*.gguf, loading/unloading models in background threads, huggingface_hub for download.

RAM-Aware Suggestion System

• Why: 3 GB target machine.
• Formula: model_budget = total_ram - max(1.5, total_ram * 0.2).
• Ranker: Priority families (Llama 3.2 → Phi-3 → Qwen → TinyLlama) filtered by file size ≤ budget.
• UI: Shows RAM, best-fit recommendation, warnings on over-budget selection.

Architectural Constants

• PyQt6 single-window (no menus/trays).
• GIL-based direct Qt updates from daemon threads (not refactored).
• model_downloader.py separate from ai_manager.py (separation of concerns).
• ./models/ directory as model store (no DB).