Feature/102 context aware chunking via llm

.env.example

@@ -48,5 +48,11 @@ OPENAI_VISION_MODEL=
 # --- chunk settings ---
 CHUNK_SIZE=
 CHUNK_OVERLAP=
+# Character-count ceiling (proxy for a token limit — no tokenizer in this
+# project) for text/PDF content sent to the LLM-based context-aware chunker
+# (POST /api/v1/ingest, semantic_boundaries/contextualize flags). Above this,
+# ingestion falls back to the plain chunk_text strategy. Leave empty for the
+# default (24000).
+CONTEXT_AWARE_CHUNKING_MAX_CHARS=
 
 

.gitignore

@@ -220,4 +220,6 @@ __marimo__/
 .streamlit/secrets.toml
 
 
-data/
+data/
+
+opencode.json

AGENTS.md

@@ -0,0 +1,94 @@
+# SprintStart AI Service
+
+FastAPI service that ingests project artifacts, indexes them via RAG, and
+answers questions / generates personalized onboarding paths.
+
+## Setup
+
+```bash
+uv sync
+cp .env.example .env   # fill in values
+uv run python -m src.main   # dev server on :8000, docs at /docs
+```
+
+Docker: `docker-compose up --build` (overrides `OLLAMA_BASE_URL` to
+`http://host.docker.internal:11434` automatically).
+
+Production Docker CMD (Dockerfile): `uvicorn api.app:app --app-dir src`.
+
+## Commands
+
+| Purpose | Command |
+|---|---|
+| Run tests | `uv run pytest` |
+| Tests + coverage | `uv run pytest --cov=src --cov-report=term-missing` |
+| Integration tests (needs Ollama) | `uv run pytest -m integration` |
+| Lint | `uv run ruff check .` |
+| Format check | `uv run ruff format --check .` |
+| Auto-format | `uv run ruff format .` |
+| Type-check | `uv run pyright src/` |
+
+Run **lint → format check → type-check → tests** before considering a change
+done. CI enforces this order:
+`gitleaks → quality (ruff lint + ruff format check + pyright) → pytest`.
+
+## Branch protection
+
+PRs to `main` must come from `dev` or `hotfix/*`. Push to `main`/`dev`
+triggers Docker image publish to `ghcr.io/sprintstartproject/sprintstart-ai`.
+
+## Architecture
+
+`src/` layout:
+
+- **`api/`** — FastAPI app (`app.py`), DI (`dependencies.py`), `schemas.py`,
+  SSE helpers (`sse.py`), and route modules under `routes/`.
+- **`agents/`** — `Agent` base class (`base.py`) with a `decision_role` /
+  `answer_system` prompt pair and `max_steps` cap (default 5).
+  `OrchestratorAgent` decides whether to delegate to sub-agents or answer
+  directly. Tools live in `agents/tools/`, registered via `ToolRegistry`.
+  `ChatOrchestrator` wraps the agent run into SSE events for `/api/v1/chat`.
+- **`ingestion/`** — Per-filetype parsers (`text_parser`, `pdf_parser`,
+  `code_parser`, `image_parser`) behind `parser.py`, then `chunker.py` and
+  `metadata_store.py`.
+- **`rag/`** — `retriever.py`, `hybrid.py` (BM25 + vector hybrid retrieval
+  with RRF fusion), `citation.py`, `query_expansion.py`, `prompt.py`.
+- **`llm/`** — `LLMClient` protocol (`base.py`) with implementations:
+  `ollama_client.py`, `openai_client.py`, `anthropic_client.py`.
+  `SplitLLMClient` lets chat and embeddings use different backends
+  (`LLM_BACKEND` vs `EMBED_BACKEND`).
+- **`store/`** — `VectorStore` protocol (`base.py`), `chroma_store.py`.
+- **`onboarding/`** — Deterministic staged pipeline (not agentic):
+  `select → filter → retrieve → synthesize → validate → emit`, yielding
+  `StageProgress` markers. Blueprints are owned by the backend and passed in
+  on each request — the service is stateless.
+
+## Conventions
+
+- `src` is on `pythonpath` for tests — import as `from agents.base import
+  Agent`, **not** `from src.agents...`.
+- Don't assume Ollama-only: check `llm/base.py`'s `LLMClient` protocol when
+  touching LLM calls. Backend is configurable per deployment via
+  `LLM_BACKEND` / `EMBED_BACKEND`.
+- New sub-agents should follow the `SynthesisAgent` / `OrchestratorAgent`
+  pattern (`Agent` base + `ToolRegistry`/`AgentTool`), not a bespoke loop.
+- `AGENT_DEBUG=1` logs each agent's reasoning (LLM text + tool calls) to
+  stderr — useful when debugging agent behavior.
+- The onboarding pipeline is intentionally deterministic/staged rather than
+  agentic: a bad LLM output degrades to a blueprint-only path instead of
+  breaking. Keep that property when modifying it.
+- `data/` is gitignored — don't commit local ChromaDB state or test fixtures
+  there.
+
+## Tests
+
+`tests/` mirrors `src/`. Key utilities in `conftest.py`:
+
+- `llm_required` / `vision_required` markers skip tests when Ollama is
+  unreachable or vision model unconfigured.
+- `parse_sse_events()` parses SSE streams into `dict` lists.
+- `clear_dependency_caches` (autouse fixture) resets `lru_cache` on
+  `get_llm`, `get_store`, `get_ingestion_metadata_store` before each test.
+
+Reuse the fakes in `tests/stubs/llm.py` (`StubLLMClient`, `ScriptedLLMClient`)
+and `tests/stubs/store.py` (`StubVectorStore`) instead of hand-rolling mocks.

README.md

@@ -63,13 +63,14 @@ The service runs on port `8000`.
 | `AGENT_DEBUG` | `0` | When set to a truthy value, logs each agent's reasoning step (LLM text and tool calls) to stderr. Disabled by default and for `0`/`false`/`no`/`off`/empty. |
 | `CHUNK_SIZE` | `512` | Maximum number of characters per chunk |
 | `CHUNK_OVERLAP` | `64` | Number of characters reused between consecutive chunks to preserve context when splitting large chunks |
+| `CONTEXT_AWARE_CHUNKING_MAX_CHARS` | `24000` | Character-count ceiling (a proxy for a token limit — there is no tokenizer in this project) for text/PDF content sent to the LLM-based context-aware chunker. Above this, ingestion falls back to the plain `chunk_text` strategy. See [Context-aware chunking](#context-aware-chunking). |
 
 ## API Endpoints
 
 | Method | Path | Description |
 |---|---|---|
 | `GET` | `/api/v1/health` | Reports service health including LLM backend status. Returns `503` if Ollama is unreachable. |
-| `POST` | `/api/v1/ingest` | Parses, chunks, and embeds a document and stores it in the vector store. Re-ingesting the same `artifact_id` replaces existing chunks. Supports text files and images (send image content as base64; requires `OLLAMA_VISION_MODEL`). |
+| `POST` | `/api/v1/ingest` | Parses, chunks, and embeds a document and stores it in the vector store. Re-ingesting the same `artifact_id` replaces existing chunks. Supports text files and images (send image content as base64; requires `OLLAMA_VISION_MODEL`). Text/PDF content can optionally be chunked using the LLM-based [context-aware chunker](#context-aware-chunking) (opt-in, off by default). |
 | `POST` | `/api/v1/chat` | Retrieves relevant chunks and streams a generated answer as Server-Sent Events (SSE). |
 | `POST` | `/api/v1/title` | Generates a short descriptive title from a user prompt. |
 | `POST` | `/api/v1/onboarding/path` | Generates a personalized onboarding path (SSE). Blueprints are passed in by the backend on each request — the service is stateless. |
@@ -87,6 +88,26 @@ The `/api/v1/chat` endpoint streams newline-delimited JSON events:
 | `done` | Signals the end of the stream |
 | `error` | Emitted on failure instead of the above |
 
+## Context-aware chunking
+
+`POST /api/v1/ingest` can optionally chunk text and PDF content using an LLM instead of the plain character-length `chunk_text` strategy (`ingestion/chunker.py`). This is opt-in per request via two independently toggleable flags on `IngestRequest`, both `false` by default:
+
+| Field | Default | Description |
+|---|---|---|
+| `semantic_boundaries` | `false` | Let the LLM choose chunk boundaries based on topic shifts / section boundaries instead of accumulating paragraphs up to `CHUNK_SIZE`. |
+| `contextualize` | `false` | Let the LLM selectively prepend a short situating "context block" (Anthropic-style *Contextual Retrieval*) to chunks that would otherwise lack context; self-contained chunks are left untouched. |
+
+Both flags share a single LLM call (see `ingestion/context_aware_chunker.py`). Leaving both at `false` (the default) skips the LLM call entirely and behaves exactly like the legacy chunker — existing callers are unaffected unless they explicitly opt in. When either is enabled, the strategy still always falls back to `chunk_text` — no request ever fails because of it — when:
+
+- the content exceeds `CONTEXT_AWARE_CHUNKING_MAX_CHARS`,
+- the LLM backend is unreachable, or
+- the LLM response is missing, malformed, or fails validation (e.g. out-of-range boundaries).
+
+Resulting chunks carry extra metadata so context/overlap portions of a chunk's content are identifiable: `has_context_block`, `context_block_range`, `has_overlap`, `overlap_range` (character ranges as `"start:end"` strings, empty when not applicable).
+
+**Scope:** only text and PDF content are affected; code and image parsing are unchanged. For PDFs, the LLM is invoked **per page** (pages are already parsed independently), so semantic boundaries and context blocks are page-scoped, not whole-document-scoped. The batch endpoint `POST /api/v1/ingest/sync` (GitHub run sync) does not expose these flags and always uses the plain chunker — changing that would mean changing the batch wire contract with the backend, which is a separate decision.
+
+
 ## AI-proposed onboarding blueprints
 
 Onboarding paths are assembled from versioned **blueprints** scoped by `global` or `area:<name>`. Blueprints carry a `source`: `authored` (human-written) or `generated` (drafted by the AI from the ingested corpus). The backend owns blueprint persistence, versioning, and rollback — the AI service is stateless and only returns generated data.

src/api/routes/ingest.py

@@ -148,7 +148,15 @@ def ingest(
         metadata_store.mark_failed(body.artifact_id, detail, _utc_now())
         raise HTTPException(status_code=413, detail=detail)
 
-    parsed_chunks = parse(body.filename, content_bytes)
+    use_context_aware_chunking = body.semantic_boundaries or body.contextualize
+
+    parsed_chunks = parse(
+        body.filename,
+        content_bytes,
+        llm=llm if use_context_aware_chunking else None,
+        semantic_boundaries=body.semantic_boundaries,
+        contextualize=body.contextualize,
+    )
 
     if not parsed_chunks:
         try:

src/api/schemas.py

@@ -46,6 +46,29 @@ class IngestRequest(BaseModel):
         ),
         examples=["primary"],
     )
+    semantic_boundaries: bool = Field(
+        default=False,
+        description=(
+            "Only affects text and PDF content. When true, an LLM chooses "
+            "chunk boundaries based on semantic coherence (topic shifts, "
+            "section boundaries) instead of the default character-length "
+            "accumulation. Falls back to the default chunker if the "
+            "content is too large for the LLM or the LLM output is "
+            "invalid. Independently toggleable from 'contextualize'."
+        ),
+    )
+    contextualize: bool = Field(
+        default=False,
+        description=(
+            "Only affects text and PDF content. When true, an LLM flags "
+            "which chunks would benefit from a short situating context "
+            "block (Anthropic-style Contextual Retrieval) and prepends it "
+            "to their content; self-contained chunks are left untouched. "
+            "Falls back to the default chunker if the content is too "
+            "large for the LLM or the LLM output is invalid. "
+            "Independently toggleable from 'semantic_boundaries'."
+        ),
+    )
 
     @field_validator("filename")
     @classmethod

src/ingestion/chunker.py

@@ -20,6 +20,10 @@ def to_parsed_chunk(
     filename: str,
     chunk_index: int,
     total_chunks_amount: int,
+    has_context_block: bool = False,
+    context_block_range: tuple[int, int] | None = None,
+    has_overlap: bool = False,
+    overlap_range: tuple[int, int] | None = None,
 ):
     """Create a ParsedChunk with standard metadata.
 
@@ -39,6 +43,24 @@ def to_parsed_chunk(
         total_chunks_amount (int):
             Total number of chunks produced for the source file.
 
+        has_context_block (bool, optional):
+            Whether a situating context block was prepended to the chunk
+            content. Defaults to ``False``.
+
+        context_block_range (tuple[int, int] | None, optional):
+            ``(start, end)`` character range of the context block within
+            ``chunk_content``, if ``has_context_block`` is ``True``.
+            Defaults to ``None``.
+
+        has_overlap (bool, optional):
+            Whether the chunk contains character/paragraph overlap carried
+            over from a neighboring chunk. Defaults to ``False``.
+
+        overlap_range (tuple[int, int] | None, optional):
+            ``(start, end)`` character range of the overlap within
+            ``chunk_content``, if ``has_overlap`` is ``True``. Defaults to
+            ``None``.
+
     Returns:
         ParsedChunk:
             Chunk instance with content, type and metadata.
@@ -50,33 +72,60 @@ def to_parsed_chunk(
             **build_metadata(Path(filename)),
             "chunk_index": str(chunk_index),
             "total_chunks": str(total_chunks_amount),
+            "has_context_block": str(has_context_block).lower(),
+            "context_block_range": (
+                f"{context_block_range[0]}:{context_block_range[1]}"
+                if context_block_range is not None
+                else ""
+            ),
+            "has_overlap": str(has_overlap).lower(),
+            "overlap_range": (
+                f"{overlap_range[0]}:{overlap_range[1]}"
+                if overlap_range is not None
+                else ""
+            ),
         },
     )
 
 
-def chunk_text(
-    filename: str,
-    text: str,
+def split_into_paragraphs(text: str) -> list[str]:
+    """Split text into non-empty, whitespace-stripped paragraphs.
+
+    Paragraphs are separated by double newlines (``\\n\\n``).
+
+    Args:
+        text (str):
+            Text content to split.
+
+    Returns:
+        list[str]:
+            Paragraphs in document order.
+    """
+    return [paragraph.strip() for paragraph in text.split("\n\n") if paragraph.strip()]
+
+
+def group_paragraphs_into_chunks(
+    paragraphs: list[str],
     chunk_size: int = chunk_size,
     chunk_overlap: int = chunk_overlap,
-) -> list[ParsedChunk]:
-    """Split text into paragraph-aware chunks.
+) -> list[str]:
+    """Accumulate paragraphs into character-limited, overlap-aware chunks.
 
-    The function preserves paragraph boundaries by splitting on
-    double newlines (``\\n\\n``). Paragraphs are accumulated until
-    adding another paragraph would exceed the configured chunk size.
+    Paragraphs are accumulated until adding another paragraph would exceed
+    ``chunk_size``. When a chunk boundary is reached, the last paragraph is
+    carried into the next chunk as overlap context. Paragraphs that exceed
+    ``chunk_size`` on their own are hard-split into overlapping character
+    chunks.
 
-    When a chunk boundary is reached, the last paragraph is carried
-    into the next chunk as overlap context. Paragraphs that exceed
-    ``chunk_size`` on their own are split into overlapping character
-    chunks by the given chunk_overlap.
+    This grouping logic is shared by :func:`chunk_text` and the LLM-based
+    context-aware chunker (when it is asked to only contextualize
+    already-fixed chunk boundaries rather than choose semantic boundaries
+    itself).
 
     Args:
-        filename (str):
-            Name of the source file.
-
-        text (str):
-            Text content to split.
+        paragraphs (list[str]):
+            Paragraphs in document order, e.g. from
+            :func:`split_into_paragraphs`.
 
         chunk_size (int, optional):
             Maximum chunk size in characters.
@@ -87,15 +136,10 @@ def chunk_text(
             Defaults to the value configured via ``CHUNK_OVERLAP``.
 
     Returns:
-        list[ParsedChunk]:
-            Paragraph-aware text chunks with metadata.
+        list[str]:
+            Raw chunk contents (paragraphs joined by ``\\n\\n``).
     """
-
     raw_chunks_content: list[str] = []
-
-    paragraphs: list[str] = [
-        paragraph.strip() for paragraph in text.split("\n\n") if paragraph.strip()
-    ]
     current_chunk_content: list[str] = []
 
     for paragraph in paragraphs:
@@ -140,6 +184,50 @@ def chunk_text(
     if current_chunk_content:
         raw_chunks_content.append("\n\n".join(current_chunk_content))
 
+    return raw_chunks_content
+
+
+def chunk_text(
+    filename: str,
+    text: str,
+    chunk_size: int = chunk_size,
+    chunk_overlap: int = chunk_overlap,
+) -> list[ParsedChunk]:
+    """Split text into paragraph-aware chunks.
+
+    The function preserves paragraph boundaries by splitting on
+    double newlines (``\\n\\n``). Paragraphs are accumulated until
+    adding another paragraph would exceed the configured chunk size.
+
+    When a chunk boundary is reached, the last paragraph is carried
+    into the next chunk as overlap context. Paragraphs that exceed
+    ``chunk_size`` on their own are split into overlapping character
+    chunks by the given chunk_overlap.
+
+    Args:
+        filename (str):
+            Name of the source file.
+
+        text (str):
+            Text content to split.
+
+        chunk_size (int, optional):
+            Maximum chunk size in characters.
+            Defaults to the value configured via ``CHUNK_SIZE``.
+
+        chunk_overlap (int, optional):
+            Overlap used when hard-splitting oversized paragraphs.
+            Defaults to the value configured via ``CHUNK_OVERLAP``.
+
+    Returns:
+        list[ParsedChunk]:
+            Paragraph-aware text chunks with metadata.
+    """
+    paragraphs = split_into_paragraphs(text)
+    raw_chunks_content = group_paragraphs_into_chunks(
+        paragraphs, chunk_size, chunk_overlap
+    )
+
     total_chunks_amount = len(raw_chunks_content)
 
     return [