codetwine

A CLI tool that analyzes source code with tree-sitter, extracts key definitions and dependencies, generates design documents via LLM, and consolidates everything into a single knowledge file. The knowledge file can be used as input material for LLM-powered code search and Q&A, such as RLM, GraphRAG, and agent search.

Dependencies are extracted at the symbol level (functions, classes)
Design documents are generated per file, taking dependencies into account
The consolidated result is written as a single JSON, a SQLite database, or both (KNOWLEDGE_FORMAT)
Outputs dependency graphs in Mermaid format

🧬 Supported Languages

Definitions, dependencies and design documents are extracted for the following languages (extensions py, java, kt, kts, js, jsx, ts, tsx, c, cpp, h, sql):

Python
Java
JavaScript
TypeScript
C
C++
Kotlin
SQL

Every other non-empty text file (.md, .yaml, .toml, Makefile, ...) is also listed in the outputs and copied to the output directory, with empty definitions, callee_usages, same_file_usages and caller_usages, a null summary and no design document. Empty files and binary files (a NUL byte in the first 8 KiB) are skipped.

🚀 Quick Start

Prerequisites

Python 3.11 or higher
uv package manager
API key for an LLM provider (Anthropic / OpenAI / Google, etc. Not required when using local LLMs like Ollama)

1. Installation

git clone https://github.com/yumeiriowl/codetwine.git
cd codetwine
uv sync

2. Configuration

cp .env.example .env

Set the following in the .env file:

# LLM API key (can be omitted for providers that don't require authentication, e.g. Ollama)
LLM_API_KEY=your-api-key-here

# LLM model name (specify the provider as a prefix in litellm format)
#   Anthropic:  anthropic/<model>
#   OpenAI:     openai/<model>
#   Google:     gemini/<model>
#   Ollama:     ollama/<model>
LLM_MODEL=anthropic/claude-sonnet-4-6

# Root directory of the project to analyze (absolute path)
DEFAULT_PROJECT_DIR=/path/to/your/project

# LLM output language (default: English)
OUTPUT_LANGUAGE=English

3. Usage

Basic Execution

uv run main.py

Analyzes the project set in DEFAULT_PROJECT_DIR in .env and outputs results to the output/ directory.

Specifying Project and Output Directories

uv run main.py --project-dir /path/to/your/project --output-dir /path/to/output

Argument	Description	Default
`--project-dir`	Root directory of the project to analyze	`DEFAULT_PROJECT_DIR` from `.env`
`--output-dir`	Output directory for analysis results	`DEFAULT_OUTPUT_DIR` from `.env` (defaults to `output/` if not set). When only `--project-dir` is specified, `DEFAULT_OUTPUT_DIR` is ignored and `output/` is used

Calling from Python

import asyncio
from codetwine.pipeline import process_all_files

run_result = asyncio.run(process_all_files(
    "/path/to/your/project", "/path/to/output", llm_client=None,
    file_list=["src/main.py", "src/utils.py"],
))

file_list (optional) holds file paths relative to the project root. When given, only these files are analyzed instead of walking the project directory; EXCLUDE_PATTERNS and the empty / binary file check still apply. llm_client is an LLMClient() when ENABLE_LLM_DOC=True.

Return key	Type	Description
`file_count`	int	Number of files analyzed
`dependency_fail_list`	string[]	Files whose dependency extraction failed. Their `file_dependencies.json` and copy are removed from the output directory
`doc_count`	int	Files with a complete design document (`0` when `ENABLE_LLM_DOC=False`)
`doc_fail_list`	string[]	Files left without a complete design document

⚙️ Configuration Options

The following options can be configured in the .env file.

LLM Settings

Variable	Description	Default
`LLM_API_KEY`	API key for the LLM provider	None
`LLM_MODEL`	Model name in litellm format (required when `ENABLE_LLM_DOC=True`)	None
`LLM_API_BASE`	API base URL (set when using non-standard endpoints, e.g. Ollama, Azure)	Not set
`OUTPUT_LANGUAGE`	Output language for design documents	`English`
`DOC_MAX_TOKENS`	Token limit for LLM output. Output cut at this limit is saved as it is and a warning is logged	`16384`

Path Settings

Variable	Description	Default
`DEFAULT_PROJECT_DIR`	Root directory of the project to analyze	Repository root
`DEFAULT_OUTPUT_DIR`	Output directory for analysis results	`output/`
`DOC_TEMPLATE_PATH`	Path to the design document template JSON file	`doc_template.json`

Performance Settings

Variable	Description	Default
`MAX_WORKERS`	Number of parallel workers for document generation	`4`
`MAX_RETRIES`	Number of retries of an LLM API call after a rate limit error (`0`: one call, no retry)	`3`
`RETRY_WAIT`	Wait time in seconds between retries	`2`
`PARSE_CACHE_MAX_FILES`	Number of files whose parse results are kept in memory at once. Lowering it reduces peak memory on large repositories at the cost of re-parsing; `0` keeps every parse result until the run ends	`200`

Output Settings

Variable	Description	Default
`KNOWLEDGE_FORMAT`	Form of the whole-project result: `json` (`project_knowledge.json`), `sqlite` (`project_knowledge.sqlite`), or `both`	`json`

The per-file JSON files are written in every case, and the SQLite database is built from them. On a repository with several thousand files, sqlite lets a reader query one file at a time instead of loading the whole consolidated JSON into memory.

The RLM QA agent example reads either form.

Analysis Settings

Variable	Description	Default
`ENABLE_LLM_DOC`	Enable/disable LLM design document generation (`True` / `False`)	`True`
`SUMMARY_MAX_CHARS`	Maximum character count for summaries	`600`
`ENABLE_CODE_SUMMARY`	Enable LLM summarization of large code when a prompt exceeds the model context window (`True` / `False`). When `False`, an oversized prompt is only reduced by dropping caller/callee context	`True`
`CODE_SUMMARY_TRIGGER_LINES`	Line span above which a definition or dependency symbol is summarized during context-overflow fallback	`40`
`CODE_SUMMARY_MAX_CHARS`	Character limit for a single code behavior summary	`400`
`EXCLUDE_PATTERNS`	Patterns to exclude during file traversal (comma-separated, fnmatch format). Every text file is collected, so use it to keep out secrets, data and lock files, and an output directory inside the project	`__pycache__,.git,.github,.venv,node_modules`

🔄 High-Level Processing Flow

Build the project-wide dependency graph
- Collects every non-empty text file from the target directory
- Analyzes import statements in each file of a supported language and identifies inter-file dependencies
Extract dependency information for each file
- Generates a syntax tree with tree-sitter and extracts definitions (functions, classes, etc.)
- Based on the dependency graph built in step 1, extracts callee and caller file paths, line numbers, and source code
- A file whose extension has no tree-sitter language gets empty definitions and usages
Generate design documents via LLM (files of a supported language only)
- Compares each source file's hash with the one recorded in its previous design document (doc.json) to identify the files whose documents must be regenerated
- Sorts files in topological order, processing from files with no dependencies toward dependent files
- Passes each file's source code, dependency information, and callee document summaries to the LLM, generating a design document section by section according to the template (doc_template.json)
- Generates a summary for each file. This summary is included as input when generating documents for subsequent dependent files, giving the LLM knowledge of each dependency's role and behavior
- Context-overflow handling: if a section prompt exceeds the model context window, it is retried with progressively smaller context — dropping caller usage snippets, then dependency document summaries, then summarizing large dependency symbols, and finally summarizing large definitions in the source itself. Code summaries are generated by the LLM and cached per symbol. Set ENABLE_CODE_SUMMARY=False to skip the summarization steps
Save all outputs
- Saved to <output directory>/<project name>
- All dependencies and design documents are consolidated into a single JSON (project_knowledge.json), a SQLite database (project_knowledge.sqlite), or both, as KNOWLEDGE_FORMAT selects
- Each file's entry is written as soon as it is read, so the consolidated result is never assembled in memory as a whole
- Dependency graphs and design documents are also output as Markdown for readability

Note: LLM API calls are only made in step 3. No LLM is used in other steps.

📁 Output Files

Running the tool generates the following files in <output directory>/<project name>/ (default: output/<project name>/).

File	Description
`project_knowledge.json`	Consolidated JSON of all file dependencies and design documents (`KNOWLEDGE_FORMAT=json` / `both`)
`project_knowledge.sqlite`	Consolidated SQLite database with the same content (`KNOWLEDGE_FORMAT=sqlite` / `both`)
`project_dependency_summary.json`	Consolidated JSON of the dependency graph + per-file summaries
`dependency_graph.md`	Dependency graph in Mermaid format
`<filename>/file_dependencies.json`	Per-file definition and dependency information
`<filename>/doc.json`	Per-file design document (JSON format)
`<filename>/doc.md`	Per-file design document (Markdown format)
`<filename>/<original filename>`	Copy of the original file

⚠️ Dependency Analysis Limitations

Dependency extraction is performed through static syntax analysis with tree-sitter, parsing import statements (including #include) in source code to identify inter-file dependencies. Dependencies may not be detected or may be incomplete in the following cases.

Common to All Languages

Dynamic imports: Patterns that construct module names as strings at runtime cannot be detected
- Python: importlib.import_module(name), __import__(name)
- JavaScript/TypeScript: import(variable)
- Java: Class.forName("com.example.Foo")

JavaScript / TypeScript

Build tool path aliases: Path aliases such as @/, ~/ defined in Webpack, Vite, tsconfig, etc. cannot be resolved. Imports using aliases are not recognized as project files and are missing from dependencies

Java / Kotlin

Wildcard imports: import com.example.* is detected as an import statement, but individual class files cannot be resolved, so they are not recognized as dependencies
Implicit same-package references: In Java/Kotlin, classes in the same package can be referenced without imports. A dependency is added when a top-level definition name of another file in the same directory is used in the source code

C / C++

Build system include paths: Include paths added via CMake or Makefile -I options are not considered. Headers that cannot be resolved from the project root or current directory as relative paths are not detected as dependencies

SQL

Object references: A dependency is added when a table, view, function, type or sequence created in another .sql file of the project is referenced. Names are compared as written, so a reference that differs in case or quoting from the CREATE statement is not detected
Unsupported syntax: PostgreSQL-style CREATE PROCEDURE, CALL, GRANT and psql \i are not parsed by the grammar. Surrounding statements are still analyzed

♻️ Incremental Processing

On subsequent runs, only the changed files and their affected scope have their design documents regenerated.

Change detection: Compares the SHA256 hash of each source file with the source_hash recorded in its doc.json to detect changed files
Dependency information: Re-extracted for all files every run to ensure consistency
Design documents: Only the changed files and files that depend on them (dependents) are regenerated; all others reuse previous results
Completeness check: Even for unchanged files, if the existing doc.json has missing sections or an empty summary (e.g. due to a previous LLM API failure), it is treated as incomplete and regenerated

🗄️ SQLite Output

With KNOWLEDGE_FORMAT=sqlite or both, the same content as project_knowledge.json is written to project_knowledge.sqlite. The database is rebuilt from the per-file JSON files on every run.

Table	Columns	Description
`meta`	`key`, `value`	`project_name`, `schema_version`, `created_at`
`files`	`file`, `summary`, `file_dependencies`, `doc`	One row per file. `file_dependencies` and `doc` hold the JSON bodies as text; both are `NULL` when the file has none
`file_edges`	`file`, `direction`, `other`	The dependency graph. `direction` is `caller` or `callee` as seen from `file`
`definitions`	`file`, `name`, `type`, `start_line`, `end_line`	Every definition, indexed by name and by file, for looking one up without reading any file body

codetwine/knowledge_db.py provides the read API:

from codetwine import knowledge_db

conn = knowledge_db.open_knowledge("output/my-project/project_knowledge.sqlite")

# One file entry at a time, in the same structure as project_knowledge.json's "files"
for entry in knowledge_db.iter_files(conn):
    print(entry["file"])

# One entry per file, in the structure of project_knowledge.json's "project_dependencies"
for dep in knowledge_db.iter_dependencies(conn):
    print(dep["file"], dep["summary"])

entry = knowledge_db.get_file(conn, "my-project/src/main_py/main.py")
callees = knowledge_db.callees_of(conn, entry["file"])
callers = knowledge_db.callers_of(conn, entry["file"])
hits = knowledge_db.find_definitions(conn, "parse_file")
partial_hits = knowledge_db.find_definitions(conn, "parse", partial=True)

file_edges holds both directions as they were analyzed. callers and callees come from two separate analyses and do not always mirror each other.

📋 Output JSON Schema

project_knowledge.json

Consolidated JSON integrating all file dependencies and design documents.

{
  "project_name": "string",
  "project_dependencies": [
    {
      "file": "string",
      "summary": "string|null",
      "callers": ["string"],
      "callees": ["string"]
    }
  ],
  "files": [
    {
      "file": "string",
      "file_dependencies": {},
      "doc": {}
    }
  ]
}

Field	Type	Description
`project_name`	string	Project name
`project_dependencies[].file`	string	Path of the source file copied to the output directory
`project_dependencies[].summary`	string\|null	Summary of the file (null when the design document is not generated, and always for a file whose extension has no tree-sitter language)
`project_dependencies[].callers`	string[]	Paths of dependent files copied to the output directory
`project_dependencies[].callees`	string[]	Paths of dependency files copied to the output directory
`files[].file`	string	Path of the source file copied to the output directory
`files[].file_dependencies`	object	Same structure as file_dependencies.json (excluding `file` field)
`files[].doc`	object	Same structure as doc.json (excluding `file` field)

project_dependency_summary.json

Consolidated JSON of the dependency graph and per-file summaries.

{
  "project_name": "string",
  "files": [
    {
      "file": "string",
      "summary": "string|null",
      "callers": ["string"],
      "callees": ["string"]
    }
  ]
}

Field	Type	Description
`project_name`	string	Project name
`files[].file`	string	Path of the source file copied to the output directory
`files[].summary`	string\|null	Summary of the file
`files[].callers`	string[]	Paths of dependent files copied to the output directory
`files[].callees`	string[]	Paths of dependency files copied to the output directory

file_dependencies.json

Per-file definition and dependency information.

{
  "file": "string",
  "definitions": [
    {
      "name": "string",
      "type": "string",
      "start_line": 0,
      "end_line": 0,
      "context": "string"
    }
  ],
  "callee_usages": [
    {
      "name": "string",
      "from": "string",
      "target_context": "string",
      "lines": [0]
    }
  ],
  "same_file_usages": [
    {
      "name": "string",
      "lines": [0]
    }
  ],
  "caller_usages": [
    {
      "name": "string",
      "file": "string",
      "usage_context": "string",
      "lines": [0]
    }
  ]
}

Field	Type	Description
`file`	string	Path of the source file copied to the output directory
`definitions[].name`	string	Function/class name
`definitions[].type`	string	Definition type (tree-sitter node type, varies by language. Python: `function_definition`, `class_definition` / Java: `class_declaration`, `method_declaration` / JS/TS: `function_declaration`, `class_declaration` / SQL: `create_table`, `create_view`, etc.)
`definitions[].start_line`	int	Start line number
`definitions[].end_line`	int	End line number
`definitions[].context`	string	Full source code of the definition
`callee_usages[].name`	string	Name of the used symbol
`callee_usages[].from`	string	Path of the dependency file copied to the output directory
`callee_usages[].target_context`	string	Full source code of the dependency symbol
`callee_usages[].lines`	int[]	Line numbers of usage within this file
`same_file_usages[].name`	string	Name of a symbol defined in this file and used in it
`same_file_usages[].lines`	int[]	Line numbers of usage within this file, outside the definition of the same name
`caller_usages[].name`	string	Name of the symbol being used
`caller_usages[].file`	string	Path of the dependent file copied to the output directory
`caller_usages[].usage_context`	string	Source code of the usage location in the dependent
`caller_usages[].lines`	int[]	Line numbers of usage in the dependent file

Members declared inside a class, struct, interface, enum or namespace are listed as their own entries in addition to the enclosing definition, so their line range and context are contained in the enclosing entry. Functions defined inside a function body are not listed.

doc.json

Per-file design document.

{
  "file": "string",
  "summary": "string",
  "sections": [
    {
      "id": "string",
      "title": "string",
      "content": "string"
    }
  ],
  "source_hash": "string"
}

Field	Type	Description
`file`	string	Path of the source file copied to the output directory
`summary`	string	Summary of the file
`sections[].id`	string	Section identifier (corresponds to id in doc_template.json)
`sections[].title`	string	Section heading
`sections[].content`	string	Section body (Markdown format)
`source_hash`	string	SHA256 hash of the source file the document was generated from

In the definitions section, each definition starts with a level-2 heading holding only the definition name in backticks (## `parse_args`).

🎨 Customizing the Design Document Template

Edit doc_template.json to customize the section structure and LLM instructions for design documents.

{
  "sections": [
    {
      "id": "overview",
      "title": "Section heading",
      "prompt": "Instruction text for the LLM"
    }
  ],
  "summary_prompt": "Instruction for generating the overall summary"
}

Operation	Method
Add section	Add a new object to the `sections` array
Remove section	Remove the corresponding element from the `sections` array
Change instructions	Edit the text in the `prompt` field
Change summary instructions	Edit the `summary_prompt` field
Use a different template	Specify the path in `DOC_TEMPLATE_PATH` in `.env`

When you modify the template sections, existing design documents whose section structure no longer matches the template are automatically regenerated on the next run.

✏️ Manual Editing of Design Documents

You can manually edit the output doc.md and have it automatically reflected in doc.json on the next run.

Edit output/<project name>/<filename>/doc.md with a text editor
On the next uv run main.py execution, if doc.md has a newer timestamp than doc.json, the Markdown section content is parsed and applied to doc.json

Notes for editing:

Do not delete or rename ## Section heading lines. The parser uses them as section delimiters, and without headings, parsing will not work correctly
The body text below section headings can be freely edited

⏭️ Running Without Design Document Generation

To output only dependency information without generating LLM design documents, set ENABLE_LLM_DOC=False in .env.

ENABLE_LLM_DOC=False

The design document generation step is skipped. Dependency information (file_dependencies.json and file copies) is still generated for each file, along with project_knowledge.json, project_dependency_summary.json, and dependency_graph.md. Since no LLM is used, it can run without API keys or model configuration. Such a run does not affect change detection: the next run with ENABLE_LLM_DOC=True regenerates the design documents of every file changed since they were generated.

💡 Usage Example: RLM QA Agent

examples/rlm_qa/ contains a sample that performs interactive Q&A against a knowledge file as a usage example for the consolidated output. It uses dspy's RLM and PythonInterpreter to generate answers by manipulating data with Python code.

The agent receives only the file graph and one summary per file. Definitions, source code and design documents are fetched per file through host-side tools (get_file_detail, search_text, read_source_file, get_files_using, graph_search), so the whole analysis is never sent into the sandbox. Against a knowledge file of a few thousand files, this keeps what the sandbox receives at a few megabytes instead of the whole consolidated output.

Sample Output

examples/sample_output/ contains sample output produced by analyzing the codetwine repository itself, as of commit 9e7ac1a (2026-09-09) with claude-sonnet-5. It is a snapshot for trying out RLM QA and is not regenerated when the code changes, so it does not reflect the current source. This output was generated using the search-oriented template examples/doc_template_search.json: one section, in prose, with an overview of the file and one entry per definition written in the words a reader would search for. It is not tied to any language. examples/doc_template_python.json is a Python-specific template that writes a fuller specification in one section. rlm_qa_agent.py references this output by default, so you can try out RLM QA immediately without running any analysis.

Note: The file field paths in project_knowledge.json refer to sources copied into the output directory and differ from the original source tree paths (e.g. codetwine/import_to_path.py → codetwine/import_to_path_py/import_to_path.py).

Additional Prerequisites

Deno runtime
dspy package (install with uv sync --extra examples)

How to Run

uv run python examples/rlm_qa/rlm_qa_agent.py

By default, the sample output in examples/sample_output/ is used. To use your own project's output, edit TARGET_KNOWLEDGE_PATH in rlm_qa_agent.py.

Either form of the knowledge file works: a path ending in .sqlite is queried per file through codetwine/knowledge_db.py, any other path is read as project_knowledge.json. Both answer the same questions. Only the SQLite form avoids holding the whole analysis in memory.

🏗️ Project Structure

codetwine/
├── README.md
├── CHANGELOG.md                # Changelog
├── LICENSE                     # License (MIT)
├── pyproject.toml              # Package configuration and dependencies
├── main.py                     # CLI entry point
├── doc_template.json           # Design document section template definition
├── .env.example                # Environment variable template
├── codetwine/
│   ├── pipeline.py             # Main pipeline (dependency graph building → document generation → output)
│   ├── file_analyzer.py        # Per-file dependency analysis
│   ├── doc_creator.py          # Design document generation via LLM
│   ├── import_to_path.py       # Import statement to file path resolution
│   ├── output.py               # JSON and Mermaid output processing
│   ├── knowledge_db.py         # SQLite output and read API
│   ├── config/
│   │   ├── settings.py         # Environment variables and per-language settings management
│   │   └── logger.py           # Logging configuration and progress output
│   ├── extractors/
│   │   ├── definitions.py      # Definition extraction (functions, classes, etc.)
│   │   ├── imports.py          # Import statement extraction
│   │   ├── usages.py           # Symbol usage location extraction
│   │   ├── usage_analysis.py   # Usage location analysis
│   │   └── dependency_graph.py # Project-wide dependency graph construction
│   ├── parsers/
│   │   └── ts_parser.py        # Source code parser using tree-sitter
│   ├── llm/
│   │   └── client.py           # LLM API client via litellm
│   └── utils/
│       └── file_utils.py       # File operation utilities
└── examples/
    ├── doc_template_python.json  # Python-optimized design document template
    ├── doc_template_search.json  # Search-oriented design document template (any language)
    ├── rlm_qa/                   # RLM QA agent sample
    │   ├── rlm_qa_agent.py       # Interactive Q&A agent
    │   ├── knowledge_store.py    # Read access to either knowledge file form
    │   └── qa_tools.py           # Tool definitions for the agent
    └── sample_output/            # Sample output (codetwine analyzed against itself)

🙏 Acknowledgments

This project uses the following libraries:

tree-sitter - Source code syntax analysis
litellm - Unified interface for multiple LLM providers

📄 License

MIT License. See LICENSE for details.

Name		Name	Last commit message	Last commit date
codetwine		codetwine
examples		examples
.env.example		.env.example
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
README.md		README.md
doc_template.json		doc_template.json
main.py		main.py
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Latest commit

History

Folders and files

Repository files navigation