Skip to content

Commit 3be3896

Browse files
bartollibartolli
committed
chore: release v0.5.10
feat(cli): add parse command for AST debugging - output tree-sitter AST nodes in JSONL format - support max-depth limiting and all-nodes flag - skip index loading for parse, config, benchmark commands - add integration tests for parse command fix(tests): resolve parallel test execution race conditions - use unique temp file names with atomic counter - move parse tests to integration test structure fix(parser): improve TypeScript nested node extraction & audit coverage - add node registration for 123/182 coverage - update all language audit reports docs: add parse command documentation - document in README and contributing guide - add tree-sitter development workflow - add tree-sitter CLI detection in scripts
1 parent e003023 commit 3be3896

34 files changed

Lines changed: 971 additions & 103 deletions

‎.gitignore‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,4 +10,12 @@
1010

1111
# IDE
1212
.idea/
13-
.vscode/
13+
.vscode/
14+
15+
# Tree-sitter development
16+
/contributing/tree-sitter/grammars/
17+
/contributing/tree-sitter/node_modules/
18+
/contributing/tree-sitter/build/
19+
/contributing/tree-sitter/target/
20+
/contributing/tree-sitter/.cache/
21+
/contributing/tree-sitter/.env

‎CHANGELOG.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,22 @@ All notable changes to this project will be documented in this file.
55
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
66
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
77

8+
## [0.5.10] - 2025-09-11
9+
10+
### Added
11+
- Parse command: output AST nodes in JSONL format for debugging
12+
- Parse command flags: --max-depth, --all-nodes, --output
13+
- Tree-sitter CLI detection in development scripts
14+
15+
### Fixed
16+
- TypeScript parser: improved nested node extraction in arrow functions and JSDoc blocks (123/182 coverage)
17+
- Test parallel execution race conditions with unique temp files
18+
- CLI startup performance for non-index commands (parse, config, benchmark)
19+
20+
### Changed
21+
- Parser audit reports now include timestamps
22+
- Parse command integration tests moved to proper test structure
23+
824
## [0.5.9] - 2025-09-07
925

1026
### Enhanced

‎Cargo.lock‎

Lines changed: 7 additions & 7 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[package]
22
name = "codanna"
3-
version = "0.5.9"
3+
version = "0.5.10"
44
authors = ["Angel Bartolli <bartolli@gmail.com>"]
55
edition = "2024"
66
description = "Code Intelligence for Large Language Models"
@@ -14,10 +14,10 @@ categories = ["development-tools", "command-line-utilities", "parsing", "text-pr
1414
autoexamples = false
1515

1616
[profile.release]
17-
# opt-level = 3
18-
# lto = true
19-
# codegen-units = 1
20-
# strip = true
17+
opt-level = 3
18+
lto = true
19+
codegen-units = 1
20+
strip = true
2121

2222
[dependencies]
2323
anyhow = "1.0.98"
@@ -36,7 +36,7 @@ num_cpus = "1.17.0"
3636
parking_lot = "0.12.4"
3737
rayon = "1.10.0"
3838
rkyv = { version = "0.8.10", features = ["bytecheck", "std"] }
39-
rmcp = { version = "0.6.1", features = ["server", "client", "transport-io", "transport-child-process", "transport-streamable-http-server", "transport-sse-server", "transport-worker"] }
39+
rmcp = { version = "0.6.4", features = ["server", "client", "transport-io", "transport-child-process", "transport-streamable-http-server", "transport-sse-server", "transport-worker"] }
4040
serde = { version = "1.0.219", features = ["derive"] }
4141
serde_json = "1.0.141"
4242
sha2 = "0.10"
@@ -52,7 +52,7 @@ tree-sitter-java = "0.23.5"
5252
tree-sitter-javascript = "0.25.0"
5353
# tree-sitter-php = "0.23.11"
5454
tree-sitter-php = "0.24.1"
55-
tree-sitter-python = "0.23.6"
55+
tree-sitter-python = "0.25.0"
5656
tree-sitter-rust = "0.24.0"
5757
tree-sitter-typescript = "0.23.2"
5858
walkdir = "2.5.0"

‎README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -333,6 +333,7 @@ All retrieve commands support `--json` flag for structured output (exit code 3 w
333333
| `codanna mcp-test` | Verify Claude can connect and list available tools | `codanna mcp-test` |
334334
| `codanna mcp <TOOL>` | Execute MCP tools without spawning server | `codanna mcp find_symbol main --json` |
335335
| `codanna benchmark` | Benchmark parser performance | `codanna benchmark rust --file my_code.rs` |
336+
| `codanna parse` | Parse file and output AST as JSON Lines | `codanna parse file.rs --all-nodes` |
336337

337338
#### Common Flags
338339

@@ -463,7 +464,7 @@ sudo dnf install pkgconfig openssl-devel
463464

464465
## Roadmap
465466

466-
### Current Release: v0.5.9
467+
### Current Release: v0.5.10
467468

468469
See [CHANGELOG.md](CHANGELOG.md) for detailed release notes and feature history.
469470

‎contributing/README.md‎

Lines changed: 76 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ Thank you for your interest in contributing to Codanna! This guide focuses on th
99
- **[Development Setup](#development-setup)** - Local environment setup
1010
- **[Testing Workflow](#testing-your-changes)** - Pre-commit and CI/CD scripts
1111

12-
## Current Status (v0.5.8)
12+
## Current Status
1313

1414
See [CHANGELOG.md](../CHANGELOG.md) for detailed release notes and feature history.
1515

@@ -81,6 +81,81 @@ codanna/
8181
└── tests/ # Language parser and integration tests
8282
```
8383

84+
## Development Tools
85+
86+
### Parse Command
87+
88+
The `codanna parse` command is essential for parser development and debugging:
89+
90+
```bash
91+
# Parse a file and output AST nodes in JSONL format
92+
codanna parse file.rs # Named nodes only (like tree-sitter)
93+
codanna parse file.rs --all-nodes # Include all nodes (punctuation, keywords)
94+
codanna parse file.rs --max-depth 3 # Limit traversal depth
95+
codanna parse file.rs -o ast.jsonl # Output to file
96+
97+
# Analyze AST structure
98+
codanna parse file.rs | jq -r .node | sort -u # List unique node types
99+
codanna parse file.rs | jq 'select(.depth == 1)' # Show top-level nodes
100+
codanna parse file.rs | jq 'select(.node == "function_item")' # Find specific nodes
101+
```
102+
103+
**Key Features:**
104+
- **Default behavior matches tree-sitter CLI** - Shows only named nodes for direct comparison
105+
- **`--all-nodes` flag** - Shows complete AST including anonymous nodes (operators, punctuation)
106+
- **JSONL format** - One JSON object per line, perfect for streaming and Unix tools
107+
- **Hierarchy tracking** - Each node includes depth, parent ID, and unique ID
108+
- **Error codes** - Proper exit codes (3=NotFound, 4=ParseError, 8=UnsupportedLanguage)
109+
110+
### Tree-sitter Integration Scripts
111+
112+
Located in `contributing/tree-sitter/scripts/`:
113+
114+
#### setup.sh
115+
Install tree-sitter grammars for testing:
116+
```bash
117+
./contributing/tree-sitter/scripts/setup.sh typescript # Install specific grammar
118+
./contributing/tree-sitter/scripts/setup.sh # Show installed grammars
119+
```
120+
121+
#### compare-nodes.sh
122+
Compare codanna parser with tree-sitter (two modes):
123+
124+
**Language mode** - Runs audit tests and generates reports:
125+
```bash
126+
./contributing/tree-sitter/scripts/compare-nodes.sh rust
127+
```
128+
This mode:
129+
- Runs `cargo test comprehensive_rust_analysis`
130+
- Generates audit reports in `contributing/parsers/rust/`:
131+
- `AUDIT_REPORT.md` - Parser coverage analysis
132+
- `GRAMMAR_ANALYSIS.md` - Node handling statistics
133+
- Compares comprehensive example files
134+
- Shows parser implementation gaps
135+
136+
**File mode** - Compares any specific file:
137+
```bash
138+
./contributing/tree-sitter/scripts/compare-nodes.sh examples/rust/main.rs
139+
```
140+
This mode:
141+
- Uses `codanna parse` to analyze the file
142+
- Compares with tree-sitter output
143+
- Saves detailed comparison to `{filename}_comparison.log`
144+
- Shows matching statistics
145+
146+
#### explore-ast.sh
147+
Quick AST exploration:
148+
```bash
149+
# Use codanna (default)
150+
./contributing/tree-sitter/scripts/explore-ast.sh file.rs
151+
152+
# Use tree-sitter
153+
./contributing/tree-sitter/scripts/explore-ast.sh file.rs tree-sitter
154+
155+
# Compare both
156+
./contributing/tree-sitter/scripts/explore-ast.sh file.rs both
157+
```
158+
84159
## Testing Your Changes
85160

86161
We provide local scripts that replicate our CI/CD pipeline:

‎contributing/development/tree-sitter-cli-setup.md‎

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
# Tree-sitter CLI Setup for Local Development
2+
3+
## Purpose
4+
5+
Set up tree-sitter CLI for local grammar exploration and testing without affecting the production build. This enables:
6+
- Testing new language grammars before integration
7+
- Debugging parser issues with official tree-sitter tools
8+
- Exploring AST structures interactively
9+
- Comparing tree-sitter output with our parser implementation
10+
11+
## Quick Start
12+
13+
```bash
14+
# 1. Install tree-sitter CLI (if not already installed)
15+
cargo install tree-sitter-cli --locked
16+
17+
# 2. Install a grammar (one-time per language)
18+
cd contributing/tree-sitter
19+
./scripts/setup.sh typescript
20+
21+
# 3. Parse a file to see AST
22+
tree-sitter parse examples/typescript/comprehensive.ts
23+
24+
# 4. Compare with our parser
25+
.codanna/scripts/compare-nodes.sh typescript
26+
```
27+
28+
## Directory Structure
29+
30+
```
31+
contributing/tree-sitter/
32+
├── README.md # User documentation
33+
├── grammars/ # Cloned grammar repositories (gitignored)
34+
│ ├── tree-sitter-typescript/
35+
│ ├── tree-sitter-python/
36+
│ └── ...
37+
└── scripts/
38+
├── setup.sh # Install grammars on-demand
39+
└── explore-ast.sh # Helper to parse files
40+
41+
.codanna/scripts/
42+
└── compare-nodes.sh # Compare tree-sitter with our parser
43+
```
44+
45+
## How It Works
46+
47+
1. **Tree-sitter Configuration**: The setup script configures `~/.config/tree-sitter/config.json` to point to our grammars directory
48+
2. **Grammar Naming**: Grammars must be named `tree-sitter-{language}` for tree-sitter to find them
49+
3. **On-Demand Installation**: Install only the languages you need with `./scripts/setup.sh <language>`
50+
4. **Automatic Language Detection**: Tree-sitter determines which grammar to use based on file extension
51+
52+
## Scripts
53+
54+
### setup.sh
55+
56+
Configures tree-sitter and installs grammars on-demand:
57+
58+
```bash
59+
# Show installed grammars
60+
./scripts/setup.sh
61+
62+
# Install specific language
63+
./scripts/setup.sh typescript
64+
./scripts/setup.sh python
65+
./scripts/setup.sh rust
66+
```
67+
68+
Supported languages: typescript, python, rust, go, php, c, cpp
69+
70+
### explore-ast.sh
71+
72+
Simple wrapper for parsing files:
73+
74+
```bash
75+
./scripts/explore-ast.sh examples/typescript/comprehensive.ts
76+
```
77+
78+
### compare-nodes.sh
79+
80+
Compares tree-sitter AST nodes with our parser implementation:
81+
82+
```bash
83+
# From project root
84+
.codanna/scripts/compare-nodes.sh typescript
85+
```
86+
87+
This script:
88+
1. Parses the comprehensive example with tree-sitter
89+
2. Runs our parser tests (which regenerates audit reports)
90+
3. Shows differences between the two parsers
91+
92+
## Usage Examples
93+
94+
### Exploring AST Structure
95+
96+
```bash
97+
# Parse any file with tree-sitter
98+
tree-sitter parse examples/typescript/comprehensive.ts
99+
100+
# See specific node types
101+
tree-sitter parse examples/python/main.py | grep "class_definition"
102+
```
103+
104+
### Debugging Parser Differences
105+
106+
```bash
107+
# Compare node recognition
108+
.codanna/scripts/compare-nodes.sh typescript
109+
110+
# Output shows:
111+
# - Nodes tree-sitter finds that we don't handle
112+
# - Nodes we report that tree-sitter doesn't find
113+
```
114+
115+
### Testing New Languages
116+
117+
```bash
118+
# Install a new grammar
119+
./contributing/tree-sitter/scripts/setup.sh go
120+
121+
# Test parsing
122+
tree-sitter parse examples/go/main.go
123+
```
124+
125+
## Benefits
126+
127+
1. **Official Reference**: Compare our parser against the official tree-sitter implementation
128+
2. **Node Discovery**: Find exact node names for parser implementation
129+
3. **Grammar Updates**: Test new grammar versions before updating dependencies
130+
4. **Debugging**: Identify parsing differences and missing node handlers
131+
5. **Report Generation**: Running compare-nodes.sh updates audit reports with current timestamps
132+
133+
## Notes
134+
135+
- Grammars are cloned with `--depth 1` for speed
136+
- The grammars directory is gitignored (contains large files)
137+
- Each developer's tree-sitter config points to their local grammar directory
138+
- No environment variables or .env files needed - uses tree-sitter's native configuration
139+
- Running compare-nodes.sh regenerates the audit reports as a side effect
140+
141+
## Integration with Development Workflow
142+
143+
1. When implementing a new language parser:
144+
- Install the grammar: `./scripts/setup.sh <language>`
145+
- Parse examples to understand AST structure
146+
- Use compare-nodes.sh to verify coverage
147+
148+
2. When debugging parsing issues:
149+
- Parse the problematic file with tree-sitter
150+
- Compare output with our parser
151+
- Identify missing or incorrectly handled nodes
152+
153+
3. When updating parser coverage:
154+
- Run compare-nodes.sh to see gaps
155+
- Implement missing node handlers
156+
- Verify with updated audit reports

‎contributing/parsers/c/AUDIT_REPORT.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# C Parser Coverage Report
22

3-
*Generated: 2025-09-11 15:39:19 UTC*
3+
*Generated: 2025-09-11 20:24:24 UTC*
44

55
## Summary
66
- Nodes in file: 145

0 commit comments

Comments
 (0)