PanGloss

A Rust morphological/phonological parsing engine, starting from a native port of HermitCrab's parser — words in, morphemes out — and growing toward a broader FST-based toolkit (spell-checking, hybrid finite-state approaches, and more) beyond HermitCrab's original scope.

Origin

rust/ here began as a from-scratch Rust reimplementation of SIL.Machine.Morphology.HermitCrab (the C# morphological parser used by SIL's FieldWorks/FLEx), built to be callable directly or switched in as a drop-in engine wherever HermitCrab is used today. It was extracted from sillsdev/machine as a single squashed commit (no line history carried over — the original repo keeps the full commit-by-commit record if it's ever needed). See docs/hermitcrab-rust-port-audit.md for exactly what was ported, what's known to still differ, and the process for tracking HermitCrab's ongoing evolution against this codebase.

The parity contract is not this repo's own history or corpora. As HermitCrab in Machine evolves, the binding correctness gate is the conformance oracle that lives there (pulled in here as a submodule once available) — PanGloss's own algorithms are free to diverge internally as its scope grows, as long as they keep passing that oracle. samples/ and docs/history/ are historical reference, not living gates.

API parity goal

PanGloss aims to keep a public surface recognizably similar to HermitCrab's — so that, where the two engines' scopes overlap, something already calling HermitCrab could call PanGloss instead with minimal changes. PanGloss is meant to be called directly (not through Machine's C# layer), so this is about a familiar shape, not wire compatibility with any specific C# interop path.

Where this is going: the hybrid FST

PanGloss's active development is porting a propose-and-verify hybrid FST analyzer — a fast path that proposes candidate analyses with an ensemble of finite-state machines in microseconds, then verifies each one exactly with the real engine (sound by construction, never a false positive) — originally built and measured in C# on the fst-advisor branch of sillsdev/machine (22-72x per-word speedups over the pooled engine, engine-pathological words answered in milliseconds). See docs/fst-plan/ for the full research and port plan: start with HYBRID_FST_FEASIBILITY.md (what it is and why it works), then HYBRID_FST_RUST_PLAN.md (the milestone-by-milestone Rust port plan this repo is executing).

Layout

• rust/ — the ported engine (crate map and build instructions in rust/README.md), including direct FieldWorks project (.fwdata) import — see that README's "Direct FieldWorks project import" section and docs/fwdata-import-plan.md.
• samples/data/ — reference grammars (Amharic, Indonesian, Sena) used by rust/'s test suite.
• docs/ — the port audit (above), the FST plan (docs/fst-plan/), the FieldWorks-import plan (docs/fwdata-import-plan.md, docs/snapshot-format.md), and historical planning documents under docs/history/.

Building

cd rust
cargo build --release
cargo test
cargo clippy --workspace --all-targets

License

MIT — see LICENSE.