Skip to content

Native Second String extension: Scala-compatible similarities and phonetics - #34

Draft
alexy wants to merge 1 commit into
sail-extensionsfrom
work/second-string-extension
Draft

alexy wants to merge 1 commit into
sail-extensionsfrom
work/second-string-extension

Conversation

@alexy

@alexy alexy commented Oct 6, 2026

Copy link
Copy Markdown
Member

Scope

Port Sem's Spark Second String into an independent native Sail extension wheel under examples/extensions/second-string. It uses the existing pysail.extensions bootstrap and DataFusion scalar Arrow FFI. All sixteen original SQL names/defaults remain, and ten configured entry points preserve the Scala DSL options. No Sail host implementation changes are included.

Reference: SemyonSinchenko/spark-second-string at a35db39fa8e9b65db2d201a45b86d11a6ca34b98. Apache-2.0 attribution includes the translated Commons Codec 1.21.0 primary Double Metaphone rules.

Verification

Published source f31c33ae01e574b146853ace4a901a3876cf5417 passes the detached package gate: Rust formatting, Clippy with warnings denied, 41 release Rust tests, Ruff, strict mypy, and 112 checks in each of local and two-process-worker Sail modes. The compiled Scala oracle contains 7,296 answers; every answer passes through foreign Arrow FFI and through shuffled Sail batches in both modes. Numeric tolerance is absolute 1e-12; phonetic strings are exact.

Runtime qualification uses the existing native release Sail host at 9f0aa7d2a50f1258544d37c05dd19b3ff3d915b3, API 1 / DataFusion 55.1.0 / Arrow 59.3.0. Host and extension identities are recorded separately.

Manual check

Build/install the release wheel as described in the package README, then start a compatible host with SAIL_EXPERIMENTAL_EXTENSIONS=1. Check:

SELECT ss_jaro_winkler('martha', 'marhta');
SELECT ss_jaccard_with_options('abc', 'abd', 2);
SELECT ss_soundex('Robert');

Expected: 0.9611111111111111, 1/3, and R163. The README also gives both native integration-test commands.

Qualification and Sem research status includes retained failed attempts and scope limits. No Spark speed comparison, Linux binary qualification, or stable ABI promise is claimed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant