Conversation
The five extended groups of the bash extglob become terms of their own: `@(a|b)` matches exactly one of its alternatives, `?(a|b)` at most one, `*(a|b)` any number of them, `+(a|b)` at least one, and `!(a|b)` matches a sub-string -- without separators, like a `*` -- that none of them matches. Requested in gobwas#3; `!(...)` also covers gobwas#47. Each alternative is a pattern in its own right, compiled as a sub-pattern and matched whole, so a `*` inside an alternative reaches only to the end of its own sub-string and never past it into what follows the group. In the engine a group finds every candidate sub-string on entry, takes the shortest and stores the rest as restart points carrying the candidate length in an extra path entry past the group's own depth, so that the backtracking walk can resume a repetition it did not begin. The compile-time passes -- the shaped terminal matchers, the terminal-star marking, needsState, minLength and requiredSuffix -- all learn about the new nodes, since a group may end a pattern, follow a star, sit inside a brace alternative or contain one. In the lexer a `(` opens a group only right after one of the five operator characters, and it binds to the character right before it, so `**(a)` is a `*` followed by the group `*(a)`, as in bash. The nesting is now a stack rather than a brace counter: the innermost open construct decides whether `,` `}` `|` `)` are special, so the groups and the `{...}` alternatives nest freely. Outside of a group `(`, `)` and `|` are plain characters, a backslash escapes them as it escapes everything else, and QuoteMeta escapes `(` and `)` so that its output still compiles to the quoted text. Match stays zero-allocation and deterministic, and terminates on every pattern, including the groups whose alternatives match the empty string such as `*(?(a))` or `*(*)`. The groups are tested against a reference matcher that re-implements the syntax independently, over 643k pattern/string/separator comparisons, and FuzzMatchRegexp translates `?( *( +( @(` into regexp quantifiers so that the existing differential fuzzing covers them too; `!(...)` is skipped there, as RE2 has no negation. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Add the five groups to the grammar, mirroring the Compile doc comment, and describe how an alternative is matched whole, how a `(` binds to the character before it, and which characters the innermost open construct makes special. Note that QuoteMeta escapes `(` and `)`, and that escaping either half of a group opener is enough to break it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add the bash extglob groups:
@(?(*(+(!(Closes #3. Closes #47.
This adds the five extended groups of the bash
extglobto the pattern syntax, the parser and the v1 matching engine.Syntax
A group opens with one of
?(*(+(@(!(, closes with), and a|separates its alternatives:@(a|b)?(a|b)*(a|b)+(a|b)!(a|b)An alternative is a whole pattern in its own right: it may be empty, and it may contain any term, including character classes, braces and further groups. So
@(ab|a)cmatchesabcandac,+(a*b|c)matchesabcc,src/**/!(*_test).gomatchessrc/a/b.gobut notsrc/a/b_test.go, and*(a?(b)|bc)matchesabcas the sub-stringsaandbc.Semantics, and where they are pinned
The interesting part of this feature is not the parsing but deciding what the groups mean at the edges, where bash's own behaviour is an artifact of its matcher rather than a rule. The semantics implemented here are the declarative ones:
[[ a == *!(a) ]]is false there, while the declarative reading makes it true -- the star takes the whole string and leaves the group an empty piece, which is nota. Comparing against bash 5.2 over 70,525 separator-free cases, all 122 disagreements are of that one shape. Since the quirk is not something anyone can rely on deliberately, and since a rule that can be stated in one sentence is worth more in a library than bug-compatibility with one shell's loop, this pins the rule.@(a*|b)cagainstaxxcgives the groupaxx; the*cannot swallow thecthat follows the group.!(...)never spans a separator, exactly like*. Bash has no separators at all, so there is nothing to copy here; this is the pathname-expansion behaviour, and it is what makessrc/**/!(*_test).gouseful.,and}inside a{...},|and)inside a group. So{@(a,b),c}has a literal comma inside the group and a separating one outside it, and in@(a|{b,c|d})the second bar is text. Groups and braces nest freely.(binds to the character right before it, as in bash:**(a)is a*followed by the group*(a), not a**and a literal(. Elsewhere(is an ordinary character, so(a)anda(b|c)keep matching themselves, and so do a bare)and a bare|outside a group.()|as it escapes everything else, andQuoteMetaescapes(and), so its output still compiles to the quoted text. A group left open is a*SyntaxErrorwith the reasonunclosed `(`and the offset at the end of the pattern, the same way an unclosed brace is reported.How it fits the v1 engine
The engine walks the matcher tree with backtracking, resuming from the most recent checkpoint on a mismatch. A repeating or negated group cannot be resumed the way a star is: on resuming it no longer knows where it began, and its candidate pieces depend on that beginning. So a group finds every candidate piece on entry, takes the shortest, and stores the others as restart points that carry the candidate length in an extra path entry past the group's own depth -- which is enough for the walk to pick up a repetition it did not start. Each alternative is compiled into a sub-pattern of its own, which is what makes "matched whole" fall out naturally.
The compile-time passes had to learn about the new nodes as well: tail folding into shaped matchers, terminal-star marking,
needsState,minLengthandrequiredSuffixall silently corrupt group bodies if left alone, since a group may end a pattern, follow a star, sit inside a brace alternative or contain one. Matching stays deterministic and terminates on every pattern, including groups whose alternatives match the empty string, such as*(?(a))or*(*).Performance
Matchkeeps performing zero allocations, and the existing benchmarks are unchanged within noise -- a pattern without groups compiles to the same matchers it did before. The one measurable cost is at compile time: the lexer's nesting stack replaces the old brace counter, so compiling a pattern that uses{...}or a group costs two small allocations more ({a,ab,abc}: 27 -> 29 allocs). Brace-free patterns are untouched.Tests
groups_test.go: hand-written tables per operator, plus the lexing,QuoteMetaand syntax-error cases, and the separator behaviour.TestGroupReference: a differential test against an independent recursive re-implementation of the whole syntax, over an exhaustive set of small patterns and strings with and without a separator -- 2,659 patterns and 643,478 comparisons, in under 0.2s.FuzzMatchRegexp:translateGlobnow translates?( *( +( @(into the regexp quantifiers, so the existing fuzzing covers the groups too; patterns containing!(are skipped there, since RE2 has no negation, and the reference matcher above covers them instead.syntax/lexer_test.go: token cases for the newgroup_open,group_separatorandgroup_close, including**(a),{@(a,b),c}and the escaped forms.gofmt -l .,go vet ./...andgo test ./...are clean, andgo test -fuzz=FuzzMatchRegexpran ~10M execs without a failure.