Skip to content

Add Bash extglob groups and update documentation - #77

Open
sawy3r wants to merge 2 commits into
gobwas:masterfrom
sawy3r:extglob
Open

sawy3r wants to merge 2 commits into
gobwas:masterfrom
sawy3r:extglob

Conversation

@sawy3r

@sawy3r sawy3r commented Sep 17, 2026

Copy link
Copy Markdown

Add the bash extglob groups: @( ?( *( +( !(

Closes #3. Closes #47.

This adds the five extended groups of the bash extglob to the pattern syntax, the parser and the v1 matching engine.

Syntax

A group opens with one of ?( *( +( @( !(, closes with ), and a | separates its alternatives:

Term Matches
@(a|b) exactly one of the alternatives
?(a|b) at most one of them
*(a|b) any number of them, including none
+(a|b) any number of them, at least one
!(a|b) any sequence of non-separator characters that none of them matches

An alternative is a whole pattern in its own right: it may be empty, and it may contain any term, including character classes, braces and further groups. So @(ab|a)c matches abc and ac, +(a*b|c) matches abcc, src/**/!(*_test).go matches src/a/b.go but not src/a/b_test.go, and *(a?(b)|bc) matches abc as the sub-strings a and bc.

Semantics, and where they are pinned

The interesting part of this feature is not the parsing but deciding what the groups mean at the edges, where bash's own behaviour is an artifact of its matcher rather than a rule. The semantics implemented here are the declarative ones:

  • Every split of the string among the terms counts. A pattern matches when the string can be split so that every term matches its piece, for some split. Bash disagrees in exactly one shape: its star loop never tries the rest of the pattern against the empty remainder, so [[ a == *!(a) ]] is false there, while the declarative reading makes it true -- the star takes the whole string and leaves the group an empty piece, which is not a. Comparing against bash 5.2 over 70,525 separator-free cases, all 122 disagreements are of that one shape. Since the quirk is not something anyone can rely on deliberately, and since a rule that can be stated in one sentence is worth more in a library than bug-compatibility with one shell's loop, this pins the rule.
  • A star inside an alternative reaches only to the end of its own piece. Each alternative is matched whole, from its first character to its last, so @(a*|b)c against axxc gives the group axx; the * cannot swallow the c that follows the group.
  • !(...) never spans a separator, exactly like *. Bash has no separators at all, so there is nothing to copy here; this is the pathname-expansion behaviour, and it is what makes src/**/!(*_test).go useful.
  • The innermost open construct decides which characters are special: , and } inside a {...}, | and ) inside a group. So {@(a,b),c} has a literal comma inside the group and a separating one outside it, and in @(a|{b,c|d}) the second bar is text. Groups and braces nest freely.
  • A ( binds to the character right before it, as in bash: **(a) is a * followed by the group *(a), not a ** and a literal (. Elsewhere ( is an ordinary character, so (a) and a(b|c) keep matching themselves, and so do a bare ) and a bare | outside a group.
  • A backslash escapes ( ) | as it escapes everything else, and QuoteMeta escapes ( and ), so its output still compiles to the quoted text. A group left open is a *SyntaxError with the reason unclosed `(` and the offset at the end of the pattern, the same way an unclosed brace is reported.

How it fits the v1 engine

The engine walks the matcher tree with backtracking, resuming from the most recent checkpoint on a mismatch. A repeating or negated group cannot be resumed the way a star is: on resuming it no longer knows where it began, and its candidate pieces depend on that beginning. So a group finds every candidate piece on entry, takes the shortest, and stores the others as restart points that carry the candidate length in an extra path entry past the group's own depth -- which is enough for the walk to pick up a repetition it did not start. Each alternative is compiled into a sub-pattern of its own, which is what makes "matched whole" fall out naturally.

The compile-time passes had to learn about the new nodes as well: tail folding into shaped matchers, terminal-star marking, needsState, minLength and requiredSuffix all silently corrupt group bodies if left alone, since a group may end a pattern, follow a star, sit inside a brace alternative or contain one. Matching stays deterministic and terminates on every pattern, including groups whose alternatives match the empty string, such as *(?(a)) or *(*).

Performance

Match keeps performing zero allocations, and the existing benchmarks are unchanged within noise -- a pattern without groups compiles to the same matchers it did before. The one measurable cost is at compile time: the lexer's nesting stack replaces the old brace counter, so compiling a pattern that uses {...} or a group costs two small allocations more ({a,ab,abc}: 27 -> 29 allocs). Brace-free patterns are untouched.

Tests

  • groups_test.go: hand-written tables per operator, plus the lexing, QuoteMeta and syntax-error cases, and the separator behaviour.
  • TestGroupReference: a differential test against an independent recursive re-implementation of the whole syntax, over an exhaustive set of small patterns and strings with and without a separator -- 2,659 patterns and 643,478 comparisons, in under 0.2s.
  • FuzzMatchRegexp: translateGlob now translates ?( *( +( @( into the regexp quantifiers, so the existing fuzzing covers the groups too; patterns containing !( are skipped there, since RE2 has no negation, and the reference matcher above covers them instead.
  • syntax/lexer_test.go: token cases for the new group_open, group_separator and group_close, including **(a), {@(a,b),c} and the escaped forms.

gofmt -l ., go vet ./... and go test ./... are clean, and go test -fuzz=FuzzMatchRegexp ran ~10M execs without a failure.

sawy3r and others added 2 commits September 18, 2026 05:34
The five extended groups of the bash extglob become terms of their own:
`@(a|b)` matches exactly one of its alternatives, `?(a|b)` at most one,
`*(a|b)` any number of them, `+(a|b)` at least one, and `!(a|b)` matches a
sub-string -- without separators, like a `*` -- that none of them matches.
Requested in gobwas#3; `!(...)` also covers gobwas#47.

Each alternative is a pattern in its own right, compiled as a sub-pattern
and matched whole, so a `*` inside an alternative reaches only to the end
of its own sub-string and never past it into what follows the group. In
the engine a group finds every candidate sub-string on entry, takes the
shortest and stores the rest as restart points carrying the candidate
length in an extra path entry past the group's own depth, so that the
backtracking walk can resume a repetition it did not begin. The
compile-time passes -- the shaped terminal matchers, the terminal-star
marking, needsState, minLength and requiredSuffix -- all learn about the
new nodes, since a group may end a pattern, follow a star, sit inside a
brace alternative or contain one.

In the lexer a `(` opens a group only right after one of the five operator
characters, and it binds to the character right before it, so `**(a)` is a
`*` followed by the group `*(a)`, as in bash. The nesting is now a stack
rather than a brace counter: the innermost open construct decides whether
`,` `}` `|` `)` are special, so the groups and the `{...}` alternatives
nest freely. Outside of a group `(`, `)` and `|` are plain characters, a
backslash escapes them as it escapes everything else, and QuoteMeta
escapes `(` and `)` so that its output still compiles to the quoted text.

Match stays zero-allocation and deterministic, and terminates on every
pattern, including the groups whose alternatives match the empty string
such as `*(?(a))` or `*(*)`.

The groups are tested against a reference matcher that re-implements the
syntax independently, over 643k pattern/string/separator comparisons, and
FuzzMatchRegexp translates `?( *( +( @(` into regexp quantifiers so that
the existing differential fuzzing covers them too; `!(...)` is skipped
there, as RE2 has no negation.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Add the five groups to the grammar, mirroring the Compile doc comment, and
describe how an alternative is matched whole, how a `(` binds to the
character before it, and which characters the innermost open construct
makes special. Note that QuoteMeta escapes `(` and `)`, and that escaping
either half of a group opener is enough to break it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use extending glob syntax ! works with only character range or a single character.

1 participant