automata: fix bug in reverse suffix/inner optimization

A minimal reproducer of this bug is on a haystack of `zabb` with the
regex `.bb|b`. The `regex` crate will report a match at `2..3`, but the
correct match is `1..4`.

While this seems like a simple regex, there are a pretty specific set
of circumstances required to trigger the bug:

1. There are no prefix literals that activate a standard prefix
   literal scan.
2. There needs to be an extractable *suffix* or *inner* literal.
3. An actual match needs to be present in the haystack.
4. The regex and haystack

Crucially, note that because of (3), this bug will never lead to
`Regex::is_match` providing a false positive *or* a false negative.
This bug is strictly about leftmost-first match semantics being
incorrect in some cases and will report an incorrect match span.

(4) could do with a bit more explanation, since it's rather subtle.
Let's trace the minimal example through the regex crate's "reverse
suffix" optimization.

During compilation, there is no prefix literal that can be extracted.
The `.` defeats that class of optimization. Moreover, there is a suffix
literal in the regex. That is, all matches for `.bb|b` must end with
`b`. The regex crate sees this and will scan for matches of `b`. It will
then attempt to match the regex in reverse at each candidate match of
`b`. Let's see what happens:

* Find first occurrence of `b` at offset `2` in `zabb`.
* Start reverse confirmation step at offset `2`.
* The second alternation branch, `b` in `.bb|b`, matches at `2..3`.
* The second alternation branch is reported as the overall match.
  This happens because the first alternation branch, `.bb`, does _not_
  have a match ending at offset `3`.

The fundamental problem here is that there is an overlap between the
reverse automaton for confirming the match and the literal scan. Small
changes, even to the haystack, can result in the bug disappearing.
For example, with a haystack of `zbb`, the correct match of `0..3` is
reported. This occurs because there is a quadratic "trip wire" that
triggers in this case that causes the search to bail out and fall back
to a DFA without using any literal optimizations.

This bug also applies to the "reverse inner" optimization. This
can happen when the literal is extracted from inside the regex
as opposed to it being a suffix literal. For example, the regex
`(?:..acbb|b)a(?:c|d)` on the haystack `xzbacbbac` reported a match at
`2..5`, but the correct match is `1..9`.

Note that #1355 technically fixes this problem and is much simpler, but
in so doing, makes the reverse suffix and inner optimizations completely
ineffective.

Fixes #1354, Closes #1355

Author: BurntSushi

regex-automata/src/meta/reverse_inner.rs

@@ -50,7 +50,9 @@ use crate::{util::prefilter::Prefilter, MatchKind};
 ///
 /// Note that this assumes leftmost-first match semantics, so callers must
 /// not call this otherwise.
-pub(crate) fn extract(hirs: &[&Hir]) -> Option<(Hir, Prefilter)> {
+pub(crate) fn extract(
+    hirs: &[&Hir],
+) -> Option<(Hir, Prefilter, Vec<Vec<u8>>)> {
     if hirs.len() != 1 {
         debug!(
             "skipping reverse inner optimization since it only \
@@ -73,7 +75,7 @@ pub(crate) fn extract(hirs: &[&Hir]) -> Option<(Hir, Prefilter)> {
     // we probably wouldn't be here looking for an inner prefilter.
     for i in 1..concat.len() {
         let hir = &concat[i];
-        let pre = match prefilter(hir) {
+        let (pre, lits) = match prefilter(hir) {
             None => continue,
             Some(pre) => pre,
         };
@@ -96,17 +98,17 @@ pub(crate) fn extract(hirs: &[&Hir]) -> Option<(Hir, Prefilter)> {
         // something better and more discriminatory by looking at the entire
         // suffix. We don't do this above to avoid making this loop worst case
         // quadratic in the length of 'concat'.
-        let pre2 = match prefilter(&concat_suffix) {
-            None => pre,
-            Some(pre2) => {
+        let (pre2, lits2) = match prefilter(&concat_suffix) {
+            None => (pre, lits),
+            Some((pre2, lits2)) => {
                 if pre2.is_fast() {
-                    pre2
+                    (pre2, lits2)
                 } else {
-                    pre
+                    (pre, lits)
                 }
             }
         };
-        return Some((concat_prefix, pre2));
+        return Some((concat_prefix, pre2, lits2));
     }
     debug!(
         "skipping reverse inner optimization because a top-level \
@@ -124,7 +126,7 @@ pub(crate) fn extract(hirs: &[&Hir]) -> Option<(Hir, Prefilter)> {
 ///
 /// Note that this assumes leftmost-first match semantics, so callers must
 /// not call this otherwise.
-fn prefilter(hir: &Hir) -> Option<Prefilter> {
+fn prefilter(hir: &Hir) -> Option<(Prefilter, Vec<Vec<u8>>)> {
     let mut extractor = literal::Extractor::new();
     extractor.kind(literal::ExtractKind::Prefix);
     let mut prefixes = extractor.extract(hir);
@@ -148,9 +150,10 @@ fn prefilter(hir: &Hir) -> Option<Prefilter> {
         prefixes.len(),
         prefixes
     );
-    prefixes
-        .literals()
-        .and_then(|lits| Prefilter::new(MatchKind::LeftmostFirst, lits))
+    let lits = prefixes.literals()?;
+    let pre = Prefilter::new(MatchKind::LeftmostFirst, lits)?;
+    let lits = lits.iter().map(|lit| lit.as_bytes().to_vec()).collect();
+    Some((pre, lits))
 }
 
 /// Looks for a "top level" HirKind::Concat item in the given HIR. This will