The Default Rust Regex Matcher

grep-regex wraps Rust’s regex engine in RegexMatcher, while retaining ripgrep-specific configuration, candidate-line acceleration, and byte-level metadata.

It exists to present regex matching through ripgrep’s generic Matcher interface while allowing the searcher to use optimized hints before running the full expression. The matcher stores the compiled regex, an optional faster regex, and bytes that cannot occur in a match.

Sources: crates/regex/src/matcher.rs:367-380

Core concepts

RegexMatcherBuilder

A builder is the caller-facing object that accumulates regex behavior before compilation. RegexMatcherBuilder stores a Config, and new initializes that configuration with Config::default().

Sources: crates/regex/src/matcher.rs:24-26, crates/regex/src/matcher.rs:36-38

ConfiguredHIR

A configured HIR is the parsed and translated representation that sits between configuration and the compiled Rust regex. ConfiguredHIR contains both the effective Config and the translated Hir.

Sources: crates/regex/src/config.rs:158-161

RegexMatcher

A matcher is the compiled implementation exposed to the rest of ripgrep. RegexMatcher stores the configured regex, an optional fast_line_regex, and a non_matching_bytes set.

Sources: crates/regex/src/matcher.rs:367-380

InnerLiterals

Inner literals are candidate byte strings extracted from a pattern so the matcher can locate promising lines before confirming a match with the original regex.

Sources: crates/regex/src/matcher.rs:53-85

Banned byte

A banned byte is a configured byte that the compiled expression must not match. ban::check recursively rejects literals and single-range classes containing that byte.

Sources: crates/regex/src/ban.rs:8-53

How a pattern becomes a Matcher

The build path converts one or more patterns into HIR, applies matcher semantics, compiles the HIR, and assembles optimization metadata.

build is the one-pattern convenience method; it delegates to build_many, which first asks Config to produce a ConfiguredHIR.

ConfiguredHIR::new either constructs an alternation of fixed-string literals or parses the combined pattern, translates it to HIR, applies the configured case and syntax options, checks the banned byte, and strips the configured line terminator when required.

build_many then wraps word or whole-line behavior around the HIR, compiles it with to_regex, computes non-matching bytes, asks InnerLiterals for one accelerated regex, and stores all results in RegexMatcher. Whole-line mode takes precedence over word mode.

The adapter exposes ordinary matching through methods such as find_at, try_find_iter, captures_at, and shortest_match_at. These delegate to the compiled Regex and convert engine matches into ripgrep Match values or capture state.

Sources: crates/regex/src/matcher.rs:45-47, crates/regex/src/matcher.rs:53-85, crates/regex/src/config.rs:166-229, crates/regex/src/matcher.rs:414-421, crates/regex/src/matcher.rs:439-455, crates/regex/src/matcher.rs:458-468, crates/regex/src/matcher.rs:471-478

How literal extraction speeds matching

Literal extraction walks the HIR and produces a bounded sequence of exact or inexact literals. The extractor handles literals, Unicode and byte classes, repetitions, captures, concatenations, and alternations.

For concatenations, extraction crosses adjacent literal sequences; when a segment ends the useful prefix, the sequence is marked as no longer a prefix, allowing the cross operation to choose the appropriate prefix-or-suffix behavior. Empty or especially good sequences can stop further work early.

Alternations union the extracted sequences until the result becomes infinite, while repetitions preserve exactness only where the repetition semantics make that safe; otherwise they mark the result inexact.

The extractor is deliberately bounded. It limits class expansion, repetition expansion, individual literal length, and total literal count. Cross products exceeding the total limit become infinite, and unions may trim literals before giving up on a finite result.

After extraction, extract_untagged optimizes the sequence for prefix preference and discards it by making it infinite when the result is not considered good enough. A finite sequence is considered good only when it has no poisonous literal, has a minimum length, and stays within size thresholds; the stricter “really good” test requires literals of at least length three and no more than eight literals.

one_regex turns the retained literals into a Rust Regex over an HIR alternation. The matcher uses that regex to search for a candidate line; if it finds one, the result is only a candidate offset, and the original regex remains responsible for confirmation. Without a fast regex, the matcher returns a confirmed result from the normal shortest-match path.

The optimization is conditional: inner literal extraction is skipped without a line terminator, when the main regex is already accelerated unless Unicode word boundaries may force a slower path, and when the HIR is already an alternation of literals.

The following data flow shows why an extracted literal is safe as a prefilter: it identifies possible lines, but does not replace the full match.

Sources: crates/regex/src/literal.rs:169-189, crates/regex/src/literal.rs:196-226, crates/regex/src/literal.rs:231-247, crates/regex/src/literal.rs:264-315, crates/regex/src/literal.rs:131-136, crates/regex/src/literal.rs:379-393, crates/regex/src/literal.rs:398-430, crates/regex/src/literal.rs:151-166, crates/regex/src/literal.rs:553-565, crates/regex/src/literal.rs:570-577, crates/regex/src/literal.rs:106-122, crates/regex/src/matcher.rs:491-506, crates/regex/src/literal.rs:54-92

Why banned syntax and bytes are enforced

The configured banned byte is checked after HIR translation. ban::check rejects a literal containing the byte, rejects a one-byte Unicode or byte class whose range contains it, and recursively checks repetitions, captures, concatenations, and alternations.

This enforces a simple exclusion invariant: a pattern containing the forbidden byte is rejected rather than compiled. The tests show that the check catches direct literals, nested expressions, alternations, repetitions, and class forms, while allowing a negated class or a larger class that also contains other bytes.

Line terminators use a related but separate enforcement path. When a line terminator is configured, strip_from_match removes that byte from classes and recursively rewrites compound HIR nodes; a literal or class that would become invalid produces an error. CRLF removes both carriage return and newline.

The reason for this restriction is operational: the matcher is line-oriented, so the configured line boundary must not be accepted as ordinary pattern content. The tests demonstrate that a pattern containing the configured newline is rejected, while a pattern using \s behaves differently depending on whether a line terminator is configured.

The matcher also computes non_matching_bytes by starting with all bytes and removing bytes represented by the HIR. This metadata is exposed through the matcher interface for downstream search optimizations.

Sources: crates/regex/src/ban.rs:8-53, crates/regex/src/ban.rs:67-82, crates/regex/src/strip.rs:39-49, crates/regex/src/strip.rs:55-119, crates/regex/src/matcher.rs:573-584, crates/regex/src/matcher.rs:589-596, crates/regex/src/non_matching.rs:10-14, crates/regex/src/matcher.rs:481-483

How it connects

The searcher consumes a generic Matcher; the grep-regex crate supplies the Rust-regex implementation, while the searcher applies a matcher to input bytes and reports results to a sink.

The high-level facade re-exports grep_regex as regex, alongside the matcher, searcher, printer, and optional PCRE2 crates. The core search code represents the selected engine as PatternMatcher::RustRegex and optionally PatternMatcher::PCRE2.

For the surrounding architecture, start with The Matcher Trait and Internal Iteration, then follow the hand-off to The Searcher Core. Engine selection and the alternative implementation are described in The PCRE2 Matcher, while the broader crate wiring is covered by Repository and Crate Map.

Sources: crates/searcher/src/lib.rs:7-24, crates/grep/src/lib.rs:15-21, crates/core/search.rs:191-197

Key takeaways

• RegexMatcherBuilder turns patterns and configuration into a ConfiguredHIR, a compiled Regex, optimization metadata, and a RegexMatcher.
• Literal extraction handles HIR structure, keeps bounded exact or inexact candidates, and may produce a fast_line_regex.
• Candidate-line detection is only a prefilter; the full regex still confirms the match.
• ban::check rejects configured bytes recursively, while line-terminator stripping rejects patterns that would violate line-oriented matching.
• The matcher exposes Rust regex behavior through ripgrep’s generic matching operations and metadata.

Sources: crates/regex/src/matcher.rs:53-85, crates/regex/src/literal.rs:169-189, crates/regex/src/literal.rs:151-166, crates/regex/src/literal.rs:106-122, crates/regex/src/matcher.rs:491-506, crates/regex/src/ban.rs:8-53, crates/regex/src/strip.rs:55-119, crates/regex/src/matcher.rs:414-421, crates/regex/src/matcher.rs:439-455, crates/regex/src/matcher.rs:481-483