The Matcher Trait and Internal Iteration

The Matcher trait is the low-level contract that lets ripgrep search bytes through different regular-expression engines without coupling the searcher to one implementation. It describes finding matches, working with capture groups, reporting errors, and exposing search-related hints.

The design exists so the searcher can drive matching directly and receive results through callbacks rather than materializing a separate iterator. That push model lets the searcher retain control while it consumes input and applies a matcher.

Sources: crates/matcher/src/lib.rs:546-1131, crates/searcher/src/sink.rs:64-74, crates/searcher/src/lib.rs:7-15

Core concepts

Backend contract

A backend is a type implementing Matcher; it supplies a concrete capture representation through type Captures, an error type through type Error, a first-match operation through find_at, and capture storage through new_captures.

Sources: crates/matcher/src/lib.rs:546-1131

Match span

A match is a byte range represented by Match, whose start and end fields identify the matched region. The iteration loop explicitly recognizes an empty match when m.start == m.end.

Sources: crates/matcher/src/lib.rs:69-72, crates/matcher/src/lib.rs:704-713

Capture collection

A capture collection is a Captures value that reports its group count with len, returns a group span with get, and treats capture group 0 as the overall match.

Sources: crates/matcher/src/lib.rs:379-461

Internal iteration

Internal iteration means the search operation owns the loop and pushes each result into a callback supplied by the caller. In the ordinary find_iter API, that callback returns bool; true continues and false stops.

Sources: crates/searcher/src/sink.rs:64-74, crates/matcher/src/lib.rs:625-637

Replacement interpolation

Interpolation turns replacement bytes containing $ references into output bytes by resolving numeric or named capture references against the current capture collection and haystack.

Sources: crates/matcher/src/interpolate.rs:14-56, crates/matcher/src/lib.rs:379-461

What Matcher requires and provides

The core required operations establish the backend boundary. find_at searches haystack from byte offset at, returns the first match after that point, and keeps offsets relative to the beginning of haystack; new_captures creates capture storage suitable for the capture APIs.

Capture support is optional. A backend without captures uses NoCaptures; the default capture_count returns 0, and the default capture_index returns None. A capture-aware backend reports its total groups and maps names to indices.

The trait also exposes convenience operations built around those primitives:

The offset-free methods delegate to offset-aware or fallible variants, while the offset-aware loops perform the repeated search and callback calls.

The offset-aware loop repeatedly calls find_at, stops when there is no match, invokes matched, and returns when that callback says to stop or reports an error.

let m = match self.find_at(haystack, last_end)? {
    None => return Ok(Ok(())),
    Some(m) => m,
};
match matched(m) {
    Ok(true) => continue,
    Ok(false) => return Ok(Ok(())),
    Err(err) => return Ok(Err(err)),
}

The loop owns search progress and callback control; the callback owns what to do with each match.

The loop also handles empty matches explicitly. After an empty match it advances the next search position by one byte, and it skips another empty match at the same end position, preventing the search from repeating forever. The capture loop applies the same policy after reading group 0.

Sources: crates/matcher/src/lib.rs:561-583, crates/matcher/src/lib.rs:585-613, crates/matcher/src/lib.rs:621-637, crates/matcher/src/lib.rs:669-678, crates/matcher/src/lib.rs:740-746, crates/matcher/src/lib.rs:753-763, crates/matcher/src/lib.rs:926-937, crates/matcher/src/lib.rs:692-732, crates/matcher/src/lib.rs:820-861

Why the API pushes instead of returning an iterator

A pull iterator would make the caller ask for the next match. This crate instead makes the search operation drive execution and call the supplied function, matching the broader searcher’s documented “push” or “internal” iteration model. The stated reason is the complexity of the searcher implementation: the searcher retains control while it consumes input and applies a matcher.

At the matcher level, the callback is also a control boundary. The ordinary callback returns false to stop iteration, while try_find_iter_at can return a caller-defined callback error separately from the matcher’s own Self::Error.

Capture iteration uses the same model but reuses a mutable Self::Captures value. The loop repeatedly calls captures_at, obtains group 0, and invokes matched with the current captures.

The public convenience methods preserve this call direction: find_iter calls find_iter_at, try_find_iter calls try_find_iter_at, and the capture variants call their offset-aware counterparts.

Sources: crates/searcher/src/sink.rs:64-74, crates/searcher/src/lib.rs:7-15, crates/matcher/src/lib.rs:625-637, crates/matcher/src/lib.rs:692-732, crates/matcher/src/lib.rs:820-861, crates/matcher/src/lib.rs:669-678, crates/matcher/src/lib.rs:740-746, crates/matcher/src/lib.rs:753-763

How capture interpolation becomes replacement output

A replacement template is processed by interpolate. It scans until it finds $, copies ordinary bytes to dst, treats $$ as a literal dollar sign, and parses a capture reference when the $ begins a valid name or number.

find_cap_ref accepts digits, ASCII letters, and underscores after $. It also accepts braces, so ${1} is a bounded reference. A numeric token becomes Ref::Number; any other valid token becomes Ref::Named.

The longest valid token is consumed before lookup. Therefore $1a is parsed as one name, while braces let the caller separate ${1} from following text. If a reference is malformed, interpolate preserves the $ as literal output and continues scanning.

For a numeric reference, interpolate calls the supplied append function with the numeric capture index. For a named reference, it first calls name_to_index; only an existing index is appended. The capture implementation then obtains that group with get and extends dst with the corresponding slice of haystack.

A missing capture contributes no bytes, which is the empty-string behavior required for invalid names or indices. The same mechanism supports $0 for the overall match because capture group 0 is defined as that match.

Replacement over a haystack preserves the bytes between matches. replace copies the unmatched prefix, invokes its callback for each Match, updates the last-match end, and finally copies the trailing suffix. The capture-aware form performs the same operation around captures_iter_at, using caps.get(0) for each overall match.

Sources: crates/matcher/src/interpolate.rs:14-56, crates/matcher/src/interpolate.rs:97-134, crates/matcher/src/interpolate.rs:138-143, crates/matcher/src/lib.rs:379-461, crates/core/flags/defs.rs:6374-6390, crates/matcher/src/lib.rs:902-919, crates/matcher/src/lib.rs:948-968

How it connects

The searcher consumes bytes, applies a Matcher, and reports search results to a Sink; this trait is therefore the boundary between regex engines and line-oriented search. See The Searcher Core for the consumer that drives this boundary.

The standard Rust regex backend implements this contract through RegexMatcher, while the PCRE2 crate provides another RegexMatcher implementation. See The Default Rust Regex Matcher and The PCRE2 Matcher for backend-specific behavior.

Replacement bytes can contain indexed or named capture references, and the printer exposes configuration for those replacement bytes. See Output Printers for how replacement results become user-facing output.

Sources: crates/searcher/src/lib.rs:7-15, crates/regex/src/lib.rs:1-18, crates/pcre2/src/lib.rs:1-16, crates/printer/src/standard.rs:280-289

Key takeaways

• Matcher standardizes first-match search, capture storage, errors, callbacks, replacement, and search hints.
• Internal iteration keeps the matcher in control and lets callbacks continue, stop, or fail.
• Empty matches advance by one byte so repeated searches make progress.
• Interpolation parses $ references, resolves names or indices, and copies capture slices into dst.
• Capture-aware replacement preserves unmatched bytes around each match while the callback supplies replacement content.

Sources: crates/matcher/src/lib.rs:546-1131, crates/matcher/src/lib.rs:692-732, crates/matcher/src/interpolate.rs:14-56, crates/matcher/src/lib.rs:379-461, crates/matcher/src/lib.rs:948-968