The Tokora Guide

Tokora writes parsers as plain Rust functions with the reach of a combinator library. It lexes on demand as the parser pulls tokens — there is no separate tokenize pass — and gives you typed, structured errors with rich diagnostics, explicit backtracking and recovery, streaming over partial input, and optional lossless concrete syntax trees that preserve every byte (whitespace and comments included) for formatters, refactoring tools, and language servers.

This guide is the tour. The reference documentation explains each API in isolation; the guide tells the story in order, building Calc — a tiny calculator language with variables — end to end, then reusing it to introduce the harder machinery.

program := stmt+
stmt    := "let" ident "=" expr ";"        bind a variable
         | "print" expr ("," expr)* ";"    print one or more values
         | expr ";"                        evaluate and discard
expr    := integers, variables, + - * / ^, unary -, ( ) grouping

The five parts

• Part I — Getting Started (this introduction) frames tokora and sketches Calc.
• Part II — Core Concepts is the tutorial: ten chapters that build Calc, from tokens and the lexer and first parsers through composition, deterministic choice, Pratt expressions, backtracking, diagnostics, recovery, partial input, and testing. Read it to build a parser.
• Part III — Design & Architecture is the internals: how the parse-while-lexing engine, the checkpoint and rewind timeline, the atomic emitter, the event-stream CST engine, and the source and slice layer actually work. Read it to understand the design, or before contributing.
• Part IV — Recipes & Applied Parsers puts it together: the anatomy of a real parser, a custom (non-logos) lexer, and four full walkthroughs — calculator, S-expressions, JSON, and C expressions — capped by lossless CSTs with Rowan. Read it for worked examples.
• Part V — Reference is the catalog: the combinator and atom reference; the errors, emitters, and context model; the Pratt reference; the reusable AST building blocks; and the vocabulary, macros, and feature flags. Read it to look something up.

Two topics ride a feature flag: Testing (Part II) needs conformance, and both Lossless CSTs (Part IV) and the event-stream CST engine (Part III) need rowan.

How to read this guide

Every non-ignored Rust fence is a doctest — the suite compiles and runs it, so the examples cannot quietly drift from the API. Later chapters may hide reduced token and error definitions to keep the visible code focused (expand an example in the HTML docs to see them). Chapters build on each other, but each states what it teaches up front, so you can jump in anywhere. Pick a path:

• New to tokora — read Part I, work Part II in order, then open the matching walkthrough in Part IV.
• Using tokora as a library — Part V is the lookup catalog; its entries point back to the chapter that teaches each API.
• Contributing, or just curious how it works — Part III is the internals tour; it assumes Part II.

The four examples/ programs in the repository (json, calculator, s_expression, and c_expression) are canonical complete programs; the applied chapters explain how to reproduce their structure without copying their source into the guide.

1. Tokens and the lexer

Calc’s source text becomes a stream of tokens before any parsing happens. This chapter defines that token type, wires it to a lexer, and states the contract the rest of the guide (and the crate) relies on.

The token type — data and kind

A tokora token is two types working together, connected by the Token trait:

• the token itself (Tok below) carries payloads — Int(i64) holds its value;
• its Kind (TokKind below) is a payload-free, Copy discriminant.

The split matters later: dispatch tables (chapter 4) and “expected one of …” diagnostics (chapter 7) need to name token classes without inventing payload values, and that is exactly what a kind is. Token::is_trivia marks tokens (like whitespace or comments) that carry no syntax; Calc has none because the lexer skips whitespace outright — languages that keep trivia tokens instead skip them with padded at the parser level.

The lexer

Any type implementing Lexer can drive tokora’s parsers. Calc does not hand-roll one: the LogosLexer adapter turns any logos-derived token enum into a conforming lexer, so the whole lexer is the #[derive(Logos)] block below. Keywords are plain #[token] patterns — logos resolves the let-versus-identifier overlap by longest match, then pattern priority.

use tokora::{
  Lexer, SimpleSpan, Token as TokenT,
  logos::{self, Logos},
};

// The lexer-level error: what lexing yields for bytes that are no token at all.
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;

impl From<()> for LexError {
  fn from(_: ()) -> Self {
    LexError
  }
}

#[derive(Debug, Clone, PartialEq, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Tok {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))]
  Int(i64),
  #[token("let")]
  Let,
  #[token("print")]
  Print,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")]
  Ident,
  #[token("+")]
  Plus,
  #[token("-")]
  Minus,
  #[token("*")]
  Star,
  #[token("/")]
  Slash,
  #[token("^")]
  Caret,
  #[token("=")]
  Assign,
  #[token(";")]
  Semi,
  #[token(",")]
  Comma,
  #[token("(")]
  LParen,
  #[token(")")]
  RParen,
}

// The payload-free discriminant. `Display` is what diagnostics print.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokKind {
  Int,
  Let,
  Print,
  Ident,
  Plus,
  Minus,
  Star,
  Slash,
  Caret,
  Assign,
  Semi,
  Comma,
  LParen,
  RParen,
}

impl core::fmt::Display for TokKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Self::Int => "integer",
      Self::Let => "`let`",
      Self::Print => "`print`",
      Self::Ident => "identifier",
      Self::Plus => "`+`",
      Self::Minus => "`-`",
      Self::Star => "`*`",
      Self::Slash => "`/`",
      Self::Caret => "`^`",
      Self::Assign => "`=`",
      Self::Semi => "`;`",
      Self::Comma => "`,`",
      Self::LParen => "`(`",
      Self::RParen => "`)`",
    })
  }
}

impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    match self {
      Tok::Int(n) => write!(f, "{n}"),
      other => core::fmt::Display::fmt(&other.kind(), f),
    }
  }
}

// The bridge: tokora's `Token` trait names the kind and the lexer error type.
impl TokenT<'_> for Tok {
  type Kind = TokKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;

  fn kind(&self) -> TokKind {
    match self {
      Tok::Int(_) => TokKind::Int,
      Tok::Let => TokKind::Let,
      Tok::Print => TokKind::Print,
      Tok::Ident => TokKind::Ident,
      Tok::Plus => TokKind::Plus,
      Tok::Minus => TokKind::Minus,
      Tok::Star => TokKind::Star,
      Tok::Slash => TokKind::Slash,
      Tok::Caret => TokKind::Caret,
      Tok::Assign => TokKind::Assign,
      Tok::Semi => TokKind::Semi,
      Tok::Comma => TokKind::Comma,
      Tok::LParen => TokKind::LParen,
      Tok::RParen => TokKind::RParen,
    }
  }

  fn is_trivia(&self) -> bool {
    false
  }
}

// The whole lexer: one type alias over the logos adapter.
type CalcLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;

// Drive it by hand once — parsers will do this for us from chapter 2 on.
let mut lexer = CalcLexer::new("let answer = 6 * 7 ;");
let mut tokens = Vec::new();
while let Some(result) = lexer.lex() {
  let tok = result.expect("every byte of this source belongs to a token");
  // `span()` and `slice()` describe the token just lexed — the slice borrows
  // straight from the source, no copy.
  tokens.push((tok.kind(), lexer.slice(), lexer.span()));
}

assert_eq!(tokens[0], (TokKind::Let, "let", SimpleSpan::new(0, 3)));
assert_eq!(tokens[1], (TokKind::Ident, "answer", SimpleSpan::new(4, 10)));
assert_eq!(tokens[2], (TokKind::Assign, "=", SimpleSpan::new(11, 12)));
assert_eq!(
  tokens[3..]
    .iter()
    .map(|(k, _, _)| *k)
    .collect::<Vec<_>>(),
  [TokKind::Int, TokKind::Star, TokKind::Int, TokKind::Semi],
);

The lexer contract, in brief

Parsers do more than drain the lexer forward: they peek, checkpoint, rewind, and re-lex. That machinery is only sound if the lexer behaves like a pure function of source, position, and state — the same position always yields the same token, spans never move backward, exhaustion is sticky, and a composite token (a string literal, say) owns every byte it spans. The full, normative statement lives in the Lexer contract; LogosLexer upholds it for you, and chapter 10 shows the conformance kit that checks a hand-rolled lexer against it mechanically.

One consequence worth internalizing now: because the lexer is deterministic and the source is immutable, rewinding is cheap — a checkpoint is a snapshot, not a journal. That is what makes the backtracking of chapter 6 and the recovery of chapter 8 affordable.

Next: chapter 2 writes the first parsers over this token stream.

2. First parsers

A tokora parser is a plain function over an InputRef: pull a token, decide, pull the next. Tokora does not eagerly materialize a whole token stream; InputRef pulls from the lexer on demand and stages/caches tokens for explicit lookahead and backtracking. That parse-while-lexing architecture is why every signature in this guide is generic over the input’s lifetime 'inp.

The two primitives this chapter leans on:

• next — consume the next token unconditionally (Ok(None) at end of input);
• try_expect — examine the next token from the cache or lexer and either commit it (the predicate matched, you get the token) or leave it staged (Ok(None)). This one-token peek-or-take is the workhorse of hand-written parsers.

A typed error and the Err channel

Parsers return Result<O, E> where E is your error type. Failures reach it through two routes: your own code returns it directly, and the crate’s machinery emits structured errors — UnexpectedTokenOf when a token mismatches, UnexpectedEot at a premature end, the token’s lexer error for unlexable bytes — through the configured Emitter. The default emitter, Fatal, converts the first emission into E via From and unwinds the parse: fail-fast, the right default for a REPL. Chapter 7 swaps in a collecting emitter without touching the parser. Your error type just needs the matching From impls (that is the FromEmitterError bound the entry points ask for).

The fluent entry points

Parser::new + apply wrap a parser function with a default fail-fast context; the Parse trait then offers parse, parse_str, parse_slice, and parse_with_state. Behind their respective source features, the same trait also provides parse_bytes, parse_bstr, and parse_hipstr.

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, PartialEq, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Tok {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))]
  Int(i64),
  #[token("let")] Let,
  #[token("print")] Print,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[token("+")] Plus,
  #[token("-")] Minus,
  #[token("*")] Star,
  #[token("/")] Slash,
  #[token("^")] Caret,
  #[token("=")] Assign,
  #[token(";")] Semi,
  #[token(",")] Comma,
  #[token("(")] LParen,
  #[token(")")] RParen,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokKind { Int, Let, Print, Ident, Plus, Minus, Star, Slash, Caret, Assign, Semi, Comma, LParen, RParen }
impl core::fmt::Display for TokKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Self::Int => "integer", Self::Let => "`let`", Self::Print => "`print`",
      Self::Ident => "identifier", Self::Plus => "`+`", Self::Minus => "`-`",
      Self::Star => "`*`", Self::Slash => "`/`", Self::Caret => "`^`",
      Self::Assign => "`=`", Self::Semi => "`;`", Self::Comma => "`,`",
      Self::LParen => "`(`", Self::RParen => "`)`",
    })
  }
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    match self {
      Tok::Int(n) => write!(f, "{n}"),
      other => core::fmt::Display::fmt(&other.kind(), f),
    }
  }
}
impl TokenT<'_> for Tok {
  type Kind = TokKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokKind {
    match self {
      Tok::Int(_) => TokKind::Int, Tok::Let => TokKind::Let, Tok::Print => TokKind::Print,
      Tok::Ident => TokKind::Ident, Tok::Plus => TokKind::Plus, Tok::Minus => TokKind::Minus,
      Tok::Star => TokKind::Star, Tok::Slash => TokKind::Slash, Tok::Caret => TokKind::Caret,
      Tok::Assign => TokKind::Assign, Tok::Semi => TokKind::Semi, Tok::Comma => TokKind::Comma,
      Tok::LParen => TokKind::LParen, Tok::RParen => TokKind::RParen,
    }
  }
  fn is_trivia(&self) -> bool { false }
}
type CalcLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
use tokora::{
  Emitter, InputRef, Parse, ParseContext, Parser,
  error::{UnexpectedEot, token::UnexpectedTokenOf},
};

// Calc's parse error. One variant per failure family; the `From` impls are
// what let the emitter's structured errors collapse into it.
#[derive(Debug, Clone, PartialEq)]
enum CalcError {
  Lex,           // bytes that are no token at all
  Unexpected,    // a wrong token in a right place
  UnexpectedEnd, // input ended mid-statement
}

impl From<LexError> for CalcError {
  fn from(_: LexError) -> Self {
    CalcError::Lex
  }
}
impl<'inp> From<UnexpectedTokenOf<'inp, CalcLexer<'inp>>> for CalcError {
  fn from(_: UnexpectedTokenOf<'inp, CalcLexer<'inp>>) -> Self {
    CalcError::Unexpected
  }
}
impl<O, Lang: ?Sized, Set: Clone + 'static> From<UnexpectedEot<O, Lang, Set>> for CalcError {
  fn from(_: UnexpectedEot<O, Lang, Set>) -> Self {
    CalcError::UnexpectedEnd
  }
}
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang>
  for CalcError
{
  fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self {
    CalcError::UnexpectedEnd
  }
}

/// Parses `let <ident> = <int> ;` and returns the binding.
///
/// The signature is the crate's idiom: generic over the parse context `Ctx`,
/// pinning only the emitter's error type. Callers choose the emitter.
fn parse_let<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<(&'inp str, i64), CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  if inp.try_expect(|t| matches!(t.data(), Tok::Let))?.is_none() {
    return Err(CalcError::Unexpected);
  }
  if inp.try_expect(|t| matches!(t.data(), Tok::Ident))?.is_none() {
    return Err(CalcError::Unexpected);
  }
  // Zero-copy: `slice()` is the just-consumed token's text, borrowed from the source.
  let name = inp.slice();
  if inp.try_expect(|t| matches!(t.data(), Tok::Assign))?.is_none() {
    return Err(CalcError::Unexpected);
  }
  let value = match inp.next()? {
    Some(tok) => match tok.into_data() {
      Tok::Int(n) => n,
      _ => return Err(CalcError::Unexpected),
    },
    None => return Err(CalcError::UnexpectedEnd),
  };
  if inp.try_expect(|t| matches!(t.data(), Tok::Semi))?.is_none() {
    return Err(CalcError::Unexpected);
  }
  Ok((name, value))
}

// The fluent entry: wrap the function, hand it a source.
let binding = Parser::new().apply(parse_let).parse_str("let answer = 42 ;");
assert_eq!(binding, Ok(("answer", 42)));

// The `Err` channel carries the typed error out.
let missing_eq = Parser::new().apply(parse_let).parse_str("let answer 42 ;");
assert_eq!(missing_eq, Err(CalcError::Unexpected));

The parse context

Every signature so far has ended with the same two lines, left unexplained until now:

where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,

A parse context is the small bundle a parse needs on top of the lexer: an Emitter — the policy that decides what becomes of each diagnostic, the fail-fast Fatal one by default — and the lookahead cache that stages the peeked tokens from this chapter’s opening. ParseContext rolls those two (plus an optional language marker) into one type, so a parser is generic over a single Ctx knob instead of separate emitter, cache, and language parameters.

Read the two clauses in that light. The first says only “Ctx is some parse context.” The second pins the one thing the body actually depends on — that the context’s emitter produces your error type, so each ? yields a CalcError. Everything else stays open, which is exactly the point: the identical function runs under whatever emitter the caller installs.

You have been choosing a context all along without naming it. Parser::new() installs the beginner default — FatalContext, a Fatal emitter over the default cache — and apply substitutes it for Ctx. So the generic form and a copy pinned to that concrete default are one and the same parser:

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, PartialEq, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Tok {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))]
  Int(i64),
  #[token("let")] Let,
  #[token("print")] Print,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[token("+")] Plus,
  #[token("-")] Minus,
  #[token("*")] Star,
  #[token("/")] Slash,
  #[token("^")] Caret,
  #[token("=")] Assign,
  #[token(";")] Semi,
  #[token(",")] Comma,
  #[token("(")] LParen,
  #[token(")")] RParen,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokKind { Int, Let, Print, Ident, Plus, Minus, Star, Slash, Caret, Assign, Semi, Comma, LParen, RParen }
impl core::fmt::Display for TokKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Self::Int => "integer", Self::Let => "`let`", Self::Print => "`print`",
      Self::Ident => "identifier", Self::Plus => "`+`", Self::Minus => "`-`",
      Self::Star => "`*`", Self::Slash => "`/`", Self::Caret => "`^`",
      Self::Assign => "`=`", Self::Semi => "`;`", Self::Comma => "`,`",
      Self::LParen => "`(`", Self::RParen => "`)`",
    })
  }
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    match self {
      Tok::Int(n) => write!(f, "{n}"),
      other => core::fmt::Display::fmt(&other.kind(), f),
    }
  }
}
impl TokenT<'_> for Tok {
  type Kind = TokKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokKind {
    match self {
      Tok::Int(_) => TokKind::Int, Tok::Let => TokKind::Let, Tok::Print => TokKind::Print,
      Tok::Ident => TokKind::Ident, Tok::Plus => TokKind::Plus, Tok::Minus => TokKind::Minus,
      Tok::Star => TokKind::Star, Tok::Slash => TokKind::Slash, Tok::Caret => TokKind::Caret,
      Tok::Assign => TokKind::Assign, Tok::Semi => TokKind::Semi, Tok::Comma => TokKind::Comma,
      Tok::LParen => TokKind::LParen, Tok::RParen => TokKind::RParen,
    }
  }
  fn is_trivia(&self) -> bool { false }
}
type CalcLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
use tokora::error::{UnexpectedEot, token::UnexpectedTokenOf};
#[derive(Debug, Clone, PartialEq)]
enum CalcError { Lex, Unexpected, UnexpectedEnd }
impl From<LexError> for CalcError { fn from(_: LexError) -> Self { CalcError::Lex } }
impl<'inp> From<UnexpectedTokenOf<'inp, CalcLexer<'inp>>> for CalcError {
  fn from(_: UnexpectedTokenOf<'inp, CalcLexer<'inp>>) -> Self { CalcError::Unexpected }
}
impl<O, Lang: ?Sized, Set: Clone + 'static> From<UnexpectedEot<O, Lang, Set>> for CalcError {
  fn from(_: UnexpectedEot<O, Lang, Set>) -> Self { CalcError::UnexpectedEnd }
}
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for CalcError {
  fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { CalcError::UnexpectedEnd }
}
use tokora::{Emitter, FatalContext, InputRef, Parse, ParseContext, Parser};

// The generic idiom, one more time — one function, any context.
fn one_int<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<i64, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  match inp.next()? {
    Some(tok) => match tok.into_data() {
      Tok::Int(n) => Ok(n),
      _ => Err(CalcError::Unexpected),
    },
    None => Err(CalcError::UnexpectedEnd),
  }
}

// The concrete context `Parser::new()` installs: a fail-fast `Fatal` emitter over the default
// cache. Pinning it by hand type-checks against the generic `one_int` above — which is exactly
// the substitution `apply` performs when you never name a context at all.
fn one_int_default<'inp>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, FatalContext<'inp, CalcLexer<'inp>, CalcError>>,
) -> Result<i64, CalcError> {
  one_int(inp)
}

assert_eq!(Parser::new().apply(one_int).parse_str("42"), Ok(42));
assert_eq!(Parser::new().apply(one_int_default).parse_str("42"), Ok(42));

Chapter 7 is where the generic knob earns its keep: the very same parsers, run under a collecting context, gather every diagnostic instead of stopping at the first. The full catalog — every context, emitter, and error leaf — is the errors, emitters & context reference.

expect: mismatches with a name

The manual try_expect-then-Err above works, but the failure says nothing about what was expected. The expect combinator consumes one token and, on a mismatch, routes an UnexpectedTokenOf through the emitter carrying an Expected — the machine-readable “expected integer, found ;” half of a diagnostic (chapter 7 renders it). At end of input it emits UnexpectedEot instead:

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, PartialEq, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Tok {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))]
  Int(i64),
  #[token("let")] Let,
  #[token("print")] Print,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[token("+")] Plus,
  #[token("-")] Minus,
  #[token("*")] Star,
  #[token("/")] Slash,
  #[token("^")] Caret,
  #[token("=")] Assign,
  #[token(";")] Semi,
  #[token(",")] Comma,
  #[token("(")] LParen,
  #[token(")")] RParen,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokKind { Int, Let, Print, Ident, Plus, Minus, Star, Slash, Caret, Assign, Semi, Comma, LParen, RParen }
impl core::fmt::Display for TokKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Self::Int => "integer", Self::Let => "`let`", Self::Print => "`print`",
      Self::Ident => "identifier", Self::Plus => "`+`", Self::Minus => "`-`",
      Self::Star => "`*`", Self::Slash => "`/`", Self::Caret => "`^`",
      Self::Assign => "`=`", Self::Semi => "`;`", Self::Comma => "`,`",
      Self::LParen => "`(`", Self::RParen => "`)`",
    })
  }
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    match self {
      Tok::Int(n) => write!(f, "{n}"),
      other => core::fmt::Display::fmt(&other.kind(), f),
    }
  }
}
impl TokenT<'_> for Tok {
  type Kind = TokKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokKind {
    match self {
      Tok::Int(_) => TokKind::Int, Tok::Let => TokKind::Let, Tok::Print => TokKind::Print,
      Tok::Ident => TokKind::Ident, Tok::Plus => TokKind::Plus, Tok::Minus => TokKind::Minus,
      Tok::Star => TokKind::Star, Tok::Slash => TokKind::Slash, Tok::Caret => TokKind::Caret,
      Tok::Assign => TokKind::Assign, Tok::Semi => TokKind::Semi, Tok::Comma => TokKind::Comma,
      Tok::LParen => TokKind::LParen, Tok::RParen => TokKind::RParen,
    }
  }
  fn is_trivia(&self) -> bool { false }
}
type CalcLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
use tokora::{
  Emitter, InputRef, Parse, ParseContext, Parser,
  error::{UnexpectedEot, token::UnexpectedTokenOf},
};
#[derive(Debug, Clone, PartialEq)]
enum CalcError { Lex, Unexpected, UnexpectedEnd }
impl From<LexError> for CalcError { fn from(_: LexError) -> Self { CalcError::Lex } }
impl<'inp> From<UnexpectedTokenOf<'inp, CalcLexer<'inp>>> for CalcError {
  fn from(_: UnexpectedTokenOf<'inp, CalcLexer<'inp>>) -> Self { CalcError::Unexpected }
}
impl<O, Lang: ?Sized, Set: Clone + 'static> From<UnexpectedEot<O, Lang, Set>> for CalcError {
  fn from(_: UnexpectedEot<O, Lang, Set>) -> Self { CalcError::UnexpectedEnd }
}
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for CalcError {
  fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { CalcError::UnexpectedEnd }
}
use tokora::{ParseInput, parser::expect, utils::Expected};

fn parse_int<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<i64, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  // The classifier names what it wants; a mismatch becomes a structured emission.
  let tok = expect(|t: &Tok| {
    if matches!(t, Tok::Int(_)) {
      Ok(())
    } else {
      Err(Expected::one(TokKind::Int))
    }
  })
  .parse_input(inp)?;
  match tok {
    Tok::Int(n) => Ok(n),
    _ => unreachable!("the classifier admits only integers"),
  }
}

assert_eq!(Parser::new().apply(parse_int).parse_str("7"), Ok(7));
// Mismatch: `expect` emits UnexpectedToken; Fatal converts it via `From`.
assert_eq!(
  Parser::new().apply(parse_int).parse_str(";"),
  Err(CalcError::Unexpected)
);
// Premature end: UnexpectedEot instead.
assert_eq!(
  Parser::new().apply(parse_int).parse_str(""),
  Err(CalcError::UnexpectedEnd)
);

parse_let and parse_int compose by ordinary function calls — that is most of tokora’s composition story already. The next chapter adds the combinator layer for the repetitive shapes. Next: chapter 3.

3. Composition

Chapter 2 composed parsers with ordinary function calls. That scales surprisingly far, but three shapes recur in every grammar — A then B, zero or more A, A separated by commas — and the combinator layer expresses them declaratively. tokora has two combinator families:

• ParseInput — a parser that must produce a value or fail. Every fn(&mut InputRef<…>) -> Result<O, E> implements it for free.
• TryParseInput — a parser that may also decline: its ParseAttempt result is either Accept(value) or Decline, and a decline consumes no valid tokens — the input is rewound so whatever comes next can look at the same tokens. (Lexer-error tokens and already-emitted diagnostics are not rolled back; see the transactional contract.) Declining elements are what let the repetition drivers stop cleanly without arbitrary lookahead.

Sequencing

then keeps both outputs as a tuple; ignore_then and then_ignore keep one side; map transforms the output, and spanned / sliced / located attach where it came from. A delimited shape is just sequencing with the brackets ignored — open.ignore_then(body).then_ignore(close) — which is how the argument-list example below wraps its comma list in parentheses. That hand-roll is the lesson here; the combinator reference packages it ready-made as the parens shape (with braces/brackets/angles and the generic delimited).

Repetition

repeated drives a TryParseInput element until it declines, and collect accumulates the values into any Container (a Vec here; arrays and bounded containers work too). If your element is a plain ParseInput and you would rather supply the stopping decision yourself, repeated_while takes an explicit peek-window condition instead — while_head and while_kind spell the common width-1 conditions (“continue while the head satisfies this”, “…while its kind is that”) without a Peeked window or a turbofish. For the very common “repeat until a sentinel token, and leave it in place” shape there is a one-liner: list_until(until) collects into a Vec and stops before the token until accepts, so the caller’s next step still sees it.

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, PartialEq, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Tok {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))]
  Int(i64),
  #[token("let")] Let,
  #[token("print")] Print,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[token("+")] Plus,
  #[token("-")] Minus,
  #[token("*")] Star,
  #[token("/")] Slash,
  #[token("^")] Caret,
  #[token("=")] Assign,
  #[token(";")] Semi,
  #[token(",")] Comma,
  #[token("(")] LParen,
  #[token(")")] RParen,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokKind { Int, Let, Print, Ident, Plus, Minus, Star, Slash, Caret, Assign, Semi, Comma, LParen, RParen }
impl core::fmt::Display for TokKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Self::Int => "integer", Self::Let => "`let`", Self::Print => "`print`",
      Self::Ident => "identifier", Self::Plus => "`+`", Self::Minus => "`-`",
      Self::Star => "`*`", Self::Slash => "`/`", Self::Caret => "`^`",
      Self::Assign => "`=`", Self::Semi => "`;`", Self::Comma => "`,`",
      Self::LParen => "`(`", Self::RParen => "`)`",
    })
  }
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    match self {
      Tok::Int(n) => write!(f, "{n}"),
      other => core::fmt::Display::fmt(&other.kind(), f),
    }
  }
}
impl TokenT<'_> for Tok {
  type Kind = TokKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokKind {
    match self {
      Tok::Int(_) => TokKind::Int, Tok::Let => TokKind::Let, Tok::Print => TokKind::Print,
      Tok::Ident => TokKind::Ident, Tok::Plus => TokKind::Plus, Tok::Minus => TokKind::Minus,
      Tok::Star => TokKind::Star, Tok::Slash => TokKind::Slash, Tok::Caret => TokKind::Caret,
      Tok::Assign => TokKind::Assign, Tok::Semi => TokKind::Semi, Tok::Comma => TokKind::Comma,
      Tok::LParen => TokKind::LParen, Tok::RParen => TokKind::RParen,
    }
  }
  fn is_trivia(&self) -> bool { false }
}
type CalcLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
use tokora::error::{UnexpectedEot, syntax::FullContainer, token::UnexpectedToken};
#[derive(Debug, Clone, PartialEq)]
enum CalcError { Lex, Unexpected, UnexpectedEnd }
impl From<LexError> for CalcError { fn from(_: LexError) -> Self { CalcError::Lex } }
impl<'a, T, K: Clone, S, Lang: ?Sized> From<UnexpectedToken<'a, T, K, S, Lang>> for CalcError {
  fn from(_: UnexpectedToken<'a, T, K, S, Lang>) -> Self { CalcError::Unexpected }
}
impl<O, Lang: ?Sized, Set: Clone + 'static> From<UnexpectedEot<O, Lang, Set>> for CalcError {
  fn from(_: UnexpectedEot<O, Lang, Set>) -> Self { CalcError::UnexpectedEnd }
}
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for CalcError {
  fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { CalcError::UnexpectedEnd }
}
impl<S, Lang: ?Sized> From<FullContainer<S, Lang>> for CalcError {
  fn from(_: FullContainer<S, Lang>) -> Self { CalcError::Unexpected }
}
use tokora::{
  Emitter, InputRef, Parse, ParseContext, Parser, TryParseInput,
  emitter::FullContainerEmitter,
  try_parse_input::ParseAttempt,
};

/// A `let` binding as a *try*-shaped element: decline unless the next token is
/// `let`, and only then commit to the strict tail of the statement.
fn try_let<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<ParseAttempt<(&'inp str, i64)>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  // The decision point: a non-`let` token is *put back* and we decline.
  if inp.try_expect(|t| matches!(t.data(), Tok::Let))?.is_none() {
    return Ok(ParseAttempt::Decline);
  }
  // Committed from here on: failures are real errors, not declines.
  if inp.try_expect(|t| matches!(t.data(), Tok::Ident))?.is_none() {
    return Err(CalcError::Unexpected);
  }
  let name = inp.slice();
  if inp.try_expect(|t| matches!(t.data(), Tok::Assign))?.is_none() {
    return Err(CalcError::Unexpected);
  }
  let value = match inp.next()? {
    Some(tok) => match tok.into_data() {
      Tok::Int(n) => n,
      _ => return Err(CalcError::Unexpected),
    },
    None => return Err(CalcError::UnexpectedEnd),
  };
  if inp.try_expect(|t| matches!(t.data(), Tok::Semi))?.is_none() {
    return Err(CalcError::Unexpected);
  }
  Ok(ParseAttempt::Accept((name, value)))
}

/// Zero or more bindings: repeat the element until it declines, collect into a `Vec`.
fn parse_bindings<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Vec<(&'inp str, i64)>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter:
    Emitter<'inp, CalcLexer<'inp>, Error = CalcError> + FullContainerEmitter<'inp, CalcLexer<'inp>>,
{
  use tokora::{Accumulator, ParseInput as _};
  try_let.repeated().collect().parse_input(inp)
}

let bindings = Parser::new()
  .apply(parse_bindings)
  .parse_str("let a = 1 ; let b = 2 ; let c = 3 ;")
  .unwrap();
assert_eq!(bindings, [("a", 1), ("b", 2), ("c", 3)]);

// The element declines on the first non-`let` token, so the repetition stops
// cleanly — an empty input is zero bindings, not an error.
let none = Parser::new().apply(parse_bindings).parse_str("").unwrap();
assert!(none.is_empty());

Separation — separators are typed punctuators

Comma-separated lists could be hand-rolled with try_expect, but separator handling is where edge cases breed: leading separators, trailing separators, doubled separators, minimum and maximum element counts. separated — and its ready-made spellings like separated_by_comma — puts the policy in one place. Two small impls wire your token type to the separator vocabulary in punct:

• PunctuatorToken tells the driver which of your kinds is a comma (semicolon, parenthesis, …);
• From<Comma<(), (), ()>> for your kind type lets the zero-sized Comma punctuator name itself in diagnostics.

The Separated driver’s knobs — its element-count bounds and leading/trailing separator policies — are documented on Separated; each reports through its own emitter trait, which is why the where clause below names them. (There is also separated_while for elements that cannot decline, where you provide the lookahead condition — and separated1_by::<Sep, _>(peek), the committed-first “light” spelling of it: one-or-more elements, an optional leading separator, a trailing one refused, collected into a Vec, with the whole policy already chosen.)

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, PartialEq, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Tok {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))]
  Int(i64),
  #[token("let")] Let,
  #[token("print")] Print,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[token("+")] Plus,
  #[token("-")] Minus,
  #[token("*")] Star,
  #[token("/")] Slash,
  #[token("^")] Caret,
  #[token("=")] Assign,
  #[token(";")] Semi,
  #[token(",")] Comma,
  #[token("(")] LParen,
  #[token(")")] RParen,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokKind { Int, Let, Print, Ident, Plus, Minus, Star, Slash, Caret, Assign, Semi, Comma, LParen, RParen }
impl core::fmt::Display for TokKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Self::Int => "integer", Self::Let => "`let`", Self::Print => "`print`",
      Self::Ident => "identifier", Self::Plus => "`+`", Self::Minus => "`-`",
      Self::Star => "`*`", Self::Slash => "`/`", Self::Caret => "`^`",
      Self::Assign => "`=`", Self::Semi => "`;`", Self::Comma => "`,`",
      Self::LParen => "`(`", Self::RParen => "`)`",
    })
  }
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    match self {
      Tok::Int(n) => write!(f, "{n}"),
      other => core::fmt::Display::fmt(&other.kind(), f),
    }
  }
}
impl TokenT<'_> for Tok {
  type Kind = TokKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokKind {
    match self {
      Tok::Int(_) => TokKind::Int, Tok::Let => TokKind::Let, Tok::Print => TokKind::Print,
      Tok::Ident => TokKind::Ident, Tok::Plus => TokKind::Plus, Tok::Minus => TokKind::Minus,
      Tok::Star => TokKind::Star, Tok::Slash => TokKind::Slash, Tok::Caret => TokKind::Caret,
      Tok::Assign => TokKind::Assign, Tok::Semi => TokKind::Semi, Tok::Comma => TokKind::Comma,
      Tok::LParen => TokKind::LParen, Tok::RParen => TokKind::RParen,
    }
  }
  fn is_trivia(&self) -> bool { false }
}
type CalcLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
use tokora::error::{
  UnexpectedEot,
  syntax::{FullContainer, MissingSyntax, TooFew, TooMany},
  token::{MissingToken, SeparatedError, UnexpectedToken},
};
#[derive(Debug, Clone, PartialEq)]
enum CalcError { Lex, Unexpected, UnexpectedEnd }
impl From<LexError> for CalcError { fn from(_: LexError) -> Self { CalcError::Lex } }
impl<'a, T, K: Clone, S, Lang: ?Sized> From<UnexpectedToken<'a, T, K, S, Lang>> for CalcError {
  fn from(_: UnexpectedToken<'a, T, K, S, Lang>) -> Self { CalcError::Unexpected }
}
impl<O, Lang: ?Sized, Set: Clone + 'static> From<UnexpectedEot<O, Lang, Set>> for CalcError {
  fn from(_: UnexpectedEot<O, Lang, Set>) -> Self { CalcError::UnexpectedEnd }
}
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for CalcError {
  fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { CalcError::UnexpectedEnd }
}
impl<O, Lang: ?Sized> From<MissingSyntax<O, Lang>> for CalcError {
  fn from(_: MissingSyntax<O, Lang>) -> Self { CalcError::Unexpected }
}
impl<'a, T, K: Clone, S, Lang: ?Sized> From<SeparatedError<'a, T, K, S, Lang>> for CalcError {
  fn from(_: SeparatedError<'a, T, K, S, Lang>) -> Self { CalcError::Unexpected }
}
impl<'a, K: Clone, O, Lang: ?Sized> From<MissingToken<'a, K, O, Lang>> for CalcError {
  fn from(_: MissingToken<'a, K, O, Lang>) -> Self { CalcError::Unexpected }
}
impl<S, Lang: ?Sized> From<FullContainer<S, Lang>> for CalcError {
  fn from(_: FullContainer<S, Lang>) -> Self { CalcError::Unexpected }
}
impl<S, Lang: ?Sized> From<TooFew<S, Lang>> for CalcError {
  fn from(_: TooFew<S, Lang>) -> Self { CalcError::Unexpected }
}
impl<S, Lang: ?Sized> From<TooMany<S, Lang>> for CalcError {
  fn from(_: TooMany<S, Lang>) -> Self { CalcError::Unexpected }
}
use tokora::{
  Emitter, InputRef, Parse, ParseContext, Parser, TryParseInput,
  try_parse_input::ParseAttempt,
};
use tokora::{
  Accumulator, ParseInput,
  emitter::{
    FullContainerEmitter, SeparatedEmitter, UnexpectedLeadingSeparatorEmitter,
    UnexpectedTrailingSeparatorEmitter,
  },
  parser::expect,
  punct::Comma,
  token::PunctuatorToken,
  utils::Expected,
};

// Wire `Tok` into the punctuator vocabulary: name which kind is the comma.
impl PunctuatorToken<'_> for Tok {
  fn comma() -> Option<TokKind> {
    Some(TokKind::Comma)
  }
}
// And let the zero-sized `Comma` punctuator name itself as a kind.
impl From<Comma<(), (), ()>> for TokKind {
  fn from(_: Comma<(), (), ()>) -> Self {
    TokKind::Comma
  }
}

/// A *try*-shaped integer element for the separated driver.
fn try_int<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<ParseAttempt<i64>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  Ok(match inp.try_expect(|t| matches!(t.data(), Tok::Int(_)))? {
    Some(tok) => match tok.into_data() {
      Tok::Int(n) => ParseAttempt::Accept(n),
      _ => unreachable!("the predicate admits only integers"),
    },
    None => ParseAttempt::Decline,
  })
}

/// `( int , int , … )` — a delimited, comma-separated list: sequencing for the
/// parentheses, `separated_by_comma` for the elements.
fn parse_args<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Vec<i64>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>
    + SeparatedEmitter<'inp, CalcLexer<'inp>>
    + FullContainerEmitter<'inp, CalcLexer<'inp>>
    + UnexpectedLeadingSeparatorEmitter<'inp, CalcLexer<'inp>>
    + UnexpectedTrailingSeparatorEmitter<'inp, CalcLexer<'inp>>,
{
  expect(|t: &Tok| {
    if matches!(t, Tok::LParen) {
      Ok(())
    } else {
      Err(Expected::one(TokKind::LParen))
    }
  })
  .ignore_then(try_int.separated_by_comma().collect())
  .then_ignore(expect(|t: &Tok| {
    if matches!(t, Tok::RParen) {
      Ok(())
    } else {
      Err(Expected::one(TokKind::RParen))
    }
  }))
  .parse_input(inp)
}

let args: Vec<i64> = Parser::new()
  .apply(parse_args)
  .parse_str("( 1 , 2 , 3 )")
  .unwrap();
assert_eq!(args, [1, 2, 3]);

// Zero elements: the element declines at `)`, the list is empty, the closer matches.
let empty: Vec<i64> = Parser::new().apply(parse_args).parse_str("( )").unwrap();
assert!(empty.is_empty());

// A doubled separator is a structured failure, not a mis-parse.
let doubled = Parser::new().apply(parse_args).parse_str("( 1 , , 2 )");
assert!(doubled.is_err());

Calc now has its print 1 , 2 ; argument shape and statement lists. What it does not have yet is a way to choose which statement parser to run based on the next token — that is dispatch. Next: chapter 4.

4. Deterministic choice

A Calc statement starts with let, print, or an integer. Choosing between alternatives is where many combinator libraries reach for speculative “try each in order” choice; tokora deliberately does not. Its choice shapes are deterministic: look at the next token’s Kind once, decide, and run exactly one branch. No branch is ever half-run and unwound, so a dispatch failure is committed — the error cannot be lost to backtracking, and its expected set is exact.

Three surfaces, one decision rule:

• peek_then_choice — you write the decision handler yourself over a peek window (any fan-in, your own failure diagnostic);
• dispatch_on_kind — the decision is a static table: table[i] is the viable first-token kind for branch i. On a miss, the emitted UnexpectedToken carries the whole table as an expected one of … set (Expected::OneOf); at end of input it is UnexpectedEnd instead. Use peek_then_choice when several kinds route to one branch; the table form is one kind per branch;
• select! — the table is written beside the patterns, one arm per kind, and the classified head moves into its arm by value. Its runtime is dispatch_take, and the declining twin try_select! / try_dispatch_take is the same shape that returns Decline on a head outside the table instead of committing.

Peeked versus fused

DispatchOnKind is the peek shape: the decision token is peeked (staged in the token cache, including a lexer-state clone), the winning branch — any ParseInput, with the token still on the input — consumes it back out. FusedDispatchOnKind, built by fused_dispatch_on_kind, is the lex-once twin: the dispatcher consumes the head token as part of classifying it and hands it to the winning arm (an FnMut(head, inp) — the ParseTokenChoice surface), skipping the cache round trip entirely. Failures are observationally identical; only the hit path differs. When each wins: hot sum-type loops (a statement loop, a JSON value loop) prefer the fused shape — the saved stage/unstage matters most when the lexer state is expensive to clone — while branches that are self-contained ParseInput parsers, reused elsewhere or wanting the head token left on the input, keep the peek shape. And per the dense-discriminant note, keep your kind enum’s discriminants dense (0, 1, 2, …) so kind matches beside the table compile to jump tables.

The third shape, select!, is the fused one with the table moved next to the patterns. Each arm is kind => (span, pattern) => value: the kinds are the table, so it is written once instead of twice and cannot drift out of step with the arms; the head is classified once against it, committed, and handed to the arm moved, so an arm binds the payload (Tok::Int(n)) rather than re-matching a token it was already routed by. There is no hand-written unreachable!() — an arm whose pattern is narrower than its kind hands the token back and the runtime builds the same whole-table UnexpectedToken a miss would get. One constraint the diagnostic does not name: the kind expressions must be const-promotable (a unit-variant path or a const), because the expansion hands a &'static [Kind] to dispatch_take; anything else fails at the invocation with E0716. All three appear below, and the loop at the end asserts they agree.

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, PartialEq, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Tok {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))]
  Int(i64),
  #[token("let")] Let,
  #[token("print")] Print,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[token("+")] Plus,
  #[token("-")] Minus,
  #[token("*")] Star,
  #[token("/")] Slash,
  #[token("^")] Caret,
  #[token("=")] Assign,
  #[token(";")] Semi,
  #[token(",")] Comma,
  #[token("(")] LParen,
  #[token(")")] RParen,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokKind { Int, Let, Print, Ident, Plus, Minus, Star, Slash, Caret, Assign, Semi, Comma, LParen, RParen }
impl core::fmt::Display for TokKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Self::Int => "integer", Self::Let => "`let`", Self::Print => "`print`",
      Self::Ident => "identifier", Self::Plus => "`+`", Self::Minus => "`-`",
      Self::Star => "`*`", Self::Slash => "`/`", Self::Caret => "`^`",
      Self::Assign => "`=`", Self::Semi => "`;`", Self::Comma => "`,`",
      Self::LParen => "`(`", Self::RParen => "`)`",
    })
  }
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    match self {
      Tok::Int(n) => write!(f, "{n}"),
      other => core::fmt::Display::fmt(&other.kind(), f),
    }
  }
}
impl TokenT<'_> for Tok {
  type Kind = TokKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokKind {
    match self {
      Tok::Int(_) => TokKind::Int, Tok::Let => TokKind::Let, Tok::Print => TokKind::Print,
      Tok::Ident => TokKind::Ident, Tok::Plus => TokKind::Plus, Tok::Minus => TokKind::Minus,
      Tok::Star => TokKind::Star, Tok::Slash => TokKind::Slash, Tok::Caret => TokKind::Caret,
      Tok::Assign => TokKind::Assign, Tok::Semi => TokKind::Semi, Tok::Comma => TokKind::Comma,
      Tok::LParen => TokKind::LParen, Tok::RParen => TokKind::RParen,
    }
  }
  fn is_trivia(&self) -> bool { false }
}
type CalcLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
use tokora::error::{
  UnexpectedEnd,
  syntax::{FullContainer, MissingSyntax, TooFew},
  token::{MissingToken, SeparatedError, UnexpectedToken},
};
#[derive(Debug, Clone, PartialEq)]
enum CalcError { Lex, Unexpected, UnexpectedEnd }
impl From<LexError> for CalcError { fn from(_: LexError) -> Self { CalcError::Lex } }
impl<'a, T, K: Clone, S, Lang: ?Sized> From<UnexpectedToken<'a, T, K, S, Lang>> for CalcError {
  fn from(_: UnexpectedToken<'a, T, K, S, Lang>) -> Self { CalcError::Unexpected }
}
impl<H, O, Lang: ?Sized, Set: Clone + 'static> From<UnexpectedEnd<H, O, Lang, Set>> for CalcError {
  fn from(_: UnexpectedEnd<H, O, Lang, Set>) -> Self { CalcError::UnexpectedEnd }
}
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for CalcError {
  fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { CalcError::UnexpectedEnd }
}
impl<'a, T, K: Clone, S, Lang: ?Sized> From<SeparatedError<'a, T, K, S, Lang>> for CalcError {
  fn from(_: SeparatedError<'a, T, K, S, Lang>) -> Self { CalcError::Unexpected }
}
impl<'a, K: Clone, O, Lang: ?Sized> From<MissingToken<'a, K, O, Lang>> for CalcError {
  fn from(_: MissingToken<'a, K, O, Lang>) -> Self { CalcError::Unexpected }
}
impl<O, Lang: ?Sized> From<MissingSyntax<O, Lang>> for CalcError {
  fn from(_: MissingSyntax<O, Lang>) -> Self { CalcError::Unexpected }
}
impl<S, Lang: ?Sized> From<FullContainer<S, Lang>> for CalcError {
  fn from(_: FullContainer<S, Lang>) -> Self { CalcError::Unexpected }
}
impl<S, Lang: ?Sized> From<TooFew<S, Lang>> for CalcError {
  fn from(_: TooFew<S, Lang>) -> Self { CalcError::Unexpected }
}
use tokora::{
  ComposableParseContext, Emitter, InputRef, Parse, ParseChoice, ParseContext, ParseInput,
  ParseTokenChoice, Parser, SimpleSpan, span::Spanned,
};

/// Calc's statement AST (expressions stay integers until chapter 5).
#[derive(Debug, Clone, PartialEq)]
enum Stmt<'a> {
  Let(&'a str, i64),
  Print(Vec<i64>),
  Bare(i64),
}

fn expect_int<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<i64, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  match inp.next()? {
    Some(tok) => match tok.into_data() {
      Tok::Int(n) => Ok(n),
      _ => Err(CalcError::Unexpected),
    },
    None => Err(CalcError::UnexpectedEnd),
  }
}
fn expect_tok<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
  want: fn(&Tok) -> bool,
) -> Result<(), CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  if inp.try_expect(|t| want(t.data()))?.is_none() {
    return Err(CalcError::Unexpected);
  }
  Ok(())
}
// (Hidden here: `expect_int` and `expect_tok`, small helpers in chapter 2's style.)

// ── The three branch parsers, peek-shaped: the head token is still on the input. ──

fn stmt_let<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Stmt<'inp>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  expect_tok(inp, |t| matches!(t, Tok::Let))?;
  expect_tok(inp, |t| matches!(t, Tok::Ident))?;
  let name = inp.slice();
  expect_tok(inp, |t| matches!(t, Tok::Assign))?;
  let value = expect_int(inp)?;
  expect_tok(inp, |t| matches!(t, Tok::Semi))?;
  Ok(Stmt::Let(name, value))
}

fn stmt_print<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Stmt<'inp>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  expect_tok(inp, |t| matches!(t, Tok::Print))?;
  let mut args = vec![expect_int(inp)?];
  while inp.try_expect(|t| matches!(t.data(), Tok::Comma))?.is_some() {
    args.push(expect_int(inp)?);
  }
  expect_tok(inp, |t| matches!(t, Tok::Semi))?;
  Ok(Stmt::Print(args))
}

fn stmt_bare<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Stmt<'inp>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  let value = expect_int(inp)?;
  expect_tok(inp, |t| matches!(t, Tok::Semi))?;
  Ok(Stmt::Bare(value))
}

/// The peek-shaped dispatcher: `table[i]` names branch `i`'s first token.
fn parse_stmt<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Stmt<'inp>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  static TABLE: [TokKind; 3] = [TokKind::Let, TokKind::Print, TokKind::Int];
  (stmt_let, stmt_print, stmt_bare)
    .dispatch_on_kind(&TABLE)
    .parse_input(inp)
}

let stmt = Parser::new().apply(parse_stmt).parse_str("print 1 , 2 ;");
assert_eq!(stmt, Ok(Stmt::Print(vec![1, 2])));

// A committed dispatch failure: `;` is in no table slot, so the error carries
// the whole table as its expected set — `let`, `print`, or an integer.
assert_eq!(
  Parser::new().apply(parse_stmt).parse_str("; 1"),
  Err(CalcError::Unexpected)
);

// ── The fused twin: arms receive the already-lexed head token. ──

fn let_arm<'inp, Ctx>(
  _head: Spanned<Tok, SimpleSpan>, // the `let` keyword, already consumed
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Stmt<'inp>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  expect_tok(inp, |t| matches!(t, Tok::Ident))?;
  let name = inp.slice();
  expect_tok(inp, |t| matches!(t, Tok::Assign))?;
  let value = expect_int(inp)?;
  expect_tok(inp, |t| matches!(t, Tok::Semi))?;
  Ok(Stmt::Let(name, value))
}

fn print_arm<'inp, Ctx>(
  _head: Spanned<Tok, SimpleSpan>, // the `print` keyword
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Stmt<'inp>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  let mut args = vec![expect_int(inp)?];
  while inp.try_expect(|t| matches!(t.data(), Tok::Comma))?.is_some() {
    args.push(expect_int(inp)?);
  }
  expect_tok(inp, |t| matches!(t, Tok::Semi))?;
  Ok(Stmt::Print(args))
}

fn bare_arm<'inp, Ctx>(
  head: Spanned<Tok, SimpleSpan>, // the integer itself — no re-consume
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Stmt<'inp>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  let value = match head.into_data() {
    Tok::Int(n) => n,
    _ => unreachable!("the table routes only integers here"),
  };
  expect_tok(inp, |t| matches!(t, Tok::Semi))?;
  Ok(Stmt::Bare(value))
}

fn parse_stmt_fused<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Stmt<'inp>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  static TABLE: [TokKind; 3] = [TokKind::Let, TokKind::Print, TokKind::Int];
  (let_arm, print_arm, bare_arm)
    .fused_dispatch_on_kind(&TABLE)
    .parse_input(inp)
}

// ── Match-first: the table lives beside the patterns. ──

/// What the head decided, with the span the arm was handed. `Tok::Int(n)` binds `n` **by
/// value** — the arm receives the moved payload, which an arm that only borrowed the head
/// could not do.
enum Head {
  Let(SimpleSpan),
  Print(SimpleSpan),
  Int(SimpleSpan, i64),
}

fn parse_stmt_select<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Stmt<'inp>, CalcError>
where
  Ctx: ComposableParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  // No `static TABLE` beside this one: the three kinds in the first column *are* the
  // table, so it cannot fall out of step with the arms.
  let head = tokora::select!(inp, {
    TokKind::Let => (span, Tok::Let) => Head::Let(span),
    TokKind::Print => (span, Tok::Print) => Head::Print(span),
    TokKind::Int => (span, Tok::Int(n)) => Head::Int(span, n),
  })?;
  // The input is borrowed for the classification, so the branch that keeps parsing runs
  // after it — and reuses the fused arms unchanged.
  match head {
    Head::Let(span) => let_arm(Spanned::new(span, Tok::Let), inp),
    Head::Print(span) => print_arm(Spanned::new(span, Tok::Print), inp),
    Head::Int(span, n) => bare_arm(Spanned::new(span, Tok::Int(n)), inp),
  }
}

// All three shapes agree — on hits and on misses.
for src in ["let x = 7 ;", "print 1 , 2 ;", "42 ;", "; nope"] {
  let peeked = Parser::new().apply(parse_stmt).parse_str(src);
  let fused = Parser::new().apply(parse_stmt_fused).parse_str(src);
  let selected = Parser::new().apply(parse_stmt_select).parse_str(src);
  assert_eq!(peeked, fused, "shapes diverged on {src:?}");
  assert_eq!(peeked, selected, "shapes diverged on {src:?}");
}

Calc still evaluates nothing but bare integers. Chapter 5 replaces them with real expressions. Next: chapter 5.

5. Expressions: Pratt parsing

Calc’s statements are done; its expressions are still bare integers. Expression grammars are the one place where plain recursive descent gets ugly: the textbook shape is one function per precedence level (expr → term → factor → atom), so every new operator level costs another function and another layer of calls, and right-associativity and prefix operators need hand-written special cases at each rung.

Pratt parsing (precedence climbing) replaces the ladder of functions with a single loop plus a table: each operator carries a binding power, and the loop keeps consuming operators while their power clears the current floor. One loop, any number of levels, and a new operator is a new table row rather than new code.

tokora has two Pratt surfaces:

• token-level — InputRef::pratt, used in this chapter. The token type itself classifies each token via PrattToken, and the folds map tokens to tokens. This is the shape to reach for when the expression’s value is itself expressible as a token — a calculator that folds 1 + 2 into Int(3).
• AST-level — the pratt combinator. You supply LHS/RHS sub-parsers and folds over your own node type, so the result is a tree. That is what a full Calc — the one whose expressions include variables, and which therefore cannot fold to a number during the parse — would use.

Both run the same engine; only the currency of the folds differs.

The power ladder

Two of those rows carry the chapter’s whole design.

Associativity is how strict the floor is, not a special case. After a left-associative operator the engine recurses with a floor that admits only powers strictly greater than the operator’s own, so an equal-power operator to the right does not clear the inner floor and folds into the outer call instead — 10 / 2 / 5 groups as (10 / 2) / 5. A right-associative operator recurses with a floor that admits its own power too, so the equal-power operator does clear it and is consumed by the inner call: 2 ^ 3 ^ 2 groups as 2 ^ (3 ^ 2) = 512. You write PrattInfix::Left or PrattInfix::Right; picking the strictness is the engine’s job. Note what it never does: step to a neighbouring level. A PrattPower is only ever compared, so the rule holds at the ends of the ladder, where power ± 1 would have run out of room and silently swapped the two behaviours.

Grouping is an operator pair below the floor. ( is a prefix operator at power -1 and ) is a postfix operator at the same power. A top-level parse starts at the default floor (0, for an integer power), so a stray ) there is below the floor: the loop leaves it on the input for the surrounding grammar. But the recursive call inside a ( prefix runs with a floor of -1, and there ) clears the floor and is consumed — closing exactly its own group. No bracket-matching code and no depth counter: the precedence rule already says it.

Binding powers are plain integers

Power defaults to i64, and tokora implements PrattPower for every standard integer type. Write 1, 2, -1 and move on. A newtype is still welcome when you want named levels and a type-checked ladder — the trait is public, and it asks for nothing beyond Default + Clone + Ord — but nothing forces one on you.

The folds must be named functions

The fold parameters are bound by for<'lt> FnMut(…, &'lt mut Emitter) — a higher-ranked bound. A closure is monomorphic in its argument lifetimes and does not satisfy it; what you get is a mismatched-types error mentioning a for<'lt> signature, which is baffling if you do not know what you are looking at. Function items are generic over their lifetime parameters and satisfy the bound for free. So: write the folds as fns. Their shapes — mind the argument order, the operator comes last for infix and postfix but first for prefix:

fn fold_prefix (operator, operand,                  EmitterView) -> Result<Spanned<Tok, Span>, Error>
fn fold_infix  (left,     right,    infix_operator, EmitterView) -> Result<Spanned<Tok, Span>, Error>
fn fold_postfix(operand,  operator,                 EmitterView) -> Result<Spanned<Tok, Span>, Error>

Calc’s expression engine

use tokora::{Token as TokenT, logos::{self, Logos}};
use tokora::EmitterView;
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, PartialEq, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Tok {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))]
  Int(i64),
  #[token("let")] Let,
  #[token("print")] Print,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[token("+")] Plus,
  #[token("-")] Minus,
  #[token("*")] Star,
  #[token("/")] Slash,
  #[token("^")] Caret,
  #[token("=")] Assign,
  #[token(";")] Semi,
  #[token(",")] Comma,
  #[token("(")] LParen,
  #[token(")")] RParen,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokKind { Int, Let, Print, Ident, Plus, Minus, Star, Slash, Caret, Assign, Semi, Comma, LParen, RParen }
impl core::fmt::Display for TokKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Self::Int => "integer", Self::Let => "`let`", Self::Print => "`print`",
      Self::Ident => "identifier", Self::Plus => "`+`", Self::Minus => "`-`",
      Self::Star => "`*`", Self::Slash => "`/`", Self::Caret => "`^`",
      Self::Assign => "`=`", Self::Semi => "`;`", Self::Comma => "`,`",
      Self::LParen => "`(`", Self::RParen => "`)`",
    })
  }
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    match self {
      Tok::Int(n) => write!(f, "{n}"),
      other => core::fmt::Display::fmt(&other.kind(), f),
    }
  }
}
impl TokenT<'_> for Tok {
  type Kind = TokKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokKind {
    match self {
      Tok::Int(_) => TokKind::Int, Tok::Let => TokKind::Let, Tok::Print => TokKind::Print,
      Tok::Ident => TokKind::Ident, Tok::Plus => TokKind::Plus, Tok::Minus => TokKind::Minus,
      Tok::Star => TokKind::Star, Tok::Slash => TokKind::Slash, Tok::Caret => TokKind::Caret,
      Tok::Assign => TokKind::Assign, Tok::Semi => TokKind::Semi, Tok::Comma => TokKind::Comma,
      Tok::LParen => TokKind::LParen, Tok::RParen => TokKind::RParen,
    }
  }
  fn is_trivia(&self) -> bool { false }
}
type CalcLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
use tokora::error::{UnexpectedEnd, token::UnexpectedToken};
#[derive(Debug, Clone, PartialEq)]
enum CalcError { Lex, Unexpected, UnexpectedEnd }
impl From<LexError> for CalcError { fn from(_: LexError) -> Self { CalcError::Lex } }
impl<'a, T, K: Clone, S, Lang: ?Sized> From<UnexpectedToken<'a, T, K, S, Lang>> for CalcError {
  fn from(_: UnexpectedToken<'a, T, K, S, Lang>) -> Self { CalcError::Unexpected }
}
impl<H, O, Lang: ?Sized, Set: Clone + 'static> From<UnexpectedEnd<H, O, Lang, Set>> for CalcError {
  fn from(_: UnexpectedEnd<H, O, Lang, Set>) -> Self { CalcError::UnexpectedEnd }
}
impl<O, Lang: ?Sized> From<tokora::error::RecursionLimitReached<O, Lang>> for CalcError {
  fn from(_: tokora::error::RecursionLimitReached<O, Lang>) -> Self { CalcError::UnexpectedEnd }
}
impl<O, Lang: ?Sized> From<tokora::error::NonAssociativeChain<O, Lang>> for CalcError {
  fn from(_: tokora::error::NonAssociativeChain<O, Lang>) -> Self { CalcError::UnexpectedEnd }
}
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for CalcError {
  fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { CalcError::UnexpectedEnd }
}
use tokora::{
  Emitter, InputRef, Parse, ParseContext, Parser, SimpleSpan,
  emitter::PrattEmitter,
  parser::{PrattInfix, PrattLHS, PrattRHS, Precedenced},
  span::Spanned,
  token::PrattToken,
};

// ── The ladder. Plain `i64`s — no newtype, because `PrattPower` is implemented for the
//    integers. The default floor is `i64::default()` = 0, and PREC_PAREN sits *below* it:
//    that is what makes `)` invisible at the top level and consumable inside a group.

const PREC_PAREN: i64 = -1; // ( )
const PREC_SUM: i64 = 1; //    + -
const PREC_PROD: i64 = 2; //   * /
const PREC_NEG: i64 = 3; //    unary -
const PREC_EXP: i64 = 4; //    ^

// ── The table, written as an impl on the token: each token says what it is at each
//    position. `None` means "not part of an expression here", so the token is left on the
//    input — which is exactly how the engine knows to stop at `;` or `,`.

impl PrattToken<'_, i64> for Tok {
  fn try_pratt_lhs(&self) -> Option<PrattLHS<(), (), i64>> {
    Some(match self {
      Tok::Int(_) => PrattLHS::Operand(()),
      Tok::Minus => PrattLHS::Prefix(Precedenced::new((), PREC_NEG)),
      Tok::LParen => PrattLHS::Prefix(Precedenced::new((), PREC_PAREN)),
      _ => return None,
    })
  }

  fn try_pratt_rhs(&self) -> Option<PrattRHS<(), (), (), (), i64>> {
    Some(match self {
      Tok::Plus => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(()), PREC_SUM)),
      Tok::Minus => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(()), PREC_SUM)),
      Tok::Star => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(()), PREC_PROD)),
      Tok::Slash => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(()), PREC_PROD)),
      // The one right-associative row in the table.
      Tok::Caret => PrattRHS::Infix(Precedenced::new(PrattInfix::Right(()), PREC_EXP)),
      Tok::RParen => PrattRHS::Postfix(Precedenced::new((), PREC_PAREN)),
      _ => return None,
    })
  }
}

// ── The folds. Named `fn`s, not closures. The token-level API's currency is
//    `Spanned<Tok, Span>`, so a computed value goes back in as a `Tok::Int`.

fn fold_prefix<'inp, E>(
  op: Spanned<Tok, SimpleSpan>,
  operand: Spanned<Tok, SimpleSpan>,
  _: EmitterView<'_, 'inp, CalcLexer<'inp>, E>,
) -> Result<Spanned<Tok, SimpleSpan>, CalcError> {
  let (span, op) = op.into_components();
  match op {
    Tok::Minus => Ok(Spanned::new(span, Tok::Int(-int(operand)?))),
    // Grouping: the `(` prefix's "operand" is the whole parenthesised expression, already
    // folded by the inner call (which also ate the `)`). Pass it through untouched.
    Tok::LParen => Ok(operand),
    _ => unreachable!("the LHS table admits only `-` and `(` as prefixes"),
  }
}

fn fold_infix<'inp, E>(
  left: Spanned<Tok, SimpleSpan>,
  right: Spanned<Tok, SimpleSpan>,
  infix: Spanned<PrattInfix<Tok, Tok, Tok>, SimpleSpan>,
  _: EmitterView<'_, 'inp, CalcLexer<'inp>, E>,
) -> Result<Spanned<Tok, SimpleSpan>, CalcError> {
  let span = left.span();
  let (l, r) = (int(left)?, int(right)?);
  // The associativity has already done its job in the engine; the fold just wants the token.
  let (PrattInfix::Left(op) | PrattInfix::Right(op) | PrattInfix::Neither(op)) =
    infix.into_data();
  let value = match op {
    Tok::Plus => l + r,
    Tok::Minus => l - r,
    Tok::Star => l * r,
    // The folds are fallible on purpose. A grown-up Calc would add a `DivByZero` variant
    // rather than reuse `Unexpected`, but the shape is the same: an `Err` out of a fold
    // aborts the expression.
    Tok::Slash => l.checked_div(r).ok_or(CalcError::Unexpected)?,
    Tok::Caret => u32::try_from(r)
      .ok()
      .and_then(|e| l.checked_pow(e))
      .ok_or(CalcError::Unexpected)?,
    _ => unreachable!("the RHS table admits only the five arithmetic infixes"),
  };
  Ok(Spanned::new(span, Tok::Int(value)))
}

fn fold_postfix<'inp, E>(
  operand: Spanned<Tok, SimpleSpan>,
  _close: Spanned<Tok, SimpleSpan>,
  _: EmitterView<'_, 'inp, CalcLexer<'inp>, E>,
) -> Result<Spanned<Tok, SimpleSpan>, CalcError> {
  Ok(operand) // `)` closed its group; the value flows on.
}

/// Unwrap a folded operand back to its integer.
fn int(tok: Spanned<Tok, SimpleSpan>) -> Result<i64, CalcError> {
  match tok.into_data() {
    Tok::Int(n) => Ok(n),
    _ => Err(CalcError::Unexpected),
  }
}

// ── The entry point: one call, the whole expression grammar.

fn calc_expr<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<i64, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter:
    Emitter<'inp, CalcLexer<'inp>, Error = CalcError> + PrattEmitter<'inp, CalcLexer<'inp>>,
{
  // `Expr` = i64 (what an expression *means*); `Power` = i64 (how tightly things bind).
  let folded = inp.pratt::<_, _, _, i64, i64>(
    fold_prefix::<Ctx::Emitter>,
    fold_infix::<Ctx::Emitter>,
    fold_postfix::<Ctx::Emitter>,
  )?;
  // `Ok(None)` means the cursor was not looking at an expression at all.
  match folded {
    Some(tok) => int(tok),
    None => Err(CalcError::UnexpectedEnd),
  }
}

let eval = |src| Parser::new().apply(calc_expr).parse_str(src);

assert_eq!(eval("1 + 2 * 3"), Ok(7)); //     `*` outranks `+`       → 1 + (2 * 3)
assert_eq!(eval("(1 + 2) * 3"), Ok(9)); //   grouping overrides     → (1 + 2) * 3
assert_eq!(eval("10 / 2 / 5"), Ok(1)); //    `/` is left-assoc      → (10 / 2) / 5
assert_eq!(eval("2 ^ 3 ^ 2"), Ok(512)); //   `^` is RIGHT-assoc     → 2 ^ (3 ^ 2)
assert_eq!(eval("-(1 + 2)"), Ok(-3)); //     prefix over a group
assert_eq!(eval("-2 ^ 2"), Ok(-4)); //       `^` outranks unary `-` → -(2 ^ 2)

// Nothing here is expression-shaped: the engine consumes nothing and says so.
assert_eq!(eval(";"), Err(CalcError::UnexpectedEnd));

// ── And it slots straight into the statement grammar. ──
fn expect_tok<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
  want: fn(&Tok) -> bool,
) -> Result<(), CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  if inp.try_expect(|t| want(t.data()))?.is_none() {
    return Err(CalcError::Unexpected);
  }
  Ok(())
}
// (Hidden here: `expect_tok`, chapter 2's one-token helper.)

fn parse_let<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<(&'inp str, i64), CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter:
    Emitter<'inp, CalcLexer<'inp>, Error = CalcError> + PrattEmitter<'inp, CalcLexer<'inp>>,
{
  expect_tok(inp, |t| matches!(t, Tok::Let))?;
  expect_tok(inp, |t| matches!(t, Tok::Ident))?;
  let name = inp.slice();
  expect_tok(inp, |t| matches!(t, Tok::Assign))?;
  // The engine stops at `;` by itself: `Semi` has no RHS table entry, so `try_pratt_rhs`
  // returns `None` and the operator loop ends with the token still on the input.
  let value = calc_expr(inp)?;
  expect_tok(inp, |t| matches!(t, Tok::Semi))?;
  Ok((name, value))
}

let binding = Parser::new()
  .apply(parse_let)
  .parse_str("let x = -2 + 3 * (4 + 1) ;")
  .unwrap();
assert_eq!(binding, ("x", 13));

Recovery: an error node is a legal operand

Everything above stops at the first bad token. An editor cannot: it needs a tree for 1 + — a line the user is halfway through typing — not an error message. The posture that produces one is rust-analyzer’s, and the typed driver already supports it with no signature change at all.

The rule that makes it work is a rule about ParsePrattLHS: only Prefix is held to “consume what you report”. A prefix report makes the driver re-enter the expression at the same position, so a zero-width one would descend forever, and the driver refuses it. An Operand report causes no recursion and no fold, so a zero-width one costs nothing — and that is the licence an error node needs. Report an operand you did not consume, and the driver folds it like any other.

So the LHS channel, when no operand is there, reports the problem and hands back a hole:

// The recovery arm of the LHS channel. `other` is the head token kind, or `None` at end of
// input. Compiled and pinned in full — see the links below.
other => {
  inp.emit_error(Spanned::new(here(inp), Diag::ExpectedExpression))?;
  let mark = inp.cst_mark();
  if !in_recovery_set(other) {
    // Nothing can consume this token, so leaving it would hand the next cycle the same
    // input. Swallow exactly one — r-a's `err_and_bump`.
    inp.next()?;
  }
  inp.cst_start_at(mark, ERROR_EXPR);
  inp.cst_finish(ERROR_EXPR);
  Ok(PrattLHS::Operand(Expr::Error))
}

Two branches, and the recovery set chooses between them. A token some enclosing construct can still use — ), an infix operator, end of input — is left in place, so the error node is zero-width and the enclosing frame is handed the token it was waiting for. Anything else is swallowed into a one-token error node, which is what guarantees the parse moves.

Deciding that by peeking is safe because of the RHS channel’s own contract: End restores whatever the deciding read consumed, so a token this expression declines is handed back untouched.

The same licence covers a case that has nothing to do with a missing operand. A token your lexer accepts can still be one your AST cannot hold — a digit run whose value overflows the integer type is the usual one, and no lexer regex rules it out, since the regex bounds a literal’s shape and not its magnitude. That conversion is the one step in an operand parser that can fail on input nothing upstream is able to reject, and the answer is the same one: report, return an error-node Operand, and let the fold complete. Here the node is as wide as the offending text rather than zero or one token, because the text is real and the tree owes the source every byte. An operand arm that panics instead gives back a parser that fails on input a user can type, which is the property this whole posture exists to avoid.

The rest follows from the driver:

• the fold completes over the hole. fold_infix is called with Expr::Error on one side and returns a node like any other, so 1 + becomes Bin(Add, Num(1), Error) rather than an absence.
• the loop continues. After that fold the RHS loop takes another cycle, so 1 + + 2 parses to ((1 + <error>) + 2) with one diagnostic — the second + is folded, not reported. This is where r-a and rustc part company: rustc propagates a PResult upward and abandons the enclosing production instead.
• several diagnostics survive one parse. Under a recording emitter (Verbose), emit_error returns Ok and the grammar keeps going; and when a production does have to give up, the ordinary Err it returns crosses the driver on the keep-and-commit path, so every diagnostic the expression had already made is still there for the caller.

That last point is what lets the two halves of the posture coexist. Recover in the channel where the production knows what to do; return Err where it does not — an unclosed group, say, whose extent an operand parser cannot guess — and catch it at the nearest enclosing recovery point with inplace_recover, which resumes at the offset the driver handed back rather than at the attempt origin the way recover does.

Where this posture stops

One class of error stays outside it, and a grammar that recovers should know where the line is. A grammar error becomes a hole and the parse continues. A resource error ends the parse: RecursionLimitReached is terminal, so inplace_recover re-raises it instead of spending it — deliberately, because a depth budget a recovery point could swallow would not bound anything. There is no position to resume from and no error node to synthesize, and the default budget is shallower than it sounds: 64 frames, which 64 nested parentheses reach.

Terminal is a property of the attempt, not a switch thrown on the parse. The trip is counted on a monotone cell of the input session, so an error type that discards the payload — () does — still cannot lose the stop; but every recovery point reads that cell relative to the attempt it is judging, snapshotting it beforehand and asking whether it moved. So the budget charges the failure it actually stopped, and grammar code that catches a trip itself and parses on gets ordinary recovery back for everything after it. Reading the cell absolutely instead would let one deep expression early in a file suppress every diagnostic in the rest of it. (Do not confuse this with a PartialSession’s terminal latch, which is a real latch and a different mechanism: it refuses later attempts over a growing buffer.) Once the parse is over, the absolute reading is the useful one and has its own door — Cst::resource_trips for a lossless parse.

What a recovering entry point owes there is therefore not recovery but not panicking: record the trip, hand back a hole, and let the caller tell “this is your program” from “this is how far we got”. If you build a tree alongside, a terminated parse leaves its tail uncovered, so the tree has to come out through finish_partial, which tiles that tail; finish refuses it, correctly, since for a parse that ran to the end an uncovered gap is a bug in the grammar.

The whole grammar — with a lossless tree whose holes are real, sometimes zero-width, error nodes — is examples/expr_recovery.rs, pinned input by input in tokora/tests/pratt_recovery.rs. For recovery between constructs — skipping to a sync point, counting holes — see chapter 8.

A floor of your own

pratt starts at Power::default(). pratt_with_min_precedence lets you name the floor instead — parse only what binds at least as tightly as some level and leave the rest to the caller’s loop. It is the same knob the ( prefix turns; here you turn it by hand.

Calc parses and evaluates real expressions now. Everything so far has been deterministic: one look at the next token decides everything, and no parser ever un-does work. The next chapter is about the cases where you genuinely must try something and be able to take it back. Next: chapter 6.

6. Backtracking

Every chapter so far has been deterministic: one look at the next token decided everything, and no parser ever un-did work. That is tokora’s default posture, and it is the right one — a decision that is never re-taken cannot lose a diagnostic. But some grammars genuinely need a second token before they can choose, and a few need an unbounded one. Calc is about to grow exactly such a shape.

Give Calc plain assignment (x = 1 ;) alongside expression statements (x + 1 ;). Both start with an identifier. Chapter 4’s dispatch cannot help: it decides on one kind, and here the kind is the same. The decision lives on the second token.

The tools, in the order you should reach for them

All of them are the same mechanism — save a checkpoint, maybe restore it — wearing a different shape. A rollback is total: position, span, lexer state, the token cache, the diagnostics emitted since the save, the lexer-error dedup watermark, and the poison boundary all return to what they were. Restoring is a snapshot copy, not a journal replay: the source is immutable, so there is nothing to undo.

Beneath all of them sits the raw save/restore pair. It is gated behind the unstable-raw feature and it is not the API you are meant to use: the guards exist because the raw pair has a last-in-first-out contract that a human must uphold by hand, and every guard upholds it by construction — a nested Transaction mutably borrows its parent, so deciding the parent while a child is undecided is a borrow error, not a runtime bug. Guards first. Always.

Closure-shaped speculation

attempt runs a closure and rolls back if it returns None. try_attempt is its Result sibling: roll back on Err, and hand the error to the caller. The difference matters, because a speculative parse has two ways to not work out — “this isn’t the shape I was looking for” (a decline; try something else) and “this is the shape, and it is broken” (a real error; report it). Keep them apart or you will report the wrong one.

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, PartialEq, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Tok {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))]
  Int(i64),
  #[token("let")] Let,
  #[token("print")] Print,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[token("+")] Plus,
  #[token("-")] Minus,
  #[token("*")] Star,
  #[token("/")] Slash,
  #[token("^")] Caret,
  #[token("=")] Assign,
  #[token(";")] Semi,
  #[token(",")] Comma,
  #[token("(")] LParen,
  #[token(")")] RParen,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokKind { Int, Let, Print, Ident, Plus, Minus, Star, Slash, Caret, Assign, Semi, Comma, LParen, RParen }
impl core::fmt::Display for TokKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Self::Int => "integer", Self::Let => "`let`", Self::Print => "`print`",
      Self::Ident => "identifier", Self::Plus => "`+`", Self::Minus => "`-`",
      Self::Star => "`*`", Self::Slash => "`/`", Self::Caret => "`^`",
      Self::Assign => "`=`", Self::Semi => "`;`", Self::Comma => "`,`",
      Self::LParen => "`(`", Self::RParen => "`)`",
    })
  }
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    match self {
      Tok::Int(n) => write!(f, "{n}"),
      other => core::fmt::Display::fmt(&other.kind(), f),
    }
  }
}
impl TokenT<'_> for Tok {
  type Kind = TokKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokKind {
    match self {
      Tok::Int(_) => TokKind::Int, Tok::Let => TokKind::Let, Tok::Print => TokKind::Print,
      Tok::Ident => TokKind::Ident, Tok::Plus => TokKind::Plus, Tok::Minus => TokKind::Minus,
      Tok::Star => TokKind::Star, Tok::Slash => TokKind::Slash, Tok::Caret => TokKind::Caret,
      Tok::Assign => TokKind::Assign, Tok::Semi => TokKind::Semi, Tok::Comma => TokKind::Comma,
      Tok::LParen => TokKind::LParen, Tok::RParen => TokKind::RParen,
    }
  }
  fn is_trivia(&self) -> bool { false }
}
type CalcLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
use tokora::error::{UnexpectedEot, token::UnexpectedToken};
#[derive(Debug, Clone, PartialEq)]
enum CalcError { Lex, Unexpected, UnexpectedEnd }
impl From<LexError> for CalcError { fn from(_: LexError) -> Self { CalcError::Lex } }
impl<'a, T, K: Clone, S, Lang: ?Sized> From<UnexpectedToken<'a, T, K, S, Lang>> for CalcError {
  fn from(_: UnexpectedToken<'a, T, K, S, Lang>) -> Self { CalcError::Unexpected }
}
impl<O, Lang: ?Sized, Set: Clone + 'static> From<UnexpectedEot<O, Lang, Set>> for CalcError {
  fn from(_: UnexpectedEot<O, Lang, Set>) -> Self { CalcError::UnexpectedEnd }
}
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for CalcError {
  fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { CalcError::UnexpectedEnd }
}
use tokora::{Emitter, InputRef, Parse, ParseContext, Parser};

/// Calc's two identifier-initial statements.
#[derive(Debug, Clone, PartialEq)]
enum Stmt<'a> {
  Assign(&'a str, Vec<&'a str>), // x = 1 + 2 ;
  Expr(Vec<&'a str>),            // x + 1 ;
}

fn expect_tok<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
  want: fn(&Tok) -> bool,
) -> Result<(), CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  if inp.try_expect(|t| want(t.data()))?.is_none() {
    return Err(CalcError::Unexpected);
  }
  Ok(())
}
/// This chapter's stand-in for chapter 5's Pratt engine: `atom (+ atom)*`, where an atom
/// is an integer or a variable. It yields the atoms' source text. (Hidden alongside it:
/// `expect_tok`, chapter 2's one-token helper.)
fn parse_expr<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Vec<&'inp str>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  let mut atoms = Vec::new();
  loop {
    expect_tok(inp, |t| matches!(t, Tok::Int(_) | Tok::Ident))?;
    atoms.push(inp.slice());
    if inp.try_expect(|t| matches!(t.data(), Tok::Plus))?.is_none() {
      return Ok(atoms);
    }
  }
}

// ── `attempt`: speculate, and decline unconditionally — an unbounded lookahead. ──

/// Answers a question by *parsing* it and then throwing the parse away. The closure always
/// returns `None`, so the input always rewinds: the answer travels out through a captured
/// variable, not through the return value.
///
/// For a fixed, shallow window `peek` is cheaper and does not re-lex. What `attempt` buys
/// is *unbounded* lookahead — the whole speculative parse — paid for by doing the work
/// twice.
fn looks_like_assignment<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> bool
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  let mut answer = false;
  let _: Option<()> = inp.attempt(|inp| {
    // A lexer error here folds into a "no". That is safe, not sloppy: the lookahead
    // consumes nothing, so the committed parse walks into the same bad token and emits
    // the diagnostic there — the rolled-back one is re-emitted, exactly once in total.
    answer = inp
      .try_expect(|t| matches!(t.data(), Tok::Ident))
      .ok()
      .flatten()
      .is_some()
      && inp
        .try_expect(|t| matches!(t.data(), Tok::Assign))
        .ok()
        .flatten()
        .is_some();
    None // always decline → the input rewinds whatever we found
  });
  answer
}

// ── `try_attempt`: speculate for real, and keep the distinction. ──

/// The speculation's own error channel. `try_attempt` rolls back on *any* `Err`, so the
/// two failure kinds must stay distinguishable on the far side of the rollback.
enum Speculation {
  NotAnAssignment,  // wrong shape — rewind and try the other branch
  Failed(CalcError) // right shape, broken — rewind, then report
}

fn parse_stmt<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Stmt<'inp>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  let attempted = inp.try_attempt(|inp| {
    let ident = inp
      .try_expect(|t| matches!(t.data(), Tok::Ident))
      .map_err(Speculation::Failed)?;
    if ident.is_none() {
      return Err(Speculation::NotAnAssignment);
    }
    let name = inp.slice();
    let eq = inp
      .try_expect(|t| matches!(t.data(), Tok::Assign))
      .map_err(Speculation::Failed)?;
    if eq.is_none() {
      // The decision point. The identifier we already consumed is put back too —
      // that is the whole reason this is an attempt and not a peek.
      return Err(Speculation::NotAnAssignment);
    }
    // Committed to `x = …` from here: a failure now is a real error, not a decline.
    let value = parse_expr(inp).map_err(Speculation::Failed)?;
    expect_tok(inp, |t| matches!(t, Tok::Semi)).map_err(Speculation::Failed)?;
    Ok(Stmt::Assign(name, value))
  });

  match attempted {
    Ok(stmt) => Ok(stmt),
    Err(Speculation::Failed(e)) => Err(e),
    Err(Speculation::NotAnAssignment) => {
      // Rolled back: the identifier is on the input again, so the expression parser
      // sees it as its own first atom.
      let value = parse_expr(inp)?;
      expect_tok(inp, |t| matches!(t, Tok::Semi))?;
      Ok(Stmt::Expr(value))
    }
  }
}

/// Runs the lookahead *and then* the real parse, so a passing assertion also proves the
/// lookahead left the input exactly where it found it.
fn stmt_with_lookahead<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<(bool, Stmt<'inp>), CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  let guessed = looks_like_assignment(inp);
  Ok((guessed, parse_stmt(inp)?))
}

assert_eq!(
  Parser::new().apply(stmt_with_lookahead).parse_str("x = 1 + 2 ;"),
  Ok((true, Stmt::Assign("x", vec!["1", "2"])))
);
// The `x` was consumed by the speculation and put back by the rollback, so the
// expression branch still finds it.
assert_eq!(
  Parser::new().apply(stmt_with_lookahead).parse_str("x + 1 ;"),
  Ok((false, Stmt::Expr(vec!["x", "1"])))
);
// Committed and broken: the error survives the rollback instead of becoming a decline.
assert_eq!(
  Parser::new().apply(stmt_with_lookahead).parse_str("x = ;"),
  Err(CalcError::Unexpected)
);

Guard-shaped speculation

A closure is a poor fit for control flow with several exits — a loop with two breaks, a match with an early return. begin hands you a Transaction guard instead: parse through it (it dereferences to the InputRef), then commit to keep the work or rollback to discard it. Say nothing and the drop decides — and the default is rollback, so an early return, a break, or a ? that propagates an error all rewind on the way out. You cannot forget to undo a speculative branch, because undoing it is what happens if you write no code at all.

The dual exists too. begin_with::<Commit> flips the drop policy — the guard keeps progress unless you roll it back explicitly. That is what an operator loop wants: every successful iteration keeps its tokens with no commit() call on the hot path, and only the branch that backs out of a half-consumed operator says so. The policy is a zero-sized typestate parameter: the choice is compiled in, not branched on.

One thing overrides the policy: a panic is not a decision. An undecided guard dropped while the thread is unwinding rolls back whatever policy it carries (std builds), because an unwind aborts the region rather than completing it — keeping half an iteration would leave the input in a state no normal execution can reach, which a host that catches would then see. ? is a return, not an unwind, so the keep-on-? behaviour above is untouched. Under no_std there is no panicking() to read and the divergence is documented rather than fixed.

And when you need several live fallback points at once — the longest-match shape, where you keep parsing and want to return to the best position you have seen — reach for begin_stacked. Its savepoints follow SQL semantics: rollback_to an older savepoint destroys every younger one (out-of-order revival is impossible by construction) while the target stays valid for a later rollback, and release forgets savepoints while keeping the parsed progress. A SavepointId is lifetime-branded to its transaction, so it cannot outlive it.

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, PartialEq, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Tok {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))]
  Int(i64),
  #[token("let")] Let,
  #[token("print")] Print,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[token("+")] Plus,
  #[token("-")] Minus,
  #[token("*")] Star,
  #[token("/")] Slash,
  #[token("^")] Caret,
  #[token("=")] Assign,
  #[token(";")] Semi,
  #[token(",")] Comma,
  #[token("(")] LParen,
  #[token(")")] RParen,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokKind { Int, Let, Print, Ident, Plus, Minus, Star, Slash, Caret, Assign, Semi, Comma, LParen, RParen }
impl core::fmt::Display for TokKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Self::Int => "integer", Self::Let => "`let`", Self::Print => "`print`",
      Self::Ident => "identifier", Self::Plus => "`+`", Self::Minus => "`-`",
      Self::Star => "`*`", Self::Slash => "`/`", Self::Caret => "`^`",
      Self::Assign => "`=`", Self::Semi => "`;`", Self::Comma => "`,`",
      Self::LParen => "`(`", Self::RParen => "`)`",
    })
  }
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    match self {
      Tok::Int(n) => write!(f, "{n}"),
      other => core::fmt::Display::fmt(&other.kind(), f),
    }
  }
}
impl TokenT<'_> for Tok {
  type Kind = TokKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokKind {
    match self {
      Tok::Int(_) => TokKind::Int, Tok::Let => TokKind::Let, Tok::Print => TokKind::Print,
      Tok::Ident => TokKind::Ident, Tok::Plus => TokKind::Plus, Tok::Minus => TokKind::Minus,
      Tok::Star => TokKind::Star, Tok::Slash => TokKind::Slash, Tok::Caret => TokKind::Caret,
      Tok::Assign => TokKind::Assign, Tok::Semi => TokKind::Semi, Tok::Comma => TokKind::Comma,
      Tok::LParen => TokKind::LParen, Tok::RParen => TokKind::RParen,
    }
  }
  fn is_trivia(&self) -> bool { false }
}
type CalcLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
use tokora::error::{UnexpectedEot, token::UnexpectedToken};
#[derive(Debug, Clone, PartialEq)]
enum CalcError { Lex, Unexpected, UnexpectedEnd }
impl From<LexError> for CalcError { fn from(_: LexError) -> Self { CalcError::Lex } }
impl<'a, T, K: Clone, S, Lang: ?Sized> From<UnexpectedToken<'a, T, K, S, Lang>> for CalcError {
  fn from(_: UnexpectedToken<'a, T, K, S, Lang>) -> Self { CalcError::Unexpected }
}
impl<O, Lang: ?Sized, Set: Clone + 'static> From<UnexpectedEot<O, Lang, Set>> for CalcError {
  fn from(_: UnexpectedEot<O, Lang, Set>) -> Self { CalcError::UnexpectedEnd }
}
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for CalcError {
  fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { CalcError::UnexpectedEnd }
}
use tokora::{Commit, Emitter, InputRef, Parse, ParseContext, Parser};

fn expect_tok<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
  want: fn(&Tok) -> bool,
) -> Result<(), CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  if inp.try_expect(|t| want(t.data()))?.is_none() {
    return Err(CalcError::Unexpected);
  }
  Ok(())
}
fn parse_expr<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Vec<&'inp str>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  let mut atoms = Vec::new();
  loop {
    expect_tok(inp, |t| matches!(t, Tok::Int(_) | Tok::Ident))?;
    atoms.push(inp.slice());
    if inp.try_expect(|t| matches!(t.data(), Tok::Plus))?.is_none() {
      return Ok(atoms);
    }
  }
}
#[derive(Debug, Clone, PartialEq)]
enum Stmt<'a> {
  Assign(&'a str, Vec<&'a str>),
  Expr(Vec<&'a str>),
}
// (Hidden: `expect_tok`, `parse_expr`, and `Stmt` from the previous example.)

// ── `begin`: rollback-on-drop, so every exit path rewinds unless you say otherwise. ──

fn parse_stmt<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Stmt<'inp>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  {
    let mut txn = inp.begin(); // ── speculative scope ──
    if txn.try_expect(|t| matches!(t.data(), Tok::Ident))?.is_some() {
      let name = txn.slice();
      if txn.try_expect(|t| matches!(t.data(), Tok::Assign))?.is_some() {
        // Committed shape. A `?` failure below still rewinds — the guard's drop runs on
        // the error path too — and the error itself propagates untouched.
        let value = parse_expr(&mut txn)?;
        expect_tok(&mut txn, |t| matches!(t, Tok::Semi))?;
        let stmt = Stmt::Assign(name, value);
        txn.commit(); // keep the work
        return Ok(stmt);
      }
    }
    // Falling out of the block drops an undecided guard: the input rewinds to the begin
    // point. No explicit rollback, and no exit path that can forget one.
  }
  let value = parse_expr(inp)?;
  expect_tok(inp, |t| matches!(t, Tok::Semi))?;
  Ok(Stmt::Expr(value))
}

assert_eq!(
  Parser::new().apply(parse_stmt).parse_str("x = 1 + 2 ;"),
  Ok(Stmt::Assign("x", vec!["1", "2"]))
);
assert_eq!(
  Parser::new().apply(parse_stmt).parse_str("x + 1 ;"),
  Ok(Stmt::Expr(vec!["x", "1"]))
);

// ── `begin_with::<Commit>`: keep-on-drop, for a loop whose common path is success. ──

/// `atom (+ atom)*`, where a dangling `+` is *not* an error: it is simply not part of the
/// expression, and must be handed back to whatever comes next.
fn parse_expr_greedy<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Vec<&'inp str>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  expect_tok(inp, |t| matches!(t, Tok::Int(_) | Tok::Ident))?;
  let mut atoms = vec![inp.slice()];
  loop {
    let mut txn = inp.begin_with::<Commit>();
    if txn.try_expect(|t| matches!(t.data(), Tok::Plus))?.is_none() {
      break; // no operator: nothing was consumed, so keeping "progress" is a no-op
    }
    if txn.try_expect(|t| matches!(t.data(), Tok::Int(_) | Tok::Ident))?.is_none() {
      txn.rollback(); // a dangling `+`: put it back and stop. The one explicit branch.
      break;
    }
    atoms.push(txn.slice());
    // Success. The guard drops here and *keeps* the `+ atom` — no `commit()` on the
    // hot path, which is the entire point of the `Commit` policy.
  }
  Ok(atoms)
}

/// Parses an expression and then reports the kind of the very next token — so an
/// assertion can see whether the dangling `+` really came back.
fn expr_then_peek<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<(Vec<&'inp str>, Option<TokKind>), CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  let atoms = parse_expr_greedy(inp)?;
  let next = inp.next()?.map(|t| t.data().kind());
  Ok((atoms, next))
}

assert_eq!(
  Parser::new().apply(expr_then_peek).parse_str("1 + 2 ;"),
  Ok((vec!["1", "2"], Some(TokKind::Semi)))
);
// The half-consumed operator was handed back, not swallowed.
assert_eq!(
  Parser::new().apply(expr_then_peek).parse_str("1 + 2 + ;"),
  Ok((vec!["1", "2"], Some(TokKind::Plus)))
);

// ── `begin_stacked`: several live fallback points, and return to the best one. ──

/// Calc's `print` takes coordinate *pairs*, so a trailing odd atom is not part of the
/// list. Take a savepoint after every complete pair and, at the end, roll back to the
/// last one — the classic longest-valid-prefix shape.
fn parse_pairs<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<usize, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  let mut txn = inp.begin_stacked();
  let mut best = txn.savepoint(); // the empty list is always a valid answer
  let mut seen = 0usize;
  loop {
    if txn
      .try_expect(|t| matches!(t.data(), Tok::Int(_) | Tok::Ident))?
      .is_none()
    {
      break;
    }
    seen += 1;
    if seen % 2 == 0 {
      best = txn.savepoint(); // a complete pair: a better place to fall back to
    }
  }
  txn.rollback_to(best); // discard the trailing half-pair, if any
  txn.commit(); // and keep everything up to it
  Ok(seen - seen % 2)
}

fn pairs_then_peek<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<(usize, Option<TokKind>), CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  let n = parse_pairs(inp)?;
  let next = inp.next()?.map(|t| t.data().kind());
  Ok((n, next))
}

assert_eq!(
  Parser::new().apply(pairs_then_peek).parse_str("1 2 3 4 ;"),
  Ok((4, Some(TokKind::Semi)))
);
// The odd `3` is rewound: the savepoint after the second atom wins.
assert_eq!(
  Parser::new().apply(pairs_then_peek).parse_str("1 2 3 ;"),
  Ok((2, Some(TokKind::Int)))
);

Speculation that outlives the call — session points

Every tool so far is lexical. A guard is a borrow of the input, so the speculative scope it opens can only end where that borrow does: inside one expression, one block, one call. Most of the time that is exactly what you want, and it is why the guards cannot be misused.

But it rules out one shape. A driver — a REPL, an IDE, an incremental reparser — is stepped through separate method calls: it marks a position on one call, parses on the next few, and only later decides whether to keep that work. Write that with a guard and you get a value that borrows the very input it is stored beside — self-referential, and rejected:

struct Driver<'a, 'inp, 'closure, Ctx> {
  inp: &'a mut InputRef<'inp, 'closure, CalcLexer<'inp>, Ctx>,
  txn: Transaction<'a, 'inp, 'closure, CalcLexer<'inp>, Ctx>, // ✗ borrows `inp`, beside `inp`
}

A session point is the non-lexical form. It is a value on the input, not a borrow of it: begin_point pushes a checkpoint onto the input’s own stack and hands back a plain SessionPointId — a Copy token, not a borrow, so nothing stays borrowed and the whole consume surface — next, try_expect, any parser you hand the input to — is still callable with the point open, in this call and in later ones. commit_point keeps the work; rollback_point takes it all back — cursor, lexer state, the token cache, and the diagnostics emitted since the mark. Both take the id. Points still settle newest-first, so the stack is the last-in, first-out order; what the id buys is that a settle names its own point — an id whose point is gone is refused instead of quietly settling whatever is newest, and it cannot come to mean a different point as the stack moves under it. points() is the live depth.

Here is the shape the guards cannot express: Speculator holds the input, marks in one call, parses in the next, and decides in a third.

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, PartialEq, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Tok {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))]
  Int(i64),
  #[token("let")] Let,
  #[token("print")] Print,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[token("+")] Plus,
  #[token("-")] Minus,
  #[token("*")] Star,
  #[token("/")] Slash,
  #[token("^")] Caret,
  #[token("=")] Assign,
  #[token(";")] Semi,
  #[token(",")] Comma,
  #[token("(")] LParen,
  #[token(")")] RParen,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokKind { Int, Let, Print, Ident, Plus, Minus, Star, Slash, Caret, Assign, Semi, Comma, LParen, RParen }
impl core::fmt::Display for TokKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Self::Int => "integer", Self::Let => "`let`", Self::Print => "`print`",
      Self::Ident => "identifier", Self::Plus => "`+`", Self::Minus => "`-`",
      Self::Star => "`*`", Self::Slash => "`/`", Self::Caret => "`^`",
      Self::Assign => "`=`", Self::Semi => "`;`", Self::Comma => "`,`",
      Self::LParen => "`(`", Self::RParen => "`)`",
    })
  }
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    match self {
      Tok::Int(n) => write!(f, "{n}"),
      other => core::fmt::Display::fmt(&other.kind(), f),
    }
  }
}
impl TokenT<'_> for Tok {
  type Kind = TokKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokKind {
    match self {
      Tok::Int(_) => TokKind::Int, Tok::Let => TokKind::Let, Tok::Print => TokKind::Print,
      Tok::Ident => TokKind::Ident, Tok::Plus => TokKind::Plus, Tok::Minus => TokKind::Minus,
      Tok::Star => TokKind::Star, Tok::Slash => TokKind::Slash, Tok::Caret => TokKind::Caret,
      Tok::Assign => TokKind::Assign, Tok::Semi => TokKind::Semi, Tok::Comma => TokKind::Comma,
      Tok::LParen => TokKind::LParen, Tok::RParen => TokKind::RParen,
    }
  }
  fn is_trivia(&self) -> bool { false }
}
type CalcLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
use tokora::error::{UnexpectedEot, token::UnexpectedToken};
#[derive(Debug, Clone, PartialEq)]
enum CalcError { Lex, Unexpected, UnexpectedEnd }
impl From<LexError> for CalcError { fn from(_: LexError) -> Self { CalcError::Lex } }
impl<'a, T, K: Clone, S, Lang: ?Sized> From<UnexpectedToken<'a, T, K, S, Lang>> for CalcError {
  fn from(_: UnexpectedToken<'a, T, K, S, Lang>) -> Self { CalcError::Unexpected }
}
impl<O, Lang: ?Sized, Set: Clone + 'static> From<UnexpectedEot<O, Lang, Set>> for CalcError {
  fn from(_: UnexpectedEot<O, Lang, Set>) -> Self { CalcError::UnexpectedEnd }
}
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for CalcError {
  fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { CalcError::UnexpectedEnd }
}
use tokora::{Emitter, InputRef, Parse, ParseContext, Parser, SessionPointId};

fn expect_tok<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
  want: fn(&Tok) -> bool,
) -> Result<(), CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  if inp.try_expect(|t| want(t.data()))?.is_none() {
    return Err(CalcError::Unexpected);
  }
  Ok(())
}
/// Chapter 2's `atom (+ atom)*`, hidden: it yields the atoms' source text.
fn parse_expr<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Vec<&'inp str>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  let mut atoms = Vec::new();
  loop {
    expect_tok(inp, |t| matches!(t, Tok::Int(_) | Tok::Ident))?;
    atoms.push(inp.slice());
    if inp.try_expect(|t| matches!(t.data(), Tok::Plus))?.is_none() {
      return Ok(atoms);
    }
  }
}
/// A driver that holds the input and is stepped through separate calls. Note what `mark` does
/// **not** return: no guard, only a plain id — so nothing stays borrowed, which is precisely
/// why `parse` below is callable with a mark still open.
struct Speculator<'a, 'inp, 'closure, Ctx>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
{
  inp: &'a mut InputRef<'inp, 'closure, CalcLexer<'inp>, Ctx>,
  open: Vec<SessionPointId<'closure>>,
}

impl<'inp, Ctx> Speculator<'_, 'inp, '_, Ctx>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  /// Call 1: mark where we are. The id goes in a field — no borrow is held.
  fn mark(&mut self) {
    let point = self.inp.begin_point();
    self.open.push(point);
  }

  /// Call 2: parse for real, *through* the open mark.
  fn parse(&mut self) -> Result<Vec<&'inp str>, CalcError> {
    parse_expr(self.inp)
  }

  /// Call 3: is the statement terminated? (More real parsing, still through the mark.)
  fn at_semi(&mut self) -> Result<bool, CalcError> {
    Ok(self.inp.try_expect(|t| matches!(t.data(), Tok::Semi))?.is_some())
  }

  /// Call 4: decide — long after the mark was made, by naming the mark.
  fn keep(&mut self) {
    let point = self.open.pop().expect("a mark is open");
    self.inp.commit_point(point);
  }

  fn undo(&mut self) {
    let point = self.open.pop().expect("a mark is open");
    self.inp.rollback_point(point);
  }

  fn depth(&self) -> usize { self.inp.points() }
}

/// Speculatively parse a statement. If it is not terminated, take the whole thing back —
/// a decision made three calls after the mark.
fn speculative_stmt<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<(Option<Vec<&'inp str>>, usize), CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  let mut spec = Speculator { inp, open: Vec::new() };

  spec.mark();                       // ── the point opens …
  assert_eq!(spec.depth(), 1);
  let atoms = spec.parse()?;         //    … real tokens are consumed …
  if spec.at_semi()? {
    spec.keep();                     //    … and it is decided here.
    Ok((Some(atoms), spec.depth()))
  } else {
    spec.undo();                     //    Everything since `mark` is gone.
    Ok((None, spec.depth()))
  }
}

// Terminated: the point commits and the work stands.
assert_eq!(
  Parser::new().apply(speculative_stmt).parse_str("x + 1 ;"),
  Ok((Some(vec!["x", "1"]), 0)),
);
// Unterminated: the rollback puts every token back, and the stack is empty again.
assert_eq!(
  Parser::new().apply(speculative_stmt).parse_str("x + 1"),
  Ok((None, 0)),
);

Two rules keep sessions honest. A point pins its base, exactly as a guard does: a rewind reaching below a live point would tear its foundation out, so it panics where it is requested instead of corrupting the timeline — which means you must settle a point before the scope that opened it ends. And dropping the input with live points does nothing for them: a session ends explicitly. Implicitly rolling one back on drop would paper over a driver that lost track of its own points — the deliberate opposite of a guard’s drop policy, and for the same reason: the failure you cannot see is the one that hurts.

Backtracking rewinds diagnostics too — which raises the question of what a diagnostic even is here, and how a parser reports more than one. That is the next chapter. Next: chapter 7.

7. Diagnostics

Everything Calc has done so far stops at the first thing it does not understand. That is correct for a config loader and useless for a compiler: a compiler that reports one error per run is a compiler nobody wants to use. But “report everything” is not a property of the grammar — it is a property of what you do with a diagnostic once you have one. So tokora puts that decision in one replaceable object, the emitter, and leaves the parser alone.

The parser calls emit_error and carries on with ?. What happens next is the emitter’s business:

• Fatal — emit_error returns the error, so the ? at the call site ends the parse. Nothing is stored, nothing is allocated; the diagnostic is the Err value the caller already gets. This is what Parser::new gives you, and every chapter so far has used it without saying so.
• Verbose — the same call records the error and returns Ok, so the ? does nothing and the parser keeps going. At the end you read the whole harvest off the emitter.

Same parser code, same ?s, opposite behaviour. That is the whole point of the design: you do not write a “collecting parser” and a “fail-fast parser” — you write a parser and hand it an emitter.

Two tiers, and only two

Severity has exactly two rungs — Error and Warning — and the tier is a classification, not a control-flow decision. A warning is never fatal: Fatal has no warning sink and drops it on the floor; Verbose files it in a channel parallel to the errors. Note that both tiers carry the same payload type — your error enum — so a warning is a value of it too.

Labels: “while parsing X”

labelled(name, parser) pushes a &'static str onto the emitter’s open -label stack for the duration of a sub-parse. Every diagnostic recorded inside is stamped with the labels open at emit time — a snapshot, not a pointer — which is what makes labels survive chapter 6: a rollback that drops a diagnostic drops its labels with it, and a re-emission on the committed path re-derives them from the then-current stack. Around a non-collecting emitter both the push and the pop inline away to nothing, so labelled is free when nobody is listening.

Reading the harvest

Verbose exposes span-keyed channels — errors(), warnings(), labels() (parallel to errors(), span-for-span and index-for-index), and skipped_regions() (chapter 8’s recovery holes) — plus one view that the maps cannot express on their own: diagnostics() walks every channel interleaved in true emission order, each entry a Diagnostic carrying its span, its label snapshot, and its DiagnosticKind. That is the view a renderer wants; tokora ships the data and takes on no dependency on ariadne, miette, or anything else.

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, PartialEq, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Tok {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))]
  Int(i64),
  #[token("let")] Let,
  #[token("print")] Print,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[token("+")] Plus,
  #[token("-")] Minus,
  #[token("*")] Star,
  #[token("/")] Slash,
  #[token("^")] Caret,
  #[token("=")] Assign,
  #[token(";")] Semi,
  #[token(",")] Comma,
  #[token("(")] LParen,
  #[token(")")] RParen,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokKind { Int, Let, Print, Ident, Plus, Minus, Star, Slash, Caret, Assign, Semi, Comma, LParen, RParen }
impl core::fmt::Display for TokKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Self::Int => "integer", Self::Let => "`let`", Self::Print => "`print`",
      Self::Ident => "identifier", Self::Plus => "`+`", Self::Minus => "`-`",
      Self::Star => "`*`", Self::Slash => "`/`", Self::Caret => "`^`",
      Self::Assign => "`=`", Self::Semi => "`;`", Self::Comma => "`,`",
      Self::LParen => "`(`", Self::RParen => "`)`",
    })
  }
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    match self {
      Tok::Int(n) => write!(f, "{n}"),
      other => core::fmt::Display::fmt(&other.kind(), f),
    }
  }
}
impl TokenT<'_> for Tok {
  type Kind = TokKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokKind {
    match self {
      Tok::Int(_) => TokKind::Int, Tok::Let => TokKind::Let, Tok::Print => TokKind::Print,
      Tok::Ident => TokKind::Ident, Tok::Plus => TokKind::Plus, Tok::Minus => TokKind::Minus,
      Tok::Star => TokKind::Star, Tok::Slash => TokKind::Slash, Tok::Caret => TokKind::Caret,
      Tok::Assign => TokKind::Assign, Tok::Semi => TokKind::Semi, Tok::Comma => TokKind::Comma,
      Tok::LParen => TokKind::LParen, Tok::RParen => TokKind::RParen,
    }
  }
  fn is_trivia(&self) -> bool { false }
}
type CalcLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
use tokora::error::{UnexpectedEot, token::UnexpectedToken};
#[derive(Debug, Clone, PartialEq)]
enum CalcError { Lex, Unexpected, UnexpectedEnd }
impl From<LexError> for CalcError { fn from(_: LexError) -> Self { CalcError::Lex } }
impl<'a, T, K: Clone, S, Lang: ?Sized> From<UnexpectedToken<'a, T, K, S, Lang>> for CalcError {
  fn from(_: UnexpectedToken<'a, T, K, S, Lang>) -> Self { CalcError::Unexpected }
}
impl<O, Lang: ?Sized, Set: Clone + 'static> From<UnexpectedEot<O, Lang, Set>> for CalcError {
  fn from(_: UnexpectedEot<O, Lang, Set>) -> Self { CalcError::UnexpectedEnd }
}
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for CalcError {
  fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { CalcError::UnexpectedEnd }
}
use tokora::{
  Emitter, InputRef, Parse, ParseContext, ParseInput, Parser,
  cache::DefaultCache,
  emitter::{Severity, Verbose},
  labelled,
  span::Spanned,
};

#[derive(Debug, Clone, PartialEq)]
enum Stmt<'a> {
  Let(&'a str, i64),
  Print(i64),
}

fn expect_int<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<i64, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  match inp.next()? {
    Some(tok) => match tok.into_data() {
      Tok::Int(n) => Ok(n),
      _ => Err(CalcError::Unexpected),
    },
    None => Err(CalcError::UnexpectedEnd),
  }
}
/// Skip to the end of the broken statement so the parse has somewhere to resume.
/// (Chapter 8 replaces this with real, nesting-aware recovery.)
fn skip_to_semi<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<(), CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  while let Some(tok) = inp.next()? {
    if matches!(tok.data(), Tok::Semi) {
      break;
    }
  }
  Ok(())
}

/// `let <ident> = <int> ;`, with the `let` already consumed by the caller.
///
/// `Ok(None)` means "reported and resynchronised" — the statement is gone, the parse is not.
/// (Hidden alongside: `expect_int`, chapter 2's helper.)
fn parse_let<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Option<Stmt<'inp>>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  if inp.try_expect(|t| matches!(t.data(), Tok::Ident))?.is_none() {
    let at = *inp.span();
    // THE line this chapter is about. Under `Fatal` this `?` propagates and the parse is
    // over; under `Verbose` the error is filed and execution simply continues to the next
    // statement. The parser does not know, and does not need to.
    inp.emit_error(Spanned::new(at, CalcError::Unexpected))?;
    skip_to_semi(inp)?;
    return Ok(None);
  }
  let name = inp.slice();
  if inp.try_expect(|t| matches!(t.data(), Tok::Assign))?.is_none() {
    return Err(CalcError::Unexpected);
  }
  let value = expect_int(inp)?;
  if inp.try_expect(|t| matches!(t.data(), Tok::Semi))?.is_none() {
    return Err(CalcError::Unexpected);
  }
  Ok(Some(Stmt::Let(name, value)))
}

/// `print <int> ;`, with the `print` already consumed.
fn parse_print<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Option<Stmt<'inp>>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  let value = expect_int(inp)?;
  if inp.try_expect(|t| matches!(t.data(), Tok::Semi))?.is_none() {
    return Err(CalcError::Unexpected);
  }
  Ok(Some(Stmt::Print(value)))
}

fn parse_program<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Vec<Stmt<'inp>>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  let mut stmts = Vec::new();
  while let Some(head) = inp.next()? {
    let at = *inp.span();
    match head.into_data() {
      // The label is pushed for the sub-parse and popped afterwards; anything the
      // sub-parse emits is stamped with it.
      Tok::Let => {
        if let Some(stmt) = labelled("a `let` binding", parse_let).parse_input(inp)? {
          stmts.push(stmt);
        }
      }
      Tok::Print => {
        if let Some(stmt) = labelled("a `print` statement", parse_print).parse_input(inp)? {
          stmts.push(stmt);
        }
      }
      // An empty statement. Worth saying, not worth stopping for — so it is a *warning*,
      // and a warning is never fatal: `Fatal` drops this on the floor and carries on.
      Tok::Semi => {
        inp.emit_warning(Spanned::new(at, CalcError::Unexpected))?;
      }
      _ => {
        inp.emit_error(Spanned::new(at, CalcError::Unexpected))?;
        skip_to_semi(inp)?;
      }
    }
  }
  Ok(stmts)
}

// A program with one stray `;` (a warning) and one broken `let` (an error).
const SRC: &str = "let x = 1 ; ; let = 2 ; print 3 ;";

// ── Fatal (what `Parser::new()` hands you): the first error is the last event. ──
assert_eq!(
  Parser::new().apply(parse_program).parse_str(SRC),
  Err(CalcError::Unexpected)
);

// ── Verbose: the very same `parse_program`, run to the end of the file. ──
let mut emitter = Verbose::<CalcError>::new();
let cache = DefaultCache::<'_, CalcLexer<'_>>::default();
let stmts = Parser::with_context((&mut emitter, cache))
  .apply(parse_program)
  .parse_str(SRC)
  .expect("Verbose never fails the parse: it files the diagnostics instead");

// The good statements all came through; the broken one is simply absent.
assert_eq!(stmts, [Stmt::Let("x", 1), Stmt::Print(3)]);

// Errors and warnings are independent channels, each keyed by span.
assert_eq!(emitter.errors().values().flatten().count(), 1);
assert_eq!(emitter.warnings().values().flatten().count(), 1);

// `labels()` is parallel to `errors()`: same span, same index. The broken `let` was
// emitted inside the labelled sub-parse, so it knows what it was doing at the time.
let (span, group) = emitter.errors().iter().next().expect("one error");
assert_eq!(group.as_slice(), &[CalcError::Unexpected]);
assert_eq!(emitter.labels()[span], vec![vec!["a `let` binding"]]);

// And `diagnostics()` interleaves every channel in *emission* order — the stray `;`
// warning was emitted before the broken `let`, and here that is visible. The span-keyed
// maps above cannot tell you this; this is the view a renderer consumes.
let timeline: Vec<Severity> = emitter.diagnostics().map(|d| d.severity()).collect();
assert_eq!(timeline, [Severity::Warning, Severity::Error]);

Expected sets come for free

Not every diagnostic is one you write. The combinators build structured errors themselves — UnexpectedToken with an Expected set, UnexpectedEnd at end of input, TooFew from a bounded repetition — and hand them to the emitter through the same two verbs. Chapter 4’s dispatch_on_kind is the clearest case: on a miss it reports the whole table as expected one of …, which it can only do because the decision is committed and never rolled back. Your error enum absorbs each of these through a From impl — that is what the From impls on CalcError have been for since chapter 2.

Choosing

Reach for Fatal when the first error ends the job anyway (a config file, a query, a protocol frame): it stores nothing, allocates nothing, and the diagnostic is the Err you already handle. Reach for Verbose when a human is going to read the output. Silent and Ignored round out the set for the cases where you want the parse and not the diagnostics.

But notice what skip_to_semi above quietly is: a hand-rolled, bracket-blind resynchroniser that would happily stop at the ; inside a parenthesised expression. Collecting many errors is only half of it — the other half is landing somewhere sane afterwards. Next: chapter 8.

8. Recovery

Chapter 7’s parser collects many errors, but it resynchronises by scanning to the next ; and that is not good enough. Consider a broken let whose garbage contains a semicolon:

let = ( 2 ; 3 ) ;

A bracket-blind skip stops at the ; inside the parentheses, resumes at 3 ) ; — which is not a statement either — and reports a second error that exists only because the first recovery landed badly. That is a cascade, and it is why a compiler that reports twenty errors for one typo is worse than useless.

sync_balanced: the skip that can count

sync_balanced skips forward to a sync point at nesting depth zero. You give it two things:

• a classifier — which token kinds open and close a pair. Any FnMut(&Kind) -> Balance<P> is one, via the blanket DelimClass impl; Balance is Open(pair), Close(pair), or Neutral.
• a sync predicate — what you want to land on. It is consulted only at depth zero, so garbage containing balanced pairs skips straight over any sync tokens buried inside them.

It returns Option<Hole> — the region it skipped (span() and skipped(), the token count) — and stops before the sync token, leaving it for the parse that resumes.

Two properties are worth naming because they are what make it safe. Depth counting is token-level: a composite token (a block string, a raw literal) is one token whose lexer already swallowed any brackets inside it, so nothing within a token can move the depth. And it is pair-blind: a closer closes the innermost open pair whatever its identity, because inside garbage the mismatched pairs are part of what is being thrown away.

One hole, one diagnostic

A skip does not report the tokens it dropped one by one — that would be the cascade again, in a different costume. A successful sync that skipped at least one token reports the whole region exactly once through emit_skipped_region (a defaulted no-op, so a fail-fast emitter pays nothing; Verbose records it, and you read it back with skipped_regions() or interleaved in diagnostics()). A sync that finds nothing emits nothing at all and rewinds without a trace: no diagnostic for a failed hole.

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, PartialEq, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Tok {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))]
  Int(i64),
  #[token("let")] Let,
  #[token("print")] Print,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[token("+")] Plus,
  #[token("-")] Minus,
  #[token("*")] Star,
  #[token("/")] Slash,
  #[token("^")] Caret,
  #[token("=")] Assign,
  #[token(";")] Semi,
  #[token(",")] Comma,
  #[token("(")] LParen,
  #[token(")")] RParen,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokKind { Int, Let, Print, Ident, Plus, Minus, Star, Slash, Caret, Assign, Semi, Comma, LParen, RParen }
impl core::fmt::Display for TokKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Self::Int => "integer", Self::Let => "`let`", Self::Print => "`print`",
      Self::Ident => "identifier", Self::Plus => "`+`", Self::Minus => "`-`",
      Self::Star => "`*`", Self::Slash => "`/`", Self::Caret => "`^`",
      Self::Assign => "`=`", Self::Semi => "`;`", Self::Comma => "`,`",
      Self::LParen => "`(`", Self::RParen => "`)`",
    })
  }
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    match self {
      Tok::Int(n) => write!(f, "{n}"),
      other => core::fmt::Display::fmt(&other.kind(), f),
    }
  }
}
impl TokenT<'_> for Tok {
  type Kind = TokKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokKind {
    match self {
      Tok::Int(_) => TokKind::Int, Tok::Let => TokKind::Let, Tok::Print => TokKind::Print,
      Tok::Ident => TokKind::Ident, Tok::Plus => TokKind::Plus, Tok::Minus => TokKind::Minus,
      Tok::Star => TokKind::Star, Tok::Slash => TokKind::Slash, Tok::Caret => TokKind::Caret,
      Tok::Assign => TokKind::Assign, Tok::Semi => TokKind::Semi, Tok::Comma => TokKind::Comma,
      Tok::LParen => TokKind::LParen, Tok::RParen => TokKind::RParen,
    }
  }
  fn is_trivia(&self) -> bool { false }
}
type CalcLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
use tokora::error::{UnexpectedEot, token::UnexpectedToken};
#[derive(Debug, Clone, PartialEq)]
enum CalcError { Lex, Unexpected, UnexpectedEnd }
impl From<LexError> for CalcError { fn from(_: LexError) -> Self { CalcError::Lex } }
impl<'a, T, K: Clone, S, Lang: ?Sized> From<UnexpectedToken<'a, T, K, S, Lang>> for CalcError {
  fn from(_: UnexpectedToken<'a, T, K, S, Lang>) -> Self { CalcError::Unexpected }
}
impl From<UnexpectedEot> for CalcError {
  fn from(_: UnexpectedEot) -> Self { CalcError::UnexpectedEnd }
}
use tokora::{
  Balance, DelimClass, Emitter, InputRef, Parse, ParseContext, Parser,
  cache::DefaultCache,
  emitter::Verbose,
  span::Spanned,
};

#[derive(Debug, Clone, PartialEq)]
enum Stmt<'a> {
  Let(&'a str, i64),
  Print(i64),
}

fn expect_int<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<i64, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  match inp.next()? {
    Some(tok) => match tok.into_data() {
      Tok::Int(n) => Ok(n),
      _ => Err(CalcError::Unexpected),
    },
    None => Err(CalcError::UnexpectedEnd),
  }
}
fn parse_let<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Stmt<'inp>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  if inp.try_expect(|t| matches!(t.data(), Tok::Ident))?.is_none() {
    return Err(CalcError::Unexpected);
  }
  let name = inp.slice();
  if inp.try_expect(|t| matches!(t.data(), Tok::Assign))?.is_none() {
    return Err(CalcError::Unexpected);
  }
  let value = expect_int(inp)?;
  if inp.try_expect(|t| matches!(t.data(), Tok::Semi))?.is_none() {
    return Err(CalcError::Unexpected);
  }
  Ok(Stmt::Let(name, value))
}
fn parse_print<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Stmt<'inp>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  let value = expect_int(inp)?;
  if inp.try_expect(|t| matches!(t.data(), Tok::Semi))?.is_none() {
    return Err(CalcError::Unexpected);
  }
  Ok(Stmt::Print(value))
}
// (Hidden: `parse_let` and `parse_print` — chapter 4's branch parsers, with the head
//  keyword already consumed. Both simply return `Err` when they do not fit.)

/// The classifier: Calc's only pair is the parenthesis. A named `fn` (not a closure), so the
/// higher-ranked `FnMut(&Kind)` bound the blanket `DelimClass` impl wants is satisfied for
/// free.
fn parens(kind: &TokKind) -> Balance<()> {
  match kind {
    TokKind::LParen => Balance::Open(()),
    TokKind::RParen => Balance::Close(()),
    _ => Balance::Neutral,
  }
}

/// A deliberately bracket-blind classifier, so the two can be compared side by side.
fn flat(_kind: &TokKind) -> Balance<()> {
  Balance::Neutral
}

/// Parse statements; on a bad one, report it and skip to the next depth-0 `;`. Returns the
/// statements it salvaged and the size of each hole it punched.
fn recover_program<'inp, Ctx, D>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
  classifier: D,
) -> Result<(Vec<Stmt<'inp>>, Vec<usize>), CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
  D: DelimClass<TokKind> + Copy,
{
  let mut stmts = Vec::new();
  let mut holes = Vec::new();
  while let Some(head) = inp.next()? {
    let at = *inp.span();
    let parsed = match head.into_data() {
      Tok::Let => parse_let(inp),
      Tok::Print => parse_print(inp),
      _ => Err(CalcError::Unexpected),
    };
    match parsed {
      Ok(stmt) => stmts.push(stmt),
      Err(_) => {
        inp.emit_error(Spanned::new(at, CalcError::Unexpected))?;
        // The skip. `pred` is only consulted at depth zero, so a `;` inside `( … )` is
        // skipped over rather than mistaken for the end of the statement.
        if let Some(hole) = inp.sync_balanced(classifier, |t| matches!(t.data(), Tok::Semi))? {
          holes.push(hole.skipped());
        }
        // The sync stops *before* the `;`. Eat it, so the next statement starts clean.
        let _ = inp.try_expect(|t| matches!(t.data(), Tok::Semi))?;
      }
    }
  }
  Ok((stmts, holes))
}

fn program_nested<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<(Vec<Stmt<'inp>>, Vec<usize>), CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  recover_program(inp, parens)
}
fn program_flat<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<(Vec<Stmt<'inp>>, Vec<usize>), CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  recover_program(inp, flat)
}
// (Hidden: `program_nested` and `program_flat`, one-line wrappers that pin the classifier.)

// The middle statement is broken, and its garbage contains a semicolon.
const SRC: &str = "print 1 ; let = ( 2 ; 3 ) ; print 4 ;";

// ── Nesting-aware: one mistake, one hole, one error. ──
let mut emitter = Verbose::<CalcError>::new();
let cache = DefaultCache::<'_, CalcLexer<'_>>::default();
let (stmts, holes) = Parser::with_context((&mut emitter, cache))
  .apply(program_nested)
  .parse_str(SRC)
  .unwrap();

assert_eq!(stmts, [Stmt::Print(1), Stmt::Print(4)]);
// `= ( 2 ; 3 )` — seven tokens minus the `;` we stop before: the inner `;` was *inside*
// the parentheses, at depth 1, so the predicate never even saw it.
assert_eq!(holes, [6]);
assert_eq!(emitter.errors().values().flatten().count(), 1);
// The skip reported itself, once, without being asked.
assert_eq!(emitter.skipped_regions().values().flatten().count(), 1);

// ── Bracket-blind: the same code with a classifier that counts nothing. ──
let mut emitter = Verbose::<CalcError>::new();
let cache = DefaultCache::<'_, CalcLexer<'_>>::default();
let (stmts, holes) = Parser::with_context((&mut emitter, cache))
  .apply(program_flat)
  .parse_str(SRC)
  .unwrap();

assert_eq!(stmts, [Stmt::Print(1), Stmt::Print(4)]);
// The cascade, measured: the first skip stopped at the `;` *inside* the parentheses and
// resumed at `3 ) ;`, which is not a statement either — so a second error and a second
// hole were invented by the recovery itself.
assert_eq!(holes, [3, 1]);
assert_eq!(emitter.errors().values().flatten().count(), 2);
assert_eq!(emitter.skipped_regions().values().flatten().count(), 2);

skip_then_retry: recovery as a combinator

Writing the loop by hand is fine, but the common shape — try the parser; if it fails, skip to a sync point and try again — is skip_then_retry. It takes the same classifier and predicate, and it carries the thing a hand-rolled retry loop usually forgets: a mandatory progress guard. A retry cycle that consumes nothing bails out with the error that triggered it rather than spinning; a cycle that fails after real progress consumes the sync token before re-syncing, so every continuing cycle advances by at least one token and the loop provably terminates.

Note the sync predicate here is where a statement may begin, not ;. Sync to what you are about to retry.

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, PartialEq, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Tok {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))]
  Int(i64),
  #[token("let")] Let,
  #[token("print")] Print,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[token("+")] Plus,
  #[token("-")] Minus,
  #[token("*")] Star,
  #[token("/")] Slash,
  #[token("^")] Caret,
  #[token("=")] Assign,
  #[token(";")] Semi,
  #[token(",")] Comma,
  #[token("(")] LParen,
  #[token(")")] RParen,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokKind { Int, Let, Print, Ident, Plus, Minus, Star, Slash, Caret, Assign, Semi, Comma, LParen, RParen }
impl core::fmt::Display for TokKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Self::Int => "integer", Self::Let => "`let`", Self::Print => "`print`",
      Self::Ident => "identifier", Self::Plus => "`+`", Self::Minus => "`-`",
      Self::Star => "`*`", Self::Slash => "`/`", Self::Caret => "`^`",
      Self::Assign => "`=`", Self::Semi => "`;`", Self::Comma => "`,`",
      Self::LParen => "`(`", Self::RParen => "`)`",
    })
  }
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    match self {
      Tok::Int(n) => write!(f, "{n}"),
      other => core::fmt::Display::fmt(&other.kind(), f),
    }
  }
}
impl TokenT<'_> for Tok {
  type Kind = TokKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokKind {
    match self {
      Tok::Int(_) => TokKind::Int, Tok::Let => TokKind::Let, Tok::Print => TokKind::Print,
      Tok::Ident => TokKind::Ident, Tok::Plus => TokKind::Plus, Tok::Minus => TokKind::Minus,
      Tok::Star => TokKind::Star, Tok::Slash => TokKind::Slash, Tok::Caret => TokKind::Caret,
      Tok::Assign => TokKind::Assign, Tok::Semi => TokKind::Semi, Tok::Comma => TokKind::Comma,
      Tok::LParen => TokKind::LParen, Tok::RParen => TokKind::RParen,
    }
  }
  fn is_trivia(&self) -> bool { false }
}
type CalcLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
use tokora::error::{UnexpectedEot, token::UnexpectedToken};
#[derive(Debug, Clone, PartialEq)]
enum CalcError { Lex, Unexpected, UnexpectedEnd }
impl From<LexError> for CalcError { fn from(_: LexError) -> Self { CalcError::Lex } }
impl<'a, T, K: Clone, S, Lang: ?Sized> From<UnexpectedToken<'a, T, K, S, Lang>> for CalcError {
  fn from(_: UnexpectedToken<'a, T, K, S, Lang>) -> Self { CalcError::Unexpected }
}
impl From<UnexpectedEot> for CalcError {
  fn from(_: UnexpectedEot) -> Self { CalcError::UnexpectedEnd }
}
use tokora::{
  Balance, Emitter, InputRef, Parse, ParseContext, ParseInput, Parser,
  cache::DefaultCache,
  emitter::Verbose,
  error::{MaybeIncomplete, MaybeTerminal},
};

// THE NEVER-RECOVERABLE LAW AND ITS TERMINAL DUAL. `skip_then_retry` requires the emitter's error
// type to answer two questions: *are you an `Incomplete`?* and *are you a terminal stop?* Either
// one is re-raised untouched, before any skip and from any retry — recovery synthesises
// progress over a *malformed* construct, while an incomplete one is merely *unfinished* (skipping
// it throws away input that has not arrived) and a terminal stop is a limit no skip can clear.
// `CalcError` is never either: it parses whole strings with no scanner limiter, it never descends
// (no pratt engine, no `InputRef::descend`), and it is not driven through a `PartialSession`, so
// no terminal stop can reach it. Note which ground does the work on the scanner side: *no limiter*,
// not *an accepting emitter*. A limiter that could refuse would reach this type by two routes, not
// one — the marked `UnexpectedEnd` when the emitter takes the diagnostic, and a bare lexer error
// when it rejects it — and having no limiter at all closes both. The traits' default answers —
// `false` — are the right ones. A grammar that *does* have a limiter, a descent budget, or a
// session stores the value and answers for it; `MaybeTerminal` has the three sources this crate
// builds, the arm each one needs, and the rule for a terminal condition it cannot name.
impl MaybeIncomplete for CalcError {}
impl MaybeTerminal for CalcError {}

#[derive(Debug, Clone, PartialEq)]
enum Stmt<'a> {
  Let(&'a str, i64),
  Print(i64),
}
fn expect_int<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<i64, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  match inp.next()? {
    Some(tok) => match tok.into_data() {
      Tok::Int(n) => Ok(n),
      _ => Err(CalcError::Unexpected),
    },
    None => Err(CalcError::UnexpectedEnd),
  }
}
fn parse_stmt<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Stmt<'inp>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  match inp.next()? {
    Some(tok) => match tok.into_data() {
      Tok::Let => {
        if inp.try_expect(|t| matches!(t.data(), Tok::Ident))?.is_none() {
          return Err(CalcError::Unexpected);
        }
        let name = inp.slice();
        if inp.try_expect(|t| matches!(t.data(), Tok::Assign))?.is_none() {
          return Err(CalcError::Unexpected);
        }
        let value = expect_int(inp)?;
        if inp.try_expect(|t| matches!(t.data(), Tok::Semi))?.is_none() {
          return Err(CalcError::Unexpected);
        }
        Ok(Stmt::Let(name, value))
      }
      Tok::Print => {
        let value = expect_int(inp)?;
        if inp.try_expect(|t| matches!(t.data(), Tok::Semi))?.is_none() {
          return Err(CalcError::Unexpected);
        }
        Ok(Stmt::Print(value))
      }
      _ => Err(CalcError::Unexpected),
    },
    None => Err(CalcError::UnexpectedEnd),
  }
}
fn parens(kind: &TokKind) -> Balance<()> {
  match kind {
    TokKind::LParen => Balance::Open(()),
    TokKind::RParen => Balance::Close(()),
    _ => Balance::Neutral,
  }
}
// (Hidden: `parse_stmt` — chapter 4's dispatcher, failing with `Err` on anything that is
//  not a statement — and the `parens` classifier from above.)

/// The whole recovery policy, as one wrapper.
fn recovered_stmt<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<Stmt<'inp>, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  parse_stmt
    .skip_then_retry(parens, |t| matches!(t.data(), Tok::Let | Tok::Print))
    .parse_input(inp)
}

// A parenthesised lump of garbage — containing a `let` and a `;` that a blind skip would
// have fallen for — followed by the real statement.
let mut emitter = Verbose::<CalcError>::new();
let cache = DefaultCache::<'_, CalcLexer<'_>>::default();
let stmt = Parser::with_context((&mut emitter, cache))
  .apply(recovered_stmt)
  .parse_str("( let y = 9 ; ) let x = 1 ;")
  .unwrap();

assert_eq!(stmt, Stmt::Let("x", 1));
// One hole, seven tokens: the entire `( … )` lump. The `let` *inside* it sat at depth 1,
// so it was never a candidate sync point.
let holes: Vec<usize> = emitter.skipped_regions().values().flatten().copied().collect();
assert_eq!(holes, [7]);

Recovery inside an expression

Everything above recovers between constructs: a statement fails, the driver skips to the next sync point, and the statement loop carries on. An expression cannot be recovered that way. There is no sync point inside 1 + , and skipping to one would discard the operand the user is about to type.

The Pratt driver’s answer is the rust-analyzer one: report the missing operand, hand the driver an error node as the operand, and let the fold and the loop continue over it. It works because the LHS channel’s Operand report — unlike Prefix — is not held to “consume what you report”, so an operand of zero width is legal. The full argument, the recovery-set rule that keeps such a parse terminating, and how the two kinds of recovery meet at an inplace_recover boundary are in chapter 5; the worked grammar is examples/expr_recovery.rs.

The law, once more

Recovery skips input. An Incomplete error says there is more input coming. Skipping past it would throw away bytes that have not arrived yet — so both skip_then_retry and Recover check is_incomplete() before they skip, and re-raise such an error untouched. Recovery is for the malformed, never for the unfinished.

That guarantee is what makes the next chapter safe to build. Next: chapter 9.

9. Partial input

Every parse so far started with the whole source in hand. A network socket does not work that way: bytes arrive in chunks, and a parser that must wait for the last one is a parser that cannot stream. But the naive fix is a bug factory. Given the bytes 1 + 2, is the final token the integer 2? Or is it the first digit of 23, whose second digit is still in flight? A parser that guesses will, sooner or later, guess wrong.

tokora’s answer is to make the question representable, in the type system.

The Completeness typestate

An input carries a Completeness parameter:

• Complete — the default, and what chapters 1-8 have been using without ever saying so. The source is all of it. Every frontier rule below is written if Cmpl::PARTIAL && …, so under Complete they are compiled out of existence: the typestate is not a runtime mode, and the fast path pays exactly nothing for the existence of the slow one.
• Partial — the source is a prefix of a stream that may still grow. It carries one runtime bit, is_final, that the driver states when the last chunk lands — parse_partial’s is_final argument.

The three frontier rules

While a partial input is non-final, three conservative rules fire at the scan chokepoint, each surfacing an Incomplete rather than committing to an answer that later bytes could contradict:

1. The holdback. A token the lexer decided by reading as far as the end of the buffer is withheld — it might be the prefix of a longer one.
2. A lexer error at the frontier is withheld the same way: garbage that abuts the end of the buffer might be the beginning of something valid.
3. End of input, when the input is not final, is not end of input. It is a request for more bytes.

“Read as far as the end” is a fact the lexer reports, through Lexer::read_frontier — not the item’s span reaching the end. The two coincide only for a lexer that never reads past what it emits (SpanEnd). A lexer that probes ahead and backtracks — which the bundled logos backend’s DFA does whenever one pattern is a proper prefix of another, an integer beside a float — decides an item at 0..1 by looking at byte 2, and rule 1 holds that item back even though its span sits behind the end. Keying on the span instead was the pre-0.10.0 proxy, and it committed exactly the items one more byte would change.

Rule 1 is the one you feel, and it has a name: frontier latency. A token whose decision consulted the end of a non-final buffer becomes visible only when more input arrives or is_final is set. That is not a limitation to be engineered around — it is the only sound answer. The sole proof that 2 is not the start of 23 is another byte, or a promise that there will not be one. How many tokens it costs is the lexer’s answer, not the buffer’s: one for a SpanEnd lexer, more for a lookahead one, and every token until the seal for one that reports Unbounded.

is_final belongs to the driver, and it only goes one way

Notice who makes that promise. is_final is not a fact about the parse — it is a fact about the world: the caller has told us no more bytes are coming. A parser combinator cannot possibly know it. Only the code holding the socket can.

So there is no set_final on an InputRef, and there never will be. You state finality where you build the input — parse_partial’s is_final argument — and the parser you hand the input to simply cannot reach it. That is enforced by the borrow checker, not by convention: the flag lives on the input, the handle borrows the input, and the borrow lasts as long as the handle does.

Two bugs fall out of that one line, and it is worth seeing both, because they are mirrors:

• A parser that could end a stream would break the holdback. Speculate, call set_final(true), fail, roll back — and the rollback would not undo it, because rolling back the world is not a thing rollback does. The next read would then hand you a token the frontier owed an Incomplete for: the very 2-that-might-be-23 this chapter is about.
• A rollback that could un-end a stream — the “obvious” fix of checkpointing the flag and restoring it — is worse. Your last chunk lands, you mark the stream final, the parser rolls back across that moment, and is_final quietly reverts to false. Now the parser asks for a refill that can never come, and your program waits forever. That trades a wrong token for a hang.

The way out is to notice that the two bugs share a premise — that a parser can touch the bit at all. Take that away and both are gone: finality is set by the driver, before any parser exists, and it is monotone (a stream cannot un-end). Nothing to roll back, and nothing that would want to.

No growable source: the caller owns the buffer

tokora has no internal growable source, and that is a deliberate architectural line, not an omission. An InputRef borrows one immutable slice for its whole life — which is precisely what makes zero-copy slices free, and makes a checkpoint a snapshot copy rather than a journalled edit. Backtracking is cheap because the source cannot move under it.

So resumption lives with the caller. It owns the byte buffer; on an incomplete result it appends the next chunk to its own buffer and rebuilds the input over the larger slice. Re-lexing the prefix each round is cheap, and it keeps the frontier rules a pure function of the current slice — which is what “Sans-I/O” means: tokora never reads, never waits, never owns a socket. It parses what you hand it and tells you when it needs more.

parse_partial wires up one round of that loop: it builds a Partial input over your slice, seals it if this is the last chunk, and drives any ParseInput<…, Partial> — a typed fn like the one below, a named combinator chain, or a parser written generic over its completeness parameter. Since 0.3.0 that bound is the partial driver’s whole signature: the closure era’s bare-FnOnce bypass is gone, and partial mode is simply the other instantiation of the same trait vocabulary the complete drivers use.

Partial mode adds exactly two requirements to your error type, both one-liners: it must implement From<Incomplete<L::Offset>>, so the frontier has a way to speak — and MaybeIncomplete, so the frontier can be recognized: your refill loop keys off is_incomplete(), and inside the parse the resilient collection loops (repeated/separated) consult the same trait to re-raise a frontier Incomplete untouched instead of spending it as a diagnostic — the never-recoverable law, enforced at the atom layer.

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, PartialEq, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Tok {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))]
  Int(i64),
  #[token("let")] Let,
  #[token("print")] Print,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[token("+")] Plus,
  #[token("-")] Minus,
  #[token("*")] Star,
  #[token("/")] Slash,
  #[token("^")] Caret,
  #[token("=")] Assign,
  #[token(";")] Semi,
  #[token(",")] Comma,
  #[token("(")] LParen,
  #[token(")")] RParen,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokKind { Int, Let, Print, Ident, Plus, Minus, Star, Slash, Caret, Assign, Semi, Comma, LParen, RParen }
impl core::fmt::Display for TokKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Self::Int => "integer", Self::Let => "`let`", Self::Print => "`print`",
      Self::Ident => "identifier", Self::Plus => "`+`", Self::Minus => "`-`",
      Self::Star => "`*`", Self::Slash => "`/`", Self::Caret => "`^`",
      Self::Assign => "`=`", Self::Semi => "`;`", Self::Comma => "`,`",
      Self::LParen => "`(`", Self::RParen => "`)`",
    })
  }
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    match self {
      Tok::Int(n) => write!(f, "{n}"),
      other => core::fmt::Display::fmt(&other.kind(), f),
    }
  }
}
impl TokenT<'_> for Tok {
  type Kind = TokKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokKind {
    match self {
      Tok::Int(_) => TokKind::Int, Tok::Let => TokKind::Let, Tok::Print => TokKind::Print,
      Tok::Ident => TokKind::Ident, Tok::Plus => TokKind::Plus, Tok::Minus => TokKind::Minus,
      Tok::Star => TokKind::Star, Tok::Slash => TokKind::Slash, Tok::Caret => TokKind::Caret,
      Tok::Assign => TokKind::Assign, Tok::Semi => TokKind::Semi, Tok::Comma => TokKind::Comma,
      Tok::LParen => TokKind::LParen, Tok::RParen => TokKind::RParen,
    }
  }
  fn is_trivia(&self) -> bool { false }
}
type CalcLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
use tokora::error::{UnexpectedEot, token::UnexpectedToken};
use tokora::{
  InputRef, Partial, parse_partial,
  cache::DefaultCache,
  emitter::Fatal,
  error::{Incomplete, MaybeIncomplete},
};

// Chapter 3's `CalcError`, plus the one variant partial mode asks for.
#[derive(Debug, Clone, PartialEq)]
enum CalcError {
  Lex,
  Unexpected,
  UnexpectedEnd,
  /// The frontier speaking: "ask me again when you have more bytes."
  Incomplete,
}
impl From<LexError> for CalcError { fn from(_: LexError) -> Self { CalcError::Lex } }
impl<'a, T, K: Clone, S, Lang: ?Sized> From<UnexpectedToken<'a, T, K, S, Lang>> for CalcError {
  fn from(_: UnexpectedToken<'a, T, K, S, Lang>) -> Self { CalcError::Unexpected }
}
impl From<UnexpectedEot> for CalcError {
  fn from(_: UnexpectedEot) -> Self { CalcError::UnexpectedEnd }
}
// THE requirement. `L::Offset` is `usize` for a `str` source: the offset the input ran out at.
impl From<Incomplete<usize>> for CalcError {
  fn from(_: Incomplete<usize>) -> Self {
    CalcError::Incomplete
  }
}

// And how a caller *recognises* it — the same trait chapter 8's recovery consults before it
// dares to skip anything.
impl MaybeIncomplete for CalcError {
  fn is_incomplete(&self) -> bool {
    matches!(self, CalcError::Incomplete)
  }
}

type CalcCtx<'a> = (Fatal<CalcError>, DefaultCache<'a, CalcLexer<'a>>);

/// Sum every integer in the chunk. The `Partial` in the signature is the whole difference:
/// the frontier rules exist for this parser and are compiled away for a `Complete` one.
fn sum<'inp>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, CalcCtx<'inp>, (), Partial>,
) -> Result<i64, CalcError> {
  // Rollback-on-drop (chapter 6). An incomplete attempt must leave *no trace*, because the
  // caller is about to re-drive this same parser over a longer buffer.
  let mut txn = inp.begin();
  let mut total = 0i64;
  while let Some(tok) = txn.next()? {
    // ↑ the frontier rules live in `next()`; a withheld token surfaces here as `Incomplete`,
    //   the `?` propagates it, and the guard's drop rewinds everything on the way out.
    match tok.into_data() {
      Tok::Int(n) => total += n,
      Tok::Plus => {}
      _ => return Err(CalcError::Unexpected),
    }
  }
  txn.commit();
  Ok(total)
}

fn fresh_ctx<'a>() -> CalcCtx<'a> {
  (Fatal::of(), DefaultCache::<'a, CalcLexer<'a>>::default())
}

// ── The flip. Same bytes, one bit of difference. ──

// Not final: the `2` touches the end of the buffer, so it is withheld. It might yet be a
// `23`, and nothing in these bytes can prove otherwise.
assert_eq!(
  parse_partial(fresh_ctx(), "1 + 2", (), false, sum),
  Err(CalcError::Incomplete)
);

// Final: the promise that no more bytes are coming. The frontier rules go inert, the token
// yields, and the parse finishes — this is now *exactly* a `Complete` parse.
assert_eq!(parse_partial(fresh_ctx(), "1 + 2", (), true, sum), Ok(3));

// ── And the loop that falls out of it: the caller owns the buffer. ──

let chunks = ["1 +", " 2", " + 30"];
let mut buffer = String::new();
let mut refills = 0;
let mut answer = None;

for (i, chunk) in chunks.iter().enumerate() {
  buffer.push_str(chunk); // the growable thing is *yours*, not tokora's
  let is_final = i + 1 == chunks.len();
  match parse_partial(fresh_ctx(), buffer.as_str(), (), is_final, sum) {
    Ok(total) => {
      answer = Some(total);
      break;
    }
    // Not a failure — a request. Append the next chunk and re-drive over the longer slice.
    Err(e) if e.is_incomplete() => refills += 1,
    Err(other) => panic!("a real parse error: {other:?}"),
  }
}

assert_eq!(answer, Some(33));
// Each non-final chunk ended mid-token, so each one cost exactly one refill: that is the
// one-token frontier latency, and it is the whole price of correctness here.
assert_eq!(refills, 2);

Write once, run in both modes

Everything above used a parser whose signature names Partial. That is one honest way to write a streaming parser — but 0.3.0’s point is that you do not have to choose. Write the parser generic over its completeness and it is one item with two instantiations: the complete combinator driver pins Cmpl = Complete, parse_partial pins Cmpl = Partial, and the frontier rules exist in exactly one of the two monomorphizations. The bound to reach for is SurfaceIncomplete — the Completeness refinement the scan chokepoint itself uses, satisfied by Complete unconditionally and by Partial wherever the error type meets the two requirements above.

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, PartialEq, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Tok {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))]
  Int(i64),
  #[token("let")] Let,
  #[token("print")] Print,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[token("+")] Plus,
  #[token("-")] Minus,
  #[token("*")] Star,
  #[token("/")] Slash,
  #[token("^")] Caret,
  #[token("=")] Assign,
  #[token(";")] Semi,
  #[token(",")] Comma,
  #[token("(")] LParen,
  #[token(")")] RParen,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokKind { Int, Let, Print, Ident, Plus, Minus, Star, Slash, Caret, Assign, Semi, Comma, LParen, RParen }
impl core::fmt::Display for TokKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Self::Int => "integer", Self::Let => "`let`", Self::Print => "`print`",
      Self::Ident => "identifier", Self::Plus => "`+`", Self::Minus => "`-`",
      Self::Star => "`*`", Self::Slash => "`/`", Self::Caret => "`^`",
      Self::Assign => "`=`", Self::Semi => "`;`", Self::Comma => "`,`",
      Self::LParen => "`(`", Self::RParen => "`)`",
    })
  }
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    match self {
      Tok::Int(n) => write!(f, "{n}"),
      other => core::fmt::Display::fmt(&other.kind(), f),
    }
  }
}
impl TokenT<'_> for Tok {
  type Kind = TokKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokKind {
    match self {
      Tok::Int(_) => TokKind::Int, Tok::Let => TokKind::Let, Tok::Print => TokKind::Print,
      Tok::Ident => TokKind::Ident, Tok::Plus => TokKind::Plus, Tok::Minus => TokKind::Minus,
      Tok::Star => TokKind::Star, Tok::Slash => TokKind::Slash, Tok::Caret => TokKind::Caret,
      Tok::Assign => TokKind::Assign, Tok::Semi => TokKind::Semi, Tok::Comma => TokKind::Comma,
      Tok::LParen => TokKind::LParen, Tok::RParen => TokKind::RParen,
    }
  }
  fn is_trivia(&self) -> bool { false }
}
type CalcLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
use tokora::error::{UnexpectedEot, token::UnexpectedToken};
use tokora::{
  InputRef, Partial, parse_partial,
  cache::DefaultCache,
  emitter::Fatal,
  error::{Incomplete, MaybeIncomplete},
};

// Chapter 3's `CalcError`, plus the one variant partial mode asks for.
#[derive(Debug, Clone, PartialEq)]
enum CalcError {
  Lex,
  Unexpected,
  UnexpectedEnd,
  /// The frontier speaking: "ask me again when you have more bytes."
  Incomplete,
}
impl From<LexError> for CalcError { fn from(_: LexError) -> Self { CalcError::Lex } }
impl<'a, T, K: Clone, S, Lang: ?Sized> From<UnexpectedToken<'a, T, K, S, Lang>> for CalcError {
  fn from(_: UnexpectedToken<'a, T, K, S, Lang>) -> Self { CalcError::Unexpected }
}
impl From<UnexpectedEot> for CalcError {
  fn from(_: UnexpectedEot) -> Self { CalcError::UnexpectedEnd }
}
// THE requirement. `L::Offset` is `usize` for a `str` source: the offset the input ran out at.
impl From<Incomplete<usize>> for CalcError {
  fn from(_: Incomplete<usize>) -> Self {
    CalcError::Incomplete
  }
}

// And how a caller *recognises* it — the same trait chapter 8's recovery consults before it
// dares to skip anything.
impl MaybeIncomplete for CalcError {
  fn is_incomplete(&self) -> bool {
    matches!(self, CalcError::Incomplete)
  }
}

type CalcCtx<'a> = (Fatal<CalcError>, DefaultCache<'a, CalcLexer<'a>>);

/// Sum every integer in the chunk. The `Partial` in the signature is the whole difference:
/// the frontier rules exist for this parser and are compiled away for a `Complete` one.
fn sum<'inp>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, CalcCtx<'inp>, (), Partial>,
) -> Result<i64, CalcError> {
  // Rollback-on-drop (chapter 6). An incomplete attempt must leave *no trace*, because the
  // caller is about to re-drive this same parser over a longer buffer.
  let mut txn = inp.begin();
  let mut total = 0i64;
  while let Some(tok) = txn.next()? {
    // ↑ the frontier rules live in `next()`; a withheld token surfaces here as `Incomplete`,
    //   the `?` propagates it, and the guard's drop rewinds everything on the way out.
    match tok.into_data() {
      Tok::Int(n) => total += n,
      Tok::Plus => {}
      _ => return Err(CalcError::Unexpected),
    }
  }
  txn.commit();
  Ok(total)
}

fn fresh_ctx<'a>() -> CalcCtx<'a> {
  (Fatal::of(), DefaultCache::<'a, CalcLexer<'a>>::default())
}
use tokora::{Parse, Parser, input::SurfaceIncomplete};

/// ONE parser. The only new thing is the `Cmpl` parameter where chapter 9's `sum` wrote
/// `Partial`: same transaction, same loop, same commit.
fn sum_generic<'inp, Cmpl>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, CalcCtx<'inp>, (), Cmpl>,
) -> Result<i64, CalcError>
where
  Cmpl: SurfaceIncomplete<'inp, CalcLexer<'inp>, CalcCtx<'inp>, ()>,
{
  let mut txn = inp.begin();
  let mut total = 0i64;
  while let Some(tok) = txn.next()? {
    match tok.into_data() {
      Tok::Int(n) => total += n,
      Tok::Plus => {}
      _ => return Err(CalcError::Unexpected),
    }
  }
  txn.commit();
  Ok(total)
}

// (a) The COMPLETE drive: the ordinary whole-input combinator API, `Cmpl = Complete`.
let complete = Parser::with_context(fresh_ctx())
  .apply(sum_generic)
  .parse_str("1 + 2 + 30");
assert_eq!(complete, Ok(33));

// (b) The PARTIAL drive: the SAME item over chunks, `Cmpl = Partial`.
let chunks = ["1 +", " 2", " + 30"];
let mut buffer = String::new();
let mut refills = 0;
let mut answer = None;
for (i, chunk) in chunks.iter().enumerate() {
  buffer.push_str(chunk);
  let is_final = i + 1 == chunks.len();
  match parse_partial(fresh_ctx(), buffer.as_str(), (), is_final, sum_generic) {
    Ok(total) => {
      answer = Some(total);
      break;
    }
    Err(e) if e.is_incomplete() => refills += 1,
    Err(other) => panic!("a real parse error: {other:?}"),
  }
}

// One parser, two modes, one answer — and the exact frontier latency of the typed version.
assert_eq!(answer, Some(33));
assert_eq!(refills, 2);

This scales past single functions: the builder methods thread the same parameter, so a whole chain — expect(…).map(…).repeated().collect() — assembled inside a Cmpl-generic function runs under both drivers too. The combinators that cannot run partial yet (the decision-window *_while/peek/dispatch families and the CST node family — see the combinator reference) stay pinned at Complete, so reaching for one from partial code is a compile error at the drive site, never a silent wrong parse.

The bigger example

The input module documents the Sans-I/O resumption loop end to end, with a hand-written lexer instead of a logos one — worth reading once, because it shows the frontier rules interacting with a lex/bump implementation you can see all of.

Why chapter 8 came first

Recall the never-recoverable law: an Incomplete is re-raised untouched by every recovery combinator, checked before any skip. Now you can see what it buys. Recovery skips input on the theory that the input is wrong. At a stream frontier the input is not wrong — it is merely unfinished, and skipping it would silently discard bytes that had not arrived yet, turning a refill request into data loss. The two features compose only because that law holds: a streaming parser can use recovery, and recovery will never eat the frontier.

Calc’s implementation is complete: it lexes, parses, dispatches, computes expressions, speculates, reports, recovers, and streams. Chapter 10, Testing, finishes the fundamentals by showing how to verify a lexer and parser.

10. Testing

Calc works. But how do you know, and — the harder question — how would you find out if it stopped?

Testing a parser’s grammar is ordinary work: feed it strings, check the trees. What is not ordinary is the layer underneath. tokora’s input machinery does things to your lexer that no hand-written driver would: it truncates the token cache after a rollback, it re-lexes a rewound region on demand, it resumes a lexer from a saved State at an arbitrary offset. Every one of those moves is sound only because the Lexer contract holds. If your lexer quietly violates it — a token whose identity depends on lookahead past its own span, a with_state + bump resume that does not reproduce the suffix — nothing fails loudly. Instead your parser is subtly wrong only after a backtrack, which is the worst bug in this entire crate to find by hand.

So do not find it by hand.

The conformance kit

The conformance module (feature conformance) ships a Harness that drives your lexer against the contract and panics, with the input index, position, operation and expected-vs-got, at the first violation. Build it over a corpus with new or over, then call run. It checks:

1. replay identity — two fresh runs produce the identical token/span/slice sequence, and identical means by value: the whole token, the whole error payload, never a rendering of either and never the token’s kind standing in for it;
2. state-resume faithfulness — at every position, saving the state and resuming there reproduces the rest of the run. This is the prefix-replay assumption, verbatim;
3. monotone progress — spans advance, none is empty, and the run terminates;
4. sticky exhaustion — once lex returns None, it keeps doing so;
5. span/slice coherence — every slice equals the source over its span;
6. gap-free tiling — opt-in via lossless, for a lexer that emits trivia as tokens rather than skipping it. Calc’s skips whitespace, so Calc does not ask for this one.

On top of the trait tier it drives a real Input session through fixed, named save/peek/drain/restore schedules and requires the committed token stream to equal the straight-lex stream — no randomness, the schedules are enumerated — and run_partial adds chapter 9’s tier: for every split point of every input, a non-final drain of the prefix must yield a prefix of the items before the cut and end incomplete, while a final drain of the whole source must reproduce the complete parse exactly. An item is a committed token or a lexer error the input layer raised — a refusal is a decision about bytes exactly as a token is, and appending can turn one into the other. That is the check that catches a lexer which is unfaithful under truncation — and truncation is exactly what a stream does to you.

It asks for a prefix rather than equality because withholding more is always sound, and a lexer that reports a read frontier past its own spans does exactly that. Calc’s lexer is logos-backed and declares SCAN_LOOKAHEAD as Unbounded — the const has no default, so that is a written choice — and this run therefore withholds every item until the input is final. The run still passes, and it still catches an item that changes — a token whose kind, span or value moved, a token that became an error, an error whose payload moved. See Lexer::read_frontier for what declaring the class buys back.

Comparing the value is why the kit asks for PartialEq on your token and on its error type. It is one derive on a data type, and it is what makes the comparison total: every field participates, including the one you add next year. The alternative, a comparison key written by hand, is a projection, and the field it forgets is exactly the field that drifts. Calc’s Tok and LexError already derive it. Both entry points ask for it: run compared an error’s Debug rendering and a token’s kind until 0.10.0, so its replay-identity green meant less than the word said — two unequal error values that print alike passed, and a payload rendering a counter reddened a conforming lexer. A vocabulary whose token or error genuinely cannot be PartialEq loses the kit and recovers it with a hand-written impl or a newtype.

One thing PartialEq does not promise is reflexivity, and a payload holding an f64 that can be NaN is not equal to itself. Eq and Ord do promise it — but they are markers, and nothing checks the promise, so a Kind, a span, an offset or a source slice can break it too. Either way it is your obligation, not the kit’s — hand-write the impl that says what equality means for such a type — and either way the kit will not misreport it: a comparison decided by a value that will not equal itself is refused, tagged non-reflexive-payload and naming the component, instead of being reported as a conformance failure of your lexer.

The kit refuses only when it has nothing better to say. A difference it can convict on — one at the discriminant, one in item count, or one between two values that each equal themselves — outranks a comparison it cannot use, wherever both are available: two runs yielding [Tok(NaN)] and [Tok(NaN), Tok(0.0)] are reported as a length mismatch, because the extra item proves the divergence whatever NaN does. The refusal is the answer of last resort, and it means the kit searched and found nothing it could stand behind.

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, PartialEq, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Tok {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))]
  Int(i64),
  #[token("let")] Let,
  #[token("print")] Print,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[token("+")] Plus,
  #[token("-")] Minus,
  #[token("*")] Star,
  #[token("/")] Slash,
  #[token("^")] Caret,
  #[token("=")] Assign,
  #[token(";")] Semi,
  #[token(",")] Comma,
  #[token("(")] LParen,
  #[token(")")] RParen,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokKind { Int, Let, Print, Ident, Plus, Minus, Star, Slash, Caret, Assign, Semi, Comma, LParen, RParen }
impl core::fmt::Display for TokKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Self::Int => "integer", Self::Let => "`let`", Self::Print => "`print`",
      Self::Ident => "identifier", Self::Plus => "`+`", Self::Minus => "`-`",
      Self::Star => "`*`", Self::Slash => "`/`", Self::Caret => "`^`",
      Self::Assign => "`=`", Self::Semi => "`;`", Self::Comma => "`,`",
      Self::LParen => "`(`", Self::RParen => "`)`",
    })
  }
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    match self {
      Tok::Int(n) => write!(f, "{n}"),
      other => core::fmt::Display::fmt(&other.kind(), f),
    }
  }
}
impl TokenT<'_> for Tok {
  type Kind = TokKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokKind {
    match self {
      Tok::Int(_) => TokKind::Int, Tok::Let => TokKind::Let, Tok::Print => TokKind::Print,
      Tok::Ident => TokKind::Ident, Tok::Plus => TokKind::Plus, Tok::Minus => TokKind::Minus,
      Tok::Star => TokKind::Star, Tok::Slash => TokKind::Slash, Tok::Caret => TokKind::Caret,
      Tok::Assign => TokKind::Assign, Tok::Semi => TokKind::Semi, Tok::Comma => TokKind::Comma,
      Tok::LParen => TokKind::LParen, Tok::RParen => TokKind::RParen,
    }
  }
  fn is_trivia(&self) -> bool { false }
}
type CalcLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
use tokora::error::{UnexpectedEot, token::UnexpectedToken};
#[derive(Debug, Clone, PartialEq)]
enum CalcError { Lex, Unexpected, UnexpectedEnd }
impl From<LexError> for CalcError { fn from(_: LexError) -> Self { CalcError::Lex } }
impl<'a, T, K: Clone, S, Lang: ?Sized> From<UnexpectedToken<'a, T, K, S, Lang>> for CalcError {
  fn from(_: UnexpectedToken<'a, T, K, S, Lang>) -> Self { CalcError::Unexpected }
}
impl<O, Lang: ?Sized, Set: Clone + 'static> From<UnexpectedEot<O, Lang, Set>> for CalcError {
  fn from(_: UnexpectedEot<O, Lang, Set>) -> Self { CalcError::UnexpectedEnd }
}
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for CalcError {
  fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { CalcError::UnexpectedEnd }
}
use tokora::{
  Emitter, InputRef, Parse, ParseContext, ParseInput, Parser,
  conformance::Harness,
  traced,
};

// A corpus: the shapes Calc actually sees, plus the empty source, which is where
// off-by-one bugs live.
const CORPUS: [&str; 5] = [
  "",
  "let x = 1 ;",
  "print 1 , 2 ;",
  "( 1 + 2 ) * 3 ^ 4 ;",
  "let ab = 12 ; print ab ;",
];

// The contract, checked. Note the absence of `.lossless()`: Calc's lexer *skips* whitespace,
// so its spans legitimately leave gaps. Ask for gap-free tiling only from a lossless lexer.
Harness::<CalcLexer<'_>>::over(CORPUS).run();

// And the streaming tier: every split point of every input, chunked-equivalence checked.
Harness::<CalcLexer<'_>>::over(CORPUS).run_partial();

// ── Debugging a parse you do not understand ──
fn parse_stmt<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<i64, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  if inp.try_expect(|t| matches!(t.data(), Tok::Print))?.is_none() {
    return Err(CalcError::Unexpected);
  }
  let value = match inp.next()? {
    Some(tok) => match tok.into_data() {
      Tok::Int(n) => n,
      _ => return Err(CalcError::Unexpected),
    },
    None => return Err(CalcError::UnexpectedEnd),
  };
  if inp.try_expect(|t| matches!(t.data(), Tok::Semi))?.is_none() {
    return Err(CalcError::Unexpected);
  }
  Ok(value)
}
// (Hidden: `parse_stmt`, a `print <int> ;` parser in chapter 2's style.)

/// Wrapping a parser in [`traced`] prints an indented `enter` / `exit` transcript to stderr
/// as it runs — including the crate's own instrumented combinators, and including the
/// backtracks, which is usually the part you could not see.
fn traced_stmt<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<i64, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  traced("statement", parse_stmt).parse_input(inp)
}

// With the `trace` feature *off*, `traced(name, p)` is literally `p` — no wrapper type, no
// branch, nothing to strip before shipping. The line above can stay where it is.
assert_eq!(
  Parser::new().apply(traced_stmt).parse_str("print 7 ;"),
  Ok(7)
);

Fuzzing the machinery, not just the grammar

The conformance kit checks your lexer against the contract. The fuzz module (feature fuzz) checks the layer above it: a deterministic operation-script fuzzer that drives the input and backtracking machinery — consume, peek, the sync family, attempt, the transaction guards, stacked savepoints, session points, partial-mode chunking — against a scriptable synthetic lexer, and verifies the documented laws after every operation. run_case runs one Case; run_seeds sweeps a range of them. They are ordinary #[test]s on stable Rust — no nightly, no external fuzzer — so the arbitrary operation orders your grammar will eventually produce get exercised long before your users produce them.

A testing ladder for your language

1. Conformance your lexer, once, over a corpus of real inputs — including the empty one, the one-token one, and the one that ends mid-token. If you use a LogosLexer this is already true, and the check is cheap insurance against the day you replace LogosLexer with a custom lexer.
2. Golden-test the grammar: source in, AST out. Ordinary table tests.
3. Golden-test the diagnostics under Verbose — chapter 7’s diagnostics() view is a stable, orderable thing to snapshot. A recovery regression shows up here as a changed hole, and nowhere else.
4. Reach for traced the moment a parse surprises you, and delete nothing afterwards — it costs nothing with the feature off.

Where to go next

Calc is finished: it lexes, parses, dispatches, folds expressions by precedence, speculates and rolls back, reports many diagnostics at once, recovers without cascading, and streams. The Calc fundamentals are complete.

The four programs in examples/ — json, calculator, s_expression, and c_expression — are canonical complete programs in the same style, each leaning on a different corner of the crate. Next: chapter 11 uses them to show how a real parser comes together. Read InputRef for the primitives, the parser module for the combinator catalogue, and the emitter module when you decide what your diagnostics should do. When something behaves in a way the documentation did not predict, that is a bug in one of them — the guide’s doctests exist so that it is never quietly the guide.

The parsing engine: parse while lexing

Every parser you wrote in Part II — the plain functions over an InputRef from chapter 2, the speculative guards of chapter 6, the streaming prefixes of chapter 9 — ran on one small engine, and never had to name it. This chapter names it. It is the opening chapter of Part III, so it sets the register the rest of the part keeps: less here is how to call it, more here is why it is shaped this way.

The engine is the thing that turns a source and a lexer into a stream of tokens a combinator can consume, one token at a time, without ever building a token buffer. Understanding its two objects — the input owner and the working handle every parser is handed — and the single surface a combinator drives is the mental model the checkpoint, emitter, and CST chapters all build on.

Lex, then parse — and why tokora does neither in that order

The textbook pipeline runs in two phases: the lexer consumes the whole source into a Vec<Token>, and then the parser consumes the vector. The phases are clean to reason about and they cost a full materialized copy of the token stream — an allocation proportional to the input, touched twice (once to fill, once to drain), with the parser starting only after the lexer has finished.

Two-phase:   Source ──▶ Lexer ──▶ [ Vec<Token> ] ──▶ Parser
                                   ↑ one allocation, sized to the whole input

Tokora runs the lexer and the parser interleaved. The parser asks for a token; the engine lexes exactly one and hands it over; the parser decides and asks for the next. There is no vector between them — only a small, fixed lookahead window, buffered on the stack, for the moments a decision needs to see a token or two ahead before committing to consume them.

Parse-while-lexing:   Source ──▶ Lexer ◀──▶ Parser
                                  └── on demand, one token at a time,
                                      no token buffer between the two

The payoff is not incidental; it is the design goal the whole input layer is bent toward:

• No token buffer. Memory is O(1) in the input length beyond the lookahead window — the engine never holds the stream, only a cursor into the source and a handful of staged tokens.
• Single pass. A token is lexed and consumed in the same breath, so it is still warm in cache when the parser reads it; there is no cold second sweep over a vector.
• Streaming falls out for free. Because the parser pulls rather than the lexer pushing, a source that is only a prefix of a growing stream works with the same machinery — the frontier rules of chapter 9 live at the one point where a token is pulled.

The cost tokora pays for this is that speculation cannot be “just re-run the lexer from a saved Vec index” — there is no vector. It has to be a genuine snapshot-and-restore of the engine’s position, which is exactly what backtracking is, and why it gets its own chapter.

Two objects: the input owner and the working handle

The engine is split in two, and the split is the load-bearing design decision of the whole layer.

• The owner holds the ground truth: the borrowed source, the live lexer state, the span of the last token, the lookahead cache, the recursion budget every descent draws on, and the bookkeeping the frontier and backtracking machinery keep (the finality flag, the lexer-error dedup watermark, the poison boundary, the checkpoint lineage). It is a crate-internal type; a parser never sees it.
• The working handle, InputRef, is what every parser is handed. It is not a copy of the owner — it is a bundle of borrows into the owner, plus a borrow of the emitter. Every combinator, every guard, every speculative branch operates through one of these.

The entry-point trait Parse is the seam between the two. Its driver builds the owner, borrows a handle out of it, and runs the parser against the handle — the whole of it:

fn parse_with_state(self, src: &'inp L::Source, state: L::State) -> Result<O, Error> {
  let Parser { mut f, ctx, .. } = self;

  let mut input = Input::with_state_and_context(src, state, ctx.provide());
  let mut input_ref = input.as_ref();
  f.parse_input(&mut input_ref)
}

Read it as three steps. The ParseContext hands over what it supplies — provide() builds an InputContext, which is an emitter, a lookahead cache, and the recursion budget every descent draws on (with_recursion_limiter is where a caller changes it; see the errors, emitters & context reference for the emitter/cache pairing). The owner is built over the immutable source, the initial lexer state and that context, and keeps all three for the parse’s life. A handle is borrowed out of the owner — the emitter borrow comes with it — and the parser runs against the handle, returning a value or the emitter’s error type.

Why two objects rather than one? Because the handle being a borrow of the owner is what makes three separate guarantees hold at once, each enforced by the borrow checker rather than by convention:

• Only the driver can end a stream. Sealing a partial stream as final takes &mut Input, and a live handle already borrows the owner — so no combinator, at any depth, can claim the stream ended. That is the finality law of chapter 9, and it is a consequence of the split, not a rule bolted on.
• A checkpoint can be a pure copy. Because the source is one immutable slice the owner merely borrows, saving a position is copying a few offsets and cloning the (typically cheap) lexer state — not journalling edits to a buffer. More on this below, and in the Checkpoint & Rewind chapter.
• The scanner keeps its registers. The hot fields the per-token path touches are packed on the owner ahead of the bookkeeping; the handle borrows them directly. The abbreviated shape:

// Abbreviated: the parked-front-token, front-report and trace/witness fields are elided.
pub struct InputRef<'inp, 'closure, L, Ctx, Lang: ?Sized = (), Cmpl = Complete>
where
  L: Lexer<'inp>,
  Ctx: ParseContext<'inp, L, Lang>,
  Cmpl: Completeness,
{
  input: &'closure &'inp L::Source,        // the immutable source, borrowed
  state: &'closure mut L::State,           // the live lexer state
  span: &'closure mut L::Span,             // span of the most recently consumed token
  cache: &'closure mut Ctx::Cache,         // the lookahead window
  finality: Cmpl::Finality,                // a read-only snapshot (see chapter 9)
  emitted_error_end: &'closure mut L::Offset,       // lexer-error dedup watermark
  poison_boundary: &'closure mut Option<L::Offset>, // sticky limit-trip frontier
  recursion: &'closure mut RecursionLimiter,        // descent depth + its budget
  session: Session<'inp, 'closure, L, Ctx::Emitter, Lang>, // lineage + the emitter borrow
  _marker: PhantomData<Lang>,
}

The cursor and the frontier

The engine’s position is a Cursor — a thin wrapper over the lexer’s offset. It marks where the next token will be lexed from: with the cache empty it is the raw lex position, and with tokens staged it points at the start of the first staged token, so it always reads as the boundary between what has been consumed and what has not. A consume advances it; a peek never does. That single invariant — a peek commits no progress — is what makes the lookahead window safe to fill speculatively, and it is what the demo at the end of this chapter asserts.

The tokens themselves come from the Lexer trait’s on-demand pull. The engine builds a lexer positioned at the cursor and calls lex to produce the next token; bump is how it fast-forwards a freshly built lexer to the offset it should resume from (the engine constructs a lexer per operation rather than holding one across the whole parse):

fn lex(&mut self) -> Option<Result<Self::Token, <Self::Token as Token<'inp>>::Error>>;
fn read_frontier(&self) -> ReadFrontier<Self::Offset>;   // required: there is no default
fn bump(&mut self, n: &Self::Offset);

lex returning None is exhaustion; Some(Ok(tok)) is a token; Some(Err(e)) is a lexer error the engine routes to the emitter. All three are required: read_frontier in particular has no default, because how far a lexer has safely read is a fact only that lexer knows — see chapter 9 for what a wrong answer costs a partial drive. It is called exactly as often as the parse demands and no more — that “no more” is the whole point.

A parser is a function over the handle

There is one trait every combinator implements, and its whole surface is a single method:

pub trait ParseInput<'inp, L, O, Ctx, Lang: ?Sized = (), Cmpl = Complete> {
  fn parse_input(
    &mut self,
    input: &mut InputRef<'inp, '_, L, Ctx, Lang, Cmpl>,
  ) -> Result<O, <Ctx::Emitter as Emitter<'inp, L, Lang>>::Error>
  where
    L: Lexer<'inp>,
    Ctx: ParseContext<'inp, L, Lang>,
    Cmpl: Completeness;
}

A parser, then, is a thing that mutates the handle and yields a value or the emitter’s error. Composition is nothing more exotic than threading the same &mut InputRef through each combinator in turn: then runs one parser, then the next, on the same handle; repeated runs its inner parser against the handle until it stops; a hand-written function calls next() and try_expect() directly. There is no separate “parser value” flowing between stages — only the handle, moving forward. The tentative sibling TryParseInput has the same shape for parsers that report “did not match” without emitting.

Since 0.3.0 the trait carries the same Completeness typestate as the handle it consumes: the trailing Cmpl parameter names whether the drive is over a whole input (Complete, the default) or a still-growing chunk of a stream (Partial). The default keeps every existing signature reading exactly as before — ParseInput<'inp, L, O, Ctx> is the complete-mode trait — while a parser written generic over Cmpl runs under both modes: Parse drives it at Complete and parse_partial drives the same item at Partial. The dispatch surface stays one vocabulary, not two (see chapter 9 for the write-once story and the frontier rules the partial instantiation activates).

The consume/peek surface the handle exposes is small and deliberate:

• next — consume the next token unconditionally (Ok(None) at end of input). This is the one call that advances the committed cursor.
• peek / peek_one — fill the lookahead window without committing. peek takes a compile-time Window capacity (typenum::U1 through U32), so the maximum lookahead a grammar uses is fixed at monomorphization — there is no unbounded, hidden lookahead. Peeked tokens land in the cache and are served from there when consumed. (The cache itself is the context’s one pluggable seam here: the default is a small, fixed, stack-inline buffer, so the shipped engine allocates nothing for lookahead — a custom Cache may choose its own storage and capacity.)
• try_expect — the peek-or-take workhorse: examine the next token and either commit it (the predicate matched) or leave it staged. A predicate that never commits is a one-token peek.
• cursor, slice, span — the position, the source text of the last token, and its span.

Because lookahead is explicit and capacity-bounded, dispatch is deterministic: a combinator looks at a fixed window, decides, and commits — there is no implicit “try this whole branch and unwind if it fails” behind the scenes. Speculation exists, but you ask for it by name (the next section), which is what keeps the cost of a parse legible.

Emission rides alongside consumption

A parser does not consume on one channel and report diagnostics on another, disconnected one. The emitter is borrowed by the handle — it lives in the same session cell as the backtracking bookkeeping — so the handle’s own emit_* forwarding methods and the crate’s structured emission paths write to the same object the consume path is driving. (The emitter is reached through the handle, never handed out as a value: a &mut Ctx::Emitter is itself installable as another parse’s emitter, which is how a recording sink could be aimed at a buffer it was not built over. The callbacks that used to receive the emitter — a *_while condition, a peek_then handler, a token-level pratt fold — receive an EmitterView for the same reason: the same operations, in a value no emitter slot can take. The read side, emitter_ref, stays open.) Consumption and emission share one handle, and — the part that matters for what follows — one timeline.

Two channels ride that timeline, and neither is a second pass:

• Diagnostics. A mismatched token, a premature end, a lexer error — these are emitted through the Emitter as they are discovered, mid-consume. Whether an emission is fatal (unwind now) or merely recorded (keep going, collect more) is the emitter’s choice, not the parser’s; the same parser runs fail-fast or collecting depending only on the context it was given. The atomic emitter design behind that is the subject of the Atomic Emitter chapter.
• Committed tokens. Every token that settles flows to one emitter hook, once. That hook is the seam the lossless CST rides: a recording sink turns each settled token into a tree event, which is how every consuming combinator becomes tree-producing with no per-combinator code. A diagnostics-only emitter leaves the hook a no-op and pays nothing. The CstEmitter capability and the event stream it feeds are the subject of the event-stream CST chapter.

The reason to introduce both channels here, in the engine chapter, is that they are why backtracking has to rewind more than a cursor.

Backtracking, at a glance

The handle can save its position and later return to it. You reach for this by name — attempt / try_attempt for a single speculative closure, begin for an imperative Transaction guard — and the tutorial covered the full surface in chapter 6.

The one fact the engine chapter needs you to carry forward is this: a rollback rewinds the whole timeline, not just the cursor. When a speculative branch is abandoned, the engine restores, as a unit, the position, the lexer state, the emitted diagnostics, the lexer-error dedup watermark, and the poison boundary — so a branch that emitted an error and then backed out leaves no diagnostic behind, exactly as it leaves no cursor movement behind. Consumption and emission were on one timeline going forward; they are on one timeline going backward too.

And because the owner merely borrows an immutable source (there is no growable internal buffer to un-edit), a saved checkpoint is a pure copy of those few facts, and a restore is copying them back — not replaying a journal of edits. That is the shape of it at a glance; the mechanism — how the copy stays cheap, how nested checkpoints keep a last-in-first-out discipline, how the emitter is told to drop exactly the abandoned branch’s emissions — is deliberately left to two later chapters:

• the Checkpoint & Rewind chapter for the snapshot/restore machinery and the Checkpoint it copies;
• the Atomic Emitter chapter for how an emitter marks and rewinds its own log in step, through Emitter::rewind.

This chapter asserts only the observable half — position returns — in code below; the emission half is the emitter chapter’s to demonstrate.

The engine, end to end

One compiling parse exercises every claim above: the lexer runs on demand, a peek commits no progress, next advances the committed cursor, a combinator rides the very same handle, and a declined attempt returns the position. The lexer here is a tiny hand-written CharLexer over single-character tokens (digits and +), so nothing but core tokora types is in play.

use core::{convert::Infallible, fmt};
use tokora::{
  FatalContext, InputRef, Lexer, SimpleSpan, Token,
  error::{UnexpectedEot, token::UnexpectedToken},
};
use tokora::span::Span as _;
#[derive(Debug, PartialEq)]
struct Error;
impl From<Infallible> for Error { fn from(e: Infallible) -> Self { match e {} } }
impl<'a, T, K: Clone, S, Lang: ?Sized> From<UnexpectedToken<'a, T, K, S, Lang>> for Error { fn from(_: UnexpectedToken<'a, T, K, S, Lang>) -> Self { Error } }
impl<O, Lang: ?Sized, Set: Clone + 'static> From<UnexpectedEot<O, Lang, Set>> for Error { fn from(_: UnexpectedEot<O, Lang, Set>) -> Self { Error } }
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for Error { fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { Error } }
impl tokora::error::MaybeIncomplete for Error {}
#[derive(Debug, Clone, PartialEq)]
enum Tok { Digit(u32), Plus }
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum Kind { Digit, Plus }
impl fmt::Display for Kind { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { write!(f, "{self:?}") } }
impl Token<'_> for Tok {
  type Kind = Kind;
  type Error = Infallible;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> Kind { match self { Tok::Digit(_) => Kind::Digit, Tok::Plus => Kind::Plus } }
  fn is_trivia(&self) -> bool { false }
}
struct CharLexer<'a> { src: &'a str, pos: usize, tok: SimpleSpan, state: () }
impl<'a> Lexer<'a> for CharLexer<'a> {
  type State = (); type Source = str; type Token = Tok; type Span = SimpleSpan; type Offset = usize;
  fn new(src: &'a str) -> Self { Self { src, pos: 0, tok: SimpleSpan::new(0, 0), state: () } }
  fn with_state(src: &'a str, _: ()) -> Self { Self::new(src) }
  fn check(&self) -> Result<(), Infallible> { Ok(()) }
  fn state(&self) -> &() { &self.state }
  fn state_mut(&mut self) -> &mut () { &mut self.state }
  fn into_state(self) -> Self::State {}
  fn source(&self) -> &'a str { self.src }
  fn span(&self) -> SimpleSpan { self.tok }
  fn slice(&self) -> &'a str { &self.src[self.tok.start()..self.tok.end()] }
  fn lex(&mut self) -> Option<Result<Tok, Infallible>> {
    let bytes = self.src.as_bytes();
    while self.pos < bytes.len() && bytes[self.pos] == b' ' { self.pos += 1; }
    if self.pos >= bytes.len() { return None; }
    let (start, c) = (self.pos, bytes[self.pos] as char);
    self.pos += 1;
    self.tok = SimpleSpan::new(start, self.pos);
    Some(Ok(match c { '+' => Tok::Plus, _ => Tok::Digit(c as u32 - '0' as u32) }))
  }
  fn read_frontier(&self) -> tokora::ReadFrontier<usize> { tokora::ReadFrontier::SpanEnd }
  fn bump(&mut self, n: &usize) { self.pos += n; }
}
type Ctx<'a> = FatalContext<'a, CharLexer<'a>, Error>;
use tokora::{Parse, Parser, ParseInput as _, parser::expect, utils::Expected};

// A parser is a plain function over the working handle: it drives the same `&mut InputRef`
// that every combinator drives.
fn engine_demo<'a>(
  inp: &mut InputRef<'a, '_, CharLexer<'a>, Ctx<'a>>,
) -> Result<Vec<(Kind, usize)>, Error> {
  // The lexer has not run yet: the committed cursor sits at offset 0.
  assert_eq!(*inp.cursor().as_inner(), 0);

  // `try_expect` with a predicate that never commits is a one-token peek. The engine lexes
  // the token on demand into the lookahead window — but leaves it staged, so the cursor
  // does NOT move. A peek commits no progress.
  let peeked = inp.try_expect(|_t| false)?;
  assert!(peeked.is_none());
  assert_eq!(*inp.cursor().as_inner(), 0, "a peek commits no progress");

  // `next()` serves that staged token from the cache (no re-lex) and commits it. The token
  // carries the span the lexer computed for it — evidence the work happened on demand.
  let first = inp.next()?.expect("a first token");
  assert_eq!(first.data().kind(), Kind::Digit);
  assert_eq!(first.span().end(), 1);
  assert_eq!(*inp.cursor().as_inner(), 1, "one token consumed, cursor advanced");

  // A combinator rides the very same handle. `expect` consumes through `InputRef` exactly as
  // the hand-written calls above do — composition is threading one handle forward.
  let plus = expect(|t: &Tok| if matches!(t, Tok::Plus) { Ok(()) }
                    else { Err(Expected::one(Kind::Plus)) })
    .parse_input(inp)?;
  assert_eq!(plus.kind(), Kind::Plus);

  // Backtracking, at a glance: consume speculatively inside `attempt`, then decline. The
  // rollback returns the cursor to the begin point (and the lexer state, and any emissions
  // with it — see the Checkpoint & Rewind chapter).
  let before = *inp.cursor().as_inner();
  let declined = inp.attempt(|inp| { let _ = inp.next(); None::<()> });
  assert!(declined.is_none());
  assert_eq!(*inp.cursor().as_inner(), before, "a declined attempt rewinds position");

  // Drain whatever remains, on demand, recording each token's kind and end offset.
  let mut rest = Vec::new();
  while let Some(tok) = inp.next()? {
    rest.push((tok.data().kind(), tok.span().end()));
  }
  Ok(rest)
}

// Drive it. `parse_str` builds the owner, borrows a handle, and runs `engine_demo` against it.
let rest = Parser::with_parser(engine_demo).parse_str("1+2").unwrap();
assert_eq!(rest, vec![(Kind::Digit, 3)]); // only the final `2` is left to drain

Where to go next

You now have the mental model the rest of Part III refines along four axes:

• How the input is stored — the Source/Slice seam that lets this same engine read &str, &[u8], or an owned reference-counted buffer without the grammar changing: the Source, Slice & storage backends chapter.
• How a checkpoint is taken and restored — the pure-copy snapshot, its last-in-first-out discipline, and the Checkpoint it copies: the Checkpoint & Rewind chapter.
• How the emitter marks and rewinds its log in step with the cursor — the atomic Emitter capability family and Emitter::rewind: the Atomic Emitter chapter.
• How committed tokens become a lossless tree — the CstEmitter hook and the cst event stream it feeds: the event-stream CST chapter.

Checkpoint, rewind, and the LIFO contract

The engine chapter closed on a promise it deferred: a rollback rewinds the whole timeline, not just the cursor — the position, the lexer state, the emitted diagnostics, the lexer-error dedup watermark, and the poison boundary all return together — and it does so by copying a snapshot back, not by replaying a journal of edits. This chapter is that promise, made precise. It is the load-bearing internals chapter of Part III, and it keeps the part’s register: less how to call it, more why it is shaped this way.

The tutorial already taught the surface. Chapter 6 walked the whole public backtracking vocabulary — attempt / try_attempt, the Transaction guard, the stacked savepoints, the session points — and stated the two laws they all obey: restores are last-in, first-out, and restoring is a snapshot copy, not a journal replay. This chapter explains the single mechanism beneath all of them, why those two laws are the shape they are, and how one checkpoint manages to rewind two independent channels — the input cursor and the emitter’s diagnostic/event log — as one unit.

What a checkpoint captures

A Checkpoint is a snapshot of one lineage: the concrete history of tokens lexed and diagnostics emitted up to the instant of the save. It is a handful of copied facts and nothing more — no borrow of the input, no journal, no diff:

// Abbreviated: the debug-only cross-input witness id and the allocator-only lineage id
// (both explained below) are elided.
pub struct Checkpoint<'inp, 'closure, L: Lexer<'inp>> {
  cursor: Cursor<'inp, 'closure, L>,  // where the next token will be lexed from
  span: L::Span,                      // the last-consumed token's span
  state: L::State,                    // the live lexer regime, cloned
  emitter_checkpoint: u64,            // the emitter's emission mark (the diagnostic channel)
  emitted_error_end: L::Offset,       // the lexer-error dedup watermark
  poison_boundary: Option<L::Offset>, // the sticky terminal frontier (a latched limit trip)
  cache_pushes: u64,                  // the cache's monotone push count
}

Saving is amortized O(1): it clones the lexer state (typically cheap — often a Copy regime enum) and copies a few offsets. It never touches the source, because the source is one immutable slice the input merely borrows — the design decision the engine chapter called load-bearing. There is no growable internal buffer to un-edit, so a saved position is genuinely just those few facts, and a restore is genuinely just copying them back.

That immutability is why the primitive is a snapshot rather than a diff. A mutable-buffer parser has to record what it changed and play the changes backward; tokora has nothing to play backward. The whole of “rewind” is: overwrite the live scanning cells with the saved ones, and truncate the one growable thing — the emitter’s log — back to the saved mark.

One save, two channels

The subtle part is that a checkpoint spans two logs that a parser writes to independently.

The position channel is the scanning state: the cursor, the last-consumed span, the lexer regime, the token cache, the dedup watermark, and the poison boundary. This is the input’s own ground truth and its lineage bookkeeping, and the checkpoint carries a copy of all of it.

The emission channel is the emitter’s log — the diagnostics a parser reports and, for a recording sink, the CST events it builds. The checkpoint does not copy that log; it copies a single u64 mark into it, taken with Emitter::checkpoint at save time. On restore, the input hands that mark back through Emitter::rewind, and the emitter drops exactly the emissions recorded after it.

One save captures both channels; one restore replays both. That is the entire reason a declined speculative branch leaves no diagnostic behind, exactly as it leaves no cursor movement behind — the two were saved as a unit and are restored as a unit. The reference emitter, Verbose, makes the mechanism concrete: its checkpoint is the length of its emission log, and its rewind pops every entry recorded past the mark, newest first. Its release — the settle for a branch that was kept rather than abandoned — is a deliberate no-op, because a kept mark is just a number going out of scope. The demo at the end of this chapter drives exactly this pair.

The engine chapter framed all of this as “consumption and emission share one handle, and one timeline”. The checkpoint is the object that makes the timeline rewindable: it is the seam where the two channels are pinned to a single point and released from it together.

Why a copy, not a merge — and the cell taxonomy

The predecessor design tried to be cleverer, and that is where the bugs lived. When restore reconciles saved and current state — as the previous Verbose rewind did, keeping or dropping live diagnostics by a span-end offset heuristic (retain every diagnostic whose span ends at or before the restore offset) rather than truncating by emission order — it cannot tell two diagnostics at the same offset apart, and it silently keeps or drops the wrong one. The golden model refuses reconciliation on principle: restore overwrites the scanning cells and truncates the emission log by mark. There is no heuristic to be wrong, because there is no merge.

Making that refusal hold as the code grows takes a discipline, because “restore” means something different for each cell an input owns, and a cell added without deciding which meaning it has is precisely the defect that has shipped — twice (a cache-push counter, then the finality bit, each added next to the backtracking bookkeeping instead of through it). So every mutable cell an input owns is classified into exactly one of six classes, and the class is the restore rule:

The three lineage memos in the middle row are the checkpoint’s emitted_error_end, poison_boundary, and cache_pushes fields: facts about the saved lineage that a last-in, first-out restore returns to exactly, so they copy back verbatim. They move together for a reason — a speculative peek that trips a resource limit latches the poison boundary, emits the limit diagnostic, and lifts the dedup watermark in one step, so a restore that unwinds that peek must put all three back paired. The two structural exceptions are still lineage memos, but their mechanics differ: the live-checkpoint stack is popped through the restored id rather than snapshot-copied, and the pin set is left untouched (a restore never changes which guards are live) — as is the open session-point stack, a restore below whose base is refused by its pin.

The world fact row is the one place the discipline must not reach, and it is enforced structurally rather than remembered. A working handle borrows the input for its whole life, so the finality bit — which only the driver can flip, and only with &mut Input — is unreachable while any parser, guard, or speculative branch runs. It cannot change during a handle’s life, so no rollback can observe it change, so a checkpoint has nothing to save. Restoring it would be the mirror bug: a rollback across a legitimate seal would un-end an ended stream, and the parser would wait forever for input that will never come. This is the finality law of chapter 9, seen from the checkpoint’s side.

The control-stack fact row is that argument arriving from the other direction. A save and the restore that returns to it sit at the same frame depth by construction, so the recursion budget cannot be observed to change across the pair and there is nothing for a checkpoint to save; while an unwind pops frames a state-restore knows nothing about, so the only witness that can be right is one that pops with the frame. That witness is the Descent guard’s destructor, which behaves identically under both drop policies and in std and no_std alike. A depth carried in checkpoint state would instead be double-restored on a std unwind and leaked on a no_std one.

The taxonomy is not a comment that hopes to stay true. A single crate-internal function destructures the input exhaustively — no .. — and binds every field. Adding a cell is therefore a compile error at the guardian, at the table that asks which class the new cell is in and what restore must do to it. It is generic and never instantiated, so it costs zero bytes; it is purely a wall. (It is greppable, too: grep CELL_CENSUS finds it from anywhere in the tree.)

The last-in, first-out contract, and how misuse is caught

Restoring a checkpoint invalidates every checkpoint saved after it. This is the one law the type system does not, on its own, enforce — so it is worth being exact about why it holds and how a violation is caught.

The reason is structural, not stylistic. Restoring an older checkpoint truncates the emission log below a younger checkpoint’s mark and un-lexes the tokens the younger position depends on. A truncated log cannot be rebuilt, so there is simply no correct state a later restore of the younger checkpoint could produce. Last-in, first-out is not a convention laid over the mechanism; it is the only order in which the mechanism has a defined answer.

Three layers guard it, strongest first:

• The lifetime brand (compile time). A Checkpoint is branded with the invariant 'closure lifetime of the handle that saved it. Every handle a parser receives arrives through a for<'closure> closure, so any two handles carry rigidly distinct brands that cannot unify — and restoring a checkpoint one handle saved into a different handle is a compile error, not a runtime check. This is what makes the Transaction guard’s nesting safe: an inner guard mutably borrows its parent for its whole life, so deciding the parent while a child is still live does not compile. The most common LIFO violation is unrepresentable.

• The pin set (every allocator build, detect-at-cause). A guard, an attempt, or a session point logically borrows the timeline from its begin point forward, so it pins that begin point on the input’s lineage. A raw restore that would pop a pinned checkpoint off the lineage — a restore reaching below a live guard’s foundation — panics at the restore itself, where the mistake is made, rather than letting the guard continue on a torn base. This is a real runtime check kept in release, not a debug assertion.

• The live-checkpoint witness (debug builds). Debug builds track the live checkpoints exactly and panic on any out-of-order restore, with a message that begins non-LIFO checkpoint restore. Because cargo test compiles with debug assertions on, exercising a parser’s backtracking paths in tests surfaces a violation immediately. A companion assert re-checks that a checkpoint belongs to this input — a backstop for the one construction the brand cannot separate (two inputs borrowed in a single crate-internal scope).

Release builds without the pin trip do not check a raw non-LIFO restore, and the contract is honest about what that costs: the input is left unspecified but bounded. Even then — no undefined behavior, no leak, no panic originating in the input layer, every scan terminates (the resource-limiter state travels inside the checkpoint, so a re-reached limit re-trips rather than rescanning without bound), and the input stays usable. What is not guaranteed is diagnostic fidelity: a diagnostic may go missing or be attributed to the wrong branch. That bounded-but-imperfect floor is the whole reason the raw triple is gated away behind a feature and the guards are the supported surface.

The attached emitter answers for itself, and one of them is louder. A violation still arrives at Emitter::rewind, and an emitter that can detect the unpaired settle it produces is allowed to say so. The recording CST Sink does — see the end of this chapter. A restore the emitter refuses is not rolled back either: the emitter’s own state is untouched, but the raw restore raises from the middle of its own rollback, so the lineage is popped through the target while the position and the reporting witnesses are not restored. That sits inside the bounded envelope above, and reaching it needs the violation being reported.

The guards are the surface; raw save/restore is the valve

The raw save / restore / commit triple is the primitive everything is built on, and it is not the API you are meant to reach for. It is public only under the unstable-raw feature; without it the three methods are crate-internal, so a downstream crate cannot even express a non-LIFO restore, and the whole hazard class of the previous section is unrepresentable. The supported surface upholds the contract by construction:

• Transaction — the guard from begin. Parse through it (it dereferences to the handle), then commit to keep the work or rollback to discard it. Say nothing and the drop decides; the default is rollback, so every early exit — a break, a ?, a return — rewinds on the way out. The drop policy is a zero-sized typestate: begin_with::<Commit> flips it to keep-on-drop for operator loops whose common path is success — except on a panic unwind, which takes the rollback arm whatever the policy says (std builds), because an unwind aborts the region rather than completing it. Deciding is one branch over an Option<Checkpoint>; there is no journaling to unwind.

• StackedTransaction — the guard from begin_stacked, for several live fallback points at once. Its savepoints follow SQL semantics: rollback_to an older one destroys every younger one (out-of-order revival is impossible by construction — the savepoint vector truncates from the top), while the target stays valid for a later rollback; release forgets savepoints while keeping the progress. A SavepointId is lifetime-branded to its transaction, so it cannot outlive it, and a foreign or stale id panics in every build.

• attempt / try_attempt and the session points round out the surface for closure-shaped and externally-driven speculation respectively. Chapter 6 is the usage reference for all of these; the point here is only that each holds its begin-point checkpoint internally, settles it in exactly one of restore-or-commit, and never hands it out — so the LIFO contract is theirs to keep, not yours.

The through-line: the raw triple has a contract a human must uphold by hand, and every guard upholds it mechanically — a nested guard’s out-of-order decision is a borrow error, a raw restore below a live guard is a pinned-base panic, and a merely-dropped guard settles safely. Guards first, always.

Composing with a tree-building emitter

The one-timeline promise has to survive one more composition: the lossless CST. When a parse builds a tree, committed tokens and structure flow to a recording emitter — a cst::Sink wrapping an inner diagnostics emitter — through the same Emitter::commit_token hook the engine chapter named. The sink buffers a second log (the event stream) and forwards every diagnostic to its inner emitter, and its checkpoint / rewind / release must rewind both under the one mark the input already manages. Get that wrong and a rolled-back branch leaves a phantom node in the tree even though its diagnostic vanished — the two channels sheared apart.

The sink keeps them together with a value-keyed inner contract, and the discipline is worth naming at the architecture level (the lossless-CST chapter — chapter 16 — and the emitter reference carry the API detail). The sink’s own mark is the event-log length; at each checkpoint it also freezes, on a small stack, the inner emitter’s own checkpoint reading — a plain u64, captured by value, not a resource to reclaim. On rewind it truncates its event log to the mark, replays its undo journal, and then rewinds the inner emitter only to a reading it knows exactly:

• the captured reading of the row being rewound to — every disciplined path (a guard, an attempt, the scan family, a correct raw pair) lands here;
• nothing at all when the mark is the current length — a rewind that truncates nothing must leave the inner alone, because the surviving events are the whole log and every inner-side record they reference must survive with them;
• the inner’s construction-time reading for a full unwind to the origin (an empty event log provably pairs with the reading the inner had at construction).

Everything else is refused rather than guessed. An out-of-range future mark — one strictly above the current length, naming a log position that does not exist yet — is a total no-op on every channel: events, the mark stack, the journal, the era ledger, and the inner alike. (Clamping it to the current length instead — the pre-redesign behavior — would let a future mark spend the live row of a real checkpoint taken at that length, desyncing the two logs.) And a truncating rewind to a mid-log mark that no live row captured has no exact inner reading anywhere — an unpaired settle, which is a parser bug rather than anything a document can provoke — so it panics in every build, release included, never fabricating a reading. It used to be a debug_assertions-only wall deferring to the input’s LIFO witness above; that witness is itself debug-only, so release builds had no wall on either layer and the two logs sheared in silence. The verdict is a preflight: it is decided against the unchanged mark stack, before the row spend, the truncation, the journal replay and the ledger write, so a caught panic leaves the sink exactly as it was rather than half rewound — a wall raised after the damage would only be narrating it. The sole exception is a panic already unwinding, where reporting would abort the process instead: there the rewind degrades to a total no-op on every channel and latches, and finish/finish_partial then refuse with FinishError::UnpairedSettle rather than return the tree of a rollback that never ran. (Named without a link: this chapter is not rowan-gated, and the variant is.) The sink hands the inner only readings it can prove, or nothing.

Two consequences fall out of “value-keyed”. First, the inner emitter must itself be value-keyed — checkpoint a pure monotone reading, rewind a drop-by-value, release a no-op — which is exactly the shape of Verbose, Fatal, Silent, and Ignored. Second, release on the sink pops its own row without forwarding to the inner, because a kept reading needs no cleanup; this is what keeps a commit-heavy loop (a Pratt operator loop saves per iteration) from stranding one dead row per committed branch. The Emitter::release hook exists precisely so the input can tell a buffering sink “this mark will never be rewound to” and let it reclaim the row — one timeline, kept bounded.

The rewind, end to end

One compiling parse exercises the headline claim: a speculative branch that consumes a token and files a diagnostic, then declines — and both the cursor and the diagnostic rewind together. The emitter is Verbose so the dropped diagnostic is observable after the parse; the lexer is the same tiny hand-written CharLexer over single-character tokens the engine chapter used, so nothing but core tokora types is in play.

use core::{convert::Infallible, fmt};
use tokora::{
  InputRef, Lexer, SimpleSpan, Token,
  error::{UnexpectedEot, token::UnexpectedToken},
};
use tokora::span::Span as _;
#[derive(Debug, PartialEq)]
struct Error;
impl From<Infallible> for Error { fn from(e: Infallible) -> Self { match e {} } }
impl<'a, T, K: Clone, S, Lang: ?Sized> From<UnexpectedToken<'a, T, K, S, Lang>> for Error { fn from(_: UnexpectedToken<'a, T, K, S, Lang>) -> Self { Error } }
impl<O, Lang: ?Sized, Set: Clone + 'static> From<UnexpectedEot<O, Lang, Set>> for Error { fn from(_: UnexpectedEot<O, Lang, Set>) -> Self { Error } }
impl tokora::error::MaybeIncomplete for Error {}
#[derive(Debug, Clone, PartialEq)]
enum Tok { Digit(u32), Plus }
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum Kind { Digit, Plus }
impl fmt::Display for Kind { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { write!(f, "{self:?}") } }
impl Token<'_> for Tok {
  type Kind = Kind;
  type Error = Infallible;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> Kind { match self { Tok::Digit(_) => Kind::Digit, Tok::Plus => Kind::Plus } }
  fn is_trivia(&self) -> bool { false }
}
struct CharLexer<'a> { src: &'a str, pos: usize, tok: SimpleSpan, state: () }
impl<'a> Lexer<'a> for CharLexer<'a> {
  type State = (); type Source = str; type Token = Tok; type Span = SimpleSpan; type Offset = usize;
  fn new(src: &'a str) -> Self { Self { src, pos: 0, tok: SimpleSpan::new(0, 0), state: () } }
  fn with_state(src: &'a str, _: ()) -> Self { Self::new(src) }
  fn check(&self) -> Result<(), Infallible> { Ok(()) }
  fn state(&self) -> &() { &self.state }
  fn state_mut(&mut self) -> &mut () { &mut self.state }
  fn into_state(self) -> Self::State {}
  fn source(&self) -> &'a str { self.src }
  fn span(&self) -> SimpleSpan { self.tok }
  fn slice(&self) -> &'a str { &self.src[self.tok.start()..self.tok.end()] }
  fn lex(&mut self) -> Option<Result<Tok, Infallible>> {
    let bytes = self.src.as_bytes();
    while self.pos < bytes.len() && bytes[self.pos] == b' ' { self.pos += 1; }
    if self.pos >= bytes.len() { return None; }
    let (start, c) = (self.pos, bytes[self.pos] as char);
    self.pos += 1;
    self.tok = SimpleSpan::new(start, self.pos);
    Some(Ok(match c { '+' => Tok::Plus, _ => Tok::Digit(c as u32 - '0' as u32) }))
  }
  fn read_frontier(&self) -> tokora::ReadFrontier<usize> { tokora::ReadFrontier::SpanEnd }
  fn bump(&mut self, n: &usize) { self.pos += n; }
}
use tokora::{
  Emitter, Parse, ParseContext, Parser,
  cache::DefaultCache, emitter::Verbose, span::Spanned,
};

// Generic over the parse context, so the very same parser can run under any emitter —
// here it will run under `Verbose`, which records diagnostics instead of failing fast.
fn rewind_demo<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CharLexer<'inp>, Ctx>,
) -> Result<Vec<Kind>, Error>
where
  Ctx: ParseContext<'inp, CharLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CharLexer<'inp>, Error = Error>,
{
  let start = *inp.cursor().as_inner();

  // A speculative branch that consumes a token AND files a diagnostic, then declines.
  // Both live on one timeline, so the decline rewinds both: the cursor returns to `start`,
  // and the diagnostic is dropped from the emitter's log.
  let declined = inp.attempt(|inp| {
    let _ = inp.next();                     // consume: the cursor moves forward
    let at = *inp.span();
    // File a real diagnostic. Under `Verbose` this is recorded (and returns `Ok`); the
    // branch is about to decline, so it should not survive.
    let _ = inp.emit_error(Spanned::new(at, Error));
    None::<()>                              // decline → roll the whole branch back
  });
  assert!(declined.is_none());
  assert_eq!(*inp.cursor().as_inner(), start, "position rewound to the begin point");

  // The real parse now sees every token the speculation had consumed — proof the cursor
  // came back. It emits nothing of its own.
  let mut kinds = Vec::new();
  while let Some(tok) = inp.next()? {
    kinds.push(tok.data().kind());
  }
  Ok(kinds)
}

let mut emitter = Verbose::<Error>::new();
let cache = DefaultCache::<'_, CharLexer<'_>>::default();
let kinds = Parser::with_context((&mut emitter, cache))
  .apply(rewind_demo)
  .parse_str("1+2")
  .expect("Verbose files diagnostics rather than failing the parse");

// Position rewound: the real parse saw all three tokens the speculation had consumed. Had
// the cursor NOT come back, it would have seen only the two after the speculated token.
assert_eq!(kinds, vec![Kind::Digit, Kind::Plus, Kind::Digit]);

// Diagnostics rewound: the error the declined branch filed is gone. Had the emission
// timeline NOT rewound with the cursor, this count would be exactly one.
assert_eq!(emitter.errors().values().flatten().count(), 0);

The two assertions are the chapter in miniature: the token count proves the position channel rewound, the error count proves the emission channel rewound, and they rewound because a single checkpoint pinned both to the begin point and a single restore released both from it.

Where to go next

The checkpoint is one of the four seams Part III opens onto the same engine:

• How the input is stored — the Source / Slice seam that lets this same machinery, and the same pure-copy checkpoint, read &str, &[u8], or an owned reference-counted buffer without the grammar changing: the Source, Slice & storage backends chapter.
• How the emitter marks and rewinds its own log — the atomic Emitter capability family, the value-keyed checkpoint / rewind / release trio this chapter leaned on, and how Fatal / Verbose / Silent get their rewind behavior: the Atomic Emitter chapter.
• How committed tokens become a lossless tree — the CstEmitter hook and the cst event stream the cst::Sink buffers and rewinds under this chapter’s mark: the event-stream CST chapter, which is where the value-keyed-inner composition sketched above is developed in full.

The atomic emitter

The engine chapter established that a parser consumes and emits on one handle, one timeline; the checkpoint chapter made the rewind precise — one save pins the input cursor and the emitter’s log together, one restore releases both. This chapter takes the second of those two channels on its own terms: what the emitter is, why its capability surface is a family of small traits rather than one large one, and how it marks and rewinds its own log in step with the cursor. Like its Part III siblings it keeps the register: less here is how to call it, more here is why it is shaped this way. The tutorial side is chapter 7; the flat catalog is the errors, emitters & context reference, which this chapter is the architecture behind.

Diagnostics are an effect, and effects are pluggable

A parser discovers a problem mid-consume: a token that does not fit, a premature end, a lexer error. What happens next — stop the parse, record the problem and press on, drop it, or fold it into a tree — is not a property of the grammar. It is a policy, and it is the same policy across the whole parse. So tokora puts that policy in one replaceable object, the emitter, borrowed by the handle alongside the scanning state, and leaves the parser writing the same line either way:

inp.emit_error(Spanned::new(at, err))?;

The protocol is a Result. Ok(()) means the diagnostic was handled as non-fatal and parsing should continue; Err(Self::Error) means it is fatal and the ? at the call site unwinds the parse. The decisive point is that the emitter returns that verdict, not the call site. Fatal’s emit_error returns Err, so the ? ends the parse; Verbose’s records the diagnostic and returns Ok, so the ? does nothing and the loop continues. The parser cannot tell which it is running under, and does not need to.

Why a borrowed channel rather than a richer return type? Because the fatal/non-fatal choice is uniform over a parse and orthogonal to any single combinator. Encoding it in return types would thread the decision through every signature and force each combinator to re-decide it; a borrowed effect object keeps the grammar policy-free and swaps the entire behavior at the driver, by handing in a different context. This is the whole of the design’s ergonomics: you do not write a “collecting parser” and a “fail-fast parser” — you write a parser and choose an emitter.

The one-timeline law of the previous two chapters is what makes that channel safe to emit from, not merely to consume from. Because the emitter rides the same rewindable timeline as consumption, a declined speculative branch drops its diagnostics with its tokens: a diagnostic filed inside an attempt that then declines never reaches the harvest, exactly as the tokens it consumed never move the committed cursor. That constraint — whatever the emitter records must be able to unwind — is the shape the rest of the trait is bent toward.

The atomic capability design

The core Emitter trait is small. Trimmed of its where L: Lexer<'a> bounds, its shape is:

pub trait Emitter<'a, L, Lang: ?Sized = ()> {
    type Error;

    // The three required diagnostic verbs. `Ok(())` is non-fatal (continue);
    // `Err(Self::Error)` is fatal (the `?` unwinds the parse).
    fn emit_lexer_error(&mut self, err: Spanned<<L::Token as Token<'a>>::Error, L::Span>) -> Result<(), Self::Error>;
    fn emit_unexpected_token(&mut self, err: UnexpectedTokenOf<'a, L, Lang>) -> Result<(), Self::Error>;
    fn emit_error(&mut self, err: Spanned<Self::Error, L::Span>) -> Result<(), Self::Error>;

    // The one non-diagnostic method with no default (see below): the emitter must
    // say how its state unwinds.
    fn rewind(&mut self, cursor: &Cursor<'a, '_, L>, checkpoint: u64);

    // Everything past here has a blanket no-op (or inert-value) default, so a
    // fail-fast emitter inherits empty bodies and the calls inline to nothing.
    fn emit_warning(&mut self, warning: Spanned<Self::Error, L::Span>) -> Result<(), Self::Error> { Ok(()) }
    fn emit_skipped_region(&mut self, span: L::Span, skipped: usize) -> Result<(), Self::Error> { Ok(()) }
    fn checkpoint(&mut self) -> u64 { 0 }
    fn release(&mut self, checkpoint: u64) {}
    fn commit_token(&mut self, tok: &L::Token, span: &L::Span) {}
    fn commit_lexer_error(&mut self, err: Spanned<..>) -> Result<(), Self::Error> { self.emit_lexer_error(err) }
    fn enter_label(&mut self, label: &'static str) {}
    fn exit_label(&mut self) {}
    fn bound_source(&self) -> Option<SourceIdentity> { None }
}

bound_source is the newest of the defaulted members and the one with a forwarding obligation: an emitter that binds a particular source — the CST Sink does — overrides it so a mismatched drive is refused, and a wrapper that forwards every emission but inherits the None default silently disables that check for whatever it wraps. It is the same obligation checkpoint/rewind/release carry.

Three verbs are not enough for a real grammar. A bounded repetition can report too few elements; a separated list can report a stray or a missing separator; a Pratt loop can report an operator with no operand; a tree build needs structure events. Folding all of those into the core trait would make every emitter — including a two-line fail-fast one — answer questions it will never be asked, and make every parser depend on the whole surface whether or not it uses it.

So tokora splits each scenario into its own focused trait, each an extension of Emitter, and lets a combinator name only the capabilities it actually uses. Call it the Lego rule: separated bounds the separator capabilities, repeated the count capabilities, a Pratt parse the PrattEmitter, and a plain expect nothing beyond the base. An emitter, symmetrically, implements only the capabilities its parsers need.

Each capability comes with a matching From…Error blanket. The method payloads are the leaf error types from [crate::error], and the trait that wires a leaf to a method — FromEmitterError for the base surface, and one per capability — is blanket-implemented off a plain From<LeafError> impl on your error enum. So implementing a capability on a pre-built emitter is nothing you do: give your error type the From impls and Fatal / Verbose / Silent gain the capability for free. The genuinely new code in a custom emitter is almost always just the base surface.

Bundling the common family

The separated/repeated machinery ends up needing most of the family at once, which would be a six-line where-clause ladder at every generic parser that drives it. ComposableEmitter is that ladder as one name:

pub trait ComposableEmitter<'inp, L, Lang: ?Sized = ()>:
    Emitter<'inp, L, Lang>
    + FullContainerEmitter<'inp, L, Lang>
    + SeparatedEmitter<'inp, L, Lang>
    + UnexpectedLeadingSeparatorEmitter<'inp, L, Lang>
    + UnexpectedTrailingSeparatorEmitter<'inp, L, Lang>
    + TooFewEmitter<'inp, L, Lang>
    + UnclosedEmitter<'inp, L, Lang>
{}

It is blanket-implemented for every emitter that satisfies the whole family, so a bound of E: ComposableEmitter is interchangeable with spelling out the seven sub-traits — that is the default-policy surface, and PolicyComposableEmitter is the same bundle widened by the three a count or separator policy needs — and its context-side twin, ComposableParseContext, collapses a whole parse context to one bound in the same way — and goes one rung further, riding FromTokenErrors on the emitter’s Error so the leaf From conversions come with it. The five capabilities outside the bundle (TooManyEmitter, the missing-separator pair, PrattEmitter, CstEmitter) are the less-common ones, named on demand by the parsers that use them; the pre-built emitters implement all of them regardless.

The one capability that binds rather than defaults

CstEmitter is the exception worth naming, because it inverts the default’s purpose. Four of its five methods (cst_start / cst_finish, and the retro-wrap pair cst_mark / cst_start_at) have no-op defaults, so Fatal / Verbose / Silent / Ignored are CstEmitter for one written line and a tree-less parse compiles the event calls to nothing. That line is the fifth method, cst_demote — the node bracket’s failing exit, and required for the same reason rewind is (the next section): an inherited no-op there is not an absence someone would notice on the first parse but a presence nothing can detect — a node the grammar retracted, left open in whatever channel sits below, on error paths only. But a tree-producing parse path bounds Ctx::Emitter: CstEmitter anyway — not to reach a method the default already provides, but to make the bound itself load-bearing: a wrapper emitter that forwarded the diagnostic surface and forgot the structure surface would produce a parse whose diagnostics flow perfectly and whose tree is silently empty. On every other capability that is an annoyance; on this one it is a wrong tree nothing downstream can detect, so CST is the one place the design binds instead of trusting a default. The recording implementation is the rowan-gated cst::Sink, the subject of the event-stream CST chapter (its vocabulary lives in [crate::cst]).

The core trait’s one method the design refuses to default

Everything past the three emit verbs is defaulted — with a single, deliberate exception: rewind has no default body. Every emitter must write it, and that is a design decision, not an oversight.

The reason is the one-timeline law. A recording emitter that inherited a no-op rewind would keep the diagnostics of branches the parse abandoned, attributed to code that never committed — a phantom diagnostic. Nothing in the crate can detect that: it surfaces as wrong output, never as a panic (the contracts on emit_warning and rewind spell this out). By refusing to default the method, the design forces every emitter author to answer how does my state unwind? For a stateless emitter the answer is a trivially empty body — Fatal, Silent, and Ignored each write one — but the compiler makes them write it, so the question is never skipped by accident. checkpoint defaults to 0 and release defaults to a no-op precisely because a stateless emitter genuinely has nothing to mark or reclaim; rewind is the one place where silence would be a correctness bug, so silence is not allowed to be the default.

CstEmitter::cst_demote is the same decision taken again on the capability trait, and the test is the same one stated generally: a default is allowed when inheriting it fails toward an absence something downstream reports, and refused when inheriting it fails toward a presence nothing can. rewind is a phantom diagnostic; cst_demote is a phantom node. Every other method on both traits keeps its default because its absence announces itself.

checkpoint / rewind / release: the emitter’s transactional surface

Three methods make the emitter’s log rewindable in step with the cursor, and together they are a value-keyed reading model — no per-save resource, no journal of edits.

• checkpoint(&self) -> u64 returns a reading: a monotone mark that names how much has been emitted so far. It borrows &self and allocates nothing — it is a measurement, not a save. It is also fail-atomic: if it unwinds, nothing may have changed. The &self receiver is the tell — an emitter that registers per-mark state needs interior mutability to do it, and owning that atomicity is the price: reserve before you publish. That is why there is no reserve_checkpoint hook to pair with it; reserve-then-publish is already expressible inside the body, and the input layer orders its own capture so nothing crate-side is pending when the call is made.
• rewind(cursor, mark) restores the emission state to that reading, dropping exactly what was recorded after it. Like release, it can be called from a drop that is already unwinding — a rolling-back guard settles in its Drop — so it must not panic there; a panic mid-unwind aborts. The same rider is on exit_label, whose pop labelled performs from a drop guard. The binding property is never abort, not never panic: an emitter that can detect an unpaired settle may report it by panicking on the normal path, so long as it checks std::thread::panicking first. Two riders come with that: the report must be raised before the rewind mutates anything, so a caught panic leaves the emitter whole rather than half rewound; and when the report is suppressed mid-unwind the emitter must latch the fact and refuse at whatever surface hands its output onward, rather than degrade in silence. The recording CST Sink (the rowan feature) is the one implementation here that has anything to detect, and it takes that door under both riders.
• release(mark) is the eviction dual: it tells the emitter a mark was kept rather than rewound, so any per-mark bookkeeping can be reclaimed.

Verbose is the reference. Its checkpoint is the length of its emission log; its rewind pops every entry recorded past the mark, newest first, each dropping from the channel the log entry names; and its release is a deliberate no-op, because its rollback state lives in the recorded values themselves — a kept mark is just a number going out of scope, with nothing to evict. That the model keys on emission order rather than on a source offset is what lets rewind drop a speculative zero-width diagnostic while keeping an earlier one at the same offset — the distinction a span-offset heuristic could not make, and the exact defect the predecessor design shipped (see the checkpoint chapter’s why-a-copy-not-a-merge discussion).

The advisory status of release is the subtle half. A mark can be abandoned with neither a rewind nor a release — a raw checkpoint merely dropped, say — so an emitter whose correctness depended on release being called would leak. It must not: release is strictly bookkeeping, and releasing may never change the observable emission state. Value-keyed emitters (Verbose, Fatal, Silent, Ignored) inherit the no-op and are correct; only an emitter that keeps a genuine per-mark table — a buffering sink with a checkpoint stack — overrides release to pop the kept row, and even a missed release there is bounded-but-unswept, reclaimed by the next enclosing rewind, never wrong. This is why the reference posture is value-keyed: the property “correctness does not depend on release” falls out of it for free.

How the input drives this trio — one save pinning both the cursor and this mark, and a recording cst::Sink buffering a second (event) log under the very same mark so the tree rewinds exactly as the diagnostics do — is the checkpoint chapter’s subject, developed there in full. This chapter owns only the emitter’s half of the contract.

Two last hooks belong to the same timeline, and they are the pair that decides what a source byte is. commit_token is called by the input exactly once per settled token — consumed, or skipped behind a scan frontier — and nowhere else. A diagnostics emitter inherits its no-op default; the recording sink overrides it to record a token event, which is what makes every consuming combinator tree-producing with zero per-combinator code.

commit_lexer_error is its refusal-side twin: the input layer calls it, from its one deduped reporting site, for a lexer error it raised over bytes it lexed and could not tokenize. Its default body forwards to emit_lexer_error, so an emitter that only collects diagnostics implements nothing and cannot tell the two apart — every lexer error arrives exactly as it always did. Only an emitter that treats the span as structural evidence overrides it, and the recording sink is the one: a recorded refusal span is what licenses a byte to have no token at all, so it must come from the layer that refused those bytes rather than from a caller who picked a range. A wrapper emitter forwards this exactly as it forwards commit_token; inheriting the default hands the wrapped sink a caller-shaped report where a refusal was, and its finish then refuses a region that was legitimately unlexable.

Those two auto-emission hooks, paired with the CstEmitter structuring surface above, are what the event-stream CST chapter is about.

The built-ins as design points

Each shipped emitter is a point in a small design space — what to do with a diagnostic — and reading them side by side is the clearest way to see the trait’s range.

• Fatal is fail-fast, and the default a bare Parser::new() installs. Every emit verb returns Err, so the first diagnostic is the Err the caller already handles; it stores nothing, allocates nothing, and its rewind is empty because there is nothing to unwind.
• Verbose is record-and-continue, and the reference value-keyed emitter. It collects hard errors and soft warnings into parallel span-keyed channels — plus a third channel for recovery holes — all threaded on one emission log, so a rewind unwinds an abandoned branch’s records together and its read-side diagnostics() view can replay every channel interleaved in true emission order. (Whether a diagnostic is an error or a warning is a Severity classification, never a control-flow decision — that stays the emitter’s.)
• Silent discards but keeps your error type E, so it slots into the same signatures as Fatal/Verbose for a best-effort parse where the diagnostics are unwanted. Ignored goes further and collapses the error type to (), for when you want the value and never the diagnostics.
• cst::Sink is the recording CstEmitter the capability section pointed at: it wraps an inner diagnostics emitter, forwards its diagnostics, and also buffers the CST event stream under the one rewind mark. Its internals — the event vocabulary and the finish materialization — are the event-stream CST chapter’s; see [crate::cst].

One assembly, two effect channels

The headline claim compiles: one parser, written once, run under two emitters that reinterpret its single emit_error line — a fail-fast stop under one, a recorded-and-continue under the other. The lexer is the same tiny hand-written CharLexer over single-character tokens the engine and checkpoint chapters used, so nothing but core tokora types is in play.

use core::{convert::Infallible, fmt};
use tokora::{
  InputRef, Lexer, SimpleSpan, Token,
  error::{UnexpectedEot, token::UnexpectedToken},
};
use tokora::span::Span as _;
#[derive(Debug, PartialEq)]
struct Error;
impl From<Infallible> for Error { fn from(e: Infallible) -> Self { match e {} } }
impl<'a, T, K: Clone, S, Lang: ?Sized> From<UnexpectedToken<'a, T, K, S, Lang>> for Error { fn from(_: UnexpectedToken<'a, T, K, S, Lang>) -> Self { Error } }
impl<O, Lang: ?Sized, Set: Clone + 'static> From<UnexpectedEot<O, Lang, Set>> for Error { fn from(_: UnexpectedEot<O, Lang, Set>) -> Self { Error } }
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for Error { fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { Error } }
impl tokora::error::MaybeIncomplete for Error {}
#[derive(Debug, Clone, PartialEq)]
enum Tok { Digit(u32), Plus }
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum Kind { Digit, Plus }
impl fmt::Display for Kind { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { write!(f, "{self:?}") } }
impl Token<'_> for Tok {
  type Kind = Kind;
  type Error = Infallible;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> Kind { match self { Tok::Digit(_) => Kind::Digit, Tok::Plus => Kind::Plus } }
  fn is_trivia(&self) -> bool { false }
}
struct CharLexer<'a> { src: &'a str, pos: usize, tok: SimpleSpan, state: () }
impl<'a> Lexer<'a> for CharLexer<'a> {
  type State = (); type Source = str; type Token = Tok; type Span = SimpleSpan; type Offset = usize;
  fn new(src: &'a str) -> Self { Self { src, pos: 0, tok: SimpleSpan::new(0, 0), state: () } }
  fn with_state(src: &'a str, _: ()) -> Self { Self::new(src) }
  fn check(&self) -> Result<(), Infallible> { Ok(()) }
  fn state(&self) -> &() { &self.state }
  fn state_mut(&mut self) -> &mut () { &mut self.state }
  fn into_state(self) -> Self::State {}
  fn source(&self) -> &'a str { self.src }
  fn span(&self) -> SimpleSpan { self.tok }
  fn slice(&self) -> &'a str { &self.src[self.tok.start()..self.tok.end()] }
  fn lex(&mut self) -> Option<Result<Tok, Infallible>> {
    let bytes = self.src.as_bytes();
    while self.pos < bytes.len() && bytes[self.pos] == b' ' { self.pos += 1; }
    if self.pos >= bytes.len() { return None; }
    let (start, c) = (self.pos, bytes[self.pos] as char);
    self.pos += 1;
    self.tok = SimpleSpan::new(start, self.pos);
    Some(Ok(match c { '+' => Tok::Plus, _ => Tok::Digit(c as u32 - '0' as u32) }))
  }
  fn read_frontier(&self) -> tokora::ReadFrontier<usize> { tokora::ReadFrontier::SpanEnd }
  fn bump(&mut self, n: &usize) { self.pos += n; }
}
use tokora::{
  Emitter, Parse, ParseContext, Parser,
  cache::DefaultCache,
  emitter::{Severity, Silent, Verbose},
  span::Spanned,
};

// One parser, generic over the parse context, so the SAME assembly runs under any emitter. Its
// only diagnostic decision is not made here: `emit_error` returns `Ok` under a collecting emitter
// and `Err` under a fail-fast one, and the `?` obeys whichever it is handed.
fn digits<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CharLexer<'inp>, Ctx>,
) -> Result<Vec<u32>, Error>
where
  Ctx: ParseContext<'inp, CharLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CharLexer<'inp>, Error = Error>,
{
  let mut out = Vec::new();
  while let Some(tok) = inp.next()? {
    let at = *inp.span();
    match tok.into_data() {
      Tok::Digit(n) => out.push(n),
      // A `+` where a digit belongs. THE line the emitter reinterprets: a fatal stop under
      // `Fatal`, filed-and-continue under `Verbose`, dropped under `Silent`.
      Tok::Plus => inp.emit_error(Spanned::new(at, Error))?,
    }
  }
  Ok(out)
}

// ── Fatal (what `Parser::new()` installs): the first emitted error is the `Err` the caller
//    already handles. Nothing is stored, nothing is allocated. ──
assert_eq!(Parser::new().apply(digits).parse_str("1+2"), Err(Error));

// ── Verbose: the very same `digits`, run to the end of the input. `emit_error` now records
//    rather than returning, so the `?` is a no-op and the parse completes. ──
let mut emitter = Verbose::<Error>::new();
let cache = DefaultCache::<'_, CharLexer<'_>>::default();
let out = Parser::with_context((&mut emitter, cache))
  .apply(digits)
  .parse_str("1+2")
  .expect("Verbose files the diagnostic rather than failing the parse");

// Both digits came through; the `+` was filed, not fatal.
assert_eq!(out, vec![1, 2]);
// The diagnostic is read back off the emitter — the effect channel — after the parse.
assert_eq!(emitter.errors().values().flatten().count(), 1);
// The read-side view classifies it: one Error-tier diagnostic, in emission order.
let tiers: Vec<Severity> = emitter.diagnostics().map(|d| d.severity()).collect();
assert_eq!(tiers, [Severity::Error]);

// ── Silent: the discard channel. Same recovered value as Verbose, but nothing is kept. ──
let cache = DefaultCache::<'_, CharLexer<'_>>::default();
let out = Parser::with_context((Silent::<Error>::new(), cache))
  .apply(digits)
  .parse_str("1+2")
  .expect("Silent drops the diagnostic and never fails");
assert_eq!(out, vec![1, 2]);

Three runs, one digits. The grammar never branched on the emitter; the emitter branched on the grammar’s behalf. That is the atomic emitter design in one line of code seen three ways.

Where to go next

The emitter is one of the four seams Part III opens onto the same engine:

• How the input drives this log in step with the cursor — one save pinning both channels, the LIFO contract, and how a cst::Sink buffers a second event log under this chapter’s mark: the Checkpoint, rewind & the LIFO contract chapter, which develops the value-keyed-inner composition in full.
• How committed tokens become a lossless tree — the CstEmitter structuring surface and the commit_token auto-emission hook this chapter named, and the cst event stream the cst::Sink buffers and rewinds: the event-stream CST chapter.
• How the input is stored — the Source / Slice seam this same machinery reads through, byte- or text-shaped, owned or borrowed: the Source, Slice & storage backends chapter.
• The flat catalog — every emitter, capability sub-trait, and read-side type as terse reference entries, plus the error taxonomy and the ParseContext / ComposableParseContext bundle: the errors, emitters & context reference.

The event-stream CST engine

The checkpoint chapter closed on a composition it sketched: a recording sink that buffers a second log — the CST event stream — under the very same mark the input already uses to rewind position and diagnostics, so a declined branch drops its tree exactly as it drops its tokens. The atomic-emitter chapter one-lined the sink itself: cst::Sink, the recording CstEmitter that wraps an inner diagnostics emitter, forwards its diagnostics, and also records structure. This chapter owns the sink’s internals — the deepest seam in Part III. It keeps the part’s register: less how to call it (that is chapter 16, the tutorial, which deliberately treats the event stream as an implementation detail), more why every mechanism is shaped the way it is.

The details it documents are load-bearing: an era that rewinds by one, a witness dropped from a mark, a gap tiled without a covering diagnostic — each is a wrong tree with no witness, the failure class the whole design is bent to make unrepresentable. So this chapter is precise about the invariants, not just the shapes.

Events, not eager nodes

The obvious way to build a tree while parsing is to call a builder as you go: open a node here, push a token there, close the node on the way out. It is also the way that does not survive backtracking. A tree builder holds interior state — a stack of half-open nodes — that a speculative branch mutates as it parses; when the branch declines, that state has to be surgically undone, node by node, or the tree keeps a phantom of work the parse rolled back. An eager builder like rowan’s GreenNodeBuilder — which tokora exposes thinly as SyntaxTreeBuilder — has no rollback of its own, so tokora drives no builder at all during the parse; it drives one exactly once, at materialization, from the finished log.

Instead — following the lineage of rust-analyzer and Biome — tokora records the parse as a flat log of events and derives the tree from the surviving events exactly once, at the end. An event is a tiny value: open a node of this kind, here is a committed token, close a node. Nothing is built while parsing; the log is just appended to. The payoff is the whole reason the design exists:

• Backtracking rewinds the tree for free. The events live in the emitter’s rewindable channel, so the one mark that truncates diagnostics on a declined branch truncates its tree events in the same motion. There is no tree surgery on decline, because there is no tree yet — only a log, and a log rewinds by throwing away its suffix. A rolled-back branch’s structure vanishes exactly as its cursor movement does.
• Materialization is a single validated pass. Building once, from a complete log, means the builder is only ever driven with already-checked operations — so rowan can never panic under it, and every losslessness and balance law is enforced in one place (see materialization below).

The event stream is therefore not an optimization detail; it is the data structure that makes “the tree participates in the one rollback contract” true rather than aspirational.

The event vocabulary

The vocabulary lives in cst::event, and it is rowan-free — it compiles in every build, because the recording half of the CST design (the marks, the node combinators) is unconditional; only the materializing half is gated on rowan. The log itself is a Vec of one crate-internal Event enum, and the governing law is stated up front: the parse-time event buffer changes in exactly two ways — append (the cst_* emission methods) and suffix-truncate (a rewind). No emission ever rewrites the kind of an interior slot: both directions of that rewrite — completing a tombstone into a real node, and un-opening a real node back into a tombstone — are encoded as appended events naming the earlier slot (StartAt and Demote below), and the kind writes they stand for land at materialization, on a buffer the consumed sink owns and no mark can still name. That two-verb discipline is what makes rewind-by-truncation exact: the prefix below any live mark is immutable, so truncating to a mark restores the buffer to precisely the state it had when the mark was captured. (There is one journaled exception — an acceleration field — developed with the sink below; it is legal only because it is reversed on rewind, so the law holds observationally.)

The six events

• StartNode { kind, forward_parent } opens a node of kind, closed by a matching FinishNode — unless kind is the reserved TOMBSTONE value (u16::MAX), in which case the slot is an inert mark that pairs with no finish and materializes into nothing. The forward_parent field is the journaled acceleration; ignore it until the sink.
• Token { kind, span } is one committed token: its dialect-mapped kind and its source span, appended exactly once per settled token. Peeks, declines, and unconsumed stoppers append nothing — only a settle records a token (the commit_token hook the emitter chapter named).
• FinishNode { kind } closes the innermost open node — plain stack discipline. The kind is the one the emitter intended to close, and it is the leaked-finish detector: materialization compares it against the frame the finish actually lands on and refuses a mismatch (MismatchedFinish), which is the one signal that separates a legal cross-checkpoint close from a finish whose start was rolled back out from under it — two histories that otherwise leave byte-identical buffers.
• StartAt { kind, target, prev } retro-opens a node of kind at the buffer position of the tombstone named by target. This is the append-only form of retro-parenting: rather than rewrite a tombstone into a real start in place, you append a StartAt that names it. Same-target StartAts open in reverse buffer order at materialization — the later wrap becomes the outer node, because its finish is necessarily appended later — and prev is the chain link that makes recovering them cheap: it holds the forward_parent value this wrap displaced on its target, so materialization walks the target’s wrap list newest-first instead of rebuilding a keyed index. The in-place alternative (rewriting the tombstone’s kind) is banned by law: an interior write below a live emitter mark would survive the truncation that was supposed to erase the branch that made it.
• Demote { target } un-opens the node the StartNode at target opened — the failing exit of an up-front bracket, and the exact mirror of StartAt: an appended event naming an earlier slot, never an in-place rewrite of it. It is what a node) bracket emits when its sub-parser unwinds with an error (see the combinator surface), so a stream carries one per failing bracket and a parse with no failing bracket carries none. Appending rather than rewriting is what gives the bracket’s two exits one rollback law: a rollback into the window between the start and the demote truncates the demote and the node is open again, exactly as it would truncate a FinishNode. A rewrite would not truncate — the rollback would keep the slot and the rewrite — so the failing exit would silently drop a node the restore contract had just promised was open again. Materialization canonicalizes the demotion: one pass over the owned buffer writes TOMBSTONE onto every surviving demote’s target before the walk, after which the abandoned node is indistinguishable from the inert slot an unspent cst_mark leaves, and the walk is blind to both. A Demote can never outlive its target — target is strictly below the demote’s own index, and truncation is a suffix operation — so the pass has nothing to reconcile.
• Diag { error_span } is a forwarded-diagnostic slot — a marker in the event log for one diagnostic that was forwarded to the wrapped emitter, on Ok and Err alike. It is skipped at materialization, with one exception that is the design’s single deliberate channel coupling: a lexer-error slot carries the offending source span in error_span, so finish can tell a byte a lexer legitimately refused from a byte a dropped token lost. Living in the event log means the span rewinds with the branch that saw it — an abandoned lexer error stops covering anything, for free. Every other diagnostic (unexpected token, missing element, …) points at tokens that settled or at zero-width absences, covers no gap, and stores None.

Balance is derived from the log, never cached beside it: a real StartNode or a StartAt is +1, a FinishNode or a Demote is −1, and a tombstone, a Token, and a Diag are all 0 — so the pair StartNode(+1) … Demote(−1) nets exactly what a canonicalized StartNode(TOMBSTONE) does, which is why an up-front bracket’s two exits cost balance the same thing whichever it takes. A malformed buffer is representable — the raw cst_* surface is sharp on purpose — but it is unrepresentable as a successful materialization: finish walks the log and returns a typed error rather than building a wrong tree.

Marks carry an era and a witness

A retro-wrap needs a handle to the tombstone it will anchor at, and an up-front bracket needs one to the start it may have to demote. Both are the same handle — an EventMark, one positional surface with one staleness rule rather than two — and the subtle part is that index-in-bounds is not validity. Truncate-and-regrow is the normal backtracking rhythm, so “the buffer has an event at index 3” says nothing about whether that event is still the slot a mark was minted for — a rewind can truncate it away and unrelated events can regrow over index 3. So a mark is a positional witness plus identity, three fields:

• index — the named slot’s position in the buffer;
• era — the truncation history the mark was issued under;
• sink — the identity of the one recording sink that minted it.

Two (index, era) pairs coincide trivially — two fresh sinks both mint (0, 0) — so identity is not optional decoration. A recording sink validates all three at every spend and panics in every build on a stale or foreign mark. This is the savepoint posture, and it is deliberate: both are parser bugs (the branch that conceived the wrap was rolled back, or the mark belongs to another parse entirely), not input-dependent conditions, and the silent alternative — wrapping whatever sits at that index — is a wrong tree nothing downstream can detect. An emitter with no event channel returns an inert mark (index u64::MAX, the reserved witness id 0), which fails a recording sink’s identity wall deterministically rather than wrapping anything.

Why brand marks at all, rather than trust the parser to only spend live ones? Because stale- and foreign-mark misuse is precisely the bug class earlier iterations shipped, and the type system cannot catch it: a mark is Copy and may legitimately outlive its combinator frame (a pratt driver holds one across arbitrarily many operator iterations, spending it once per fold). Branding moves the detection from “hope” to “panics at the spend, at the cause.”

The rewindable sink

cst::Sink is where the vocabulary meets the emitter contract. It wraps an inner emitter E, forwards the entire emitter trait family to it (so any context bound E satisfies, Sink<E> satisfies too), and buffers the event stream — one rewindable timeline for tree and diagnostics alike.

A sink is minted by the parse, never constructed beside it: parse_lossless (and its partial sibling) takes the source once and uses that one argument for both the sink it builds and the input it drives, so the buffer the tree’s text is sliced out of and the buffer the parse reads cannot be two different buffers. Its where clause pins the context’s emitter to Sink by name, which is what leaves a forwarding wrapper no slot to occupy at the entry. What comes back is a Cst — the spent sink, holding the one door to the tree, and deliberately not an emitter, so the artefact cannot be re-aimed at a second parse. Its cells are classified by the same CELL_CENSUS discipline the checkpoint chapter described for the input layer: a crate-internal function destructures the sink exhaustively — no .. — so a new field cannot be added without declaring which class it is in and what a rewind must do to it.

One timeline: events beside diagnostics

The sink’s checkpoint is simply the event-log length: one positional mark over one unified log, exactly Verbose’s architecture. Every diagnostic forwarded to the inner emitter occupies a Diag slot inside the event buffer, appended by one census-marked helper on Ok and Err alike (record-then-propagate: a fatal unwind that skipped the slot on the Err edge would drop an error_span a later finish needs). So the whole of rewind is: truncate the buffer to the mark, reverse-replay the undo journal, and rewind the inner emitter to the reading its mark-stack row captured. One mark governs both channels because both channels are the one log — the tree events and the diagnostic order-slots interleaved on a single timeline.

There is deliberately no &mut accessor to the inner emitter, only a shared inner_ref. A caller who could drive the inner emitter’s rewind directly would shear the event log from the diagnostic log with no witness — the exact desync the one-timeline law forbids. Ownership of the inner comes back only from materialization, which consumes the Cst handle.

The mark stack, the journal, and the era ledger

Three cells carry the rewind machinery, and each has a distinct restore rule — the reason the census matters.

The mark stack (rows) holds one MarkRow per live checkpoint capture. A row is three frozen facts: the captured mark (the event-log length), the derived open-node depth at capture time, and the inner emitter’s own checkpoint reading, captured by value. Depth is a frozen fact about a prefix, never a live counter — there is no depth counter anywhere in the sink; every query recounts the events above the nearest frozen row (or the released floor, a memo of the newest settled row that keeps recounts short across commit-heavy loops). A cached counter would need its own restore rule; a derived one is restored by truncation for free. Each row is spent by exactly one of release (the branch was kept) or rewind (it was abandoned) — the settle discipline the input layer’s release census locks.

The undo journal exists for the one law-breaking write the design permits. When cst_start_at appends a StartAt, it also writes back onto its target tombstone a forward_parent: the relative offset to the newest StartAt naming that tombstone. This is an in-place mutation of an interior slot — otherwise banned outright — and it is legal only because every write is journaled. The journal records (at_len, index, old_forward_parent), and a rewind reverse-replays the entries whose StartAt died, restoring each overwritten value newest-first. The pointer is never required for correctness — materialization recovers every wrap from the StartAt events themselves — but it is both an acceleration and an integrity canary: finish checks that a set forward_parent still names a live StartAt of its target, and the dangling pointer of an abandoned branch (a DanglingForwardParent) is exactly the silent corruption the journal exists to kill. In-place mutation plus a reverse-replay journal is the pure-copy discipline of the checkpoint chapter, lifted to events.

The era ledger (TruncationLedger) is the cell that makes stale marks detectable, and it is the one most worth being exact about. It is two parts — a monotone era source and a merged truncation stack — and both are monotone, never rewound. This inverts the usual rewind instinct, and the inversion is the point:

• The era source is bumped by +1 on every recorded truncation and never rolled back. Rewinding it would reissue an era a dead mark was minted under, and let that dead mark validate.
• The truncation stack is a witness of truncations: a rewind appends to it (a rewind is a truncation) and never removes from it. Forgetting a truncation would false-accept exactly the stale mark the record existed to kill.

A mark is stale iff some truncation younger than the mark’s era reached the mark’s index or below. The staleness query is one binary search: the stack is kept strictly increasing in both era and low-water mark (a new truncation subsumes every recorded truncation at an equal-or-higher low-water mark — anything an older, shallower entry would invalidate, the newer, deeper one also invalidates — so subsumed entries are merged away on push). The entries younger than a mark’s era are therefore a suffix, the smallest low-water mark among them is that suffix’s first entry, and a single lookup decides. A truncation strictly above a mark’s index leaves it live; one that reaches its index kills it forever; and a mark issued after a truncation is untouched by it. Regrow-then-truncate- shallow keeps both records, because they invalidate different ranges. This is the mechanism behind the flat claim “truncation makes old marks stale forever.”

The sink’s identity — the witness stamped into every mark — comes from a process-unique, 1-based atomic counter (0 is reserved for the inert mark). It is minted unconditionally in every build, because the witness is the every-build half of mark validation, and it is never reissued: the allocator is a fetch_update that aborts on overflow rather than wrapping usize::MAX back to 0 — a wrap would be doubly wrong (0 is the inert id, and every id after it reissues a live one). Sinks move and a dead sink’s address can be reused, so an address would not do; a monotone counter is never reused for the process’s life.

The value-keyed inner, developed

Here is the composition the checkpoint chapter sketched and deferred, developed in full. The sink composes with its wrapped emitter through checkpoint readings, never mark resources: checkpoint captures inner.checkpoint() onto the mark-stack row as a plain u64, rewind hands a captured reading back to inner.rewind, and release pops the sink’s own row without forwarding — the inner is never told about kept branches, because a kept reading is just a number going out of scope. This requires the inner to be value-keyed: a pure monotone checkpoint, a drop-by-value rewind, a no-op release — the shape of Verbose, Fatal, Silent, and Ignored, and of every Verbose-shaped collector. (A table-keyed inner that allocated per-checkpoint bookkeeping is explicitly unsupported here; it belongs at the input layer’s direct seam, where the settle discipline is 1:1.)

The rule that keeps the two logs pinned together is that the sink rewinds the inner only to a reading it knows exactly — it never fabricates one. On a rewind to mark:

• The sink spends the mark-stack captures at or above mark: everything strictly above dies with the branch, and the newest capture at exactly mark is the row being rewound to — its stored inner reading is the exact target. Every disciplined path (a guard, an attempt, the scan family, a correct raw save/restore) lands here.
• If nothing was truncated — mark equals the current length — the inner is left untouched. The surviving events are the whole log, so every inner-side record they reference must survive too; this is the trait’s rewind-to-current no-op law, upheld on every channel.
• For a no-row unwind to the origin (mark == 0, an empty event log), the target is the inner’s construction-time reading. That reading is primed lazily at the first inner-advancing touch (a forwarded diagnostic or a settled token — the sink’s only two advancing surfaces), and it provably equals the reading at construction: the sink exposes no &mut path to the inner, so the inner cannot advance before the sink’s own first advancing call, and every advancing surface primes the base before forwarding. An empty event log therefore pairs with exactly the construction reading.

Everything else is refused rather than guessed. An out-of-range future mark — one strictly above the current length, naming a log position that does not exist yet — is a total no-op on every channel: events, the mark stack, the floor, the journal, the era ledger, and the inner alike. (Clamping it to the current length instead — the pre-redesign behavior — would let a future mark spend the live row of a real checkpoint taken at that length, and that checkpoint’s own later rewind would then find no row: the desync.) And a truncating rewind to a mid-log mark that no live row captured has no exact inner reading anywhere — the mark was never returned by checkpoint, or its capture was already spent. That is an unpaired settle: a parser bug, never something a document can provoke, so it panics in every build. The wall used to be debug_assertions-only, deferring to the input layer’s LIFO witness one level up; that witness is itself debug-only, so a release build had no wall at all and the event log silently sheared away from the diagnostic log — bounded only because materialization reads the whole log at once, and a wrong tree the moment any of it is flushed incrementally. The condition is decided by a preflight over the unchanged mark stack, ahead of the row spend, the truncation, the journal replay and the ledger write, so a host that catches the panic holds the sink it had rather than the sheared one the wall exists to prevent. The one exception is a panic already unwinding, where raising a second one aborts the process rather than reporting anything: there the rewind degrades to a total no-op — both logs left describing the same history, rather than one of them rewound — and latches, so finish and finish_partial refuse afterwards with UnpairedSettle. Silence there is permissible only because it is recorded. The sink hands the inner only readings it can prove, or nothing.

One more append-shaped write deserves a note, because it is the single censused exception to “append + suffix-truncate”: the recovery-hole wrap. When chapter 8’s recovery skips a garbage region, the sink brackets the hole’s already-buffered token events in a StartNode(error_kind) … FinishNode pair. Those tokens are the buffer’s suffix by construction — they settled during the scan, after every live mark was captured, and the scanner runs no user code — so the wrap is a prefix-preserving splice entirely above every live mark: one insert at the first hole token, one appended finish, with the journal’s positions bumped to stay exact. It never disturbs a slot any live mark can name, which is why it is a lawful member of the append family rather than a violation of it.

Materialization: one walk that builds and validates

finish consumes the sink and turns the surviving events into a rowan green tree in a single forward walk — driving the builder only with operations it has already checked, so rowan can never panic under it, and returning a typed FinishError on the first violation instead. It never panics. The one walk enforces, together: balance (an orphan finish or a leftover open is a typed error — rowan’s silent one-level absorb under a root wrapper is unreachable, because the walk refuses first); retro-wrap integrity (a StartAt whose target is not a live tombstone is a StaleStartAt; a dangling forward_parent is the journal’s finish-time canary); kind hygiene (the reserved tombstone band); span discipline (monotone, non-overlapping, in-bounds, u32-fitting); and the two losslessness laws below. One refusal precedes the walk instead of arising from it: a sink that had to degrade a rewind it could not perform refuses here (UnpairedSettle) rather than materialize a log that describes a rollback that never happened.

Two pre-passes run ahead of that walk, and both are legal for one reason: finish has consumed the sink, so no EventMark and no checkpoint row can still name a slot, and nothing can observe the buffer between them and the walk. This is the one moment an interior kind write is unconditionally safe.

Canonicalization is the first, and it is what makes the failing exit’s appended event equivalent to the eager rewrite it replaced: one linear pass applies every surviving Demote to its target (events[target].kind = TOMBSTONE), after which the walk simply skips the Demote and sees an abandoned node as the inert slot it is. The pass is latched off when the parse took no failing bracket exit — which is every parse of a predictive grammar — so a grammar that never demotes never pays for it. A demote whose target is not a live, un-demoted StartNode sitting strictly below the demote itself is a StaleDemote: the release-build half of the double-demote calibration (the debug half panics at the emit site), and the backstop for the positional shape a raw injection could otherwise use to erase a node with a perfectly balanced buffer left behind.

Same-target wraps are resolved in the second pre-pass, which groups the StartAts by target and validates every forward_parent canary; the main walk then opens each target’s wraps latest-first at the tombstone’s position (so the last-declared wrap is the outermost node), and a hoisted wrap that would close before its own declaration is an ImproperWrap — a wrap crossing a node boundary instead of enclosing whole subtrees.

Gap-tiling and the coverage law

Losslessness — tree.text() == source, byte for byte — is structural, not a property the lexer is trusted to provide. As the walk lays down committed tokens in source order, any run of source bytes no committed token covers is tiled with a gap_kind token in the currently open node. A skipped- whitespace region, an undrained tail, a poisoned truncation: whatever the events left uncovered becomes a gap tile, so the round-trip holds for every input, lexer errors included.

“The currently open node” is more precise than it looks, and the precision is the whole of the placement rule: a gap is tiled where it opens, not where it is noticed. An uncovered run opens the instant the token before it settles — that is the moment the parse stopped covering the source — so it is emitted immediately after that token, in the node open then, and it is in the tree before the next event is read. The trailing run is therefore not a case: the bytes after the last committed token are tiled by that token like any other run, so Root[Document[Tok] Gap] is Root[Document[Tok Gap]] and the node widens over the tail. The one clause left over is a run that no token precedes — a source that begins with bytes no token claims — which has no such moment and so tiles where the walk first sees it: at the first committed token, or, with no committed token anywhere, at the end of the walk, in whatever node is open there. A wholly unlexable source consequently keeps its run beside the document node rather than inside it, and finish_partial tiles before it closes the frames an unbalanced stream left open, so there the fallback lands in the innermost open node.

Two invariances follow, and they are why the rule is stated at the opening moment rather than at any later one. First, nothing that follows a run can move it: two streams sharing a prefix through the token a run trails place that run identically, including when one of them simply stops there — so placement never depends on whether more input happened to follow. Second, no diagnostic can move it either, because placement reads the token and structure events only. That second one is load-bearing rather than tidy: a prefilled lookahead cache emits the lexer errors it crosses when it crosses them, so prefetching hoists a lexer-class diagnostic earlier in the event stream. The token stream is exactly invariant under prefill and the diagnostic stream is not, so a rule that read a diagnostic’s position would make the materialized tree a function of how far the caller happened to peek. Coverage — the next paragraph’s subject — still consults every diagnostic, but through the merged set of recorded spans, which is order-independent for the same reason.

But tiling is not unconditional, and the condition is the design’s one deliberate coupling of the diagnostic and CST channels. Elsewhere the two are independent — a Diag slot is invisible to the tree. At finish, though, a tiled byte must be explained: finish tiles a gap only where a recorded lexer-error diagnostic covers it (the lexer saw bytes it could not tokenize, said so, and committed no token there). A gap with no covering error and no covering token is a dropped committed token — the partial-forwarding-wrapper signature — and is refused as an UncoveredGap rather than dressed up as a plausible-but-lossy tree. This is why the lexer-error span rides in the event log (the error_span of a Diag slot): so it rewinds with its branch, and so its span is available to license its gap at exactly the one moment the channels are allowed to cross.

“Recorded” is narrower than “reported”, and the narrowing is the whole guarantee. A licence has to be earned by the same thing a token span is earned by — the input layer lexing those bytes and refusing them — so the layer’s own reports reach the sink through commit_lexer_error, the refusal-side twin of commit_token, and only those carry an error_span. Every caller-facing spelling of the same report — a parser’s InputRef::emit_lexer_error, a callback’s EmitterView::emit_lexer_error, a wrapper forwarding either — lands on Emitter::emit_lexer_error, takes its Diag slot, reaches the inner emitter, and records no span. Without that split the licence was a caller-chosen span with nothing consumed for it — the shape CstEmitter::cst_token had on the token channel before it was deleted — and it reached across buffers: through the orphan-rule wrapper the parse_lossless docs describe, a foreign parse’s refusals excused bytes of this sink’s source. The capability is untouched, only its structural side effect: a callback can still report a malformed input inline, with no rewind, which is what the decide family exists for.

Two more walls guard the same seam. A balanced stream that builds structure but carries no committed token at all over a nonempty source no lexer error explains is a StructureWithoutTokens: the signature of a wrapper emitter that forwarded the CstEmitter structuring surface but inherited the no-op commit_token, so every token silently vanished. It is refused ahead of the gap-coverage law so the all-dropped case earns that precise message rather than an uncovered-gap report over the whole source. The qualifier is what keeps the wall off a legitimate shape it would otherwise share a symptom with: the wall reads the same evidence the coverage law does, and a lossless grammar opens its root node before it can know whether any token follows — so a buffer holding nothing lexable (an unterminated string mid-edit) reaches finish as one error span, one open-then-closed node, and zero tokens. Every byte is explained there, so that tree tiles rather than being refused. And finish_partial is the tooling door for an incomplete parse: it closes open nodes rather than reporting UnclosedNodes, and tiles every gap rather than refusing an uncovered one — the two ways an incomplete parse legitimately differs from a complete one (a fatal abort leaves nodes open; a fail-fast lexer error leaves an un-diagnosed tail). Every other law is enforced identically; the exemptions are exactly the incompleteness signals, nothing more.

The compile-time trivia wall

The coverage law only closes if every source byte reaches the sink as a token or a reported lexer error. A lexer that silently skips whitespace breaks that premise: a skipped-whitespace gap would be indistinguishable from a dropped committed token, and the sink could not tell the lossless case from the corrupt one. So parse_lossless refuses a skipping lexer at compile time: an inline-const assertion on L::SURFACES_TRIVIA fires a post-monomorphization error at the offending call site (at build/test/doc time — not under cargo check, which never monomorphizes the call). A lossless sink structurally requires a trivia-surfacing lexer; the wall makes that a type-level fact rather than a runtime hope. parse_lossless itself demonstrates both sides of the wall, as a compiling and a compile_fail doctest.

One materialization-time policy remains configurable: TriviaPolicy. Its only variant today is the provable one, AsEmitted — a committed trivia token materializes into whichever node was open when it settled (call-site placement), which is deterministic, cache- transparent, and origin-blind. This is deliberately not the Roslyn/Swift “leading trivia attaches forward” policy; a token-attached view is a later materialization-time extension, which is the only reason the enum exists at all.

The combinator surface

You rarely emit events by hand. The node) family is the blessed bracketing over the sink, and its encoding is worth one architectural note because it explains why backtracking stays clean: every combinator in the family is a both-exits bracket whose exits are all appends. They differ only in when the node’s kind is named, and that split is measured rather than stylistic.

Up front. node) driven as a plain parser cannot decline, so its kind is known at entry: it appends a StartNode immediately (cst_start, which hands back the mark naming that slot) and closes on both exits — cst_finish on success, cst_demote on an error-path unwind. Two events per successful node instead of three, no tombstone whose only purpose is to be named later, and an open node that exists only inside the frame.

Retro. Every other shape — node over a declining parser, node_opt), and node_at) — mints an inert tombstone at entry and spends it as a retro-wrap (cst_start_at + cst_finish) only on a successful exit, so no node is ever open between entry and exit. That is not a leftover: a declining parser’s kind is not knowable at entry, and node_at’s whole purpose is a kind decided after its first child was parsed.

Neither shape leaves a dangling StartNode for a later finish to mispair with, and neither leaves anything for a rollback to surgically undo. On a non-success return the two materialize identically — the retro bracket by never spending its mark, the up-front bracket by appending a Demote that canonicalization applies to its own start — so the buffers differ and the trees do not. The labelled finish-on-both-exits discipline, made structural rather than dutiful.

node_at) spends a caller-held mark — the retro-wrap shape: mark, parse a prefix, then decide it was the start of something bigger. For the common single-wrap decision, Marker wraps a raw mark in a compile-time single-use typestate: complete spends it into a node and yields a CompletedMarker, abandon consumes it leaving the tombstone inert, and precede — a further outer wrap — exists only on CompletedMarker, so wrapping an abandoned or still-open intent is unrepresentable rather than merely checked. The raw EventMark stays Copy and multi-spend for the pratt shape that needs it. The combinator reference catalogs the full surface; the point here is only that every one of these lowers to the append-only vocabulary above.

The rewind, in one parse

The headline claim compiles: a speculative branch builds a whole node — tokens and structure, all recorded — then declines, and its events truncate as if they never happened; the real parse then builds the tree for keeps, and finish materializes a lossless tree. The proof is an equivalence: the same source, parsed straight and parsed through the declined speculation, materializes byte-identical green trees. That equivalence is a tested law of the sink, not an accident of the example. The lexer is a tiny hand-written lossless CharLexer — every byte surfaces as a token, so SURFACES_TRIVIA is honestly true — and nothing but core tokora types is in play.

use core::{convert::Infallible, fmt};
use tokora::{
  InputRef, Lexer, SimpleSpan, Token as TokenT,
  error::{UnexpectedEot, token::UnexpectedToken},
};
use tokora::span::Span as _;
#[derive(Debug, PartialEq)]
struct Error;
impl From<Infallible> for Error { fn from(e: Infallible) -> Self { match e {} } }
impl<'a, T, K: Clone, S, Lang: ?Sized> From<UnexpectedToken<'a, T, K, S, Lang>> for Error { fn from(_: UnexpectedToken<'a, T, K, S, Lang>) -> Self { Error } }
impl<O, Lang: ?Sized, Set: Clone + 'static> From<UnexpectedEot<O, Lang, Set>> for Error { fn from(_: UnexpectedEot<O, Lang, Set>) -> Self { Error } }
impl tokora::error::MaybeIncomplete for Error {}
#[derive(Debug, Clone, Copy, PartialEq)]
enum Tok { Num, Plus }
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum Kind { Num, Plus }
impl fmt::Display for Kind { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { write!(f, "{self:?}") } }
impl TokenT<'_> for Tok {
  type Kind = Kind;
  type Error = Infallible;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  // The lexer surfaces every byte — the compile-time wall on `Sink::new` requires it.
  const SURFACES_TRIVIA: bool = true;
  fn kind(&self) -> Kind { match self { Tok::Num => Kind::Num, Tok::Plus => Kind::Plus } }
  fn is_trivia(&self) -> bool { false }
}
struct CharLexer<'a> { src: &'a str, pos: usize, tok: SimpleSpan, state: () }
impl<'a> Lexer<'a> for CharLexer<'a> {
  type State = (); type Source = str; type Token = Tok; type Span = SimpleSpan; type Offset = usize;
  fn new(src: &'a str) -> Self { Self { src, pos: 0, tok: SimpleSpan::new(0, 0), state: () } }
  fn with_state(src: &'a str, _: ()) -> Self { Self::new(src) }
  fn check(&self) -> Result<(), Infallible> { Ok(()) }
  fn state(&self) -> &() { &self.state }
  fn state_mut(&mut self) -> &mut () { &mut self.state }
  fn into_state(self) -> Self::State {}
  fn source(&self) -> &'a str { self.src }
  fn span(&self) -> SimpleSpan { self.tok }
  fn slice(&self) -> &'a str { &self.src[self.tok.start()..self.tok.end()] }
  fn lex(&mut self) -> Option<Result<Tok, Infallible>> {
    let bytes = self.src.as_bytes();
    // No skipping: every byte becomes a token, so the round trip is structural.
    if self.pos >= bytes.len() { return None; }
    let (start, c) = (self.pos, bytes[self.pos]);
    self.pos += 1;
    self.tok = SimpleSpan::new(start, self.pos);
    Some(Ok(if c == b'+' { Tok::Plus } else { Tok::Num }))
  }
  fn read_frontier(&self) -> tokora::ReadFrontier<usize> { tokora::ReadFrontier::SpanEnd }
  fn bump(&mut self, n: &usize) { self.pos += n; }
}
use rowan::Language;
use tokora::{
  Emitter, InputRef as In, ParseContext, ParseInput,
  cache::DefaultCache,
  cst::{CstProfile, KindValidator, parse_lossless},
  emitter::{CstEmitter, Fatal},
  parser::node,
};

// The dialect's whole u16 kind space: token images, then the node kind, then bookkeeping.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
#[repr(u16)]
enum K { Num, Plus, Expr, Error, Gap, Root }

// The sink-side mapper: committed tokens enter the tree through this one match.
fn map_token(t: &Tok) -> u16 {
  (match t { Tok::Num => K::Num, Tok::Plus => K::Plus }) as u16
}

// The dialect's CST profile: the mapper, the predicate that names the whole kind space
// (every discriminant up to `Root`), and the two bookkeeping kinds.
fn profile() -> CstProfile<Tok> {
  CstProfile::new(
    map_token,
    KindValidator::new(|k| k <= K::Root as u16),
    K::Error as u16,
    K::Gap as u16,
  )
}

// Rowan's raw <-> typed bargain.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
enum Lang {}
impl Language for Lang {
  type Kind = K;
  fn kind_from_raw(raw: rowan::SyntaxKind) -> K {
    const KINDS: [K; 6] = [K::Num, K::Plus, K::Expr, K::Error, K::Gap, K::Root];
    KINDS[raw.0 as usize]
  }
  fn kind_to_raw(k: K) -> rowan::SyntaxKind { rowan::SyntaxKind(k as u16) }
}

type Ln<'a> = CharLexer<'a>;

// One `Expr` node wrapping every token the sub-parse commits. Driven through `parse_input`
// this is the UP-FRONT bracket: `cst_start` opens the node, and the exit closes it either
// way — `cst_finish` here, `cst_demote` had the sub-parse errored. Both exits are appends,
// which is exactly why a rolled-back branch leaves nothing behind.
fn expr<'inp, Ctx>(inp: &mut In<'inp, '_, Ln<'inp>, Ctx>) -> Result<(), Error>
where
  Ctx: ParseContext<'inp, Ln<'inp>>,
  Ctx::Emitter: CstEmitter<'inp, Ln<'inp>> + Emitter<'inp, Ln<'inp>, Error = Error>,
{
  node(K::Expr as u16, |inp: &mut In<'inp, '_, Ln<'inp>, Ctx>| {
    // Each consumed token settles, so `commit_token` records a `Token` event for it.
    while inp.next()?.is_some() {}
    Ok(())
  })
  .parse_input(inp)
}

// Build the WHOLE node speculatively, then decline: every event the branch buffered — the
// node's start, the token settles, its finish — truncates on the one rewind mark. Then
// build it again, for keeps.
fn decline_then_parse<'inp, Ctx>(inp: &mut In<'inp, '_, Ln<'inp>, Ctx>) -> Result<(), Error>
where
  Ctx: ParseContext<'inp, Ln<'inp>>,
  Ctx::Emitter: CstEmitter<'inp, Ln<'inp>> + Emitter<'inp, Ln<'inp>, Error = Error>,
{
  let declined: Option<()> = inp.attempt(|inp| {
    expr(inp).ok()?;
    None // the branch did real work; declining rewinds all of it
  });
  assert!(declined.is_none());
  expr(inp)
}

let src = "1+2";

// Straight drive. The driver mints the sink from `src` itself, so the buffer the tree's
// text comes from and the buffer the parse reads are the same argument of the same call.
let (straight, parsed) = parse_lossless(
  src,
  (),
  Fatal::<Error>::new(),
  profile(),
  DefaultCache::<Ln<'_>>::default(),
  expr,
);
parsed.unwrap();
let (green_straight, _) = straight.finish(K::Root as u16);

// Same source, through the declined speculation.
let (backtracked, parsed) = parse_lossless(
  src,
  (),
  Fatal::<Error>::new(),
  profile(),
  DefaultCache::<Ln<'_>>::default(),
  decline_then_parse,
);
parsed.unwrap();
let (green_backtracked, _) = backtracked.finish(K::Root as u16);

// The declined branch left NO phantom: the two green trees are byte-identical.
let green = green_straight.unwrap();
assert_eq!(green, green_backtracked.unwrap());

// And the round-trip law holds — the reason to build a CST at all.
let tree = rowan::SyntaxNode::<Lang>::new_root(green);
assert_eq!(tree.text().to_string(), src);

// The structure is the grammar's: Root > Expr > [Num "1", Plus "+", Num "2"].
let expr_node = tree.first_child().unwrap();
assert_eq!(expr_node.kind(), K::Expr);
assert_eq!(expr_node.children_with_tokens().count(), 3);

Had the declined branch’s events not rewound, the backtracked tree would carry a phantom Expr and a duplicate run of tokens, and both the byte-identity assertion and the round-trip would fail. They hold because one checkpoint mark pinned the event log, the diagnostics, and the cursor to the begin point, and one rewind released all three from it — the tree rewound for free.

Where to go next

This chapter is the last of Part III’s four seams onto the one engine:

• The mark the sink rides — the pure-copy checkpoint, its last-in-first-out contract, and the cell taxonomy this chapter’s own census mirrors: the Checkpoint, rewind & the LIFO contract chapter, where the value-keyed-inner composition developed above was first sketched.
• The emitter the sink wraps — the atomic Emitter capability family, the value-keyed checkpoint / rewind / release trio, and the CstEmitter structuring surface the node combinators drive: the Atomic Emitter chapter.
• The tutorial, end to end — building a real GraphQL-shaped CST with the node combinators, the typed tree views, recovery error-nodes, and the round-trip oracle, without ever touching an event: chapter 16.
• The flat catalog — every node combinator, the marks, and the emitter surface as terse reference entries: the combinator reference and the errors, emitters & context reference.

Source, Slice, and storage backends

Every parser in this book has been generic over a lifetime 'inp and a lexer L, and has run over &str without ever saying why it could run over anything else. This chapter is the why.

Tokora’s engine — the on-demand, parse-while-lexing machinery introduced in chapter 2, and the immutable-slice model of chapter 9 — never touches a str or a [u8] directly. It reads the input through two small traits, Source and Slice. Everything else — owned or borrowed, text or raw bytes, std or bare-metal — is a matter of which type you hand it.

Like the previous chapter, this one keeps Part III’s register: less here is how to use it, more here is why it is shaped this way.

The problem: one engine, two axes of representation

A parser combinator library that hard-codes &str cannot lex a binary format; one that hard-codes &[u8] throws away UTF-8’s guarantees and has to re-check code-point boundaries by hand. And neither can accept an owned, reference-counted buffer of the kind an async I/O stack hands you — the bytes::Bytes your socket already filled — without a copy back down to a borrow.

The input a real program has on hand varies along two independent axes:

• owned vs borrowed — a &str you are lending the parser, versus a bytes::Bytes or a HipStr the parser can hold a cheap clone of;
• text-shaped vs byte-shaped — a source whose atom is a Unicode scalar value (char, with code-point boundaries to respect) versus one whose atom is a raw u8 (any index is a valid cut).

Tokora refuses to pick for you. Instead it names the seam — the handful of operations the engine actually needs from an input — as a trait, implements that trait once for str and once for [u8], and lets feature-gated backends add more. A parser written against the seam is representation-agnostic for free; it never mentions a concrete source type at all.

The seam is two traits

The engine needs to ask two different questions, so there are two traits:

• Source is implemented on the input medium — the whole thing being lexed (str, [u8], bytes::Bytes, …). It answers how long are you, and give me the sub-range a..b.
• Slice is implemented on what a span of that medium looks like — the value a lexer yields for one token (&str, &[u8], a cheap Bytes clone, …). It answers what are your characters, and how many.

They are bound together by one associated type: Source::Slice<'a>: Slice<'a>. Slicing a Source produces something that is itself a Slice. That is the whole contract, and it is why SliceOf<'inp, L> — the projection <L::Source as Source>::Slice<'inp> — is the type generic parser code reaches for whenever it wants the raw text of a token.

'a is a validity requirement, not merely a label on that projection: Slice<'a>: 'a means that both the slice value and the data it represents remain valid for at least 'a. The canonical Slice implementations therefore live on the representations themselves — str, [u8], BStr, and the optional backend values — rather than on a selected set of reference spellings. In particular, the lifetime-carrying HipStr<'data> and HipByt<'data> implement Slice<'source> when 'data: 'source.

Shared references are forwarded uniformly. If T: Slice<'source> and a reference to T lives for 'data: 'source, then &'data T: Slice<'source>; the same law applies repeatedly to nested references. That makes &str, &&str, &[u8], and borrowed backend values retain the representation, character type, and iterator behavior of their underlying T.

Source: addressing the medium

pub trait Source<Cursor>: core::fmt::Debug {
    type Slice<'source>: Slice<'source> where Self: 'source;

    fn is_empty(&self) -> bool;
    fn len(&self) -> Cursor;
    fn as_slice(&self) -> Self::Slice<'_>;
    fn slice<R>(&self, range: R) -> Option<Self::Slice<'_>>
    where R: RangeBounds<Cursor>;

    fn find_boundary(&self, index: Cursor) -> Cursor { index } // default
    fn is_boundary(&self, index: Cursor) -> bool;
}

Three things are worth reading closely.

• Cursor is a type parameter, not usize. A Source is addressed by whatever offset type its lexer uses; the core impls fix it to usize, but the trait does not. This is the same Offset a Lexer declares (Lexer::Source: Source<Lexer::Offset>), so the offset arithmetic the engine does and the addressing the source understands are the same type by construction.
• slice is fallible and zero-copy. It returns Option<Self::Slice<'_>> — None for an out-of-range or (for text) boundary-splitting range, mirroring slice::get. The '_ selects the associated slice for that call. Borrowing sources such as str tie it to the borrow of self; explicit reference sources such as &'data str may preserve their longer carried lifetime instead. Either way, a token payload can remain a view into the source rather than a fresh allocation.
• find_boundary and is_boundary are the entire text/byte distinction. This is the design decision that keeps the rest of the engine shape-blind.

Source deliberately does not have a blanket implementation for &T. Such an implementation can only select T::Slice using the lifetime of the outer borrow, which shortens source-carried lifetimes: slicing a briefly borrowed &'data str would produce a slice tied to the brief borrow instead of 'data.

The borrowed core media therefore have explicit implementations: &'data str, &'data [u8], and (with bstr_1) &'data BStr all return slices that preserve 'data. Owned backends such as Bytes, HipStr, and the smol-bytes types implement Source on the owner itself. They can still be borrowed normally to call source methods, but &Bytes is intentionally not a distinct source type. This keeps the associated lifetime truthful rather than trading it for blanket convenience.

The boundary discipline

is_boundary(i) asks is i a legal place to cut? find_boundary(i) asks what is the nearest legal cut at or below i? The two core impls answer differently, and that difference is the only place in tokora where “text” and “bytes” mean different things:

For bytes every in-range index is a boundary, so find_boundary is the identity and costs nothing. For text, find_boundary walks down from i until it lands on a code-point boundary (indices at or past the end are returned unchanged, matching the byte behavior). A lexer that advances by a byte count it computed from a regex can therefore call find_boundary and be guaranteed a slice position that will not split a multi-byte scalar — the same call is a no-op on a byte source and a safety net on a text source. This is why Lexer::bump can promise it never lands “in the middle of a UTF-8 code point (does not apply when lexing raw &[u8])”: the promise is delegated to the Source impl, made once, and inherited by every backend of the same shape.

Slice: reading a span

pub trait Slice<'source>: PartialEq + Eq + core::fmt::Debug + 'source {
    type Char: Copy + core::fmt::Debug + PartialEq + Eq + core::hash::Hash;
    type Iter<'a>: Iterator<Item = Self::Char> where Self: 'a;
    type PositionedIter<'a>: Iterator<Item = (usize, Self::Char)> where Self: 'a;

    fn iter<'a>(&'a self) -> Self::Iter<'a> where Self: 'a;
    fn positioned_iter<'a>(&'a self) -> Self::PositionedIter<'a> where Self: 'a;
    fn len(&self) -> usize;
    fn is_empty(&self) -> bool { self.len() == 0 } // default
}

Slice::Char is the shape marker in the type system: char for a text slice, u8 for a byte slice, and it must match the character type the underlying lexer works in. The two iterators are the whole reading interface — iter for the characters, positioned_iter for (offset, character) pairs — and the canonical core impls forward to the standard-library iterators that already do the right thing: str::Chars / str::CharIndices for str, and Copied<slice::Iter> / Enumerate<…> for [u8]. Shared-reference forwarding supplies their usual &str and &[u8] forms without separate implementations. There is no bespoke UTF-8 decoding in tokora; Slice is a thin, uniform face over machinery core already ships.

(Do not confuse Slice with the Sliced<D, Src> struct that lives in the same module. Slice is the span of a source; Sliced is an unrelated provenance wrapper — a value paired with which source it came from, e.g. a file name — the counterpart to Spanned’s where within a source.)

The backends

Beyond the two always-present core impls, four optional crates each add Source/Slice impls for their buffer types. Every one reuses the same iterator machinery as the core — the byte-shaped backends borrow <[u8]>::iter().copied(), the text-shaped ones borrow str::Chars — so a backend is really just an owner type plus its slicing and its boundary rule. Byte-shaped backends delegate is_boundary straight to the [u8] rule (i <= len); the text-shaped ones (HipStr, Utf8Bytes) replicate the str char-boundary logic verbatim.

The single most important column is Slice<'a>. For the borrowed backends it is a plain reference, as you would expect. But for the owned backends it is the owner type again, not a &-borrow — bytes::Bytes::Slice = Bytes, HipStr::Slice = HipStr, and so on. That is not a copy: these types slice in O(1) by bumping a reference count (bytes, smol-bytes shared) or, for a small enough span, by inlining ≤62 bytes into the handle (smol-bytes). An owned source is therefore still zero-copy to slice — tokora does not force borrowing to get the property; it lets each representation express its own cheapest slice, and refcounted buffers happen to have a very cheap one.

A quick tour of what each backend is for:

• bytes_1 exposes bytes::Bytes — the de-facto owned, cheaply cloneable byte buffer of the async ecosystem. Reach for it when the bytes you want to parse already arrived as a Bytes and you want to keep token slices alive past the parse without copying.
• bstr_1 exposes bstr::BStr, a borrowed, byte-shaped view whose slices are &BStr. It is the “bytes that are conventionally text but not guaranteed UTF-8” case; it retains that byte-string representation while forwarding its byte iteration and boundary behavior to the same underlying rules as [u8].
• hipstr_0_8 exposes the hipstr inline-or-shared-or-borrowed hybrids in both shapes: HipStr (text) and HipByt (bytes). A HipStr may be a small string stored inline, a reference-counted heap share, or a borrow — and slicing it preserves whichever it is, so it is the flexible choice when you do not know in advance whether inputs are tiny or huge.
• smol_bytes_0_1 exposes three impls from smol-bytes: the byte-shaped shared::Bytes (the default; ref-counted heap with a 62-byte inline small-buffer optimization, and zero-copy convertible with bytes::Bytes), the byte-shaped compact::Bytes (same inline threshold, but it re-inlines a shrinking view to release its allocation), and the text-shaped Utf8Bytes (a UTF-8 wrapper over the shared strategy — the char-shaped counterpart to HipStr). All three are owned and all three slice cheaply.

Why the feature names carry versions

The feature that turns a backend on is bytes_1, not bytes; hipstr_0_8, not hipstr. Each versioned feature enables a package-renamed optional dependency pinned to one SemVer-major line of the upstream crate (bytes_1 = { package = "bytes", version = "1", … }), and a bare alias forwards to the current one (bytes = ["bytes_1"]). This is the same discipline the crate uses for the logos adapter (logos_0_16, the only supported major — 0.14 and 0.15 have been retired): the version in the name is what lets a future major be added as a purely additive feature, no breaking change to anyone pinned to the current one.

no_std posture

The category list in Cargo.toml includes no-std::no-alloc, and that is a load-bearing claim, not an aspiration. The two core impls — Source/Slice for str and [u8] — carry no feature gate at all. They are core-only: no std, no alloc, no allocator. A parser whose lexer sources from &str or &[u8] compiles and runs on bare metal, and that is the baseline the whole abstraction rests on.

The feature graph layers up from there:

• no feature → core only. Core str / [u8] sources, the parser itself, and its stack-buffered lookahead machinery (peek windows of 1-32 tokens over a small inline cache) — no allocator in sight.

  Stack-buffered means the lookahead is on your stack, so the bound is worth stating. A peek::<W>() reserves one window and its worst case is the whole array live at once:

  W::CAPACITY × size_of::<Maybe<CachedToken<&Token, &State, &Span>,
                                CachedToken<Token, State, Span>>>()
  

  which for every realistic type is W::CAPACITY × (size_of::<Token>() + size_of::<State>() + size_of::<Span>()) plus per-entry padding and a discriminant. W::CAPACITY is at most 32 — U32 is the widest window the crate offers — so that expression, evaluated at 32, is the maximum a single peek can cost. Everything else on the frame is O(1) in the window width: single-entry temporaries, one clone of the lexer (size_of::<L>(), which contains State), and a small fixed part.

  The coefficient is one, not two, and that is recent: through 0.7.3 a peek that looked past the cache staged the extra tokens in a second W::CAPACITY-slot array, so a cache miss cost twice the figure above. At U32 over a 1 KiB token beside a 1 KiB lexer state — 2,072 bytes an entry — that second array was 66,312 bytes of stack nobody asked for, and the peek’s owned storage was 132,632 bytes rather than 66,320. Those tokens are now staged in the window the caller already owns.

  Token, State and Span are yours and unconstrained in size, so this is the one place an embedded target has to do the arithmetic itself: pick the narrowest window that decides the production, and use peek_kind / head_satisfies — which run at U1 — for a head test. Nothing here is heap-allocated and nothing scales with the length of the input.

• alloc → adds the allocator-backed pieces (growable containers, the session stack) while staying no_std.

• std (default) → everything, plus it turns on the upstream backends’ own default features.

The backend features themselves are uniform about std: none implies it.

• bytes_1, bstr_1, hipstr_0_8, and smol_bytes_0_1 all bring their crates in with default-features = false, so enabling a backend does not drag in std. Every backend compiles with neither std nor alloc turned on in tokora (the owned buffers still need a global allocator to link into a final binary, but tokora’s feature graph leaves that choice to you rather than forcing it).
• smol_bytes_0_1 additionally turns on smol-bytes’ own alloc feature — the tier its buffer types live on — so it requires smol-bytes ≥ 0.1.2, the first rlib-only release with an alloc tier. (Earlier 0.1.x built a cdylib alongside the rlib, which needed std’s global allocator and panic handler even to check; that is why this feature once implied std.)

However far down you turn the graph, the parser you write does not change. It is generic over L: Lexer, and L::Source is some Source; the concrete choice of representation lives at the call site, not in the grammar.

The entry-point family

The Parse trait’s methods are the ergonomic front door to all of this. parse / parse_with_state are the general form — they take &L::Source for whatever source your lexer declared, so a lexer whose Source is bytes::Bytes or smol_bytes::compact::Bytes is driven through exactly these. On top of them sit conveniences: parse_str and parse_slice for the core str / [u8] shapes, and — behind their respective backend features — parse_bytes, parse_bstr, and parse_hipstr.

There is a subtlety worth naming, because it explains why those last three exist. parse_bytes, parse_bstr, and parse_hipstr are convenience over a core-sourced lexer: each requires L::Source to be [u8] (or str) and simply borrows your owned buffer down to &[u8] / &str before parsing. They are for “I am holding a bytes::Bytes but my lexer reads [u8].” The owned-type Source impls in the table above are the other path — they are what a lexer uses when its Source associated type genuinely is the owned type, and it is that path that gives you owned, refcount-sliced tokens.

The abstraction, exercised

Nothing above needs a lexer to demonstrate — the Source and Slice traits stand on their own. The behavior starts with canonical str and [u8] implementations. Slice behavior is inherited through its shared-reference forwarding law, while Source additionally has explicit &str and &[u8] implementations that preserve the input reference’s lifetime. The distinction is intentional: Slice only reads an already-produced span, while Source::Slice<'a> must accurately describe how long a newly produced span remains valid.

Here is one function generic over Source and one generic over Slice, each run over both a text source and a byte source, with the boundary discipline doing its job:

use tokora::{Slice, Source};

// Representation-agnostic: take the leading `n` cursor units of any source,
// snapping `n` to a valid boundary so the returned slice is always well-formed.
// For `str` that means a UTF-8 code-point boundary; for `[u8]` every index is a
// boundary, so nothing moves.
fn head<S>(src: &S, n: usize) -> Option<S::Slice<'_>>
where
    S: Source<usize> + ?Sized,
{
    let end = src.find_boundary(n.min(src.len()));
    src.slice(..end)
}

// Slice-level: how many *elements* does this span iterate? The element type is
// the slice's `Char` — `char` for text, `u8` for bytes.
fn elements<'s, S: Slice<'s>>(span: &S) -> usize {
    span.iter().count()
}

// "héllo": 'é' is a two-byte code point, so the text is 6 bytes long.
let text: &str = "héllo";
assert_eq!(text.len(), 6);

// Asking for 2 bytes lands *inside* 'é'. The str source snaps down to a
// boundary, so `head` yields "h" — never a panic, never a split scalar.
assert_eq!(head(text, 2), Some("h"));

// The exact same code over bytes keeps both bytes: every index is valid there.
let bytes: &[u8] = b"h\xC3\xA9llo"; // the UTF-8 encoding of "héllo"
assert_eq!(head(bytes, 2), Some(b"h\xC3".as_slice()));

// One text, two shapes: `str` iterates 5 scalar values; `[u8]` iterates 6 bytes.
assert_eq!(elements(&text), 5);
assert_eq!(elements(&text.as_bytes()), 6);

Note the asymmetry the two signatures reveal: head is generic over S = str / [u8] — the medium, which is what implements Source — while elements is generic over S = &str / &[u8] — the span, which is what implements Slice. Source::Slice<'a>: Slice<'a> is the hinge that connects them, and it is the only line of glue the engine needs to be blind to representation.

The same head would compile unchanged for a lexer sourced from bytes::Bytes or HipStr; the only thing that changes is the concrete S::Slice<'_> it returns — a refcount bump instead of a borrow. That is the payoff of naming the seam: the grammar is written once, and the choice of how the input is stored is somebody else’s, made later, at the edge.

11. Anatomy of a real Tokora parser

Prerequisites: chapters 1–10.

The Calc chapters isolate one idea at a time. A maintained parser program has to make those ideas cooperate: it owns a result model, a token model, a lexer, an error conversion boundary, an entry point, and assertions that exercise the program as users run it. This chapter gives that assembly order, then points to the complete programs that remain the canonical sources.

Start with the output

Decide what a successful parse returns before choosing combinators. A calculator can fold to an f64; an S-expression parser needs an AST that an evaluator consumes later; JSON borrows scalar text while allocating collection nodes; a C expression parser builds an AST. That decision determines whether a parser is manual recursive descent, token-level Pratt, AST-level Pratt, or a combinator composition.

Keep parser functions generic over Ctx. The grammar only requires the capabilities it uses, while the caller chooses a fail-fast or collecting emitter and the cache policy. The small binding parser below shows the complete plumbing without becoming a second full example.

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { Self } }
#[derive(Debug, Clone, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Tok {
  #[token("let")] Let,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[token("=")] Assign,
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))] Int(i64),
  #[token(";")] Semi,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
enum Kind { Let, Ident, Assign, Int, Semi }
impl core::fmt::Display for Kind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self { Self::Let => "let", Self::Ident => "identifier", Self::Assign => "=", Self::Int => "integer", Self::Semi => ";" })
  }
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    core::fmt::Display::fmt(&self.kind(), f)
  }
}
impl TokenT<'_> for Tok {
  type Kind = Kind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> Kind {
    match self { Self::Let => Kind::Let, Self::Ident => Kind::Ident, Self::Assign => Kind::Assign, Self::Int(_) => Kind::Int, Self::Semi => Kind::Semi }
  }
  fn is_trivia(&self) -> bool { false }
}
type BindingLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
#[derive(Debug, PartialEq)]
enum ParseError { Lex, Unexpected, End }
impl From<LexError> for ParseError { fn from(_: LexError) -> Self { Self::Lex } }
impl<O, Lang: ?Sized, Set: Clone + 'static> From<tokora::error::UnexpectedEot<O, Lang, Set>> for ParseError { fn from(_: tokora::error::UnexpectedEot<O, Lang, Set>) -> Self { Self::End } }
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for ParseError { fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { Self::End } }
impl<'inp> From<tokora::error::token::UnexpectedTokenOf<'inp, BindingLexer<'inp>>> for ParseError {
  fn from(_: tokora::error::token::UnexpectedTokenOf<'inp, BindingLexer<'inp>>) -> Self { Self::Unexpected }
}
use tokora::{Emitter, InputRef, Parse, ParseContext, Parser};

fn parse_binding<'inp, Ctx>(
  input: &mut InputRef<'inp, '_, BindingLexer<'inp>, Ctx>,
) -> Result<(&'inp str, i64), ParseError>
where
  Ctx: ParseContext<'inp, BindingLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, BindingLexer<'inp>, Error = ParseError>,
{
  if input.try_expect(|token| matches!(token.data(), Tok::Let))?.is_none() {
    return Err(ParseError::Unexpected);
  }
  if input.try_expect(|token| matches!(token.data(), Tok::Ident))?.is_none() {
    return Err(ParseError::Unexpected);
  }
  let name = input.slice();
  if input.try_expect(|token| matches!(token.data(), Tok::Assign))?.is_none() {
    return Err(ParseError::Unexpected);
  }
  let value = match input.next()? {
    Some(token) => match token.into_data() {
      Tok::Int(value) => value,
      _ => return Err(ParseError::Unexpected),
    },
    None => return Err(ParseError::End),
  };
  if input.try_expect(|token| matches!(token.data(), Tok::Semi))?.is_none() {
    return Err(ParseError::Unexpected);
  }
  Ok((name, value))
}

assert_eq!(
  Parser::new().apply(parse_binding).parse_str("let answer = 42;"),
  Ok(("answer", 42)),
);

The public surface used here is Token, Lexer, lexer::LogosLexer, InputRef, InputRef::next, InputRef::try_expect, InputRef::slice, ParseContext, Emitter, ParseInput, TryParseInput, Parser::new, Parser::apply, and Parse::parse_str. Values returned from the input are spanned, so a real parser can retain source locations as well as data.

Build the lexical layer

Give the token enum payloads only where parsing needs values. Pair it with a payload-free kind enum for dispatch and diagnostics, and convert lexer and structured parser errors into one application error type. A Logos lexer is a good default; a custom Lexer is the variation point when a language needs stateful or non-Logos scanning.

Choose a parser shape

Use manual recursive descent when the next token directly chooses a grammar case. Use combinators for regular sequencing, repetition, separators, and delimiters. Use token-level Pratt when folds can return a token-shaped value; use AST-level Pratt when folds construct a separate tree. Each shape can call the others—there is no all-or-nothing parser style.

Wire the entry point

The executable boundary is deliberately boring: Parser::new().apply(entry).parse_str(source). Put the user-visible conversion from Result to reporting or evaluation there, not inside low-level parser functions. This keeps the grammar reusable in tests, a CLI, and a language server.

Test the complete program

The four programs exercise public behavior from main and are also compiled as examples:

cargo run -p tokora --example calculator --features logos
cargo test -p tokora --no-default-features --features std,logos,rowan,combinators --examples

Small doctests verify local API contracts; the maintained binaries verify their complete integration, including their entry points and assertion tables.

Map the maintained examples

With an output model, lexical layer, parser shape, entry point, and assertions chosen, you can start a real parser and know which maintained program to follow. Next: the custom-lexer recipe, then the walkthroughs, starting with chapter 12.

Recipe: writing a custom lexer

Every parser in this guide runs over a token stream, and that stream comes from a Lexer. Chapter 1 took Calc’s lexer as given; this recipe turns the seam around and builds one from scratch for a small language — a mini config dialect of key = value entries with # comments:

# ports
http  = 8080
debug = true

By the end you will have: a token vocabulary, a working lexer over it, a parser driven by that lexer, the token capabilities that unlock the vocabulary layer, and a clear rule for the one lexer decision that a lossless CST depends on. The primary path is logos-backed — the LogosLexer adapter turns a #[derive(Logos)] enum into a conforming lexer, so you write scanning rules, not a scanner. A brief detour covers hand-writing the Lexer trait directly for the cases logos cannot express.

We build it in two passes. Steps 1–4 grow a syntactic lexer that skips trivia — the right choice for an AST or evaluator — starting with whitespace. Step 5 then makes it lossless by surfacing whitespace and comments as tokens, which is the shape a lossless CST needs.

Step 1 — the token vocabulary

A tokora token is two types linked by the Token trait (the split is chapter 1’s subject): the token carries payloads (Int(i64) holds its value), and its Kind is a payload-free Copy discriminant that dispatch tables and “expected one of …” diagnostics name. With logos, the token enum is the lexer: each #[token]/#[regex] is a scanning rule, and a top-level skip drops trivia at the lexer level — here, whitespace (Step 5 adds comments).

use tokora::{
  Lexer, Token,
  logos::{self, Logos},
};

// The lexer-level error: what lexing yields for bytes that are no token at all.
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;

impl From<()> for LexError {
  fn from(_: ()) -> Self {
    LexError
  }
}

// The raw scanner. Each attribute is a rule; `skip` discards whitespace *at the lexer level*,
// so it never reaches the parser (Step 5 revisits that decision).
#[derive(Debug, Clone, PartialEq, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Tok {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))]
  Int(i64),
  // Two spellings, one variant: logos gives explicit tokens priority over the identifier
  // regex, so `true`/`false` win the overlap.
  #[token("true", |_| true)]
  #[token("false", |_| false)]
  Bool(bool),
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")]
  Ident,
  #[token("=")]
  Eq,
}

// The payload-free discriminant. Its `Display` is what diagnostics print.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokKind {
  Int,
  Bool,
  Ident,
  Eq,
}

impl core::fmt::Display for TokKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Self::Int => "integer",
      Self::Bool => "boolean",
      Self::Ident => "identifier",
      Self::Eq => "`=`",
    })
  }
}

// The bridge. `Token` names the kind and the lexer error, and classifies trivia.
impl Token<'_> for Tok {
  type Kind = TokKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;

  // This lexer *skips* whitespace and comments, so no surviving token is ever trivia and
  // `SURFACES_TRIVIA` keeps its default `false`. Step 5 covers when to flip both.
  fn kind(&self) -> TokKind {
    match self {
      Tok::Int(_) => TokKind::Int,
      Tok::Bool(_) => TokKind::Bool,
      Tok::Ident => TokKind::Ident,
      Tok::Eq => TokKind::Eq,
    }
  }

  fn is_trivia(&self) -> bool {
    false
  }
}

// The whole lexer: one type alias over the logos adapter.
type ConfigLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;

// Drive it by hand once. `slice()` borrows straight from the source — no copy — and skipped
// whitespace has already vanished from the stream.
let mut lexer = ConfigLexer::new("http = 8080\n");
let mut items = Vec::new();
while let Some(result) = lexer.lex() {
  let tok = result.expect("every non-trivia byte belongs to a token");
  items.push((tok.kind(), lexer.slice()));
}
assert_eq!(
  items,
  [
    (TokKind::Ident, "http"),
    (TokKind::Eq, "="),
    (TokKind::Int, "8080"),
  ],
);

That is the whole lexer. LogosLexer::new builds it, lex pulls one token at a time (returning None at end of input), and slice and span describe the token just produced.

Step 2 — drive a parse

A tokora parser is a plain function over an InputRef: it pulls tokens with next and peeks-or-takes with try_expect, exactly as chapter 2 introduces. The parser is generic over its parse context Ctx (the emitter+cache bundle — see chapter 2 and the errors & context reference), pinning only the emitter’s error type. Parser::new then hands it a default fail-fast (Fatal) context and runs it against a source.

Entry points do not enforce end of input, so a whole-document parser loops until the stream runs dry itself:

use tokora::{Lexer, Token, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, PartialEq, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Tok {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))]
  Int(i64),
  #[token("true", |_| true)]
  #[token("false", |_| false)]
  Bool(bool),
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[token("=")] Eq,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokKind { Int, Bool, Ident, Eq }
impl core::fmt::Display for TokKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self { Self::Int => "integer", Self::Bool => "boolean", Self::Ident => "identifier", Self::Eq => "`=`" })
  }
}
impl Token<'_> for Tok {
  type Kind = TokKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokKind {
    match self { Tok::Int(_) => TokKind::Int, Tok::Bool(_) => TokKind::Bool, Tok::Ident => TokKind::Ident, Tok::Eq => TokKind::Eq }
  }
  fn is_trivia(&self) -> bool { false }
}
type ConfigLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
use tokora::{
  Emitter, InputRef, Parse, ParseContext, Parser,
  error::{UnexpectedEot, token::UnexpectedTokenOf},
};

// A parsed value, and the parser's error. The `From` impls are what let the crate's
// structured errors collapse into it (the `FromEmitterError` bound the entry points ask for).
#[derive(Debug, PartialEq)]
enum Value {
  Int(i64),
  Bool(bool),
}

#[derive(Debug, Clone, PartialEq)]
enum ConfigError {
  Lex,           // bytes that are no token at all
  Unexpected,    // a wrong token in a right place
  UnexpectedEnd, // input ended mid-entry
}

impl From<LexError> for ConfigError {
  fn from(_: LexError) -> Self {
    ConfigError::Lex
  }
}
impl<'inp> From<UnexpectedTokenOf<'inp, ConfigLexer<'inp>>> for ConfigError {
  fn from(_: UnexpectedTokenOf<'inp, ConfigLexer<'inp>>) -> Self {
    ConfigError::Unexpected
  }
}
impl<O, Lang: ?Sized, Set: Clone + 'static> From<UnexpectedEot<O, Lang, Set>> for ConfigError {
  fn from(_: UnexpectedEot<O, Lang, Set>) -> Self {
    ConfigError::UnexpectedEnd
  }
}
impl<'inp, L: Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for ConfigError {
  fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self {
    ConfigError::UnexpectedEnd
  }
}

/// Parses a run of `<ident> = <int|bool>` entries to end of input.
fn config<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, ConfigLexer<'inp>, Ctx>,
) -> Result<Vec<(&'inp str, Value)>, ConfigError>
where
  Ctx: ParseContext<'inp, ConfigLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, ConfigLexer<'inp>, Error = ConfigError>,
{
  let mut entries = Vec::new();
  // Peek for a key; end of input (or any non-identifier) ends the loop cleanly.
  while inp.try_expect(|t| matches!(t.data(), Tok::Ident))?.is_some() {
    let key = inp.slice(); // zero-copy: the just-consumed identifier's text
    if inp.try_expect(|t| matches!(t.data(), Tok::Eq))?.is_none() {
      return Err(ConfigError::Unexpected);
    }
    let value = match inp.next()? {
      Some(tok) => match tok.into_data() {
        Tok::Int(n) => Value::Int(n),
        Tok::Bool(b) => Value::Bool(b),
        _ => return Err(ConfigError::Unexpected),
      },
      None => return Err(ConfigError::UnexpectedEnd),
    };
    entries.push((key, value));
  }
  Ok(entries)
}

// `Parser::new()` supplies the default fail-fast context; `.parse_str` runs it.
let parsed = Parser::new()
  .apply(config)
  .parse_str("http = 8080\ndebug = true\n");
assert_eq!(
  parsed,
  Ok(vec![("http", Value::Int(8080)), ("debug", Value::Bool(true))]),
);

// The typed error carries a wrong token out through the `Err` channel.
let bad = Parser::new().apply(config).parse_str("http 8080");
assert_eq!(bad, Err(ConfigError::Unexpected));

Step 3 — token capabilities

The hand-written matches! and try_expect above work, but tokora ships a vocabulary layer — ready-made punctuators, a keyword! generator, delimiters — that parses against your token if the token opts into the matching capability trait. Each capability is a subtrait of Token you implement by pointing a few methods at your Kinds:

• PunctuatorToken — map ASCII punctuation to kinds (equal() -> Some(TokKind::Eq)); unlocks the ~80 built-in punct types and their parse/try_parse, plus token-level predicates like is_equal().
• KeywordToken — report a token’s canonical spelling; unlocks keyword!-generated keyword parsers.
• IdentifierToken — flag identifier tokens.
• LitToken and PrattToken round out the set for literals and Pratt expressions.

You implement only what your language uses; every method defaults to “not me”. The vocabulary reference is the full catalog — here is the opt-in and one parse of each kind:

use tokora::{Lexer, Token, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, PartialEq, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Tok {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))]
  Int(i64),
  #[token("true", |_| true)]
  #[token("false", |_| false)]
  Bool(bool),
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[token("=")] Eq,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokKind { Int, Bool, Ident, Eq }
impl core::fmt::Display for TokKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self { Self::Int => "integer", Self::Bool => "boolean", Self::Ident => "identifier", Self::Eq => "`=`" })
  }
}
impl Token<'_> for Tok {
  type Kind = TokKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokKind {
    match self { Tok::Int(_) => TokKind::Int, Tok::Bool(_) => TokKind::Bool, Tok::Ident => TokKind::Ident, Tok::Eq => TokKind::Eq }
  }
  fn is_trivia(&self) -> bool { false }
}
type ConfigLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
use tokora::error::{UnexpectedEot, token::UnexpectedTokenOf};
#[derive(Debug, Clone, PartialEq)]
enum ConfigError { Lex, Unexpected, UnexpectedEnd }
impl From<LexError> for ConfigError { fn from(_: LexError) -> Self { ConfigError::Lex } }
impl<'inp> From<UnexpectedTokenOf<'inp, ConfigLexer<'inp>>> for ConfigError {
  fn from(_: UnexpectedTokenOf<'inp, ConfigLexer<'inp>>) -> Self { ConfigError::Unexpected }
}
impl<O, Lang: ?Sized, Set: Clone + 'static> From<UnexpectedEot<O, Lang, Set>> for ConfigError { fn from(_: UnexpectedEot<O, Lang, Set>) -> Self { ConfigError::UnexpectedEnd } }
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for ConfigError { fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { ConfigError::UnexpectedEnd } }
use tokora::{
  Emitter, InputRef, Parse, ParseContext, Parser,
  keyword,
  punct::Equal,
  token::{IdentifierToken, KeywordToken, PunctuatorToken, PunctuatorTokenExt},
};

// Opt in: teach the token which kinds are which punctuator / keyword / identifier.
impl PunctuatorToken<'_> for Tok {
  fn equal() -> Option<TokKind> {
    Some(TokKind::Eq)
  }
}
impl KeywordToken<'_> for Tok {
  fn keyword(&self) -> Option<&'static str> {
    match self {
      Tok::Bool(true) => Some("true"),
      Tok::Bool(false) => Some("false"),
      _ => None,
    }
  }
}
impl IdentifierToken<'_> for Tok {
  fn is_identifier(&self) -> bool {
    matches!(self, Tok::Ident)
  }
}

// Token-level predicates now read the classification without matching on `Kind`.
assert_eq!(<Tok as PunctuatorToken>::equal(), Some(TokKind::Eq));
assert!(Tok::Eq.is_equal());
assert_eq!(Tok::Bool(true).keyword(), Some("true"));
assert!(Tok::Ident.is_identifier());

// A `keyword!` type parses against `KeywordToken`; it matches when the token's spelling agrees.
keyword! {
    /// The `true` literal keyword.
    (True, "TRUE", "true"),
}

// The built-in `Equal` punctuator replaces the hand-written `=` check.
fn eq_sign<'inp, Ctx>(inp: &mut InputRef<'inp, '_, ConfigLexer<'inp>, Ctx>) -> Result<(), ConfigError>
where
  Ctx: ParseContext<'inp, ConfigLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, ConfigLexer<'inp>, Error = ConfigError>,
{
  Equal::parse(inp)?;
  Ok(())
}
fn a_true<'inp, Ctx>(inp: &mut InputRef<'inp, '_, ConfigLexer<'inp>, Ctx>) -> Result<(), ConfigError>
where
  Ctx: ParseContext<'inp, ConfigLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, ConfigLexer<'inp>, Error = ConfigError>,
{
  True::parse(inp)?;
  Ok(())
}

assert!(Parser::new().apply(eq_sign).parse_str("=").is_ok());
assert!(Parser::new().apply(eq_sign).parse_str("x").is_err());
assert!(Parser::new().apply(a_true).parse_str("true").is_ok());
assert!(Parser::new().apply(a_true).parse_str("false").is_err());

Equal::parse and True::parse needed no new error From impls beyond UnexpectedTokenOf and UnexpectedEot — the same two the hand-written parser already carried. See the vocabulary reference for the punctuator!/keyword! macros, delimiter types, and the Lang-generic spellings.

Step 4 — the hand-written path

Lexer is a plain trait. When logos does not fit — a whitespace-sensitive or indentation-based grammar, a lexer that must thread nesting depth through State, a bespoke source type — implement it directly. The surface is small:

trait Lexer<'inp> {
    type State: State;                             // resume "mode"; cloned on every checkpoint — keep it cheap
    type Source: Source<Self::Offset> + ?Sized;   // str, [u8], a custom backend
    type Token:  Token<'inp>;
    type Span:   Span<Offset = Self::Offset> + …;
    type Offset: …;                               // usize for str / [u8]
    const SURFACES_TRIVIA: bool = <Self::Token>::SURFACES_TRIVIA;   // defaults to the vocabulary's

    fn new(src: &'inp Self::Source) -> Self;
    fn with_state(src: &'inp Self::Source, state: Self::State) -> Self;  // the *resume* constructor
    fn check(&self) -> Result<(), TokenError>;    // e.g. a resource-limit probe
    fn state(&self) -> &Self::State;
    fn state_mut(&mut self) -> &mut Self::State;
    fn into_state(self) -> Self::State;
    fn source(&self) -> &'inp Self::Source;
    fn span(&self)  -> Self::Span;                 // the current token's span
    fn slice(&self) -> SliceOf<'inp, Self>;        // the current token's text (zero-copy)
    fn lex(&mut self) -> Option<Result<Self::Token, TokenError>>;  // None = end of input (sticky)
    fn read_frontier(&self) -> ReadFrontier<Self::Offset>;   // no default: how far have you safely read?
    fn bump(&mut self, n: &Self::Offset);
}

The combinator reference carries a complete, compiling hand-written lexer — CharLexer, a byte-per-character scanner — that you can copy as a starting skeleton.

What makes it correct is the lexer contract, because the input layer rebuilds a fresh lexer and re-lexes on demand for lookahead and backtracking. In brief: scanning is a pure function of source, offset, and State (so replay after a rewind is identical); lex exhaustion is sticky (None stays None); spans are monotone and nonempty; span and slice agree; and a composite token owns its contents — a string literal or block comment is one token whose span swallows every delimiter inside it, so a { buried in a string never perturbs balanced recovery. LogosLexer upholds every clause for you; a hand-written lexer must uphold them itself, and the conformance kit (chapter 10) checks a lexer against the contract mechanically. Partial/streaming input adds one more clause (chapter 9).

Step 5 — trivia and losslessness

The one lexer decision with a downstream consequence: does the lexer skip trivia (whitespace, comments) at the lexer level, or surface it as real tokens?

• Skip it (the config lexer above, and Calc). Trivia never reaches the parser, is_trivia is always false, and SURFACES_TRIVIA keeps its default false. This is the right choice for a purely syntactic parse — an AST, an evaluator, a REPL.
• Surface it — drop the skip rules, add trivia variants whose is_trivia returns true, and declare SURFACES_TRIVIA = true. That constant is a totality promise: every source byte is covered by an emitted token (trivia included) or a reported lexer error, none silently discarded.

A lossless CST (see [crate::cst] and the lossless-CST chapter) needs the surfacing lexer: its gap-filling cst::Sink tiles the whole source, and a skipped-trivia gap is indistinguishable at the event level from a dropped token. So the lossless (gap_kind) sink refuses at compile time to be built over a lexer that does not declare SURFACES_TRIVIA — the guarantee is enforced, not hoped for. (Declaring true while still skipping is a contract violation surfaced as cst::FinishError::UncoveredGap, never UB.)

Here is the config vocabulary rebuilt to surface trivia — the shape a lossless parse requires:

use tokora::{Lexer, Token, span::Span as _, logos::{self, Logos}};

#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError {
  fn from(_: ()) -> Self {
    LexError
  }
}

// A *lossless* vocabulary: whitespace and comments are real tokens — note the absence of `skip`.
#[derive(Debug, Clone, PartialEq, Logos)]
#[logos(crate = logos, error = LexError)]
enum Lossless {
  #[regex(r"[ \t\r\n]+")]
  Whitespace,
  #[regex(r"#[^\r\n]*", allow_greedy = true)]
  Comment,
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))]
  Int(i64),
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")]
  Ident,
  #[token("=")]
  Eq,
}

#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum LosslessKind {
  Whitespace,
  Comment,
  Int,
  Ident,
  Eq,
}

impl core::fmt::Display for LosslessKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Self::Whitespace => "whitespace",
      Self::Comment => "comment",
      Self::Int => "integer",
      Self::Ident => "identifier",
      Self::Eq => "`=`",
    })
  }
}

impl Token<'_> for Lossless {
  type Kind = LosslessKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;

  // The totality promise the lossless CST sink checks at compile time.
  const SURFACES_TRIVIA: bool = true;

  fn kind(&self) -> LosslessKind {
    match self {
      Lossless::Whitespace => LosslessKind::Whitespace,
      Lossless::Comment => LosslessKind::Comment,
      Lossless::Int(_) => LosslessKind::Int,
      Lossless::Ident => LosslessKind::Ident,
      Lossless::Eq => LosslessKind::Eq,
    }
  }

  // The per-token identity half: which surfaced tokens are trivia.
  fn is_trivia(&self) -> bool {
    matches!(self, Lossless::Whitespace | Lossless::Comment)
  }
}

// The `LogosLexer` adapter is one blanket impl for every token type, so a logos-backed dialect
// declares `SURFACES_TRIVIA` on its `Token` impl (above); `Lexer::SURFACES_TRIVIA` inherits it.
// A hand-written lexer whose skipping differs from its vocabulary overrides the `Lexer` const.
type LosslessLexer<'a> = tokora::lexer::LogosLexer<'a, Lossless>;

// Every byte is now a token — trivia included — so the spans tile the source with no gaps.
let mut lexer = LosslessLexer::new("x = 1 # note");
let mut spans = Vec::new();
while let Some(result) = lexer.lex() {
  result.expect("every byte belongs to a token");
  spans.push((lexer.span().start(), lexer.span().end()));
}
assert_eq!(spans.len(), 7); // ident, ws, `=`, ws, int, ws, comment
assert_eq!(spans.first().map(|s| s.0), Some(0)); // cover starts at the first byte …
assert_eq!(spans.last().map(|s| s.1), Some(12)); // … and reaches the last, contiguously

That is the entire difference between a syntactic lexer and a lossless one: no skip, trivia variants that answer is_trivia, and the SURFACES_TRIVIA promise. Pick the first for an AST or evaluator; pick the second when you need to reconstruct the source exactly — the lossless-CST chapter builds a full typed tree on top of it.

Next: the walkthroughs (calculator, JSON) each define a lexer with these steps, then build a real grammar over it.

12. Walkthrough: calculator

Prerequisites: chapters 5, 10, and 11.

This walkthrough builds the maintained calculator.rs end to end, inline. It is a token-level Pratt evaluator: the parser classifies tokens, then folds directly to an f64 rather than allocating an expression AST.

Chapter 5 taught the token-level Pratt engine with a plain-i64 ladder; this chapter is the maintained instantiation of that same engine, with two deliberate differences worth watching for as they appear below: a named Power newtype for the precedence ladder (instead of a bare integer) and f64 arithmetic (so ^ is powf and folds can produce fractional results). Every part — lexer, self-classifying token, folds, and the one-call entry point — is shown as a compiling doctest, so you can follow the whole calculator without leaving the page.

Define token, kind, lexer alias, and CalcError

The enum carries numeric payloads in Token::Num(f64) and leaves classification to a separate TokenKind (a fieldless enum, so it can be Copy + Eq + Hash as the Token trait requires of its Kind). The program derives the Logos lexer, aliases it as CalcLexer, and has one CalcError family for lexical errors, an unexpected token, and an unexpected end. The From conversions are what let a generic Ctx::Emitter return the application error — including the two Pratt-specific expression-end errors, UnexpectedEoLhs and UnexpectedEoRhs, that the token-level engine reports through the emitter when an operator runs out of operand.

use tokora::{Token as TokenT, logos::{self, Logos}};

#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { Self } }

#[derive(Debug, Clone, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Token {
  #[regex(r"[0-9]+(\.[0-9]+)?", |lex| lex.slice().parse::<f64>().map_err(|_| LexError))]
  Num(f64),
  #[token("+")] Plus,
  #[token("-")] Minus,
  #[token("*")] Star,
  #[token("/")] Slash,
  #[token("^")] Caret,
  #[token("(")] LParen,
  #[token(")")] RParen,
}

#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokenKind { Num, Plus, Minus, Star, Slash, Caret, LParen, RParen }

impl core::fmt::Display for TokenKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Self::Num => "number", Self::Plus => "+", Self::Minus => "-", Self::Star => "*",
      Self::Slash => "/", Self::Caret => "^", Self::LParen => "(", Self::RParen => ")",
    })
  }
}
impl core::fmt::Display for Token {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    core::fmt::Display::fmt(&self.kind(), f)
  }
}

// Classification lives on a separate type; `kind()` just projects into it.
impl From<&Token> for TokenKind {
  fn from(t: &Token) -> Self {
    match t {
      Token::Num(_) => Self::Num, Token::Plus => Self::Plus, Token::Minus => Self::Minus,
      Token::Star => Self::Star, Token::Slash => Self::Slash, Token::Caret => Self::Caret,
      Token::LParen => Self::LParen, Token::RParen => Self::RParen,
    }
  }
}
impl TokenT<'_> for Token {
  type Kind = TokenKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokenKind { TokenKind::from(self) }
  fn is_trivia(&self) -> bool { false }
}

type CalcLexer<'a> = tokora::lexer::LogosLexer<'a, Token>;

#[derive(Debug)]
enum CalcError { Lex(LexError), UnexpectedToken, UnexpectedEot }

impl From<LexError> for CalcError { fn from(e: LexError) -> Self { Self::Lex(e) } }
impl<'inp> From<tokora::error::token::UnexpectedTokenOf<'inp, CalcLexer<'inp>>> for CalcError {
  fn from(_: tokora::error::token::UnexpectedTokenOf<'inp, CalcLexer<'inp>>) -> Self { Self::UnexpectedToken }
}
// Both Pratt expression-end errors collapse to the same "ran out of input" variant.
impl From<tokora::error::UnexpectedEot> for CalcError { fn from(_: tokora::error::UnexpectedEot) -> Self { Self::UnexpectedEot } }
impl From<tokora::error::UnexpectedEoLhs> for CalcError { fn from(_: tokora::error::UnexpectedEoLhs) -> Self { Self::UnexpectedEot } }
impl From<tokora::error::UnexpectedEoRhs> for CalcError { fn from(_: tokora::error::UnexpectedEoRhs) -> Self { Self::UnexpectedEot } }
impl<O, Lang: ?Sized> From<tokora::error::RecursionLimitReached<O, Lang>> for CalcError { fn from(_: tokora::error::RecursionLimitReached<O, Lang>) -> Self { Self::UnexpectedEot } }
impl<O, Lang: ?Sized> From<tokora::error::NonAssociativeChain<O, Lang>> for CalcError { fn from(_: tokora::error::NonAssociativeChain<O, Lang>) -> Self { Self::UnexpectedEot } }

assert_eq!(Token::Star.kind(), TokenKind::Star);
assert_eq!(Token::Num(1.5).kind(), TokenKind::Num);

The relevant public APIs are Token, token::PrattToken, parser::PrattPower, parser::PrattLHS, parser::PrattRHS, parser::Precedenced, parser::PrattInfix, InputRef::pratt, PrattEmitter, Spanned, Parser, and Parse::parse_str.

Define the precedence constants and grouping sentinel

Power(i32) names this language’s ladder. It is useful for making the domain explicit, not for orphan-rule reasons: Tokora implements PrattPower for the standard integer types too, so a bare i64 (chapter 5’s choice) would also work. The grouping sentinel is below the default floor so an opening parenthesis can recurse at that lower floor and consume its matching closing parenthesis without exposing it to the outer expression.

use tokora::parser::PrattPower;

#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, PartialOrd, Ord)]
struct Power(i32);

// Nothing to implement: a binding power is only ever compared, never stepped.
impl PrattPower for Power {}

const PREC_PAREN: Power = Power(-1); // ( )       — below the floor
const PREC_SUM: Power = Power(1);    // + -
const PREC_PROD: Power = Power(2);   // * /
const PREC_NEG: Power = Power(3);    // unary -
const PREC_EXP: Power = Power(4);    // ^

// A top-level parse starts at the default floor.
assert_eq!(Power::default(), Power(0));
// `(` sits *below* that floor, so a stray `)` is invisible at the top level and left for the
// caller — but consumable inside the recursive call a `(` prefix opens (whose floor is PREC_PAREN).
assert!(PREC_PAREN < Power::default());
// Associativity is how strictly the engine compares against a level, not a move along the
// ladder: after a left-associative `+` the recursion admits only powers strictly above
// PREC_SUM, so `*` gets in and another `+` does not.
assert!(PREC_PROD > PREC_SUM);
assert!(PREC_NEG < PREC_EXP);

Implement try_pratt_lhs and try_pratt_rhs

The PrattToken implementation turns that ladder into the engine’s classifier: the token type describes itself at each position. try_pratt_lhs accepts a number (an operand), a prefix minus, or an opening parenthesis; try_pratt_rhs accepts the infix operators and the closing-parenthesis postfix sentinel. Returning None tells the engine that the token is not part of this expression here, so it is left on the input and the loop stops. ^ is the one Right-associative row; (/) share PREC_PAREN.

use tokora::{Token as TokenT, logos::{self, Logos}};
use tokora::EmitterView;
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { Self } }
#[derive(Debug, Clone, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Token {
  #[regex(r"[0-9]+(\.[0-9]+)?", |lex| lex.slice().parse::<f64>().map_err(|_| LexError))] Num(f64),
  #[token("+")] Plus, #[token("-")] Minus, #[token("*")] Star, #[token("/")] Slash,
  #[token("^")] Caret, #[token("(")] LParen, #[token(")")] RParen,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokenKind { Num, Plus, Minus, Star, Slash, Caret, LParen, RParen }
impl core::fmt::Display for TokenKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self { Self::Num => "number", Self::Plus => "+", Self::Minus => "-", Self::Star => "*", Self::Slash => "/", Self::Caret => "^", Self::LParen => "(", Self::RParen => ")" })
  }
}
impl core::fmt::Display for Token { fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { core::fmt::Display::fmt(&self.kind(), f) } }
impl From<&Token> for TokenKind {
  fn from(t: &Token) -> Self { match t { Token::Num(_) => Self::Num, Token::Plus => Self::Plus, Token::Minus => Self::Minus, Token::Star => Self::Star, Token::Slash => Self::Slash, Token::Caret => Self::Caret, Token::LParen => Self::LParen, Token::RParen => Self::RParen } }
}
impl TokenT<'_> for Token {
  type Kind = TokenKind; type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokenKind { TokenKind::from(self) }
  fn is_trivia(&self) -> bool { false }
}
use tokora::parser::PrattPower;
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, PartialOrd, Ord)]
struct Power(i32);
impl PrattPower for Power {}
const PREC_PAREN: Power = Power(-1);
const PREC_SUM: Power = Power(1);
const PREC_PROD: Power = Power(2);
const PREC_NEG: Power = Power(3);
const PREC_EXP: Power = Power(4);
use tokora::parser::{PrattInfix, PrattLHS, PrattRHS, Precedenced};
use tokora::token::PrattToken;

impl PrattToken<'_, f64, Power> for Token {
  fn try_pratt_lhs(&self) -> Option<PrattLHS<(), (), Power>> {
    Some(match self {
      Token::Num(_) => PrattLHS::Operand(()),
      Token::Minus => PrattLHS::Prefix(Precedenced::new((), PREC_NEG)),
      Token::LParen => PrattLHS::Prefix(Precedenced::new((), PREC_PAREN)),
      _ => return None,
    })
  }

  fn try_pratt_rhs(&self) -> Option<PrattRHS<(), (), (), (), Power>> {
    Some(match self {
      Token::Plus => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(()), PREC_SUM)),
      Token::Minus => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(()), PREC_SUM)),
      Token::Star => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(()), PREC_PROD)),
      Token::Slash => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(()), PREC_PROD)),
      // The one right-associative row: `2 ^ 3 ^ 2` groups as `2 ^ (3 ^ 2)`.
      Token::Caret => PrattRHS::Infix(Precedenced::new(PrattInfix::Right(()), PREC_EXP)),
      // `)` is a postfix at PREC_PAREN, consumed only inside the group `(` opened.
      Token::RParen => PrattRHS::Postfix(Precedenced::new((), PREC_PAREN)),
      _ => return None,
    })
  }
}

// A number is an operand; `-` and `(` open the left edge; everything else declines.
assert!(matches!(Token::Num(1.0).try_pratt_lhs(), Some(PrattLHS::Operand(()))));
assert!(matches!(Token::Minus.try_pratt_lhs(), Some(PrattLHS::Prefix(_))));
assert!(Token::Plus.try_pratt_lhs().is_none());
// `^` is an infix, `)` a postfix; a bare number has no right-hand-side role.
assert!(matches!(Token::Caret.try_pratt_rhs(), Some(PrattRHS::Infix(_))));
assert!(matches!(Token::RParen.try_pratt_rhs(), Some(PrattRHS::Postfix(_))));
assert!(Token::Num(1.0).try_pratt_rhs().is_none());

Implement the named prefix, infix, and postfix folds

Use named functions rather than closures because the token-level fold traits require a higher-ranked lifetime bound on the emitter (for<'lt> FnMut(…, &'lt mut Emitter)); a closure is monomorphic in that lifetime and does not satisfy it, while a fn item is generic over its lifetimes and satisfies it for free. fold_prefix negates a number or passes a grouped value through; fold_infix extracts the operator from PrattInfix and computes the next f64 (here ^ is powf); fold_postfix acknowledges a closing parenthesis and returns its operand. Each fold trades in Spanned<Token>, so the evaluated value goes back in as a Token::Num. The emitter parameter is unused here, so the folds can even be exercised directly with E = ():

use tokora::{Token as TokenT, logos::{self, Logos}};
use tokora::EmitterView;
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { Self } }
#[derive(Debug, Clone, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Token {
  #[regex(r"[0-9]+(\.[0-9]+)?", |lex| lex.slice().parse::<f64>().map_err(|_| LexError))] Num(f64),
  #[token("+")] Plus, #[token("-")] Minus, #[token("*")] Star, #[token("/")] Slash,
  #[token("^")] Caret, #[token("(")] LParen, #[token(")")] RParen,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokenKind { Num, Plus, Minus, Star, Slash, Caret, LParen, RParen }
impl core::fmt::Display for TokenKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self { Self::Num => "number", Self::Plus => "+", Self::Minus => "-", Self::Star => "*", Self::Slash => "/", Self::Caret => "^", Self::LParen => "(", Self::RParen => ")" })
  }
}
impl core::fmt::Display for Token { fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { core::fmt::Display::fmt(&self.kind(), f) } }
impl From<&Token> for TokenKind {
  fn from(t: &Token) -> Self { match t { Token::Num(_) => Self::Num, Token::Plus => Self::Plus, Token::Minus => Self::Minus, Token::Star => Self::Star, Token::Slash => Self::Slash, Token::Caret => Self::Caret, Token::LParen => Self::LParen, Token::RParen => Self::RParen } }
}
impl TokenT<'_> for Token {
  type Kind = TokenKind; type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokenKind { TokenKind::from(self) }
  fn is_trivia(&self) -> bool { false }
}
type CalcLexer<'a> = tokora::lexer::LogosLexer<'a, Token>;
#[derive(Debug)]
enum CalcError { Lex(LexError), UnexpectedToken, UnexpectedEot }
impl From<LexError> for CalcError { fn from(e: LexError) -> Self { Self::Lex(e) } }
impl<'inp> From<tokora::error::token::UnexpectedTokenOf<'inp, CalcLexer<'inp>>> for CalcError { fn from(_: tokora::error::token::UnexpectedTokenOf<'inp, CalcLexer<'inp>>) -> Self { Self::UnexpectedToken } }
impl From<tokora::error::UnexpectedEot> for CalcError { fn from(_: tokora::error::UnexpectedEot) -> Self { Self::UnexpectedEot } }
impl From<tokora::error::UnexpectedEoLhs> for CalcError { fn from(_: tokora::error::UnexpectedEoLhs) -> Self { Self::UnexpectedEot } }
impl From<tokora::error::UnexpectedEoRhs> for CalcError { fn from(_: tokora::error::UnexpectedEoRhs) -> Self { Self::UnexpectedEot } }
impl<O, Lang: ?Sized> From<tokora::error::RecursionLimitReached<O, Lang>> for CalcError { fn from(_: tokora::error::RecursionLimitReached<O, Lang>) -> Self { Self::UnexpectedEot } }
impl<O, Lang: ?Sized> From<tokora::error::NonAssociativeChain<O, Lang>> for CalcError { fn from(_: tokora::error::NonAssociativeChain<O, Lang>) -> Self { Self::UnexpectedEot } }
use tokora::{SimpleSpan, parser::PrattInfix, span::Spanned};

fn fold_prefix<'inp, E>(
  op: Spanned<Token, SimpleSpan>,
  operand: Spanned<Token, SimpleSpan>,
  _: EmitterView<'_, 'inp, CalcLexer<'inp>, E>,
) -> Result<Spanned<Token, SimpleSpan>, CalcError> {
  let (span, op) = op.into_components();
  match op {
    Token::Minus => {
      let n = match operand.into_data() { Token::Num(n) => n, _ => unreachable!() };
      Ok(Spanned::new(span, Token::Num(-n)))
    }
    // Grouping: the `(` prefix's "operand" is the whole parenthesised expression, already folded
    // by the inner call (which also ate the `)`). Pass it through untouched.
    Token::LParen => Ok(operand),
    _ => unreachable!("the LHS table admits only `-` and `(` as prefixes"),
  }
}

fn fold_infix<'inp, E>(
  left: Spanned<Token, SimpleSpan>,
  right: Spanned<Token, SimpleSpan>,
  infix: Spanned<PrattInfix<Token, Token, Token>, SimpleSpan>,
  _: EmitterView<'_, 'inp, CalcLexer<'inp>, E>,
) -> Result<Spanned<Token, SimpleSpan>, CalcError> {
  let (span, left_tok) = left.into_components();
  let l = match left_tok { Token::Num(n) => n, _ => unreachable!() };
  let r = match right.into_data() { Token::Num(n) => n, _ => unreachable!() };
  // Associativity has already done its job in the engine; the fold just wants the operator.
  let (PrattInfix::Left(op) | PrattInfix::Right(op) | PrattInfix::Neither(op)) = infix.into_data();
  let value = match op {
    Token::Plus => l + r,
    Token::Minus => l - r,
    Token::Star => l * r,
    Token::Slash => l / r,
    Token::Caret => l.powf(r),
    _ => unreachable!("the RHS table admits only the five arithmetic infixes"),
  };
  Ok(Spanned::new(span, Token::Num(value)))
}

fn fold_postfix<'inp, E>(
  operand: Spanned<Token, SimpleSpan>,
  _close: Spanned<Token, SimpleSpan>,
  _: EmitterView<'_, 'inp, CalcLexer<'inp>, E>,
) -> Result<Spanned<Token, SimpleSpan>, CalcError> {
  Ok(operand) // `)` closed its group; the value flows on
}

// The folds are pure arithmetic over `Spanned<Token>`, so they run with no parser context: pick
// `E = ()` and lend its (empty) operations through an `EmitterView`. Building one needs a `&mut
// E` in hand, which is why it grants nothing a caller did not already have — and why a fold body
// stays unit-testable without standing up a parse.
let span = SimpleSpan::new(0, 0);
let sum = fold_infix::<()>(
  Spanned::new(span, Token::Num(2.0)),
  Spanned::new(span, Token::Num(3.0)),
  Spanned::new(span, PrattInfix::Left(Token::Star)),
  EmitterView::new(&mut ()),
).unwrap();
assert!(matches!(sum.into_data(), Token::Num(n) if n == 6.0));

let neg = fold_prefix::<()>(
  Spanned::new(span, Token::Minus),
  Spanned::new(span, Token::Num(2.0)),
  EmitterView::new(&mut ()),
).unwrap();
assert!(matches!(neg.into_data(), Token::Num(n) if n == -2.0));

let grouped = fold_postfix::<()>(
  Spanned::new(span, Token::Num(9.0)),
  Spanned::new(span, Token::RParen),
  EmitterView::new(&mut ()),
).unwrap();
assert!(matches!(grouped.into_data(), Token::Num(n) if n == 9.0));

Build calc_expr

calc_expr calls InputRef::pratt with the three folds, then unwraps the final Token::Num. The turbofish fixes the two type parameters the engine cannot infer: Expr = f64 (what an expression means) and Power (how tightly things bind). Its Ctx bounds add PrattEmitter to the ordinary Emitter bound because Pratt-specific diagnostics travel through the emitter too; a FatalContext satisfies both with no extra work. The five assertions below are the maintained evaluator’s behavior contract, now executable inline:

use tokora::{Token as TokenT, logos::{self, Logos}};
use tokora::EmitterView;
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { Self } }
#[derive(Debug, Clone, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Token {
  #[regex(r"[0-9]+(\.[0-9]+)?", |lex| lex.slice().parse::<f64>().map_err(|_| LexError))] Num(f64),
  #[token("+")] Plus, #[token("-")] Minus, #[token("*")] Star, #[token("/")] Slash,
  #[token("^")] Caret, #[token("(")] LParen, #[token(")")] RParen,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokenKind { Num, Plus, Minus, Star, Slash, Caret, LParen, RParen }
impl core::fmt::Display for TokenKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self { Self::Num => "number", Self::Plus => "+", Self::Minus => "-", Self::Star => "*", Self::Slash => "/", Self::Caret => "^", Self::LParen => "(", Self::RParen => ")" })
  }
}
impl core::fmt::Display for Token { fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { core::fmt::Display::fmt(&self.kind(), f) } }
impl From<&Token> for TokenKind {
  fn from(t: &Token) -> Self { match t { Token::Num(_) => Self::Num, Token::Plus => Self::Plus, Token::Minus => Self::Minus, Token::Star => Self::Star, Token::Slash => Self::Slash, Token::Caret => Self::Caret, Token::LParen => Self::LParen, Token::RParen => Self::RParen } }
}
impl TokenT<'_> for Token {
  type Kind = TokenKind; type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokenKind { TokenKind::from(self) }
  fn is_trivia(&self) -> bool { false }
}
type CalcLexer<'a> = tokora::lexer::LogosLexer<'a, Token>;
#[derive(Debug)]
enum CalcError { Lex(LexError), UnexpectedToken, UnexpectedEot }
impl From<LexError> for CalcError { fn from(e: LexError) -> Self { Self::Lex(e) } }
impl<'inp> From<tokora::error::token::UnexpectedTokenOf<'inp, CalcLexer<'inp>>> for CalcError { fn from(_: tokora::error::token::UnexpectedTokenOf<'inp, CalcLexer<'inp>>) -> Self { Self::UnexpectedToken } }
impl<O, Lang: ?Sized, Set: Clone + 'static> From<tokora::error::UnexpectedEot<O, Lang, Set>> for CalcError { fn from(_: tokora::error::UnexpectedEot<O, Lang, Set>) -> Self { Self::UnexpectedEot } }
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for CalcError { fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { Self::UnexpectedEot } }
impl From<tokora::error::UnexpectedEoLhs> for CalcError { fn from(_: tokora::error::UnexpectedEoLhs) -> Self { Self::UnexpectedEot } }
impl From<tokora::error::UnexpectedEoRhs> for CalcError { fn from(_: tokora::error::UnexpectedEoRhs) -> Self { Self::UnexpectedEot } }
impl<O, Lang: ?Sized> From<tokora::error::RecursionLimitReached<O, Lang>> for CalcError { fn from(_: tokora::error::RecursionLimitReached<O, Lang>) -> Self { Self::UnexpectedEot } }
impl<O, Lang: ?Sized> From<tokora::error::NonAssociativeChain<O, Lang>> for CalcError { fn from(_: tokora::error::NonAssociativeChain<O, Lang>) -> Self { Self::UnexpectedEot } }
use tokora::parser::PrattPower;
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, PartialOrd, Ord)]
struct Power(i32);
impl PrattPower for Power {}
const PREC_PAREN: Power = Power(-1);
const PREC_SUM: Power = Power(1);
const PREC_PROD: Power = Power(2);
const PREC_NEG: Power = Power(3);
const PREC_EXP: Power = Power(4);
use tokora::parser::{PrattInfix, PrattLHS, PrattRHS, Precedenced};
use tokora::token::PrattToken;
impl PrattToken<'_, f64, Power> for Token {
  fn try_pratt_lhs(&self) -> Option<PrattLHS<(), (), Power>> {
    Some(match self {
      Token::Num(_) => PrattLHS::Operand(()),
      Token::Minus => PrattLHS::Prefix(Precedenced::new((), PREC_NEG)),
      Token::LParen => PrattLHS::Prefix(Precedenced::new((), PREC_PAREN)),
      _ => return None,
    })
  }
  fn try_pratt_rhs(&self) -> Option<PrattRHS<(), (), (), (), Power>> {
    Some(match self {
      Token::Plus => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(()), PREC_SUM)),
      Token::Minus => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(()), PREC_SUM)),
      Token::Star => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(()), PREC_PROD)),
      Token::Slash => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(()), PREC_PROD)),
      Token::Caret => PrattRHS::Infix(Precedenced::new(PrattInfix::Right(()), PREC_EXP)),
      Token::RParen => PrattRHS::Postfix(Precedenced::new((), PREC_PAREN)),
      _ => return None,
    })
  }
}
use tokora::{SimpleSpan, span::Spanned};
fn fold_prefix<'inp, E>(op: Spanned<Token, SimpleSpan>, operand: Spanned<Token, SimpleSpan>, _: EmitterView<'_, 'inp, CalcLexer<'inp>, E>) -> Result<Spanned<Token, SimpleSpan>, CalcError> {
  let (span, op) = op.into_components();
  match op {
    Token::Minus => { let n = match operand.into_data() { Token::Num(n) => n, _ => unreachable!() }; Ok(Spanned::new(span, Token::Num(-n))) }
    Token::LParen => Ok(operand),
    _ => unreachable!(),
  }
}
fn fold_infix<'inp, E>(left: Spanned<Token, SimpleSpan>, right: Spanned<Token, SimpleSpan>, infix: Spanned<PrattInfix<Token, Token, Token>, SimpleSpan>, _: EmitterView<'_, 'inp, CalcLexer<'inp>, E>) -> Result<Spanned<Token, SimpleSpan>, CalcError> {
  let (span, left_tok) = left.into_components();
  let l = match left_tok { Token::Num(n) => n, _ => unreachable!() };
  let r = match right.into_data() { Token::Num(n) => n, _ => unreachable!() };
  let (PrattInfix::Left(op) | PrattInfix::Right(op) | PrattInfix::Neither(op)) = infix.into_data();
  let value = match op { Token::Plus => l + r, Token::Minus => l - r, Token::Star => l * r, Token::Slash => l / r, Token::Caret => l.powf(r), _ => unreachable!() };
  Ok(Spanned::new(span, Token::Num(value)))
}
fn fold_postfix<'inp, E>(operand: Spanned<Token, SimpleSpan>, _close: Spanned<Token, SimpleSpan>, _: EmitterView<'_, 'inp, CalcLexer<'inp>, E>) -> Result<Spanned<Token, SimpleSpan>, CalcError> { Ok(operand) }
use tokora::{Emitter, InputRef, Parse, ParseContext, Parser, emitter::PrattEmitter};

fn calc_expr<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<f64, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter:
    Emitter<'inp, CalcLexer<'inp>, Error = CalcError> + PrattEmitter<'inp, CalcLexer<'inp>>,
{
  let folded = inp.pratt::<_, _, _, f64, Power>(
    fold_prefix::<Ctx::Emitter>,
    fold_infix::<Ctx::Emitter>,
    fold_postfix::<Ctx::Emitter>,
  )?;
  // `Ok(None)` means the cursor was not looking at an expression at all.
  match folded {
    Some(tok) => match tok.into_data() {
      Token::Num(n) => Ok(n),
      _ => unreachable!(),
    },
    None => Err(CalcError::UnexpectedEot),
  }
}

let eval = |src| Parser::new().apply(calc_expr).parse_str(src);

// `^` folds through `f64::powf`, and Miri intentionally perturbs transcendental floats to catch
// code that assumes bit-exact results across platforms — so this table compares with a small
// epsilon instead of `assert_eq!`. The taught values themselves are unchanged.
fn assert_close(got: f64, want: f64) {
  assert!((got - want).abs() < 1e-9, "{got} != {want}");
}
assert_close(eval("1 + 2 * 3").unwrap(), 7.0);   // `*` binds tighter than `+`  → 1 + (2 * 3)
assert_close(eval("(1 + 2) * 3").unwrap(), 9.0); // grouping overrides          → (1 + 2) * 3
assert_close(eval("2 ^ 3 ^ 2").unwrap(), 512.0); // `^` is RIGHT-assoc          → 2 ^ (3 ^ 2)
assert_close(eval("-2 ^ 2").unwrap(), -4.0);     // `^` outranks unary `-`      → -(2 ^ 2)
assert_close(eval("10 / 2 / 5").unwrap(), 1.0);  // `/` is left-assoc           → (10 / 2) / 5

Reproduce the maintained assertion table

The assertions above are the maintained binary’s assertion table: precedence (1 + 2 * 3), parentheses, right-associative 2 ^ 3 ^ 2, unary minus versus exponentiation, and left-associative division. They are the behavior contract for the evaluator. For the full runnable program — the same code driven from a main that prints each result — run:

cargo run -p tokora --example calculator --features logos

You have now followed the complete calculator inline: a Logos lexer, a self-classifying PrattToken, three named folds, and a one-call calc_expr, evaluating real expressions to f64 with no AST allocated. Next: chapter 13.

13. Walkthrough: S-expressions

Prerequisites: chapters 2 and 11, plus familiarity with Box and Vec.

The maintained s_expression.rs uses manual recursive descent. It deliberately avoids Pratt parsing and combinators: each branch consumes exactly the tokens that its grammar form owns, and evaluation happens after parsing.

Define tokens and the AST/value types

The lexer owns keyword strings and produces integers, booleans, built-ins, parentheses, and a quote token. The output model distinguishes syntax (Expr) from evaluated values (Atom), while BuiltIn makes functions first-class values. The public parser APIs are Token, lexer::LogosLexer, InputRef::next, InputRef::try_expect, ParseContext, Emitter, Parser, and Parse::parse_str.

Implement atom and built-in branches in parse_expr

The parser consumes one token with next and immediately maps atom-like tokens into the AST. The reduced example keeps only numbers and lists, but has the same recursive-descent shape as the maintained program.

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { Self } }
#[derive(Clone, Debug, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Tok {
  #[regex(r"-?[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))] Int(i64),
  #[token("(")] Open,
  #[token(")")] Close,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
enum Kind { Int, Open, Close }
impl core::fmt::Display for Kind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self { Self::Int => "integer", Self::Open => "(", Self::Close => ")" })
  }
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { core::fmt::Display::fmt(&self.kind(), f) }
}
impl TokenT<'_> for Tok {
  type Kind = Kind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> Kind { match self { Self::Int(_) => Kind::Int, Self::Open => Kind::Open, Self::Close => Kind::Close } }
  fn is_trivia(&self) -> bool { false }
}
type SExprLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
#[derive(Debug, PartialEq)]
enum SExprError { Lex, Unexpected, End }
impl From<LexError> for SExprError { fn from(_: LexError) -> Self { Self::Lex } }
impl<O, Lang: ?Sized, Set: Clone + 'static> From<tokora::error::UnexpectedEot<O, Lang, Set>> for SExprError { fn from(_: tokora::error::UnexpectedEot<O, Lang, Set>) -> Self { Self::End } }
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for SExprError { fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { Self::End } }
impl<'inp> From<tokora::error::token::UnexpectedTokenOf<'inp, SExprLexer<'inp>>> for SExprError {
  fn from(_: tokora::error::token::UnexpectedTokenOf<'inp, SExprLexer<'inp>>) -> Self { Self::Unexpected }
}
use tokora::{Emitter, InputRef, Parse, ParseContext, Parser};

#[derive(Debug, PartialEq)]
enum Expr { Int(i64), List(Vec<Expr>) }

fn parse_expr<'inp, Ctx>(
  input: &mut InputRef<'inp, '_, SExprLexer<'inp>, Ctx>,
) -> Result<Expr, SExprError>
where
  Ctx: ParseContext<'inp, SExprLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, SExprLexer<'inp>, Error = SExprError>,
{
  match input.next()? {
    Some(token) => match token.into_data() {
      Tok::Int(value) => Ok(Expr::Int(value)),
      Tok::Open => Ok(Expr::List(parse_list(input)?)),
      Tok::Close => Err(SExprError::Unexpected),
    },
    None => Err(SExprError::End),
  }
}

fn parse_list<'inp, Ctx>(
  input: &mut InputRef<'inp, '_, SExprLexer<'inp>, Ctx>,
) -> Result<Vec<Expr>, SExprError>
where
  Ctx: ParseContext<'inp, SExprLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, SExprLexer<'inp>, Error = SExprError>,
{
  let mut values = Vec::new();
  while input.try_expect(|token| matches!(token.data(), Tok::Close))?.is_none() {
    values.push(parse_expr(input)?);
  }
  Ok(values)
}

let parsed = Parser::new().apply(parse_expr).parse_str("(1 (2 3))");
assert_eq!(parsed, Ok(Expr::List(vec![Expr::Int(1), Expr::List(vec![Expr::Int(2), Expr::Int(3)])])));

Implement quote and parenthesized branches

In the complete program, Quote requires an opening parenthesis and then delegates to parse_list. An opening parenthesis first probes for if; if it is present, the branch parses condition, then-expression, and optional else-expression. Otherwise it parses a function expression followed by its argument list. This is direct control flow, not speculative parser choice.

Implement parse_list, including the closing parenthesis

The loop’s try_expect is the important detail: it consumes ) when present and otherwise leaves the next token for parse_expr. The list parser is therefore responsible for both the empty list and the close delimiter; callers never consume a second close token.

Implement eval and apply

Parsing builds syntax; evaluation reduces it. That boundary keeps parser errors separate from runtime errors such as division by zero or applying a non-function.

#[derive(Clone, Debug, PartialEq)]
enum BuiltIn { Add, Not }
#[derive(Clone, Debug, PartialEq)]
enum Atom { Number(i64), Bool(bool), Function(BuiltIn) }
enum Expr {
  Constant(Atom),
  If { condition: Box<Expr>, then: Box<Expr>, otherwise: Option<Box<Expr>> },
  Application(Box<Expr>, Vec<Expr>),
}

fn apply(function: BuiltIn, arguments: Vec<Atom>) -> Result<Atom, String> {
  match function {
    BuiltIn::Add => arguments.into_iter().try_fold(0_i64, |sum, value| match value {
      Atom::Number(value) => Ok(sum + value),
      other => Err(format!("expected number, got {other:?}")),
    }).map(Atom::Number),
    BuiltIn::Not => match arguments.as_slice() {
      [Atom::Bool(value)] => Ok(Atom::Bool(!value)),
      _ => Err("not expects one boolean".into()),
    },
  }
}

fn eval(expr: Expr) -> Result<Atom, String> {
  match expr {
    Expr::Constant(atom) => Ok(atom),
    Expr::If { condition, then, otherwise } => match eval(*condition)? {
      Atom::Bool(true) => eval(*then),
      Atom::Bool(false) => otherwise.map(|expr| eval(*expr)).unwrap_or(Ok(Atom::Bool(false))),
      _ => Err("if condition must be boolean".into()),
    },
    Expr::Application(function, arguments) => match eval(*function)? {
      Atom::Function(function) => apply(function, arguments.into_iter().map(eval).collect::<Result<_, _>>()?),
      _ => Err("application target is not a function".into()),
    },
  }
}

let expr = Expr::Application(
  Box::new(Expr::Constant(Atom::Function(BuiltIn::Add))),
  vec![Expr::Constant(Atom::Number(1)), Expr::Constant(Atom::Number(2))],
);
assert_eq!(eval(expr), Ok(Atom::Number(3)));

Exercise the maintained forms

Run s_expression.rs to exercise literals, built-ins, conditionals, applications, and quoted lists:

cargo run -p tokora --example s_expression --features logos

The result is a complete recursive-descent parser/interpreter with no Pratt or combinator machinery. Next: chapter 14.

14. Walkthrough: JSON

Prerequisites: chapters 3, 4, and 11.

The maintained json.rs combines borrowed scalar values with allocated Vec nodes for arrays and objects. It is not a zero-allocation parser: string slices borrow from the input, while collection structure is owned. The maintained sample.json supplies the end-to-end input.

Define borrowed tokens, JsonError, punctuator mappings, and JsonValue

Token<'inp>::String(&'inp str) borrows from the source. JsonValue<'inp> preserves that borrow for scalars, but List(Vec<JsonValue<'inp>>) and Object(Vec<(&'inp str, JsonValue<'inp>)>) allocate their container nodes. The lexer maps punctuation through PunctuatorToken so the delimiter types can remain generic.

Strings arrive as two token variants, String and EscapedString, both reporting TokenKind::String. The split is what spelling RFC 8259’s surrogate-pairing rule as a regex — a \uXXXX escape is admissible only outside D800–DFFF, or as a high half immediately followed by a \u low half — buys for nothing: the two rules are disjoint (one forbids a backslash, the other requires one), so the automaton is already deciding which applies. A consumer that needs the decoded value therefore knows from the variant whether the slice can be used as it borrows or has to be unescaped into an owned String. Validating the escape in a callback instead measured 1.65× the shipped rule over the bundled sample.json; the folded regex measures 1.02–1.07×.

use tokora::{
  Token as TokenT,
  punct::{Brace, Bracket, Colon, Comma},
  token::PunctuatorToken,
};

#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
enum Kind { BraceOpen, BraceClose, BracketOpen, BracketClose, Colon, Comma }
impl core::fmt::Display for Kind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Self::BraceOpen => "{", Self::BraceClose => "}", Self::BracketOpen => "[",
      Self::BracketClose => "]", Self::Colon => ":", Self::Comma => ",",
    })
  }
}
#[derive(Clone, Debug)]
enum Token { BraceOpen, BraceClose, BracketOpen, BracketClose, Colon, Comma }
impl core::fmt::Display for Token {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { core::fmt::Display::fmt(&self.kind(), f) }
}
impl TokenT<'_> for Token {
  type Kind = Kind;
  type Error = ();
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> Kind {
    match self {
      Self::BraceOpen => Kind::BraceOpen, Self::BraceClose => Kind::BraceClose,
      Self::BracketOpen => Kind::BracketOpen, Self::BracketClose => Kind::BracketClose,
      Self::Colon => Kind::Colon, Self::Comma => Kind::Comma,
    }
  }
  fn is_trivia(&self) -> bool { false }
}
impl PunctuatorToken<'_> for Token {
  fn comma() -> Option<Kind> { Some(Kind::Comma) }
  fn colon() -> Option<Kind> { Some(Kind::Colon) }
  fn open_brace() -> Option<Kind> { Some(Kind::BraceOpen) }
  fn close_brace() -> Option<Kind> { Some(Kind::BraceClose) }
  fn open_bracket() -> Option<Kind> { Some(Kind::BracketOpen) }
  fn close_bracket() -> Option<Kind> { Some(Kind::BracketClose) }
}
impl From<Comma> for Kind { fn from(_: Comma) -> Self { Self::Comma } }
impl From<Colon> for Kind { fn from(_: Colon) -> Self { Self::Colon } }
impl From<Brace> for Kind { fn from(_: Brace) -> Self { Self::BraceOpen } }
impl From<Bracket> for Kind { fn from(_: Bracket) -> Self { Self::BracketOpen } }

assert_eq!(<Token as PunctuatorToken>::comma(), Some(Kind::Comma));
assert_eq!(<Token as PunctuatorToken>::open_bracket(), Some(Kind::BracketOpen));

The public surface includes token::PunctuatorToken, parser::expect, ParseInput::map, ParseInput::ignored, ParseInput::then, ParseInput::then_ignore, ParseInput::separated_by_comma_while, TryParseInput::separated_by_comma, Accumulator::collect, punct::Colon, punct::Bracket, punct::Brace, ParseChoice::peek_then_choice, and ParseChoice::peek_then_try_choice.

Build boolean, null, number, and string

Each scalar uses expect to state the accepted token kind. map extracts a Boolean, number, or borrowed string; ignored turns null into unit. These small parsers stay mandatory once the value dispatcher has selected them, which gives invalid scalar starts a precise expected-kind diagnostic.

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { Self } }
#[derive(Clone, Debug, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Token { #[token("true", |_| true)] Bool(bool), #[token("null")] Null }
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
enum Kind { Bool, Null }
impl core::fmt::Display for Kind { fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { f.write_str(match self { Self::Bool => "bool", Self::Null => "null" }) } }
impl core::fmt::Display for Token { fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { core::fmt::Display::fmt(&self.kind(), f) } }
impl TokenT<'_> for Token {
  type Kind = Kind; type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> Kind { match self { Self::Bool(_) => Kind::Bool, Self::Null => Kind::Null } }
  fn is_trivia(&self) -> bool { false }
}
type JsonLexer<'a> = tokora::lexer::LogosLexer<'a, Token>;
#[derive(Debug, PartialEq)]
enum JsonError { Lex, Unexpected, End }
impl From<LexError> for JsonError { fn from(_: LexError) -> Self { Self::Lex } }
impl<'inp> From<tokora::error::token::UnexpectedTokenOf<'inp, JsonLexer<'inp>>> for JsonError {
  fn from(_: tokora::error::token::UnexpectedTokenOf<'inp, JsonLexer<'inp>>) -> Self { Self::Unexpected }
}
impl<H, O, Lang: ?Sized, Set: Clone + 'static> From<tokora::error::UnexpectedEnd<H, O, Lang, Set>> for JsonError {
  fn from(_: tokora::error::UnexpectedEnd<H, O, Lang, Set>) -> Self { Self::End }
}
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for JsonError {
  fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { Self::End }
}
use tokora::{Emitter, InputRef, Parse, ParseContext, ParseInput, Parser, parser::expect, utils::Expected};

fn boolean<'inp, Ctx>(
  input: &mut InputRef<'inp, '_, JsonLexer<'inp>, Ctx>,
) -> Result<bool, JsonError>
where
  Ctx: ParseContext<'inp, JsonLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, JsonLexer<'inp>, Error = JsonError>,
{
  expect(|token: &Token| if matches!(token, Token::Bool(_)) {
    Ok(())
  } else {
    Err(Expected::one(Kind::Bool))
  })
  .map(|token| match token { Token::Bool(value) => value, Token::Null => unreachable!() })
  .parse_input(input)
}

fn null<'inp, Ctx>(
  input: &mut InputRef<'inp, '_, JsonLexer<'inp>, Ctx>,
) -> Result<(), JsonError>
where
  Ctx: ParseContext<'inp, JsonLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, JsonLexer<'inp>, Error = JsonError>,
{
  expect(|token: &Token| if matches!(token, Token::Null) {
    Ok(())
  } else {
    Err(Expected::one(Kind::Null))
  })
  .ignored()
  .parse_input(input)
}

fn true_then_null<'inp, Ctx>(
  input: &mut InputRef<'inp, '_, JsonLexer<'inp>, Ctx>,
) -> Result<bool, JsonError>
where
  Ctx: ParseContext<'inp, JsonLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, JsonLexer<'inp>, Error = JsonError>,
{
  boolean.then_ignore(null).parse_input(input)
}

assert_eq!(Parser::new().apply(true_then_null).parse_str("true null"), Ok(true));

Build arrays with tentative values, comma separation, delimiters, and collection

list passes try_json_value to separated_by_comma, collects accepted values into a Vec, and sequences the bracket parsers around that comma-separated core. A tentative element must decline without consuming when the closer is next; malformed separators remain errors. This focused parser follows the same comma-separated array path and makes those failures executable. For the ready-made single-region alternative to this bracket hand-roll, see the brackets free shape (and its parens/braces/angles/delimited siblings):

use tokora::{
  Accumulator, Emitter, InputRef, Parse, ParseContext, ParseInput, Parser, Token as TokenT,
  TryParseInput,
  emitter::{FullContainerEmitter, SeparatedEmitter, UnexpectedLeadingSeparatorEmitter, UnexpectedTrailingSeparatorEmitter},
  error::{
    UnexpectedEot,
    syntax::{FullContainer, MissingSyntaxOf},
    token::{MissingTokenOf, SeparatedErrorOf, UnexpectedTokenOf},
  },
  logos::{self, Logos},
  punct::{CloseBracket, Comma, OpenBracket},
  token::PunctuatorToken,
  try_parse_input::ParseAttempt,
};
#[derive(Clone, Debug, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+")]
enum JsonToken {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<u64>().map_err(|_| ()))] Number(u64),
  #[token("[")] OpenBracket,
  #[token("]")] CloseBracket,
  #[token(",")] Comma,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
enum JsonKind { Number, OpenBracket, CloseBracket, Comma }
impl core::fmt::Display for JsonKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self { Self::Number => "number", Self::OpenBracket => "[", Self::CloseBracket => "]", Self::Comma => "," })
  }
}
impl core::fmt::Display for JsonToken {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { core::fmt::Display::fmt(&self.kind(), f) }
}
impl TokenT<'_> for JsonToken {
  type Kind = JsonKind;
  type Error = ();
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> JsonKind {
    match self { Self::Number(_) => JsonKind::Number, Self::OpenBracket => JsonKind::OpenBracket, Self::CloseBracket => JsonKind::CloseBracket, Self::Comma => JsonKind::Comma }
  }
  fn is_trivia(&self) -> bool { false }
}
impl PunctuatorToken<'_> for JsonToken {
  fn comma() -> Option<JsonKind> { Some(JsonKind::Comma) }
  fn open_bracket() -> Option<JsonKind> { Some(JsonKind::OpenBracket) }
  fn close_bracket() -> Option<JsonKind> { Some(JsonKind::CloseBracket) }
}
impl From<Comma> for JsonKind { fn from(_: Comma) -> Self { Self::Comma } }
impl From<OpenBracket> for JsonKind { fn from(_: OpenBracket) -> Self { Self::OpenBracket } }
impl From<CloseBracket> for JsonKind { fn from(_: CloseBracket) -> Self { Self::CloseBracket } }
type JsonLexer<'a> = tokora::lexer::LogosLexer<'a, JsonToken>;
#[derive(Debug, PartialEq)]
enum JsonError { Lex, Unexpected, End, Missing, Separator, Full }
impl From<()> for JsonError { fn from(_: ()) -> Self { Self::Lex } }
impl<'inp> From<UnexpectedTokenOf<'inp, JsonLexer<'inp>>> for JsonError {
  fn from(_: UnexpectedTokenOf<'inp, JsonLexer<'inp>>) -> Self { Self::Unexpected }
}
impl<O, Lang: ?Sized, Set: Clone + 'static> From<UnexpectedEot<O, Lang, Set>> for JsonError {
  fn from(_: UnexpectedEot<O, Lang, Set>) -> Self { Self::End }
}
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for JsonError {
  fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { Self::End }
}
impl<'inp> From<MissingTokenOf<'inp, JsonLexer<'inp>>> for JsonError {
  fn from(_: MissingTokenOf<'inp, JsonLexer<'inp>>) -> Self { Self::Missing }
}
impl<'inp> From<MissingSyntaxOf<'inp, JsonLexer<'inp>>> for JsonError {
  fn from(_: MissingSyntaxOf<'inp, JsonLexer<'inp>>) -> Self { Self::Missing }
}
impl<'inp> From<SeparatedErrorOf<'inp, JsonLexer<'inp>>> for JsonError {
  fn from(_: SeparatedErrorOf<'inp, JsonLexer<'inp>>) -> Self { Self::Separator }
}
impl<S, Lang: ?Sized> From<FullContainer<S, Lang>> for JsonError {
  fn from(_: FullContainer<S, Lang>) -> Self { Self::Full }
}
fn try_number<'inp, Ctx>(
  input: &mut InputRef<'inp, '_, JsonLexer<'inp>, Ctx>,
) -> Result<ParseAttempt<u64>, JsonError>
where
  Ctx: ParseContext<'inp, JsonLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, JsonLexer<'inp>, Error = JsonError>,
{
  Ok(match input.try_expect(|token| matches!(token.data(), JsonToken::Number(_)))? {
    Some(token) => match token.into_data() { JsonToken::Number(value) => ParseAttempt::Accept(value), _ => unreachable!() },
    None => ParseAttempt::Decline,
  })
}
fn array<'inp, Ctx>(
  input: &mut InputRef<'inp, '_, JsonLexer<'inp>, Ctx>,
) -> Result<Vec<u64>, JsonError>
where
  Ctx: ParseContext<'inp, JsonLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, JsonLexer<'inp>, Error = JsonError>
    + SeparatedEmitter<'inp, JsonLexer<'inp>>
    + FullContainerEmitter<'inp, JsonLexer<'inp>>
    + UnexpectedLeadingSeparatorEmitter<'inp, JsonLexer<'inp>>
    + UnexpectedTrailingSeparatorEmitter<'inp, JsonLexer<'inp>>,
{
  OpenBracket::parse
    .ignore_then(try_number.separated_by_comma().collect())
    .then_ignore(CloseBracket::parse)
    .parse_input(input)
}
assert_eq!(Parser::new().apply(array).parse_str("[1,2]"), Ok(vec![1, 2]));
assert!(Parser::new().apply(array).parse_str("[1,,2]").is_err());
assert!(Parser::new().apply(array).parse_str("[1,]").is_err());

Build fields and objects with separated_by_comma_while

A field is string.then_ignore(Colon::parse).then(json_value). object uses separated_by_comma_while with JsonValue::decide, an external continuation decision over a one-token cache::Peeked window. The decision returns parser::Action to stop before a closing brace or continue for another field.

Implement tentative try_json_value

The six value branches are selected with ParseChoice::peek_then_try_choice. The chooser returns Ok(None) for a token that cannot begin a JSON value, which becomes a try_parse_input::ParseAttempt decline. That is the right behavior for an array element when the next token is a delimiter.

Implement committed json_value with an expected-kind diagnostic

json_value uses ParseChoice::peek_then_choice instead. Its chooser maps a valid start to a Branch and constructs an unexpected-token error with utils::Expected for every other token. The difference is semantic: tentative choice says not an element here; committed choice says a JSON value is required here.

Parse sample.json and test separators

Run the maintained parser against its bundled source, then retain focused malformed-separator tests beside any extension:

cargo run -p tokora --example json --features logos

You can now reproduce the maintained JSON parser while knowing exactly where borrowed values end, collection allocation begins, and tentative choice is required. Next: chapter 15.

15. Walkthrough: C expressions

Prerequisites: chapters 5 and 11; chapter 12 is helpful.

This walkthrough builds the maintained c_expression.rs end to end, inline. Where chapter 12 folded tokens straight to an f64 with the token-level Pratt engine, this is the AST-level engine (pratt): parse_lhs and parse_rhs are full parser functions with the InputRef in hand, and the folds build a typed Expr tree over your own node type. Three things here are unique to this chapter, worth watching for as they appear below:

• The classifiers are parsers, not a token trait. parse_lhs/parse_rhs return PrattLHS/PrattRHS values; a non-operator is PrattRHS::End, not a None.
• Postfix operators that consume more input. fold_postfix receives the InputRef first, so [i], (args...), and ? t : f read the tokens they need — the token-level postfix fold could not.
• C’s deep precedence ladder, with the ternary as a low-precedence postfix.

Every part — lexer, AST types, precedence ladder, the two classifiers, the three folds, and the one-call entry point — is shown as a compiling doctest, so you can follow the whole parser without leaving the page.

This chapter’s public surface is parser::pratt, parser::PrattLHS, parser::PrattRHS, parser::PrattInfix, parser::Precedenced, parser::PrattPower, InputRef::next, InputRef::try_expect, ParseInput::parse_input, and Parse::parse_str. The Pratt reference catalogs the whole surface, token-level and AST-level, side by side.

Define the lexer, token kinds, and CExprError

The Logos lexer carries payloads in Token::Num(i64) and Token::Ident(String); the remaining variants are punctuation. Multi-character operators (==, <<, ++, …) are listed before their single-character prefixes so Logos’ longest-match rule tokenizes == as one token, not two. As in the calculator, classification lives on a separate fieldless TokenKind (so it can be Copy + Eq + Hash, as Token requires of its Kind), and Display on the kind doubles as the diagnostic name.

The error family is simpler than the calculator’s: AST-level Pratt does not route expression-end errors through a PrattEmitter (parse_lhs reports “ran out of operand” itself, as you will see) — an ordinary From impl is enough, where the token-level engine needs the extra capability trait. That does not make From<UnexpectedEoLhs> and From<UnexpectedEoRhs> optional, though: the Pratt engine raises them itself, marked terminal, when parse_lhs/parse_rhs breaks its own contract — reports an operator and consumes nothing, a grammar bug rather than the ordinary end of input parse_lhs already reports as UnexpectedEot. CExprError carries both conversions below, alongside the lexical error and the unexpected token.

use tokora::{
  Token as TokenT,
  error::{
    NonAssociativeChain, RecursionLimitReached, UnexpectedEoLhs, UnexpectedEoRhs,
    token::UnexpectedTokenOf,
  },
  logos::{self, Logos},
};

#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { Self } }

#[derive(Clone, Debug, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Token {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))]
  Num(i64),
  #[regex(r"[a-zA-Z_][a-zA-Z0-9_]*", |lex| lex.slice().to_string())]
  Ident(String),
  // Multi-character operators — before the single-char variants (longest match wins).
  #[token("++")] PlusPlus,
  #[token("--")] MinusMinus,
  #[token("==")] EqEq,
  #[token("!=")] BangEq,
  #[token("<=")] LtEq,
  #[token(">=")] GtEq,
  #[token("&&")] AmpAmp,
  #[token("||")] PipePipe,
  #[token("<<")] Shl,
  #[token(">>")] Shr,
  // Single-character operators.
  #[token("+")] Plus,
  #[token("-")] Minus,
  #[token("*")] Star,
  #[token("/")] Slash,
  #[token("%")] Percent,
  #[token("&")] Amp,
  #[token("|")] Pipe,
  #[token("^")] Caret,
  #[token("~")] Tilde,
  #[token("!")] Bang,
  #[token("?")] Question,
  #[token(":")] Colon,
  #[token("<")] Lt,
  #[token(">")] Gt,
  #[token(",")] Comma,
  // Delimiters.
  #[token("(")] LParen,
  #[token(")")] RParen,
  #[token("[")] LBracket,
  #[token("]")] RBracket,
}

#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
enum TokenKind {
  Num, Ident, PlusPlus, MinusMinus, EqEq, BangEq, LtEq, GtEq, AmpAmp, PipePipe, Shl, Shr, Plus,
  Minus, Star, Slash, Percent, Amp, Pipe, Caret, Tilde, Bang, Question, Colon, Lt, Gt, Comma,
  LParen, RParen, LBracket, RBracket,
}

impl core::fmt::Display for TokenKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Self::Num => "number", Self::Ident => "identifier", Self::PlusPlus => "++",
      Self::MinusMinus => "--", Self::EqEq => "==", Self::BangEq => "!=", Self::LtEq => "<=",
      Self::GtEq => ">=", Self::AmpAmp => "&&", Self::PipePipe => "||", Self::Shl => "<<",
      Self::Shr => ">>", Self::Plus => "+", Self::Minus => "-", Self::Star => "*",
      Self::Slash => "/", Self::Percent => "%", Self::Amp => "&", Self::Pipe => "|",
      Self::Caret => "^", Self::Tilde => "~", Self::Bang => "!", Self::Question => "?",
      Self::Colon => ":", Self::Lt => "<", Self::Gt => ">", Self::Comma => ",",
      Self::LParen => "(", Self::RParen => ")", Self::LBracket => "[", Self::RBracket => "]",
    })
  }
}
impl core::fmt::Display for Token {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    core::fmt::Display::fmt(&self.kind(), f)
  }
}

impl From<&Token> for TokenKind {
  fn from(t: &Token) -> Self {
    match t {
      Token::Num(_) => Self::Num, Token::Ident(_) => Self::Ident, Token::PlusPlus => Self::PlusPlus,
      Token::MinusMinus => Self::MinusMinus, Token::EqEq => Self::EqEq, Token::BangEq => Self::BangEq,
      Token::LtEq => Self::LtEq, Token::GtEq => Self::GtEq, Token::AmpAmp => Self::AmpAmp,
      Token::PipePipe => Self::PipePipe, Token::Shl => Self::Shl, Token::Shr => Self::Shr,
      Token::Plus => Self::Plus, Token::Minus => Self::Minus, Token::Star => Self::Star,
      Token::Slash => Self::Slash, Token::Percent => Self::Percent, Token::Amp => Self::Amp,
      Token::Pipe => Self::Pipe, Token::Caret => Self::Caret, Token::Tilde => Self::Tilde,
      Token::Bang => Self::Bang, Token::Question => Self::Question, Token::Colon => Self::Colon,
      Token::Lt => Self::Lt, Token::Gt => Self::Gt, Token::Comma => Self::Comma,
      Token::LParen => Self::LParen, Token::RParen => Self::RParen, Token::LBracket => Self::LBracket,
      Token::RBracket => Self::RBracket,
    }
  }
}

impl TokenT<'_> for Token {
  type Kind = TokenKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokenKind { TokenKind::from(self) }
  fn is_trivia(&self) -> bool { false }
}

type CExprLexer<'a> = tokora::lexer::LogosLexer<'a, Token>;

#[derive(Debug)]
enum CExprError { Lex(LexError), UnexpectedToken, UnexpectedEot }

impl From<LexError> for CExprError { fn from(e: LexError) -> Self { Self::Lex(e) } }
impl<'inp> From<UnexpectedTokenOf<'inp, CExprLexer<'inp>>> for CExprError {
  fn from(_: UnexpectedTokenOf<'inp, CExprLexer<'inp>>) -> Self { Self::UnexpectedToken }
}
// The Pratt engine's own terminal exits — required by `Pratt`'s `ParseInput` impl regardless of
// whether this grammar ever trips them. Both name a contract violation in `parse_lhs`/`parse_rhs`
// (an operator reported but not consumed), never ordinary operand exhaustion, so both fold into
// the same `UnexpectedEot` this parser already reports for that.
impl<O, Lang: ?Sized, Set: Clone + 'static> From<UnexpectedEoLhs<O, Lang, Set>> for CExprError {
  fn from(_: UnexpectedEoLhs<O, Lang, Set>) -> Self { Self::UnexpectedEot }
}
impl<O, Lang: ?Sized, Set: Clone + 'static> From<UnexpectedEoRhs<O, Lang, Set>> for CExprError {
  fn from(_: UnexpectedEoRhs<O, Lang, Set>) -> Self { Self::UnexpectedEot }
}
// The two the engines RETURN rather than emit: a recursion-limit trip at a frame prologue, and a
// second same-power non-associative operator. No `emit_*` hook sees either, so neither is part of
// the `FromPrattError` bundle — the pratt entry points name these two `From`s themselves.
impl<O, Lang: ?Sized> From<RecursionLimitReached<O, Lang>> for CExprError {
  fn from(_: RecursionLimitReached<O, Lang>) -> Self { Self::UnexpectedEot }
}
impl<O, Lang: ?Sized> From<NonAssociativeChain<O, Lang>> for CExprError {
  fn from(_: NonAssociativeChain<O, Lang>) -> Self { Self::UnexpectedEot }
}

assert_eq!(Token::Star.kind(), TokenKind::Star);
assert_eq!(Token::PlusPlus.kind(), TokenKind::PlusPlus);
assert_eq!(Token::Star.to_string(), "*");

Define UnaryOp, BinOp, PostfixOp, and Expr

The AST has typed variants for prefix, binary, postfix increment/decrement, index, call, and ternary expressions. Separating operator tags from tree nodes lets the folds stay small and keeps the Display implementation useful as an assertion oracle: every node prints fully parenthesised, so to_string() is a compact, exact check on the tree’s shape. PostfixOp is the tag parse_rhs hands to fold_postfix; its Index/Call/Ternary variants are instructions to the fold to consume more input. There is no “not an operator” tag: that answer is PrattRHS::End.

#[derive(Clone, Copy, Debug)]
enum UnaryOp { Neg, Pos, Not, BNot, PreInc, PreDec }

#[derive(Clone, Copy, Debug)]
enum BinOp {
  Add, Sub, Mul, Div, Mod, Or, And, BOr, BXor, BAnd, Eq, Neq, Lt, Gt, Lte, Gte, Shl, Shr,
}

// The tag `parse_rhs` passes to `fold_postfix`. `Index`/`Call`/`Ternary` tell the fold to consume
// more input. Every variant here is a real operator.
#[derive(Clone, Copy, Debug)]
enum PostfixOp { Inc, Dec, Index, Call, Ternary }

#[derive(Clone, Debug)]
enum Expr {
  Num(i64),
  Var(String),
  Prefix { op: UnaryOp, operand: Box<Expr> },
  Binary { op: BinOp, left: Box<Expr>, right: Box<Expr> },
  PostfixInc(Box<Expr>),
  PostfixDec(Box<Expr>),
  Index { base: Box<Expr>, index: Box<Expr> },
  Call { func: Box<Expr>, args: Vec<Expr> },
  Ternary { cond: Box<Expr>, then: Box<Expr>, otherwise: Box<Expr> },
}

impl core::fmt::Display for UnaryOp {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      UnaryOp::Neg => "-", UnaryOp::Pos => "+", UnaryOp::Not => "!", UnaryOp::BNot => "~",
      UnaryOp::PreInc => "++", UnaryOp::PreDec => "--",
    })
  }
}
impl core::fmt::Display for BinOp {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      BinOp::Add => "+", BinOp::Sub => "-", BinOp::Mul => "*", BinOp::Div => "/", BinOp::Mod => "%",
      BinOp::Or => "||", BinOp::And => "&&", BinOp::BOr => "|", BinOp::BXor => "^", BinOp::BAnd => "&",
      BinOp::Eq => "==", BinOp::Neq => "!=", BinOp::Lt => "<", BinOp::Gt => ">", BinOp::Lte => "<=",
      BinOp::Gte => ">=", BinOp::Shl => "<<", BinOp::Shr => ">>",
    })
  }
}
impl core::fmt::Display for Expr {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    match self {
      Expr::Num(n) => write!(f, "{n}"),
      Expr::Var(s) => write!(f, "{s}"),
      Expr::Prefix { op, operand } => write!(f, "({op}{operand})"),
      Expr::Binary { op, left, right } => write!(f, "({left} {op} {right})"),
      Expr::PostfixInc(e) => write!(f, "({e}++)"),
      Expr::PostfixDec(e) => write!(f, "({e}--)"),
      Expr::Index { base, index } => write!(f, "({base}[{index}])"),
      Expr::Ternary { cond, then, otherwise } => write!(f, "({cond} ? {then} : {otherwise})"),
      Expr::Call { func, args } => {
        write!(f, "{func}(")?;
        for (i, a) in args.iter().enumerate() {
          if i > 0 { write!(f, ", ")?; }
          write!(f, "{a}")?;
        }
        write!(f, ")")
      }
    }
  }
}

// The Display impl is the assertion oracle used throughout this chapter.
let two_times_three = Expr::Binary {
  op: BinOp::Mul,
  left: Box::new(Expr::Num(2)),
  right: Box::new(Expr::Num(3)),
};
assert_eq!(two_times_three.to_string(), "(2 * 3)");
assert_eq!(
  Expr::Prefix { op: UnaryOp::Neg, operand: Box::new(Expr::Var("a".to_string())) }.to_string(),
  "(-a)",
);
assert_eq!(
  Expr::Call {
    func: Box::new(Expr::Var("f".to_string())),
    args: vec![Expr::Num(1), Expr::Num(2)],
  }
  .to_string(),
  "f(1, 2)",
);

Define the precedence ladder

The ladder runs from the ternary through the logical and bitwise operators, comparison, shifts, arithmetic, prefix, and the high-power postfix forms. The precise numeric values matter only relative to one another; named constants make that relationship auditable. One row carries C-specific character: the ternary is a postfix operator, but a very low-precedence one (a || b ? c : d parses as (a || b) ? c : d). Every constant here is a real operator — “not an operator” is PrattRHS::End, which is not a power and needs no room below the floor.

use tokora::parser::PrattPower;

#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord)]
struct Power(i32);
// Nothing to implement: a binding power is only ever compared, never stepped.
impl PrattPower for Power {}

const PREC_TERNARY: Power = Power(2);  // ?:        (postfix, low precedence)
const PREC_OR: Power = Power(3);       // ||
const PREC_AND: Power = Power(4);      // &&
const PREC_BOR: Power = Power(5);      // |
const PREC_BXOR: Power = Power(6);     // ^
const PREC_BAND: Power = Power(7);     // &
const PREC_EQ: Power = Power(8);       // == !=
const PREC_CMP: Power = Power(9);      // < > <= >=
const PREC_SHIFT: Power = Power(10);   // << >>
const PREC_ADD: Power = Power(11);     // + -
const PREC_MUL: Power = Power(12);     // * / %
const PREC_PREFIX: Power = Power(13);  // unary - + ! ~ ++ --
const PREC_POSTFIX: Power = Power(14); // ++ -- [] ()  (postfix)

// Every operator outranks the default floor a top-level parse starts at.
assert!(PREC_TERNARY > Power::default());
// The whole ladder is one strictly-increasing chain; only the relative order matters.
assert!(
  PREC_TERNARY < PREC_OR && PREC_OR < PREC_AND && PREC_AND < PREC_BOR && PREC_BOR < PREC_BXOR
    && PREC_BXOR < PREC_BAND && PREC_BAND < PREC_EQ && PREC_EQ < PREC_CMP && PREC_CMP < PREC_SHIFT
    && PREC_SHIFT < PREC_ADD && PREC_ADD < PREC_MUL && PREC_MUL < PREC_PREFIX
    && PREC_PREFIX < PREC_POSTFIX
);
// Left-associativity means the engine recurses admitting only powers strictly above the
// operator's own, so `*` binds inside a `+`'s right operand and a second `+` does not.
assert!(PREC_MUL > PREC_ADD);

Implement parse_lhs and parse_rhs

parse_lhs reads the left edge of a (sub-)expression: an operand (number or identifier), a parenthesised group, or a prefix operator (-, +, !, ~, ++, --). Unlike the calculator’s PrattToken::try_pratt_lhs — a pure classifier on the token — this is a full parser function with the InputRef in hand, which is exactly what lets the ( arm recurse into parse_cexpr and then consume its own ) with try_expect. Running out of input here is this function’s error to report (CExprError::UnexpectedEot); the AST engine does not synthesize one.

parse_rhs classifies what follows an operand: an infix operator (mapped to a left-associative PrattInfix), a postfix trigger (++, --, [, (, ?), or — for anything else — PrattRHS::End. On End the Pratt engine restores whatever parse_rhs consumed, leaving that token on the input for the surrounding grammar (a ), ], :, or , closing an enclosing form) instead of losing it. This is the AST-level counterpart of a token-level try_pratt_rhs returning None: parse_rhs must always return a value, and End is the value that means “the expression stops here”.

use tokora::{Token as TokenT, ParseInput, error::token::UnexpectedTokenOf, logos::{self, Logos}, parser::{PrattPower, pratt}};
#[derive(Clone, Debug, Default, PartialEq)] struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { Self } }
#[derive(Clone, Debug, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Token {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))] Num(i64),
  #[regex(r"[a-zA-Z_][a-zA-Z0-9_]*", |lex| lex.slice().to_string())] Ident(String),
  #[token("++")] PlusPlus, #[token("--")] MinusMinus, #[token("==")] EqEq, #[token("!=")] BangEq,
  #[token("<=")] LtEq, #[token(">=")] GtEq, #[token("&&")] AmpAmp, #[token("||")] PipePipe,
  #[token("<<")] Shl, #[token(">>")] Shr, #[token("+")] Plus, #[token("-")] Minus,
  #[token("*")] Star, #[token("/")] Slash, #[token("%")] Percent, #[token("&")] Amp,
  #[token("|")] Pipe, #[token("^")] Caret, #[token("~")] Tilde, #[token("!")] Bang,
  #[token("?")] Question, #[token(":")] Colon, #[token("<")] Lt, #[token(">")] Gt,
  #[token(",")] Comma, #[token("(")] LParen, #[token(")")] RParen, #[token("[")] LBracket,
  #[token("]")] RBracket,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
enum TokenKind { Num, Ident, PlusPlus, MinusMinus, EqEq, BangEq, LtEq, GtEq, AmpAmp, PipePipe, Shl, Shr, Plus, Minus, Star, Slash, Percent, Amp, Pipe, Caret, Tilde, Bang, Question, Colon, Lt, Gt, Comma, LParen, RParen, LBracket, RBracket }
impl core::fmt::Display for TokenKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self { Self::Num => "number", Self::Ident => "identifier", Self::PlusPlus => "++", Self::MinusMinus => "--", Self::EqEq => "==", Self::BangEq => "!=", Self::LtEq => "<=", Self::GtEq => ">=", Self::AmpAmp => "&&", Self::PipePipe => "||", Self::Shl => "<<", Self::Shr => ">>", Self::Plus => "+", Self::Minus => "-", Self::Star => "*", Self::Slash => "/", Self::Percent => "%", Self::Amp => "&", Self::Pipe => "|", Self::Caret => "^", Self::Tilde => "~", Self::Bang => "!", Self::Question => "?", Self::Colon => ":", Self::Lt => "<", Self::Gt => ">", Self::Comma => ",", Self::LParen => "(", Self::RParen => ")", Self::LBracket => "[", Self::RBracket => "]" })
  }
}
impl core::fmt::Display for Token { fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { core::fmt::Display::fmt(&self.kind(), f) } }
impl From<&Token> for TokenKind {
  fn from(t: &Token) -> Self { match t { Token::Num(_) => Self::Num, Token::Ident(_) => Self::Ident, Token::PlusPlus => Self::PlusPlus, Token::MinusMinus => Self::MinusMinus, Token::EqEq => Self::EqEq, Token::BangEq => Self::BangEq, Token::LtEq => Self::LtEq, Token::GtEq => Self::GtEq, Token::AmpAmp => Self::AmpAmp, Token::PipePipe => Self::PipePipe, Token::Shl => Self::Shl, Token::Shr => Self::Shr, Token::Plus => Self::Plus, Token::Minus => Self::Minus, Token::Star => Self::Star, Token::Slash => Self::Slash, Token::Percent => Self::Percent, Token::Amp => Self::Amp, Token::Pipe => Self::Pipe, Token::Caret => Self::Caret, Token::Tilde => Self::Tilde, Token::Bang => Self::Bang, Token::Question => Self::Question, Token::Colon => Self::Colon, Token::Lt => Self::Lt, Token::Gt => Self::Gt, Token::Comma => Self::Comma, Token::LParen => Self::LParen, Token::RParen => Self::RParen, Token::LBracket => Self::LBracket, Token::RBracket => Self::RBracket } }
}
impl TokenT<'_> for Token { type Kind = TokenKind; type Error = LexError; const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded; fn kind(&self) -> TokenKind { TokenKind::from(self) } fn is_trivia(&self) -> bool { false } }
type CExprLexer<'a> = tokora::lexer::LogosLexer<'a, Token>;
#[derive(Debug)] enum CExprError { Lex(LexError), UnexpectedToken, UnexpectedEot }
impl From<LexError> for CExprError { fn from(e: LexError) -> Self { Self::Lex(e) } }
impl<O, Lang: ?Sized, Set: Clone + 'static> From<tokora::error::UnexpectedEot<O, Lang, Set>> for CExprError { fn from(_: tokora::error::UnexpectedEot<O, Lang, Set>) -> Self { Self::UnexpectedEot } }
impl<O, Lang: ?Sized, Set: Clone + 'static> From<tokora::error::UnexpectedEoLhs<O, Lang, Set>> for CExprError { fn from(_: tokora::error::UnexpectedEoLhs<O, Lang, Set>) -> Self { Self::UnexpectedEot } }
impl<O, Lang: ?Sized, Set: Clone + 'static> From<tokora::error::UnexpectedEoRhs<O, Lang, Set>> for CExprError { fn from(_: tokora::error::UnexpectedEoRhs<O, Lang, Set>) -> Self { Self::UnexpectedEot } }
impl<O, Lang: ?Sized> From<tokora::error::RecursionLimitReached<O, Lang>> for CExprError { fn from(_: tokora::error::RecursionLimitReached<O, Lang>) -> Self { Self::UnexpectedEot } }
impl<O, Lang: ?Sized> From<tokora::error::NonAssociativeChain<O, Lang>> for CExprError { fn from(_: tokora::error::NonAssociativeChain<O, Lang>) -> Self { Self::UnexpectedEot } }
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for CExprError { fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { Self::UnexpectedEot } }
impl<'inp> From<UnexpectedTokenOf<'inp, CExprLexer<'inp>>> for CExprError { fn from(_: UnexpectedTokenOf<'inp, CExprLexer<'inp>>) -> Self { Self::UnexpectedToken } }
#[derive(Clone, Copy, Debug)] enum UnaryOp { Neg, Pos, Not, BNot, PreInc, PreDec }
#[derive(Clone, Copy, Debug)] enum BinOp { Add, Sub, Mul, Div, Mod, Or, And, BOr, BXor, BAnd, Eq, Neq, Lt, Gt, Lte, Gte, Shl, Shr }
#[derive(Clone, Copy, Debug)] enum PostfixOp { Inc, Dec, Index, Call, Ternary }
#[derive(Clone, Debug)] enum Expr { Num(i64), Var(String), Prefix { op: UnaryOp, operand: Box<Expr> }, Binary { op: BinOp, left: Box<Expr>, right: Box<Expr> }, PostfixInc(Box<Expr>), PostfixDec(Box<Expr>), Index { base: Box<Expr>, index: Box<Expr> }, Call { func: Box<Expr>, args: Vec<Expr> }, Ternary { cond: Box<Expr>, then: Box<Expr>, otherwise: Box<Expr> } }
impl core::fmt::Display for UnaryOp { fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { f.write_str(match self { UnaryOp::Neg => "-", UnaryOp::Pos => "+", UnaryOp::Not => "!", UnaryOp::BNot => "~", UnaryOp::PreInc => "++", UnaryOp::PreDec => "--" }) } }
impl core::fmt::Display for BinOp { fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { f.write_str(match self { BinOp::Add => "+", BinOp::Sub => "-", BinOp::Mul => "*", BinOp::Div => "/", BinOp::Mod => "%", BinOp::Or => "||", BinOp::And => "&&", BinOp::BOr => "|", BinOp::BXor => "^", BinOp::BAnd => "&", BinOp::Eq => "==", BinOp::Neq => "!=", BinOp::Lt => "<", BinOp::Gt => ">", BinOp::Lte => "<=", BinOp::Gte => ">=", BinOp::Shl => "<<", BinOp::Shr => ">>" }) } }
impl core::fmt::Display for Expr { fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { match self { Expr::Num(n) => write!(f, "{n}"), Expr::Var(s) => write!(f, "{s}"), Expr::Prefix { op, operand } => write!(f, "({op}{operand})"), Expr::Binary { op, left, right } => write!(f, "({left} {op} {right})"), Expr::PostfixInc(e) => write!(f, "({e}++)"), Expr::PostfixDec(e) => write!(f, "({e}--)"), Expr::Index { base, index } => write!(f, "({base}[{index}])"), Expr::Ternary { cond, then, otherwise } => write!(f, "({cond} ? {then} : {otherwise})"), Expr::Call { func, args } => { write!(f, "{func}(")?; for (i, a) in args.iter().enumerate() { if i > 0 { write!(f, ", ")?; } write!(f, "{a}")?; } write!(f, ")") } } } }
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord)] struct Power(i32);
impl PrattPower for Power {}
const PREC_TERNARY: Power = Power(2);
const PREC_OR: Power = Power(3);
const PREC_AND: Power = Power(4);
const PREC_BOR: Power = Power(5);
const PREC_BXOR: Power = Power(6);
const PREC_BAND: Power = Power(7);
const PREC_EQ: Power = Power(8);
const PREC_CMP: Power = Power(9);
const PREC_SHIFT: Power = Power(10);
const PREC_ADD: Power = Power(11);
const PREC_MUL: Power = Power(12);
const PREC_PREFIX: Power = Power(13);
const PREC_POSTFIX: Power = Power(14);
fn fold_prefix<'inp, Ctx>(_inp: &mut InputRef<'inp, '_, CExprLexer<'inp>, Ctx>, operand: Box<Expr>, op: Precedenced<UnaryOp, Power>) -> Result<Box<Expr>, CExprError> where Ctx: ParseContext<'inp, CExprLexer<'inp>>, Ctx::Emitter: Emitter<'inp, CExprLexer<'inp>, Error = CExprError> { Ok(Box::new(Expr::Prefix { op: op.into_data(), operand })) }
fn fold_infix<'inp, Ctx>(_inp: &mut InputRef<'inp, '_, CExprLexer<'inp>, Ctx>, left: Box<Expr>, right: Box<Expr>, op: Precedenced<PrattInfix<BinOp, BinOp, BinOp>, Power>) -> Result<Box<Expr>, CExprError> where Ctx: ParseContext<'inp, CExprLexer<'inp>>, Ctx::Emitter: Emitter<'inp, CExprLexer<'inp>, Error = CExprError> { let (PrattInfix::Left(op) | PrattInfix::Right(op) | PrattInfix::Neither(op)) = op.into_data(); Ok(Box::new(Expr::Binary { op, left, right })) }
fn fold_postfix<'inp, Ctx>(inp: &mut InputRef<'inp, '_, CExprLexer<'inp>, Ctx>, operand: Box<Expr>, op: Precedenced<PostfixOp, Power>) -> Result<Box<Expr>, CExprError> where Ctx: ParseContext<'inp, CExprLexer<'inp>>, Ctx::Emitter: Emitter<'inp, CExprLexer<'inp>, Error = CExprError> {
  match op.into_data() {
    PostfixOp::Inc => Ok(Box::new(Expr::PostfixInc(operand))),
    PostfixOp::Dec => Ok(Box::new(Expr::PostfixDec(operand))),
    PostfixOp::Index => { let index = parse_cexpr(inp)?; if inp.try_expect(|t| matches!(t.data, Token::RBracket))?.is_none() { return Err(CExprError::UnexpectedToken); } Ok(Box::new(Expr::Index { base: operand, index })) }
    PostfixOp::Call => { let mut args: Vec<Expr> = Vec::new(); if inp.try_expect(|t| matches!(t.data, Token::RParen))?.is_some() { return Ok(Box::new(Expr::Call { func: operand, args })); } args.push(*parse_cexpr(inp)?); loop { if inp.try_expect(|t| matches!(t.data, Token::RParen))?.is_some() { break; } if inp.try_expect(|t| matches!(t.data, Token::Comma))?.is_none() { return Err(CExprError::UnexpectedToken); } args.push(*parse_cexpr(inp)?); } Ok(Box::new(Expr::Call { func: operand, args })) }
    PostfixOp::Ternary => { let then = parse_cexpr(inp)?; if inp.try_expect(|t| matches!(t.data, Token::Colon))?.is_none() { return Err(CExprError::UnexpectedToken); } let otherwise = parse_cexpr(inp)?; Ok(Box::new(Expr::Ternary { cond: operand, then, otherwise })) }
  }
}
fn parse_cexpr<'inp, Ctx>(inp: &mut InputRef<'inp, '_, CExprLexer<'inp>, Ctx>) -> Result<Box<Expr>, CExprError> where Ctx: ParseContext<'inp, CExprLexer<'inp>>, Ctx::Emitter: Emitter<'inp, CExprLexer<'inp>, Error = CExprError> { pratt(parse_lhs, parse_rhs, fold_prefix, fold_infix, fold_postfix).parse_input(inp) }
use tokora::{
  Emitter, InputRef, Parse, ParseContext, Parser,
  parser::{PrattInfix, PrattLHS, PrattRHS, Precedenced},
};

fn parse_lhs<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CExprLexer<'inp>, Ctx>,
) -> Result<PrattLHS<Box<Expr>, UnaryOp, Power>, CExprError>
where
  Ctx: ParseContext<'inp, CExprLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CExprLexer<'inp>, Error = CExprError>,
{
  match inp.next()? {
    None => Err(CExprError::UnexpectedEot),
    Some(tok) => match tok.into_data() {
      Token::Num(n) => Ok(PrattLHS::Operand(Box::new(Expr::Num(n)))),
      Token::Ident(s) => Ok(PrattLHS::Operand(Box::new(Expr::Var(s)))),
      // Grouping: recurse, then require the matching `)`. The parser owns the delimiter.
      Token::LParen => {
        let e = parse_cexpr(inp)?;
        if inp.try_expect(|t| matches!(t.data, Token::RParen))?.is_none() {
          return Err(CExprError::UnexpectedToken);
        }
        Ok(PrattLHS::Operand(e))
      }
      Token::Minus => Ok(PrattLHS::Prefix(Precedenced::new(UnaryOp::Neg, PREC_PREFIX))),
      Token::Plus => Ok(PrattLHS::Prefix(Precedenced::new(UnaryOp::Pos, PREC_PREFIX))),
      Token::Bang => Ok(PrattLHS::Prefix(Precedenced::new(UnaryOp::Not, PREC_PREFIX))),
      Token::Tilde => Ok(PrattLHS::Prefix(Precedenced::new(UnaryOp::BNot, PREC_PREFIX))),
      Token::PlusPlus => Ok(PrattLHS::Prefix(Precedenced::new(UnaryOp::PreInc, PREC_PREFIX))),
      Token::MinusMinus => Ok(PrattLHS::Prefix(Precedenced::new(UnaryOp::PreDec, PREC_PREFIX))),
      _ => Err(CExprError::UnexpectedToken),
    },
  }
}

fn parse_rhs<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CExprLexer<'inp>, Ctx>,
) -> Result<PrattRHS<BinOp, BinOp, BinOp, PostfixOp, Power>, CExprError>
where
  Ctx: ParseContext<'inp, CExprLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CExprLexer<'inp>, Error = CExprError>,
{
  macro_rules! infix_l {
    ($op:expr, $prec:expr) => { PrattRHS::Infix(Precedenced::new(PrattInfix::Left($op), $prec)) };
  }
  match inp.next()? {
    // Nothing left: the expression stops here.
    None => Ok(PrattRHS::End),
    Some(tok) => Ok(match tok.into_data() {
      Token::PipePipe => infix_l!(BinOp::Or, PREC_OR),
      Token::AmpAmp => infix_l!(BinOp::And, PREC_AND),
      Token::Pipe => infix_l!(BinOp::BOr, PREC_BOR),
      Token::Caret => infix_l!(BinOp::BXor, PREC_BXOR),
      Token::Amp => infix_l!(BinOp::BAnd, PREC_BAND),
      Token::EqEq => infix_l!(BinOp::Eq, PREC_EQ),
      Token::BangEq => infix_l!(BinOp::Neq, PREC_EQ),
      Token::Lt => infix_l!(BinOp::Lt, PREC_CMP),
      Token::Gt => infix_l!(BinOp::Gt, PREC_CMP),
      Token::LtEq => infix_l!(BinOp::Lte, PREC_CMP),
      Token::GtEq => infix_l!(BinOp::Gte, PREC_CMP),
      Token::Shl => infix_l!(BinOp::Shl, PREC_SHIFT),
      Token::Shr => infix_l!(BinOp::Shr, PREC_SHIFT),
      Token::Plus => infix_l!(BinOp::Add, PREC_ADD),
      Token::Minus => infix_l!(BinOp::Sub, PREC_ADD),
      Token::Star => infix_l!(BinOp::Mul, PREC_MUL),
      Token::Slash => infix_l!(BinOp::Div, PREC_MUL),
      Token::Percent => infix_l!(BinOp::Mod, PREC_MUL),
      // Postfix triggers. `parse_rhs` consumes only the trigger token; `fold_postfix` reads the
      // rest. `?` binds at the low ternary level; the rest at the high postfix level.
      Token::PlusPlus => PrattRHS::Postfix(Precedenced::new(PostfixOp::Inc, PREC_POSTFIX)),
      Token::MinusMinus => PrattRHS::Postfix(Precedenced::new(PostfixOp::Dec, PREC_POSTFIX)),
      Token::LBracket => PrattRHS::Postfix(Precedenced::new(PostfixOp::Index, PREC_POSTFIX)),
      Token::LParen => PrattRHS::Postfix(Precedenced::new(PostfixOp::Call, PREC_POSTFIX)),
      Token::Question => PrattRHS::Postfix(Precedenced::new(PostfixOp::Ternary, PREC_TERNARY)),
      // Anything else is not an operator here; the engine restores what this call consumed and
      // leaves the token for the surrounding grammar.
      _ => PrattRHS::End,
    }),
  }
}

// Precedence, associativity, grouping, and prefix operators are all decided by these two
// classifiers plus the engine. (The folds — hidden here — build the tree; they are revealed next.)
let parse = |src| Parser::new().apply(parse_cexpr).parse_str(src).map(|e: Box<Expr>| e.to_string());
assert_eq!(parse("1 + 2 * 3").unwrap(), "(1 + (2 * 3))");    // `*` outranks `+`
assert_eq!(parse("(1 + 2) * 3").unwrap(), "((1 + 2) * 3)");  // grouping overrides
assert_eq!(parse("a + b + c").unwrap(), "((a + b) + c)");    // `+` is left-associative
assert_eq!(parse("-a").unwrap(), "(-a)");                    // prefix
assert_eq!(parse("~bits | flags").unwrap(), "((~bits) | flags)"); // prefix binds tighter than `|`

Implement the three folds

Use named fns, not closures: the fold traits carry a higher-ranked lifetime bound (here on the InputRef’s inner lifetime) that a monomorphic closure cannot satisfy but a generic fn item satisfies for free. fold_prefix wraps an operand in Expr::Prefix; fold_infix extracts the operator from PrattInfix (the engine has already applied associativity) and builds Expr::Binary. Both are pure tree builders — they never touch the input.

fold_postfix is where the AST-level API earns its keep. It receives the InputRef first (the calculator’s token-level postfix fold gets no input at all), so a postfix trigger can go on to consume the tokens it needs. parse_rhs has already eaten the trigger ([, (, or ?); the fold reads the rest:

• index parses an expression, then requires ];
• call accepts ) for an empty argument list or loops over comma-separated expressions;
• ternary parses the then-expression, requires :, and parses the otherwise-expression.

Crucially, each enclosed sub-expression is just a recursive parse_cexpr call, and each of those stops before the delimiter this fold then consumes itself — the parser stays in control of every delimiter.

use tokora::{Token as TokenT, ParseInput, error::token::UnexpectedTokenOf, logos::{self, Logos}, parser::{PrattLHS, PrattPower, PrattRHS, pratt}};
#[derive(Clone, Debug, Default, PartialEq)] struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { Self } }
#[derive(Clone, Debug, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Token {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))] Num(i64),
  #[regex(r"[a-zA-Z_][a-zA-Z0-9_]*", |lex| lex.slice().to_string())] Ident(String),
  #[token("++")] PlusPlus, #[token("--")] MinusMinus, #[token("==")] EqEq, #[token("!=")] BangEq,
  #[token("<=")] LtEq, #[token(">=")] GtEq, #[token("&&")] AmpAmp, #[token("||")] PipePipe,
  #[token("<<")] Shl, #[token(">>")] Shr, #[token("+")] Plus, #[token("-")] Minus,
  #[token("*")] Star, #[token("/")] Slash, #[token("%")] Percent, #[token("&")] Amp,
  #[token("|")] Pipe, #[token("^")] Caret, #[token("~")] Tilde, #[token("!")] Bang,
  #[token("?")] Question, #[token(":")] Colon, #[token("<")] Lt, #[token(">")] Gt,
  #[token(",")] Comma, #[token("(")] LParen, #[token(")")] RParen, #[token("[")] LBracket,
  #[token("]")] RBracket,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
enum TokenKind { Num, Ident, PlusPlus, MinusMinus, EqEq, BangEq, LtEq, GtEq, AmpAmp, PipePipe, Shl, Shr, Plus, Minus, Star, Slash, Percent, Amp, Pipe, Caret, Tilde, Bang, Question, Colon, Lt, Gt, Comma, LParen, RParen, LBracket, RBracket }
impl core::fmt::Display for TokenKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self { Self::Num => "number", Self::Ident => "identifier", Self::PlusPlus => "++", Self::MinusMinus => "--", Self::EqEq => "==", Self::BangEq => "!=", Self::LtEq => "<=", Self::GtEq => ">=", Self::AmpAmp => "&&", Self::PipePipe => "||", Self::Shl => "<<", Self::Shr => ">>", Self::Plus => "+", Self::Minus => "-", Self::Star => "*", Self::Slash => "/", Self::Percent => "%", Self::Amp => "&", Self::Pipe => "|", Self::Caret => "^", Self::Tilde => "~", Self::Bang => "!", Self::Question => "?", Self::Colon => ":", Self::Lt => "<", Self::Gt => ">", Self::Comma => ",", Self::LParen => "(", Self::RParen => ")", Self::LBracket => "[", Self::RBracket => "]" })
  }
}
impl core::fmt::Display for Token { fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { core::fmt::Display::fmt(&self.kind(), f) } }
impl From<&Token> for TokenKind {
  fn from(t: &Token) -> Self { match t { Token::Num(_) => Self::Num, Token::Ident(_) => Self::Ident, Token::PlusPlus => Self::PlusPlus, Token::MinusMinus => Self::MinusMinus, Token::EqEq => Self::EqEq, Token::BangEq => Self::BangEq, Token::LtEq => Self::LtEq, Token::GtEq => Self::GtEq, Token::AmpAmp => Self::AmpAmp, Token::PipePipe => Self::PipePipe, Token::Shl => Self::Shl, Token::Shr => Self::Shr, Token::Plus => Self::Plus, Token::Minus => Self::Minus, Token::Star => Self::Star, Token::Slash => Self::Slash, Token::Percent => Self::Percent, Token::Amp => Self::Amp, Token::Pipe => Self::Pipe, Token::Caret => Self::Caret, Token::Tilde => Self::Tilde, Token::Bang => Self::Bang, Token::Question => Self::Question, Token::Colon => Self::Colon, Token::Lt => Self::Lt, Token::Gt => Self::Gt, Token::Comma => Self::Comma, Token::LParen => Self::LParen, Token::RParen => Self::RParen, Token::LBracket => Self::LBracket, Token::RBracket => Self::RBracket } }
}
impl TokenT<'_> for Token { type Kind = TokenKind; type Error = LexError; const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded; fn kind(&self) -> TokenKind { TokenKind::from(self) } fn is_trivia(&self) -> bool { false } }
type CExprLexer<'a> = tokora::lexer::LogosLexer<'a, Token>;
#[derive(Debug)] enum CExprError { Lex(LexError), UnexpectedToken, UnexpectedEot }
impl From<LexError> for CExprError { fn from(e: LexError) -> Self { Self::Lex(e) } }
impl<O, Lang: ?Sized, Set: Clone + 'static> From<tokora::error::UnexpectedEot<O, Lang, Set>> for CExprError { fn from(_: tokora::error::UnexpectedEot<O, Lang, Set>) -> Self { Self::UnexpectedEot } }
impl<O, Lang: ?Sized, Set: Clone + 'static> From<tokora::error::UnexpectedEoLhs<O, Lang, Set>> for CExprError { fn from(_: tokora::error::UnexpectedEoLhs<O, Lang, Set>) -> Self { Self::UnexpectedEot } }
impl<O, Lang: ?Sized, Set: Clone + 'static> From<tokora::error::UnexpectedEoRhs<O, Lang, Set>> for CExprError { fn from(_: tokora::error::UnexpectedEoRhs<O, Lang, Set>) -> Self { Self::UnexpectedEot } }
impl<O, Lang: ?Sized> From<tokora::error::RecursionLimitReached<O, Lang>> for CExprError { fn from(_: tokora::error::RecursionLimitReached<O, Lang>) -> Self { Self::UnexpectedEot } }
impl<O, Lang: ?Sized> From<tokora::error::NonAssociativeChain<O, Lang>> for CExprError { fn from(_: tokora::error::NonAssociativeChain<O, Lang>) -> Self { Self::UnexpectedEot } }
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for CExprError { fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { Self::UnexpectedEot } }
impl<'inp> From<UnexpectedTokenOf<'inp, CExprLexer<'inp>>> for CExprError { fn from(_: UnexpectedTokenOf<'inp, CExprLexer<'inp>>) -> Self { Self::UnexpectedToken } }
#[derive(Clone, Copy, Debug)] enum UnaryOp { Neg, Pos, Not, BNot, PreInc, PreDec }
#[derive(Clone, Copy, Debug)] enum BinOp { Add, Sub, Mul, Div, Mod, Or, And, BOr, BXor, BAnd, Eq, Neq, Lt, Gt, Lte, Gte, Shl, Shr }
#[derive(Clone, Copy, Debug)] enum PostfixOp { Inc, Dec, Index, Call, Ternary }
#[derive(Clone, Debug)] enum Expr { Num(i64), Var(String), Prefix { op: UnaryOp, operand: Box<Expr> }, Binary { op: BinOp, left: Box<Expr>, right: Box<Expr> }, PostfixInc(Box<Expr>), PostfixDec(Box<Expr>), Index { base: Box<Expr>, index: Box<Expr> }, Call { func: Box<Expr>, args: Vec<Expr> }, Ternary { cond: Box<Expr>, then: Box<Expr>, otherwise: Box<Expr> } }
impl core::fmt::Display for UnaryOp { fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { f.write_str(match self { UnaryOp::Neg => "-", UnaryOp::Pos => "+", UnaryOp::Not => "!", UnaryOp::BNot => "~", UnaryOp::PreInc => "++", UnaryOp::PreDec => "--" }) } }
impl core::fmt::Display for BinOp { fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { f.write_str(match self { BinOp::Add => "+", BinOp::Sub => "-", BinOp::Mul => "*", BinOp::Div => "/", BinOp::Mod => "%", BinOp::Or => "||", BinOp::And => "&&", BinOp::BOr => "|", BinOp::BXor => "^", BinOp::BAnd => "&", BinOp::Eq => "==", BinOp::Neq => "!=", BinOp::Lt => "<", BinOp::Gt => ">", BinOp::Lte => "<=", BinOp::Gte => ">=", BinOp::Shl => "<<", BinOp::Shr => ">>" }) } }
impl core::fmt::Display for Expr { fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { match self { Expr::Num(n) => write!(f, "{n}"), Expr::Var(s) => write!(f, "{s}"), Expr::Prefix { op, operand } => write!(f, "({op}{operand})"), Expr::Binary { op, left, right } => write!(f, "({left} {op} {right})"), Expr::PostfixInc(e) => write!(f, "({e}++)"), Expr::PostfixDec(e) => write!(f, "({e}--)"), Expr::Index { base, index } => write!(f, "({base}[{index}])"), Expr::Ternary { cond, then, otherwise } => write!(f, "({cond} ? {then} : {otherwise})"), Expr::Call { func, args } => { write!(f, "{func}(")?; for (i, a) in args.iter().enumerate() { if i > 0 { write!(f, ", ")?; } write!(f, "{a}")?; } write!(f, ")") } } } }
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord)] struct Power(i32);
impl PrattPower for Power {}
const PREC_TERNARY: Power = Power(2);
const PREC_OR: Power = Power(3);
const PREC_AND: Power = Power(4);
const PREC_BOR: Power = Power(5);
const PREC_BXOR: Power = Power(6);
const PREC_BAND: Power = Power(7);
const PREC_EQ: Power = Power(8);
const PREC_CMP: Power = Power(9);
const PREC_SHIFT: Power = Power(10);
const PREC_ADD: Power = Power(11);
const PREC_MUL: Power = Power(12);
const PREC_PREFIX: Power = Power(13);
const PREC_POSTFIX: Power = Power(14);
fn parse_lhs<'inp, Ctx>(inp: &mut InputRef<'inp, '_, CExprLexer<'inp>, Ctx>) -> Result<PrattLHS<Box<Expr>, UnaryOp, Power>, CExprError> where Ctx: ParseContext<'inp, CExprLexer<'inp>>, Ctx::Emitter: Emitter<'inp, CExprLexer<'inp>, Error = CExprError> {
  match inp.next()? {
    None => Err(CExprError::UnexpectedEot),
    Some(tok) => match tok.into_data() {
      Token::Num(n) => Ok(PrattLHS::Operand(Box::new(Expr::Num(n)))),
      Token::Ident(s) => Ok(PrattLHS::Operand(Box::new(Expr::Var(s)))),
      Token::LParen => { let e = parse_cexpr(inp)?; if inp.try_expect(|t| matches!(t.data, Token::RParen))?.is_none() { return Err(CExprError::UnexpectedToken); } Ok(PrattLHS::Operand(e)) }
      Token::Minus => Ok(PrattLHS::Prefix(Precedenced::new(UnaryOp::Neg, PREC_PREFIX))),
      Token::Plus => Ok(PrattLHS::Prefix(Precedenced::new(UnaryOp::Pos, PREC_PREFIX))),
      Token::Bang => Ok(PrattLHS::Prefix(Precedenced::new(UnaryOp::Not, PREC_PREFIX))),
      Token::Tilde => Ok(PrattLHS::Prefix(Precedenced::new(UnaryOp::BNot, PREC_PREFIX))),
      Token::PlusPlus => Ok(PrattLHS::Prefix(Precedenced::new(UnaryOp::PreInc, PREC_PREFIX))),
      Token::MinusMinus => Ok(PrattLHS::Prefix(Precedenced::new(UnaryOp::PreDec, PREC_PREFIX))),
      _ => Err(CExprError::UnexpectedToken),
    },
  }
}
fn parse_rhs<'inp, Ctx>(inp: &mut InputRef<'inp, '_, CExprLexer<'inp>, Ctx>) -> Result<PrattRHS<BinOp, BinOp, BinOp, PostfixOp, Power>, CExprError> where Ctx: ParseContext<'inp, CExprLexer<'inp>>, Ctx::Emitter: Emitter<'inp, CExprLexer<'inp>, Error = CExprError> {
  match inp.next()? {
    None => Ok(PrattRHS::End),
    Some(tok) => Ok(match tok.into_data() {
      Token::PipePipe => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Or), PREC_OR)),
      Token::AmpAmp => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::And), PREC_AND)),
      Token::Pipe => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::BOr), PREC_BOR)),
      Token::Caret => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::BXor), PREC_BXOR)),
      Token::Amp => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::BAnd), PREC_BAND)),
      Token::EqEq => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Eq), PREC_EQ)),
      Token::BangEq => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Neq), PREC_EQ)),
      Token::Lt => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Lt), PREC_CMP)),
      Token::Gt => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Gt), PREC_CMP)),
      Token::LtEq => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Lte), PREC_CMP)),
      Token::GtEq => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Gte), PREC_CMP)),
      Token::Shl => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Shl), PREC_SHIFT)),
      Token::Shr => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Shr), PREC_SHIFT)),
      Token::Plus => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Add), PREC_ADD)),
      Token::Minus => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Sub), PREC_ADD)),
      Token::Star => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Mul), PREC_MUL)),
      Token::Slash => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Div), PREC_MUL)),
      Token::Percent => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Mod), PREC_MUL)),
      Token::PlusPlus => PrattRHS::Postfix(Precedenced::new(PostfixOp::Inc, PREC_POSTFIX)),
      Token::MinusMinus => PrattRHS::Postfix(Precedenced::new(PostfixOp::Dec, PREC_POSTFIX)),
      Token::LBracket => PrattRHS::Postfix(Precedenced::new(PostfixOp::Index, PREC_POSTFIX)),
      Token::LParen => PrattRHS::Postfix(Precedenced::new(PostfixOp::Call, PREC_POSTFIX)),
      Token::Question => PrattRHS::Postfix(Precedenced::new(PostfixOp::Ternary, PREC_TERNARY)),
      _ => PrattRHS::End,
    }),
  }
}
fn parse_cexpr<'inp, Ctx>(inp: &mut InputRef<'inp, '_, CExprLexer<'inp>, Ctx>) -> Result<Box<Expr>, CExprError> where Ctx: ParseContext<'inp, CExprLexer<'inp>>, Ctx::Emitter: Emitter<'inp, CExprLexer<'inp>, Error = CExprError> { pratt(parse_lhs, parse_rhs, fold_prefix, fold_infix, fold_postfix).parse_input(inp) }
use tokora::{
  Emitter, InputRef, Parse, ParseContext, Parser,
  parser::{PrattInfix, Precedenced},
};

fn fold_prefix<'inp, Ctx>(
  _inp: &mut InputRef<'inp, '_, CExprLexer<'inp>, Ctx>,
  operand: Box<Expr>,
  op: Precedenced<UnaryOp, Power>,
) -> Result<Box<Expr>, CExprError>
where
  Ctx: ParseContext<'inp, CExprLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CExprLexer<'inp>, Error = CExprError>,
{
  Ok(Box::new(Expr::Prefix { op: op.into_data(), operand }))
}

fn fold_infix<'inp, Ctx>(
  _inp: &mut InputRef<'inp, '_, CExprLexer<'inp>, Ctx>,
  left: Box<Expr>,
  right: Box<Expr>,
  op: Precedenced<PrattInfix<BinOp, BinOp, BinOp>, Power>,
) -> Result<Box<Expr>, CExprError>
where
  Ctx: ParseContext<'inp, CExprLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CExprLexer<'inp>, Error = CExprError>,
{
  // Associativity has already done its job in the engine; the fold just wants the operator.
  let (PrattInfix::Left(op) | PrattInfix::Right(op) | PrattInfix::Neither(op)) = op.into_data();
  Ok(Box::new(Expr::Binary { op, left, right }))
}

fn fold_postfix<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CExprLexer<'inp>, Ctx>,
  operand: Box<Expr>,
  op: Precedenced<PostfixOp, Power>,
) -> Result<Box<Expr>, CExprError>
where
  Ctx: ParseContext<'inp, CExprLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CExprLexer<'inp>, Error = CExprError>,
{
  match op.into_data() {
    PostfixOp::Inc => Ok(Box::new(Expr::PostfixInc(operand))),
    PostfixOp::Dec => Ok(Box::new(Expr::PostfixDec(operand))),
    // e[index]: parse the index (stops before `]`), then require `]`.
    PostfixOp::Index => {
      let index = parse_cexpr(inp)?;
      if inp.try_expect(|t| matches!(t.data, Token::RBracket))?.is_none() {
        return Err(CExprError::UnexpectedToken);
      }
      Ok(Box::new(Expr::Index { base: operand, index }))
    }
    // e(args): an empty `)`, or a comma-separated argument loop.
    PostfixOp::Call => {
      let mut args: Vec<Expr> = Vec::new();
      if inp.try_expect(|t| matches!(t.data, Token::RParen))?.is_some() {
        return Ok(Box::new(Expr::Call { func: operand, args }));
      }
      args.push(*parse_cexpr(inp)?);
      loop {
        if inp.try_expect(|t| matches!(t.data, Token::RParen))?.is_some() {
          break;
        }
        if inp.try_expect(|t| matches!(t.data, Token::Comma))?.is_none() {
          return Err(CExprError::UnexpectedToken);
        }
        args.push(*parse_cexpr(inp)?);
      }
      Ok(Box::new(Expr::Call { func: operand, args }))
    }
    // cond ? then : otherwise — parse `then` (stops before `:`), require `:`, parse `otherwise`.
    PostfixOp::Ternary => {
      let then = parse_cexpr(inp)?;
      if inp.try_expect(|t| matches!(t.data, Token::Colon))?.is_none() {
        return Err(CExprError::UnexpectedToken);
      }
      let otherwise = parse_cexpr(inp)?;
      Ok(Box::new(Expr::Ternary { cond: operand, then, otherwise }))
    }
  }
}

// The postfix forms — increment/decrement, indexing, calls, and the ternary — each proven by the
// fold consuming exactly the input it needs after the trigger token.
let parse = |src| Parser::new().apply(parse_cexpr).parse_str(src).map(|e: Box<Expr>| e.to_string());
assert_eq!(parse("x++").unwrap(), "(x++)");                       // postfix ++ (same token as prefix)
assert_eq!(parse("arr[i + 1]").unwrap(), "(arr[(i + 1)])");       // index consumes an expr and `]`
assert_eq!(parse("f()").unwrap(), "f()");                         // empty argument list
assert_eq!(parse("f(a + b, c * d)").unwrap(), "f((a + b), (c * d))"); // comma-separated args
assert_eq!(parse("a ? b : c").unwrap(), "(a ? b : c)");           // ternary consumes `t`, `:`, `f`

Close the recursion in parse_cexpr

parse_cexpr calls pratt with the two classifiers and three folds, then drives it with parse_input. Note what is absent from its bounds: there is no PrattEmitter. The AST engine keeps error-reporting inside the folds and parse_lhs, so the ordinary Emitter bound is all it needs — a FatalContext satisfies it with no extra work. The mutual recursion (grouping in parse_lhs, index/call/ternary in fold_postfix) rides ordinary call-stack frames through these named fns; no recursive parser type is involved.

The five assertions below extend to the maintained binary’s full assertion table — precedence, grouping, associativity, unary operators, increment, ternary, indexing, calls, shifts, and bitwise operators — now executable inline:

use tokora::{Token as TokenT, error::token::UnexpectedTokenOf, logos::{self, Logos}, parser::{PrattInfix, PrattLHS, PrattPower, PrattRHS, Precedenced}};
#[derive(Clone, Debug, Default, PartialEq)] struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { Self } }
#[derive(Clone, Debug, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Token {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))] Num(i64),
  #[regex(r"[a-zA-Z_][a-zA-Z0-9_]*", |lex| lex.slice().to_string())] Ident(String),
  #[token("++")] PlusPlus, #[token("--")] MinusMinus, #[token("==")] EqEq, #[token("!=")] BangEq,
  #[token("<=")] LtEq, #[token(">=")] GtEq, #[token("&&")] AmpAmp, #[token("||")] PipePipe,
  #[token("<<")] Shl, #[token(">>")] Shr, #[token("+")] Plus, #[token("-")] Minus,
  #[token("*")] Star, #[token("/")] Slash, #[token("%")] Percent, #[token("&")] Amp,
  #[token("|")] Pipe, #[token("^")] Caret, #[token("~")] Tilde, #[token("!")] Bang,
  #[token("?")] Question, #[token(":")] Colon, #[token("<")] Lt, #[token(">")] Gt,
  #[token(",")] Comma, #[token("(")] LParen, #[token(")")] RParen, #[token("[")] LBracket,
  #[token("]")] RBracket,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
enum TokenKind { Num, Ident, PlusPlus, MinusMinus, EqEq, BangEq, LtEq, GtEq, AmpAmp, PipePipe, Shl, Shr, Plus, Minus, Star, Slash, Percent, Amp, Pipe, Caret, Tilde, Bang, Question, Colon, Lt, Gt, Comma, LParen, RParen, LBracket, RBracket }
impl core::fmt::Display for TokenKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self { Self::Num => "number", Self::Ident => "identifier", Self::PlusPlus => "++", Self::MinusMinus => "--", Self::EqEq => "==", Self::BangEq => "!=", Self::LtEq => "<=", Self::GtEq => ">=", Self::AmpAmp => "&&", Self::PipePipe => "||", Self::Shl => "<<", Self::Shr => ">>", Self::Plus => "+", Self::Minus => "-", Self::Star => "*", Self::Slash => "/", Self::Percent => "%", Self::Amp => "&", Self::Pipe => "|", Self::Caret => "^", Self::Tilde => "~", Self::Bang => "!", Self::Question => "?", Self::Colon => ":", Self::Lt => "<", Self::Gt => ">", Self::Comma => ",", Self::LParen => "(", Self::RParen => ")", Self::LBracket => "[", Self::RBracket => "]" })
  }
}
impl core::fmt::Display for Token { fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { core::fmt::Display::fmt(&self.kind(), f) } }
impl From<&Token> for TokenKind {
  fn from(t: &Token) -> Self { match t { Token::Num(_) => Self::Num, Token::Ident(_) => Self::Ident, Token::PlusPlus => Self::PlusPlus, Token::MinusMinus => Self::MinusMinus, Token::EqEq => Self::EqEq, Token::BangEq => Self::BangEq, Token::LtEq => Self::LtEq, Token::GtEq => Self::GtEq, Token::AmpAmp => Self::AmpAmp, Token::PipePipe => Self::PipePipe, Token::Shl => Self::Shl, Token::Shr => Self::Shr, Token::Plus => Self::Plus, Token::Minus => Self::Minus, Token::Star => Self::Star, Token::Slash => Self::Slash, Token::Percent => Self::Percent, Token::Amp => Self::Amp, Token::Pipe => Self::Pipe, Token::Caret => Self::Caret, Token::Tilde => Self::Tilde, Token::Bang => Self::Bang, Token::Question => Self::Question, Token::Colon => Self::Colon, Token::Lt => Self::Lt, Token::Gt => Self::Gt, Token::Comma => Self::Comma, Token::LParen => Self::LParen, Token::RParen => Self::RParen, Token::LBracket => Self::LBracket, Token::RBracket => Self::RBracket } }
}
impl TokenT<'_> for Token { type Kind = TokenKind; type Error = LexError; const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded; fn kind(&self) -> TokenKind { TokenKind::from(self) } fn is_trivia(&self) -> bool { false } }
type CExprLexer<'a> = tokora::lexer::LogosLexer<'a, Token>;
#[derive(Debug)] enum CExprError { Lex(LexError), UnexpectedToken, UnexpectedEot }
impl From<LexError> for CExprError { fn from(e: LexError) -> Self { Self::Lex(e) } }
impl<O, Lang: ?Sized, Set: Clone + 'static> From<tokora::error::UnexpectedEot<O, Lang, Set>> for CExprError { fn from(_: tokora::error::UnexpectedEot<O, Lang, Set>) -> Self { Self::UnexpectedEot } }
impl<O, Lang: ?Sized, Set: Clone + 'static> From<tokora::error::UnexpectedEoLhs<O, Lang, Set>> for CExprError { fn from(_: tokora::error::UnexpectedEoLhs<O, Lang, Set>) -> Self { Self::UnexpectedEot } }
impl<O, Lang: ?Sized, Set: Clone + 'static> From<tokora::error::UnexpectedEoRhs<O, Lang, Set>> for CExprError { fn from(_: tokora::error::UnexpectedEoRhs<O, Lang, Set>) -> Self { Self::UnexpectedEot } }
impl<O, Lang: ?Sized> From<tokora::error::RecursionLimitReached<O, Lang>> for CExprError { fn from(_: tokora::error::RecursionLimitReached<O, Lang>) -> Self { Self::UnexpectedEot } }
impl<O, Lang: ?Sized> From<tokora::error::NonAssociativeChain<O, Lang>> for CExprError { fn from(_: tokora::error::NonAssociativeChain<O, Lang>) -> Self { Self::UnexpectedEot } }
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for CExprError { fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { Self::UnexpectedEot } }
impl<'inp> From<UnexpectedTokenOf<'inp, CExprLexer<'inp>>> for CExprError { fn from(_: UnexpectedTokenOf<'inp, CExprLexer<'inp>>) -> Self { Self::UnexpectedToken } }
#[derive(Clone, Copy, Debug)] enum UnaryOp { Neg, Pos, Not, BNot, PreInc, PreDec }
#[derive(Clone, Copy, Debug)] enum BinOp { Add, Sub, Mul, Div, Mod, Or, And, BOr, BXor, BAnd, Eq, Neq, Lt, Gt, Lte, Gte, Shl, Shr }
#[derive(Clone, Copy, Debug)] enum PostfixOp { Inc, Dec, Index, Call, Ternary }
#[derive(Clone, Debug)] enum Expr { Num(i64), Var(String), Prefix { op: UnaryOp, operand: Box<Expr> }, Binary { op: BinOp, left: Box<Expr>, right: Box<Expr> }, PostfixInc(Box<Expr>), PostfixDec(Box<Expr>), Index { base: Box<Expr>, index: Box<Expr> }, Call { func: Box<Expr>, args: Vec<Expr> }, Ternary { cond: Box<Expr>, then: Box<Expr>, otherwise: Box<Expr> } }
impl core::fmt::Display for UnaryOp { fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { f.write_str(match self { UnaryOp::Neg => "-", UnaryOp::Pos => "+", UnaryOp::Not => "!", UnaryOp::BNot => "~", UnaryOp::PreInc => "++", UnaryOp::PreDec => "--" }) } }
impl core::fmt::Display for BinOp { fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { f.write_str(match self { BinOp::Add => "+", BinOp::Sub => "-", BinOp::Mul => "*", BinOp::Div => "/", BinOp::Mod => "%", BinOp::Or => "||", BinOp::And => "&&", BinOp::BOr => "|", BinOp::BXor => "^", BinOp::BAnd => "&", BinOp::Eq => "==", BinOp::Neq => "!=", BinOp::Lt => "<", BinOp::Gt => ">", BinOp::Lte => "<=", BinOp::Gte => ">=", BinOp::Shl => "<<", BinOp::Shr => ">>" }) } }
impl core::fmt::Display for Expr { fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { match self { Expr::Num(n) => write!(f, "{n}"), Expr::Var(s) => write!(f, "{s}"), Expr::Prefix { op, operand } => write!(f, "({op}{operand})"), Expr::Binary { op, left, right } => write!(f, "({left} {op} {right})"), Expr::PostfixInc(e) => write!(f, "({e}++)"), Expr::PostfixDec(e) => write!(f, "({e}--)"), Expr::Index { base, index } => write!(f, "({base}[{index}])"), Expr::Ternary { cond, then, otherwise } => write!(f, "({cond} ? {then} : {otherwise})"), Expr::Call { func, args } => { write!(f, "{func}(")?; for (i, a) in args.iter().enumerate() { if i > 0 { write!(f, ", ")?; } write!(f, "{a}")?; } write!(f, ")") } } } }
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord)] struct Power(i32);
impl PrattPower for Power {}
const PREC_TERNARY: Power = Power(2);
const PREC_OR: Power = Power(3);
const PREC_AND: Power = Power(4);
const PREC_BOR: Power = Power(5);
const PREC_BXOR: Power = Power(6);
const PREC_BAND: Power = Power(7);
const PREC_EQ: Power = Power(8);
const PREC_CMP: Power = Power(9);
const PREC_SHIFT: Power = Power(10);
const PREC_ADD: Power = Power(11);
const PREC_MUL: Power = Power(12);
const PREC_PREFIX: Power = Power(13);
const PREC_POSTFIX: Power = Power(14);
fn parse_lhs<'inp, Ctx>(inp: &mut InputRef<'inp, '_, CExprLexer<'inp>, Ctx>) -> Result<PrattLHS<Box<Expr>, UnaryOp, Power>, CExprError> where Ctx: ParseContext<'inp, CExprLexer<'inp>>, Ctx::Emitter: Emitter<'inp, CExprLexer<'inp>, Error = CExprError> {
  match inp.next()? {
    None => Err(CExprError::UnexpectedEot),
    Some(tok) => match tok.into_data() {
      Token::Num(n) => Ok(PrattLHS::Operand(Box::new(Expr::Num(n)))),
      Token::Ident(s) => Ok(PrattLHS::Operand(Box::new(Expr::Var(s)))),
      Token::LParen => { let e = parse_cexpr(inp)?; if inp.try_expect(|t| matches!(t.data, Token::RParen))?.is_none() { return Err(CExprError::UnexpectedToken); } Ok(PrattLHS::Operand(e)) }
      Token::Minus => Ok(PrattLHS::Prefix(Precedenced::new(UnaryOp::Neg, PREC_PREFIX))),
      Token::Plus => Ok(PrattLHS::Prefix(Precedenced::new(UnaryOp::Pos, PREC_PREFIX))),
      Token::Bang => Ok(PrattLHS::Prefix(Precedenced::new(UnaryOp::Not, PREC_PREFIX))),
      Token::Tilde => Ok(PrattLHS::Prefix(Precedenced::new(UnaryOp::BNot, PREC_PREFIX))),
      Token::PlusPlus => Ok(PrattLHS::Prefix(Precedenced::new(UnaryOp::PreInc, PREC_PREFIX))),
      Token::MinusMinus => Ok(PrattLHS::Prefix(Precedenced::new(UnaryOp::PreDec, PREC_PREFIX))),
      _ => Err(CExprError::UnexpectedToken),
    },
  }
}
fn parse_rhs<'inp, Ctx>(inp: &mut InputRef<'inp, '_, CExprLexer<'inp>, Ctx>) -> Result<PrattRHS<BinOp, BinOp, BinOp, PostfixOp, Power>, CExprError> where Ctx: ParseContext<'inp, CExprLexer<'inp>>, Ctx::Emitter: Emitter<'inp, CExprLexer<'inp>, Error = CExprError> {
  match inp.next()? {
    None => Ok(PrattRHS::End),
    Some(tok) => Ok(match tok.into_data() {
      Token::PipePipe => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Or), PREC_OR)),
      Token::AmpAmp => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::And), PREC_AND)),
      Token::Pipe => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::BOr), PREC_BOR)),
      Token::Caret => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::BXor), PREC_BXOR)),
      Token::Amp => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::BAnd), PREC_BAND)),
      Token::EqEq => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Eq), PREC_EQ)),
      Token::BangEq => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Neq), PREC_EQ)),
      Token::Lt => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Lt), PREC_CMP)),
      Token::Gt => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Gt), PREC_CMP)),
      Token::LtEq => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Lte), PREC_CMP)),
      Token::GtEq => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Gte), PREC_CMP)),
      Token::Shl => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Shl), PREC_SHIFT)),
      Token::Shr => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Shr), PREC_SHIFT)),
      Token::Plus => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Add), PREC_ADD)),
      Token::Minus => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Sub), PREC_ADD)),
      Token::Star => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Mul), PREC_MUL)),
      Token::Slash => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Div), PREC_MUL)),
      Token::Percent => PrattRHS::Infix(Precedenced::new(PrattInfix::Left(BinOp::Mod), PREC_MUL)),
      Token::PlusPlus => PrattRHS::Postfix(Precedenced::new(PostfixOp::Inc, PREC_POSTFIX)),
      Token::MinusMinus => PrattRHS::Postfix(Precedenced::new(PostfixOp::Dec, PREC_POSTFIX)),
      Token::LBracket => PrattRHS::Postfix(Precedenced::new(PostfixOp::Index, PREC_POSTFIX)),
      Token::LParen => PrattRHS::Postfix(Precedenced::new(PostfixOp::Call, PREC_POSTFIX)),
      Token::Question => PrattRHS::Postfix(Precedenced::new(PostfixOp::Ternary, PREC_TERNARY)),
      _ => PrattRHS::End,
    }),
  }
}
fn fold_prefix<'inp, Ctx>(_inp: &mut InputRef<'inp, '_, CExprLexer<'inp>, Ctx>, operand: Box<Expr>, op: Precedenced<UnaryOp, Power>) -> Result<Box<Expr>, CExprError> where Ctx: ParseContext<'inp, CExprLexer<'inp>>, Ctx::Emitter: Emitter<'inp, CExprLexer<'inp>, Error = CExprError> { Ok(Box::new(Expr::Prefix { op: op.into_data(), operand })) }
fn fold_infix<'inp, Ctx>(_inp: &mut InputRef<'inp, '_, CExprLexer<'inp>, Ctx>, left: Box<Expr>, right: Box<Expr>, op: Precedenced<PrattInfix<BinOp, BinOp, BinOp>, Power>) -> Result<Box<Expr>, CExprError> where Ctx: ParseContext<'inp, CExprLexer<'inp>>, Ctx::Emitter: Emitter<'inp, CExprLexer<'inp>, Error = CExprError> { let (PrattInfix::Left(op) | PrattInfix::Right(op) | PrattInfix::Neither(op)) = op.into_data(); Ok(Box::new(Expr::Binary { op, left, right })) }
fn fold_postfix<'inp, Ctx>(inp: &mut InputRef<'inp, '_, CExprLexer<'inp>, Ctx>, operand: Box<Expr>, op: Precedenced<PostfixOp, Power>) -> Result<Box<Expr>, CExprError> where Ctx: ParseContext<'inp, CExprLexer<'inp>>, Ctx::Emitter: Emitter<'inp, CExprLexer<'inp>, Error = CExprError> {
  match op.into_data() {
    PostfixOp::Inc => Ok(Box::new(Expr::PostfixInc(operand))),
    PostfixOp::Dec => Ok(Box::new(Expr::PostfixDec(operand))),
    PostfixOp::Index => { let index = parse_cexpr(inp)?; if inp.try_expect(|t| matches!(t.data, Token::RBracket))?.is_none() { return Err(CExprError::UnexpectedToken); } Ok(Box::new(Expr::Index { base: operand, index })) }
    PostfixOp::Call => { let mut args: Vec<Expr> = Vec::new(); if inp.try_expect(|t| matches!(t.data, Token::RParen))?.is_some() { return Ok(Box::new(Expr::Call { func: operand, args })); } args.push(*parse_cexpr(inp)?); loop { if inp.try_expect(|t| matches!(t.data, Token::RParen))?.is_some() { break; } if inp.try_expect(|t| matches!(t.data, Token::Comma))?.is_none() { return Err(CExprError::UnexpectedToken); } args.push(*parse_cexpr(inp)?); } Ok(Box::new(Expr::Call { func: operand, args })) }
    PostfixOp::Ternary => { let then = parse_cexpr(inp)?; if inp.try_expect(|t| matches!(t.data, Token::Colon))?.is_none() { return Err(CExprError::UnexpectedToken); } let otherwise = parse_cexpr(inp)?; Ok(Box::new(Expr::Ternary { cond: operand, then, otherwise })) }
  }
}
use tokora::{Emitter, InputRef, Parse, ParseContext, ParseInput, Parser, parser::pratt};

fn parse_cexpr<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CExprLexer<'inp>, Ctx>,
) -> Result<Box<Expr>, CExprError>
where
  Ctx: ParseContext<'inp, CExprLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CExprLexer<'inp>, Error = CExprError>,
{
  pratt(parse_lhs, parse_rhs, fold_prefix, fold_infix, fold_postfix).parse_input(inp)
}

// The maintained binary's assertion table — the parser's behavior contract — now inline. Each row
// parses a C expression and checks its fully-parenthesised `Display`.
let parse = |src| Parser::new().apply(parse_cexpr).parse_str(src).map(|e: Box<Expr>| e.to_string());
assert_eq!(parse("1 + 2 * 3").unwrap(), "(1 + (2 * 3))");
assert_eq!(parse("(1 + 2) * 3").unwrap(), "((1 + 2) * 3)");
assert_eq!(parse("a + b + c").unwrap(), "((a + b) + c)");
assert_eq!(parse("-a").unwrap(), "(-a)");
assert_eq!(parse("!flag").unwrap(), "(!flag)");
assert_eq!(parse("~bits").unwrap(), "(~bits)");
assert_eq!(parse("++x").unwrap(), "(++x)");
assert_eq!(parse("x++").unwrap(), "(x++)");
assert_eq!(parse("x++ + ++y").unwrap(), "((x++) + (++y))");
assert_eq!(parse("a ? b : c").unwrap(), "(a ? b : c)");
assert_eq!(parse("a ? b : c ? d : e").unwrap(), "(a ? b : (c ? d : e))");
assert_eq!(parse("arr[0]").unwrap(), "(arr[0])");
assert_eq!(parse("f()").unwrap(), "f()");
assert_eq!(parse("f(1, 2)").unwrap(), "f(1, 2)");
assert_eq!(parse("a == b && c != d").unwrap(), "((a == b) && (c != d))");
assert_eq!(parse("~bits | flags").unwrap(), "((~bits) | flags)");
assert_eq!(parse("arr[i + 1]").unwrap(), "(arr[(i + 1)])");
assert_eq!(parse("f(a + b, c * d)").unwrap(), "f((a + b), (c * d))");
assert_eq!(parse("x << 2 | y >> 1").unwrap(), "((x << 2) | (y >> 1))");

Reproduce the maintained assertion table

The assertions above are the maintained binary’s assertion table: precedence, grouping, left-associativity, unary operators, prefix and postfix increment and the way the two bind against one infix operator, the ternary and its right-associative nesting, indexing, calls, mixed precedence chains, and C’s low-precedence bitwise and shift operators. They are the behavior contract for the parser. For the full runnable program — the same code driven from a main that prints each parsed expression — run:

cargo run -p tokora --example c_expression --features logos

You have now followed the complete C-expression parser inline: a Logos lexer, a typed Expr AST, C’s precedence ladder, two classifier functions, and three folds — the postfix one consuming its own delimiters for indexing, calls, and the ternary — closed into a one-call parse_cexpr. The Pratt reference catalogs both Pratt surfaces side by side.

The optional chapter 16, Lossless CSTs with Rowan, takes a different route: it records source tokens rather than reducing them to an AST. It requires the rowan feature, so it is named here without a rustdoc link.

16. Lossless CSTs with Rowan

Prerequisites: chapters 1 and 2. Chapter 6 explains the backtracking this chapter gets for free, and chapter 8 the recovery machinery it reuses.

A concrete syntax tree keeps what an AST throws away: whitespace, comments, exact token text, even the garbage inside a syntax error — every byte of the source, in order. Formatters, linters, IDEs, and refactoring tools live on that property. Tokora’s CST support is lossless by configuration: you write one parser assembly, and the emitter you run it with decides whether it also builds a tree.

Two crates share the work:

• tokora parses and records. Committed tokens flow to the emitter on their own (commit_token fires once per settled token, everywhere — you never call it), and node structure is declared with the node) combinators. This half is rowan-free and compiles in every build.
• rowan stores the finished tree. Under the rowan feature, parse_lossless drives the parse with a Sink emitter minted from the same source, buffering it as a flat event stream, and finish materializes the returned Cst once into a rowan green tree. The source is named once, to the driver: the buffer the tree’s text comes from is the buffer the parse read, by construction rather than by convention.

The event stream between the two is an implementation detail: you never construct, inspect, or replay events. (The cst::event module documents the vocabulary and its laws normatively, for the curious.) What matters is where the events live — in the emitter’s rewindable channel. The same checkpoint/rewind mark that unwinds diagnostics unwinds tree events, so attempt, the Transaction guards, and pratt rollback rewind the tree for free.

If you read this chapter before 0.2: it taught a manual builder walkthrough — a recording shim around every consume, a builder parameter threaded through every parser function — and ended by warning that input rollback “cannot roll back external Rowan builder state”. That caveat is now the headline feature (tree recording participates in the one rollback contract), and the manual threading is simply gone: no parser signature changes when a tree is wanted.

Enable Rowan

Rowan is an optional dependency and tokora does not re-export it, so a tree-building crate names both:

[dependencies]
tokora = { version = "0.10", features = ["logos", "rowan"] }
rowan = "0.17"

The rowan feature implies std (rowan itself requires it); it does not imply logos. Only the materializing half — Sink and the typed tree views — lives behind the feature. The recording half (CstEmitter, the node) combinators, the marks) is unconditional, which is what lets a grammar crate stay rowan-free while its tooling consumers opt in.

One enum owns the kind space

Rowan trees are dynamically typed: every node and token carries a raw u16 kind, and the dialect gives those numbers meaning through a [rowan::Language] implementation. The convention that keeps the numbering sane is: one enum, one space — node kinds and token kinds live in the same #[repr(u16)] enum, declared in the dialect crate. Lexer tokens enter the tree only as images under a mapper function you hand the sink, never as raw lexer discriminants, so a collision between a token kind and a node kind is unrepresentable rather than merely checked.

This chapter builds Query, a GraphQL-shaped slice: selection sets, fields with optional aliases, and integer arguments. Its lossless lexer (hidden below, a logos derive like chapter 1’s — just without a skip rule, so whitespace, comments, and commas are real tokens with is_trivia returning true) produces Tok. Because the lexer surfaces every byte, its Tok declares const SURFACES_TRIVIA = true, and the lossless Sink refuses at compile time to wrap a trivia-skipping lexer — a skipped-whitespace gap is indistinguishable from a dropped committed token. The unified kind space maps it like this:

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Logos)]
#[logos(crate = logos, error = LexError)]
enum Tok {
  #[regex(r"[ \t\r\n]+")] Whitespace,
  #[regex(r"#[^\r\n]*", allow_greedy = true)] Comment,
  #[token(",")] Comma,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[regex(r"-?[0-9]+")] Int,
  #[token("{")] LBrace,
  #[token("}")] RBrace,
  #[token("(")] LParen,
  #[token(")")] RParen,
  #[token(":")] Colon,
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Tok::Whitespace => "whitespace", Tok::Comment => "comment", Tok::Comma => "`,`",
      Tok::Ident => "identifier", Tok::Int => "integer", Tok::LBrace => "`{`",
      Tok::RBrace => "`}`", Tok::LParen => "`(`", Tok::RParen => "`)`", Tok::Colon => "`:`",
    })
  }
}
impl TokenT<'_> for Tok {
  type Kind = Tok;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  const SURFACES_TRIVIA: bool = true;
  fn kind(&self) -> Tok { *self }
  fn is_trivia(&self) -> bool { matches!(self, Tok::Whitespace | Tok::Comment | Tok::Comma) }
}
use rowan::Language;
use tokora::cst::{CstProfile, KindValidator};

/// The dialect's whole u16 space: token images first, node kinds after, plus the three
/// bookkeeping kinds. One enum means one place to look and no way to collide. (One value
/// is reserved crate-wide: `u16::MAX`, the tombstone — never map anything to it.)
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
#[repr(u16)]
enum SyntaxKind {
  // Token images — committed tokens enter the tree only through `map_token` below.
  Whitespace, Comment, Comma, Ident, Int, LBrace, RBrace, LParen, RParen, Colon,
  // Node kinds — the grammar's shapes, declared by the `node()` calls you will meet next.
  SelectionSet, Field, Alias, Arguments, Argument,
  // Bookkeeping: recovery holes, materialization gap tiles, and the synthetic root.
  Error, Gap, Root,
}
type K = SyntaxKind;

impl SyntaxKind {
  /// The raw value the event channel speaks.
  const fn raw(self) -> u16 {
    self as u16
  }
}

/// The sink-side mapper: one compiler-exhaustive match from lexer token to unified kind.
/// Add a token variant and this match — the whole cost of keeping the spaces aligned —
/// fails to compile until you place its image.
fn map_token(tok: &Tok) -> u16 {
  (match tok {
    Tok::Whitespace => K::Whitespace, Tok::Comment => K::Comment, Tok::Comma => K::Comma,
    Tok::Ident => K::Ident, Tok::Int => K::Int, Tok::LBrace => K::LBrace,
    Tok::RBrace => K::RBrace, Tok::LParen => K::LParen, Tok::RParen => K::RParen,
    Tok::Colon => K::Colon,
  }) as u16
}

/// The dialect's CST profile: the mapper, the predicate that says which raw u16s this
/// language can name, and the two bookkeeping kinds. Stated once, reused at every
/// construction.
fn query_profile() -> CstProfile<Tok> {
  CstProfile::new(
    map_token,
    KindValidator::new(|kind| kind <= K::Root.raw()),
    K::Error.raw(),
    K::Gap.raw(),
  )
}

/// Rowan's side of the bargain: raw ↔ typed kind conversion.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
enum QueryLang {}

impl Language for QueryLang {
  type Kind = SyntaxKind;

  fn kind_from_raw(raw: rowan::SyntaxKind) -> SyntaxKind {
    // `#[repr(u16)]` with default discriminants: the raw value is the declaration index.
    const KINDS: [SyntaxKind; 18] = [
      K::Whitespace, K::Comment, K::Comma, K::Ident, K::Int, K::LBrace, K::RBrace,
      K::LParen, K::RParen, K::Colon, K::SelectionSet, K::Field, K::Alias, K::Arguments,
      K::Argument, K::Error, K::Gap, K::Root,
    ];
    KINDS[raw.0 as usize]
  }

  fn kind_to_raw(kind: SyntaxKind) -> rowan::SyntaxKind {
    rowan::SyntaxKind(kind as u16)
  }
}

assert_eq!(map_token(&Tok::Colon), SyntaxKind::Colon.raw());
assert_eq!(
  QueryLang::kind_from_raw(rowan::SyntaxKind(SyntaxKind::Field.raw())),
  SyntaxKind::Field,
);

That is the entire dialect setup. The sink-facing part is smaller still: the mapper plus two kind choices at construction — error_kind (what wraps a recovery hole’s skipped tokens) and gap_kind (what tiles source bytes no committed token covered). Everything else is rowan’s ordinary price, paid once per dialect.

A note for real languages: keep contextual keywords out of the token images. GraphQL’s query lexes as an identifier and should map to Ident — let the typed layer classify by text. Baking nineteen *Kw kinds into the image space forces the mapper to re-classify identifiers on the hot path for no structural gain.

The grammar declares the tree

Here is the heart of the chapter. node(kind, parser)) wraps a parser so that, on success, everything the sub-parse committed — tokens, trivia, nested nodes — becomes the children of one syntax node of that kind. Structure is declared exactly where the grammar already is; nothing else about the parser changes. Compare these functions with chapter 2’s: the signatures are identical except for one bound — CstEmitter where chapter 2 wrote Emitter — and the bound appears only on functions that declare tree structure. Helpers that merely consume keep the plain emitter bound.

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Logos)]
#[logos(crate = logos, error = LexError)]
enum Tok {
  #[regex(r"[ \t\r\n]+")] Whitespace,
  #[regex(r"#[^\r\n]*", allow_greedy = true)] Comment,
  #[token(",")] Comma,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[regex(r"-?[0-9]+")] Int,
  #[token("{")] LBrace,
  #[token("}")] RBrace,
  #[token("(")] LParen,
  #[token(")")] RParen,
  #[token(":")] Colon,
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Tok::Whitespace => "whitespace", Tok::Comment => "comment", Tok::Comma => "`,`",
      Tok::Ident => "identifier", Tok::Int => "integer", Tok::LBrace => "`{`",
      Tok::RBrace => "`}`", Tok::LParen => "`(`", Tok::RParen => "`)`", Tok::Colon => "`:`",
    })
  }
}
impl TokenT<'_> for Tok {
  type Kind = Tok;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  const SURFACES_TRIVIA: bool = true;
  fn kind(&self) -> Tok { *self }
  fn is_trivia(&self) -> bool { matches!(self, Tok::Whitespace | Tok::Comment | Tok::Comma) }
}
type QueryLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
#[derive(Debug, Clone, PartialEq)]
enum QueryError { Lex, Unexpected }
impl From<LexError> for QueryError { fn from(_: LexError) -> Self { QueryError::Lex } }
impl<'a, T, Kd: Clone, S, Lang: ?Sized> From<tokora::error::token::UnexpectedToken<'a, T, Kd, S, Lang>> for QueryError {
  fn from(_: tokora::error::token::UnexpectedToken<'a, T, Kd, S, Lang>) -> Self { QueryError::Unexpected }
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
#[repr(u16)]
enum SyntaxKind {
  Whitespace, Comment, Comma, Ident, Int, LBrace, RBrace, LParen, RParen, Colon,
  SelectionSet, Field, Alias, Arguments, Argument,
  Error, Gap, Root,
}
type K = SyntaxKind;
impl SyntaxKind {
  const fn raw(self) -> u16 { self as u16 }
}
fn map_token(tok: &Tok) -> u16 {
  (match tok {
    Tok::Whitespace => K::Whitespace, Tok::Comment => K::Comment, Tok::Comma => K::Comma,
    Tok::Ident => K::Ident, Tok::Int => K::Int, Tok::LBrace => K::LBrace,
    Tok::RBrace => K::RBrace, Tok::LParen => K::LParen, Tok::RParen => K::RParen,
    Tok::Colon => K::Colon,
  }) as u16
}
fn query_profile() -> CstProfile<Tok> {
  CstProfile::new(
    map_token,
    KindValidator::new(|kind| kind <= K::Root.raw()),
    K::Error.raw(),
    K::Gap.raw(),
  )
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
enum QueryLang {}
impl rowan::Language for QueryLang {
  type Kind = SyntaxKind;
  fn kind_from_raw(raw: rowan::SyntaxKind) -> SyntaxKind {
    const KINDS: [SyntaxKind; 18] = [
      K::Whitespace, K::Comment, K::Comma, K::Ident, K::Int, K::LBrace, K::RBrace,
      K::LParen, K::RParen, K::Colon, K::SelectionSet, K::Field, K::Alias, K::Arguments,
      K::Argument, K::Error, K::Gap, K::Root,
    ];
    KINDS[raw.0 as usize]
  }
  fn kind_to_raw(kind: SyntaxKind) -> rowan::SyntaxKind { rowan::SyntaxKind(kind as u16) }
}
use tokora::{
  Emitter, InputRef, Parse, ParseContext, ParseInput, Parser, TryParseInput,
  cache::DefaultCache,
  cst::{CstProfile, KindValidator, parse_lossless},
  emitter::{CstEmitter, Fatal},
  parser::{node, node_at},
  try_parse_input::ParseAttempt,
};

/// Chapter shorthand for the input reference.
type QueryIn<'inp, 'x, Ctx> = InputRef<'inp, 'x, QueryLexer<'inp>, Ctx>;

/// The typed result. The AST does not go away when a tree is wanted — the tree is a side
/// effect of consuming, and the parser still returns whatever it returned before.
#[derive(Debug, Clone, PartialEq)]
struct Field {
  alias: Option<String>,
  name: String,
  args: usize,
  children: Vec<Field>,
}

/// Commits any leading trivia, then reports the next token's kind without consuming it
/// (`None` at end of input). Committing trivia during a peek is safe over a lossless
/// stream: trivia belongs to the parse — and to the tree — no matter which branch wins.
fn sig_peek<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<Option<Tok>, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  inp.skip_while(|t| t.is_trivia())?;
  let mut ahead = None;
  inp.try_expect(|t| {
    ahead = Some(t.data().kind());
    false
  })?;
  Ok(ahead)
}

fn expect_tok<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>, want: Tok) -> Result<(), QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  inp.skip_while(|t| t.is_trivia())?;
  match inp.try_expect(|t| t.data().kind() == want)? {
    Some(_) => Ok(()),
    None => Err(QueryError::Unexpected),
  }
}
fn ident<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<String, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  inp.skip_while(|t| t.is_trivia())?;
  match inp.try_expect(|t| matches!(t.data().kind(), Tok::Ident))? {
    Some(_) => Ok(inp.slice().to_string()),
    None => Err(QueryError::Unexpected),
  }
}
fn try_colon<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<ParseAttempt<()>, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  inp.skip_while(|t| t.is_trivia())?;
  Ok(match inp.try_expect(|t| matches!(t.data().kind(), Tok::Colon))? {
    Some(_) => ParseAttempt::Accept(()),
    None => ParseAttempt::Decline,
  })
}
// (Hidden: `expect_tok` and `ident` — chapter 2's committed one-token parsers, with a
//  leading trivia skip; and `try_colon`, a declining attempt at a `:`.)

/// `selection_set := "{" field* "}"` — one `node()` bracket over the whole shape: the
/// braces, the trivia, and every child selection land inside the `SelectionSet` node.
fn selection_set<'inp, Ctx>(
  inp: &mut QueryIn<'inp, '_, Ctx>,
) -> Result<Vec<Field>, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: CstEmitter<'inp, QueryLexer<'inp>>
    + Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  node(K::SelectionSet.raw(), |inp: &mut QueryIn<'inp, '_, Ctx>| {
    expect_tok(inp, Tok::LBrace)?;
    let mut fields = Vec::new();
    loop {
      match sig_peek(inp)? {
        Some(Tok::Ident) => fields.push(field(inp)?),
        Some(Tok::RBrace) => {
          expect_tok(inp, Tok::RBrace)?;
          return Ok(fields);
        }
        _ => return Err(QueryError::Unexpected),
      }
    }
  })
  .parse_input(inp)
}

fn field<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<Field, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: CstEmitter<'inp, QueryLexer<'inp>>
    + Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  node(K::Field.raw(), |inp: &mut QueryIn<'inp, '_, Ctx>| {
    let mark = inp.cst_mark();
    let first = ident(inp)?;
    let (alias, name) = match node_at(mark, K::Alias.raw(), try_colon).try_parse_input(inp)? {
      ParseAttempt::Accept(()) => (Some(first), ident(inp)?),
      _ => (None, first),
    };
    let args = opt_arguments(inp)?;
    let children = match sig_peek(inp)? {
      Some(Tok::LBrace) => selection_set(inp)?,
      _ => Vec::new(),
    };
    Ok(Field { alias, name, args, children })
  })
  .parse_input(inp)
}
// (Hidden: `field` — the next section builds it around the alias ambiguity.)

/// `arguments := "(" argument* ")"`, or nothing at all. Dispatch by PEEK, then let the
/// bracketed parser consume the `(` — so the parenthesis lands *inside* the `Arguments`
/// node. And when there are no arguments, no node is ever opened: an absent optional
/// shape must not leave an empty node behind.
fn opt_arguments<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<usize, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: CstEmitter<'inp, QueryLexer<'inp>>
    + Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  match sig_peek(inp)? {
    Some(Tok::LParen) => node(K::Arguments.raw(), |inp: &mut QueryIn<'inp, '_, Ctx>| {
      expect_tok(inp, Tok::LParen)?;
      let mut count = 0;
      loop {
        match sig_peek(inp)? {
          Some(Tok::Ident) => {
            argument(inp)?;
            count += 1;
          }
          Some(Tok::RParen) => {
            expect_tok(inp, Tok::RParen)?;
            return Ok(count);
          }
          _ => return Err(QueryError::Unexpected),
        }
      }
    })
    .parse_input(inp),
    _ => Ok(0),
  }
}

/// `argument := ident ":" int` — `Argument[Ident, Colon, Int]`, plus whatever trivia was
/// consumed along the way.
fn argument<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<(), QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: CstEmitter<'inp, QueryLexer<'inp>>
    + Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  node(K::Argument.raw(), |inp: &mut QueryIn<'inp, '_, Ctx>| {
    ident(inp)?;
    expect_tok(inp, Tok::Colon)?;
    expect_tok(inp, Tok::Int)
  })
  .parse_input(inp)
}

let src = "{ user(id: 4) { name } }";

// `parse_lossless` mints the sink FROM `src`: the buffer the parse reads and the buffer
// the tree's text is sliced out of are the same argument of the same call, so they cannot
// disagree. It takes the ordinary emitter to forward to (fail-fast `Fatal` here) and the
// dialect corner — the mapper and the two bookkeeping kinds — and hands back the spent
// handle, because materialization happens after the parse. The `()` is the lexer's `State`:
// `LogosLexer` inherits logos' `Extras`, which this dialect leaves empty.
let (cst, parsed) = parse_lossless(
  src,
  (),
  Fatal::<QueryError>::new(),
  query_profile(),
  DefaultCache::<QueryLexer<'_>>::default(),
  selection_set,
);
let fields = parsed.unwrap();

// The typed result, exactly as if no tree existed:
assert_eq!(fields.len(), 1);
let user = &fields[0];
assert_eq!(user.alias, None);
assert_eq!(user.name, "user");
assert_eq!(user.args, 1);
assert_eq!(user.children.len(), 1);
assert_eq!(user.children[0].name, "name");

// Materialize once. The handle is consumed; the inner emitter comes back with the tree,
// so collected diagnostics (chapter 7) survive materialization.
let (green, _emitter) = cst.finish(K::Root.raw());
let tree = rowan::SyntaxNode::<QueryLang>::new_root(green.unwrap());

// The round-trip law — the reason to build a CST at all:
assert_eq!(tree.text().to_string(), src);

// And the structure is the grammar's:
//
//   Root
//   └─ SelectionSet
//      ├─ "{"  " "
//      ├─ Field
//      │  ├─ Ident "user"
//      │  ├─ Arguments ["(", Argument [Ident "id", ":", " ", Int "4"], ")"]
//      │  ├─ " "
//      │  └─ SelectionSet ["{", " ", Field [Ident "name", " "], "}"]
//      ├─ " "
//      └─ "}"
let sel = tree.first_child().unwrap();
assert_eq!(sel.kind(), SyntaxKind::SelectionSet);
let user_node = sel.first_child().unwrap();
assert_eq!(user_node.kind(), SyntaxKind::Field);
assert_eq!(
  user_node.children().map(|n| n.kind()).collect::<Vec<_>>(),
  [SyntaxKind::Arguments, SyntaxKind::SelectionSet],
);
assert_eq!(user_node.first_child().unwrap().text().to_string(), "(id: 4)");

The bracket contract

node() is a bracket, and its exits are total:

• Success wraps precisely the region committed since entry.
• A decline (the inner parser is a try_ parser that declined) records no node — not even an empty one. opt_arguments above leans on this; node_opt) packages the same shape as an Option.
• An error-path unwind (? out of the inner parser) records no node and leaves no dangling half-open bracket: materialization stays balanced, whatever already committed stays in the tree, and gap tiling (below) keeps the round trip.

There is no “finish the node on every path” duty anywhere in the grammar — the bracket is append-only under the hood (an inert mark at entry, spent only on success), which is why no exit can leave the tree in a wrong state.

node_at: wrap what you already parsed

Some shapes are only knowable in hindsight. A GraphQL field may open with an alias — author: user — but when the parser reads the first identifier it cannot know whether that identifier is the field’s name or an alias: only a following : decides. Rewriting the grammar to lookahead twice would contort it; wrapping too eagerly would put a wrong node in the tree.

node_at) is the retro-wrap: take a mark before the first identifier, parse it, and spend the mark only when the colon shows up — the new node wraps everything recorded since the mark, including tokens committed before the wrap was conceivable.

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Logos)]
#[logos(crate = logos, error = LexError)]
enum Tok {
  #[regex(r"[ \t\r\n]+")] Whitespace,
  #[regex(r"#[^\r\n]*", allow_greedy = true)] Comment,
  #[token(",")] Comma,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[regex(r"-?[0-9]+")] Int,
  #[token("{")] LBrace,
  #[token("}")] RBrace,
  #[token("(")] LParen,
  #[token(")")] RParen,
  #[token(":")] Colon,
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Tok::Whitespace => "whitespace", Tok::Comment => "comment", Tok::Comma => "`,`",
      Tok::Ident => "identifier", Tok::Int => "integer", Tok::LBrace => "`{`",
      Tok::RBrace => "`}`", Tok::LParen => "`(`", Tok::RParen => "`)`", Tok::Colon => "`:`",
    })
  }
}
impl TokenT<'_> for Tok {
  type Kind = Tok;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  const SURFACES_TRIVIA: bool = true;
  fn kind(&self) -> Tok { *self }
  fn is_trivia(&self) -> bool { matches!(self, Tok::Whitespace | Tok::Comment | Tok::Comma) }
}
type QueryLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
#[derive(Debug, Clone, PartialEq)]
enum QueryError { Lex, Unexpected }
impl From<LexError> for QueryError { fn from(_: LexError) -> Self { QueryError::Lex } }
impl<'a, T, Kd: Clone, S, Lang: ?Sized> From<tokora::error::token::UnexpectedToken<'a, T, Kd, S, Lang>> for QueryError {
  fn from(_: tokora::error::token::UnexpectedToken<'a, T, Kd, S, Lang>) -> Self { QueryError::Unexpected }
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
#[repr(u16)]
enum SyntaxKind {
  Whitespace, Comment, Comma, Ident, Int, LBrace, RBrace, LParen, RParen, Colon,
  SelectionSet, Field, Alias, Arguments, Argument,
  Error, Gap, Root,
}
type K = SyntaxKind;
impl SyntaxKind {
  const fn raw(self) -> u16 { self as u16 }
}
fn map_token(tok: &Tok) -> u16 {
  (match tok {
    Tok::Whitespace => K::Whitespace, Tok::Comment => K::Comment, Tok::Comma => K::Comma,
    Tok::Ident => K::Ident, Tok::Int => K::Int, Tok::LBrace => K::LBrace,
    Tok::RBrace => K::RBrace, Tok::LParen => K::LParen, Tok::RParen => K::RParen,
    Tok::Colon => K::Colon,
  }) as u16
}
fn query_profile() -> CstProfile<Tok> {
  CstProfile::new(
    map_token,
    KindValidator::new(|kind| kind <= K::Root.raw()),
    K::Error.raw(),
    K::Gap.raw(),
  )
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
enum QueryLang {}
impl rowan::Language for QueryLang {
  type Kind = SyntaxKind;
  fn kind_from_raw(raw: rowan::SyntaxKind) -> SyntaxKind {
    const KINDS: [SyntaxKind; 18] = [
      K::Whitespace, K::Comment, K::Comma, K::Ident, K::Int, K::LBrace, K::RBrace,
      K::LParen, K::RParen, K::Colon, K::SelectionSet, K::Field, K::Alias, K::Arguments,
      K::Argument, K::Error, K::Gap, K::Root,
    ];
    KINDS[raw.0 as usize]
  }
  fn kind_to_raw(kind: SyntaxKind) -> rowan::SyntaxKind { rowan::SyntaxKind(kind as u16) }
}
use tokora::{
  Emitter, InputRef, Parse, ParseContext, ParseInput, Parser, TryParseInput,
  cache::DefaultCache,
  cst::{CstProfile, KindValidator, parse_lossless},
  emitter::{CstEmitter, Fatal},
  parser::{node, node_at},
  try_parse_input::ParseAttempt,
};
type QueryIn<'inp, 'x, Ctx> = InputRef<'inp, 'x, QueryLexer<'inp>, Ctx>;
#[allow(dead_code)]
#[derive(Debug, Clone, PartialEq)]
struct Field {
  alias: Option<String>,
  name: String,
  args: usize,
  children: Vec<Field>,
}
fn sig_peek<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<Option<Tok>, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  inp.skip_while(|t| t.is_trivia())?;
  let mut ahead = None;
  inp.try_expect(|t| {
    ahead = Some(t.data().kind());
    false
  })?;
  Ok(ahead)
}
fn expect_tok<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>, want: Tok) -> Result<(), QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  inp.skip_while(|t| t.is_trivia())?;
  match inp.try_expect(|t| t.data().kind() == want)? {
    Some(_) => Ok(()),
    None => Err(QueryError::Unexpected),
  }
}
fn ident<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<String, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  inp.skip_while(|t| t.is_trivia())?;
  match inp.try_expect(|t| matches!(t.data().kind(), Tok::Ident))? {
    Some(_) => Ok(inp.slice().to_string()),
    None => Err(QueryError::Unexpected),
  }
}
fn try_colon<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<ParseAttempt<()>, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  inp.skip_while(|t| t.is_trivia())?;
  Ok(match inp.try_expect(|t| matches!(t.data().kind(), Tok::Colon))? {
    Some(_) => ParseAttempt::Accept(()),
    None => ParseAttempt::Decline,
  })
}
fn selection_set<'inp, Ctx>(
  inp: &mut QueryIn<'inp, '_, Ctx>,
) -> Result<Vec<Field>, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: CstEmitter<'inp, QueryLexer<'inp>>
    + Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  node(K::SelectionSet.raw(), |inp: &mut QueryIn<'inp, '_, Ctx>| {
    expect_tok(inp, Tok::LBrace)?;
    let mut fields = Vec::new();
    loop {
      match sig_peek(inp)? {
        Some(Tok::Ident) => fields.push(field(inp)?),
        Some(Tok::RBrace) => {
          expect_tok(inp, Tok::RBrace)?;
          return Ok(fields);
        }
        _ => return Err(QueryError::Unexpected),
      }
    }
  })
  .parse_input(inp)
}
fn opt_arguments<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<usize, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: CstEmitter<'inp, QueryLexer<'inp>>
    + Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  match sig_peek(inp)? {
    Some(Tok::LParen) => node(K::Arguments.raw(), |inp: &mut QueryIn<'inp, '_, Ctx>| {
      expect_tok(inp, Tok::LParen)?;
      let mut count = 0;
      loop {
        match sig_peek(inp)? {
          Some(Tok::Ident) => {
            argument(inp)?;
            count += 1;
          }
          Some(Tok::RParen) => {
            expect_tok(inp, Tok::RParen)?;
            return Ok(count);
          }
          _ => return Err(QueryError::Unexpected),
        }
      }
    })
    .parse_input(inp),
    _ => Ok(0),
  }
}
fn argument<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<(), QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: CstEmitter<'inp, QueryLexer<'inp>>
    + Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  node(K::Argument.raw(), |inp: &mut QueryIn<'inp, '_, Ctx>| {
    ident(inp)?;
    expect_tok(inp, Tok::Colon)?;
    expect_tok(inp, Tok::Int)
  })
  .parse_input(inp)
}
/// `field := (ident ":")? ident arguments? selection_set?`
fn field<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<Field, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: CstEmitter<'inp, QueryLexer<'inp>>
    + Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  node(K::Field.raw(), |inp: &mut QueryIn<'inp, '_, Ctx>| {
    // An inert mark: costs one buffer slot, promises nothing.
    let mark = inp.cst_mark();
    let first = ident(inp)?;
    let (alias, name) = match node_at(mark, K::Alias.raw(), try_colon).try_parse_input(inp)? {
      // The colon was there — `first` was an alias all along. `node_at` spent the mark:
      // the tree now holds `Alias[Ident, Colon]` wrapped around the identifier that was
      // parsed BEFORE the wrap was known. The real name follows.
      ParseAttempt::Accept(()) => (Some(first), ident(inp)?),
      // No colon: the attempt declined and the mark was left unspent. An unspent mark
      // materializes into nothing — `first` was the name, and no `Alias` node exists.
      _ => (None, first),
    };
    let args = opt_arguments(inp)?;
    let children = match sig_peek(inp)? {
      Some(Tok::LBrace) => selection_set(inp)?,
      _ => Vec::new(),
    };
    Ok(Field { alias, name, args, children })
  })
  .parse_input(inp)
}

let src = "{ author: user(id: 4) { name } }";
let (cst, parsed) = parse_lossless(
  src,
  (),
  Fatal::<QueryError>::new(),
  query_profile(),
  DefaultCache::<QueryLexer<'_>>::default(),
  selection_set,
);
let fields = parsed.unwrap();

assert_eq!(fields[0].alias.as_deref(), Some("author"));
assert_eq!(fields[0].name, "user");

let (green, _emitter) = cst.finish(K::Root.raw());
let tree = rowan::SyntaxNode::<QueryLang>::new_root(green.unwrap());
assert_eq!(tree.text().to_string(), src);

// The retro-wrap in the finished tree: Field's first child is the Alias node, spanning
// the identifier and the colon that revealed it.
let field_node = tree.first_child().unwrap().first_child().unwrap();
assert_eq!(field_node.kind(), SyntaxKind::Field);
let alias_node = field_node.first_child().unwrap();
assert_eq!(alias_node.kind(), SyntaxKind::Alias);
assert_eq!(alias_node.text().to_string(), "author:");

Two safety properties keep caller-held marks honest. A mark whose branch was rolled back is stale, and spending it panics in every build — the alternative would be silently wrapping whatever the retry parsed over the same buffer positions, a wrong tree nothing downstream can detect. And for the common single-wrap decision tree, the Marker typestate makes double-spends and wrap-before-complete compile errors rather than conventions.

Tokens reach the tree on their own

Notice what the grammar above never does: it never records a token. There is no builder.token(...), no recording wrapper around next, no per-atom plumbing. Every committed token — consumed by try_expect, drained from the lookahead cache, or settled by a scan like skip_while — flows to the emitter at the moment it settles, through one crate-internal chokepoint. Peeks, declines, and rolled-back speculation record nothing, because nothing was committed.

That is why trivia handling costs zero code: the trivia skips sprinkled through the helpers (skip_while(|t| t.is_trivia()), or the padded combinator, which does the same) commit the trivia tokens they cross, so the whitespace lands in the tree even though no grammar rule mentions it. A trivia token materializes into whichever node was open where it committed (the TriviaPolicy::AsEmitted placement — deterministic, and exactly where the consuming code stood). Capturing trivia wrappers that collect Vecs of trivia per node remain useful for consumers that want formatting data without a tree in the dependency closure; under a sink they are redundant.

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Logos)]
#[logos(crate = logos, error = LexError)]
enum Tok {
  #[regex(r"[ \t\r\n]+")] Whitespace,
  #[regex(r"#[^\r\n]*", allow_greedy = true)] Comment,
  #[token(",")] Comma,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[regex(r"-?[0-9]+")] Int,
  #[token("{")] LBrace,
  #[token("}")] RBrace,
  #[token("(")] LParen,
  #[token(")")] RParen,
  #[token(":")] Colon,
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Tok::Whitespace => "whitespace", Tok::Comment => "comment", Tok::Comma => "`,`",
      Tok::Ident => "identifier", Tok::Int => "integer", Tok::LBrace => "`{`",
      Tok::RBrace => "`}`", Tok::LParen => "`(`", Tok::RParen => "`)`", Tok::Colon => "`:`",
    })
  }
}
impl TokenT<'_> for Tok {
  type Kind = Tok;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  const SURFACES_TRIVIA: bool = true;
  fn kind(&self) -> Tok { *self }
  fn is_trivia(&self) -> bool { matches!(self, Tok::Whitespace | Tok::Comment | Tok::Comma) }
}
type QueryLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
#[derive(Debug, Clone, PartialEq)]
enum QueryError { Lex, Unexpected }
impl From<LexError> for QueryError { fn from(_: LexError) -> Self { QueryError::Lex } }
impl<'a, T, Kd: Clone, S, Lang: ?Sized> From<tokora::error::token::UnexpectedToken<'a, T, Kd, S, Lang>> for QueryError {
  fn from(_: tokora::error::token::UnexpectedToken<'a, T, Kd, S, Lang>) -> Self { QueryError::Unexpected }
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
#[repr(u16)]
enum SyntaxKind {
  Whitespace, Comment, Comma, Ident, Int, LBrace, RBrace, LParen, RParen, Colon,
  SelectionSet, Field, Alias, Arguments, Argument,
  Error, Gap, Root,
}
type K = SyntaxKind;
impl SyntaxKind {
  const fn raw(self) -> u16 { self as u16 }
}
fn map_token(tok: &Tok) -> u16 {
  (match tok {
    Tok::Whitespace => K::Whitespace, Tok::Comment => K::Comment, Tok::Comma => K::Comma,
    Tok::Ident => K::Ident, Tok::Int => K::Int, Tok::LBrace => K::LBrace,
    Tok::RBrace => K::RBrace, Tok::LParen => K::LParen, Tok::RParen => K::RParen,
    Tok::Colon => K::Colon,
  }) as u16
}
fn query_profile() -> CstProfile<Tok> {
  CstProfile::new(
    map_token,
    KindValidator::new(|kind| kind <= K::Root.raw()),
    K::Error.raw(),
    K::Gap.raw(),
  )
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
enum QueryLang {}
impl rowan::Language for QueryLang {
  type Kind = SyntaxKind;
  fn kind_from_raw(raw: rowan::SyntaxKind) -> SyntaxKind {
    const KINDS: [SyntaxKind; 18] = [
      K::Whitespace, K::Comment, K::Comma, K::Ident, K::Int, K::LBrace, K::RBrace,
      K::LParen, K::RParen, K::Colon, K::SelectionSet, K::Field, K::Alias, K::Arguments,
      K::Argument, K::Error, K::Gap, K::Root,
    ];
    KINDS[raw.0 as usize]
  }
  fn kind_to_raw(kind: SyntaxKind) -> rowan::SyntaxKind { rowan::SyntaxKind(kind as u16) }
}
use tokora::{
  Emitter, InputRef, Parse, ParseContext, ParseInput, Parser, TryParseInput,
  cache::DefaultCache,
  cst::{CstProfile, KindValidator, parse_lossless},
  emitter::{CstEmitter, Fatal},
  parser::{node, node_at},
  try_parse_input::ParseAttempt,
};
type QueryIn<'inp, 'x, Ctx> = InputRef<'inp, 'x, QueryLexer<'inp>, Ctx>;
#[allow(dead_code)]
#[derive(Debug, Clone, PartialEq)]
struct Field {
  alias: Option<String>,
  name: String,
  args: usize,
  children: Vec<Field>,
}
fn sig_peek<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<Option<Tok>, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  inp.skip_while(|t| t.is_trivia())?;
  let mut ahead = None;
  inp.try_expect(|t| {
    ahead = Some(t.data().kind());
    false
  })?;
  Ok(ahead)
}
fn expect_tok<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>, want: Tok) -> Result<(), QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  inp.skip_while(|t| t.is_trivia())?;
  match inp.try_expect(|t| t.data().kind() == want)? {
    Some(_) => Ok(()),
    None => Err(QueryError::Unexpected),
  }
}
fn ident<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<String, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  inp.skip_while(|t| t.is_trivia())?;
  match inp.try_expect(|t| matches!(t.data().kind(), Tok::Ident))? {
    Some(_) => Ok(inp.slice().to_string()),
    None => Err(QueryError::Unexpected),
  }
}
fn try_colon<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<ParseAttempt<()>, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  inp.skip_while(|t| t.is_trivia())?;
  Ok(match inp.try_expect(|t| matches!(t.data().kind(), Tok::Colon))? {
    Some(_) => ParseAttempt::Accept(()),
    None => ParseAttempt::Decline,
  })
}
fn selection_set<'inp, Ctx>(
  inp: &mut QueryIn<'inp, '_, Ctx>,
) -> Result<Vec<Field>, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: CstEmitter<'inp, QueryLexer<'inp>>
    + Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  node(K::SelectionSet.raw(), |inp: &mut QueryIn<'inp, '_, Ctx>| {
    expect_tok(inp, Tok::LBrace)?;
    let mut fields = Vec::new();
    loop {
      match sig_peek(inp)? {
        Some(Tok::Ident) => fields.push(field(inp)?),
        Some(Tok::RBrace) => {
          expect_tok(inp, Tok::RBrace)?;
          return Ok(fields);
        }
        _ => return Err(QueryError::Unexpected),
      }
    }
  })
  .parse_input(inp)
}
fn field<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<Field, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: CstEmitter<'inp, QueryLexer<'inp>>
    + Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  node(K::Field.raw(), |inp: &mut QueryIn<'inp, '_, Ctx>| {
    let mark = inp.cst_mark();
    let first = ident(inp)?;
    let (alias, name) = match node_at(mark, K::Alias.raw(), try_colon).try_parse_input(inp)? {
      ParseAttempt::Accept(()) => (Some(first), ident(inp)?),
      _ => (None, first),
    };
    let args = opt_arguments(inp)?;
    let children = match sig_peek(inp)? {
      Some(Tok::LBrace) => selection_set(inp)?,
      _ => Vec::new(),
    };
    Ok(Field { alias, name, args, children })
  })
  .parse_input(inp)
}
fn opt_arguments<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<usize, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: CstEmitter<'inp, QueryLexer<'inp>>
    + Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  match sig_peek(inp)? {
    Some(Tok::LParen) => node(K::Arguments.raw(), |inp: &mut QueryIn<'inp, '_, Ctx>| {
      expect_tok(inp, Tok::LParen)?;
      let mut count = 0;
      loop {
        match sig_peek(inp)? {
          Some(Tok::Ident) => {
            argument(inp)?;
            count += 1;
          }
          Some(Tok::RParen) => {
            expect_tok(inp, Tok::RParen)?;
            return Ok(count);
          }
          _ => return Err(QueryError::Unexpected),
        }
      }
    })
    .parse_input(inp),
    _ => Ok(0),
  }
}
fn argument<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<(), QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: CstEmitter<'inp, QueryLexer<'inp>>
    + Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  node(K::Argument.raw(), |inp: &mut QueryIn<'inp, '_, Ctx>| {
    ident(inp)?;
    expect_tok(inp, Tok::Colon)?;
    expect_tok(inp, Tok::Int)
  })
  .parse_input(inp)
}
// Comments, newlines, commas: no grammar rule mentions them, all of them survive.
let src = "{ # every byte survives\n  a, b }";
let (cst, parsed) = parse_lossless(
  src,
  (),
  Fatal::<QueryError>::new(),
  query_profile(),
  DefaultCache::<QueryLexer<'_>>::default(),
  selection_set,
);
parsed.unwrap();
let (green, _emitter) = cst.finish(K::Root.raw());
let tree = rowan::SyntaxNode::<QueryLang>::new_root(green.unwrap());
assert_eq!(tree.text().to_string(), src);

let tokens: Vec<_> = tree
  .descendants_with_tokens()
  .filter_map(|el| el.into_token())
  .map(|t| (t.kind(), t.text().to_string()))
  .collect();
assert!(tokens.contains(&(SyntaxKind::Comment, "# every byte survives".to_string())));
assert!(tokens.contains(&(SyntaxKind::Comma, ",".to_string())));
// Nothing was gap-tiled: every byte was covered by a real committed token.
assert!(tokens.iter().all(|(kind, _)| *kind != SyntaxKind::Gap));

// And when bytes are NOT covered — here `%` is no token of the language, the lexer
// reports it and fail-fast `Fatal` aborts the parse before the tail is even lexed — that
// tail is un-diagnosed. Strict `finish` refuses it (an unexplained gap is, to `finish`,
// indistinguishable from a dropped token); the tooling door `finish_partial` tiles it, so
// an aborted parse still round-trips its text.
let src = "{ a % b }";
let (cst, res) = parse_lossless(
  src,
  (),
  Fatal::<QueryError>::new(),
  query_profile(),
  DefaultCache::<QueryLexer<'_>>::default(),
  selection_set,
);
assert_eq!(res, Err(QueryError::Lex));

let (green, _emitter) = cst.finish_partial(K::Root.raw());
let tree = rowan::SyntaxNode::<QueryLang>::new_root(green.unwrap());
assert_eq!(tree.text().to_string(), src, "aborted parse, intact text");
assert!(
  tree
    .descendants_with_tokens()
    .filter_map(|el| el.into_token())
    .any(|t| t.kind() == SyntaxKind::Gap),
  "the unparsed region is a gap token, not a hole in the text"
);

One assembly, two configurations

The grammar functions above bound their emitter as CstEmitter — and every diagnostics-only emitter the crate ships (Fatal, Verbose, Silent, and Ignored) already implements it, through defaulted no-op event methods. So the same functions run in a plain fail-fast context with no sink anywhere in sight — no rowan feature, no tree, and no cost: the no-op event calls take references, inline to empty bodies, and compile to nothing.

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Logos)]
#[logos(crate = logos, error = LexError)]
enum Tok {
  #[regex(r"[ \t\r\n]+")] Whitespace,
  #[regex(r"#[^\r\n]*", allow_greedy = true)] Comment,
  #[token(",")] Comma,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[regex(r"-?[0-9]+")] Int,
  #[token("{")] LBrace,
  #[token("}")] RBrace,
  #[token("(")] LParen,
  #[token(")")] RParen,
  #[token(":")] Colon,
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Tok::Whitespace => "whitespace", Tok::Comment => "comment", Tok::Comma => "`,`",
      Tok::Ident => "identifier", Tok::Int => "integer", Tok::LBrace => "`{`",
      Tok::RBrace => "`}`", Tok::LParen => "`(`", Tok::RParen => "`)`", Tok::Colon => "`:`",
    })
  }
}
impl TokenT<'_> for Tok {
  type Kind = Tok;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  const SURFACES_TRIVIA: bool = true;
  fn kind(&self) -> Tok { *self }
  fn is_trivia(&self) -> bool { matches!(self, Tok::Whitespace | Tok::Comment | Tok::Comma) }
}
type QueryLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
#[derive(Debug, Clone, PartialEq)]
enum QueryError { Lex, Unexpected }
impl From<LexError> for QueryError { fn from(_: LexError) -> Self { QueryError::Lex } }
impl<'a, T, Kd: Clone, S, Lang: ?Sized> From<tokora::error::token::UnexpectedToken<'a, T, Kd, S, Lang>> for QueryError {
  fn from(_: tokora::error::token::UnexpectedToken<'a, T, Kd, S, Lang>) -> Self { QueryError::Unexpected }
}
impl<O, Lang: ?Sized, Set: Clone + 'static> From<tokora::error::UnexpectedEot<O, Lang, Set>> for QueryError { fn from(_: tokora::error::UnexpectedEot<O, Lang, Set>) -> Self { QueryError::Unexpected } }
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for QueryError { fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { QueryError::Unexpected } }
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
#[repr(u16)]
enum SyntaxKind {
  Whitespace, Comment, Comma, Ident, Int, LBrace, RBrace, LParen, RParen, Colon,
  SelectionSet, Field, Alias, Arguments, Argument,
  Error, Gap, Root,
}
type K = SyntaxKind;
impl SyntaxKind {
  const fn raw(self) -> u16 { self as u16 }
}
use tokora::{
  Emitter, InputRef, ParseContext, ParseInput, TryParseInput,
  emitter::CstEmitter,
  parser::{node, node_at},
  try_parse_input::ParseAttempt,
};
type QueryIn<'inp, 'x, Ctx> = InputRef<'inp, 'x, QueryLexer<'inp>, Ctx>;
#[derive(Debug, Clone, PartialEq)]
struct Field {
  alias: Option<String>,
  name: String,
  args: usize,
  children: Vec<Field>,
}
fn sig_peek<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<Option<Tok>, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  inp.skip_while(|t| t.is_trivia())?;
  let mut ahead = None;
  inp.try_expect(|t| {
    ahead = Some(t.data().kind());
    false
  })?;
  Ok(ahead)
}
fn expect_tok<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>, want: Tok) -> Result<(), QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  inp.skip_while(|t| t.is_trivia())?;
  match inp.try_expect(|t| t.data().kind() == want)? {
    Some(_) => Ok(()),
    None => Err(QueryError::Unexpected),
  }
}
fn ident<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<String, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  inp.skip_while(|t| t.is_trivia())?;
  match inp.try_expect(|t| matches!(t.data().kind(), Tok::Ident))? {
    Some(_) => Ok(inp.slice().to_string()),
    None => Err(QueryError::Unexpected),
  }
}
fn try_colon<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<ParseAttempt<()>, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  inp.skip_while(|t| t.is_trivia())?;
  Ok(match inp.try_expect(|t| matches!(t.data().kind(), Tok::Colon))? {
    Some(_) => ParseAttempt::Accept(()),
    None => ParseAttempt::Decline,
  })
}
fn selection_set<'inp, Ctx>(
  inp: &mut QueryIn<'inp, '_, Ctx>,
) -> Result<Vec<Field>, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: CstEmitter<'inp, QueryLexer<'inp>>
    + Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  node(K::SelectionSet.raw(), |inp: &mut QueryIn<'inp, '_, Ctx>| {
    expect_tok(inp, Tok::LBrace)?;
    let mut fields = Vec::new();
    loop {
      match sig_peek(inp)? {
        Some(Tok::Ident) => fields.push(field(inp)?),
        Some(Tok::RBrace) => {
          expect_tok(inp, Tok::RBrace)?;
          return Ok(fields);
        }
        _ => return Err(QueryError::Unexpected),
      }
    }
  })
  .parse_input(inp)
}
fn field<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<Field, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: CstEmitter<'inp, QueryLexer<'inp>>
    + Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  node(K::Field.raw(), |inp: &mut QueryIn<'inp, '_, Ctx>| {
    let mark = inp.cst_mark();
    let first = ident(inp)?;
    let (alias, name) = match node_at(mark, K::Alias.raw(), try_colon).try_parse_input(inp)? {
      ParseAttempt::Accept(()) => (Some(first), ident(inp)?),
      _ => (None, first),
    };
    let args = opt_arguments(inp)?;
    let children = match sig_peek(inp)? {
      Some(Tok::LBrace) => selection_set(inp)?,
      _ => Vec::new(),
    };
    Ok(Field { alias, name, args, children })
  })
  .parse_input(inp)
}
fn opt_arguments<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<usize, QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: CstEmitter<'inp, QueryLexer<'inp>>
    + Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  match sig_peek(inp)? {
    Some(Tok::LParen) => node(K::Arguments.raw(), |inp: &mut QueryIn<'inp, '_, Ctx>| {
      expect_tok(inp, Tok::LParen)?;
      let mut count = 0;
      loop {
        match sig_peek(inp)? {
          Some(Tok::Ident) => {
            argument(inp)?;
            count += 1;
          }
          Some(Tok::RParen) => {
            expect_tok(inp, Tok::RParen)?;
            return Ok(count);
          }
          _ => return Err(QueryError::Unexpected),
        }
      }
    })
    .parse_input(inp),
    _ => Ok(0),
  }
}
fn argument<'inp, Ctx>(inp: &mut QueryIn<'inp, '_, Ctx>) -> Result<(), QueryError>
where
  Ctx: ParseContext<'inp, QueryLexer<'inp>>,
  Ctx::Emitter: CstEmitter<'inp, QueryLexer<'inp>>
    + Emitter<'inp, QueryLexer<'inp>, Error = QueryError>,
{
  node(K::Argument.raw(), |inp: &mut QueryIn<'inp, '_, Ctx>| {
    ident(inp)?;
    expect_tok(inp, Tok::Colon)?;
    expect_tok(inp, Tok::Int)
  })
  .parse_input(inp)
}
use tokora::{Parse, Parser};

// The exact same `selection_set` — chapter 2's default fail-fast context, no sink,
// no tree. This code needs no `rowan` feature to compile.
let fields = Parser::new()
  .apply(selection_set)
  .parse_str("{ user(id: 4) { name } }")
  .unwrap();
assert_eq!(fields[0].name, "user");
assert_eq!(fields[0].children[0].name, "name");