10. Testing

Calc works. But how do you know, and — the harder question — how would you find out if it stopped?

Testing a parser’s grammar is ordinary work: feed it strings, check the trees. What is not ordinary is the layer underneath. tokora’s input machinery does things to your lexer that no hand-written driver would: it truncates the token cache after a rollback, it re-lexes a rewound region on demand, it resumes a lexer from a saved State at an arbitrary offset. Every one of those moves is sound only because the Lexer contract holds. If your lexer quietly violates it — a token whose identity depends on lookahead past its own span, a with_state + bump resume that does not reproduce the suffix — nothing fails loudly. Instead your parser is subtly wrong only after a backtrack, which is the worst bug in this entire crate to find by hand.

So do not find it by hand.

The conformance kit

The conformance module (feature conformance) ships a Harness that drives your lexer against the contract and panics, with the input index, position, operation and expected-vs-got, at the first violation. Build it over a corpus with new or over, then call run. It checks:

replay identity — two fresh runs produce the identical token/span/slice sequence, and identical means by value: the whole token, the whole error payload, never a rendering of either and never the token’s kind standing in for it;
state-resume faithfulness — at every position, saving the state and resuming there reproduces the rest of the run. This is the prefix-replay assumption, verbatim;
monotone progress — spans advance, none is empty, and the run terminates;
sticky exhaustion — once lex returns None, it keeps doing so;
span/slice coherence — every slice equals the source over its span;
gap-free tiling — opt-in via lossless, for a lexer that emits trivia as tokens rather than skipping it. Calc’s skips whitespace, so Calc does not ask for this one.

On top of the trait tier it drives a real Input session through fixed, named save/peek/drain/restore schedules and requires the committed token stream to equal the straight-lex stream — no randomness, the schedules are enumerated — and run_partial adds chapter 9’s tier: for every split point of every input, a non-final drain of the prefix must yield a prefix of the items before the cut and end incomplete, while a final drain of the whole source must reproduce the complete parse exactly. An item is a committed token or a lexer error the input layer raised — a refusal is a decision about bytes exactly as a token is, and appending can turn one into the other. That is the check that catches a lexer which is unfaithful under truncation — and truncation is exactly what a stream does to you.

It asks for a prefix rather than equality because withholding more is always sound, and a lexer that reports a read frontier past its own spans does exactly that. Calc’s lexer is logos-backed and declares SCAN_LOOKAHEAD as Unbounded — the const has no default, so that is a written choice — and this run therefore withholds every item until the input is final. The run still passes, and it still catches an item that changes — a token whose kind, span or value moved, a token that became an error, an error whose payload moved. See Lexer::read_frontier for what declaring the class buys back.

Comparing the value is why the kit asks for PartialEq on your token and on its error type. It is one derive on a data type, and it is what makes the comparison total: every field participates, including the one you add next year. The alternative, a comparison key written by hand, is a projection, and the field it forgets is exactly the field that drifts. Calc’s Tok and LexError already derive it. Both entry points ask for it: run compared an error’s Debug rendering and a token’s kind until 0.10.0, so its replay-identity green meant less than the word said — two unequal error values that print alike passed, and a payload rendering a counter reddened a conforming lexer. A vocabulary whose token or error genuinely cannot be PartialEq loses the kit and recovers it with a hand-written impl or a newtype.

One thing PartialEq does not promise is reflexivity, and a payload holding an f64 that can be NaN is not equal to itself. Eq and Ord do promise it — but they are markers, and nothing checks the promise, so a Kind, a span, an offset or a source slice can break it too. Either way it is your obligation, not the kit’s — hand-write the impl that says what equality means for such a type — and either way the kit will not misreport it: a comparison decided by a value that will not equal itself is refused, tagged non-reflexive-payload and naming the component, instead of being reported as a conformance failure of your lexer.

The kit refuses only when it has nothing better to say. A difference it can convict on — one at the discriminant, one in item count, or one between two values that each equal themselves — outranks a comparison it cannot use, wherever both are available: two runs yielding [Tok(NaN)] and [Tok(NaN), Tok(0.0)] are reported as a length mismatch, because the extra item proves the divergence whatever NaN does. The refusal is the answer of last resort, and it means the kit searched and found nothing it could stand behind.

use tokora::{Token as TokenT, logos::{self, Logos}};
#[derive(Clone, Debug, Default, PartialEq)]
struct LexError;
impl From<()> for LexError { fn from(_: ()) -> Self { LexError } }
#[derive(Debug, Clone, PartialEq, Logos)]
#[logos(crate = logos, skip r"[ \t\r\n]+", error = LexError)]
enum Tok {
  #[regex(r"[0-9]+", |lex| lex.slice().parse::<i64>().map_err(|_| LexError))]
  Int(i64),
  #[token("let")] Let,
  #[token("print")] Print,
  #[regex(r"[A-Za-z_][A-Za-z0-9_]*")] Ident,
  #[token("+")] Plus,
  #[token("-")] Minus,
  #[token("*")] Star,
  #[token("/")] Slash,
  #[token("^")] Caret,
  #[token("=")] Assign,
  #[token(";")] Semi,
  #[token(",")] Comma,
  #[token("(")] LParen,
  #[token(")")] RParen,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
enum TokKind { Int, Let, Print, Ident, Plus, Minus, Star, Slash, Caret, Assign, Semi, Comma, LParen, RParen }
impl core::fmt::Display for TokKind {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    f.write_str(match self {
      Self::Int => "integer", Self::Let => "`let`", Self::Print => "`print`",
      Self::Ident => "identifier", Self::Plus => "`+`", Self::Minus => "`-`",
      Self::Star => "`*`", Self::Slash => "`/`", Self::Caret => "`^`",
      Self::Assign => "`=`", Self::Semi => "`;`", Self::Comma => "`,`",
      Self::LParen => "`(`", Self::RParen => "`)`",
    })
  }
}
impl core::fmt::Display for Tok {
  fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
    match self {
      Tok::Int(n) => write!(f, "{n}"),
      other => core::fmt::Display::fmt(&other.kind(), f),
    }
  }
}
impl TokenT<'_> for Tok {
  type Kind = TokKind;
  type Error = LexError;
  const SCAN_LOOKAHEAD: tokora::ScanLookahead = tokora::ScanLookahead::Unbounded;
  fn kind(&self) -> TokKind {
    match self {
      Tok::Int(_) => TokKind::Int, Tok::Let => TokKind::Let, Tok::Print => TokKind::Print,
      Tok::Ident => TokKind::Ident, Tok::Plus => TokKind::Plus, Tok::Minus => TokKind::Minus,
      Tok::Star => TokKind::Star, Tok::Slash => TokKind::Slash, Tok::Caret => TokKind::Caret,
      Tok::Assign => TokKind::Assign, Tok::Semi => TokKind::Semi, Tok::Comma => TokKind::Comma,
      Tok::LParen => TokKind::LParen, Tok::RParen => TokKind::RParen,
    }
  }
  fn is_trivia(&self) -> bool { false }
}
type CalcLexer<'a> = tokora::lexer::LogosLexer<'a, Tok>;
use tokora::error::{UnexpectedEot, token::UnexpectedToken};
#[derive(Debug, Clone, PartialEq)]
enum CalcError { Lex, Unexpected, UnexpectedEnd }
impl From<LexError> for CalcError { fn from(_: LexError) -> Self { CalcError::Lex } }
impl<'a, T, K: Clone, S, Lang: ?Sized> From<UnexpectedToken<'a, T, K, S, Lang>> for CalcError {
  fn from(_: UnexpectedToken<'a, T, K, S, Lang>) -> Self { CalcError::Unexpected }
}
impl<O, Lang: ?Sized, Set: Clone + 'static> From<UnexpectedEot<O, Lang, Set>> for CalcError {
  fn from(_: UnexpectedEot<O, Lang, Set>) -> Self { CalcError::UnexpectedEnd }
}
impl<'inp, L: tokora::Lexer<'inp>, Lang: ?Sized> tokora::emitter::FromUnclosed<'inp, L, Lang> for CalcError {
  fn from_unclosed<D>(_: tokora::error::Unclosed<D, L::Span, Lang>) -> Self { CalcError::UnexpectedEnd }
}
use tokora::{
  Emitter, InputRef, Parse, ParseContext, ParseInput, Parser,
  conformance::Harness,
  traced,
};

// A corpus: the shapes Calc actually sees, plus the empty source, which is where
// off-by-one bugs live.
const CORPUS: [&str; 5] = [
  "",
  "let x = 1 ;",
  "print 1 , 2 ;",
  "( 1 + 2 ) * 3 ^ 4 ;",
  "let ab = 12 ; print ab ;",
];

// The contract, checked. Note the absence of `.lossless()`: Calc's lexer *skips* whitespace,
// so its spans legitimately leave gaps. Ask for gap-free tiling only from a lossless lexer.
Harness::<CalcLexer<'_>>::over(CORPUS).run();

// And the streaming tier: every split point of every input, chunked-equivalence checked.
Harness::<CalcLexer<'_>>::over(CORPUS).run_partial();

// ── Debugging a parse you do not understand ──
fn parse_stmt<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<i64, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  if inp.try_expect(|t| matches!(t.data(), Tok::Print))?.is_none() {
    return Err(CalcError::Unexpected);
  }
  let value = match inp.next()? {
    Some(tok) => match tok.into_data() {
      Tok::Int(n) => n,
      _ => return Err(CalcError::Unexpected),
    },
    None => return Err(CalcError::UnexpectedEnd),
  };
  if inp.try_expect(|t| matches!(t.data(), Tok::Semi))?.is_none() {
    return Err(CalcError::Unexpected);
  }
  Ok(value)
}
// (Hidden: `parse_stmt`, a `print <int> ;` parser in chapter 2's style.)

/// Wrapping a parser in [`traced`] prints an indented `enter` / `exit` transcript to stderr
/// as it runs — including the crate's own instrumented combinators, and including the
/// backtracks, which is usually the part you could not see.
fn traced_stmt<'inp, Ctx>(
  inp: &mut InputRef<'inp, '_, CalcLexer<'inp>, Ctx>,
) -> Result<i64, CalcError>
where
  Ctx: ParseContext<'inp, CalcLexer<'inp>>,
  Ctx::Emitter: Emitter<'inp, CalcLexer<'inp>, Error = CalcError>,
{
  traced("statement", parse_stmt).parse_input(inp)
}

// With the `trace` feature *off*, `traced(name, p)` is literally `p` — no wrapper type, no
// branch, nothing to strip before shipping. The line above can stay where it is.
assert_eq!(
  Parser::new().apply(traced_stmt).parse_str("print 7 ;"),
  Ok(7)
);

Fuzzing the machinery, not just the grammar

The conformance kit checks your lexer against the contract. The fuzz module (feature fuzz) checks the layer above it: a deterministic operation-script fuzzer that drives the input and backtracking machinery — consume, peek, the sync family, attempt, the transaction guards, stacked savepoints, session points, partial-mode chunking — against a scriptable synthetic lexer, and verifies the documented laws after every operation. run_case runs one Case; run_seeds sweeps a range of them. They are ordinary #[test]s on stable Rust — no nightly, no external fuzzer — so the arbitrary operation orders your grammar will eventually produce get exercised long before your users produce them.

A testing ladder for your language

Conformance your lexer, once, over a corpus of real inputs — including the empty one, the one-token one, and the one that ends mid-token. If you use a LogosLexer this is already true, and the check is cheap insurance against the day you replace LogosLexer with a custom lexer.
Golden-test the grammar: source in, AST out. Ordinary table tests.
Golden-test the diagnostics under Verbose — chapter 7’s diagnostics() view is a stable, orderable thing to snapshot. A recovery regression shows up here as a changed hole, and nowhere else.
Reach for traced the moment a parse surprises you, and delete nothing afterwards — it costs nothing with the feature off.

Where to go next

Calc is finished: it lexes, parses, dispatches, folds expressions by precedence, speculates and rolls back, reports many diagnostics at once, recovers without cascading, and streams. The Calc fundamentals are complete.

The four programs in examples/ — json, calculator, s_expression, and c_expression — are canonical complete programs in the same style, each leaning on a different corner of the crate. Next: chapter 11 uses them to show how a real parser comes together. Read InputRef for the primitives, the parser module for the combinator catalogue, and the emitter module when you decide what your diagnostics should do. When something behaves in a way the documentation did not predict, that is a bug in one of them — the guide’s doctests exist so that it is never quietly the guide.

Keyboard shortcuts

The Tokora Guide

10. Testing

The conformance kit

Fuzzing the machinery, not just the grammar

A testing ladder for your language

Where to go next